Multi-dimensional voice emotion analysis method and system

Abstract

The invention discloses a multi-dimensional voice emotion analysis method and system, and belongs to the technical field of computers. The method comprises the following steps: S1, acquiring audio data of a conversation between a to-be-detected person and a user; s2, converting the audio data into a structured text file; s3, analyzing the text file, and judging whether the audio information segment in the text file hits a preset model rule or not; s4, when the audio information segment in the text file accords with a preset model rule, marking an emotion mark represented by the model rule on the corresponding audio data; the model rule is divided into a text rule and a voice rule. The voice emotion is effectively analyzed from multiple dimensions of keywords, speed, volume, silence, interruption and the like, and the sharing index is more comprehensive; and the text rule comprises a plurality of logic rules, so that the comprehensiveness and accuracy of emotion analysis can be improved by effectively combining the text rule and the voice rule.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and specifically relates to a multi-dimensional speech emotion analysis method and system. Background Art

[0002] With the development of society and the advancement of science and technology, people have a richer material life and are increasingly pursuing high-quality services. For service personnel, one of the factors that affect service quality is their own emotions. The following example takes the seat staff who receive customers as an example.

[0003] Agents understand customers' problems through conversations and solve them. Customers encountered during work hours are of all kinds. The psychological and physical pressure accumulated over time is difficult to relieve, which easily leads to negative emotions. This may lead to reduced work efficiency, poor service quality, poor communication with users, and affect customers' emotions. In the existing technology, it is difficult to accurately and objectively monitor the standardized conversation process between agents and customers, and it is impossible to effectively warn of abnormal conversation processes caused by negative emotions. The analysis of emotional information contained in the conversation between agents and customers is not accurate enough.

[0004] After searching, the Chinese invention patent: A method and system for intelligent emotion determination (publication number: CN106548788A, publication date: 2020.01.07) provides an intelligent emotion determination method and system, which obtains audio information of a conversation between a person to be detected and a user; determines an abnormal emotion audio information segment from each audio information segment constituting the audio information, wherein the abnormal emotion audio information segment is an audio information segment containing preset audio information for characterizing the abnormal emotion of the person to be detected that meets the corresponding preset conditions; when the abnormal emotion audio information segment can be determined, it is determined that the person to be detected corresponding to the audio information segment has abnormal emotions. It is more objective and the analysis of whether the person to be detected has abnormal emotions is more accurate. However, the shortcomings of this application are that the analysis indicators of the voice text are not comprehensive enough, and there is a lack of recognition and consideration of the key behavior of interruption. Summary of the invention

[0005] 1. Technical problem to be solved by the invention

[0006] The purpose of the present invention is to solve the problem that the existing speech emotion analysis is not comprehensive and accurate enough.

[0007] 2. Technical solution

[0008] In order to achieve the above object, the technical solution provided by the present invention is:

[0009] A multi-dimensional speech emotion analysis method of the present invention comprises the following steps:

[0010] S1. Obtaining audio data of the conversation between the person to be detected and the user;

[0011] S2, converting audio data into structured text files;

[0012] S3, parsing the text file to determine whether the audio information segment in the text file hits the preset model rule;

[0013] S4, when the audio information segment in the text file meets the preset model rule, the corresponding audio data is marked with the emotion tag represented by the model rule;

[0014] The model rules are divided into text rules and voice rules.

[0015] Preferably, the text rules and the voice rules are specifically:

[0016] Text rules: Combine keywords through logical symbols, and then match them with the text of the audio information segment to filter out the desired audio information segment under the corresponding logical text. There are five logical operation symbols, namely "&, |,!,#,()";

[0017] Among them, multiple text rules can form a complete speech process through logical operation symbols. There are five logical operation symbols, namely "&, |,!,@,()";

[0018] The speech rules include at least one of the following objects: the conversation contains a preset keyword and meets its attributes, the mute information in the conversation meets the attributes of the preset mute rule, the speech speed information in the conversation meets the attributes of the preset speech speed rule, the volume information in the conversation meets the attributes of the preset volume rule, and the interruption information in the conversation meets the attributes of the preset interruption rule.

[0019] Preferably, the step S2 further includes

[0020] Convert the audio data into a structured audio text file, where the audio text file should include: audio text of the agent channel and the customer channel, the number of sentences, the number of text words, the channel, start time, end time, speaking speed and volume corresponding to each sentence, the average speaking speed, average volume, number of sentences and text words of the entire audio, and the start and end time of silence.

[0021] Preferably, the step S3 also includes

[0022] Determine, from the audio text file constituting the audio information, an audio information segment corresponding to the text that conforms to the text rule; when a plurality of preset text rules are combined according to logical operation symbols to form a rule combination, determine whether the plurality of audio segments that conform to the text rule conform to the complete text rule combination;

[0023] Determining an audio information segment that meets the voice rule from the audio information segments constituting the audio information specifically includes:

[0024] Determining an audio information segment containing a preset keyword rule from the audio information segments;

[0025] Determine, from the audio information segments, an audio information segment whose speech rate information meets a preset speech rate rule;

[0026] Determining, from the audio information segments, an audio information segment whose volume information complies with a preset volume rule;

[0027] Determining, from the audio information segments, an audio information segment whose silence information complies with a preset silence rule;

[0028] An audio information segment whose interruption information complies with a preset interruption rule is determined from the audio information segments.

[0029] Preferably, the step S3 also includes

[0030] Determine whether the audio text segment corresponding to the audio information segment in the audio text file conforms to the preset text rule information, wherein the text rule includes keywords and logical symbols, and the logical symbols specifically include:

[0031] #: means "Near", a primary connector, a connector between simple objects. The object connected by "#" is a complex object. There can be one or more "#" in a complex object, but the number of text characters between the first and last simple objects is no more than 10 by default. At the same time, the object before "#" must appear before the object after "#". The simple object refers to a single keyword or word; the complex object refers to an object connected by multiple keywords or words through "#";

[0032] (): means "combination", which means combining one or several simple or complex objects into a desired object;

[0033] &: means "and", a secondary connector, which means that the simple or complex objects connected by "&" must appear at the same time in the recording text. There is no limit to the order in which the connected keywords appear. It can also connect a combination of simple or complex objects;

[0034] |: means "or", a secondary connector, which means that at least one of the simple or complex objects connected by "|" appears in the recording text. It can also connect a combination of simple or complex objects;

[0035] !: means "non", a secondary connector, which means that the simple or complex object connected by "!" is not allowed to appear in the recording text. It can also connect a combination of simple or complex objects;

[0036] When it is determined that the audio text corresponding to the audio information segment hits a certain preset text rule, the start time and end time corresponding to the audio information segment that hits the text rule are recorded.

[0037] Preferably, the step S3 also includes

[0038] After determining multiple audio information segments that meet the text rules, it is determined whether the multiple audio information segments meet the logical combination of multiple text rules to form a complete speech process, wherein the logical combination of the text rules includes:

[0039] @: means "sequence", which means that the order of simple or complex text rules connected by "@" in the entire audio text must be consistent with the order of text rule objects before and after "@". It can also connect a combination of simple or complex voice rules;

[0040] (): means "combination", which means combining one or several simple or complex text rules into an object you want;

[0041] &: means "and", a secondary connector, which means that the simple or complex text rules connected by "&" must appear at the same time in the entire audio information text. The order in which the connected text rules appear is not limited, and it can also connect a combination of simple or complex text rules;

[0042] |: means "or", a secondary connector, which means that at least one of the simple or complex text rules connected by "|" appears in the entire audio information text. It can also connect a combination of simple or complex text rules;

[0043] !: means "non", a secondary connector, which means that the simple or complex text rules connected with "!" are not allowed to appear in the entire audio information text. It can also connect a combination of simple or complex text rules.

[0044] Preferably, determining the preset voice rule information that each audio text segment corresponding to each of the audio information segments in the audio text file complies with specifically includes:

[0045] When the preset voice rule includes keyword information, for each audio information segment in each audio information segment, the audio text segment corresponding to the audio information segment is compared with the preset keyword; the preset keyword included in the audio text segment, and the start time and end time of the appearance of the preset keyword are determined;

[0046] When the preset voice rule information includes speech rate rule information, for each audio information segment in each audio information segment, according to the speech rate characteristics of each sentence of the two parties in the audio text segment corresponding to the audio information segment; determining the speech rate characteristics that meet the preset rules and the start time and end time of the speech rate characteristics contained in the audio text segment;

[0047] When the preset voice rule information includes volume rule information, for each audio information segment in each audio information segment, according to the volume characteristics of each sentence of the two parties in the audio text segment corresponding to the audio information segment; determine the volume characteristics that meet the preset rules and the start time and end time of the volume characteristics contained in the audio text segment;

[0048] When the preset voice rule information includes silence rule information, for each audio information segment in each audio information segment, according to the start time, end time, and duration of the audio corresponding to each sentence of the two parties in the audio text segment corresponding to the audio information segment, determine the silence feature contained in the audio corresponding to the person to be detected in the audio text segment; determine the silence feature contained in the audio text segment that meets the preset rule, as well as the start time and end time of the occurrence of the contained silence feature;

[0049] When the preset voice rule information includes interruption rule information, for each audio information segment in each audio information segment, the interruption feature contained in the audio corresponding to the person to be detected in the audio text segment is determined according to the start time, end time, and duration of the audio corresponding to each sentence of the two parties in the audio text segment corresponding to the audio information segment; the interruption feature contained in the audio text segment that meets the preset rules is determined, as well as the start time and end time of the interruption feature.

[0050] Preferably, when the preset voice rule is a keyword rule, the following attributes may be added to the keyword rule, including:

[0051] Absolute position, i.e., the specific time position of the keyword in the entire audio information; relative position, for two recording rules, i.e., the relative position of the keyword rule with respect to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker of the most recent speech after the keyword appears, whether it is the customer channel or the agent channel; previous channel, i.e., the speaker of the most recent speech before the keyword appears, whether it is the customer channel or the agent channel; channel, i.e., the speaker identity of the keyword, whether the keyword is spoken by the customer or the agent; number, i.e., the number of times the keyword appears in the entire audio information; rule, i.e., the text rule of the keyword itself;

[0052] When the preset voice rule is a speech rate rule, the following attributes may be added to the speech rate rule, including:

[0053] Absolute position, i.e., the specific time position where the speech rate rule appears in the entire audio information; relative position, for two recording rules, i.e., the relative relationship of the speech rate rule to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker who spoke the most recent content after the speech rate rule appears, whether it is the customer channel or the agent channel; previous channel, i.e., the channel of the speaker who spoke the most recent content before the speech rate rule appears, whether it is the customer channel or the agent channel; average, i.e., the average speech speed of the entire audio information; channel, indicating whether the speech rate rule acts on the customer or the agent channel; number of times, i.e., the number of times the audio information segment that meets the speech rate rule appears in the entire audio information; single sentence, i.e., the number of times a single sentence that meets the speech rate rule appears in the entire audio information;

[0054] When the preset voice rule is a volume rule, the following attributes may be added to the volume rule, including:

[0055] Absolute position, i.e., the specific time position where the volume rule appears in the entire audio information; relative position, for two recording rules, i.e., the relative position of the volume rule with respect to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker who spoke the most recent content after the volume rule appears, whether it is the customer channel or the agent channel; previous channel, i.e., the channel of the speaker who spoke the most recent content before the volume rule appears, whether it is the customer channel or the agent channel; channel, indicating whether the volume rule acts on the customer channel or the agent channel; number of times, i.e., the number of times the audio information segment that meets the volume rule appears in the entire audio information; volume, i.e., the volume size of the specified volume rule;

[0056] When the preset voice rule is a mute rule, the following attributes may be added to the mute rule, including:

[0057] Absolute position, i.e. the specific time position of silence in the whole audio information; relative position, for two recording rules, i.e. the relative position of silence rule with respect to another voice rule in the whole audio information; subsequent channel, i.e. the channel of the speaker who spoke the most recent content after the silence ended, whether it was the customer channel or the agent channel; previous channel, i.e. the channel of the speaker who spoke the most recent content before the silence started, whether it was the customer channel or the agent channel; number of times, i.e. the number of times the audio information segment that meets the silence rule appears in the whole audio information; duration, i.e. the duration of the specified silence;

[0058] When the preset voice rule is an interruption rule, the following attributes may be added to the interruption rule, including:

[0059] The overlapping duration refers to the duration of the overlap of the audio of the two channels after the interruption occurs; the number of times refers to the number of times the audio information segment that meets the interruption rule appears in the entire audio information; the subsequent channel refers to the channel of the speaker who spoke the most recent content after the interruption, whether it is the customer channel or the agent channel; the previous channel refers to the channel of the speaker who spoke the most recent content before the interruption began, whether it is the customer channel or the agent channel; the absolute position refers to the specific time position of the interruption in the entire audio information; the relative position refers to the relative position of the interruption rule relative to the other voice rule in the entire audio information for the two recording rules.

[0060] Preferably, the specific process of step S3 includes:

[0061] Process S301, the structured text information transcribed by the audio transcription module enters the audio analysis module;

[0062] Enter process S302 to determine whether text rules are configured in the model rules;

[0063] When no text rules are configured in S302, the process proceeds to process S303 to determine whether a voice rule is configured in the model rule;

[0064] When the voice rules are not configured in S303, the process proceeds to process S305, where it is considered that no model rules are currently configured, and the structured text file transcribed from the audio transcription module cannot be parsed, and the process ends;

[0065] When the voice rules are configured in S303, the process enters S306 to parse the audio and text information according to the configured voice rules; wherein the voice rules include the following attributes: preset keyword rules, the silence duration meets the preset duration, the speech speed information of both parties of the call reaches the corresponding indicator threshold, the volume information reaches the corresponding indicator threshold, and the interruption rule meets the preset conditions;

[0066] Entering process S308, determining whether the voice rule is matched;

[0067] When S308 does not match the voice rule, the process enters S310, and it is considered that the audio does not match the model rule, and the process ends;

[0068] When S308 matches the voice rule, the process enters S319 and considers that the audio meets the preset model rule;

[0069] When the text rules are configured in S302, the process proceeds to process S304, where the text is parsed according to the keywords and logical symbols configured in the text rules;

[0070] Enter process S307 to determine whether the text rule is matched;

[0071] When no text rule is matched in S307, the process proceeds to S313 to determine whether a voice rule is configured;

[0072] When S307 matches the text rule, the process goes to S309 to determine whether a rule combination of multiple text rules is configured;

[0073] When the rule combination of multiple text rules is not configured in S309, the process proceeds to process S313 to determine whether a voice rule is configured;

[0074] When a rule combination of multiple text rules is configured in S309, the process proceeds to process S311, and the text information is parsed according to the rule combination composed of the configured multiple text rules and logical operators;

[0075] Entering process S312, determining whether a rule combination of multiple text rules is matched;

[0076] When S312 does not match the rule combination of multiple text rules, the process proceeds to process S313 to determine whether a voice rule is configured;

[0077] When S312 matches a rule combination of multiple text rules, the process goes to process S313 to determine whether a voice rule is configured;

[0078] Enter process S313 to determine whether voice rules are configured;

[0079] When no voice rule is configured in S313, the process proceeds to S314 to check whether a text rule or rule combination is hit;

[0080] When S314 does not match the text rule or rule combination, the process proceeds to S316, where it is considered that the audio does not match the model rule, and the process ends;

[0081] When S314 matches a text rule or a rule combination, the process proceeds to S319, where it is considered that the audio meets the preset model rule;

[0082] When the voice rules are configured in S313, the process enters S315 to parse the audio and text information according to the configured voice rules; wherein the voice rules include the following attributes: preset keyword rules, the silence duration meets the preset duration, the speech speed information of both parties of the call reaches the corresponding indicator threshold, the volume information reaches the corresponding indicator threshold, and the interruption rule meets the preset conditions;

[0083] Enter process S317 to determine whether the voice rule is matched;

[0084] When S317 matches the voice rule, the process enters S319 and considers that the audio meets the preset model rule;

[0085] When the voice rule is not hit in S317, the process goes to S318 to check whether the text rule or rule combination is hit;

[0086] When S318 does not match the text rule or rule combination, the process proceeds to S320, where it is considered that the audio does not match the model rule, and the process ends;

[0087] When S318 matches a text rule or a rule combination, the process proceeds to S319, where it is considered that the audio meets the preset model rule;

[0088] Entering process S319, confirming that the audio meets the preset model rules, where the model rules include text rules and voice rules;

[0089] Entering process S321, after confirming that the audio meets the preset model rules, the audio is marked with the emotion tag corresponding to the model rules, and it is considered that the audio meets the emotion corresponding to the model rules, and the process ends.

[0090] A multi-dimensional speech emotion analysis system, the system is used to perform the above method, the system comprises

[0091] An audio collection module is used to obtain audio information of the conversation between the person to be detected and the user;

[0092] The audio transcription module is used to convert the audio obtained by the above audio collection module into a structured audio text file, where the audio text file should include: audio text of the agent channel and the customer channel, the number of sentences, the number of text words, the channel, start time, end time, speaking speed and volume corresponding to each sentence, the average speaking speed, average volume, number of sentences and number of text words for the entire audio, and the start time and end time of silence.

[0093] The audio analysis module is used to parse the audio text file of the above-mentioned audio transcription module to determine whether the audio information segment in the audio text file conforms to the preset model rules, namely the text rules and the voice rules. If the two rules are configured at the same time, the audio information segment must conform to the two rules at the same time. After determining that the audio information segment hits the model rule, it is considered that the emotion represented by the model rule exists in the detected audio, and the corresponding audio data is marked with the emotion represented by the model rule.

[0094] 3. Beneficial effects

[0095] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0096] A multi-dimensional speech emotion analysis method and system of the present invention includes the following steps S1, obtaining audio data of the conversation between the person to be detected and the user; S2, converting the audio data into a structured text file; S3, parsing the text file to determine whether the audio information segment in the text file hits the preset model rule; S4, when the audio information segment in the text file meets the preset model rule, marking the corresponding audio data with the emotion mark represented by the model rule; the model rule is divided into text rules and voice rules. By converting the audio information of the conversation between the person to be detected and the user, and performing model analysis based on text rules and voice rules, the speech emotion is effectively analyzed from multiple dimensions such as keywords, speaking speed, volume, mute, interruption, etc., and the sharing indicators are more comprehensive; and the text rules contain several kinds of logical rules, and the effective combination of text rules and voice rules can improve the comprehensiveness and accuracy of emotion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 A flowchart of a multi-dimensional speech emotion analysis method of the present invention;

[0098] Figure 2 It is a detailed flow logic block diagram of step S3 of the present invention. DETAILED DESCRIPTION

[0099] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0100] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0101] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0102] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.

[0103] In addition, the terms "installed", "set", "provided with", "connected", "connected", and "socketed" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be an internal connection between two devices, elements, or components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0104] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0105] Example 1

[0106] See attached Figure 1 and Figure 2 A multi-dimensional speech emotion analysis method of this embodiment includes the following steps:

[0107] S1. Obtaining audio data of the conversation between the person to be detected and the user;

[0108] S2, converting audio data into structured text files;

[0109] S3, parsing the text file to determine whether the audio information segment in the text file hits the preset model rule;

[0110] S4, when the audio information segment in the text file meets the preset model rule, the corresponding audio data is marked with the emotion tag represented by the model rule;

[0111] The model rules are divided into text rules and voice rules.

[0112] The text rules and the voice rules are specifically:

[0113] Text rules: Combine keywords through logical symbols, and then match them with the text of the audio information segment to filter out the desired audio information segment under the corresponding logical text. There are five logical operation symbols, namely "&, |,!,#,()";

[0114] Among them, multiple text rules can form a complete speech process through logical operation symbols. There are five logical operation symbols, namely "&, |,!,@,()";

[0115] The speech rules include at least one of the following objects: the conversation contains a preset keyword and meets its attributes, the mute information in the conversation meets the attributes of the preset mute rule, the speech speed information in the conversation meets the attributes of the preset speech speed rule, the volume information in the conversation meets the attributes of the preset volume rule, and the interruption information in the conversation meets the attributes of the preset interruption rule.

[0116] The step S2 also includes

[0117] Convert the audio data into a structured audio text file, where the audio text file should include: audio text of the agent channel and the customer channel, the number of sentences, the number of text words, the channel, start time, end time, speaking speed and volume corresponding to each sentence, the average speaking speed, average volume, number of sentences and text words of the entire audio, and the start and end time of silence.

[0118] In this step, text indexes can be established through existing speech transcription methods to convert unstructured speech files into

[0119] Convert it into structured text information, namely audio text file, to lay the foundation for subsequent analysis and processing of audio information.

[0120] Furthermore, for the system for multi-dimensional analysis of speech emotions provided by the embodiment of the present invention, the execution body of this step can be an audio transcription module. The subsequent step of analyzing the audio information can be performed by the speech waveform analysis module in the system. Therefore, the audio analysis module can store the audio text file generated in this step in the corresponding directory of the server in an indexed manner so that the audio analysis module can retrieve and use it when analyzing the audio information.

[0121] The above-mentioned indexing method creates a corresponding directory in the server based on the region to which the recording belongs and the year, month, day, hour and date information of the day when the recording was generated collected by the audio collection module, and stores the generated structured text file in the directory. Subsequently, the audio analysis module only needs to follow the same indexing method to quickly find the corresponding audio file.

[0122] The above audio text file includes the following basic information: audio text corresponding to the agent channel and the customer channel, the number of sentences and words in the text, the channel, start time, end time, speech speed information and volume information corresponding to each sentence of the two parties in the conversation, the average speech speed, average volume, number of sentences and text words of the entire audio, and the start and end time of silence. In addition, the serial number corresponding to the recording, the agent work number, store information, the start time and end time of the recording, voiceprint characteristics and other information can also be recorded for use by the audio analysis module when analyzing the audio information.

[0123] Furthermore, since what is analyzed is the audio information of the agents and customers, and different places have different dialects, the embodiments of the present invention also take into account the problem of local accents. When actually using the multi-dimensional speech emotion analysis method provided by the embodiments of the present invention, in order to make the conversion of audio information into a structured audio text file more accurate, it is also necessary to adapt to the local accents of each region to optimize the acoustic model so that it can cover a wide range of local accents. It is also necessary to optimize the speech model in combination with business knowledge and hotline service scope to improve the accuracy of speech transcription.

[0124] The step S3 also includes

[0125] Determine, from the audio text file constituting the audio information, an audio information segment corresponding to the text that conforms to the text rule; when a plurality of preset text rules are combined according to logical operation symbols to form a rule combination, determine whether the plurality of audio segments that conform to the text rule conform to the complete text rule combination;

[0126] Determining an audio information segment that meets the voice rule from the audio information segments constituting the audio information specifically includes:

[0127] Determining an audio information segment containing a preset keyword rule from the audio information segments;

[0128] Determine, from the audio information segments, an audio information segment whose speech rate information meets a preset speech rate rule;

[0129] Determining, from the audio information segments, an audio information segment whose volume information complies with a preset volume rule;

[0130] Determining, from the audio information segments, an audio information segment whose silence information complies with a preset silence rule;

[0131] An audio information segment whose interruption information complies with a preset interruption rule is determined from the audio information segments.

[0132] The step S3 also includes

[0133] Determine whether the audio information segment in the audio text file corresponds to the preset text rule information, wherein the text rule combines keywords through logical symbols, and then matches them with the text of the audio information segment to filter out the desired audio information segment under the corresponding logical text. There are five logical operation symbols, namely "&, |,!,#,()";

[0134] The following is an explanation and example of the logical operation symbols in the above text rules;

[0135] #: means "Near", a primary connector, a connector between simple objects. The object connected by "#" is a complex object. There can be one or more "#" in a complex object, but the number of text characters between the first and last simple objects is no more than 10 by default, and the object before "#" must appear before the object after "#". The simple object refers to a single keyword or word; the complex object refers to an object with multiple keywords or words connected by "#". In addition, you can add "#n" at the end to customize the number of characters, where n represents the maximum number of characters between simple objects.

[0136] Examples: a. The rule is "Excuse me#ID number", "Excuse me" appears in the audio text, and "ID number" appears within 14 words in the following text (because "ID number" itself occupies 4 words), then this audio will be filtered out under this rule; b. The rule is "Hello#Store#5", "Hello" appears in the audio text, and "Store" appears within 7 words in the following text (because "Store" itself occupies 2 words), then this audio will be filtered out under this rule; c. The rule is "Hello#Pufa#Excuse me", which means that "Hello" must be followed by "Store" and "Excuse me", and the number of words between "Hello", "Store" and "Excuse me" cannot exceed 10; d. The rule is "Hello#Seat#Excuse me#15", which means that "Hello" must be followed by "Seat" and "Excuse me", and the number of words between "Hello", "Seat" and "Excuse me" cannot exceed 15.

[0137] (): means "combination", which means combining one or several simple or complex objects into an object you want.

[0138] Example: The rule is "((Xiaomi|Samsung)&(Android))|(Apple|iPhone)". When the recording text contains "Xiaomi&Android", "Samsung&Android" or "Apple or iPhone", the recording will be filtered out under this rule.

[0139] &: means "and", a secondary connector, which means that the simple or complex objects connected by "&" must appear at the same time in the recorded text. There is no limit on the order in which the connected keywords appear. It can also connect a combination of simple or complex objects.

[0140] For example, if the rule is "Hello & Goodbye", when the transcribed text of the audio contains both the keywords "Hello" and "Goodbye", the recording will be filtered out under this rule.

[0141] |: means "or", a secondary connector, which means that at least one of the simple or complex objects connected by "|" appears in the recorded text. It can also connect a combination of simple or complex objects.

[0142] Example: The rule "Hello|Hello" means that when the recording text contains: a. only "Hello"; b. only "Hello"; c. both "Hello" and "Hello", the recordings in these three situations will be filtered out under this rule.

[0143] !: It means "non", a secondary connector, which means that the simple or complex objects connected with "!" are not allowed to appear in the recording text. It can also connect a combination of simple or complex objects.

[0144] For example, if the rule is "!Hello", when the keyword "Hello" does not appear in the transcribed text of the recording, the recording will be filtered out under this rule; when the text rule is (Hello) & (!(Hello)), it means that "Hello" must not appear in the recording text even if "Hello" appears.

[0145] Furthermore, when multiple text rules are preset and multiple text rules are combined into a speech flow through logical operation symbols, if the audio text hits all the text rules and conforms to the logic of the rule combination, then the audio is considered to hit the complete speech flow, where there are five logical operation symbols in the rule combination, namely "&, |,!,@,()";

[0146] The following is an explanation and example of the logical operation symbols in the above rule combination;

[0147] @: means "sequence", which means that the order of simple or complex text rules connected by "@" in the entire audio text must be consistent with the order of text rule objects before and after "@". It can also connect a combination of simple or complex voice rules.

[0148] Example: The rule combination is "text rule 1@text rule 2@text rule 3", which means that the audio text must match Rule 1, Rule 2 and Rule 3 in sequence.

[0149] (): means "combination", which means combining one or several simple or complex text rules into an object you want.

[0150] Example: The rule combination is "((text rule 1|text rule 2)&(text rule 3))|(text rule 4|text rule 5)". When the audio text hits "text rule 1&text rule 3", "text rule 2&text rule 3" or "text rule 4|text rule 5", the audio will be considered to have hit the complete speech flow.

[0151] &: means "and", a secondary connector, which means that the simple or complex text rules connected by "&" must appear at the same time in the entire audio information text. There is no limit on the order in which the connected text rules appear. It can also connect a combination of simple or complex text rules.

[0152] Example: The rule combination is "Text Rule 1 & Text Rule 2". When the transcribed text of the audio hits both Text Rule 1 and Text Rule 2, the audio will be considered to have hit the complete speech flow.

[0153] |: means "or", a secondary connector, which means that at least one of the simple or complex text rules connected by "|" appears in the entire audio information text. It can also connect a combination of simple or complex text rules.

[0154] Example: The rule is "text rule 1 | text rule 2", which means that when the audio text hits: a. only text rule 1; b. only text rule 2; c. hits both text rule 1 and text rule 2, in these three cases, the audio will be regarded as hitting the complete speech process.

[0155] !: means "non", a secondary connector, which means that the simple or complex text rules connected with "!" are not allowed to appear in the entire audio information text. It can also connect a combination of simple or complex text rules.

[0156] Example: The rule combination is "!Text Rule 1". When the transcribed text of the audio does not match Text Rule 1, the audio will be considered to have matched the complete speech flow.

[0157] The above-mentioned voice rules, in parallel with the text rules, analyze the text, speaking speed, volume and silence recorded by the audio transcription model, and the voice rules include at least one of the following rules: containing preset keywords, the silence duration meets the preset duration, the speaking speed information of both parties of the call reaches the corresponding indicator threshold, the volume information reaches the corresponding indicator threshold, and the interruption rule meets the preset conditions.

[0158] Determining the preset voice rule information that each audio text segment corresponding to each of the audio information segments in the audio text file complies with specifically includes:

[0159] When the preset voice rule includes keyword information, for each audio information segment in each audio information segment, the audio text segment corresponding to the audio information segment is compared with the preset keyword; the preset keyword included in the audio text segment, and the start time and end time of the appearance of the preset keyword are determined;

[0160] When the preset voice rule information includes speech rate rule information, for each audio information segment in each audio information segment, according to the speech rate characteristics of each sentence of the two parties in the audio text segment corresponding to the audio information segment; determining the speech rate characteristics that meet the preset rules and the start time and end time of the speech rate characteristics contained in the audio text segment;

[0161] When the preset voice rule information includes volume rule information, for each audio information segment in each audio information segment, according to the volume characteristics of each sentence of the two parties in the audio text segment corresponding to the audio information segment; determine the volume characteristics that meet the preset rules and the start time and end time of the volume characteristics contained in the audio text segment;

[0162] When the preset voice rule information includes silence rule information, for each audio information segment in each audio information segment, according to the start time, end time, and duration of the audio corresponding to each sentence of the two parties in the audio text segment corresponding to the audio information segment, determine the silence feature contained in the audio corresponding to the person to be detected in the audio text segment; determine the silence feature contained in the audio text segment that meets the preset rule, as well as the start time and end time of the occurrence of the contained silence feature;

[0163] When the preset voice rule information includes interruption rule information, for each audio information segment in each audio information segment, the interruption feature contained in the audio corresponding to the person to be detected in the audio text segment is determined according to the start time, end time, and duration of the audio corresponding to each sentence of the two parties in the audio text segment corresponding to the audio information segment; the interruption feature contained in the audio text segment that meets the preset rules is determined, as well as the start time and end time of the interruption feature.

[0164] When the preset voice rule is a keyword rule, the following attributes may be added to the keyword rule, including:

[0165] Absolute position, i.e., the specific time position of the keyword in the entire audio information; relative position, for two recording rules, i.e., the relative position of the keyword rule with respect to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker of the most recent speech after the keyword appears, whether it is the customer channel or the agent channel; previous channel, i.e., the speaker of the most recent speech before the keyword appears, whether it is the customer channel or the agent channel; channel, i.e., the speaker identity of the keyword, whether the keyword is spoken by the customer or the agent; number, i.e., the number of times the keyword appears in the entire audio information; rule, i.e., the text rule of the keyword itself;

[0166] When the preset voice rule is a speech rate rule, the following attributes may be added to the speech rate rule, including:

[0167] Absolute position, i.e., the specific time position where the speech rate rule appears in the entire audio information; relative position, for two recording rules, i.e., the relative relationship of the speech rate rule to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker who spoke the most recent content after the speech rate rule appears, whether it is the customer channel or the agent channel; previous channel, i.e., the channel of the speaker who spoke the most recent content before the speech rate rule appears, whether it is the customer channel or the agent channel; average, i.e., the average speech speed of the entire audio information; channel, indicating whether the speech rate rule acts on the customer or the agent channel; number of times, i.e., the number of times the audio information segment that meets the speech rate rule appears in the entire audio information; single sentence, i.e., the number of times a single sentence that meets the speech rate rule appears in the entire audio information;

[0168] When the preset voice rule is a volume rule, the following attributes may be added to the volume rule, including:

[0169] Absolute position, i.e., the specific time position where the volume rule appears in the entire audio information; relative position, for two recording rules, i.e., the relative position of the volume rule with respect to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker who spoke the most recent content after the volume rule appears, whether it is the customer channel or the agent channel; previous channel, i.e., the channel of the speaker who spoke the most recent content before the volume rule appears, whether it is the customer channel or the agent channel; channel, indicating whether the volume rule acts on the customer channel or the agent channel; number of times, i.e., the number of times the audio information segment that meets the volume rule appears in the entire audio information; volume, i.e., the volume size of the specified volume rule;

[0170] When the preset voice rule is a mute rule, the following attributes may be added to the mute rule, including:

[0171] Absolute position, i.e. the specific time position of silence in the whole audio information; relative position, for two recording rules, i.e. the relative position of silence rule with respect to another voice rule in the whole audio information; subsequent channel, i.e. the channel of the speaker who spoke the most recent content after the silence ended, whether it was the customer channel or the agent channel; previous channel, i.e. the channel of the speaker who spoke the most recent content before the silence started, whether it was the customer channel or the agent channel; number of times, i.e. the number of times the audio information segment that meets the silence rule appears in the whole audio information; duration, i.e. the duration of the specified silence;

[0172] When the preset voice rule is an interruption rule, the following attributes may be added to the interruption rule, including:

[0173] The overlapping duration refers to the duration of the overlap of the audio of the two channels after the interruption occurs; the number of times refers to the number of times the audio information segment that meets the interruption rule appears in the entire audio information; the subsequent channel refers to the channel of the speaker who spoke the most recent content after the interruption, whether it is the customer channel or the agent channel; the previous channel refers to the channel of the speaker who spoke the most recent content before the interruption began, whether it is the customer channel or the agent channel; the absolute position refers to the specific time position of the interruption in the entire audio information; the relative position refers to the relative position of the interruption rule relative to the other voice rule in the entire audio information for the two recording rules.

[0174] The specific rule objects in the above speech rules are explained and illustrated below;

[0175] Keyword rules contain at least one of the following attributes:

[0176] Rules, that is, the text rules of the keywords themselves, are the basis of keyword rules. You can configure a single keyword, or combine multiple keywords using the same logical symbols as the above text rules. For example, when multiple keywords are separated by the "|" symbol, it means that as long as one of these keywords appears in the transcribed text of the audio, the audio can be considered to meet the keyword attributes in the voice rules. Similarly, when multiple keywords are separated by the "&" symbol, it means that these keywords must appear at the same time in the transcribed text of the audio.

[0177] Number of times, this attribute is based on the above rule attributes, that is, the number of times the keyword appears in the entire audio information; for example, when the number of keywords is set to "equal to 0", it means exclusion, all keywords configured in the above rule attributes will not appear in the entire audio information, and the audio is considered to meet the number of times attribute of the keyword rule; similarly, when the number of times is configured to be greater than or equal to 3, and multiple keywords connected by "|" are configured in the above rule attributes, then as long as the total number of times multiple keywords appear in the audio information is 3 times or more, the audio is considered to meet the number of times attribute of the keyword rule.

[0178] Channel, that is, to limit whether the keyword is spoken by the customer or the agent. For example, when the channel selects agent, only when the agent says the preset keyword, the audio is considered to meet the channel attributes of the keyword rule. When the channel selects all, no matter whether the keyword is spoken by the agent or the customer, it is considered to meet the channel attributes of the keyword rule.

[0179] Absolute position refers to the specific time position where the keyword appears in the entire audio information. The specific time can be preset in seconds. For example, when the absolute position of the keyword is set to be less than 10 seconds, the keyword appears within the first 10 seconds of the recording, and the audio is deemed to meet the absolute position attribute of the keyword rule. Similarly, when the absolute position of the keyword is set to the last 10 seconds of the recording, the keyword appears within the last 10 seconds of the recording, and the audio is deemed to meet the absolute position attribute of the keyword rule.

[0180] Relative position: This attribute refers to two recording rules, that is, the relative position of the keyword rule with respect to another voice rule in the entire audio information; for example, when there is another audio rule 1, and the relative position attribute of the keyword rule is set to 10 seconds after audio rule 1, as long as the entire audio hits the preset keyword rule within 10 seconds after the time point when audio rule 1 is hit, the audio is considered to meet the relative position attribute of the keyword rule.

[0181] The previous channel is the channel of the speaker of the most recent speech before the keyword appears, which is the customer channel or the agent channel. For example, if the previous channel is set to the agent, when the keyword rule hits the speaker identity of the most recent speech content is the agent, the audio is considered to meet the previous channel attribute of the keyword rule.

[0182] The subsequent channel is the channel of the speaker of the most recent speech after the keyword appears, which is the customer channel or the agent channel. For example, if the subsequent channel is set to the agent, when the keyword rule hits the speaker identity of the most recent speech content is the agent, the audio is considered to meet the subsequent channel attribute of the keyword rule.

[0183] The speech rate rule includes at least one of the following attributes:

[0184] Average: the average speaking speed of the entire audio information, where the unit of speaking speed is words / second. For example, when the average attribute of the speaking speed rule is set to be greater than 5 words / second, the speaking speed of the entire audio is used as the benchmark. When the average speaking speed is greater than 5 words / second, the audio is considered to meet the average attribute of the speaking speed rule.

[0185] Single sentence: the number of times a single sentence that complies with the speech speed rules appears in the entire audio information; for example, when the single sentence attribute is set to greater than 3 words / second, if the speaking speed of a single sentence of an agent or customer in the entire audio is greater than 3 words / second, the audio is considered to comply with the single sentence attribute of the speech speed rules.

[0186] The number of times refers to the number of times the audio information segment that complies with the speech speed rules appears in the entire audio information. It is generally used in conjunction with a single sentence. For example, when the single sentence attribute is set to greater than 3 words / second and the number of times is set to greater than 2 times, if a single speech of an agent or customer appears in the entire audio at a speed greater than 3 words / second and the number of times it appears is greater than 2 times, the audio is considered to comply with the single sentence and number attributes of the speech speed rules.

[0187] Channel: indicates whether the speech rate rule applies to the customer channel or the agent channel. For example, when the channel is set to the customer channel and the average attribute is greater than 5 words / second, the speech rate of the customer channel in the entire audio segment is used as the benchmark. If the average speech rate is greater than 5 words / second, the audio is considered to meet the channel and average attributes of the speech rate rule. When the channel is set to the customer channel and the single sentence attribute is greater than 5 words / second, the speech rate of a single sentence in the customer channel in the entire audio segment is used as the benchmark. If the single sentence speech rate is greater than 5 words / second, the audio is considered to meet the channel and single sentence attributes of the speech rate rule.

[0188] Absolute position, used in conjunction with single sentence attributes, is the specific time position where the single sentence speech rate rule appears in the entire audio information. When the single sentence attribute is set to greater than 3 words / second and the absolute position is less than 10 seconds, if a single sentence with a speech rate greater than 3 words / second appears within the first 10 seconds of the audio, the audio is considered to meet the single sentence and absolute position attributes of the speech rate rule.

[0189] Relative position: This attribute is used with the single sentence attribute for two recording rules. For example, when there is another audio rule 1, the relative position attribute of the speech rate rule is set to 10 seconds after audio rule 1, and the single sentence attribute is greater than 3 words / second, as long as the entire audio segment has a single sentence with a speech rate greater than 3 words / second within 10 seconds after the time point when audio rule 1 is hit, the audio segment is considered to meet the single sentence and relative position attributes of the speech rate rule.

[0190] The previous channel is the channel of the speaker of the most recent speech before the speech rate rule appears. It is the customer channel or the agent channel, and is used together with the single sentence attribute. For example, if the previous channel is set to the agent and the single sentence attribute is greater than 3 words / second, when the single sentence rule hits the speaker of the most recent speech and the speaker is the agent, the audio is considered to meet the previous channel and single attribute of the speech rate rule.

[0191] The subsequent channel is the channel of the speaker of the most recent speech after the speech rate rule appears. It is the customer channel or the agent channel, and is used in conjunction with the single sentence attribute. For example, if the subsequent channel is set to the agent and the single sentence attribute is greater than 3 words / second, when the single sentence rule hits the speaker of the most recent speech and the speaker is the agent, the audio is considered to meet the subsequent channel and single attribute of the speech rate rule.

[0192] Volume rules include at least one of the following attributes:

[0193] Volume, that is, the volume size of the specified volume rule, in decibels. This attribute is the basis of the volume rule. For example, when the volume attribute is set to be greater than 50 decibels, and a speech with a volume greater than 50 decibels appears in the entire audio, the audio is considered to meet the volume attribute of the volume rule.

[0194] The number of times the audio information segment that meets the volume rule appears in the entire audio information; for example, when setting

[0195] The volume attribute is greater than 50 decibels and the number of times is greater than 3. When the number of times a sentence with a volume greater than 50 decibels appears in the entire audio exceeds 3 times, the audio is considered to meet the volume and number attributes of the volume rules.

[0196] Channel: indicates whether the volume rule applies to the customer or agent channel, that is, the relative position of the volume rule relative to another voice rule in the entire audio information. For example, when the volume attribute is set to be greater than 50 decibels, the channel is the agent channel, and the agent speaks at a volume greater than 50 decibels in the entire audio, the audio is considered to meet the volume and channel attributes of the volume rule.

[0197] The absolute position refers to the specific time position where the volume rule appears in the entire audio information. When the volume attribute is set to be greater than 50 decibels and the absolute position is less than 10 seconds, if a speech with a volume greater than 50 decibels appears within the first 10 seconds of the audio, the audio is considered to comply with the volume and absolute position attributes of the volume rule.

[0198] Relative position: This attribute is for two recording rules. For example, when there is another audio rule 1, the relative position attribute of the speech rate rule is set to 10 seconds after audio rule 1, and the volume attribute is greater than 50 decibels. As long as the entire audio has a speech with a volume greater than 50 decibels within 10 seconds after the time point when audio rule 1 is hit, the audio is considered to meet the volume and relative position attributes of the volume rule.

[0199] The previous channel is the channel of the speaker of the most recent speech before the volume rule appears, whether it is the customer channel or the agent channel; for example, if the previous channel is set to the agent and the volume attribute is greater than 50 decibels, when the speaker of the most recent speech before the speech with a volume greater than 50 decibels is the agent, the audio is considered to meet the previous channel and volume attributes of the volume rule.

[0200] The previous channel, the speaker channel of the most recent speech after the volume rule appears, is it the customer channel or the agent channel; for example, when the subsequent channel is set to the agent and the volume attribute is greater than 50 decibels, when the speaker of the most recent speech after the speech with a volume greater than 50 decibels is the agent, the audio is considered to meet the subsequent channel and volume attributes of the volume rule.

[0201] A mute rule contains at least one of the following attributes:

[0202] Duration: specifies the duration of silence in seconds. For example, when the duration of silence is set to be greater than 5 seconds, the audio is considered to meet the duration attribute of the silence rule as long as there is a silence period greater than 5 seconds in the entire audio.

[0203] The number of times refers to the number of times the audio information segment that complies with the silence rule appears in the entire audio information. For example, when the number of times attribute is set to be greater than 3 times, the number of silent time periods that appear in the entire audio exceeds 3 times, and the audio is deemed to comply with the number attribute of the silence rule. It can also be used in conjunction with the duration attribute. When the silence duration attribute is set to be greater than 5 seconds and the number of times attribute is greater than 3 times, the number of silent time periods that last more than 5 seconds that appear in the entire audio exceeds 3 times, and the audio is deemed to comply with both the number and duration attributes of the silence rule.

[0204] Absolute position refers to the specific time position where the silence rule appears in the entire audio information. For example, when the absolute position attribute of silence is set to be less than 10 seconds, silence appears within the first 10 seconds of the recording, and the audio is deemed to comply with the absolute position attribute of the silence rule. Similarly, when the absolute position of silence is set to the last 10 seconds of the recording, silence appears within the last 10 seconds of the recording, and the audio is deemed to comply with the absolute position attribute of the silence rule.

[0205] Relative position: This attribute refers to two recording rules, that is, the relative position of the mute rule relative to another voice rule in the entire audio information. For example, when there is another audio rule 1, and the relative position attribute of the mute rule is set to 10 seconds after audio rule 1, as long as the entire audio hits the preset mute rule within 10 seconds after the time point when audio rule 1 is hit, the audio is considered to meet the relative position attribute of the keyword rule.

[0206] Previous channel, the channel of the speaker of the most recent speech before the mute occurs, whether it is a customer channel or an agent channel; for example, if the previous channel is set to an agent, when the mute rule matches that the speaker of the most recent speech is an agent, the audio is considered to meet the previous channel attribute of the mute rule.

[0207] The subsequent channel is the channel of the speaker of the most recent speech after the mute occurs, which is the customer channel or the agent channel. For example, if the subsequent channel is set to the agent, when the mute rule hits the speaker identity of the most recent speech content and the mute rule is hit, the audio is considered to meet the subsequent channel attribute of the mute rule.

[0208] Interruption rules contain at least one of the following attributes:

[0209] Overlap duration refers to the duration of overlap between the audios of the two channels after an interruption occurs. For example, when the overlap duration of an interruption is set to be greater than 5 seconds, in the entire audio, as long as one channel is speaking and the other channel interrupts, and the overlap duration of the two channels is greater than 5 seconds, the audio is considered to meet the silence duration attribute of the interruption rule.

[0210] The number of times refers to the number of times the interruption occurs in the entire audio information. For example, when the number of times attribute is set to be greater than 3 times, the number of interruptions in the entire audio exceeds 3 times, and the audio is deemed to meet the number of interruption rule attributes. It can also be used in conjunction with the overlapping duration attribute. When the overlapping duration attribute of the interruption is set to be greater than 5 seconds and the number of times attribute is greater than 3 times, the number of overlapping time periods with a duration of more than 5 seconds in the entire audio exceeds 3 times, and the audio is deemed to meet the number of interruption rule and overlapping duration attributes.

[0211] Absolute position refers to the specific time position where the interruption rule appears in the entire audio information. For example, when the absolute position attribute of the interruption is set to be less than 10 seconds, the interruption appears within the first 10 seconds of the recording, and the audio is deemed to meet the absolute position attribute of the interruption rule. Similarly, when the absolute position of the interruption is set to the last 10 seconds of the recording, the interruption appears within the last 10 seconds of the recording, and the audio is deemed to meet the absolute position attribute of the interruption rule.

[0212] Relative position: This property is for two recording rules, that is, the relative position of the interruption rule relative to another voice rule in the entire audio information. For example, when there is another audio rule 1, and the relative position property of the interruption rule is set to 10 seconds after audio rule 1, as long as the entire audio hits the preset interruption rule within 10 seconds after the time point when audio rule 1 is hit, the audio is considered to meet the relative position property of the interruption rule.

[0213] Previous channel, the channel of the speaker of the most recent speech before the interruption occurs, whether it is the customer channel or the agent channel; for example, if the previous channel is set to the agent, when the speaker of the most recent speech before the interruption occurs is the agent, the audio is considered to meet the previous channel attribute of the interruption rule.

[0214] The subsequent channel is the channel of the speaker of the most recent speech after the interruption occurs, which is the customer channel or the agent channel. For example, if the subsequent channel is set to the agent, when the interruption rule hits the speaker identity of the most recent speech content and the speaker is the agent, the audio is considered to meet the subsequent channel attribute of the interruption rule.

[0215] A multi-dimensional speech emotion analysis system, characterized in that: the system is used to perform the above method, the system includes

[0216] An audio collection module is used to obtain audio information of the conversation between the person to be detected and the user;

[0217] The audio transcription module is used to convert the audio obtained by the audio collection module into a structured audio text file, wherein the audio text file should include: the audio text of the agent channel and the customer channel, the number of sentences, the number of text words, the channel, start time, end time, speaking speed and volume corresponding to each sentence, the average speaking speed, average volume, number of sentences and number of text words of the entire audio, and the start time and end time of silence;

[0218] The audio analysis module is used to parse the audio text file of the above-mentioned audio transcription module to determine whether the audio information segment in the audio text file conforms to the preset model rules, namely the text rules and the voice rules. If the two rules are configured at the same time, the audio information segment must conform to the two rules at the same time. After determining that the audio information segment hits the model rule, it is considered that the emotion represented by the model rule exists in the detected audio, and the corresponding audio data is marked with the emotion represented by the model rule.

[0219] See attached Figure 2 , the parsing process of the text information in step S3 is described;

[0220] First, in process S301, the structured text information transcribed in the audio transcription module enters the audio analysis module;

[0221] Enter process S302 to determine whether text rules are configured in the model rules;

[0222] When no text rules are configured in S302, the process proceeds to process S303 to determine whether a voice rule is configured in the model rule;

[0223] When the voice rules are not configured in S303, the process enters process S305, and it is considered that no model rules are currently configured, and the structured text file transcribed in the audio transcription module cannot be parsed, and the process ends.

[0224] When the voice rules are configured in S303, the process enters S306 to parse the audio and text information according to the configured voice rules; wherein the voice rules include the following attributes: preset keyword rules, the silence duration meets the preset duration, the speech speed information of both parties of the call reaches the corresponding indicator threshold, the volume information reaches the corresponding indicator threshold, and the interruption rule meets the preset conditions;

[0225] Entering process S308, determining whether the voice rule is matched;

[0226] When S308 does not match the voice rule, the process enters process S310, and it is considered that the audio does not match the model rule, and the process ends.

[0227] When S308 matches the voice rule, the process enters S319 and considers that the audio meets the preset model rule;

[0228] When the text rules are configured in S302, the process proceeds to process S304, where the text is parsed according to the keywords and logical symbols configured in the text rules;

[0229] Enter process S307 to determine whether the text rule is matched;

[0230] When no text rule is matched in S307, the process proceeds to S313 to determine whether a voice rule is configured;

[0231] When S307 matches the text rule, the process goes to S309 to determine whether a rule combination of multiple text rules is configured;

[0232] When the rule combination of multiple text rules is not configured in S309, the process proceeds to process S313 to determine whether a voice rule is configured;

[0233] When a rule combination of multiple text rules is configured in S309, the process proceeds to process S311, and the text information is parsed according to the rule combination composed of the configured multiple text rules and logical operators;

[0234] Entering process S312, determining whether a rule combination of multiple text rules is matched;

[0235] When S312 does not match the rule combination of multiple text rules, the process proceeds to process S313 to determine whether a voice rule is configured;

[0236] When S312 matches a rule combination of multiple text rules, the process goes to process S313 to determine whether a voice rule is configured;

[0237] Enter process S313 to determine whether voice rules are configured;

[0238] When no voice rule is configured in S313, the process proceeds to S314 to check whether a text rule or rule combination is hit;

[0239] When S314 does not match the text rule or rule combination, the process enters process S316, and it is considered that the audio does not match the model rule, and the process ends.

[0240] When S314 matches a text rule or a rule combination, the process proceeds to S319, where it is considered that the audio meets the preset model rule;

[0241] When the voice rules are configured in S313, the process enters S315 to parse the audio and text information according to the configured voice rules; wherein the voice rules include the following attributes: preset keyword rules, the silence duration meets the preset duration, the speech speed information of both parties of the call reaches the corresponding indicator threshold, the volume information reaches the corresponding indicator threshold, and the interruption rule meets the preset conditions;

[0242] Enter process S317 to determine whether the voice rule is matched;

[0243] When S317 matches the voice rule, the process enters S319 and considers that the audio meets the preset model rule;

[0244] When the voice rule is not hit in S317, the process goes to S318 to check whether the text rule or rule combination is hit;

[0245] When S318 does not match the text rule or rule combination, the process enters process S320, and it is considered that the audio does not match the model rule, and the process ends.

[0246] When S318 matches a text rule or a rule combination, the process proceeds to S319, where it is considered that the audio meets the preset model rule;

[0247] Entering process S319, confirming that the audio meets the preset model rules, where the model rules include text rules and voice rules;

[0248] Entering process S321, after confirming that the audio meets the preset model rules, the audio is marked with the emotion tag corresponding to the model rules, and it is considered that the audio meets the emotion corresponding to the model rules, and the process ends.

[0249] S4, marks the audio that hits the model rules in S3 with corresponding emotion tags. It should be understood that each model is created to detect a certain preset emotion. There is a one-to-many relationship between emotion and model. The same emotion may include multiple models. When the audio hits the model rules, it is considered that the audio meets a certain preset emotion and is marked for subsequent operation and maintenance personnel to view.

[0250] A multi-dimensional speech emotion analysis method and system of the present embodiment converts the audio information of the conversation between the person to be tested and the user, and performs model analysis based on text rules and speech rules, effectively analyzes the speech emotion from multiple dimensions such as keywords, speaking speed, volume, silence, interruption, etc., and shares more comprehensive indicators; and the text rules include several kinds of logical rules. The effective combination of text rules and speech rules can improve the comprehensiveness and accuracy of emotion analysis.

[0251] The above-described embodiments only express a certain implementation mode of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A multi-dimensional speech emotion analysis method, characterized in that: The following steps are included S1. Obtaining audio data of the conversation between the person to be detected and the user; S2, converting audio data into structured text files; S3, parsing the text file to determine whether the audio information segment in the text file hits the preset model rule; S4, when the audio information segment in the text file meets the preset model rule, the corresponding audio data is marked with the emotion tag represented by the model rule; The model rules are divided into text rules and voice rules.

2. A multi-dimensional speech emotion analysis method according to claim 1, characterized in that: The text rules and the voice rules are specifically: Text rules: Combine keywords through logical symbols, and then match them with the text of the audio information segment to filter out the desired audio information segment under the corresponding logical text. There are five logical operation symbols, namely "&, |,!,#,()"; Among them, multiple text rules can form a complete speech process through logical operation symbols. There are five logical operation symbols, namely "&, |,!,@,()"; The speech rules include at least one of the following objects: the conversation contains a preset keyword and meets its attributes, the mute information in the conversation meets the attributes of the preset mute rule, the speech speed information in the conversation meets the attributes of the preset speech speed rule, the volume information in the conversation meets the attributes of the preset volume rule, and the interruption information in the conversation meets the attributes of the preset interruption rule.

3. A multi-dimensional speech emotion analysis method according to claim 2, characterized in that: The step S2 also includes Convert the audio data into a structured audio text file, where the audio text file should include: audio text of the agent channel and the customer channel, the number of sentences, the number of text words, the channel, start time, end time, speaking speed and volume corresponding to each sentence, the average speaking speed, average volume, number of sentences and text words of the entire audio, and the start and end time of silence.

4. The multi-dimensional speech emotion analysis method according to claim 2, characterized in that: The step S3 also includes Determine, from the audio text file constituting the audio information, an audio information segment corresponding to the text that conforms to the text rule; when a plurality of preset text rules are combined according to logical operation symbols to form a rule combination, determine whether the plurality of audio segments that conform to the text rule conform to the complete text rule combination; Determining an audio information segment that meets the voice rule from the audio information segments constituting the audio information specifically includes: Determining an audio information segment containing a preset keyword rule from the audio information segments; Determine, from the audio information segments, an audio information segment whose speech rate information meets a preset speech rate rule; Determining, from the audio information segments, an audio information segment whose volume information complies with a preset volume rule; Determining, from the audio information segments, an audio information segment whose silence information complies with a preset silence rule; An audio information segment whose interruption information complies with a preset interruption rule is determined from the audio information segments.

5. The multi-dimensional speech emotion analysis method according to claim 4, characterized in that: The step S3 also includes Determine whether the audio text segment corresponding to the audio information segment in the audio text file conforms to the preset text rule information, wherein the text rule includes keywords and logical symbols, and the logical symbols specifically include: #: means "Near", a primary connector, a connector between simple objects. The object connected by "#" is a complex object. There can be one or more "#" in a complex object, but the number of text characters between the first and last simple objects is no more than 10 by default. At the same time, the object before "#" must appear before the object after "#". The simple object refers to a single keyword or word; the complex object refers to an object connected by multiple keywords or words through "#"; (): means "combination", which means combining one or several simple or complex objects into a desired object; &: means "and", a secondary connector, which means that the simple or complex objects connected by "&" must appear at the same time in the recording text. There is no limit to the order in which the connected keywords appear. It can also connect a combination of simple or complex objects; |: means "or", a secondary connector, which means that at least one of the simple or complex objects connected by "|" appears in the recording text. It can also connect a combination of simple or complex objects; !: It means "non", a secondary connector, which means that the simple or complex object connected by "!" is not allowed to appear in the recording text. It can also connect a combination of simple or complex objects; When it is determined that the audio text corresponding to the audio information segment hits a certain preset text rule, the start time and end time corresponding to the audio information segment that hits the text rule are recorded.

6. A multi-dimensional speech emotion analysis method according to claim 5, characterized in that: The step S3 also includes After determining multiple audio information segments that meet the text rules, it is determined whether the multiple audio information segments meet the logical combination of multiple text rules to form a complete speech process, wherein the logical combination of the text rules includes: @: means "sequence", which means that the order of simple or complex text rules connected by "@" in the entire audio text must be consistent with the order of text rule objects before and after "@". It can also connect a combination of simple or complex voice rules; (): means "combination", which means combining one or several simple or complex text rules into an object you want; &: means "and", a secondary connector, which means that the simple or complex text rules connected by "&" must appear at the same time in the entire audio information text. The order in which the connected text rules appear is not limited, and it can also connect a combination of simple or complex text rules; |: means "or", a secondary connector, which means that at least one of the simple or complex text rules connected by "|" appears in the entire audio information text. It can also connect a combination of simple or complex text rules; !: means "non", a secondary connector, which means that the simple or complex text rules connected with "!" are not allowed to appear in the entire audio information text. It can also connect a combination of simple or complex text rules.

7. The multi-dimensional speech emotion analysis method according to claim 6, characterized in that: Determining the preset voice rule information that each audio text segment corresponding to each of the audio information segments in the audio text file complies with specifically includes: When the preset voice rule includes keyword information, for each audio information segment in each audio information segment, the audio text segment corresponding to the audio information segment is compared with the preset keyword; the preset keyword included in the audio text segment, and the start time and end time of the appearance of the preset keyword are determined; When the preset voice rule information includes speech rate rule information, for each audio information segment in each audio information segment, according to the speech rate characteristics of each sentence of the two parties in the audio text segment corresponding to the audio information segment; determining the speech rate characteristics that meet the preset rules and the start time and end time of the speech rate characteristics contained in the audio text segment; When the preset voice rule information includes volume rule information, for each audio information segment in each audio information segment, according to the volume characteristics of each sentence of the two parties in the audio text segment corresponding to the audio information segment; determine the volume characteristics that meet the preset rules and the start time and end time of the volume characteristics contained in the audio text segment; When the preset voice rule information includes silence rule information, for each audio information segment in each audio information segment, according to the start time, end time, and duration of the audio corresponding to each sentence of the two parties in the audio text segment corresponding to the audio information segment, determine the silence feature contained in the audio corresponding to the person to be detected in the audio text segment; determine the silence feature contained in the audio text segment that meets the preset rule, as well as the start time and end time of the occurrence of the contained silence feature; When the preset voice rule information includes interruption rule information, for each audio information segment in each audio information segment, the interruption feature contained in the audio corresponding to the person to be detected in the audio text segment is determined according to the start time, end time, and duration of the audio corresponding to each sentence of the two parties in the audio text segment corresponding to the audio information segment; the interruption feature contained in the audio text segment that meets the preset rules is determined, as well as the start time and end time of the interruption feature.

8. The multi-dimensional speech emotion analysis method according to claim 7, characterized in that: When the preset voice rule is a keyword rule, the following attributes may be added to the keyword rule, including: Absolute position, i.e., the specific time position of the keyword in the entire audio information; relative position, for two recording rules, i.e., the relative position of the keyword rule with respect to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker of the most recent speech after the keyword appears, whether it is the customer channel or the agent channel; previous channel, i.e., the speaker of the most recent speech before the keyword appears, whether it is the customer channel or the agent channel; channel, i.e., the speaker identity of the keyword, whether the keyword is spoken by the customer or the agent; number, i.e., the number of times the keyword appears in the entire audio information; rule, i.e., the text rule of the keyword itself; When the preset voice rule is a speech rate rule, the following attributes may be added to the speech rate rule, including: Absolute position, i.e., the specific time position where the speech rate rule appears in the entire audio information; relative position, for two recording rules, i.e., the relative relationship of the speech rate rule to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker who spoke the most recent content after the speech rate rule appears, whether it is the customer channel or the agent channel; previous channel, i.e., the channel of the speaker who spoke the most recent content before the speech rate rule appears, whether it is the customer channel or the agent channel; average, i.e., the average speech speed of the entire audio information; channel, indicating whether the speech rate rule acts on the customer or the agent channel; number of times, i.e., the number of times the audio information segment that meets the speech rate rule appears in the entire audio information; single sentence, i.e., the number of times a single sentence that meets the speech rate rule appears in the entire audio information; When the preset voice rule is a volume rule, the following attributes may be added to the volume rule, including: Absolute position, i.e., the specific time position where the volume rule appears in the entire audio information; relative position, for two recording rules, i.e., the relative position of the volume rule with respect to another voice rule in the entire audio information; subsequent channel, i.e., the channel of the speaker who spoke the most recent content after the volume rule appears, whether it is the customer channel or the agent channel; previous channel, i.e., the channel of the speaker who spoke the most recent content before the volume rule appears, whether it is the customer channel or the agent channel; channel, indicating whether the volume rule acts on the customer channel or the agent channel; number of times, i.e., the number of times the audio information segment that meets the volume rule appears in the entire audio information; volume, i.e., the volume size of the specified volume rule; When the preset voice rule is a mute rule, the following attributes may be added to the mute rule, including: Absolute position, i.e. the specific time position of silence in the whole audio information; relative position, for two recording rules, i.e. the relative position of silence rule with respect to another voice rule in the whole audio information; subsequent channel, i.e. the channel of the speaker who spoke the most recent content after the silence ended, whether it was the customer channel or the agent channel; previous channel, i.e. the channel of the speaker who spoke the most recent content before the silence started, whether it was the customer channel or the agent channel; number of times, i.e. the number of times the audio information segment that meets the silence rule appears in the whole audio information; duration, i.e. the duration of the specified silence; When the preset voice rule is an interruption rule, the following attributes may be added to the interruption rule, including: The overlapping duration refers to the duration of the overlap of the audio of the two channels after the interruption occurs; the number of times refers to the number of times the audio information segment that meets the interruption rule appears in the entire audio information; the subsequent channel refers to the channel of the speaker who spoke the most recent content after the interruption, whether it is the customer channel or the agent channel; the previous channel refers to the channel of the speaker who spoke the most recent content before the interruption began, whether it is the customer channel or the agent channel; the absolute position refers to the specific time position of the interruption in the entire audio information; the relative position refers to the relative position of the interruption rule relative to the other voice rule in the entire audio information for the two recording rules.

9. The multi-dimensional speech emotion analysis method according to claim 7, characterized in that: The specific process of step S3 includes: Process S301, the structured text information transcribed by the audio transcription module enters the audio analysis module; Enter process S302 to determine whether text rules are configured in the model rules; When no text rules are configured in S302, the process proceeds to process S303 to determine whether a voice rule is configured in the model rule; When the voice rules are not configured in S303, the process proceeds to process S305, where it is considered that no model rules are currently configured, and the structured text file transcribed from the audio transcription module cannot be parsed, and the process ends; When the voice rules are configured in S303, the process enters S306 to parse the audio and text information according to the configured voice rules; wherein the voice rules include the following attributes: preset keyword rules, the silence duration meets the preset duration, the speech speed information of both parties of the call reaches the corresponding indicator threshold, the volume information reaches the corresponding indicator threshold, and the interruption rule meets the preset conditions; Entering process S308, determining whether the voice rule is matched; When S308 does not match the voice rule, the process enters S310, and it is considered that the audio does not match the model rule, and the process ends; When S308 matches the voice rule, the process enters S319 and considers that the audio meets the preset model rule; When the text rules are configured in S302, the process proceeds to process S304, where the text is parsed according to the keywords and logical symbols configured in the text rules; Enter process S307 to determine whether the text rule is matched; When no text rule is matched in S307, the process proceeds to S313 to determine whether a voice rule is configured; When S307 matches the text rule, the process goes to S309 to determine whether a rule combination of multiple text rules is configured; When the rule combination of multiple text rules is not configured in S309, the process proceeds to process S313 to determine whether a voice rule is configured; When a rule combination of multiple text rules is configured in S309, the process proceeds to process S311, and the text information is parsed according to the rule combination composed of the configured multiple text rules and logical operators; Entering process S312, determining whether a rule combination of multiple text rules is matched; When S312 does not match the rule combination of multiple text rules, the process proceeds to process S313 to determine whether a voice rule is configured; When S312 matches a rule combination of multiple text rules, the process goes to process S313 to determine whether a voice rule is configured; Enter process S313 to determine whether voice rules are configured; When no voice rule is configured in S313, the process proceeds to S314 to check whether a text rule or rule combination is hit; When S314 does not match the text rule or rule combination, the process proceeds to S316, where it is considered that the audio does not match the model rule, and the process ends; When S314 matches a text rule or a rule combination, the process proceeds to S319, where it is considered that the audio meets the preset model rule; When the voice rules are configured in S313, the process enters S315 to parse the audio and text information according to the configured voice rules; wherein the voice rules include the following attributes: preset keyword rules, the silence duration meets the preset duration, the speech speed information of both parties of the call reaches the corresponding indicator threshold, the volume information reaches the corresponding indicator threshold, and the interruption rule meets the preset conditions; Enter process S317 to determine whether the voice rule is matched; When S317 matches the voice rule, the process enters S319 and considers that the audio meets the preset model rule; When the voice rule is not hit in S317, the process goes to S318 to check whether the text rule or rule combination is hit; When S318 does not match the text rule or rule combination, the process proceeds to S320, where it is considered that the audio does not match the model rule, and the process ends; When S318 matches a text rule or a rule combination, the process proceeds to S319, where it is considered that the audio meets the preset model rule; Entering process S319, confirming that the audio meets the preset model rules, where the model rules include text rules and voice rules; Entering process S321, after confirming that the audio meets the preset model rules, the audio is marked with the emotion tag corresponding to the model rules, and it is considered that the audio meets the emotion corresponding to the model rules, and the process ends.

10. A multi-dimensional speech emotion analysis system, characterized in that: The system is used to execute the method described in any one of claims 1 to 9, and the system comprises An audio collection module is used to obtain audio information of the conversation between the person to be detected and the user; The audio transcription module is used to convert the audio obtained by the audio collection module into a structured audio text file, wherein the audio text file should include: the audio text of the agent channel and the customer channel, the number of sentences, the number of text words, the channel, start time, end time, speaking speed and volume corresponding to each sentence, the average speaking speed, average volume, number of sentences and number of text words of the entire audio, and the start time and end time of silence; The audio analysis module is used to parse the audio text file of the above-mentioned audio transcription module to determine whether the audio information segment in the audio text file conforms to the preset model rules, namely the text rules and the voice rules. If the two rules are configured at the same time, the audio information segment must conform to the two rules at the same time. After determining that the audio information segment hits the model rule, it is considered that the emotion represented by the model rule exists in the detected audio, and the corresponding audio data is marked with the emotion represented by the model rule.

Images