Voice call detection method, detection device, computer device and storage medium

Abstract

The application relates to a voice call detection method, a voice call detection device, computer equipment and a storage medium. The method comprises the following steps: when a customer service agent carries out voice call for a financial service in processing, acquiring voice audio generated by the customer service agent and call text corresponding to the voice audio; extracting audio features from the voice audio and text features from the call text; respectively carrying out quality detection processing on the audio features and the text features to obtain emotion detection results for the audio features and text detection results for the text features; the quality detection processing is used for quality evaluation on voice emotion expressed by the audio features and quality evaluation on text meaning expressed by the text features; and based on the emotion detection results and the text detection results, call detection results for the voice audio are obtained. The method can improve the efficiency of voice call detection and improve the accuracy of voice call detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a voice call detection method, a voice call detection device, a computer device, and a storage medium. Background Technology

[0002] In the banking industry, telephone communication between human customer service representatives and customers has become a widely used method. To detect and correct any misconduct by human customer service representatives during these telephone conversations, audio analysis of the recorded calls is necessary.

[0003] Traditional audio detection mainly involves manually reviewing the transcripts of recorded calls after a customer complaint or similar incident to identify the points in the phone conversation where violations occurred. This approach leads to a delay in recognizing violations during phone conversations and results in low detection quality and accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a voice call detection method, a voice call detection device, a computer device, a computer-readable storage medium, and a computer program product to address the aforementioned technical problems.

[0005] According to a first aspect of the present disclosure, a voice call detection method is provided, comprising:

[0006] When a customer service agent makes a voice call regarding a financial transaction in progress, the voice audio generated by the customer service agent and the corresponding call text are obtained.

[0007] Audio features are extracted from the audio voice, and text features are extracted from the call text;

[0008] The audio features and text features are subjected to quality detection processing to obtain emotion detection results for the audio features and text detection results for the text features; the quality detection processing is used to evaluate the quality of the speech emotion expressed by the audio features and the quality of the text meaning expressed by the text features.

[0009] Based on the emotion detection results and the text detection results, a call detection result is obtained for the voice audio, and the call detection result is fed back to the customer service agent.

[0010] In one exemplary embodiment, the audio features and the text features are represented based on corresponding feature vectors, and the emotion detection result and the text detection result are represented based on corresponding detection result.

[0011] The quality detection processing of the audio features and the text features includes:

[0012] The feature vectors corresponding to the audio features and the text features are respectively input into a first recurrent neural network to perform convolution processing on the feature vectors, thereby obtaining first state features for each time step, convolution features for the feature vectors, and region features. The time step represents the time domain in which the feature vector is located, and the time step corresponds to a neural network layer in the first recurrent neural network. The first state feature represents the transfer feature between two adjacent neural network layers, the convolution feature represents the output feature obtained after the feature vector is convolved through each neural network layer, and the region feature represents the local features within the feature enhancement range in the output feature.

[0013] The first state feature, the convolutional feature, and the region feature are input into an attention network to perform attention evaluation processing on the convolutional feature to obtain a first evaluation result, and attention evaluation processing on the region feature to obtain a second evaluation result; the attention evaluation processing is used to evaluate the degree of matching between the convolutional feature and the region feature and the first state feature, respectively.

[0014] The first evaluation result, the second evaluation result, and the first state feature are convolved based on the second recurrent neural network to obtain the second state feature for each time step.

[0015] The second state feature is subjected to linear activation processing to obtain the feature value probability distribution for the feature vector, and the detection result for the quality detection process is determined based on the feature values ​​included in the feature value probability distribution.

[0016] In an exemplary embodiment, the first recurrent neural network is a long short-term memory network; the step of convolving the feature vector through the first recurrent neural network to obtain first state features for each time step includes:

[0017] The feature vector is convolved by the Long Short-Term Memory network to obtain the initial state features of the feature vector corresponding to each time step, which are transmitted between the neural network layers of the Long Short-Term Memory network.

[0018] Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, hidden state features for each time step are obtained; the hidden state features characterize the state features of the feature vector transmitted between two adjacent neural network layers of the Long Short-Term Memory network.

[0019] In an exemplary embodiment, the step of performing attention evaluation processing on the convolutional features to obtain a first evaluation result, and performing attention evaluation processing on the region features to obtain a second evaluation result, includes:

[0020] Attention matching is performed on the first state feature and the convolutional feature based on the attention evaluation function to determine a first matching degree between the first state feature and the convolutional feature; and attention matching is performed on the first state feature and the region feature based on the attention evaluation function to determine a second matching degree between the first state feature and the region feature.

[0021] The first matching degree is subjected to linear activation processing to obtain a first probability value, and the first probability value is used as a first evaluation result for the convolutional feature; and the second matching degree is subjected to linear activation processing to obtain a second probability value, and the second probability value is used as a second evaluation result for the region feature.

[0022] In an exemplary embodiment, the step of performing convolution processing on the first evaluation result, the second evaluation result, and the first state feature to obtain a second state feature for each time step includes:

[0023] The first evaluation result and the convolutional feature are weighted and summed to obtain a first weighted value, and the first weighted value is used as a first context feature for the convolutional feature; and the second evaluation result and the region are weighted and summed to obtain a second weighted value, and the second weighted value is used as a second context feature for the convolutional feature.

[0024] The first context feature, the second context feature, and the first state feature are convolved to obtain the second state feature.

[0025] In an exemplary embodiment, the second recurrent neural network is a long short-term memory network; the step of convolving the first context feature, the second context feature, and the first state feature to obtain the second state feature includes:

[0026] Summing the first context feature and the second context feature yields the third context feature;

[0027] The third context feature and the first state feature are concatenated to obtain the concatenated feature;

[0028] The splicing features are convolved through the Long Short-Term Memory network to obtain the initial state features of the splicing features corresponding to each time step that are transmitted between the neural network layers of the Long Short-Term Memory network.

[0029] Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, hidden state features for each time step are obtained, and the hidden state features are used as the second state features for each time step in the second recurrent neural network.

[0030] In an exemplary embodiment, the step of performing linear activation processing on the second state feature to obtain a feature value probability distribution for the feature vector, and determining the detection result for the quality detection process based on the feature values ​​included in the feature value probability distribution, includes:

[0031] The second state feature is subjected to linear transformation and function activation processing to determine the probability distribution of each feature value for the second state feature;

[0032] In the probability distribution of each of the aforementioned feature values, the feature value with the largest corresponding probability distribution is taken as the detection result.

[0033] In one exemplary embodiment, the emotion detection result is represented based on emotion detection values, and the text detection result is represented based on text detection values.

[0034] The step of obtaining call detection results for the voice audio based on the emotion detection results and the text detection results includes:

[0035] The emotion detection value and the text detection value are weighted and summed to obtain a corresponding weighted value, which is then used as the call detection result for the speech audio.

[0036] In one exemplary embodiment, the method further includes:

[0037] If the weighted value is less than a preset threshold, the voice call is determined to be abnormal, and a corresponding abnormal prompt message is generated;

[0038] The error message will be relayed to the customer service representative.

[0039] The abnormality alert information includes one of the following: first information regarding the emotion detection result, second information regarding the text detection result, and third information regarding both the text detection result and the text detection result.

[0040] The first information is used to provide an abnormal prompt regarding the emotional state expressed in the voice audio.

[0041] The second information is used to provide an anomaly alert regarding the meaning of the text expressed in the call text;

[0042] The third piece of information is used to provide abnormal prompts for both the emotional tone expressed in the audio and the meaning of the text in the call.

[0043] According to a second aspect of the present disclosure, a voice call detection device is provided, comprising:

[0044] The information acquisition unit is configured to acquire the voice audio generated by the customer service agent and the corresponding call text when the customer service agent makes a voice call regarding a financial transaction being processed.

[0045] The feature extraction unit is configured to extract audio features from the speech audio and text features from the call text.

[0046] The feature detection unit is configured to perform quality detection processing on the audio features and the text features respectively, to obtain an emotion detection result for the audio features and a text detection result for the text features; the quality detection processing is used to evaluate the quality of the speech emotion expressed by the audio features and to evaluate the quality of the text meaning expressed by the text features.

[0047] The detection result unit is configured to perform a call detection result based on the emotion detection result and the text detection result, obtain a call detection result for the voice audio, and feed the call detection result back to the customer service agent.

[0048] According to a third aspect of the present disclosure, a computer device is provided, comprising:

[0049] processor;

[0050] Memory for storing the executable instructions of the processor;

[0051] The processor is configured to execute the executable instructions to implement the voice call detection method as described in any of the preceding claims.

[0052] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium including a computer program that, when executed by a processor of a computer device, enables the computer device to perform the voice call detection verification method as described in any of the preceding claims.

[0053] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including program instructions that, when executed by a processor of a computer device, enable the computer device to perform the voice call detection method as described in any of the preceding claims.

[0054] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0055] This method first acquires the audio and corresponding text of a customer service call conducted by a customer service agent regarding an ongoing financial transaction. Audio features are extracted from the audio, and text features are extracted from the text. Quality detection is then performed on both audio and text features to obtain emotion detection results for the audio features and text detection results for the text features. The quality detection process evaluates the quality of the emotional expression conveyed by the audio features and the text meaning conveyed by the text features. Based on the emotion detection and text detection results, a call detection result for the audio is obtained and fed back to the customer service agent. In this way, on the one hand, by acquiring the voice audio and corresponding call text generated by customer service agents when handling financial business, and then extracting the features of each voice audio and call text, the quality of the voice audio and call text can be detected using these features. This optimizes the process of voice call detection, reduces the consumption of manpower and time costs, and improves the efficiency of voice call detection. On the other hand, based on the extraction of features of each voice audio and call text, further quality evaluation is performed on the voice emotion expressed by the audio features and the text meaning expressed by the text features. By utilizing multiple factors for voice call detection, the accuracy of voice call detection is improved, enabling real-time monitoring of voice calls and helping to promptly detect and stop violations during calls.

[0056] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0057] Figure 1 This is an application environment diagram illustrating a voice call detection method according to an exemplary embodiment;

[0058] Figure 2 This is a flowchart illustrating a voice call detection method according to an exemplary embodiment;

[0059] Figure 3 This is a flowchart illustrating a step for obtaining a detection result according to an exemplary embodiment;

[0060] Figure 4 This is a flowchart illustrating a step for obtaining a second state feature according to an exemplary embodiment;

[0061] Figure 5 This is a flowchart illustrating a step for providing feedback error message according to an exemplary embodiment;

[0062] Figure 6 This is a flowchart illustrating a voice call detection method according to another exemplary embodiment;

[0063] Figure 7 This is a schematic diagram of a module for a voice call detection method according to another exemplary embodiment;

[0064] Figure 8 This is a flowchart illustrating a quality inspection process performed by a deep learning model according to an exemplary embodiment.

[0065] Figure 9 This is a block diagram illustrating a voice call detection device according to an exemplary embodiment;

[0066] Figure 10 This is a block diagram illustrating a computer device for voice call detection according to an exemplary embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, although the terms "first," "second," etc., are used repeatedly to describe various operations (or various thresholds, or various applications, or various instructions, or various elements), these operations (or thresholds, applications, instructions, or elements) should not be limited by these terms. These terms are only used to distinguish one operation (or threshold, application, instruction, or element) from another operation (or threshold, application, instruction, or element). For example, a first recurrent neural network can be called a second recurrent neural network, and a second recurrent neural network can be called a first recurrent neural network, without departing from the scope of this application. Both the first recurrent neural network and the second recurrent neural network relate to a type of neural network model in a detection model; they are simply not the same neural network model.

[0069] It should be noted that if the technical solution of this application involves personal information, the user must be clearly informed of the rules for processing personal information and their consent obtained before the product corresponding to the technical solution of this application processes the user's personal information. If the technical solution of this application involves sensitive personal information of the user, the user's individual consent must be obtained before the product corresponding to the technical solution of this application processes the user's sensitive personal information, and the requirement of "express consent" must also be met. For example, on the personal information processing device, the personal information processing rules can be clearly indicated by signage / information, or the user's personal authorization can be obtained through pop-up messages or by requesting the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0070] The voice call detection method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other network servers.

[0071] In some embodiments, reference Figure 1 First, when a customer service agent conducts a voice call regarding an ongoing financial transaction, server 104 acquires the audio and corresponding text of the voice call generated by the agent. Then, server 104 extracts audio features from the audio and text features from the text. Next, server 104 performs quality detection processing on the audio and text features respectively, obtaining emotion detection results for the audio features and text detection results for the text features. The quality detection processing is used to evaluate the quality of the emotional expression conveyed by the audio features and the text meaning conveyed by the text features. Finally, based on the emotion detection results and text detection results, server 104 obtains the call detection results for the audio and text and feeds these results back to the customer service agent.

[0072] In some embodiments, terminal 102 (such as a mobile terminal or a fixed terminal) can be implemented in various forms. Terminal 102 can be a mobile terminal including mobile phones, smartphones, laptops, portable handheld devices, personal digital assistants (PDAs), tablet computers (PADs), etc., or a fixed terminal including automated teller machines (ATMs), automated kiosks, digital TVs, desktop computers, fixed-line computers, etc.

[0073] Hereinafter, it is assumed that terminal 102 is a fixed terminal. However, those skilled in the art will understand that, if there are operations or elements specifically designed for mobile purposes, the construction according to the embodiments disclosed in this application can also be applied to mobile type terminal 102.

[0074] In some embodiments, the data processing component running on server 104 may load any of the various additional server applications and / or middleware applications being executed, such as HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), CGI (Common Gateway Interface), RDBMS (Relational Database Management System), etc.

[0075] In some embodiments, server 104 may be implemented using a standalone server or a server cluster consisting of multiple servers. Server 104 may be adapted to run one or more application services or software components that provide the terminal 102 described in the foregoing disclosure.

[0076] In some embodiments, the application service may include a service interface that provides information authentication to the user, as well as corresponding program services, etc. The software component may include, for example, an application (SDK) or client (APP) that performs information authentication functions based on authentication requests initiated by applications installed on the electronic device.

[0077] In some embodiments, the application or client with voice call detection function provided by server 104 includes a portal port that provides one-to-one application services to users in the foreground and multiple business systems that perform data processing in the background, so as to extend the voice call detection function to the APP or client, so that users can use and access the voice call detection function at any time and any place.

[0078] In some embodiments, users can input corresponding code data or control parameters into the APP or client through a preset input device or automatic control program to execute application services of the computer program in the server 104 and display application services in the user interface.

[0079] In some embodiments, the operating system running the app or client may include various versions of Microsoft... Apple and / or Linux operating system, various commercial or similar Operating systems (including but not limited to various GNU / Linux operating systems, Google) OS and / or mobile operating systems, such as Phone OS OS OS operating systems, as well as other online or offline operating systems, are not specifically limited here.

[0080] In some embodiments, such as Figure 2 As shown, a voice call detection method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the method includes the following steps:

[0081] Step S11: When a customer service agent makes a voice call regarding a financial transaction in progress, obtain the voice audio generated by the customer service agent and the corresponding call text.

[0082] In one embodiment, the customer seat is a customer service representative in the banking industry, and the financial business refers to the business in the banking industry that requires voice calls with bank customers, such as payment consultation between bank customers and customer service representatives, anti-fraud services between bank customers and customer service representatives, etc.

[0083] In one embodiment, when a customer service representative is making a voice call with a customer regarding a financial transaction, the server captures the voice audio generated by the customer service representative in real time and converts the voice audio into an audio sequence and a text sequence.

[0084] The voice server first identifies the call text in the voice audio, converts the call text into a text sequence word by word, and then converts the voice audio into a corresponding audio sequence based on the text sequence.

[0085] In this context, the text in the text sequence corresponds to the audio segments in the audio sequence.

[0086] Step S12: Extract audio features from the speech audio and extract text features from the call text.

[0087] In one embodiment, the server extracts features from the audio sequence corresponding to the speech audio to obtain audio features in the speech audio, and extracts features from the text sequence corresponding to the call text to obtain text features in the speech audio.

[0088] In one embodiment, audio features and text features are represented based on corresponding feature vectors, and the feature vectors are vector sequences corresponding to audio sequences / text sequences.

[0089] In one embodiment, the audio features include intonation features, speech rate features, and silence duration features, which are used to express the emotional features of the voice audio, that is, the audio features can express the emotions of the customer service personnel and their customers during the voice call.

[0090] In one embodiment, text features include keywords (such as prohibited words), text meaning features, etc., which are used to express the meaning features of voice audio, that is, the text features can express the meaning of the conversation between the customer service personnel and the customer during the voice call.

[0091] Step S13: Perform quality detection processing on the audio features and text features respectively to obtain the emotion detection results for the audio features and the text detection results for the text features.

[0092] In one embodiment, the quality detection process is used to evaluate the quality of the speech emotion expressed by the audio features and to evaluate the quality of the text meaning expressed by the text features.

[0093] In one embodiment, the emotion detection result is based on the emotion detection value representation, and the text detection result is based on the text detection value representation.

[0094] As an example, the server inputs the audio features of the call speech into a pre-trained audio quality inspection model. This model performs quality checks on the call speech to assess the emotion conveyed, obtaining a corresponding emotion detection value, which is then used as the emotion detection result. The emotion detection value is directly proportional to the quality assessment of the speech emotion; that is, a higher emotion detection value indicates a better expression of emotion, while a lower value indicates a worse expression of emotion.

[0095] As another example, the server inputs the text features of the call speech into a pre-trained text quality inspection model. This model performs a text meaning quality check on the call speech, obtaining a corresponding text detection value, which is then used as the text detection result. The text detection value is directly proportional to the quality assessment of the text meaning; that is, a higher text detection value for a given audio recording indicates a better understanding of the corresponding text meaning, while a lower text detection value indicates a worse understanding of the corresponding text meaning.

[0096] Step S14: Based on the emotion detection results and text detection results, obtain the call detection results for the voice audio, and feed the call detection results back to the customer service agent.

[0097] In one embodiment, the server obtains the call detection result for the voice audio by: performing a weighted summation of the emotion detection value representation and the text detection value to obtain the corresponding weighted value, and using the weighted value as the call detection result for the voice audio.

[0098] As an example, if the emotion detection score for the voice audio is 80 points (out of 100) and the text detection score for the call text is 70 points (out of 100), then the server calculates the corresponding weighted value using the formula 80×a + 70×b = C. Here, a and b are the corresponding weight values, and C is the calculated weighted value. The server then uses this weighted value C as the call detection result for the voice audio and sends it back to the customer service representative.

[0099] In the aforementioned voice call detection process, the server first acquires the audio and corresponding text of the voice call generated by the customer service agent when the agent conducts a voice call regarding the financial business being processed. Audio features are extracted from the audio, and text features are extracted from the text. Quality detection processing is then performed on the audio and text features to obtain emotion detection results for the audio features and text detection results for the text features. This quality detection processing is used to evaluate the quality of the emotional expression conveyed by the audio features and the text meaning conveyed by the text features. Based on the emotion detection results and text detection results, the call detection result for the audio is obtained and fed back to the customer service agent. In this way, on the one hand, by acquiring the voice audio and corresponding call text generated by customer service agents when handling financial business, and then extracting the features of each voice audio and call text, the quality of the voice audio and call text can be detected using these features. This optimizes the process of voice call detection, reduces the consumption of manpower and time costs, and improves the efficiency of voice call detection. On the other hand, based on the extraction of features of each voice audio and call text, further quality evaluation is performed on the voice emotion expressed by the audio features and the text meaning expressed by the text features. By utilizing multiple factors for voice call detection, the accuracy of voice call detection is improved, enabling real-time monitoring of voice calls and helping to promptly detect and stop violations during calls.

[0100] Those skilled in the art will understand that the methods disclosed in the above-described specific embodiments can be implemented in more concrete ways. For example, the above-described implementation of the server performing quality detection processing on audio features and text features separately is merely illustrative.

[0101] For example, the way the server evaluates the quality of the speech emotion expressed by the audio features; or the way the server evaluates the quality of the text meaning expressed by the text features, etc., is just one set of methods. In actual implementation, there may be other ways of dividing it. For example, the emotion detection results and the text detection results may be combined or integrated into another system, or some features may be ignored or not executed.

[0102] In one exemplary embodiment, see Figure 3 , Figure 3 This is a schematic flowchart illustrating an embodiment of obtaining detection results in this application. In step S13, the server performs quality detection processing on the audio features and text features respectively to obtain emotion detection results for the audio features and text detection results for the text features. This process can be implemented in the following way:

[0103] Step S131: Input the feature vectors corresponding to the audio features and text features into the first recurrent neural network respectively, so as to perform convolution processing on the feature vectors through the first recurrent neural network to obtain the first state features for each time step, the convolution features for the feature vectors, and the region features.

[0104] The first recurrent neural network is an LSTM network (Long Short-Term Memory).

[0105] In this context, "time step" refers to a time stamp, which is unique to sequence data (sound, text, etc.) models. For audio files or text, each time step can be a segment of speech or a word; that is, data is input into the model in units of time steps.

[0106] Here, the time step represents the time domain in which the feature vector is located, and the time step corresponds to the neural network layer in the first recurrent neural network.

[0107] In one embodiment, the first state feature characterizes the transmission features between two adjacent neural network layers.

[0108] In one embodiment, convolutional features characterize the output features obtained after the feature vector is convolved through each neural network layer.

[0109] In one embodiment, region features represent local features that fall within the feature enhancement range in the output features.

[0110] In one embodiment, the server performs convolution processing on the feature vectors using a first recurrent neural network to obtain first state features for each time step, including the following steps:

[0111] Step 1: Perform convolution processing on the feature vectors using a Long Short-Term Memory (LSTM) network to obtain the initial state features of the feature vectors corresponding to each time step as they are transmitted between the neural network layers of the LSM network.

[0112] The initial state feature is the cell state ct1 generated by the first LSTM network at time t.

[0113] Step 2: Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, obtain the hidden state features for each time step.

[0114] Among them, the hidden state features represent the state features that the feature vector is transmitted between two adjacent neural network layers in the Long Short-Term Memory network.

[0115] The hidden state feature is the hidden state ht1 generated by the first LSTM network at time t.

[0116] As an example, this convolution operation can be characterized based on the following formula:

[0117]

[0118] in, These represent the hidden state ht1 and cell state ct1 generated by the first-layer LSTM network at time t, respectively. fc_feats represents the average features of the audio / text, and [;] represents the connection symbol. This represents the word embedding matrix to be learned, where m represents the total number of words in the vocabulary and n represents the word encoding dimension.

[0119] LSTM is a Long Short-Term Memory network, and the hidden state ht1 and cell state ct1 can be represented based on the following formula:

[0120]

[0121] c t =·f t ⊙c t-1 +·i t ⊙g t ;

[0122]

[0123] Among them, i t f t o t These represent the input gate, forget gate, and output gate, respectively. σ represents the Sigmoid activation function, T represents linear change, ⊙ represents element-wise multiplication, and ct represents the cell state. That is, first obtain the cell state ct1, and then obtain the hidden state ht1 based on the cell state ct1.

[0124] Specifically, the VGGish model is used to perform convolution operations on the audio encoding vector sequence to obtain the convolution features corresponding to the audio encoding vector sequence.

[0125] Step S132: Input the first state features, convolutional features and region features into the attention network, so as to obtain the first evaluation result by performing attention evaluation processing on the convolutional features and the second evaluation result by performing attention evaluation processing on the region features.

[0126] In one embodiment, the attention evaluation process is used to evaluate the degree of matching between the convolutional features and the region features and the first state features, respectively.

[0127] In one embodiment, the server performs attention evaluation processing on convolutional features to obtain a first evaluation result, and performs attention evaluation processing on region features to obtain a second evaluation result, including the following steps:

[0128] Step 1: Perform attention matching processing on the first state features and convolutional features based on the attention evaluation function to determine the first matching degree between the first state features and convolutional features; and perform attention matching processing on the first state features and region features based on the attention evaluation function to determine the second matching degree between the first state features and region features.

[0129] As an example, the first matching degree e1 and the second matching degree e2 are characterized based on the following formula:

[0130]

[0131]

[0132]

[0133] Among them, f att It is an attention scoring function; These are the parameters learned by the attention enhancement network, which is a type of MLP (fully connected neural network). d is the hidden layer size of the attention enhancement network, r is the hidden layer size of the first LSTM network, and N is the size of the visual features in the attention enhancement network. It is a pointer vector full of 1s, where conv_feats represents the convolutional features of audio / text and pool_feats represents the regional features of audio / text.

[0134] Step 2: Perform linear activation on the first matching degree to obtain a first probability value, and use the first probability value as the first evaluation result for the convolutional feature; and perform linear activation on the second matching degree to obtain a second probability value, and use the second probability value as the second evaluation result for the region feature.

[0135] As an example, the first attention weight αt1 and the second attention weight αt2 can be calculated based on the following formula:

[0136]

[0137]

[0138] The sum of the first attention weight αt1 and the second attention weight αt2 is 1.

[0139] Here, soft max represents the activation function corresponding to linear activation processing.

[0140] Step S133: Perform convolution processing on the first evaluation result, the second evaluation result, and the first state features based on the second recurrent neural network to obtain the second state features for each time step.

[0141] In one embodiment, the second recurrent neural network is also an LSTM network. The second recurrent neural network has a similar structure and function to the first recurrent neural network, both of which are used to determine the state features of the corresponding time step.

[0142] In one exemplary embodiment, see Figure 4 , Figure 4 This is a schematic flowchart illustrating an embodiment of obtaining the second state feature in this application. In step S14, the process by which the server obtains the second state feature for each time step can be implemented in the following manner:

[0143] Step a1: Perform weighted summation on the first evaluation result and the convolutional features to obtain a first weighted value, and use the first weighted value as the first context feature for the convolutional features; and perform weighted summation on the second evaluation result and the region with features to obtain a second weighted value, and use the second weighted value as the second context feature for the convolutional features.

[0144] Step a2: Perform convolution processing on the first context feature, the second context feature, and the first state feature to obtain the second state feature.

[0145] In one embodiment, the server obtains the second state characteristic by the following steps:

[0146] Step 1: Summate the first context feature and the second context feature to obtain the third context feature.

[0147] Step 2: Concatenate the third context feature and the first state feature to obtain the concatenated feature.

[0148] Step 3: Perform convolution processing on the spliced ​​features through a Long Short-Term Memory (LSTM) network to obtain the initial state features of the spliced ​​features corresponding to each time step as they are transmitted between the neural network layers of the LSM network.

[0149] Step 4: Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, obtain the hidden state features for each time step, and use the hidden state features as the second state features for each time step in the second recurrent neural network.

[0150] As an example, the server first performs a convolution operation on the concatenated comprehensive context vector a3 and the first state feature ht1 to determine the cell state ct2 corresponding to each time step, and then obtains the hidden state feature ht2 based on the cell state ct2.

[0151] The convolution operation is characterized by the following formula:

[0152]

[0153] In this context, cell state ct2 is the initial state feature generated in this LSTM network, hidden state ht2 is the second state feature generated in this LSTM network, cell state ct1 is the initial state feature generated in the previous LSTM network, and hidden state ht1 is the second state feature generated in the previous LSTM network.

[0154] Step S134: Perform linear activation processing on the second state features to obtain the feature value probability distribution for the feature vector.

[0155] In one embodiment, the server performs linear transformation and function activation processing on the second state feature to determine the probability distribution of each feature value for the second state feature.

[0156] The linear transformation is performed using the MLP transformation, and the function activation is performed using the soft max function activation.

[0157] The probability distribution of each feature value for the second state feature can be represented by the following formula:

[0158]

[0159] in, These are the parameters of the LSTM, r is the size of the LSTM hidden layer, and D is the size of the vocabulary in the LSTM.

[0160] In one embodiment, if the input feature is an audio feature, then the probability distribution of each feature value of the second state feature at the current time step can be defined as p(y t |I, y0, ..., y t-1 If the input features are text features, then the probability distribution of each feature value of the second state feature at the current time step can be defined as p(y). txt |I, y0, ..., y t-1) representation.

[0161] Step S135: Determine the detection result for quality detection processing based on the feature values ​​included in the feature value probability distribution.

[0162] In one embodiment, the server selects the feature value with the largest probability distribution from the probability distributions of each feature value as the detection result.

[0163] In one exemplary embodiment, see Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of the feedback of abnormal prompt information in this application. After step S14, that is, after the server obtains the call detection results for the voice audio, it can be implemented in the following way:

[0164] Step b1: If the weighted value is less than the preset threshold, determine that the voice call is abnormal and generate the corresponding abnormal prompt information.

[0165] Step b2: Report the error message to customer service.

[0166] The abnormality alert information includes one of the following: first information based on the emotion detection result, second information based on the text detection result, and third information based on both the text detection result and the text detection result.

[0167] The first piece of information is used to provide abnormal prompts for the emotional state expressed in the audio.

[0168] The second piece of information is used to provide anomalies in the meaning of the text expressed in the call.

[0169] The third piece of information is used to provide abnormal prompts for both the emotional tone expressed in the audio and the meaning of the text in the call.

[0170] To more clearly illustrate the voice call detection method provided in this disclosure, a specific embodiment is described below. In an exemplary embodiment, reference is made to... Figure 6 and Figure 7 , Figure 6 This is a flowchart illustrating a voice call detection method according to another exemplary embodiment. Figure 7 This is a block diagram illustrating a voice call detection method according to another exemplary embodiment. The voice call detection method is used in a server and specifically includes the following:

[0171] Step S21: The voice customer service representative initiates a voice call request to the user from the outbound call workbench, thereby enabling a voice call with the user.

[0172] Step S22: Obtain the voice audio of the voice customer service and convert the voice audio into an audio sequence and a text sequence.

[0173] The voice server first identifies the call text in the voice audio, converts the call text into a text sequence word by word, and then converts the voice audio into a corresponding audio sequence based on the text sequence.

[0174] In this context, the text in the text sequence corresponds to the audio segments in the audio sequence.

[0175] Step S23: Extract the text feature sequence from the text sequence and extract the audio feature sequence from the audio sequence.

[0176] Step S24: Send the text feature sequence to the encoder for feature encoding to obtain the text encoded vector sequence, and send the audio feature sequence to the encoder for feature encoding to obtain the audio encoded vector sequence.

[0177] Step S25: Send the text encoding vector sequence to the deep learning model for text content quality inspection to obtain the text quality inspection result for speech audio; and send the audio encoding vector sequence to the deep neural network for audio content quality inspection to obtain the audio quality inspection result for speech audio.

[0178] Among them, text content quality inspection processing is used to detect illegal text content generated by voice customer service during the call; audio content quality inspection processing is used to detect the tone, speaking speed and silence duration of voice customer service during the call audio.

[0179] The deep learning model consists of a first recurrent neural network (an LSTM network), an attention enhancement network, a second recurrent neural network (an LSTM network), and an activation network, which are connected in sequence.

[0180] In one exemplary embodiment, reference is made to Figure 8 , Figure 8 The flowchart illustrating the quality inspection process of a deep learning model according to an exemplary embodiment specifically includes the following steps S31-S40:

[0181] Step S31: Use the text encoding vector sequence / audio encoding vector sequence as input to the first recurrent neural network.

[0182] Each encoded vector in the text encoded vector sequence / audio encoded vector sequence corresponds to a time step in the first LSTM network.

[0183] In this context, "time step" refers to a time stamp, which is unique to sequence data (sound, text, etc.) models. For audio files or text, each time step can be a segment of speech or a word; that is, data is input into the model in units of time steps.

[0184] Step S32: The first recurrent neural network performs convolution operations on the text encoding vector sequence and the audio encoding vector sequence respectively, determines the hidden state ht1 and cell state ct1 corresponding to each time step, and obtains the convolution features and region features corresponding to the text encoding vector sequence and the audio encoding vector sequence.

[0185] Among them, the regional features are the feature regions where the convolutional features to be enhanced are located.

[0186] The convolution operation is characterized by the following formula:

[0187]

[0188] in, These represent the hidden state ht1 and cell state ct1 generated by the first-layer LSTM network at time t, respectively. fc_feats represents the average features of the audio / text, and [;] represents the connection symbol. This represents the word embedding matrix to be learned, where m represents the total number of words in the vocabulary and n represents the word encoding dimension.

[0189] Among them, the recurrent neural network is a long short-term memory network, and the hidden state ht1 and the cell state ct1 can be represented based on the following formula:

[0190]

[0191] c t =·f t ⊙c t-1 +·i t ⊙g t ;

[0192]

[0193] Among them, i t f t o t These represent the input gate, forget gate, and output gate, respectively. σ represents the Sigmoid activation function, T represents linear change, ⊙ represents element-wise multiplication, and ct represents the cell state. That is, first obtain the cell state ct1, and then obtain the hidden state ht1 based on the cell state ct1.

[0194] Specifically, the VGGish model is used to perform convolution operations on the audio encoding vector sequence to obtain the convolution features corresponding to the audio encoding vector sequence.

[0195] Step S33: The first recurrent neural network uses the convolutional features, region features, and the hidden state ht1 at each time step corresponding to the text encoding vector sequence / audio encoding vector sequence as input to the attention enhancement network.

[0196] Step S34: The attention enhancement network calculates the matching degree between the convolutional features of the text encoding vector sequence / audio encoding vector sequence and the hidden state ht1, and obtains the matching score et1; and the attention enhancement network calculates the matching degree between the regional features of the text encoding vector sequence / audio encoding vector sequence and the hidden state ht1, and obtains the matching score et2.

[0197] The matching scores e1 and e2 are represented by the following formulas:

[0198]

[0199]

[0200]

[0201] Among them, f att It is an attention scoring function; These are the parameters learned by the attention enhancement network, which is a type of MLP (fully connected neural network). d is the hidden layer size of the attention enhancement network, r is the hidden layer size of the first recurrent neural network, and N is the size of the visual features in the attention enhancement network. It is a pointer vector full of 1s, where conv_feats represents the convolutional features of audio / text and pool_feats represents the regional features of audio / text.

[0202] Step S35: Calculate the attention weight αt1 for matching score et1 and the attention weight αt2 for matching score et2 using the activation function.

[0203] The activation function can be the softmax function, and the attention weights can be calculated based on the following formula:

[0204]

[0205]

[0206] The sum of attention weights at1 and at2 is 1.

[0207] Step S36: Perform a weighted summation on the attention weights ɑt1 to obtain the context vector a1 for the convolutional features; and perform a weighted summation on the attention weights ɑt2 to obtain the context vector a2 for the region features.

[0208] The weighted summation of attention weights can be calculated based on the following formula:

[0209]

[0210]

[0211] Step S37: Add the context vector a1 for convolutional features and the context vector a2 for region features to obtain the comprehensive context vector a3.

[0212] Step S38: Concatenate the integrated context vector a3 and the hidden state ht1 corresponding to each time step, and use it as the input of the second recurrent neural network.

[0213] Step S39: The second recurrent neural network performs a convolution operation on the concatenated integrated context vector a3 and the hidden state ht1 to determine the hidden state ht2 and the cell state ct2 corresponding to each time step.

[0214] The convolution operation is characterized by the following formula:

[0215]

[0216] First, the cell state ct2 is obtained, and then the hidden state ht2 is obtained based on the cell state ct2.

[0217] Step S40: The hidden state ht2 is subjected to feature transformation and function activation processing for the fully connected neural network in sequence, generating the final quality inspection score probability distribution.

[0218] The final quality inspection score probability distribution is shown in the following formula:

[0219]

[0220] in, These are the training parameters, r is the size of the LSTM hidden layer, and D is the size of the vocabulary.

[0221] Given audio / text I, the decoder needs to generate a corresponding quality inspection score y. The goal of training the model is to maximize the probability of finding the correct quality inspection score by optimizing the parameter values ​​θ. * The specific implementation is shown in the following formula:

[0222]

[0223] Where θ represents the model parameters, I represents the input audio / text, and y represents the quality control score.

[0224] Using the chain rule, the joint probability distribution described above can be broken down into the following formula:

[0225]

[0226] The probability distribution of the audio quality inspection score generated at the current time step is defined as p(y t |I, y0, ..., y t-1 The text quality inspection probability distribution generated at the current time step is based on p(y). txt |I, y0, ..., y t-1 ) representation.

[0227] Step S26: Generate call quality inspection results based on text quality inspection results and audio quality inspection results.

[0228] The call quality inspection results include whether the quality inspection passed or failed.

[0229] Step S27: If the call quality inspection result is a failure, a prompt message will be generated.

[0230] Step S28: Send the prompt message to the outbound call console to remind the voice customer service representative, and send the call quality inspection results to the voice monitoring console for further processing.

[0231] In this way, on the one hand, by acquiring the voice audio and corresponding call text generated by customer service agents when handling financial business, and then extracting the features of each voice audio and call text, the quality of the voice audio and call text can be detected using these features. This optimizes the process of voice call detection, reduces the consumption of manpower and time costs, and improves the efficiency of voice call detection. On the other hand, based on the extraction of features of each voice audio and call text, further quality evaluation is performed on the voice emotion expressed by the audio features and the text meaning expressed by the text features. By utilizing multiple factors for voice call detection, the accuracy of voice call detection is improved, enabling real-time monitoring of voice calls and helping to promptly detect and stop violations during calls.

[0232] It should be understood that, although Figures 2-8 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-8At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0233] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0234] Figure 9 This is a block diagram for voice call detection provided in an embodiment of this application. (Refer to...) Figure 9 The voice call detection 20 includes: an information acquisition unit 21, a feature extraction unit 22, a feature detection unit 23, and a detection result unit 24.

[0235] The information acquisition unit 21 is configured to acquire the voice audio generated by the customer service agent and the corresponding call text when the customer service agent makes a voice call for the financial business being processed.

[0236] The feature extraction unit 22 is configured to extract audio features from the speech audio and text features from the call text.

[0237] The feature detection unit 23 is configured to perform quality detection processing on the audio features and the text features respectively, to obtain emotion detection results for the audio features and text detection results for the text features; the quality detection processing is used to evaluate the quality of the speech emotion expressed by the audio features and to evaluate the quality of the text meaning expressed by the text features.

[0238] The detection result unit 24 is configured to perform a call detection result based on the emotion detection result and the text detection result, and then feed the call detection result back to the customer service agent.

[0239] In one exemplary embodiment, the audio features and the text features are represented based on corresponding feature vectors, and the emotion detection result and the text detection result are represented based on corresponding detection result.

[0240] The quality detection processing of the audio features and the text features includes:

[0241] The feature vectors corresponding to the audio features and the text features are respectively input into a first recurrent neural network to perform convolution processing on the feature vectors, thereby obtaining first state features for each time step, convolution features for the feature vectors, and region features. The time step represents the time domain in which the feature vector is located, and the time step corresponds to a neural network layer in the first recurrent neural network. The first state feature represents the transfer feature between two adjacent neural network layers, the convolution feature represents the output feature obtained after the feature vector is convolved through each neural network layer, and the region feature represents the local features within the feature enhancement range in the output feature.

[0242] The first state feature, the convolutional feature, and the region feature are input into an attention network to perform attention evaluation processing on the convolutional feature to obtain a first evaluation result, and attention evaluation processing on the region feature to obtain a second evaluation result; the attention evaluation processing is used to evaluate the degree of matching between the convolutional feature and the region feature and the first state feature, respectively.

[0243] The first evaluation result, the second evaluation result, and the first state feature are convolved based on the second recurrent neural network to obtain the second state feature for each time step.

[0244] The second state feature is subjected to linear activation processing to obtain the feature value probability distribution for the feature vector, and the detection result for the quality detection process is determined based on the feature values ​​included in the feature value probability distribution.

[0245] In an exemplary embodiment, the first recurrent neural network is a long short-term memory network; the step of convolving the feature vector through the first recurrent neural network to obtain first state features for each time step includes:

[0246] The feature vector is convolved by the Long Short-Term Memory network to obtain the initial state features of the feature vector corresponding to each time step, which are transmitted between the neural network layers of the Long Short-Term Memory network.

[0247] Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, hidden state features for each time step are obtained; the hidden state features characterize the state features of the feature vector transmitted between two adjacent neural network layers of the Long Short-Term Memory network.

[0248] In an exemplary embodiment, the step of performing attention evaluation processing on the convolutional features to obtain a first evaluation result, and performing attention evaluation processing on the region features to obtain a second evaluation result, includes:

[0249] Attention matching is performed on the first state feature and the convolutional feature based on the attention evaluation function to determine a first matching degree between the first state feature and the convolutional feature; and attention matching is performed on the first state feature and the region feature based on the attention evaluation function to determine a second matching degree between the first state feature and the region feature.

[0250] The first matching degree is subjected to linear activation processing to obtain a first probability value, and the first probability value is used as a first evaluation result for the convolutional feature; and the second matching degree is subjected to linear activation processing to obtain a second probability value, and the second probability value is used as a second evaluation result for the region feature.

[0251] In an exemplary embodiment, the step of performing convolution processing on the first evaluation result, the second evaluation result, and the first state feature to obtain a second state feature for each time step includes:

[0252] The first evaluation result and the convolutional feature are weighted and summed to obtain a first weighted value, and the first weighted value is used as a first context feature for the convolutional feature; and the second evaluation result and the region are weighted and summed to obtain a second weighted value, and the second weighted value is used as a second context feature for the convolutional feature.

[0253] The first context feature, the second context feature, and the first state feature are convolved to obtain the second state feature.

[0254] In an exemplary embodiment, the second recurrent neural network is a long short-term memory network; the step of convolving the first context feature, the second context feature, and the first state feature to obtain the second state feature includes:

[0255] Summing the first context feature and the second context feature yields the third context feature;

[0256] The third context feature and the first state feature are concatenated to obtain the concatenated feature;

[0257] The splicing features are convolved through the Long Short-Term Memory network to obtain the initial state features of the splicing features corresponding to each time step that are transmitted between the neural network layers of the Long Short-Term Memory network.

[0258] Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, hidden state features for each time step are obtained, and the hidden state features are used as the second state features for each time step in the second recurrent neural network.

[0259] In an exemplary embodiment, the step of performing linear activation processing on the second state feature to obtain a feature value probability distribution for the feature vector, and determining the detection result for the quality detection process based on the feature values ​​included in the feature value probability distribution, includes:

[0260] The second state feature is subjected to linear transformation and function activation processing to determine the probability distribution of each feature value for the second state feature;

[0261] In the probability distribution of each of the aforementioned feature values, the feature value with the largest corresponding probability distribution is taken as the detection result.

[0262] In one exemplary embodiment, the emotion detection result is represented based on emotion detection values, and the text detection result is represented based on text detection values.

[0263] The step of obtaining call detection results for the voice audio based on the emotion detection results and the text detection results includes:

[0264] The emotion detection value and the text detection value are weighted and summed to obtain a corresponding weighted value, which is then used as the call detection result for the speech audio.

[0265] In an exemplary embodiment, the voice call detection device 20 is further configured to:

[0266] If the weighted value is less than a preset threshold, the voice call is determined to be abnormal, and a corresponding abnormal prompt message is generated;

[0267] The error message will be relayed to the customer service representative.

[0268] The abnormality alert information includes one of the following: first information regarding the emotion detection result, second information regarding the text detection result, and third information regarding both the text detection result and the text detection result.

[0269] The first information is used to provide an abnormal prompt regarding the emotional state expressed in the voice audio.

[0270] The second information is used to provide an anomaly alert regarding the meaning of the text expressed in the call text;

[0271] The third piece of information is used to provide abnormal prompts for both the emotional tone expressed in the audio and the meaning of the text in the call.

[0272] Figure 10 This is a block diagram of a computer device 30 provided in an embodiment of this application. For example, the computer device 30 can be an electronic device, an electronic component, or a server array, etc. (Refer to...) Figure 10 The computer device 30 includes a processor 31, which may be a collection of processors, including one or more processors. The computer device 30 also includes memory resources represented by memory 32, where computer programs, such as application programs, are stored. The computer programs stored in memory 32 may include one or more modules, each corresponding to a set of executable instructions. Furthermore, the processor 31 is configured to implement the voice call detection method described above when executing the executable instructions.

[0273] In some embodiments, computer device 30 is an electronic device whose computing system can run one or more operating systems, including any operating systems discussed above and any commercially available server operating system. Computer device 30 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, super servers, database servers, etc. Exemplary database servers include, but are not limited to, commercially available database servers from companies such as IBM.

[0274] In some embodiments, the processor 31 typically controls the overall operation of the computer device 30, such as operations associated with display, data processing, data communication, and recording operations. The processor 31 may include one or more processor components to execute computer programs to perform all or part of the steps of the methods described above. Furthermore, the processor components may include one or more modules to facilitate interaction between the processor components and other components. For example, the processor components may include a multimedia module to facilitate control of the interaction between the user computer device 30 and the processor 31 using multimedia components.

[0275] In some embodiments, the processor component in processor 31 may also be referred to as a CPU (Central Processing Unit). The processor component may be an electronic chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor component. Furthermore, the processor component may be implemented using integrated circuit chips.

[0276] In some embodiments, memory 32 is configured to store various types of data to support operation on computer device 30. Examples of such data include instructions for any application or method operating on computer device 30, acquired data, messages, images, videos, etc. Memory 32 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0277] In some embodiments, the memory 32 can be a memory module, TF card, etc., and can store all information in the computer device 30, including the input raw data, computer programs, intermediate running results, and final running results. In some embodiments, it stores and retrieves information according to the location specified by the processor. In some embodiments, the computer device 30 has a memory function and can ensure normal operation because of the memory 32. In some embodiments, the memory 32 of the computer device 30 can be classified into main memory (RAM) and auxiliary memory (external memory) according to its purpose, or it can be classified into external memory and internal memory. External memory is usually magnetic media or optical discs, which can store information for a long time. RAM refers to the storage component on the motherboard, which is used to store the currently executing data and programs, but it is only used to temporarily store programs and data. The data will be lost when the power is turned off or disconnected.

[0278] In some embodiments, the computer device 30 may further include: a power supply component 33 configured to perform power management of the computer device 30, a wired or wireless network interface 34 configured to connect the computer device 30 to a network, and an input / output (I / O) interface 35. The computer device 30 may operate on an operating system stored in memory 32, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0279] In some embodiments, power supply component 33 provides power to various components of computer device 30. Power supply component 33 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to computer device 30.

[0280] In some embodiments, the wired or wireless network interface 34 is configured to facilitate wired or wireless communication between the computer device 30 and other devices. The computer device 30 may access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof.

[0281] In some embodiments, the wired or wireless network interface 34 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the wired or wireless network interface 34 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0282] In some embodiments, the input / output (I / O) interface 35 provides an interface between the processor 31 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a power button, and a lock button.

[0283] This application provides a block diagram of a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the voice call detection method described above.

[0284] If the integrated units of the various functional units in the various embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer-readable storage medium includes a number of instructions in a computer program to cause a computer device (which may be a personal computer, system server, or network device, etc.), an electronic device (e.g., MP3, MP4, etc., or a mobile phone, tablet computer, wearable device, etc., or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0285] This application provides a block diagram of a computer program product. The computer program product includes program instructions that can be executed by a server's processor to implement the voice call detection method described above.

[0286] Those skilled in the art will understand that embodiments of this application can provide a voice call detection method, a voice call detection device 20, a computer device 30, a computer-readable storage medium, or a computer program product. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer program instructions (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0287] This application is described with reference to flowchart illustrations and / or block diagrams of a voice call detection method, voice call detection apparatus 20, computer device 30, computer-readable storage medium, or computer program product according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by the computer program product. These computer program products can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that program instructions executable by the processor of the computer or other programmable data processing device generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0288] These computer program products may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that program instructions stored in the computer program product produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0289] These program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing program instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0290] It should be noted that the various methods, apparatuses, electronic devices, computer-readable storage media, computer program products, etc. described above may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0291] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0292] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for detecting voice calls, characterized in that, The method includes: When a customer service agent makes a voice call regarding a financial transaction in progress, the voice audio generated by the customer service agent and the corresponding call text are obtained. Audio features are extracted from the speech audio, and text features are extracted from the call text; the audio features and the text features are represented based on corresponding feature vectors; The feature vectors corresponding to the audio features and the text features are respectively input into a first recurrent neural network to perform convolution processing on the feature vectors, thereby obtaining first state features for each time step, convolution features for the feature vectors, and region features. The time step represents the time domain in which the feature vector is located, and the time step corresponds to a neural network layer in the first recurrent neural network. The first state feature represents the transfer feature between two adjacent neural network layers, the convolution feature represents the output feature obtained after the feature vector is convolved through each neural network layer, and the region feature represents the local features within the feature enhancement range in the output feature. The first state feature, the convolutional feature, and the region feature are input into an attention network to perform attention evaluation processing on the convolutional feature to obtain a first evaluation result, and attention evaluation processing on the region feature to obtain a second evaluation result; the attention evaluation processing is used to evaluate the degree of matching between the convolutional feature and the region feature and the first state feature, respectively. The first evaluation result, the second evaluation result, and the first state feature are convolved based on the second recurrent neural network to obtain the second state feature for each time step. Linear activation processing is applied to the second state feature to obtain a feature value probability distribution for the feature vector. Based on the feature values ​​included in the feature value probability distribution, the detection results of quality detection processing for the audio feature and the text feature are determined respectively, resulting in an emotion detection result for the audio feature and a text detection result for the text feature. The emotion detection result and the text detection result are represented based on the corresponding detection results. The quality detection processing is used to evaluate the quality of the speech emotion expressed by the audio feature and the quality of the text meaning expressed by the text feature. Based on the emotion detection results and the text detection results, a call detection result is obtained for the voice audio, and the call detection result is fed back to the customer service agent.

2. The method according to claim 1, characterized in that, The first recurrent neural network is a long short-term memory network; the step of convolving the feature vector through the first recurrent neural network to obtain the first state features for each time step includes: The feature vector is convolved by the Long Short-Term Memory network to obtain the initial state features of the feature vector corresponding to each time step, which are transmitted between the neural network layers of the Long Short-Term Memory network. Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, hidden state features for each time step are obtained; the hidden state features characterize the state features of the feature vector transmitted between two adjacent neural network layers of the Long Short-Term Memory network.

3. The method according to claim 2, characterized in that, The process of performing attention evaluation processing on the convolutional features to obtain a first evaluation result, and performing attention evaluation processing on the region features to obtain a second evaluation result, includes: Attention matching is performed on the first state feature and the convolutional feature based on the attention evaluation function to determine a first matching degree between the first state feature and the convolutional feature; and attention matching is performed on the first state feature and the region feature based on the attention evaluation function to determine a second matching degree between the first state feature and the region feature. The first matching degree is subjected to linear activation processing to obtain a first probability value, and the first probability value is used as a first evaluation result for the convolutional feature; and the second matching degree is subjected to linear activation processing to obtain a second probability value, and the second probability value is used as a second evaluation result for the region feature.

4. The method according to any one of claims 2-3, characterized in that, The convolutional processing of the first evaluation result, the second evaluation result, and the first state feature to obtain the second state feature for each time step includes: The first evaluation result and the convolutional feature are weighted and summed to obtain a first weighted value, and the first weighted value is used as a first context feature for the convolutional feature; and the second evaluation result and the region are weighted and summed to obtain a second weighted value, and the second weighted value is used as a second context feature for the convolutional feature. The first context feature, the second context feature, and the first state feature are convolved to obtain the second state feature.

5. The method according to claim 4, characterized in that, The second recurrent neural network is a long short-term memory network; the step of convolving the first context feature, the second context feature, and the first state feature to obtain the second state feature includes: Summing the first context feature and the second context feature yields the third context feature; The third context feature and the first state feature are concatenated to obtain the concatenated feature; The splicing features are convolved through the Long Short-Term Memory network to obtain the initial state features of the splicing features corresponding to each time step that are transmitted between the neural network layers of the Long Short-Term Memory network. Based on the initial state features corresponding to each time step and the output gate features corresponding to each neural network layer, hidden state features for each time step are obtained, and the hidden state features are used as the second state features for each time step in the second recurrent neural network.

6. The method according to claim 4, characterized in that, The step of performing linear activation processing on the second state feature to obtain a feature value probability distribution for the feature vector, and determining the detection result for the quality detection process based on the feature values ​​included in the feature value probability distribution, includes: The second state feature is subjected to linear transformation and function activation processing to determine the probability distribution of each feature value for the second state feature; In the probability distribution of each of the aforementioned feature values, the feature value with the largest corresponding probability distribution is taken as the detection result.

7. The method according to claim 1, characterized in that, The emotion detection result is based on the emotion detection value representation, and the text detection result is based on the text detection value representation; The step of obtaining call detection results for the voice audio based on the emotion detection results and the text detection results includes: The emotion detection value and the text detection value are weighted and summed to obtain a corresponding weighted value, which is then used as the call detection result for the speech audio.

8. The method according to claim 7, characterized in that, The method further includes: If the weighted value is less than a preset threshold, the voice call is determined to be abnormal, and a corresponding abnormal prompt message is generated; The error message will be relayed to the customer service representative. The abnormality alert information includes one of the following: first information regarding the emotion detection result, second information regarding the text detection result, and third information regarding both the text detection result and the text detection result. The first information is used to provide an abnormal prompt regarding the emotional state expressed in the voice audio. The second information is used to provide an anomaly alert regarding the meaning of the text expressed in the call text; The third piece of information is used to provide abnormal prompts for both the emotional tone expressed in the audio and the meaning of the text in the call.

9. A voice call detection device, characterized in that, include: The information acquisition unit is configured to acquire the voice audio generated by the customer service agent and the corresponding call text when the customer service agent makes a voice call regarding a financial transaction being processed. The feature extraction unit is configured to extract audio features from the speech audio and text features from the call text; the audio features and the text features are represented based on corresponding feature vectors. The feature detection unit is configured to input the feature vectors corresponding to the audio features and the text features into a first recurrent neural network, respectively, so as to perform convolution processing on the feature vectors through the first recurrent neural network to obtain first state features for each time step, convolution features for the feature vectors, and region features; the time step represents the time domain in which the feature vectors are located, and the time step corresponds to the neural network layer in the first recurrent neural network; the first state feature represents the transfer feature between two adjacent neural network layers, the convolution feature represents the output feature obtained after the feature vectors are convolved through each of the neural network layers, and the region feature represents the local features in the output features that are within the feature enhancement range; The first state feature, the convolutional feature, and the region feature are input into an attention network to perform attention evaluation processing on the convolutional feature to obtain a first evaluation result, and attention evaluation processing on the region feature to obtain a second evaluation result; the attention evaluation processing is used to evaluate the degree of matching between the convolutional feature and the region feature and the first state feature, respectively. Convolution processing is performed based on the first evaluation result, the second evaluation result, and the first state feature to obtain the second state feature for each time step; Linear activation processing is applied to the second state feature to obtain a feature value probability distribution for the feature vector. Based on the feature values ​​included in the feature value probability distribution, the detection results of quality detection processing for the audio feature and the text feature are determined respectively, resulting in an emotion detection result for the audio feature and a text detection result for the text feature. The emotion detection result and the text detection result are represented based on the corresponding detection results. The quality detection processing is used to evaluate the quality of the speech emotion expressed by the audio feature and the quality of the text meaning expressed by the text feature. The detection result unit is configured to perform a call detection result based on the emotion detection result and the text detection result, obtain a call detection result for the voice audio, and feed the call detection result back to the customer service agent.

10. A computer device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the voice call detection method as described in any one of claims 1 to 8.

11. A computer-readable storage medium comprising program data, characterized in that, When the program data is executed by the processor of a computer device, the computer device is able to perform the voice call detection method as described in any one of claims 1 to 8.

12. A computer program product, the computer program product comprising program instructions, characterized in that, When the program instructions are executed by the processor of a computer device, the computer device is able to perform the voice call detection method as described in any one of claims 1 to 8.

Images