A language behavior dialogue strategy method based on multimodal deep learning
Through the multimodal deep learning language behavior dialogue strategy method, combined with the modal features of speech, text and image, the problem of insufficient accuracy and reliability of single-modal speech recognition in complex digital sequence recognition is solved, and the bank card number is realized Efficient identification.
Patent Information
- Application Number
- CN202411336660.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-25
AI Technical Summary
Existing single-modal speech recognition technology is difficult to take into account accuracy and reliability when processing complex digital sequences such as bank card numbers, especially affected by customer accents and irregular pauses.
Using a language behavior dialogue strategy method based on multimodal deep learning, the complex speech text input by the user is obtained for grouping, and the pre-trained speech recognition network is used to analyze the attention mechanism within and between groups, combining position encoding, text and images Modal features are multimodal fusion to improve identification accuracy and reliability.
It improves the accuracy and reliability of bank card number identification, reduces interference from customer accents and irregular pauses, enhances the rationality and reliability of identification results, and realizes accurate identification of complex digital sequences.
Smart Images

Figure CN118865979B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal deep learning language behavior dialogue strategy method. Background Art
[0002] Bank card number voice recognition is a key function of the intelligent customer system. The current voice recognition system is mainly based on the recurrent neural network model, which performs well in processing general voice tasks. When faced with complex digital sequences of 16 to 19 digits such as bank card numbers, it is difficult to balance the accuracy of digital recognition and the reliability of the results. Customers usually read out the numbers in groups. For example, the complex digital sequence read out is "abcdefjhijklmnop". Different customers have different grouping methods, and the customer's accent, speaking speed, irregular pauses and background noise will have a certain impact on voice recognition. Therefore, the accuracy of single-modal voice recognition through neural networks is currently low.
[0003] The patent with publication number CN111462733B discloses a multimodal speech recognition model training method, which uses multimodal data, such as image data and speech signals, as training samples to train the speech recognition model, thereby improving the accuracy of speech recognition. However, the speech recognition part of this method still cannot overcome the interference caused by the customer's accent and irregular pauses, and only considers two modal data, such as image data and speech signals. The accuracy of speech recognition still needs to be improved. Summary of the invention
[0004] In view of this, an embodiment of the present invention provides a language behavior dialogue strategy method based on multimodal deep learning to solve the problem of poor accuracy of current single-modal speech recognition.
[0005] In an embodiment of the present invention, a method for a multimodal deep learning language behavior dialogue strategy is provided. The method for a multimodal deep learning language behavior dialogue strategy comprises the following steps:
[0006] Acquire complex voice text input by a user, and group the complex voice text to obtain one or more voice data groups;
[0007] Inputting the speech data group into a pre-trained speech recognition network, performing an intra-group attention mechanism analysis on the speech data group according to the speech recognition network, and determining an intra-group context vector of the speech data group;
[0008] Obtaining the position code of each of the language data groups, and performing an inter-group attention mechanism analysis on the speech data group based on the position code and the intra-group context vector to obtain a global context vector corresponding to the complex speech text;
[0009] According to the global context vector, obtaining recognition information output by the speech recognition network, and obtaining recognition confidence according to the recognition information, wherein the recognition information includes a verification probability of the speech data, one or more candidate recognition results, and a predicted probability of each of the candidate recognition results;
[0010] Acquire feature data corresponding to each of the text modality and the image modality, and based on the recognition confidence, perform multimodal fusion on the recognition information with the feature data corresponding to each of the text modality and the image modality to determine a target recognition result of the complex speech text input by the user; wherein the feature data of the text modality is extracted based on the text input by the user, and the feature data of the image modality is extracted based on the image provided by the user.
[0011] Preferably, performing an intra-group attention mechanism analysis on the speech data group according to the speech recognition network to determine an intra-group context vector of the speech data group includes:
[0012] encoding the speech data group according to the speech recognition network to obtain a group code of the speech data group, wherein the group code includes a hidden state of each data element in the speech data group;
[0013] Determining an attention score of each data element in the speech data set according to a hidden state of the corresponding data element and a first learnable parameter set in the speech recognition network;
[0014] Normalizing the attention scores of the data elements to obtain attention weights of the corresponding data elements;
[0015] A weighted sum is performed according to the attention weight and the hidden state of each data element in the speech data group to obtain an intra-group context vector of the speech data group.
[0016] Preferably, the performing of an inter-group attention mechanism analysis on the speech data group based on the position code and the intra-group context vector to obtain a global context vector corresponding to the complex speech text includes:
[0017] Determining an attention score corresponding to the speech data group based on the position code, the in-group context vector, and a second learnable parameter group in the speech recognition network;
[0018] Normalizing the attention scores of the speech data group to obtain the attention weights of the speech data group;
[0019] Weighting the intra-group context vector of the speech data group according to the attention weight of the speech data group to obtain a weighted vector of the speech data group;
[0020] The sum of the weighted vectors of all the speech data groups is calculated as the global context vector corresponding to the complex speech text.
[0021] Preferably, the performing of an inter-group attention mechanism analysis on the speech data group based on the position code and the intra-group context vector to obtain a global context vector corresponding to the complex speech text includes:
[0022] Obtaining a structural code of the speech data group, and determining an attention score corresponding to the speech data group according to the position code, the structural code, the intra-group context vector, and a third learnable parameter group in the speech recognition network;
[0023] Normalizing the attention scores of the speech data group to obtain the attention weights of the speech data group;
[0024] Weighting the intra-group context vector of the speech data group according to the attention weight of the speech data group to obtain a weighted vector of the speech data group;
[0025] The sum of the weighted vectors of all the speech data groups is calculated as the global context vector corresponding to the complex speech text.
[0026] Preferably, obtaining the recognition information output by the speech recognition network according to the global context vector includes:
[0027] Acquire the original hidden state of the complex speech text according to the speech recognition network;
[0028] Concatenate the original hidden state with the global context vector to obtain a concatenated vector;
[0029] Determine one or more candidate recognition results and a predicted probability of each of the candidate recognition results based on the concatenated vector and a fourth learnable parameter group in the speech recognition network;
[0030] The verification probability is determined based on the global context vector and a fifth learnable parameter group in the speech recognition network.
[0031] Preferably, determining one or more candidate recognition results and a predicted probability of each candidate recognition result according to the concatenated vector and a fourth learnable parameter group in the speech recognition network comprises:
[0032] Determining one or more predicted values and corresponding predicted probability distributions for each data element based on the concatenated vector and a fourth set of learnable parameters in the speech recognition network;
[0033] One or more predicted values of all data elements are randomly combined to obtain one or more candidate recognition results;
[0034] The predicted probability corresponding to the candidate recognition result is determined according to the predicted probability distribution of all data elements in the candidate recognition result.
[0035] Preferably, the method for obtaining recognition confidence includes:
[0036] Determine the maximum prediction probability among the candidate recognition results, and use the candidate recognition result corresponding to the maximum prediction probability as the predicted recognition result;
[0037] The maximum prediction probability and the verification probability are weighted and summed to obtain the recognition confidence.
[0038] Preferably, based on the recognition confidence, multimodally fusing the recognition information with the feature data corresponding to the text modality and the image modality to determine the target recognition result of the complex speech text input by the user includes:
[0039] In response to the recognition confidence being less than a confidence threshold, the predicted recognition result in the recognition information is feature fused with feature data of a text modality and feature data of an image modality to obtain a target recognition result.
[0040] Preferably, the training process of the speech recognition network includes:
[0041] Acquire training samples, each of the training samples comprising one or more data groups;
[0042] Inputting the training sample into a speech recognition network to be trained, and extracting, by the speech recognition network to be trained, a context vector of each data group in the training sample and a global context vector corresponding to the training sample;
[0043] Outputting prediction identification information according to the global context vector corresponding to the training sample, the prediction identification information including the prediction result and the prediction verification result of the training sample;
[0044] The current optimization loss of the speech recognition network to be trained is determined according to the prediction result and the prediction verification result, and the speech recognition network to be trained is adjusted and trained according to the optimization loss to obtain a pre-trained speech recognition network.
[0045] Preferably, determining the current optimization loss of the speech recognition network to be trained according to the prediction result and the prediction verification result includes:
[0046] Determine a difference value for each data element according to the prediction result, when the predicted value of the data element is the same as the true value, the corresponding difference value is 0, and when the predicted value of the data element is different from the true value, the corresponding difference value is 1;
[0047] Obtaining data loss according to the sum of the difference values of all the data elements;
[0048] Calculating the binary cross entropy loss between the predicted check bit probability and the true check bit probability in the predicted check result to obtain the check bit loss;
[0049] Determining the length loss according to the number of data elements in the prediction result, the predicted data length and the actual data length;
[0050] Determine a context vector loss according to the cosine similarity between the context vector of the first data group and the context vector of the last data group in the prediction result;
[0051] Determining a structural loss according to a weighted sum of the length loss and the context vector loss;
[0052] The data loss, the check bit loss and the structural loss are weightedly summed to obtain the optimization loss.
[0053] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by grouping the complex voice text of the user, the complex digital sequence is divided into multiple voice data groups for analysis, thereby reducing the amount of calculation for independent analysis of each data, the data groups are input into a pre-trained voice recognition network, the voice recognition network performs an intra-group attention mechanism analysis on the voice data group, obtains an intra-group context vector, assigns different attention scores to different numbers in the group to improve the recognition accuracy, further determines the position coding of the voice data group, uses the position coding to distinguish numbers at different positions and extracts the relationship between numbers at different positions, performs an inter-group attention mechanism analysis based on the position coding and the intra-group context vector, obtains a global context vector, and the global context vector more comprehensively covers the complex numbers. The digital information and position information of the word sequence are used to determine the recognition information based on the global context vector to improve the recognition accuracy. The recognition information also includes the check probability. The check probability is the probability obtained by analyzing the check digit in the bank card number. The validity of the bank card number is judged to make the recognition result of the bank card number more reasonable and reliable, and reduce the interference caused by the customer's accent and irregular pauses. The recognition confidence is calculated according to the recognition information to determine the reliability of the current candidate recognition result. When the recognition confidence shows that the recognition result is not reliable enough, the voice modality is combined with the text modality and the image modality according to the recognition confidence. The target recognition result is determined based on the multi-modal feature fusion, which increases the reliability and accuracy of the target recognition result. It can accurately recognize and process complex digital sequences with good recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0055] Figure 1 This is a method flow chart of a method based on multimodal deep learning language behavior dialogue strategy method provided in Example 1 of the present invention. DETAILED DESCRIPTION
[0056] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.
[0057] It should be noted that the terms "first", "second", etc. in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.
[0058] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.
[0059] See also Figure 1 , is a method flow chart of a method based on multimodal deep learning language behavior dialogue strategy method provided by the first embodiment of the present invention, such as Figure 1 As shown, the multimodal deep learning language behavior dialogue strategy method may include:
[0060] S101, obtaining complex voice text input by a user, and grouping the complex voice text to obtain one or more voice data groups.
[0061] The complex voice text input by the user is the voice data related to the bank card number read by the user. The bank card number currently read by the user is identified from the complex voice text to ensure the accuracy of subsequent intelligent services. It is worth noting that after the bank and the user sign an agreement, the user becomes an authorized user of the bank and has the authority to conduct telephone business with the bank, and the user's identity information, voiceprint information and telephone number recorded in the agreement will be stored in the database. Therefore, before obtaining the voice data related to the bank card number read by the user, the user's identity information is verified according to the bank's database. Once the verification is passed, the user's subsequent intelligent services can be carried out to ensure the security of the user's privacy.
[0062] Taking into account the user's reading habits, in order to improve the accuracy of complex voice text recognition, this embodiment groups the complex voice text to obtain multiple voice data groups. For example, a 16-digit bank card number "abcdefjhijklmnop" can be divided into four voice data groups, namely "abcd", "efjh", "ijkl" and "mnop"; for the complex voice text actually input by the user, this embodiment can divide the voice data into 4 voice data groups based on the pause positions in the voice data, thereby analyzing the features within and between voice data groups to obtain more accurate voice recognition results.
[0063] Optionally, before grouping the complex speech text, feature extraction can be performed on the complex speech text to obtain a feature sequence, and the feature sequence can be grouped to obtain multiple speech data groups, for example, the Mel-scale Frequency Cepstral Coefficients (MFCC) of each speech training sample can be calculated to remove the interference of noise in the complex speech sample and improve the accuracy of complex speech text recognition.
[0064] S102, inputting the speech data group into a pre-trained speech recognition network, performing an intra-group attention mechanism analysis on the speech data group according to the speech recognition network, and determining an intra-group context vector of the speech data group.
[0065] Optionally, the pre-trained speech recognition network can be a recurrent neural network (RNN). The current RNN network may have gradient vanishing or gradient exploding problems when processing complex digital sequences. When recognizing the numbers in the middle of the sequence, it may have "forgotten" the beginning of the sequence, resulting in overall recognition errors. For the bank card number recognition scenario, each digit is particularly important, so the traditional RNN network is less effective in the bank card number speech recognition scenario.
[0066] In order to solve the problem of low recognition accuracy of traditional RNN networks, this embodiment adds a group attention mechanism to the RNN network to optimize the recognition results of the RNN network and improve the recognition accuracy of the RNN network, wherein the training process of the RNN network can be: obtaining training samples, each training sample includes one or more data groups; inputting the training samples into the speech recognition network to be trained, and the speech recognition network to be trained extracts the context vector of each data group in the training sample and the global context vector corresponding to the training sample; outputting predicted recognition information according to the global context vector corresponding to the training sample, the predicted recognition information includes the prediction result and the prediction verification result of the training sample; determining the current optimization loss of the speech recognition network to be trained according to the prediction result and the prediction verification result, and adjusting the training of the speech recognition network to be trained according to the optimization loss to obtain a pre-trained speech recognition network.
[0067] Specifically, the input sample is first determined; in some implementations, multiple groups of historically recognized voice training samples can be obtained from a voice database, the voice training samples are voice data corresponding to bank card numbers with complex digital sequences, and the voice training samples can be voice data read out in a natural grouping manner for users of different ages, genders, accents, and speaking speeds, so as to improve the diversity of samples.
[0068] To improve recognition accuracy, feature extraction is performed on speech training samples, such as calculating the Mel frequency cepstral coefficients (MFCC) of each speech training sample to obtain a speech feature sequence. ,in, yes Moment Witt Eigenvector, is the total length of the sequence. By extracting MFCC features from speech training samples, the feature part is subsequently analyzed to remove noise interference in the speech training samples. The speech training samples are further annotated to determine the accurate text transcription corresponding to each speech training sample, including digital sequences, pauses, and grouping marks. The text annotation form can be ,in, , Indicates a pause mark; optionally, the bank card type information can also be recorded , for example, different types such as savings cards, debit cards or credit cards, the data structures corresponding to different bank card types may be different.
[0069] Furthermore, the check digit annotation can be obtained to judge the validity of the recognition result and obtain the check digit of the speech training sample. ,use The algorithm verifies the validity of the card number. If the calculated check digit is the same as the actual last digit of the card number, the check digit is marked as If they are not the same, the check bit mark value is 0. Finally, in this embodiment, the processed speech feature sequence , text annotation sequence , card type information The digits and check digits are labeled to form a training sample, and the RNN network is trained through each training sample.
[0070] In some implementations, before inputting the training samples into the RNN network, the training samples may be grouped to obtain multiple groups, and then each group is encoded according to the RNN network to obtain the group encoding corresponding to the training samples: ,in, , For the groups, each containing 4 consecutive hidden states, is the first hidden state of the group, , and are the second hidden state, the third hidden state, and the fourth hidden state of the group respectively; G is the input sequence after grouping, For the The hidden state of each time step is obtained based on the existing processing method of the RNN network, which will not be described in detail in this embodiment. Each time step corresponds to a data element. is the total number of groups after grouping, and N is 4 in this embodiment; inputting the grouped encoded training samples into the RNN network for processing helps the RNN network to better understand the structure of the digital sequence and reduce the recognition difficulty caused by long sequences. The longer sequence is divided into 4 groups for processing instead of multiple independent numbers, thereby reducing the processing complexity.
[0071] Furthermore, in order to solve the "forgetting" problem of traditional RNN when processing long sequences, this embodiment adds an attention mechanism to the RNN network for feature extraction, and applies the attention mechanism within each data group into which the four-digit number is divided, so that the model can accurately capture the features of each data group. Optionally, the processing of the intra-group attention mechanism can be:
[0072]
[0073]
[0074]
[0075] in, Indicates Group The attention score of an element indicates the degree of attention paid by the RNN network to the element. The higher the attention score, the more important the RNN network considers the element to be in the current task. is a learnable vector parameter, is a column vector, It means to transpose this column vector into a row vector; is a learnable weight matrix used to transform the hidden state Mapping to a new feature space; is a learnable bias vector; For the Group The hidden state of an element; It is Group The attention weight of each element; It is The context vector of a group is the weighted sum of the attention scores of all data elements, which integrates the information of all data elements in the data group; is the hyperbolic tangent activation function, which compresses the input into the range ; Used to normalize attention weights.
[0076] It is understandable that the RNN network includes multiple learnable parameters, such as the above-mentioned learnable and During the training process of the RNN network, the learnable parameters are continuously adjusted and learned to affect the output results of the RNN network. When the RNN network training is completed, each learnable parameter is also adjusted to obtain the final model parameters.
[0077] Through the processing of the above-mentioned intra-group attention mechanism, an attention score is dynamically assigned to each data element in the data group to determine the importance of each data element in the data group, so that the RNN network can better adapt to the characteristics of different bank card numbers and focus on some important numbers in the bank card number to improve the recognition accuracy.
[0078] Furthermore, the context vector obtained from each data group is Processing all data groups with the inter-group attention mechanism to obtain the relationship between different data groups helps the RNN network understand the global structure of the entire bank card number, thereby improving the accuracy of recognition. For example, different types of account numbers (such as debit cards and credit cards) often have different digital patterns. When the model recognizes that a certain digital group may be a bank identification code, it needs to pay more attention to the specific pattern of subsequent digital groups to improve recognition accuracy. The processing of the inter-group attention mechanism can be:
[0079]
[0080]
[0081]
[0082] in, For the The attention score of the group indicates the RNN network's attention to the entire The degree of attention of the group. The higher the attention score, the more important the RNN network considers the data grouping in the overall sequence. is a learnable vector parameter, which can be understood as a “query” vector used to evaluate the importance of the transformed group features; A learnable weight matrix for processing the context vector; A learnable weight matrix for processing positional encoding. The RNN network is able to consider the positional characteristics of the digit grouping in the entire sequence. This is particularly important when recognizing bank card numbers, because digit groups at different positions may have different meanings (such as bank code, account number, etc.); is the learnable bias vector; It is The context vector of the group, calculated by the intra-group attention mechanism; It is The positional encoding of the group provides information about the position of the digital grouping in the entire sequence; For the The attention weight of the group indicates the importance that the model ultimately decides to give to the data group. The attention weight is obtained by The function normalizes the attention score; It is a global context vector that integrates the information of all data groups. The weight is determined by the inter-group attention mechanism and represents the global representation of the entire bank card number sequence.
[0083] For the position coding of each data packet, in this embodiment, a fixed coding form is used, for example, as follows: , and They represent the position codes corresponding to the first data group and the second data group respectively. In this embodiment, the position codes are generated by using sine function and cosine function coding, wherein the sine coding function is: ; The cosine encoding function is: ;in It's the location. is the index of the data group, and in this embodiment, the values are 1, 2, 3 and 4. is the total encoding dimension; when encoding is performed based on the sine encoding function and the cosine encoding function, the sine encoding function and the cosine encoding function are used alternately, the sine encoding function is used at the even index position, and the cosine encoding function is used at the odd index position.
[0084] It is understandable that the RNN network can capture the relationship between different data groups through the processing of the inter-group attention mechanism. The digital elements at different positions in the bank card number often have different meanings, such as bank code, account number, etc. By calculating the position code , the RNN network model can distinguish the digital elements in different positions and learn the relationship between them, rather than just identifying each number in isolation, thereby improving the accuracy of bank card number recognition.
[0085] In some implementations, when recognizing the bank card number of the user input voice data, it is necessary to verify the correctness of the bank card number to avoid reducing the customer experience due to the wrong recognition of the bank card number. Therefore, structural coding information can be added to the RNN network model so that the RNN network model can learn the institutional characteristics in the bank card number; optionally, a structural encoder can be added to the RNN network, and the structural encoder encodes each data group according to the bank card type information of the bank card. The processing process of the structural encoder can be expressed as:
[0086]
[0087] in, Indicates The structural code of the group includes the position and function information of the data group in the bank card number structure; It is a two-dimensional dictionary, the first-level key is the group index, the second-level key is the card type, and each value is an encoding vector representing the combination of the position and the card type; is the index of the data group, indicating which data group is currently being processed; : Bank card type, such as savings card, debit card or credit card. Different types of bank cards may have different structural information. The bank card type is an input parameter.
[0088] By adding the structural information of the bank card number to the RNN network, the RNN network can learn the different structural features of the bank card number, so as to identify a more accurate bank card number according to the context vector in speech recognition. Therefore, the inter-group attention mechanism that integrates the structural information can be expressed as:
[0089]
[0090] in, For the The attention score of the group indicates the RNN network's attention to the entire The degree of attention of the group. The optimized attention score not only considers the context and location information, but also includes the structural information. The higher the attention score, the more important the RNN network considers the data group to be in the overall bank card number sequence. is the transpose of the learnable vector parameters, which is a row vector used to The output of the function is subjected to dot product operation, and the multi-dimensional features can be compressed into a scalar attention score through transposition operation; A learnable weight matrix for processing the context vector; It is The context vector of the group; A learnable weight matrix for handling positional encodings; It is The position encoding of the group reflects the information about the position of the digital grouping in the entire sequence, enabling the RNN network to distinguish between numbers in different positions; is a learnable weight matrix used to process structural encoding; For the A structural code of the group, generated by a structural encoder, wherein the structural code vector contains specific information about the data group in the bank card number structure, such as whether the data group may be a bank identification code, part of an account number, or a check digit; is a learnable bias vector that increases the flexibility of the model.
[0091] Furthermore, based on the updated attention score Perform normalization to obtain the updated attention weights of the data grouping , according to the attention weight and attention score Get the global context vector c of all data groups; It should be noted that the acquisition of the inter-group attention mechanism in the RNN network can adopt any of the above two acquisition methods, that is, two RNN networks can be trained for analysis. The RNN network with added structural coding has more calculation processes than the RNN network without structural coding, but the addition of the structural coding part will increase the accuracy of bank card number recognition. Therefore, in the scenario of long sequence data of bank card number recognition, the inter-group attention mechanism integrated with structural coding can be used to obtain the global context vector to improve the accuracy of the final recognition.
[0092] Furthermore, the output layer of the RNN network outputs prediction identification information according to the global context vector. The prediction identification information may include a prediction result and a prediction verification result. The prediction probability of each data element in the prediction result can be expressed as:
[0093]
[0094] in, is a given observation The following states The probability distribution of the state of each data element is the probability of the state corresponding to each data element. In this embodiment, the state refers to the data element being a different number. The dimension of the vector is equal to the number of possible states. For example, the predicted value of the identified data element in this embodiment may be 0-9 and pause. There are 11 dimensions in total; is an activation function that converts any real vector into a probability distribution and ensures that the sum of all output elements is , each element is in within the scope; is the learnable weight matrix of the output layer; is the learnable bias vector of the output layer; is the original hidden state, which contains the local information accumulated during sequence processing; It is the global context vector obtained through the attention mechanism, which integrates the information of the entire input sequence; express and The original hidden state and the global context vector obtained through the attention mechanism Connecting in the vertical direction allows the RNN network to utilize local information at the same time and global information To make the final prediction, the original hidden state And the context vector obtained by the attention mechanism Combined, the RNN network can use both local information (original hidden state) and global information (contextual information captured by the attention mechanism) when making the final prediction, solving the problem of "forgetting" information that may occur in traditional RNN networks when processing long sequences.
[0095] It can be understood that the prediction probability distribution of each time step, that is, the prediction probability distribution of each data element, can be obtained according to the above prediction identification information. The prediction probability distribution of each time step can be expressed as: ;in, Indicates the value category, including numbers and special marks , express Corresponding value category in the vector elements.
[0096] In some implementations, after determining the global context vector based on the updated inter-group attention mechanism, the check digit prediction can also be performed based on the global context vector, so that when the RNN network determines that the current number is the last number of the check digit, it pays more attention to the previous digital elements, because the check digit is usually calculated from the previous digital elements, thereby improving the accuracy of the RNN network prediction; the check digit prediction in this embodiment can be expressed as:
[0097]
[0098] in, Indicates that given an observation The probability of the next check digit being correct is arrive In the context of bank card number recognition, the predicted check digit probability can represent the confidence of the RNN network in the entire card number sequence it recognizes. The higher the predicted check digit probability, the more likely the current recognition result is a valid bank card number. is the activation function, which compresses the input to interval; is the learnable weight matrix for check digit prediction, mapping the global context vector to the check digit prediction space; is the global context vector, which contains the information of the entire sequence; is the learnable bias vector for check digit prediction.
[0099] The check digit of the bank card number is a number calculated according to an algorithm based on the previous number. Therefore, in this embodiment, the acquisition of the predicted check digit through the RNN network can improve the effect of bank card number recognition and ensure the effectiveness of bank card number recognition.
[0100] Based on the above description of the RNN network training process, it can be known that in this embodiment, the RNN network can extract intra-group attention of the input data grouping to obtain the context vector of each data group, and further perform inter-group attention extraction based on the context vector of each data group to obtain a global context vector. In the acquisition of the global context vector, the structural information of the bank card number is added to improve the accuracy of bank card number recognition; the output layer of the RNN network further performs prediction results and prediction verification results based on the acquired global context vector to improve the credibility of the current prediction results.
[0101] It can be understood that the training process of the RNN network is to continuously adjust the learnable weight parameters in the RNN network so that the RNN network outputs different prediction results, and further determine the accuracy of the current prediction based on the gap between the prediction result and the true label result, that is, calculate the current loss function of the RNN network, and use the loss function to reflect the accuracy of the current output prediction result. Based on the feedback of the loss function, the learnable parameters in the RNN network are adjusted so that the output result of the RNN network gradually approaches the true result until the loss function converges to complete the training of the RNN network; in this embodiment, the loss function of the RNN network may include data loss, check bit loss and structural loss, and the sum of data loss, check bit loss and structural loss is used as the final optimization loss, so as to optimize and adjust the RNN network according to the optimization loss to achieve the training purpose.
[0102] In some implementations, the difference value of each data element can be determined based on the prediction result. When the predicted value of the data element is the same as the true value, the corresponding difference value is 0, and when the predicted value of the data element is different from the true value, the corresponding difference value is 1; the data loss is obtained by summing the difference values of all data elements; the binary cross entropy loss between the predicted check bit probability and the true check bit probability in the prediction check result is calculated to obtain the check bit loss; the length loss is determined based on the number of data elements in the prediction result, the predicted data length, and the true data length; the context vector loss is determined based on the cosine similarity of the context vector of the first data group and the context vector of the last data group in the prediction result; the structural loss is determined based on the weighted sum of the length loss and the context vector loss; the data loss, the check bit loss, and the structural loss are weighted summed to obtain the optimization loss.
[0103] Alternatively, the optimization loss can be expressed as:
[0104]
[0105] in, represents the optimization loss; Indicates data loss; Indicates check bit loss; Indicates structural damage; and They represent the hyperparameters of check bit loss and structural loss respectively. In this embodiment, In other embodiments, it can be adjusted according to actual conditions and will not be described in detail here.
[0106] For data loss in optimization loss , the real data sample in the speech data annotated by the sample can be obtained, and the data loss can be determined according to the difference between the prediction result and the real label data at each time step. For example, in this embodiment, the bank card number is 16 bits, then it is determined whether each predicted data is the same as the real data. If it is the same as the real data, the local loss corresponding to the position is 0. Correspondingly, if it is different from the real data, the local loss corresponding to the position is 1. The local loss of each time step in the recognition result is obtained and summed up, so as to obtain the data loss of the current prediction result. .
[0107] For the check bit loss in the optimization loss , the real check digit data in the sample annotation can be obtained, and the check digit loss can be determined according to the difference between the real check digit data result and the predicted check digit probability; in this embodiment, the check digit loss is determined based on the binary cross entropy loss calculation method, which is expressed as: ,in, It is the real check bit data.
[0108] For the structural loss in the optimization loss , the length loss of the prediction result and the context vector loss can be combined to obtain the structure loss in this embodiment: ,in is the length loss, is the context vector loss; and is a preset parameter. In this embodiment, , ; The length loss is: , is the non-pause predicted by the model The number of markers, is the actual card number length, is the maximum possible card number length to be predicted; the context vector loss is: , represents the cosine similarity calculation, are the context vectors corresponding to the first and last digital groups, respectively, which can be extracted by the RNN network; if and The higher the similarity, the near , is close to 0; accordingly, if and If the similarity is low, near , Close to 1.
[0109] The prediction results of the RNN network are evaluated according to the above loss function. The closer the prediction results are to the actual results, the better the loss is. The smaller it is, the learnable parameters in the RNN network, such as the weight matrix and the offset vector, are reversely adjusted according to the optimization loss L, and a new prediction result of the RNN network is obtained according to the updated learnable parameters. Then, the new prediction result is compared with the actual result of the sample to determine the corresponding optimization loss L, and so on. When the optimization loss L converges, the training of the RNN network is completed to obtain a pre-trained RNN network. The pre-trained RNN network is also the pre-trained speech recognition network in this embodiment, and the values of each learnable parameter in the pre-trained RNN network are known.
[0110] On the basis of the above training process, after the speech data group is input into the pre-trained speech recognition network, the speech data group can be encoded according to the speech recognition network to obtain a group encoding of the speech data group, wherein the group encoding includes a hidden state of each data element in the speech data group; according to the hidden state of each data element in the speech data group and the first learnable parameter group in the speech recognition network, the attention score of the corresponding data element is determined; the attention score of the data element is normalized to obtain the attention weight of the corresponding data element; the weighted summation is performed according to the attention weight and hidden state of each data element in the speech data group to obtain the intra-group context vector of the speech data group; in this embodiment, the first learnable parameter group includes , and .
[0111] That is, the speech recognition network performs an intra-group attention mechanism analysis on each speech data group. The intra-group attention mechanism analysis process refers to the speech recognition network training process in this step, so as to obtain the attention score of each data element in the data group, and normalize the attention score of each data element to obtain the attention weight corresponding to each data element, and then perform weighted summation based on the attention score and attention weight of the data element to obtain the intra-group context vector corresponding to the speech data group.
[0112] S103, obtaining the position code of each speech data group, and performing an inter-group attention mechanism analysis on the speech data group based on the position code and the intra-group context vector, to obtain a global context vector corresponding to the complex speech text.
[0113] Optionally, the position code of each data group can be obtained, and the attention score of the corresponding voice data group can be determined according to the position code, the context vector within the group and the second learnable parameter group in the speech recognition network; the attention score of the voice data group is normalized to obtain the attention weight of the voice data group; the context vector within the group of the voice data group is weighted according to the attention weight of the voice data group to obtain the weighted vector of the voice data group; the sum of the weighted vectors of all voice data groups is calculated to be the global context vector corresponding to the voice data. In this embodiment, the second learnable parameter group includes: , , and .
[0114] It can be understood that, according to the above description of the training process of the RNN network process, the position code of each data packet is obtained. , according to the position code , the context vector within the speech data group and the second learnable parameter group to obtain the attention score of the speech data group , and then the attention score of each speech data group Normalize to get the corresponding attention weight , according to the attention weight of the speech data group and the in-group context vector The weighted sum of , we get the global context vector of all speech data groups , which is the global context vector The global context vector includes the global information of the speech data. The result prediction of the speech data based on the global context vector is more accurate.
[0115] In some implementations, the structural coding of the speech data group and the position coding of the speech data group can also be obtained, and the attention score of the corresponding speech data group is determined according to the position coding, the structural coding, the intra-group context vector and the third learnable parameter group in the speech recognition network; the attention score of the speech data group is normalized to obtain the attention weight of the speech data group; the intra-group context vector of the speech data group is weighted according to the attention weight of the speech data group to obtain the weighted vector of the speech data group; the sum of the weighted vectors of all speech data groups is calculated to be the global context vector corresponding to the speech data; the third learnable parameter group in this embodiment includes , , , and .
[0116] Based on the above statement, it can be known that the pre-trained RNN network can also be an RNN network with added structural coding. When the inter-group feature extraction is performed on the speech data group based on the RNN network with added structural coding, the structural coding of the current speech data group is obtained. and position encoding , and according to the structural encoding , Positional encoding , the context vector within the group and the third learnable parameter group to obtain the attention score of the speech data group, and normalize the attention score of the speech data group to obtain the corresponding attention weight , according to the attention weight of the speech data group and the in-group context vector The weighted sum of , we get the global context vector of all speech data groups , which is the global context vector , the global context vector subsequently used in this embodiment is a vector obtained based on the attention score obtained by structure encoding and position encoding.
[0117] S104, obtaining recognition information output by the speech recognition network according to the global context vector, and acquiring recognition confidence according to the recognition information.
[0118] The recognition information includes the verification probability of the complex speech text, one or more candidate recognition results, and the predicted probability of each candidate recognition result.
[0119] It can be understood that the process of obtaining the recognition information output by the speech recognition network based on the global context vector can be referred to step S102. According to the global context vector, the output layer of the RNN network outputs the predicted probability distribution of each data element and the predicted check probability of the check bit, and the check probability is obtained based on the global context vector.
[0120] In some implementations, due to the influence of the customer's accent or environmental noise, the pre-trained speech recognition network does not output only one corresponding value for each data element, that is, each data element may output one or more predicted values, and each predicted value will have a corresponding predicted probability distribution. For example, the third data element may be 4 or 7, and the predicted probability distribution of 4 is 60%, and the predicted probability distribution of 7 is 75%. Then, by combining the multiple predicted values of each data element, multiple candidate recognition results will be obtained, and the prediction probability corresponding to each candidate recognition result is different.
[0121] That is, the original hidden state of the complex speech text can be obtained according to the speech recognition network; the original hidden state is concatenated with the global context vector to obtain a concatenated vector; one or more candidate recognition results and the predicted probability of each candidate recognition result are determined according to the concatenated vector and the fourth learnable parameter group in the speech recognition network; the verification probability is determined according to the global context vector and the fifth learnable parameter group in the speech recognition network; in this embodiment, the fourth learnable parameter group includes and , the fifth learnable parameter group includes and .
[0122] In some implementations, one or more predicted values and corresponding predicted probability distributions of each data element can be determined based on the splicing vector and the fourth learnable parameter group in the speech recognition network; one or more candidate recognition results are obtained by randomly combining the one or more predicted values of all data elements; and the predicted probability distributions of all data elements in the candidate recognition results are determined to correspond to the predicted probabilities of the candidate recognition results; that is, according to the splicing vector and the fourth set of learnable parameters determines the predicted probability distribution , one or more candidate recognition results are obtained by randomly combining the data elements predicted at each position, and the prediction probability of each candidate recognition result can be the average value of the prediction probability distribution of all the data elements therein.
[0123] In some implementations, the maximum prediction probability among the candidate recognition results can be determined, and the candidate recognition result corresponding to the maximum prediction probability can be used as the predicted recognition result; the maximum prediction probability and the verification probability are weighted and summed to obtain the recognition confidence.
[0124] Since there may be multiple candidate recognition results in this embodiment, the prediction probability of each candidate recognition result is different, and the higher the prediction probability, the greater the corresponding possibility. Therefore, this embodiment can select the numerical value of the highest probability prediction distribution of each data element to form a predicted recognition result. The prediction probability corresponding to the predicted recognition result can be obtained by the average value of the prediction probability distribution of each data element, and then the confidence is calculated based on the prediction probability of the predicted recognition result to determine the credibility of the recognized recognition result.
[0125] Optionally, the calculation of recognition confidence can be:
[0126]
[0127] in, To identify confidence; is the maximum prediction probability, that is, the prediction probability of the recognition result composed of the values corresponding to the maximum prediction probability distribution of each position; represents the verification probability; and They are the corresponding weight values. , .
[0128] The recognition confidence of the current predicted recognition result is determined by the prediction probability and the verification probability results, so that the recognition confidence can more fully reflect the accuracy and reliability of the current predicted recognition result. The higher the recognition confidence, the greater the reliability of the current predicted recognition result. In this embodiment, a confidence threshold can also be set. By comparing the recognition confidence with the confidence threshold, it is judged whether the current predicted recognition result has reference value. For example, in this embodiment, the confidence threshold is divided into a high confidence threshold and a low confidence threshold. The high confidence threshold , low confidence threshold ; When it is determined that the recognition confidence is higher than the high confidence threshold, it means that the reliability of the current predicted recognition result is high; when it is determined that the recognition confidence is less than the high confidence threshold of 0.9 but greater than or equal to the low confidence threshold of 0.6, it means that the reliability of the current predicted recognition result is general, which may be affected by the customer's accent. The complex speech text is auxiliary recognized in combination with the results of other modalities, and the analysis weight of the predicted recognition result is reduced in subsequent analysis; when it is determined that the recognition confidence is lower than the low confidence threshold of 0.6, it means that the reliability of the current predicted recognition result is poor, so the user can be reminded to re-enter the complex speech text and analyze and recognize the complex speech text again to obtain a more reliable predicted recognition result.
[0129] S105, acquiring feature data corresponding to the text modality and the image modality respectively, and based on the recognition confidence, performing multimodal fusion of the recognition information with the feature data corresponding to the text modality and the image modality respectively, to determine the target recognition result of the complex speech text input by the user.
[0130] It can be understood that the present embodiment can fuse the speech modality with other modalities, such as text modality and image modality, to realize a more comprehensive multimodal dialogue system. The existing multimodal deep learning method is adopted. The multimodal dialogue system mainly includes feature extraction, feature alignment, multimodal fusion, dialogue state tracking, dialogue strategy generation and response generation. The feature data of the text modality is extracted based on the user input text, and the feature data of the image modality is extracted based on the image provided by the user.
[0131] In some implementations, in response to the recognition confidence being less than the confidence threshold, the predicted recognition result in the recognition information is feature fused with the feature data of the text modality and the feature data of the image modality to obtain the target recognition result. That is, when the recognition confidence is small, in this embodiment, when the recognition confidence is less than the high confidence threshold of 0.9 but greater than or equal to the low confidence threshold of 0.6, the predicted recognition result is fused with the features of other modalities to obtain a more accurate target recognition result.
[0132] This embodiment is based on speech recognition. When the recognition confidence of the predicted recognition result is low, text modality and image modality are added for fusion judgment. In other embodiments, it can also be based on the image modality. For example, certain input conditions cannot be met in some environments, such as image recognition in a dark environment will cause problems. At this time, in order to improve the recognition accuracy, the recognition results of the text modality and the speech modality can be fused for judgment; on the basis of the text modality, for example, considering the possibility of errors due to the high complexity of text input, the image modality or the speech modality can be combined for fusion judgment; it can be understood that visually impaired users can give priority to voice interaction, and hearing-impaired users can choose text or image interaction, and there is no specific modality type limitation to ensure that cross-validation information of different modalities can be performed for different users to improve the overall recognition accuracy.
[0133] As an example, the multimodal dialogue system obtains corresponding feature data from each modality. In this embodiment, the text modality can perform feature extraction on the bank card number text input by the user, and the image modality can perform feature extraction on the bank card image provided by the user. The RNN in this embodiment is used in the voice modality to extract corresponding features, that is, to predict the recognition result; for the text modality, word embedding technology can be used to extract feature data; for the image modality, convolutional neural network can be used to extract feature data. The feature data corresponding to each modality captures the essential characteristics of different modal information.
[0134] Furthermore, the multimodal dialogue system can map features of different modalities to the same semantic space to ensure that they can be effectively fused, and establish semantic associations between different modal information through feature alignment to lay the foundation for subsequent fusion; on the basis of feature alignment, multimodal fusion technology is used to integrate the features of each modality to generate a unified multimodal representation; this embodiment can adopt an attention mechanism, a gating mechanism or a more complex fusion network, for example, using a self-attention mechanism to capture feature relationships within the same modality, and then using a cross-attention mechanism to model the interaction between different modalities. This fusion method can make full use of the complementary information of each modality to improve the system's understanding ability.
[0135] This embodiment uses a multi-stage approach to extract, align and fuse features of different modalities. For example, in the feature extraction stage, for the speech modality, an optimized RNN network is used for processing, including grouping complex speech texts, performing intra-group and inter-group attention mechanism analysis, and structural encoding, and finally obtaining a global context vector as a feature vector. ; For text modality, word embedding technology is used to extract feature vectors ; For image modality, a convolutional neural network is used to extract feature vectors In the feature alignment stage, a linear projection layer is used to transform the features of each modality and map them to a common feature space, which can be:
[0136]
[0137]
[0138]
[0139] : A learnable weight matrix used to map the features of each modality to a common feature space; : The learnable bias vector.
[0140] After mapping the features to a common feature space, the features are subjected to multimodal fusion. This embodiment uses a combination of a self-attention mechanism and a cross-attention mechanism to achieve multimodal fusion:
[0141] Self-attention mechanism (taking speech modality as an example):
[0142]
[0143]
[0144]
[0145]
[0146] in, : A learnable weight matrix used to generate queries (Q), keys (K), and values (V). : The scaling factor of the attention mechanism, equal to the square root of the feature dimension. : The output of self-attention captures the internal relationship of speech features.
[0147] Cross-attention mechanism (taking voice-text interaction as an example):
[0148]
[0149]
[0150]
[0151]
[0152] : The output of the cross attention captures the relationship between speech and text features.
[0153] After obtaining the multi-modal output features, feature fusion is performed. In this embodiment, the self-attention mechanism obtains , and , and the cross-attention mechanism , and The features are concatenated in sequence to obtain a feature concatenation vector, which is then linearly transformed to obtain the final fusion feature.
[0154] Optionally, the process of performing linear transformation on the feature concatenation vector may be:
[0155]
[0156] in, is a learnable weight matrix used to fuse different attention outputs; : learnable bias vector; Feature concatenation vector; The final fusion feature contains information from all modalities and the interaction information between modalities. The final bank card number recognition result is obtained based on the final fusion feature.
[0157] Furthermore, the system can be updated and the dialogue state can be maintained based on the fused multimodality. The dialogue state tracking module records and updates key information such as user intentions and dialogue history to provide a basis for subsequent decision-making. This embodiment can use a recurrent neural network or an attention-based model to capture the dynamic characteristics of the dialogue; based on the dialogue state, the optimal dialogue strategy is generated using a reinforcement learning method, where algorithms such as a deep Q network or a policy gradient can be used; the best system behavior is selected according to the current dialogue state, and the goal of the dialogue strategy is to maximize long-term rewards (such as user satisfaction or task completion rate); finally, the final response is generated based on the generated dialogue strategy and multimodal information. The response generation module can be a template-based method or a more advanced neural generation model (such as Transformer or GPT, etc.), and the generated response should not only meet the guidance of the dialogue strategy but also naturally integrate multimodal information.
[0158] Through the above-mentioned multimodal fusion and intelligent dialogue strategy, this embodiment can not only accurately identify the bank card number, but also understand a wider range of user intentions and contextual information. According to the user's voice input, ID card image and text message, the user's identity authentication status is comprehensively judged. When identifying the bank card number, if the confidence of voice recognition is not high, the user can be requested to present the bank card image or confirm through text input to improve the accuracy of recognition; in addition, based on multimodal information and dialogue history, the user's query intention can be understood more accurately. For example, when the user asks about the account balance, the system can not only correctly identify the bank card number but also actively provide more comprehensive account information (such as recent transaction records or possible abnormal transaction reminders) based on the user's historical query habits, current time and other factors. When processing complex queries, the system's dialogue strategy can be flexibly adjusted. For example, for users who use the service for the first time, a more detailed guidance strategy will be adopted; while for old users who are familiar with the service, a more direct dialogue method will be selected. This personalized dialogue strategy can greatly improve the user experience.
[0159] To summarize, in this embodiment, the complex voice text read out by the user is divided into multiple voice data groups and input into the pre-trained voice recognition network for processing. The voice recognition network performs intra-group feature analysis on each voice data group to obtain an intra-group context vector, and then performs inter-group feature analysis on each voice data group based on the intra-group context vector to obtain a global context vector, fully extracting local and global information in the voice data, and obtaining prediction results based on the global context vector to improve the accuracy of complex voice text recognition. After determining the recognition information of the voice data, the recognition confidence is calculated based on the recognition information to judge the reliability of the current predicted recognition result. When the recognition confidence determines that the predicted recognition result is not reliable enough, it is fused with features of other modalities to obtain an accurate target recognition result, thereby improving bank card number recognition. accuracy and user experience; and in the training process of the pre-trained speech recognition network in this embodiment, through structural encoding, intra-group attention mechanism and inter-group attention mechanism, the features in the training samples are fully extracted for recognition and learning, and the corresponding optimization loss function is constructed based on multiple losses, and the speech recognition network is trained with a more accurate and comprehensive optimization loss function to improve the recognition performance of the speech recognition network, and a speech recognition network with better training effect is obtained. The prediction results are obtained based on a more comprehensive speech recognition network, and the recognition results are more accurate. Further integration with other modalities realizes the combination of speech recognition technology with multimodal fusion and intelligent dialogue strategy, providing a comprehensive and intelligent service system that can accurately identify bank card numbers while understanding complex user intentions, providing personalized services, and improving user satisfaction.
[0160] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A multimodal deep learning language behavior dialogue strategy method, characterized in that: The multimodal deep learning language behavior dialogue strategy method includes: Acquire complex voice text input by a user, and group the complex voice text to obtain one or more voice data groups; Inputting the speech data group into a pre-trained speech recognition network, performing an intra-group attention mechanism analysis on the speech data group according to the speech recognition network, and determining an intra-group context vector of the speech data group; Obtaining the position code of each of the speech data groups, and performing an inter-group attention mechanism analysis on the speech data groups based on the position code and the intra-group context vector to obtain a global context vector corresponding to the complex speech text; According to the global context vector, obtaining recognition information output by the speech recognition network, and obtaining recognition confidence according to the recognition information, wherein the recognition information includes a verification probability of the speech data, one or more candidate recognition results, and a predicted probability of each of the candidate recognition results; Acquire feature data corresponding to each of the text modality and the image modality, and based on the recognition confidence, perform multimodal fusion on the recognition information and the feature data corresponding to each of the text modality and the image modality to determine a target recognition result of the complex speech text input by the user; wherein the feature data of the text modality is extracted based on the text input by the user, and the feature data of the image modality is extracted based on the image provided by the user; The performing an intra-group attention mechanism analysis on the speech data group according to the speech recognition network to determine an intra-group context vector of the speech data group includes: encoding the speech data group according to the speech recognition network to obtain a group code of the speech data group, wherein the group code includes a hidden state of each data element in the speech data group; Determining an attention score of each data element in the speech data set according to a hidden state of the corresponding data element and a first learnable parameter set in the speech recognition network; Normalizing the attention scores of the data elements to obtain attention weights of the corresponding data elements; Performing weighted summation according to the attention weight and hidden state of each data element in the speech data group to obtain an intra-group context vector of the speech data group; The performing of an inter-group attention mechanism analysis on the speech data group based on the position code and the intra-group context vector to obtain a global context vector corresponding to the complex speech text includes: Determining an attention score corresponding to the speech data group based on the position code, the in-group context vector, and a second learnable parameter group in the speech recognition network; Normalizing the attention scores of the speech data group to obtain the attention weights of the speech data group; Weighting the intra-group context vector of the speech data group according to the attention weight of the speech data group to obtain a weighted vector of the speech data group; Calculating the sum of the weighted vectors of all the speech data groups as the global context vector corresponding to the complex speech text; The performing of an inter-group attention mechanism analysis on the speech data group based on the position code and the intra-group context vector to obtain a global context vector corresponding to the complex speech text includes: Obtaining a structural code of the speech data group, and determining an attention score corresponding to the speech data group according to the position code, the structural code, the intra-group context vector, and a third learnable parameter group in the speech recognition network; Normalizing the attention scores of the speech data group to obtain the attention weights of the speech data group; Weighting the intra-group context vector of the speech data group according to the attention weight of the speech data group to obtain a weighted vector of the speech data group; The sum of the weighted vectors of all the speech data groups is calculated as the global context vector corresponding to the complex speech text.
2. According to claim 1, a multimodal deep learning language behavior dialogue strategy method is characterized in that: The obtaining, according to the global context vector, recognition information output by the speech recognition network comprises: Acquire the original hidden state of the complex speech text according to the speech recognition network; Concatenate the original hidden state with the global context vector to obtain a concatenated vector; Determine one or more candidate recognition results and a predicted probability of each of the candidate recognition results based on the concatenated vector and a fourth learnable parameter group in the speech recognition network; The verification probability is determined based on the global context vector and a fifth learnable parameter group in the speech recognition network.
3. According to claim 2, a multimodal deep learning language behavior dialogue strategy method is characterized in that: Determining one or more candidate recognition results and a predicted probability of each candidate recognition result according to the concatenated vector and a fourth learnable parameter group in the speech recognition network includes: Determining one or more predicted values and corresponding predicted probability distributions for each data element based on the concatenated vector and a fourth set of learnable parameters in the speech recognition network; One or more predicted values of all data elements are randomly combined to obtain one or more candidate recognition results; The predicted probability corresponding to the candidate recognition result is determined according to the predicted probability distribution of all data elements in the candidate recognition result.
4. According to claim 3, a multimodal deep learning language behavior dialogue strategy method is characterized in that: The method for obtaining the recognition confidence comprises: Determine the maximum prediction probability among the candidate recognition results, and use the candidate recognition result corresponding to the maximum prediction probability as the predicted recognition result; The maximum prediction probability and the verification probability are weighted and summed to obtain the recognition confidence.
5. According to claim 4, a multimodal deep learning language behavior dialogue strategy method is characterized in that: The step of performing multimodal fusion of the recognition information with feature data corresponding to the text modality and the image modality based on the recognition confidence to determine a target recognition result of the complex speech text input by the user includes: In response to the recognition confidence being less than a confidence threshold, the predicted recognition result in the recognition information is feature fused with feature data of a text modality and feature data of an image modality to obtain a target recognition result.
6. The method of multimodal deep learning language behavior dialogue strategy according to claim 1, characterized in that: The training process of the speech recognition network includes: Acquire training samples, each of the training samples comprising one or more data groups; Inputting the training sample into a speech recognition network to be trained, and extracting, by the speech recognition network to be trained, a context vector of each data group in the training sample and a global context vector corresponding to the training sample; Outputting prediction identification information according to the global context vector corresponding to the training sample, the prediction identification information including the prediction result and the prediction verification result of the training sample; The current optimization loss of the speech recognition network to be trained is determined according to the prediction result and the prediction verification result, and the speech recognition network to be trained is adjusted and trained according to the optimization loss to obtain a pre-trained speech recognition network.
7. The method of multimodal deep learning language behavior dialogue strategy according to claim 6 is characterized in that: The step of determining the current optimization loss of the speech recognition network to be trained according to the prediction result and the prediction verification result includes: Determine a difference value for each data element according to the prediction result, when the predicted value of the data element is the same as the true value, the corresponding difference value is 0, and when the predicted value of the data element is different from the true value, the corresponding difference value is 1; Obtaining data loss according to the sum of the difference values of all the data elements; Calculating the binary cross entropy loss between the predicted check bit probability and the true check bit probability in the predicted check result to obtain the check bit loss; Determining the length loss according to the number of data elements in the prediction result, the predicted data length and the actual data length; Determine a context vector loss according to the cosine similarity between the context vector of the first data group and the context vector of the last data group in the prediction result; Determining a structural loss according to a weighted sum of the length loss and the context vector loss; The data loss, the check bit loss and the structural loss are weightedly summed to obtain the optimization loss.
Citation Information
Patent Citations
Multimodal speech recognition model training method, device, equipment and storage medium
CN111462733B
Methods and systems for synthesising speech from text
EP4177882A1
Methods and systems for synthesising speech from text
GB2612624A
Speech recognition with sequence-to-sequence models
US20200126538A1
Learning Word-Level Confidence for Subword End-To-End Automatic Speech Recognition
US20220270597A1
