Operation execution method, device, electronic device and storage medium

Abstract

The present application discloses an operation execution method, device, electronic device and storage medium, which belongs to the field of semantic recognition technology. The operation execution method includes: receiving a first input from a user; in response to the first input, obtaining the corresponding word vector, position vector and tone vector for each character in the first text respectively, and adding them to form an initial vector of the first text; wherein the first text is the text corresponding to the first input; based on the scene classification module, the initial vector is deeply encoded and dimensionally converted to obtain a first scene type; based on the intent recognition module and slot extraction module corresponding to the first scene type, the initial vector is deeply encoded and dimensionally converted to obtain first intent information and first slot information; according to the first intent information and the first slot information, the operation corresponding to the first input is executed.

Description

Technical Field

[0001] The present application belongs to the field of semantic recognition technology, and specifically relates to an operation execution method, device, electronic device and storage medium. Background Art

[0002] With the development of science and technology, electronic devices are becoming more and more intelligent. When using intelligent electronic devices, users can often input some short text instructions to the electronic devices, and the electronic devices will make corresponding decisions and give responses after understanding the semantics of the text. For example, the voice assistant of a mobile phone can complete intelligent interaction by receiving the user's voice instructions. How to accurately understand the short text instructions input by the user has become a problem that needs to be solved urgently.

[0003] At present, existing semantic understanding models can only recognize and semantically understand texts with accurate reception and pronunciation. However, Automatic Speech Recognition (ASR) errors caused by inaccurate reception or dialect pronunciation often lead to failure of semantic understanding, making it impossible to perform corresponding operations. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an operation execution method, device, electronic device and storage medium, which can solve the problem in the prior art that semantic understanding fails and corresponding operations cannot be performed due to inaccurate sound reception or dialect pronunciation.

[0005] In a first aspect, an embodiment of the present application provides an operation execution method, the method comprising:

[0006] receiving a first input from a user;

[0007] In response to the first input, obtaining a corresponding word vector, position vector and tone vector for each character in the first text respectively, and adding them together to form an initial vector of the first text; wherein the first text is the text corresponding to the first input;

[0008] Based on the scene classification module, the initial vector is deeply encoded and dimensionally converted to obtain a first scene type;

[0009] Based on the intent recognition module and the slot extraction module corresponding to the first scene type, the initial vector is deep-encoded and dimension-converted to obtain first intent information and first slot information;

[0010] An operation corresponding to the first input is performed according to the first intention information and the first slot information.

[0011] In a second aspect, an embodiment of the present application provides an operation execution device, the device comprising:

[0012] An input receiving module, used for receiving a first input from a user;

[0013] a vector acquisition module, configured to, in response to the first input, acquire a corresponding word vector, a position vector, and a tone vector for each character in a first text, and add them together to form an initial vector of the first text; wherein the first text is the text corresponding to the first input;

[0014] A scene classification module, configured to perform depth encoding and dimension conversion on the initial vector based on the scene classification module to obtain a first scene type;

[0015] an identification and extraction module, configured to perform depth encoding and dimension conversion on the initial vector based on the intent identification module and the slot extraction module corresponding to the first scene type, so as to obtain first intent information and first slot information;

[0016] An operation execution module is used to execute an operation corresponding to the first input according to the first intention information and the first slot information.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect.

[0020] In an embodiment of the present application, upon receiving the user's first input, in addition to extracting the word vector and position vector corresponding to each character in the first text corresponding to the first input, the tone vector of each character is also extracted, and then based on the initial vector of the first text formed by adding these vectors, scene classification, intent recognition and slot extraction are performed to identify the semantics of the first text and perform operations corresponding to the first input. In this way, since the embodiment of the present application adds the tone vector when generating the initial vector of the first text corresponding to the first input, even in the case of ASR errors, it is possible to extract important information in the sentence and achieve accurate semantic understanding, thereby improving the fault tolerance of semantic understanding and avoiding the problem of semantic understanding failure and inability to perform corresponding operations due to inaccurate reception or dialect pronunciation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is one of the flowcharts of the operation execution method according to an exemplary embodiment;

[0022] Figure 2 is an architecture diagram of an application scenario of an operation execution method according to an exemplary embodiment;

[0023] Figure 3 is a schematic diagram of corpus comparison according to an exemplary embodiment;

[0024] Figure 4 is a second flowchart of an operation execution method according to an exemplary embodiment;

[0025] Figure 5 is a schematic diagram of a character prediction scenario based on a KTM training method according to an exemplary embodiment;

[0026] Figure 6 is a schematic diagram of generating an initial vector according to an exemplary embodiment;

[0027] Figure 7 is a schematic diagram of a ROSITA model training structure according to an exemplary embodiment;

[0028] Figure 8 is a flowchart of a model pre-training method according to an exemplary embodiment;

[0029] Fig. 9 is a structural block diagram of an operation execution device according to an exemplary embodiment;

[0030] Fig.10 is a structural block diagram of an electronic device according to an exemplary embodiment;

[0031] Fig.11A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0033] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0034] The following is a detailed description of the operation execution method provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0035] The operation execution method provided in the present application can be applied to control scenarios based on voice recognition, such as scenarios in which users control electronic devices through voice assistants.

[0036] Currently, in the above scenarios, there are often some automatic speech recognition (ASR) errors caused by radio or dialect pronunciation. For example, "Open Yingyue player to play You San Nian" should actually be "Open music player to play You Shan Lian". Because the app name and playback entity cannot be obtained from the original text, it creates obstacles to scene classification and subsequent semantic understanding.

[0037] In response to the above problems, an embodiment of the present application provides an operation execution method, that is, receiving a first input from a user; in response to the first input, obtaining the corresponding word vector, position vector and tone vector for each character in the first text, and adding them to form an initial vector of the first text; wherein the first text is the text corresponding to the first input; based on the scene classification module, deep encoding and dimension conversion are performed on the initial vector to obtain a first scene type; based on the intent recognition module and slot extraction module corresponding to the first scene type, deep encoding and dimension conversion are performed on the initial vector to obtain first intent information and first slot information; according to the first intent information and the first slot information, execute the operation corresponding to the first input.

[0038] In this way, since the embodiment of the present application adds the tone vector when generating the initial vector of the first text corresponding to the first input, even if the ASR is erroneous, it is still possible to extract important information in the sentence and achieve accurate semantic understanding, thereby improving the fault tolerance of semantic understanding and avoiding the problem of failure of semantic understanding and inability to perform corresponding operations due to inaccurate reception or dialect pronunciation.

[0039] In the operation execution method provided in the embodiment of the present application, the execution subject may be an operation execution device, or a control module in the operation execution device for executing the operation execution method. The operation execution device may be a server or a terminal device, which is not limited here. In the embodiment of the present application, the operation execution method provided in the embodiment of the present application is illustrated by taking the operation execution device executing the operation execution method as an example.

[0040] Figure 1 The invention is a flowchart showing an operation execution method according to an exemplary embodiment.

[0041] like Figure 1 As shown, the operation execution method may include the following steps:

[0042] S110: Receive a first input from a user.

[0043] Here, the first input may be voice input or text input, which is not limited here.

[0044] S120. In response to the first input, obtain the corresponding word vector, position vector and tone vector for each character in the first text respectively, and add them to form an initial vector of the first text; wherein the first text is the text corresponding to the first input.

[0045] Here, in the case where the first input is a voice input, the first text may be a text obtained after voice recognition of the first input. The initial vector may be an initial feature vector extracted from the recognized first text and capable of representing the content of the text.

[0046] In the embodiment of the present application, tone information can be incorporated into the feature extraction process, so as to improve the model's fault tolerance for the input text. For example, for the input text "turn on the phone", the corresponding word vector, position vector and tone vector can be obtained for each word in the text, and then the word vector, position vector and tone vector corresponding to each word are added together to form the initial vector of the first text.

[0047] Exemplarily, taking the scene classification using the scene classification module as an example, in the scene classification process, there are often some ASR wrong texts that cause scene classification errors. For example, for the text "Open the Yingyue player to play by Sannian", because there are keywords such as "open" and "play", the final probability distribution generated may be {mobile phone function: 0.1, music: 0.2, radio: 0.3, video 0.3, ...}. Therefore, it is likely to be misclassified into a radio or video scene, rather than a music scene. In the text semantic recognition process, by incorporating tone information, the fault tolerance of the recognition system can be improved to avoid the occurrence of such errors: the tone vectors of "Yingyue player" and "music player", "by Sannian" and "Youshanlian" are almost the same. After adding the tone information, the initial vectors corresponding to the two texts are more likely to converge, and the correct input and the ASR wrong input will have similar probability distributions, so that the same prediction results can be obtained, which has the effect of error correction. After the query with ASR error is input into the scene classification module for scene classification, it can be correctly classified into the music scene.

[0048] S130. Based on the scene classification module, perform depth encoding and dimension conversion on the initial vector to obtain a first scene type.

[0049] Here, the scene classification module can be a module with scene classification function obtained by fine-tuning the pre-trained model using the training data corresponding to the scene classification function. The scene classification module can be used to perform scene classification on text.

[0050] Among them, the pre-trained model can be a model obtained through pre-training. The pre-trained model can be, for example, a pre-trained BERT (Bidirectional Encoder Representation from Transformers) model, and of course, it can also be a model optimized for the short text field based on the BERT model, that is, the ROSITA (Oral Short Text by Voice Assistant Tapt, based on the voice assistant field adaptation oral short text) model designed in the embodiment of the present application.

[0051] In addition, the scene classification module involved in the embodiment of the present application may include an encoder and a fully connected layer. Among them, the encoder is used to perform depth encoding on the initial vector, and the fully connected layer is used to perform dimension conversion on the encoded vector obtained after the depth encoding. The encoder may include one or more encoding layers, for example, a 12-layer Transformer Encoder is used as an encoder.

[0052] In a specific example, the initial vector corresponding to the input text "open the phone" is input into the ROSITA-based scene classification module, and the semantics are deeply encoded through the 12-layer stacked Transformer Encoder layer to generate the encoding vector, and then the dimension is transformed through the fully connected layer Feed Forward trained during the ROSITA model fine-tuning to generate the corresponding probability distribution. If there are 5 scenes in total, that is, there is a probability distribution corresponding to the 5 scenes {mobile phone function: 0.6, music: 0.2, video 0.1...}, the mobile phone function scene with the highest probability is finally selected as the predicted scene type, which is also the first scene type.

[0053] S140. Based on the intent recognition module and the slot extraction module corresponding to the first scene type, perform depth encoding and dimension conversion on the initial vector to obtain first intent information and first slot information.

[0054] Here, different scene types may correspond to different intent recognition modules and slot extraction modules. Exemplarily, the above-mentioned pre-trained model may be fine-tuned according to the training data corresponding to the intent recognition function under different scene types to obtain the intent recognition module, and the above-mentioned pre-trained model may be fine-tuned according to the training data corresponding to the slot extraction function under different scene types to obtain the slot extraction module.

[0055] Among them, the intent recognition module can be used to recognize the intent of the text, and the slot extraction module can be used to extract slot information in the text.

[0056] In addition, the intent recognition module and slot extraction module involved in the embodiments of the present application may also include an encoder and a fully connected layer. Since the pre-trained model used by the scene classification module involved in the above-mentioned fine-tuning training is used, the structures of these three modules are the same and will not be repeated here.

[0057] Exemplarily, the intent recognition module can classify the intent of the input text in the current scenario. For example, in the ROSITA-based intent recognition module in the music scenario, for the input text "Play Blue and White Porcelain", the initial vector corresponding to the input text can be deeply encoded for text semantics through 12 stacked Transformer Encoder layers to generate an encoded vector, and then the dimension transformation is performed through the Feed Forward fully connected layer retrained during model fine-tuning to generate the corresponding probability distribution {Play music: 0.6, Switch songs: 0.2,...}. Among them, the number of intent categories included in the probability distribution is the same as the total number of intent categories in the current scenario. Finally, the intent type of "Play music" with the highest probability is selected as the predicted intent type, and the name corresponding to this intent type is used as the intent information of the input text.

[0058] In addition, in the ROSITA-based slot extraction module, for example, for the input text "Open the Appointment Player to play You San Nian", first, the initial vector corresponding to the text is deeply encoded for semantics through 12 stacked Transformer Encoder layers to generate an encoded vector, and then the dimension transformation is performed through the Feed Forward fully connected layer retrained during model fine-tuning to generate the corresponding encoded vector. The length of the vector is the same as the sentence length, and the dimension size is equal to the number of slot types. That is, the probability distribution corresponding to the character "da" is {O (no slot): 0.7, Singer: 0.1...}, the probability distribution corresponding to the character "kai" is {O (no slot): 0.8, Song name: 0.1...},..., the probability distribution corresponding to the character "ying" is {O (no slot): 0.2, APP name: 0.7...}, the probability distribution corresponding to the character "yue" is {O (no slot): 0.2, APP name: 0.7...},..., the probability distribution corresponding to the character "you" is {O (no slot): 0.1, Song name: 0.7...}, the probability distribution corresponding to the character "san" is {O (no slot): 0.1, Song name: 0.7...}, and the probability distribution corresponding to the character "nian" is {O (no slot): 0.1, Song name: 0.7...}. Select the type with the highest probability as the slot output of the current character, and the slot of the APP name "Appointment Player" and the slot of the song name "You San Nian" can be extracted. Then, error correction can be performed according to simple pitch similarity, and it is corrected to "Music Player" and "You Shan Lian". That is, the slot information obtained is "APP name: Music Player" and "Song name: You Shan Lian".

[0059] S150. Execute the operation corresponding to the first input according to the first intent information and the first slot information.

[0060] In an embodiment of the present application, after obtaining the first intent information and the first slot information, the text semantics of the first input can be understood through the first intent information and the first slot information, and then the operation corresponding to the first input can be performed, such as playing a certain music, turning on a mobile phone, etc.

[0061] In this way, when receiving the user's first input, in addition to extracting the word vector and position vector corresponding to each character in the first text corresponding to the first input, the tone vector of each character is also extracted, and then based on the initial vector of the first text formed by adding these vectors, scene classification, intent recognition and slot extraction are performed to identify the semantics of the first text and perform operations corresponding to the first input. In this way, since the embodiment of the present application adds the tone vector when generating the initial vector of the first text corresponding to the first input, even if the ASR is wrong, it can extract important information in the sentence and achieve accurate semantic understanding, thereby improving the fault tolerance of semantic understanding and avoiding the problem of failure of semantic understanding and inability to perform corresponding operations due to inaccurate reception or dialect pronunciation.

[0062] In addition, in an optional implementation, after S130, the operation execution method provided in the embodiment of the present application may further include:

[0063] Based on the classification verification module corresponding to the first scene type, the initial vector is deeply encoded and dimensionally converted to verify whether the scene type of the first input is the first scene type, and a verification result is obtained;

[0064] Based on the intent recognition module and the slot extraction module corresponding to the scene type, the initial vector is deeply encoded and dimensionally converted to obtain the first intent information and the first slot information, including:

[0065] When it is determined according to the verification result that the scene type of the first input is the first scene type, the initial vector is deep encoded and dimensionally converted based on the intent recognition module and slot extraction module corresponding to the first scene type to obtain first intent information and first slot information.

[0066] Here, the classification verification module can be a module generated by fine-tuning the pre-trained model mentioned above using the corresponding binary classification training data under the current scene type, and the classification verification module can be used to identify whether the text belongs to the current scene type. In addition, the classification verification module involved in the embodiment of the present application may also include an encoder and a fully connected layer. Since the pre-trained model used by the three modules involved above is used in the fine-tuning training, the structures of these four modules are the same and will not be repeated here.

[0067] Exemplarily, the classification verification module can be used to reclassify the first text classified as the first scene type to determine whether the text belongs to the current scene type. Only if it is determined to be the current scene type will the next step of intent recognition and slot extraction operations be performed. In the classification verification module based on ROSITA, for example, for the input text "turn on the phone", the initial vector corresponding to the text can be deeply encoded through 12 stacked Transformer Encoder layers to generate an encoding vector, and then the fully connected layer Feed Forward obtained by retraining during model fine-tuning is used for dimensional transformation to generate the corresponding probability distribution {It is a mobile phone function scene: 0.8, it is not a mobile phone function scene: 0.2}, and the one with the largest probability is selected as the prediction result, that is, at this time, the input text "turn on the phone" will be determined to belong to the mobile phone function scene, and then the next step of intent recognition and slot extraction operations will be continued.

[0068] In addition, when the determination result is that the first text does not belong to the first scene type, the process may return to S130 to reclassify the scene type of the first text.

[0069] In this way, by utilizing the classification verification module to re-confirm the scene classification of text that has been scene classified, the accuracy of the scene classification determination result of the text can be improved, thereby preventing the erroneous processing of misclassified data.

[0070] Based on this, in an optional implementation manner, the above-mentioned depth encoding and dimension conversion of the initial vector may specifically include:

[0071] The initial vector is deeply encoded to obtain an encoded vector;

[0072] Transform the dimension of the encoded vector to obtain the corresponding probability distribution;

[0073] According to the probability distribution, the information with the largest probability value is determined as the target information;

[0074] Among them, the target information is the information obtained based on the target function module through the above-mentioned operation steps. The target function module includes a scene classification module, an intent recognition module, a slot extraction module and a classification verification module. The target information includes a first scene type, first intent information, first slot information and a verification result.

[0075] The scene classification module, intent recognition module, slot extraction module and classification verification module involved in the embodiments of the present application may include an encoder and a fully connected layer, wherein the encoder may be an encoder obtained by pre-training and fine-tuning the pre-trained model, and the fully connected layer may be a fully connected layer adapted to the respective functions by re-training the pre-trained model during the fine-tuning training process.

[0076] Exemplarily, the initial vector corresponding to the text can be passed through a 12-layer stacked Transformer Encoder layer to deeply encode the text semantics to generate an encoded vector, which is then transformed into a dimension through the fully connected layer FeedForward obtained by retraining during model fine-tuning to generate a probability distribution corresponding to each function, and then the scene type, intent information, slot information or verification result with the largest probability value is obtained from the probability distribution as the final output target information.

[0077] In this way, by performing deep encoding and dimensionality conversion based on different functional modules to obtain the information required by each, the training efficiency of the model can be improved, eliminating the need to train modules with different functions from scratch.

[0078] Based on the various implementation methods of the above operation execution method, a specific example is given below to more clearly describe the entire technical solution.

[0079] like Figure 2 As shown, it is an architecture diagram of an application scenario of the operation execution method provided in an embodiment of the present application, wherein:

[0080] Scenario classification is to classify the search terms (queries) entered by users into corresponding vertical categories.

[0081] Binary classification is to classify the input vertical category into two categories to determine whether the query belongs to the current vertical category. It is equivalent to fault-tolerant processing of scene classification to prevent erroneous processing of misclassified data.

[0082] Intent recognition under each vertical category is to classify the intent of the query under the current vertical category to determine the user's intent.

[0083] The slot extraction under each vertical category is based on the currently determined intent, and the slots required for the intent are extracted.

[0084] Finally, the result is returned to the user based on the intent and slot.

[0085] For example, if the user's query is "Play a song called Can't Live Without You by Zhang San", the scene classification module will classify it into the music vertical category. The binary classification module of the music vertical category determines that the current query does belong to the music vertical category, and then performs intent recognition and slot extraction. The result of intent recognition is "play a song", and the slot extraction module can extract "singer name (Zhang San)" and "song name (Can't Live Without You)" based on the slots required for the intent of "play a song". Finally, the song "Can't Live Without You" by Zhang San is fed back to the user.

[0086] In addition, before using each module obtained by fine-tuning training based on a pre-trained model, it is also necessary to pre-train the initial model to obtain a trained pre-trained model.

[0087] Among them, the pre-trained model can be a model optimized for the short text field based on the BERT model, that is, the ROSITA model designed in the embodiments of the present application.

[0088] For the traditional BERT model, due to the training method and training corpus, its semantic understanding of short texts in the field of spoken short texts similar to voice assistants is usually not very accurate, which is mainly reflected in the following aspects:

[0089] In terms of the use of the training corpus, as Figure 3 shown, the training corpus adopted by the BERT model is a sentence with a formal word usage, a longer sentence length, and a strong context relevance on the right. However, the corpus of voice assistants is generally short, the sentence pattern is more colloquial, and the context relevance is weak. If the BERT model is directly applied to the voice assistant scenario, the differences between the corpora will result in a large accuracy loss when the BERT model is used.

[0090] In terms of the model training method, for example, in a text like "Play the song 'Can't Live Without You' by Zhang San", the traditional BERT model uses the pre-training method of MLM (Mask Language Model). By randomly covering some words in the model, and these words may be some words with low contribution to the overall semantics and are easy to predict. For example, in the pre-training method of MLM, "play" may be covered, and based on the adjacent character "播", the model can easily predict "play". Therefore, the BERT model trained using the pre-training method of MLM cannot capture long-distance semantic dependency features. Another example is in the text "Zhang San watched a tennis match". In fact, it should be a tennis scenario, but because the singer name Zhang San is easily misclassified as a music scenario by the model, and the relationship between Zhang San, tennis, and the match cannot be captured, resulting in a deviation in semantic understanding. In addition, during the model training process, each training sample corresponds to only one sample label and there is only one training task. Therefore, to a certain extent, the robustness of the model is limited.

[0091] In response to the above problems, an embodiment of the present application provides a training method for a pre-trained model, namely, obtaining multiple corpora and their corresponding scene information, intent information and slot information according to the user's historical query log; identifying at least one keyword contained in the original text corresponding to each corpus; performing character replacement on at least part of the target characters in the keywords to obtain a model input text corresponding to each corpus; wherein the target characters are part of the characters in the keywords; taking the target characters as the first sample label, the scene type corresponding to the scene information and the intent type corresponding to the intent information as the second sample label, and the slot type corresponding to the slot information as the third sample label, Add three sample labels to the model input text to obtain multiple training samples; use the multiple training samples to train the initial model to obtain a pre-trained model; wherein the training tasks corresponding to the initial model include a first training task, a second training task and a third training task, the first training task corresponds to the first sample label, the second training task corresponds to the second sample label, and the third training task corresponds to the third sample label; fine-tune the pre-trained model according to the training data corresponding to the target module function to obtain a target function module having the target module function; wherein the target module functions include a scene classification function, an intent recognition function, a slot extraction function and a classification verification function.

[0092] In this way, on the one hand, by constructing training samples based on the short text corpus in the user's historical query log and its corresponding scene information, intent information and slot information, where one training sample contains three sample labels, multi-task joint training is carried out by combining self-supervised training of the first training task with supervised training of the second training task and the third training task. In this way, the robustness of the model can be enhanced while fitting the application scenario of short text semantic understanding based on speech recognition.

[0093] On the other hand, the embodiment of the present application also replaces the target characters in the original text of each corpus for keywords that contribute highly to the model prediction results. Compared with the traditional training method of randomly masking any characters for model prediction, the embodiment of the present application can more accurately capture the dependency between semantics, so that the training effect of the pre-trained model is better and it is easier to capture the semantics of short texts, thereby improving the semantic understanding effect of the model while improving the robustness of the model.

[0094] Based on this, in one possible embodiment, as Figure 4 As shown, before S110, the operation execution method provided in the embodiment of the present application may further include: S410-S460, and the specific steps are as follows:

[0095] S410. Obtain multiple corpora and their corresponding scene information, intent information, and slot information according to the user's historical query log.

[0096] In the embodiment of the present application, before performing model pre-training, the training samples required for model pre-training can be first constructed based on the user's historical query log. The user's historical query log can be a log recorded when a resource query is performed using the text corresponding to the user's historical input. The log can include multiple corpora, and one corpus contains a text of the user's historical input. The user's historical input can be voice input or text input, which is not limited here.

[0097] In addition, the scene information may be the scene type information to which the corpus content belongs. For example, for the corpus "Play a song called "I Can't Live Without You" by Zhang San", the corresponding scene information may be the music scene, and for the corpus "Turn down the volume of the mobile phone", the corresponding scene information may be the mobile phone function scene.

[0098] Intent information can be type information that represents the intent of the corpus, which is related to the scene information. For example, in a music scene, "playing a song called Can't Live Without You by Zhang San" can be the intent to play a song, and "switch to the next song" can be the intent to switch songs. Different scene types can contain different intent types. Of course, different scene types can also contain the same intent type, which is not limited here.

[0099] Slot information can be the key information that the device needs to collect from the user, and slot filling is the process of collecting user information and completing the vague or missing intentions of the user, and the slot words are the specific key information expressed by the user. For example, "Play a song by Zhang San that I can't live without you" contains two slot information, one is "singer name: Zhang San" and the other is "song name: I can't live without you".

[0100] In a specific example, relevant data can be obtained through online logs, which may include a large amount of original text corresponding to the queries input by users, as well as the scene information, intent information and slot information corresponding to each query, and of course other relevant information required in the semantic parsing process.

[0101] S420: Identify at least one keyword contained in the original text corresponding to each corpus.

[0102] Here, the entity words and slot words in the original text may be used as keywords. Therefore, the keywords corresponding to the text may be obtained by identifying the entity words and slot words in the original text.

[0103] For example, before identifying at least one keyword in the original text, the corpus obtained from the log may be preprocessed by cleaning, deduplication, etc. Then, the slot words and entity words are annotated for the cleaned corpus to determine the slot words and entity words in the original text corresponding to the corpus.

[0104] S430, performing character replacement on target characters in at least part of the keywords to obtain model input text corresponding to each corpus; wherein the target characters are part of the characters in the keywords.

[0105] In the embodiment of the present application, the KTM (Key Token Mask) method is mainly used to train the pre-trained model, wherein KTM is a pre-training method proposed based on the ROSITA model of the embodiment of the present application, that is, certain characters in the keywords in the corpus are selectively masked, and then the model is allowed to predict.

[0106] For the KTM training method proposed in the embodiment of the present application, specifically, character replacement can be performed for some or all of all keywords, wherein the number of characters replaced in each keyword is less than the total number of characters contained in the keyword. Since the character replacement ratio is generally 15% of the sentence length, this can avoid the problem of low keyword coverage caused by the contradiction between the character replacement ratio and the sentence length limit, and try to ensure that all keywords can have characters replaced. At the same time, it can also avoid the problem of slot openness caused by completely replacing all characters contained in the keyword.

[0107] Therefore, compared with the MLM pre-training method in the prior art, the present application embodiment proposes a more effective KTM method, which first identifies the keywords in the data and then replaces the words in the keywords. This can increase the training task while forcing the model to learn the semantic dependencies in the entire sentence. Figure 5As shown in the figure, the name entity "Zhang San" and the song entity "Can't Leave You" can be identified through the log. If you choose to process "Zhang San" and then randomly replace the word "San", then in the case of the existence of the names Zhang Er, Zhang San, and Zhang Si, the possible prediction results are "Er" (Zhang Er), "San" (Zhang San), and "Si" (Zhang Si). Through the previous "Play a Song", it is highly likely that the current name is a singer. Therefore, in the case that only Zhang San and Zhang Si are singers, "Er" can be excluded. According to the following song name "Can't Leave You", it can be determined that the current word is "San". In this way, through training, the model can learn the dependency between semantic information. At the same time, the entire KTM pre-training method is combined with the actual downstream application scenarios. By focusing on the keywords in the text that contribute more to the results, the dependency between semantics is captured, which improves the training effect of the model and the semantic understanding effect of the trained model on short texts.

[0108] In one optional implementation manner, the above S430 may specifically include:

[0109] Using preset mask characters, target characters in at least part of the keywords are replaced to obtain model input text corresponding to each corpus.

[0110] Here, the preset mask character may be, for example, the symbol [mask]. For example, the symbol [mask] may be used to replace the target character to be replaced in each keyword to cover the target character, thereby obtaining the text covered by the character, that is, the model input text corresponding to the corpus. The target character may be one or more characters randomly selected from the corresponding keyword.

[0111] In a specific example, for the corpus "play a song by Zhang San that I can't live without you", the keywords are "Zhang San" and "can't live without you", and one of the characters in each keyword can be randomly selected for replacement, for example, "three" in "Zhang San" can be replaced, and "leaving" in "can't live without you" can be replaced, and then the model input text corresponding to the corpus can be obtained "play a song by [mask] that I can't live without you".

[0112] In this way, by using preset mask characters to replace the target characters in at least part of the keywords, the pre-trained model can predict the original characters replaced by the preset mask characters during the training process, thereby increasing the difficulty of model training while forcing the model to learn the semantic dependency features between keywords, thereby improving the training effect of the pre-trained model.

[0113] In addition to the above-mentioned method of replacing characters using a preset mask character, different target characters may also be replaced according to words similar to the keyword. In another optional implementation, the above-mentioned S430 may specifically include:

[0114] Perform the following steps for each keyword in at least some of the keywords to obtain the model input text corresponding to each corpus:

[0115] Get target words similar to keywords;

[0116] According to the target word, the target characters in the keyword are replaced.

[0117] Here, the target character may be one or more characters in the keyword that are different from the target word. The target word may be a synonym of the keyword or a word whose similarity exceeds a preset threshold, which is not limited here.

[0118] In a specific example, for the corpus "Play a song by Zhang San that I can't live without you", the keywords are "Zhang San" and "Can't live without you". A word similar to the keyword "Zhang San" may be, for example, "Zhang Si", and a word similar to the keyword "Can't live without you" may be, for example, "Can't let go of you". In this way, the "three" in "Zhang San" can be replaced with "four", and the "leave" in "Can't live without you" can be replaced with "let go", and then the model input text "Play a song by Zhang Si that I can't let go of you" corresponding to the corpus can be obtained.

[0119] In this way, by adopting the method of similar word replacement, we can avoid the problem of character replacement mismatch between pre-training and downstream tasks, increase the difficulty of pre-training tasks, and improve the model's ability to represent learning.

[0120] S440. Take the target character as the first sample label, the scene type corresponding to the scene information and the intent type corresponding to the intent information as the second sample label, and the slot type corresponding to the slot information as the third sample label. Add three sample labels to the model input text corresponding to each corpus to obtain multiple training samples.

[0121] Here, in the process of constructing training samples, three sample labels may be added to the corpus in each training sample to construct training samples corresponding to the three pre-training tasks. Different training tasks may correspond to training samples with different sample labels added.

[0122] Exemplarily, the ROSITA model provided in the embodiment of the present application may include three training tasks, namely, the prediction task corresponding to KTM, the intent recognition task, and the NER (Named Entity Recognition) task. The model input texts of these three tasks are the same, but due to the different training tasks, the added sample labels will be different. For example, the prediction task corresponding to KTM predicts the word corresponding to the replaced character, so the replaced original character can be used as the prediction label; the intent recognition task in a specific scenario is used to classify the intent of the entire model input text, so that a more fine-grained classification representation information can be obtained, so the intent category of the text in the corresponding scenario is used as the prediction label; the NER task performs entity recognition on the input text, so the entity category (that is, the slot category) of each word in the input text is marked as the prediction label, and the word that is not an entity is replaced with the character "O".

[0123] S450. Use multiple training samples to train an initial model to obtain a pre-trained model; wherein the training tasks corresponding to the initial model include a first training task, a second training task and a third training task, the first training task corresponds to a first sample label, the second training task corresponds to a second sample label, and the third training task corresponds to a third sample label.

[0124] Here, the initial model may be an untrained ROSITA model, and correspondingly, the pre-trained model may be a trained ROSITA model. The first training task may be the prediction task corresponding to the KTM mentioned above, and the training task may use the first sample label of the training sample for model training; the second training task may be the intention recognition task mentioned above, and the training task may use the second sample label of the training sample for model training; the third training task may be the NER task mentioned above, and the training task may use the third sample label of the training sample for model training.

[0125] In the embodiment of the present application, a model training method combining self-supervision and supervision is adopted to carry out joint training of multiple tasks. Among them, for the prediction task corresponding to KTM, a self-supervision model training method is adopted; for the intent recognition task and NER task, a supervised model training method is adopted.

[0126] In addition, in the slot extraction task, that is, the NER task, since the prediction task corresponding to the KTM proposed in the embodiment of the present application and the NER task are added during the pre-training process of the model, the model learns long-distance feature dependencies while having a stronger sense of the boundaries of the slot values.

[0127] S460. Fine-tune the pre-trained model according to the training data corresponding to the target module function to obtain a target function module with the target module function; wherein the target module function includes a scene classification function, an intent recognition function, a slot extraction function and a classification verification function.

[0128] Here, using the training data corresponding to the scene classification function, the pre-trained model is fine-tuned and trained to obtain a scene classification module with a scene classification function; using the training data corresponding to the intent recognition function in a specific scene type, the pre-trained model is fine-tuned and trained to obtain an intent recognition module with an intent recognition function corresponding to the specific scene type; using the training data corresponding to the slot extraction function in the specific scene type, the pre-trained model is fine-tuned and trained to obtain a slot extraction module with a slot extraction function corresponding to the specific scene type; using the training data corresponding to the classification verification function in the specific scene type, the pre-trained model is fine-tuned and trained to obtain a slot extraction module with a classification verification function corresponding to the specific scene type.

[0129] In this way, on the one hand, by relying on the rich structured data in the downstream tasks during the pre-training process of the pre-training model, the intention information and slot information of each corpus in the user's historical query log in the actual downstream application scenario are used as sample labels, and the self-supervised training of the first training task is combined with the supervised training of the second training task and the third training task to perform multi-task joint training. In this way, the robustness of the model can be enhanced while fitting the downstream application scenario. On the other hand, the embodiment of the present application also replaces the target characters in the keywords by paying attention to the keywords with high contribution to the model prediction results in the original text, so that the pre-training model predicts the original characters corresponding to the replaced characters. Compared with the method of randomly covering any characters for model prediction in MLM, the embodiment of the present application can more accurately capture the dependency between semantics, so that the training effect of the pre-training model is better, and it is easier to capture the semantics of short texts, thereby improving the semantic understanding effect of the model while improving the robustness of the model.

[0130] In addition, the addition of tone information can also enhance the model's sensitivity to special pronunciation slots. For example, for a user's voice input with a dialect "open the music player to play You Shan Lian", the APP slot is often misidentified as "yingyue player" and "yousannian". In an optional implementation, the above S450 may specifically include:

[0131] For each training sample, obtain the corresponding word vector, position vector and tone vector for each character in the model input text, and add them together to form the initial vector of the model input text;

[0132] An initial model is trained according to initial vectors of model input texts corresponding to a plurality of training samples to obtain a pre-trained model.

[0133] Here, the tone information of the characters can be incorporated into the training process of the pre-trained model, such as Figure 6 As shown in the figure, for each character in the text, the corresponding word vector, position vector and tone vector can be obtained respectively, and these three vectors are added together to form the initial vector representation of the text, that is, the initial vector corresponding to the sample. The initial vector is used as the input vector and input into the initial model for pre-training of the model to obtain a pre-trained model.

[0134] In this way, by summarizing and incorporating intonation information during the model pre-training process, it is possible to extract important information from sentences even when the ASR is incorrect, and achieve accurate semantic understanding, thereby improving the model's error correction capabilities. In addition, by incorporating intonation information during the model pre-training process, since the pre-training corpus is larger than the general model training corpus and the pre-training model structure is more complex, more general features can be learned and have a certain degree of migration capability. When migrating to other tasks, there is no need to train the model from scratch.

[0135] In an optional embodiment, the initial model may include at least one encoding layer, a first fully connected layer, a second fully connected layer and a third fully connected layer; wherein the first fully connected layer corresponds to the first training task, the second fully connected layer corresponds to the second training task, and the third fully connected layer corresponds to the third training task.

[0136] Based on this, the initial model is trained according to the initial vectors of the model input texts corresponding to the multiple training samples to obtain the pre-trained model, which may specifically include:

[0137] Perform the following steps on each of the multiple training samples until the initial model converges to obtain a pre-trained model:

[0138] Inputting the initial vector of the model input text corresponding to the training sample into at least one encoding layer, performing deep encoding on the initial vector of the model input text using at least one encoding layer, and outputting an encoding vector;

[0139] Input the encoding vector to the first fully connected layer, the second fully connected layer and the third fully connected layer respectively, and use the first fully connected layer, the second fully connected layer and the third fully connected layer to transform the encoding vector dimension respectively, so as to obtain the first vector output by the first fully connected layer, the second vector output by the second fully connected layer and the third vector output by the third fully connected layer;

[0140] Determine a predicted character according to the first vector, determine an intention type in a predicted scenario according to the second vector, and determine a predicted slot type according to the third vector;

[0141] Based on the first training task, determining a first loss value according to the predicted character and the first sample label, based on the second training task, determining a second loss value according to the intent type in the predicted scenario and the second sample label, and, based on the third training task, determining a third loss value according to the predicted slot type and the third sample label;

[0142] Calculate a model loss value according to the first loss value, the second loss value, and the third loss value;

[0143] According to the model loss value, adjust the model parameters of the initial model.

[0144] Here, the initial model ROSITA provided in the embodiment of the present application may be composed of an encoder and fully connected layers corresponding to three training tasks respectively. The encoder may include one or more encoding layers, for example, a 12-layer TransformerEncoder is used as the encoder.

[0145] For example, Figure 7 As shown in the figure, the model training process is as follows: the original text is used to generate the model input text through the mask mechanism of KTM, and the corresponding word vector is obtained for each word in the text. The position vector and the tone vector are added to form the initial vector of the text. The initial vector representation is passed through a 12-layer stacked Transformer Encoder layer to deeply encode the semantics and generate an encoding vector. The encoding vector is transformed into dimensions through three fully connected layers Feed Forward, and each Feed Forward is trained for a training task.

[0146] Specifically, the output vector of the last layer of Transformer Encoder can be output as a vector representation after the Feed Forward of the prediction task corresponding to KTM, which is also the first vector. The length of the vector representation is equal to the sentence length of the original text, and the dimension size is equal to the size of the vocabulary, which is equivalent to the output of each position word corresponding to a probability distribution of the size of the vocabulary. For the replaced character position, the character with the highest probability is selected as the predicted character of the current position. The loss is then calculated based on the predicted character and the actual character label, and the loss at each replaced character position is summed up as the total KTM loss value, which is also the first loss value.

[0147] In addition, after the output vector of the last layer of Transformer Encoder passes through the FeedForward of the intent recognition task, a vector representation can be obtained, which is also the second vector. The length of the vector representation is equal to the sentence length of the original text, and the dimension size is equal to the number of all intent types. Select the vector corresponding to any character (for example, the first character) in the vector representation for intent classification, and obtain the intent type in the predicted scenario. Then calculate the loss based on the intent type and the actual type label to obtain the loss value of the intent classification, which is also the second loss value.

[0148] In addition, after the output vector of the last layer Transformer Encoder passes through the FeedForward of the NER task, a vector representation can also be obtained, which is the third vector. The length of this vector representation is equal to the sentence length of the original text, and the dimension size is equal to the number of all slot types. The corresponding slot type is predicted for the character at each position to obtain the predicted slot type. The loss is then calculated based on the slot type and the actual type label, and the loss of each character is summed up as the total NER loss, which is the third loss value.

[0149] Finally, the total loss value of the model, that is, the model loss value, is calculated using the following formula:

[0150] The total loss value of the model = first loss value + second loss value + third loss value

[0151] After calculating the model loss value, the gradient is calculated according to the model loss value to update the model parameters. At the same time, the model is verified during the model training process. If the verification effect meets the standard, that is, the model converges, the training ends; if the verification effect does not meet the standard, the training continues.

[0152] In this way, the pre-trained model obtained through the above model training process adopts a multi-task joint training method that combines self-supervision and supervision. Therefore, it can fit the downstream application scenarios while enhancing the robustness of the model.

[0153] Based on the various implementation methods of the above model pre-training method, a specific example is given below to more clearly describe the entire technical solution. Figure 8 As shown, the model pre-training method provided in the embodiment of the present application may include the following steps:

[0154] S801, collect data;

[0155] S802, data preprocessing;

[0156] Among them, preprocessing includes but is not limited to data cleaning, deduplication, masking of characters in keywords based on the KTM training mechanism, task data alignment, data segmentation, etc.

[0157] S803, constructing training samples;

[0158] Specifically, one training sample can correspond to three training labels, corresponding to three training tasks: the prediction task corresponding to KTM, the intent recognition task, and the NER task.

[0159] S804, using the training samples to perform multi-task joint training on the pre-trained model;

[0160] Specifically, a combination of self-supervision and supervision is adopted to conduct multi-task joint training on the model with the three training tasks corresponding to KTM, namely prediction task, intention recognition task and NER task. In addition, intonation information can also be incorporated into the model training process to improve the model's error correction ability.

[0161] S805: Verify the pre-trained model and determine whether the verification result meets the requirements. If so, end the training process; if not, return to execute S804.

[0162] In this way, by adopting the KTM training mechanism and training the pre-trained model using a multi-task joint training method that combines self-supervision and supervision, the robustness of the pre-trained model can be enhanced while fitting the application scenario of short text semantic understanding of the voice assistant.

[0163] Based on the pre-trained model obtained by the above training, the embodiment of the present application can use the pre-trained model to perform fine-tuning training to obtain the various functional modules required in the downstream tasks, which will not be repeated here.

[0164] Based on the same inventive concept, the present application also provides an operation execution device. Fig. 9 The operation execution device provided in the embodiment of the present application is described in detail.

[0165] Fig. 9 The figure is a structural block diagram of an operation execution device according to an exemplary embodiment.

[0166] like Fig. 9 As shown, the operation execution device 900 may include:

[0167] An input receiving module 901 is used to receive a first input from a user;

[0168] A vector acquisition module 902 is used to obtain a corresponding word vector, position vector and tone vector for each character in a first text in response to a first input, and add them together to form an initial vector of the first text; wherein the first text is the text corresponding to the first input;

[0169] A scene classification module 903, configured to perform depth encoding and dimension conversion on the initial vector based on the scene classification module to obtain a first scene type;

[0170] The recognition and extraction module 904 is used to perform depth encoding and dimension conversion on the initial vector based on the intent recognition module and the slot extraction module corresponding to the first scene type to obtain the first intent information and the first slot information;

[0171] The operation execution module 905 is used to execute the operation corresponding to the first input according to the first intention information and the first slot information.

[0172] The operation execution device 900 is described in detail below, as follows:

[0173] In one embodiment, the operation execution device 900 may further include:

[0174] A scene verification module, configured to perform deep encoding and dimensional conversion on the initial vector based on the scene classification module to obtain a first scene type, and then perform deep encoding and dimensional conversion on the initial vector based on a classification verification module corresponding to the first scene type to verify whether the scene type of the first input is the first scene type, and obtain a verification result;

[0175] Identification and extraction module, specifically used for:

[0176] When it is determined according to the verification result that the scene type of the first input is the first scene type, the initial vector is deep encoded and dimensionally converted based on the intent recognition module and slot extraction module corresponding to the first scene type to obtain first intent information and first slot information.

[0177] In one embodiment, the scene classification module, the recognition and extraction module, and the scene verification module may include:

[0178] The encoding submodule is used to perform deep encoding on the initial vector to obtain an encoding vector;

[0179] The conversion submodule is used to convert the dimension of the encoding vector to obtain the corresponding probability distribution;

[0180] A determination submodule is used to determine, according to the probability distribution, the information with the largest probability value as the target information;

[0181] Among them, the target information is the information obtained based on the target function module through the above-mentioned operation steps. The target function module includes a scene classification module, an intent recognition module, a slot extraction module and a classification verification module. The target information includes a first scene type, first intent information, first slot information and a verification result.

[0182] In one embodiment, the operation execution device 900 may further include:

[0183] An information acquisition module, used for acquiring a plurality of corpora and their corresponding scene information, intent information and slot information according to the user's historical query log before receiving the user's first input;

[0184] A keyword recognition module, used to recognize at least one keyword contained in the original text corresponding to each corpus;

[0185] A character replacement module is used to replace target characters in at least part of the keywords to obtain model input texts corresponding to each corpus; wherein the target characters are part of the keywords;

[0186] A label adding module is used to take the target character as the first sample label, the scene type corresponding to the scene information and the intent type corresponding to the intent information as the second sample label, and the slot type corresponding to the slot information as the third sample label, and add three sample labels to the model input text corresponding to each corpus to obtain multiple training samples;

[0187] A model training module is used to train an initial model using multiple training samples to obtain a pre-trained model; wherein the training tasks corresponding to the initial model include a first training task, a second training task and a third training task, the first training task corresponds to a first sample label, the second training task corresponds to a second sample label, and the third training task corresponds to a third sample label;

[0188] The fine-tuning training module is used to fine-tune the pre-trained model according to the training data corresponding to the target module function to obtain a target function module with the target module function; wherein the target module function includes scene classification function, intent recognition function, slot extraction function and classification verification function.

[0189] In one embodiment, the above-mentioned model training module includes:

[0190] The vector acquisition submodule is used to obtain the corresponding word vector, position vector and tone vector for each character in the model input text for each training sample, and add them together to form the initial vector of the model input text;

[0191] The model training submodule is used to train the initial model according to the initial vectors of the model input text corresponding to the multiple training samples to obtain the pre-trained model.

[0192] In one embodiment, the initial model includes at least one encoding layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer; wherein the first fully connected layer corresponds to the first training task, the second fully connected layer corresponds to the second training task, and the third fully connected layer corresponds to the third training task;

[0193] The above-mentioned model training submodule may specifically include:

[0194] The model training unit is used to perform the following operation steps on each training sample in the multiple training samples until the initial model converges to obtain a pre-trained model:

[0195] Among them, the model training unit includes:

[0196] A vector encoding subunit, used to input the initial vector of the model input text corresponding to the training sample into at least one encoding layer, perform deep encoding on the initial vector of the model input text using at least one encoding layer, and output an encoded vector;

[0197] a dimension conversion subunit, used to input the encoding vector into the first fully connected layer, the second fully connected layer and the third fully connected layer respectively, and perform dimension conversion on the encoding vector using the first fully connected layer, the second fully connected layer and the third fully connected layer respectively, to obtain a first vector output by the first fully connected layer, a second vector output by the second fully connected layer and a third vector output by the third fully connected layer;

[0198] A prediction determination subunit, configured to determine a predicted character according to the first vector, determine an intention type in a prediction scenario according to the second vector, and determine a predicted slot type according to the third vector;

[0199] A loss determination subunit, configured to determine a first loss value based on the first training task, according to the predicted character and the first sample label, determine a second loss value based on the second training task, according to the intent type in the predicted scenario and the second sample label, and determine a third loss value based on the third training task, according to the predicted slot type and the third sample label;

[0200] A loss calculation subunit, used to calculate a model loss value according to the first loss value, the second loss value and the third loss value;

[0201] The parameter adjustment subunit is used to adjust the model parameters of the initial model according to the model loss value.

[0202] In one embodiment, the character replacement module may specifically include:

[0203] The character replacement submodule is used to use preset mask characters to replace the target characters in at least part of the keywords to obtain the model input text corresponding to each corpus.

[0204] In one embodiment, the character replacement module may specifically include:

[0205] The text acquisition submodule is used to perform the following operation steps on each keyword in at least some of the keywords to obtain the model input text corresponding to each corpus:

[0206] Among them, the text acquisition submodule includes:

[0207] A word acquisition unit, used for acquiring target words similar to the keyword;

[0208] The character replacement unit is used to replace target characters in the keyword according to the target word.

[0209] In this way, when receiving the user's first input, in addition to extracting the word vector and position vector corresponding to each character in the first text corresponding to the first input, the tone vector of each character is also extracted, and then based on the initial vector of the first text formed by adding these vectors, scene classification, intent recognition and slot extraction are performed to identify the semantics of the first text and perform operations corresponding to the first input. In this way, since the embodiment of the present application adds the tone vector when generating the initial vector of the first text corresponding to the first input, even if the ASR is wrong, it can extract important information in the sentence and achieve accurate semantic understanding, thereby improving the fault tolerance of semantic understanding and avoiding the problem of failure of semantic understanding and inability to perform corresponding operations due to inaccurate reception or dialect pronunciation.

[0210] The operation execution device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0211] The operation execution device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0212] The operation execution device provided in the embodiment of the present application can realize Figures 1 to 8 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0213] Alternatively, if Fig.10 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1001, a memory 1002, and a program or instruction stored in the memory 1002 and executable on the processor 1001. When the program or instruction is executed by the processor 1001, each process of the above-mentioned operation execution method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0214] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0215] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0216] The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, and a processor 1110 and other components.

[0217] Those skilled in the art will appreciate that the electronic device 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0218] The input unit 1104 is used to receive a first input from a user;

[0219] Processor 1110 is used to, in response to a first input, obtain the corresponding word vector, position vector and tone vector for each character in the first text, and add them together to form an initial vector of the first text; wherein the first text is the text corresponding to the first input; based on the scene classification module, perform deep encoding and dimension conversion on the initial vector to obtain a first scene type; based on the intent recognition module and slot extraction module corresponding to the first scene type, perform deep encoding and dimension conversion on the initial vector to obtain first intent information and first slot information; and execute an operation corresponding to the first input according to the first intent information and the first slot information.

[0220] In this way, when receiving the user's first input, in addition to extracting the word vector and position vector corresponding to each character in the first text corresponding to the first input, the tone vector of each character is also extracted, and then based on the initial vector of the first text formed by adding these vectors, scene classification, intent recognition and slot extraction are performed to identify the semantics of the first text and perform operations corresponding to the first input. In this way, since the embodiment of the present application adds the tone vector when generating the initial vector of the first text corresponding to the first input, even if the ASR is wrong, it can extract important information in the sentence and achieve accurate semantic understanding, thereby improving the fault tolerance of semantic understanding and avoiding the problem of failure of semantic understanding and inability to perform corresponding operations due to inaccurate reception or dialect pronunciation.

[0221] Optionally, processor 1110 is also used to perform depth encoding and dimensional conversion on the initial vector based on a classification verification module corresponding to the first scene type to verify whether the scene type of the first input is the first scene type to obtain a verification result; when it is determined that the scene type of the first input is the first scene type based on the verification result, the initial vector is deep encoded and dimensional converted based on an intent recognition module and a slot extraction module corresponding to the first scene type to obtain first intent information and first slot information.

[0222] Optionally, processor 1110 is further used to perform deep encoding on the initial vector to obtain a coding vector; perform dimension conversion on the coding vector to obtain a corresponding probability distribution; and determine, based on the probability distribution, information with the largest probability value as target information; wherein the target information is information obtained based on the target function module through the above-mentioned operation steps, the target function module includes a scene classification module, an intent recognition module, a slot extraction module and a classification verification module, and the target information includes a first scene type, first intent information, first slot information and a verification result.

[0223] Optionally, the processor 1110 is further used to obtain multiple corpora and their corresponding scene information, intent information and slot information according to the user's historical query log; identify at least one keyword contained in the original text corresponding to each corpus; replace the target characters in at least part of the keywords to obtain the model input text corresponding to each corpus; wherein the target characters are part of the characters in the keywords; take the target characters as the first sample label, the scene type corresponding to the scene information and the intent type corresponding to the intent information as the second sample label, and the slot type corresponding to the slot information as the third sample label, add three sample labels to the model input text corresponding to each corpus to obtain multiple training samples; use the multiple training samples to train the initial model to obtain a pre-trained model; wherein the training tasks corresponding to the initial model include a first training task, a second training task and a third training task, the first training task corresponds to the first sample label, the second training task corresponds to the second sample label, and the third training task corresponds to the third sample label;

[0224] The pre-trained model is fine-tuned according to the training data corresponding to the target module function to obtain a target function module with the target module function; wherein the target module function includes a scene classification function, an intent recognition function, a slot extraction function and a classification verification function.

[0225] Optionally, the processor 1110 is also used to obtain the corresponding word vector, position vector and tone vector for each character in the model input text for each training sample, and add them together to form an initial vector of the model input text; train the initial model according to the initial vectors of the model input text corresponding to multiple training samples, so as to obtain a pre-trained model.

[0226] Optionally, the processor 1110 is further used to perform the following operation steps on each training sample in the multiple training samples until the initial model converges to obtain a pre-trained model: inputting an initial vector of the model input text corresponding to the training sample into at least one encoding layer, using at least one encoding layer to perform deep encoding on the initial vector of the model input text, and outputting an encoding vector; inputting the encoding vector into a first fully connected layer, a second fully connected layer, and a third fully connected layer, respectively, using the first fully connected layer, the second fully connected layer, and the third fully connected layer to perform dimensional transformation on the encoding vector, respectively, to obtain a first vector output by the first fully connected layer, a second vector output by the second fully connected layer, and a third vector output by the third fully connected layer. vector and a third vector output by the third fully connected layer; determine the predicted character according to the first vector, determine the intent type in the prediction scenario according to the second vector, and determine the predicted slot type according to the third vector; determine a first loss value based on the first training task, according to the predicted character and the first sample label, determine a second loss value based on the second training task, according to the intent type in the prediction scenario and the second sample label, and determine a third loss value based on the third training task, according to the predicted slot type and the third sample label; calculate the model loss value according to the first loss value, the second loss value and the third loss value; adjust the model parameters of the initial model according to the model loss value.

[0227] Optionally, the processor 1110 is further configured to use preset mask characters to replace target characters in at least part of the keywords to obtain model input text corresponding to each corpus.

[0228] Optionally, the processor 1110 is further used to perform the following operation steps on each keyword in at least some of the keywords to obtain the model input text corresponding to each corpus: obtaining target words similar to the keywords; and replacing target characters in the keywords according to the target words.

[0229] In this way, the embodiment of the present application proposes a new pre-training model ROSITA through an operation execution scenario based on voice recognition, and designs targeted pre-training tasks for actual application scenarios, thereby improving the semantic understanding effect of the voice assistant while improving the robustness of the model.

[0230] It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processor (Graphics Processing Unit, GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 1109 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 1110 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1110.

[0231] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned operation execution method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0232] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0233] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned operation execution method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0234] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0235] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0236] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0237] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An operation execution method, characterized in that: include: receiving a first input from a user; In response to the first input, obtaining a corresponding word vector, position vector and tone vector for each character in the first text respectively, and adding them together to form an initial vector of the first text; wherein the first text is the text corresponding to the first input; Based on the scene classification module, the initial vector is deeply encoded and dimensionally converted to obtain a first scene type; Based on the intent recognition module and the slot extraction module corresponding to the first scene type, the initial vector is deep-encoded and dimension-converted to obtain first intent information and first slot information; According to the first intention information and the first slot information, perform an operation corresponding to the first input; The step of performing depth encoding and dimension conversion on the initial vector includes: Performing deep encoding on the initial vector to obtain an encoded vector; Performing dimension conversion on the encoding vector to obtain a corresponding probability distribution; According to the probability distribution, determine the information with the largest probability value as the target information; Among them, the target information is information obtained based on the target function module through the above-mentioned operation steps, the target function module includes a scene classification module, an intention recognition module and a slot extraction module, and the target information includes a first scene type, first intention information and first slot information.

2. The method according to claim 1, characterized in that After performing depth encoding and dimension conversion on the initial vector based on the scene classification module to obtain the first scene type, the method further includes: Based on the classification verification module corresponding to the first scene type, perform depth encoding and dimension conversion on the initial vector to verify whether the scene type of the first input is the first scene type, and obtain a verification result; The intention recognition module and the slot extraction module corresponding to the scene type perform depth encoding and dimension conversion on the initial vector to obtain first intention information and first slot information, including: When it is determined according to the verification result that the scene type of the first input is the first scene type, based on the intent recognition module and the slot extraction module corresponding to the first scene type, performing depth encoding and dimension conversion on the initial vector to obtain first intent information and first slot information; Among them, the target function module also includes a classification verification module, and the target information also includes a verification result.

3. The method according to claim 1, characterized in that Before receiving the first input from the user, the method further includes: According to the user's historical query log, multiple corpora and their corresponding scene information, intent information and slot information are obtained; Identify at least one keyword contained in the original text corresponding to each corpus; Performing character replacement on at least part of the target characters in the keywords to obtain model input texts corresponding to each corpus; wherein the target characters are part of the characters in the keywords; The target character is used as the first sample label, the scene type corresponding to the scene information and the intent type corresponding to the intent information are used as the second sample label, and the slot type corresponding to the slot information is used as the third sample label. Three sample labels are added to the model input text corresponding to each corpus to obtain multiple training samples. Using the multiple training samples to train an initial model to obtain a pre-trained model; wherein the training tasks corresponding to the initial model include a first training task, a second training task and a third training task, the first training task corresponds to the first sample label, the second training task corresponds to the second sample label, and the third training task corresponds to the third sample label; The pre-trained model is fine-tuned and trained according to the training data corresponding to the target module function to obtain a target function module having the target module function; wherein the target module function includes a scene classification function, an intent recognition function, a slot extraction function and a classification verification function.

4. The method according to claim 3, characterized in that The using the multiple training samples to train the initial model to obtain a pre-trained model includes: For each of the training samples, after obtaining the corresponding word vector, position vector and tone vector for each character in the model input text, the vectors are added to form an initial vector of the model input text; The initial model is trained according to the initial vectors of the model input texts respectively corresponding to the multiple training samples to obtain the pre-trained model.

5. The method according to claim 3, characterized in that: The step of replacing target characters in at least part of the keywords to obtain model input texts corresponding to each corpus includes: Perform the following operation steps for each keyword in at least some of the keywords to obtain the model input text corresponding to each corpus: Obtain target words similar to the keywords; According to the target word, character replacement is performed on the target characters in the keyword.

6. An operation execution device, characterized in that: include: An input receiving module, used for receiving a first input from a user; a vector acquisition module, configured to, in response to the first input, acquire a corresponding word vector, a position vector, and a tone vector for each character in a first text, and add them together to form an initial vector of the first text; wherein the first text is the text corresponding to the first input; A scene classification module, configured to perform depth encoding and dimension conversion on the initial vector based on the scene classification module to obtain a first scene type; an identification and extraction module, configured to perform depth encoding and dimension conversion on the initial vector based on the intent identification module and the slot extraction module corresponding to the first scene type, so as to obtain first intent information and first slot information; an operation execution module, configured to execute an operation corresponding to the first input according to the first intention information and the first slot information; The scene classification module and the recognition and extraction module include: An encoding submodule, used for performing deep encoding on the initial vector to obtain an encoding vector; A conversion submodule, used to perform dimension conversion on the encoding vector to obtain a corresponding probability distribution; A determination submodule, used to determine, according to the probability distribution, the information with the largest probability value as the target information; Among them, the target information is information obtained based on the target function module through the above-mentioned operation steps, the target function module includes a scene classification module, an intention recognition module and a slot extraction module, and the target information includes a first scene type, first intention information and first slot information.

7. The device according to claim 6, characterized in that The device also includes: A scene verification module, configured to, after performing depth encoding and dimension conversion on the initial vector based on the scene classification module to obtain a first scene type, perform depth encoding and dimension conversion on the initial vector based on a classification verification module corresponding to the first scene type to verify whether the scene type of the first input is the first scene type, and obtain a verification result; The identification and extraction module is specifically used for: When it is determined according to the verification result that the scene type of the first input is the first scene type, based on the intent recognition module and the slot extraction module corresponding to the first scene type, performing depth encoding and dimension conversion on the initial vector to obtain first intent information and first slot information; Among them, the target function module also includes a classification verification module, and the target information also includes a verification result.

8. The device according to claim 6, characterized in that The device also includes: An information acquisition module, used for acquiring a plurality of corpora and their corresponding scene information, intent information and slot information according to the user's historical query log before receiving the user's first input; A keyword recognition module, used to recognize at least one keyword contained in the original text corresponding to each corpus; A character replacement module, used to replace target characters in at least part of the keywords to obtain model input texts corresponding to each corpus; wherein the target characters are part of the characters in the keywords; a label adding module, configured to use the target character as a first sample label, the scene type corresponding to the scene information and the intent type corresponding to the intent information as a second sample label, and the slot type corresponding to the slot information as a third sample label, and add three sample labels to the model input text corresponding to each corpus to obtain multiple training samples; A model training module, configured to train an initial model using the multiple training samples to obtain a pre-trained model; wherein the training tasks corresponding to the initial model include a first training task, a second training task and a third training task, the first training task corresponds to the first sample label, the second training task corresponds to the second sample label, and the third training task corresponds to the third sample label; A fine-tuning training module is used to fine-tune the pre-trained model according to the training data corresponding to the target module function to obtain a target function module with the target module function; wherein the target module function includes a scene classification function, an intent recognition function, a slot extraction function and a classification verification function.

9. The device according to claim 8, characterized in that The model training module includes: A vector acquisition submodule, for acquiring, for each of the training samples, the corresponding word vector, position vector and tone vector for each character in the model input text, and adding them together to form an initial vector of the model input text; The model training submodule is used to train the initial model according to the initial vectors of the model input texts corresponding to the multiple training samples respectively, so as to obtain the pre-trained model.

10. The device according to claim 8, characterized in that The character replacement module comprises: The text acquisition submodule is used to perform the following operation steps on each keyword in at least part of the keywords to obtain the model input text corresponding to each corpus: Wherein, the text acquisition submodule includes: A word acquisition unit, used for acquiring a target word similar to the keyword; The character replacement unit is used to replace the target characters in the keyword according to the target word.

11. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the operation execution method as described in any one of claims 1 to 5.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the operation execution method according to any one of claims 1 to 5 are implemented.

Images