Data processing method and device

Abstract

The embodiment of the invention provides a data processing method and device, and the method comprises the steps: obtaining answer audio submitted for a question text, and inputting the question text and the answer audio into an audio recognition model; encoding the question text and the answer audio by using an encoding module of the audio recognition model to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio; and decoding the audio vector by using the text vector through a semantic analysis module of the audio recognition model to obtain an answer text corresponding to the answer audio output by the audio recognition model. And the answer audio is identified in combination with the question text corresponding to the answer audio, so that the accuracy of converting the answer audio into the answer text is improved.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of data processing technology, and more particularly to a data processing method and device. Background Art

[0002] With the development of computer technology, the application scenarios of conversational interaction are becoming more and more extensive. Speech recognition models can provide speech recognition services in a variety of scenarios such as conversational learning, voice customer service, and speech-to-text. In oral conversational interaction scenarios, speech recognition is required to convert the speech submitted by the user into text, so as to facilitate the provision of subsequent voice interaction services to the user.

[0003] In the prior art, it is usually necessary to establish a "hot word library" and combine the words stored in the "hot word library" to assist in speech recognition. However, this method requires the maintenance of the "hot word library", which will consume a lot of human resources to update the "hot word library". The influence of the "hot word library" will reduce the recognition accuracy of the speech recognition model. Therefore, a more effective method is urgently needed to solve the above problem. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a data processing method. One or more embodiments of this specification also relate to a data processing device, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:

[0006] Obtaining an answer audio submitted for a question text, and inputting the question text and the answer audio into an audio recognition model;

[0007] Using the encoding module of the audio recognition model to encode the question text and the answer audio respectively, to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio;

[0008] The audio vector is decoded using the text vector through the semantic analysis module of the audio recognition model to obtain an answer text corresponding to the answer audio output by the audio recognition model.

[0009] Optionally, the encoding module of the audio recognition model is used to encode the question text and the answer audio respectively to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio, including:

[0010] Using a text encoding unit in an encoding module of the audio recognition model to encode the question text to obtain a text vector corresponding to the question text;

[0011] The answer audio is encoded using the audio encoding unit in the encoding module of the audio recognition model to obtain an audio vector corresponding to the answer audio.

[0012] Optionally, determining the text vector corresponding to the question text includes:

[0013] Using the mask subunit in the text encoding unit to perform mask processing on the question text to obtain a mask vector corresponding to the question text;

[0014] The mask vector is encoded by using the first text encoding subunit in the text encoding unit to obtain a text vector corresponding to the question text.

[0015] Optionally, determining the audio vector corresponding to the answer audio includes:

[0016] Processing the answer audio by using the audio sequence processing subunit in the audio encoding unit to obtain an answer sequence corresponding to the answer audio;

[0017] The answer sequence is encoded using the audio encoding subunit in the audio encoding unit to obtain an audio vector corresponding to the answer audio.

[0018] Optionally, after the audio vector is decoded by the text vector using the semantic analysis module of the audio recognition model to obtain the answer text corresponding to the answer audio output by the audio recognition model, the method further includes:

[0019] Determining question semantic information of the question text and answer semantic information of the answer text;

[0020] Matching the question semantic information with the answer semantic information, and detecting the answer text according to the matching result;

[0021] Feedback data corresponding to the answer audio is determined based on the detection result, and the feedback data is sent to the client that submitted the answer audio.

[0022] Optionally, the training of the audio recognition model includes:

[0023] Determine a target sample pair in the sample set;

[0024] Inputting the question text sample and the answer audio sample in the target sample pair into an initial audio recognition model;

[0025] Processing the question text sample and the answer audio sample using the initial audio recognition model to obtain answer recognition text corresponding to the answer audio sample output by the audio recognition model;

[0026] A loss value is calculated based on the recognition reference text and the answer recognition text in the target sample pair, and the parameters of the initial audio recognition model are adjusted based on the loss value until an audio recognition model that meets the training stop condition is obtained.

[0027] Optionally, obtaining the answer audio submitted for the question text includes:

[0028] Obtaining initial answer audio submitted to the question text;

[0029] Performing shearing processing on the initial answer audio, and determining the answer audio to be processed according to the shearing processing result;

[0030] The answer audio to be processed is subjected to noise reduction processing, and the answer audio is determined according to the noise reduction processing result.

[0031] Optionally, after the audio vector is decoded by the text vector using the semantic analysis module of the audio recognition model to obtain the answer text corresponding to the answer audio output by the audio recognition model, the method further includes:

[0032] Using the question text and the answer audio as model tuning samples, and using the answer text as a model tuning label;

[0033] A model tuning sample pair is composed of the model tuning sample and the model tuning label and stored in a model tuning sample set, wherein the model tuning sample set is used to perform tuning training on the audio recognition model.

[0034] Optionally, determining the text vector corresponding to the question text includes:

[0035] Using the word segmentation subunit in the text encoding unit to perform word segmentation processing on the question text to obtain text keywords corresponding to the question text;

[0036] The text keywords are encoded using the second text encoding subunit in the text encoding unit to obtain a text vector corresponding to the question text.

[0037] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:

[0038] An acquisition module is configured to acquire an answer audio submitted for a question text, and input the question text and the answer audio into an audio recognition model;

[0039] An encoding module is configured to use the encoding module of the audio recognition model to encode the question text and the answer audio respectively to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio;

[0040] The decoding module is configured to decode the audio vector using the text vector through the semantic analysis module of the audio recognition model to obtain the answer text corresponding to the answer audio output by the audio recognition model.

[0041] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0042] Memory and processor;

[0043] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above data processing method are implemented.

[0044] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0045] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0046] An embodiment of the present specification obtains the answer audio submitted for the question text, and inputs the question text and the answer audio into the audio recognition model; uses the encoding module of the audio recognition model to encode the question text and the answer audio respectively, and obtains the text vector corresponding to the question text, and the audio vector corresponding to the answer audio; through the semantic analysis module of the audio recognition model, uses the text vector to decode the audio vector, and obtains the answer text corresponding to the answer audio output by the audio recognition model. Without the need to manually maintain any vocabulary, the answer audio can be recognized in combination with the question text corresponding to the answer audio, thereby improving the accuracy of converting the answer audio into the answer text. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of an answer audio recognition process of a data processing method provided by an embodiment of this specification;

[0048] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0049] Figure 3 is a processing flow chart of a data processing method provided by an embodiment of this specification;

[0050] Figure 4 is a schematic diagram of an audio recognition model of a data processing method provided by an embodiment of this specification;

[0051] Figure 5 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0052] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0053] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0054] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0056] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0057] First, the terms involved in one or more embodiments of this specification are explained.

[0058] Transformer: A sequence model based on the attention mechanism. Transformer only uses the self-attention mechanism to process the input sequence and the output sequence, so it can be calculated in parallel. Transformer is essentially an Encoder-Decoder structure. The encoder consists of 6 encoding blocks (each encoder block consists of self-attention and FFNN), and the decoder consists of 6 decoding blocks (each decoder block consists of self-attention, encoder-decoder attention and FFNN). Like all generative models, the output of the encoder will be used as the input of the decoder.

[0059] Fbank: A commonly used method for extracting speech signal features, its full name is filter bank feature.

[0060] specAugment: A data augmentation method commonly used in speech recognition. This method can make the model recognize better in noisy environments and improve the generalization ability of the model. specAugment mainly performs three operations: frequency masking, time masking, and time-frequency masking.

[0061] Frequency Masking: Select a continuous frequency band in the audio spectrum and set its value to zero. The purpose of this operation is to simulate possible frequency distortion and train the model to perform accurate speech recognition even when some frequency information is missing.

[0062] Time Masking: Selects a contiguous time range in the audio spectrum and sets its value to zero. This simulates possible temporal distortions, such as short silences in speech or pauses in the speaker.

[0063] Time-frequency Masking: A combination of time-frequency masking and frequency masking.

[0064] Convolution subsampling: subsampling selects the output of each specific interval through convolution, thereby reducing the length of the sequence and retaining most of the important features. On the other hand, it can also significantly reduce the computational cost.

[0065] Softmax: Normalized exponential function. It is a generalization of the binary classification function to multi-classification, with the purpose of presenting the results of multi-classification in the form of probability. It maps the outputs of multiple neurons to the interval (0, 1), which can be understood as probability, so as to perform multi-classification.

[0066] Figure 1 This is a schematic diagram of the answer audio recognition process of a data processing method provided by an embodiment of this specification; Figure 1 As shown, in a dialogue interaction scenario, the answer audio submitted by the user for the question text is obtained. The question text and the answer audio are input into the audio recognition model, and the audio recognition model can convert the answer audio into text in combination with the semantics of the question text. The encoding module of the audio recognition model is used to encode the question text and the answer audio respectively to obtain the text vector corresponding to the question text and the audio vector corresponding to the answer audio. Through the semantic analysis module of the audio recognition model, the audio vector is decoded using the text vector, and the audio vector is identified in combination with the semantic features of the question vector to obtain the answer text corresponding to the answer audio output by the audio recognition model. There is no need to manually maintain any vocabulary, and the answer audio can be recognized in combination with the question text corresponding to the answer audio, thereby improving the accuracy of converting the answer audio into the answer text.

[0067] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0068] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0069] Step S202: Obtain answer audio submitted for the question text, and input the question text and the answer audio into an audio recognition model.

[0070] Specifically, the question text refers to the question generated in the dialogue scenario, or any sentence that needs to be answered; correspondingly, the answer audio is the answer content in audio form corresponding to the question text; the audio recognition model is used to recognize the answer audio as text content. The text question can be the question text given by the question-answering system in the knowledge question-answering scenario, and the user is required to give an answer to the question. For example, in the English oral dialogue scenario, the question-answering system gives "What is your favorite color to use?", and the user is required to answer the question. The content of the user's answer is the answer audio.

[0071] Based on this, the question text is determined, and the answer audio submitted for the question text is obtained. The question text and the answer audio are input into the audio recognition model as a set of data consisting of questions and answers, so as to obtain the answer text corresponding to the answer audio output by the audio recognition model.

[0072] Furthermore, considering that the answer audio may contain noise and blank audio due to the influence of the environment and the speaking habits of the person submitting the answer audio, in order to improve the recognition accuracy of the subsequent audio recognition model, after obtaining the initial answer audio, the initial answer can be cut and denoised to obtain the answer audio that can be input into the audio recognition model. The specific implementation is as follows:

[0073] Acquire the initial answer audio submitted for the question text; perform shearing processing on the initial answer audio, and determine the answer audio to be processed according to the shearing processing result; perform noise reduction processing on the answer audio to be processed, and determine the answer audio according to the noise reduction processing result.

[0074] Specifically, the initial answer audio refers to the unprocessed audio submitted by the user for the question text; the cutting process refers to cropping the audio frames in the initial answer audio, cropping the blank frames in the initial answer audio, or if the initial answer audio is long, cropping the initial answer audio in segments to obtain the answer audio to be processed; the noise reduction process refers to processing the noise in the answer audio to be processed, so as to achieve the purpose of data enhancement of the answer audio to be processed; the noise reduction process can be achieved by methods such as linear filters, spectral subtraction, and real-time noise reduction algorithms based on statistical models; this embodiment does not impose any limitation on the method used for noise reduction.

[0075] Based on this, the question text is determined, and the initial answer audio submitted for the question text is obtained. The audio frames to be cut contained in the initial answer audio are determined, and the audio frames to be cut in the initial answer audio are cut, and the answer audio to be processed is determined according to the cutting processing result. The noise reduction algorithm is used to perform noise reduction processing on the answer audio to be processed, and the answer audio is determined according to the noise reduction processing result.

[0076] For example, the question text may be "what color do you like?", and the user gives an audio answer to this question in the form of "en, en, i, i like blue". For the answer audio, "en, en, i," in the answer can be cut, and the blank audio frames generated in the answer due to the user's thinking can be cut. After cutting the initial answer audio to obtain the answer audio to be processed, the answer audio to be processed can also be subjected to noise reduction processing to achieve the purpose of retaining the user's voice and removing environmental noise.

[0077] To summarize, after obtaining the initial answer audio, the initial answer is cut and denoised to obtain the answer audio that can be input into the audio recognition model, thereby improving the recognition accuracy of the subsequent audio recognition model.

[0078] Furthermore, the audio recognition model can be obtained by training based on sample pairs in the sample set, which is specifically implemented as follows:

[0079] Determine a target sample pair in a sample set; input the question text sample and the answer audio sample in the target sample pair into an initial audio recognition model; use the initial audio recognition model to process the question text sample and the answer audio sample to obtain an answer recognition text corresponding to the answer audio sample output by the audio recognition model; calculate a loss value based on the recognition reference text and the answer recognition text in the target sample pair, and adjust the parameters of the initial audio recognition model based on the loss value until an audio recognition model that meets the training stop condition is obtained.

[0080] Specifically, the sample set is used to store sample pairs, the sample pairs include training samples and sample labels, and the sample pairs are used to train the initial audio recognition model; the target sample pair is the sample pair selected in the sample set when training the initial audio recognition model; the question text sample and the answer audio sample in the target sample pair are training samples, the recognition reference text in the target sample pair is the sample label, the question text sample is the question text generated in the actual dialogue scene, and correspondingly, the answer audio sample is the answer audio submitted by the user to the question text in the actual dialogue scene; the recognition reference text can be the text obtained by recognizing the answer audio sample; the answer recognition text is the recognition result of the corresponding answer audio sample output by the initial audio recognition model after the answer audio sample is input into the initial audio recognition model; the training stop condition can be that the training of the initial audio recognition model reaches a preset training round, it can be that the loss value corresponding to the initial audio recognition model reaches a preset loss value threshold, or it can be that the trained initial audio recognition model is tested based on the test set to achieve a preset recognition accuracy. This embodiment does not impose any limitation on the training stop condition.

[0081] Based on this, a sample set for training the initial audio recognition model is obtained, and a target sample pair is determined in the sample set. The question text sample and the answer audio sample in the target sample pair are input into the initial audio recognition model, and the encoding module in the initial audio recognition model is used to encode the question text sample and the answer audio sample respectively, to obtain the question text sample vector corresponding to the question text sample, and the answer audio text vector corresponding to the answer audio sample; through the semantic analysis module in the initial audio recognition model, the answer audio text vector is decoded using the question text sample vector to obtain the answer recognition text corresponding to the answer audio sample output by the initial audio recognition model. The loss value is calculated based on the loss function of the initial audio recognition model, the answer recognition text, and the recognition reference text, and the initial audio recognition model is adjusted based on the loss value. Then, a sample pair for continuing model training is selected in the sample set, and so on, until an audio recognition model that meets the training stop condition is obtained.

[0082] Using the above example, the sample pairs in the sample set can be conversation data in an actual conversation scenario, question text: "what color do you like?", answer audio, and the label red determined based on the question text. You can also manually record and annotate samples to construct sample pairs. Then, train the initial audio recognition model based on the determined sample set.

[0083] In summary, the initial audio recognition model is trained based on the sample pairs in the sample set, so that the obtained audio recognition model has better audio recognition ability.

[0084] Step S204: Utilize the encoding module of the audio recognition model to encode the question text and the answer audio respectively, and obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio.

[0085] Specifically, after obtaining the answer audio submitted for the question text and inputting the question text and the answer audio into the audio recognition model, the encoding module of the audio recognition model can be used to encode the question text and the answer audio respectively to obtain the text vector corresponding to the question text and the audio vector corresponding to the answer audio, wherein the encoding module is used to encode the question text and the answer audio, convert the question text into a text vector, and convert the answer audio into an audio vector, that is, convert the audio into a feature code to facilitate further processing by the audio recognition model. For the question text, the question text can be converted into a feature vector, and for the answer audio, the answer audio can be converted into a spectrogram, and the feature vector corresponding to the spectrogram is used as the audio vector.

[0086] Based on this, after obtaining the answer audio submitted for the question text and inputting the question text and the answer audio into the audio recognition model, the encoding module of the audio recognition model is used to encode the question text and convert the question text into a text vector in the form of feature expression, and the encoding module of the audio recognition model is used to encode the answer audio to obtain the audio feature encoding corresponding to the answer audio, that is, the audio vector.

[0087] Furthermore, considering that the data types of the question text and the answer audio are different, they can be encoded by a text encoding unit and an audio encoding unit respectively, and the specific implementation is as follows:

[0088] The question text is encoded using a text encoding unit in the encoding module of the audio recognition model to obtain a text vector corresponding to the question text; the answer audio is encoded using an audio encoding unit in the encoding module of the audio recognition model to obtain an audio vector corresponding to the answer audio.

[0089] Specifically, the text encoding unit is used to encode the question text, that is, the object processed by the text encoding unit is text. Correspondingly, the audio encoding unit is used to encode the answer audio, and the object processed by the audio encoding unit is audio; the encoding of the question text and the answer audio can be achieved through an encoder.

[0090] Based on this, the text encoding unit in the encoding module of the audio recognition model is determined, and the question text is encoded using the text encoding unit to encode the question text into a text vector. The audio encoding unit in the encoding module of the audio recognition model is determined, and the answer audio is encoded using the audio encoding unit to obtain an audio vector corresponding to the answer audio. This facilitates other processing units of the audio recognition model to recognize the audio vector in combination with the text vector and recognize the audio vector as the answer text.

[0091] Continuing with the above example, after determining the question text "what color do you like?" and obtaining the corresponding answer audio, a text encoder is used to encode the question text "what color do you like?", and an audio encoder is used to encode the answer audio, to obtain a text vector corresponding to the question text and a high-dimensional spectrum corresponding to the answer audio, that is, an audio vector.

[0092] To summarize, the question text is encoded using the text encoding unit in the encoding module of the audio recognition model, and the answer audio is encoded using the audio encoding unit in the encoding module of the audio recognition model, thereby achieving targeted encoding of the question text and answer audio, and determining different encoding methods according to the data types corresponding to the question text and answer audio, respectively.

[0093] Furthermore, due to the different lengths of the question texts, consistency issues may arise when the question texts are processed by the audio recognition model. Masking the question texts can improve the consistency of the text length of the encoding unit input to the audio recognition model. The specific implementation is as follows:

[0094] The mask subunit in the text encoding unit is used to perform mask processing on the question text to obtain a mask vector corresponding to the question text; the first text encoding subunit in the text encoding unit is used to perform encoding processing on the mask vector to obtain a text vector corresponding to the question text.

[0095] Specifically, the mask subunit is used to perform mask processing on the text to simulate the prediction process of the question text; the mask vector is the feature vector obtained by masking the question text and then encoding it; the first text encoding subunit is used to encode the mask vector.

[0096] Based on this, a mask subunit is determined in the text encoding unit of the audio recognition model, and the mask subunit is used to perform mask processing on the question text to obtain a mask vector corresponding to the question text. A first text encoding subunit for encoding the question text is determined in the text encoding unit of the audio recognition model, and the mask vector is encoded using the first text encoding subunit to obtain a text vector corresponding to the question text.

[0097] Using the above example, the question text "what color do you like?" is input into the audio recognition model, and the mask subunit in the audio recognition model is used to mask the question text and convert the question text into a mask vector. The mask vector is then processed using the first text encoding subunit in the audio recognition model to obtain a text vector in the form of a high-dimensional vector corresponding to the question text.

[0098] To summarize, the mask vector is encoded using the first text encoding subunit in the text encoding unit to obtain a text vector corresponding to the question text, thereby incorporating the semantic information of the question text into the answer audio recognition process, thereby improving the accuracy of the audio recognition model in recognizing the answer audio.

[0099] Furthermore, considering that the obtained answer audio may contain noise and the length of the answer audio is also different, in order to ensure the processing efficiency of the audio recognition model and reduce the computing cost, the audio can be preprocessed and the sequence length corresponding to the answer audio can be reduced. The specific implementation is as follows:

[0100] The answer audio is processed by the audio sequence processing subunit in the audio encoding unit to obtain an answer sequence corresponding to the answer audio; the answer sequence is encoded by the audio encoding subunit in the audio encoding unit to obtain an audio vector corresponding to the answer audio.

[0101] Specifically, the audio sequence processing subunit is used to perform feature extraction and data enhancement on the answer audio, and to reduce the sequence length corresponding to the answer audio by convolution, so as to obtain an answer sequence that retains most of the important audio features.

[0102] Based on this, the audio sequence processing subunit in the audio encoding unit in the audio recognition model is determined, and the audio sequence processing subunit is used to perform feature extraction, data enhancement, and sequence length reduction on the answer audio to obtain the answer sequence corresponding to the answer audio. The audio encoding subunit in the audio encoding unit in the audio recognition model is determined, and the audio encoding subunit is used to encode the answer sequence to obtain the audio vector corresponding to the answer audio. This facilitates the subsequent recognition of the audio vector to obtain the answer text corresponding to the audio vector.

[0103] Following the above example, after obtaining the answer audio, feature extraction is performed on the answer audio. Feature extraction can be implemented using the fbank method. Then, data enhancement is performed on the audio features corresponding to the answer audio, so that the audio recognition model can obtain better recognition results even in a noisy environment. The data enhancement process includes but is not limited to frequency masking, time masking, and time-frequency masking. Then, convolution subsampling can be performed on the answer audio after data enhancement. Through convolution, the output of each specific interval is selected, thereby reducing the length of the sequence and retaining most of the important features.

[0104] To summarize, the audio encoding subunit in the audio encoding unit is used to encode the answer sequence to obtain the audio vector corresponding to the answer audio, thereby ensuring that the sequence length corresponding to the answer audio processed by the audio recognition model is the same, reducing the amount of calculation and reducing the computational cost.

[0105] Furthermore, considering that there are non-keywords and keywords in the question text, and keywords have a greater impact on the recognition accuracy of the answer audio, in order to improve the accuracy of converting the answer audio into the answer text, the question text can be segmented and then converted into a text vector. The specific implementation is as follows:

[0106] The word segmentation subunit in the text encoding unit is used to perform word segmentation processing on the question text to obtain text keywords corresponding to the question text; the second text encoding subunit in the text encoding unit is used to encode the text keywords to obtain a text vector corresponding to the question text.

[0107] Specifically, word segmentation refers to splitting the question text into multiple words. The word segmentation can adopt methods such as word segmentation based on string matching, maximum matching method, reverse maximum matching method, etc.; this embodiment does not impose any limitation on the algorithm or method adopted for word segmentation; text keywords refer to words that can affect the semantics of the question text, for example: "What is your favorite color to use?", then words such as your favorite color are text keywords; the second text encoding subunit is used to realize encoding processing of text keywords corresponding to the question text, and convert text keywords into feature vectors through feature extraction.

[0108] Based on this, the text encoding unit in the audio recognition model is determined, and the word segmentation subunit in the text encoding unit is used to perform word segmentation processing on the question text to obtain the text keywords corresponding to the question text. The second text encoding subunit in the text encoding unit uses the second text encoding subunit to encode the text keywords, and converts the text keywords into the form of feature vectors to obtain the text vector corresponding to the question text.

[0109] Using the above example, after determining that the question text is "What is your favorite color to use?", the question text is segmented to obtain text keywords such as your, favorite, color, etc. Then, the second text encoding subunit in the text encoding unit is used to encode the text keywords such as your, favorite, color, etc. to obtain the vector expressions corresponding to the text keywords such as your, favorite, color, etc.

[0110] In summary, after the question text is segmented, it is converted into a text vector to improve the accuracy of the subsequent conversion of the answer audio into the answer text.

[0111] Step S206: The audio vector is decoded using the text vector by the semantic analysis module of the audio recognition model to obtain an answer text corresponding to the answer audio output by the audio recognition model.

[0112] Specifically, after the encoding module of the audio recognition model is used to encode the question text and the answer audio respectively to obtain the text vector corresponding to the question text and the audio vector corresponding to the answer audio, the semantic analysis module of the audio recognition model can be used to decode the audio vector using the text vector to obtain the answer text corresponding to the answer audio output by the audio recognition model, wherein the semantic analysis module is used to decode the audio vector in combination with the semantic features of the text vector to convert the audio vector into a text expression; the decoding process is the opposite of the encoding process, and is used to convert the vector (feature code) into text, and the answer text is the text obtained after the answer audio is recognized by the audio recognition model, which represents the answer to the question text.

[0113] Based on this, after the encoding module of the audio recognition model is used to encode the question text and the answer audio respectively, and the text vector corresponding to the question text and the audio vector corresponding to the answer audio are obtained, the text vector and the audio vector are further processed through the semantic analysis module of the audio recognition model. The semantic feature information of the text vector is obtained, and the audio vector is decoded based on the semantic feature information of the text vector to obtain the answer text corresponding to the answer audio output by the audio recognition model. The answer text is the reply content to the question text.

[0114] Furthermore, after the answer audio is recognized to obtain the answer text, considering that the answer audio is submitted by the user through the client, it is also necessary to send feedback data to the client based on the answer text obtained by recognition, and the user performs question-answering feedback on the answer audio submitted by the user. The specific implementation is as follows:

[0115] Determine the question semantic information of the question text and the answer semantic information of the answer text; match the question semantic information and the answer semantic information, and detect the answer text according to the matching result; determine feedback data corresponding to the answer audio based on the detection result, and send the feedback data to the client that submitted the answer audio.

[0116] Specifically, question semantic information refers to the semantic information of the question text, which is used to represent the semantics of the question text. Correspondingly, the answer semantic information of the answer text is the semantic information of the answer text, which is used to represent the semantics of the answer text. For example, if the question text is “what color do you like”, the question semantic information may be “favorite color”. Feedback data refers to positive or negative feedback determined based on the detection result after the answer text is correctly or incorrectly detected. When the question text is “what color do you like” and the answer text is “I like blue”, the question text and the answer text form a corresponding relationship. The answer text is correct for the question text, and therefore the feedback data “Good” can be generated.

[0117] Based on this, the question semantic information of the question text is determined by semantic extraction of the question text, and the answer semantic information of the answer text is determined by semantic extraction of the answer text. The question semantic information and the answer semantic information are matched, and the answer text is detected according to the matching result; if it is determined based on the detection result that the question semantic information and the answer semantic information can match, positive feedback data corresponding to the answer audio is generated, and the positive feedback data is sent to the client that submitted the answer audio. If it is determined based on the detection result that the question semantic information and the answer semantic information cannot match, negative feedback data corresponding to the answer audio is generated, and the negative feedback data is sent to the client that submitted the answer audio.

[0118] Continuing with the above example, when the question text is "what color do you like", and the answer text "I like Apple" is obtained by recognizing the answer audio, the semantics of the question text is the favorite color, while the semantics of the answer text is apple. The semantics of the question text and the answer text cannot match, and negative feedback data such as "sorry" is generated.

[0119] To sum up, the feedback data is sent to the client that submitted the answer audio, so that the user who submitted the answer audio can get feedback on the answer audio, thereby improving the user experience and promoting user learning and thinking.

[0120] Furthermore, considering that the audio recognition model also needs to be continuously optimized with the continuous iteration of dialogue interaction, a model tuning sample can be constructed based on the answer audio processed during the application of the audio recognition model and the answer text and question text corresponding to the answer audio, and the audio recognition model can be tuned and trained. The specific implementation is as follows:

[0121] The question text and the answer audio are used as model tuning samples, and the answer text is used as a model tuning label; a model tuning sample pair is composed of the model tuning sample and the model tuning label, and is stored in a model tuning sample set, wherein the model tuning sample set is used to perform tuning training on the audio recognition model.

[0122] Specifically, the model tuning sample refers to the sample used for tuning and training the audio recognition model, that is, retraining; the model tuning label is the sample label corresponding to the answer audio as the sample; the model tuning samples are stored in the model tuning sample set, and in the process of the audio recognition model recognizing the answer audio, the model tuning samples are constructed based on the recognized answer audio, the question text corresponding to the answer audio, and the recognition result (answer text) and stored in the model tuning sample set. When the number of model tuning samples contained in the model tuning sample set reaches a quantity threshold, the audio recognition model is tuned and trained based on the model tuning sample set.

[0123] Based on this, the question text and answer audio are used as model tuning samples, and the answer text is used as a model tuning label; the model tuning sample and the model tuning label form a model tuning sample pair, which is stored in the model tuning sample set. The model tuning samples stored in the model tuning sample set are used to tune and train the audio recognition model. In the process of processing the question text and answer audio based on the audio recognition model, the model tuning samples are constructed based on the answer audio and question text processed by the audio recognition model and stored in the model tuning sample set to update the model tuning sample set.

[0124] Continuing with the above example, in the process of processing the answer audio submitted by the user, the question text "Did you visit any indoor places like museums or shopping malls?" and the answer audio of the question text are determined, and after the answer audio is recognized based on the audio recognition model, a model tuning sample is constructed based on the recognized answer text, the question text "Did you visit any indoor places like museums or shopping malls?" and the answer audio of the question text, and stored in the model tuning sample set for tuning and training the audio recognition model.

[0125] In summary, the audio recognition model is tuned and trained based on the model tuning sample set, so as to update the audio recognition model and improve the recognition accuracy of the audio recognition model.

[0126] An embodiment of the present specification obtains the answer audio submitted for the question text, and inputs the question text and the answer audio into the audio recognition model; uses the encoding module of the audio recognition model to encode the question text and the answer audio respectively, and obtains the text vector corresponding to the question text, and the audio vector corresponding to the answer audio; through the semantic analysis module of the audio recognition model, uses the text vector to decode the audio vector, and obtains the answer text corresponding to the answer audio output by the audio recognition model. Without the need to manually maintain any vocabulary, the answer audio can be recognized in combination with the question text corresponding to the answer audio, thereby improving the accuracy of converting the answer audio into the answer text.

[0127] The following combination Figure 3 , taking the application of the data processing method provided in this specification in speech recognition as an example, the data processing method is further described. Figure 3 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0128] Step S302: Obtain the initial answer audio submitted for the question text, pre-process the initial answer audio, and determine the answer audio based on the processing result.

[0129] In the oral dialogue interaction scenario, the answer audio submitted by the user to the question text needs to be recognized as text. The initial answer audio submitted by the user to the question "Which fruit do you like?" is obtained, and the initial answer audio is pre-processed such as noise reduction to obtain the answer audio.

[0130] Step S304: Input the question text and answer audio into the audio recognition model.

[0131] Step S306: Encode the question text using the text encoding unit in the encoding module of the audio recognition model to obtain a text vector corresponding to the question text.

[0132] like Figure 4 As shown in the figure, the question text is input into the audio recognition model, and the question text is processed in sequence through output encoding, position encoding, masked multi-head attention mechanism and normalization to obtain the text vector corresponding to the question text.

[0133] Step S308: Encode the answer audio using the audio encoding unit in the encoding module of the audio recognition model to obtain an audio vector corresponding to the answer audio.

[0134] The answer audio is input into the audio recognition model, and is processed in sequence through spectrum enhancement, convolution subsampling, linear layer, and encoder to obtain the audio vector corresponding to the answer audio.

[0135] Step S310: The audio vector is decoded using the text vector through the semantic analysis module of the audio recognition model to obtain the answer text corresponding to the answer audio output by the audio recognition model.

[0136] After obtaining the text vector corresponding to the question text and the audio vector corresponding to the answer audio, the audio vector is decoded in combination with the text vector through the masked multi-head attention mechanism, normalization layer, forward module, normalization layer, linear layer and normalized exponential function in the semantic analysis module of the audio recognition model to obtain the answer text corresponding to the answer audio output by the audio recognition model.

[0137] In the decoding process, we use the autoregressive decoding process to input the topic of the current conversation into the decoder, and then pass it through a softmax to take the Chinese character with the highest score as the output, and at the same time, take the input at this time point as the output at the next moment. Since there is a correlation between conversations, the recognition result of the previous sentence can be added to the decoder to improve the recognition accuracy of the next sentence.

[0138] Step S312: Determine the question semantic information of the question text and the answer semantic information of the answer text.

[0139] Step S314: Match the question semantic information and the answer semantic information, and detect the answer text according to the matching result.

[0140] After the answer audio is recognized and the answer text corresponding to the answer audio is obtained, semantic recognition and detection can be used to determine whether the answer audio of the user is correct. If the question text is "Which fruit do you like?" and the answer text obtained by recognition is "Apple", the answer submitted by the user is determined to be correct; if the answer text is "blue", the answer submitted by the user is determined to be incorrect.

[0141] Step S316: Determine feedback data corresponding to the answer audio based on the detection result, and send the feedback data to the client that submitted the answer audio.

[0142] When the answer submitted by the user is correct, feedback information such as "Good" can be generated to indicate affirmation of the user's answer; on the contrary, when the answer submitted by the user is incorrect, feedback information such as "sorry" can be generated to indicate negation of the user's answer, thereby improving the user experience.

[0143] An embodiment of the present specification obtains the answer audio submitted for the question text, and inputs the question text and the answer audio into the audio recognition model; uses the encoding module of the audio recognition model to encode the question text and the answer audio respectively, and obtains the text vector corresponding to the question text, and the audio vector corresponding to the answer audio; through the semantic analysis module of the audio recognition model, uses the text vector to decode the audio vector, and obtains the answer text corresponding to the answer audio output by the audio recognition model. Without the need to manually maintain any vocabulary, the answer audio can be recognized in combination with the question text corresponding to the answer audio, thereby improving the accuracy of converting the answer audio into the answer text.

[0144] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0145] An acquisition module 502 is configured to acquire an answer audio submitted for a question text, and input the question text and the answer audio into an audio recognition model;

[0146] The encoding module 504 is configured to use the encoding module of the audio recognition model to encode the question text and the answer audio respectively to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio;

[0147] The decoding module 506 is configured to decode the audio vector using the text vector through the semantic analysis module of the audio recognition model to obtain the answer text corresponding to the answer audio output by the audio recognition model.

[0148] In an optional embodiment, the encoding module 504 is further configured to:

[0149] Using a text encoding unit in an encoding module of the audio recognition model to encode the question text to obtain a text vector corresponding to the question text;

[0150] The answer audio is encoded using the audio encoding unit in the encoding module of the audio recognition model to obtain an audio vector corresponding to the answer audio.

[0151] In an optional embodiment, the encoding module 504 is further configured to:

[0152] Using the mask subunit in the text encoding unit to perform mask processing on the question text to obtain a mask vector corresponding to the question text;

[0153] The mask vector is encoded by using the first text encoding subunit in the text encoding unit to obtain a text vector corresponding to the question text.

[0154] In an optional embodiment, the encoding module 504 is further configured to:

[0155] Processing the answer audio by using the audio sequence processing subunit in the audio encoding unit to obtain an answer sequence corresponding to the answer audio;

[0156] The answer sequence is encoded using the audio encoding subunit in the audio encoding unit to obtain an audio vector corresponding to the answer audio.

[0157] In an optional embodiment, the decoding module 506 is further configured to:

[0158] Determining question semantic information of the question text and answer semantic information of the answer text;

[0159] Matching the question semantic information with the answer semantic information, and detecting the answer text according to the matching result;

[0160] Feedback data corresponding to the answer audio is determined based on the detection result, and the feedback data is sent to the client that submitted the answer audio.

[0161] In an optional embodiment, the acquisition module 502 is further configured to:

[0162] Determine a target sample pair in the sample set;

[0163] Inputting the question text sample and the answer audio sample in the target sample pair into an initial audio recognition model;

[0164] Processing the question text sample and the answer audio sample using the initial audio recognition model to obtain answer recognition text corresponding to the answer audio sample output by the audio recognition model;

[0165] A loss value is calculated based on the recognition reference text and the answer recognition text in the target sample pair, and the parameters of the initial audio recognition model are adjusted based on the loss value until an audio recognition model that meets the training stop condition is obtained.

[0166] In an optional embodiment, the acquisition module 502 is further configured to:

[0167] Obtaining initial answer audio submitted to the question text;

[0168] Performing shearing processing on the initial answer audio, and determining the answer audio to be processed according to the shearing processing result;

[0169] The answer audio to be processed is subjected to noise reduction processing, and the answer audio is determined according to the noise reduction processing result.

[0170] In an optional embodiment, the decoding module 506 is further configured to:

[0171] Using the question text and the answer audio as model tuning samples, and using the answer text as a model tuning label;

[0172] A model tuning sample pair is composed of the model tuning sample and the model tuning label and stored in a model tuning sample set, wherein the model tuning sample set is used to perform tuning training on the audio recognition model.

[0173] In an optional embodiment, the encoding module 504 is further configured to:

[0174] Using the word segmentation subunit in the text encoding unit to perform word segmentation processing on the question text to obtain text keywords corresponding to the question text;

[0175] The text keywords are encoded using the second text encoding subunit in the text encoding unit to obtain a text vector corresponding to the question text.

[0176] In summary, an embodiment of the present specification obtains the answer audio submitted for the question text, and inputs the question text and the answer audio into the audio recognition model; uses the encoding module of the audio recognition model to encode the question text and the answer audio respectively, and obtains the text vector corresponding to the question text, and the audio vector corresponding to the answer audio; through the semantic analysis module of the audio recognition model, uses the text vector to decode the audio vector, and obtains the answer text corresponding to the answer audio output by the audio recognition model. Without the need to manually maintain any vocabulary, the answer audio can be recognized in combination with the question text corresponding to the answer audio, thereby improving the accuracy of converting the answer audio into the answer text.

[0177] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0178] Figure 6The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0179] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0180] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0181] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0182] The processor 620 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.

[0183] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0184] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0185] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0186] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0187] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.

[0188] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0189] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0190] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0191] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0192] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, characterized in that: include: Obtaining an answer audio submitted for a question text, and inputting the question text and the answer audio into an audio recognition model; Using the encoding module of the audio recognition model to encode the question text and the answer audio respectively, to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio; The audio vector is decoded using the text vector through the semantic analysis module of the audio recognition model to obtain an answer text corresponding to the answer audio output by the audio recognition model.

2. The method according to claim 1, characterized in that: The encoding module of the audio recognition model is used to encode the question text and the answer audio respectively to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio, including: Using a text encoding unit in an encoding module of the audio recognition model to encode the question text to obtain a text vector corresponding to the question text; The answer audio is encoded using the audio encoding unit in the encoding module of the audio recognition model to obtain an audio vector corresponding to the answer audio.

3. The method according to claim 2, characterized in that Determining the text vector corresponding to the question text includes: Using the mask subunit in the text encoding unit to perform mask processing on the question text to obtain a mask vector corresponding to the question text; The mask vector is encoded by using the first text encoding subunit in the text encoding unit to obtain a text vector corresponding to the question text.

4. The method according to claim 2, characterized in that: Determining the audio vector corresponding to the answer audio includes: Processing the answer audio by using the audio sequence processing subunit in the audio encoding unit to obtain an answer sequence corresponding to the answer audio; The answer sequence is encoded using the audio encoding subunit in the audio encoding unit to obtain an audio vector corresponding to the answer audio.

5. The method according to claim 1, characterized in that After the audio vector is decoded by the semantic analysis module of the audio recognition model using the text vector to obtain the answer text corresponding to the answer audio output by the audio recognition model, the method further includes: Determining question semantic information of the question text and answer semantic information of the answer text; Matching the question semantic information with the answer semantic information, and detecting the answer text according to the matching result; Feedback data corresponding to the answer audio is determined based on the detection result, and the feedback data is sent to the client that submitted the answer audio.

6. The method according to claim 1, characterized in that The training of the audio recognition model includes: Determine a target sample pair in the sample set; Inputting the question text sample and the answer audio sample in the target sample pair into an initial audio recognition model; Processing the question text sample and the answer audio sample using the initial audio recognition model to obtain answer recognition text corresponding to the answer audio sample output by the audio recognition model; A loss value is calculated based on the recognition reference text and the answer recognition text in the target sample pair, and the parameters of the initial audio recognition model are adjusted based on the loss value until an audio recognition model that meets the training stop condition is obtained.

7. The method according to claim 1, characterized in that The step of obtaining the answer audio submitted for the question text includes: Obtaining initial answer audio submitted to the question text; Performing a shearing process on the initial answer audio, and determining the answer audio to be processed according to the shearing process result; The answer audio to be processed is subjected to noise reduction processing, and the answer audio is determined according to the noise reduction processing result.

8. The method according to claim 1, characterized in that After the audio vector is decoded by the semantic analysis module of the audio recognition model using the text vector to obtain the answer text corresponding to the answer audio output by the audio recognition model, the method further includes: Using the question text and the answer audio as model tuning samples, and using the answer text as a model tuning label; A model tuning sample pair is composed of the model tuning sample and the model tuning label and stored in a model tuning sample set, wherein the model tuning sample set is used to perform tuning training on the audio recognition model.

9. The method according to claim 2, characterized in that: Determining the text vector corresponding to the question text includes: Using the word segmentation subunit in the text encoding unit to perform word segmentation processing on the question text to obtain text keywords corresponding to the question text; The text keywords are encoded using the second text encoding subunit in the text encoding unit to obtain a text vector corresponding to the question text.

10. A data processing device, characterized in that: include: An acquisition module is configured to acquire an answer audio submitted for a question text, and input the question text and the answer audio into an audio recognition model; An encoding module is configured to use the encoding module of the audio recognition model to encode the question text and the answer audio respectively to obtain a text vector corresponding to the question text and an audio vector corresponding to the answer audio; The decoding module is configured to decode the audio vector using the text vector through the semantic analysis module of the audio recognition model to obtain the answer text corresponding to the answer audio output by the audio recognition model.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the data processing method described in any one of claims 1 to 9.

Images