Voice question and answer method and device, electronic equipment and storage medium

Abstract

The application provides a voice question and answer method and device, electronic equipment and storage medium, wherein the method comprises the following steps: determining a question voice; performing text error correction based on the transcription text of the question voice to obtain a corrected text; performing feature extraction on the corrected text and the question voice respectively to obtain text features and voice features; fusing the text features and the voice features based on the correlation between the text features and the voice features to obtain question fusion features; and matching the question fusion features with candidate question fusion features of each candidate question, and determining the answer corresponding to the candidate question that matches successfully as the answer of the question voice. Through explicit error correction of the transcription text and fault tolerance processing at the feature level in the feature fusion process, the negative influence of transcription errors in the transcription text on the downstream question and answer task is minimized, the defect that the voice question and answer effect in the traditional scheme is poor in the vertical field is overcome, and the accuracy of the voice question and answer is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a voice question-and-answer method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, voice question answering has been gradually applied to all aspects of social production and life. However, the voice question answering process usually requires transcribing the input speech into text for subsequent processing. Unfortunately, many unavoidable transcription errors often occur during speech transcription, and these errors accumulate in subsequent processes, affecting the final voice question answering results. To ensure the effectiveness of voice question answering, it is necessary to correct the transcribed text during the voice question answering process.

[0003] Currently, there are two main types of text correction schemes. The first is text correction methods based on statistical language models. These methods perform well on low-level transcription errors, but due to inherent limitations of the models, they cannot capture the semantic relationships in sentences well. Therefore, they are difficult to correct transcription errors with semantic relationships. The second is text correction methods based on deep learning models. In vertical domain speech question answering scenarios, due to the lack of relevant training data, the models are prone to overfitting during training, resulting in poor performance in vertical domain applications. Summary of the Invention

[0004] This invention provides a voice question answering method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies, such as the inability to correct long-distance dependency errors with semantic associations and poor voice question answering performance in vertical domains. It achieves fault-tolerant processing at the feature level and improves the accuracy of voice question answering.

[0005] This invention provides a voice question-and-answer method, comprising:

[0006] Identify the problematic audio;

[0007] Text correction is performed on the transcribed text of the problematic speech to obtain the corrected text;

[0008] Feature extraction is performed on the error correction text and the question speech respectively to obtain text features and speech features. Based on the correlation between the text features and the speech features, the text features and the speech features are fused to obtain the question fusion features.

[0009] The question fusion feature is matched with the candidate question fusion feature of each candidate question, and the answer corresponding to the successfully matched candidate question is determined as the answer to the question speech.

[0010] According to a voice question-answering method provided by the present invention, the method involves extracting features from the error-correction text and the question speech respectively to obtain text features and speech features, and fusing the text features and speech features based on the correlation between the text features and the speech features to obtain question fusion features, including:

[0011] Based on the feature extraction model, features are extracted from the error correction text and the question speech respectively to obtain text features and speech features. Based on the correlation between the text features and the speech features, the text features and the speech features are fused to obtain the question fusion features.

[0012] The feature extraction model is trained based on sample question pairs and whether the semantics of the two sample question voices in the sample question pair are the same.

[0013] According to a voice question-answering method provided by the present invention, the feature extraction model is trained based on the following steps:

[0014] Based on the initial feature extraction model, the sample question fusion features of the sample question speech are determined;

[0015] From the sample question speech, two sample question speech with the same semantics are selected as positive sample question pairs, and from the sample question speech, two sample question speech with different semantics are selected as negative sample question pairs.

[0016] Based on the feature similarity between the sample question fusion features of the two sample question speech in the positive sample question pair and the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair, the initial feature extraction model is iterated to obtain the feature extraction model.

[0017] According to a speech question answering method provided by the present invention, the feature extraction model includes a speech feature extraction network, a text feature extraction network, and a feature fusion network;

[0018] The speech feature extraction network is used to extract features from the problematic speech to obtain the speech features of the problematic speech;

[0019] The text feature extraction network is used to extract features from the error-corrected text to obtain the text features of the error-corrected text;

[0020] The feature fusion network is used to fuse the text features and the speech features based on the correlation between them to obtain question fusion features.

[0021] According to a speech question answering method provided by the present invention, the feature fusion network includes a feature weighting layer and a feature fusion layer, wherein the feature weighting layer and the feature fusion layer are residually connected;

[0022] The feature weighting layer is used to determine the weight of the speech feature based on the correlation between the text feature and the speech feature, and to weight the speech feature based on the weight to obtain weighted speech features;

[0023] The feature fusion layer is used to determine a first speech feature based on the weighted speech features and the speech features, and to fuse the first speech feature and the text features to obtain the question fusion feature.

[0024] According to a voice question-answering method provided by the present invention, the step of performing text correction on the transcribed text based on the voice of the question to obtain corrected text includes:

[0025] Based on the text correction model, the transcribed text of the problematic speech is corrected to obtain the corrected text.

[0026] The text correction model is trained using sample text and sample correction texts of the sample text, based on a statistical language model.

[0027] According to a speech question answering method provided by the present invention, the initial feature extraction model includes an initial speech feature extraction network, an initial text feature extraction network, and an initial feature fusion network;

[0028] The initial speech feature extraction network is used to extract features from the sample question speech to obtain sample speech features;

[0029] The initial text feature extraction network is used to extract features from the sample error correction text of the sample problematic speech to obtain sample text features. The sample error correction text is obtained by correcting text errors based on the sample transcribed text of the sample problematic speech.

[0030] The initial feature fusion network is used to fuse the sample text features and the sample speech features based on the correlation between them, to obtain sample question fusion features;

[0031] The initial speech feature extraction network is built on the basis of the speech pre-trained model, and the initial text feature extraction network is built on the basis of the language model.

[0032] The present invention also provides a voice question-and-answer device, comprising:

[0033] The speech determination unit is used to determine the problematic speech.

[0034] The text correction unit is used to perform text correction based on the transcribed text of the problematic speech to obtain the corrected text;

[0035] The feature fusion unit is used to extract features from the error-corrected text and the problematic speech respectively to obtain text features and speech features, and to fuse the text features and speech features based on the correlation between the text features and the speech features to obtain the problem fusion features;

[0036] The answer determination unit is used to match the question fusion feature with the candidate question fusion feature of each candidate question, and determine the answer corresponding to the successfully matched candidate question as the answer to the question speech.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the voice question-and-answer method as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the voice question-and-answer method as described above.

[0039] The speech question answering method, apparatus, electronic device, and storage medium provided by this invention perform text correction on the transcribed text of the question speech to obtain the corrected text. Features are then extracted from both the corrected text and the question speech to obtain text features and speech features, respectively. These features are fused using the correlation between them to obtain question fusion features. The question fusion features are matched with the candidate question fusion features of each candidate question. The answer corresponding to the successfully matched candidate question is determined as the answer to the question speech. Through explicit error correction of the transcribed text and fault tolerance processing at the feature level during feature fusion, the negative impact of transcription errors in the transcribed text on downstream question answering tasks can be minimized. This overcomes the shortcomings of traditional solutions, such as the inability to correct long-distance dependencies with semantic relationships and poor speech question answering performance in vertical domains. By explicitly correcting and tolerating transcription errors, the accuracy of speech question answering can be greatly improved. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1This is a flowchart illustrating the voice question-and-answer method provided by the present invention;

[0042] Figure 2 This is a flowchart illustrating the model training process provided by the present invention;

[0043] Figure 3 This is a framework example diagram of the feature fusion process provided by the present invention;

[0044] Figure 4 This is a general framework diagram of the voice question-answering method provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of the voice question-and-answer device provided by the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] In recent years, voice question answering has been gradually applied to all aspects of social production and daily life. The voice question answering process first requires transcribing the collected input speech into text for subsequent processing. However, many unavoidable transcription errors often occur during this process, and these errors accumulate in the subsequent text feature extraction process, thus significantly negatively impacting the final voice question answering results. Therefore, to ensure the final voice question answering effect, it is necessary to correct the transcribed text during the voice question answering process.

[0049] However, when general speech-to-text models are transferred to small-sample speech-to-text in vertical domains, due to a lack of extensive training data or the use of external speech-to-text technologies, they often make errors in transcribing specialized vocabulary, contextual slang, and fixed collocations when transcribing input speech. Therefore, in the entire speech question-answering process, it is necessary to study the fault tolerance of downstream tasks for small-sample speech-to-text transcription errors in vertical domains, that is, to study how to reduce the interference of transcription errors on downstream tasks so that downstream tasks can return the correct answer without being affected by transcription errors.

[0050] Currently, the main text correction solutions for voice question answering processes are as follows:

[0051] One approach is a rule-based text correction method. This method constructs a dictionary of easily confused words and a dictionary of easily confused pinyin words. If a word is in either of these two dictionaries, it indicates a possible error. Therefore, candidate words can be recalled based on the replacement words in the dictionary, and the perplexity of the sentence after being replaced with the candidate words can be calculated. Replacement words with low perplexity are then selected based on the perplexity ranking.

[0052] The second is a text correction method based on statistical language models. This method mainly uses the probabilistic dependency relationship between consecutive words in a sentence as the basis for error localization, and uses homophones and similar words for recall scoring to determine the recall word with the lowest perplexity as the final replacement word. This method performs well in fixed phrase error correction.

[0053] Thirdly, there is the text correction method based on deep learning models. Based on a large amount of relevant training data, the model can leverage the powerful semantic feature modeling capabilities of the pre-trained model, especially its ability to predict text keywords or key speech segments, which can effectively solve the problem of long-distance dependency correction with semantic association.

[0054] However, while the aforementioned text correction method based on statistical language models performs well on low-level transcription errors, it suffers from inherent limitations. Specifically, it cannot effectively capture semantic relationships within sentences. Furthermore, extending the sliding window of the statistical language model to 4, 5, or even higher results in numerous null values, leading to smoothed scores that often exceed the set threshold, thus failing to detect transcription errors. Therefore, this method struggles to correct semantically related transcription errors.

[0055] While text correction methods based on pre-trained deep learning models have achieved further performance improvements, in vertical domain voice question answering scenarios, the lack of relevant training data, i.e., the lack of pairs of original sentences and correct sentences in the vertical domain, easily leads to the model getting overfitted during training, resulting in poor performance of the model in practical applications in the vertical domain, i.e., poor application effect in the vertical domain.

[0056] In summary, current text correction schemes mainly perform explicit error correction on the transcribed text at the text level, which has achieved good results to a certain extent. However, there is little research on error tolerance mechanisms for texts that are difficult to correct.

[0057] To address this issue, this invention provides a voice question answering method that, after explicitly correcting the transcribed text, proposes a fault-tolerance mechanism to minimize the negative impact of transcription errors on downstream question answering tasks. It overcomes the shortcomings of statistical language models in handling long-distance dependencies with semantic connections during voice question answering, as well as the overfitting of pre-trained deep learning models in vertical domains due to a lack of relevant training data, resulting in poor application performance. This invention achieves fault-tolerance processing for transcription errors, contributing to improved accuracy in voice question answering. Figure 1 This is a flowchart illustrating the voice question-and-answer method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0058] Step 110: Identify the problematic audio.

[0059] Specifically, before conducting voice question and answer, it is necessary to first determine the voice to be answered, which is the question voice. The question voice can be directly input by the user, or it can be a segment of voice extracted from the voice data stream acquired in real time by the voice acquisition module, or it can be a segment of voice extracted from historical voice data. This embodiment of the invention does not make specific limitations on this.

[0060] The question audio contains a question statement that needs to be answered. This question statement can be raised by the user based on actual needs, inspired by the context, or after seeing a report or journal article in a specific vertical field. This embodiment of the invention does not impose specific limitations on this. For example, it could be "How to control rice planthoppers?", "What are the symptoms of wheat powdery mildew?", or "How to treat swine dysentery?"

[0061] It should be noted that the question audio here can be a single segment or multiple segments. When the question audio is in multiple segments, the answer to each segment needs to be determined to achieve voice question answering. However, in this process, it is also necessary to perform text correction on the transcribed text of each segment of question audio and to perform error tolerance processing at the feature level to minimize the impact of transcription errors on the subsequent question answering process, thereby improving the accuracy of voice question answering.

[0062] Step 120: Perform text correction based on the transcribed text of the problematic speech to obtain the corrected text;

[0063] Specifically, after obtaining the problematic speech in step 110, step 120 can be executed to perform text correction on the transcribed text of the problematic speech, thereby obtaining the corrected text. The specific process includes the following steps:

[0064] First, the speech of the question can be transcribed into text, thus obtaining the transcribed text of the question. The speech transcription process can be achieved through conventional speech transcription methods, such as speech transcription software, speech transcription models, etc.

[0065] Subsequently, text correction can be performed on the transcribed text of the problematic speech to obtain the corrected text. This text correction is actually explicit correction, which means correcting transcription errors at the level of homophones, similar words, fixed collocations, etc. in the transcribed text. Specifically, it can be done by locating errors based on the probabilistic dependencies between consecutive words in the transcribed text, using homophones and similar words for recall scoring, calculating the perplexity, and using the recall word with the lowest perplexity as the final replacement word. By explicitly correcting errors, the surface-level errors in the transcribed text can be corrected, and finally, the corrected text after explicit correction can be obtained.

[0066] Here, the explicit error correction process for transcribed text can be achieved through a text correction model. Specifically, the transcribed text of the problematic speech can be input into the text correction model, and then the text correction model can perform text correction on the input transcribed text to correct transcription errors such as homophones, similar words, and fixed collocations, and finally obtain the corrected text output by the text correction model.

[0067] Before inputting the transcribed text of the problematic speech into the text correction model, the text correction model can be pre-trained. The training process of the text correction model includes the following steps: First, a large number of sample texts are collected, and sample correction texts for the sample texts are determined. Then, based on the sample texts and their sample correction texts, the initial text correction model can be trained to obtain the final text correction model. This initial text correction model can be a statistical language model, such as an n-gram.

[0068] In this embodiment of the invention, by using probabilistic dependency-based error word localization and pinyin-based error word replacement in explicit error correction, surface-level error correction in transcribed text can be achieved, laying a data foundation for subsequent error-tolerant processing and improving the accuracy of voice question answering.

[0069] Step 130: Extract features from the error correction text and the question speech respectively to obtain text features and speech features. Based on the correlation between the text features and speech features, fuse the text features and speech features to obtain the question fusion features.

[0070] Considering the performance limitations of statistical language models and pre-trained deep learning models in explicit error correction in traditional solutions, this embodiment of the invention performs implicit error correction (error-tolerant processing) at the feature level. The reason is that implicit error correction at the feature level is easier for the model to learn. Therefore, in this embodiment of the invention, based on the traditional explicit error correction based on statistical language models, the fusion features of the question speech and the transcribed text are used to perform implicit error correction (error-tolerant processing) at the feature level, so that the transcription error has the least impact on the downstream question answering task, providing key assistance for improving the accuracy of speech question answering.

[0071] Specifically, in step 120, after obtaining the corrected text from the transcribed text through text correction, step 130 can be executed to extract features from the corrected text and the problematic speech, respectively, to obtain text features and speech features. Based on the correlation between the text features and speech features, the text features and speech features are fused to obtain the problematic fused features. This process may specifically include the following steps:

[0072] First, feature extraction can be performed on the error-correction text and the question speech separately to extract the relevant features of the question sentences contained in both, thereby obtaining the text features of the error-correction text and the speech features of the question speech. Since there are significant differences between the speech waveforms and speech spectrograms of different question sentences, and these differences can reflect the semantic differences between the semantic information of the question sentences, extracting the speech features of the question speech and using the semantic information contained in these speech features can effectively compensate for the semantic information lost in the transcribed text due to transcription errors.

[0073] Here, the feature extraction process for the error correction text and the problematic speech can be implemented through a feature extraction network. Specifically, the error correction text and the problematic speech can be input into a text feature extraction network and a speech feature extraction network, respectively. The text feature extraction network extracts features from the input error correction text, and the speech feature extraction network extracts features from the problematic speech. Then, the text feature extraction network and the speech feature extraction network output the text features of the error correction text and the speech features of the problematic speech, respectively.

[0074] Before extracting features from the error-corrected text and the problematic speech, a pre-trained feature extraction network can be applied. It is worth noting that the initial text feature extraction network during the training process is built on the basis of the language model, while the speech feature extraction network is built on the basis of a general speech pre-trained model.

[0075] Subsequently, considering the differences in information level, information angle, and information focus of the semantic information of the problem statement represented by text features and speech features, and the fact that the error correction text based on the transcribed text loses some information due to transcription errors during the speech transcription process, while the semantic information contained in the speech features can just supplement this part of the information, therefore, in this embodiment of the invention, the two can be fused so that the text features and speech features can complement each other. The fusion process can be based on the correlation between the text features and speech features, that is, the text features of the error correction text and the speech features of the problem speech can be fused according to the correlation between the text features and speech features to obtain the problem fusion features;

[0076] It should be noted that fusing based on the correlation between the two is actually equivalent to fusing them based on an attention mechanism. The essence of the attention mechanism is feature weighting. The introduction of the attention mechanism can focus on the information that is more critical to the current task in high-dimensional multimodal features (text modality and speech modality), while reducing the attention to other information, and even filtering out irrelevant information. Specifically, in the embodiments of this invention, fusing the two through the attention mechanism can make the fusion process increase the weight of relevant features for fault tolerance processing and weaken the weight of irrelevant features, thereby effectively solving the problem of information overload, improving the accuracy of task processing, and finally obtaining the problem fusion feature.

[0077] Here, the attention mechanism used is similar to a non-local block structure. The resulting question fusion features not only contain semantic information about the question statement in the question speech, but also encompass the apparent information of transcription error-related features in the error correction text. Furthermore, based on the correlation between text features and speech features, feature fusion can add subtle features (semantics, tone, etc.) related to the question statement to the resulting question fusion features. These subtle features play a crucial role in the error correction and error tolerance of homophones and similar words. That is, it can identify words with the same pinyin and similar words, minimizing their impact on the subsequent question-and-answer process, thus contributing to the improvement of speech question-and-answer accuracy.

[0078] Step 140: Match the question fusion features with the candidate question fusion features of each candidate question, and determine the answer corresponding to the successfully matched candidate question as the answer to the question speech.

[0079] Specifically, after obtaining the question fusion features through the above steps, step 140 involves matching the question fusion features with the candidate question fusion features of each candidate question, and determining the answer to the question speech based on the matching results, thus completing the speech question-answering process. The specific process may include:

[0080] First, candidate questions may be needed. These candidate questions are multiple questions collected in advance to match the question speech. They can be in speech form and, like the question speech, their corresponding candidate question fusion features can be obtained through steps 110 to 130 above.

[0081] It should be noted that, in order to narrow down the matching range, save computing resources, and speed up the voice question-and-answer process when screening candidate questions, the domain corresponding to the question statement contained in the question speech can be used as a screening condition to select multiple questions in the same or similar domain as the question statement as candidate questions.

[0082] Subsequently, the problem fusion features and the candidate problem fusion features of each candidate problem can be matched to obtain the matching results. Specifically, the feature similarity between the problem fusion features and the candidate problem fusion features can be calculated. The matching relationship between the two can be measured by the feature similarity. That is, when the feature similarity is greater than or equal to the preset similarity threshold, the two are determined to be matched, and the matching result is a successful match. Correspondingly, when the feature similarity is less than the preset similarity threshold, the two are determined to be mismatched, and the matching result is a failed match.

[0083] Here, the feature similarity between the problem fusion feature and the fusion features of each candidate problem can be calculated using cosine similarity, Euclidean distance, Minkowski distance, etc.

[0084] After that, the answer to the question speech can be determined based on the matching results. In other words, the candidate question can be determined if the matching is successful. Successful matching indicates that there is a high degree of similarity between the question fusion features and the corresponding candidate question fusion features. In other words, the semantic information of the question statement represented by the question speech is very close to that of the question statement contained in the candidate question. Therefore, the answer corresponding to the candidate question in this case can be directly used as the answer to the question speech.

[0085] The speech question answering method provided by this invention performs text correction on the transcribed text of the question speech to obtain the corrected text, and extracts features from both the corrected text and the question speech to obtain text features and speech features respectively. These features are then fused using the correlation between them to obtain the question fusion features. The question fusion features are matched with the candidate question fusion features of each candidate question, and the answer corresponding to the successfully matched candidate question is determined as the answer to the question speech. Through explicit error correction of the transcribed text and error tolerance processing at the feature level during feature fusion, the negative impact of transcription errors in the transcribed text on the downstream question answering task can be minimized. This overcomes the shortcomings of traditional solutions, such as the inability to correct long-distance dependency errors with semantic relationships and poor speech question answering performance in vertical domains. By explicitly correcting and tolerating transcription errors, the accuracy of speech question answering can be greatly improved.

[0086] Based on the above embodiments, step 130 includes:

[0087] Based on the feature extraction model, features are extracted from the error correction text and the question speech respectively to obtain text features and speech features. Based on the correlation between the text features and speech features, the text features and speech features are fused to obtain the question fusion features.

[0088] The feature extraction model is trained based on sample question pairs and whether the semantics of the two sample question voices in the sample question pair are the same.

[0089] Specifically, in step 130, features are extracted from the error correction text and the problematic speech to obtain text features and speech features, respectively. Based on the correlation between the text features and speech features, the text features and speech features are fused to obtain the problem fused features. This process can be achieved with the help of a feature extraction model, and the specific process may include the following steps:

[0090] First, feature extraction models can be used to extract features from the correction text and the question speech, respectively, to obtain the text features of the correction text and the speech features of the question speech. This process can be implemented through the text feature extraction network and the speech feature extraction network in the feature extraction model. Specifically, the correction text and the question speech are input into the text feature extraction network and the speech feature extraction network in the feature extraction model, respectively. The text feature extraction network extracts features from the input correction text, and the speech feature extraction network extracts features from the input question speech. Then, the text feature extraction network and the speech feature extraction network output the text features of the correction text and the speech features of the question speech, respectively.

[0091] Subsequently, considering the differences in information level, information angle, and information focus between the semantic information of the problem sentences represented by text features and speech features, and the fact that the error correction text based on the transcribed text loses some information due to transcription errors during the speech transcription process, while the semantic information contained in the speech features can just supplement this part of the information, in this embodiment of the invention, the two can be fused in the feature extraction model so that the text features and speech features can complement each other. The fusion process can be based on the correlation between the text features and speech features, that is, the feature extraction model can fuse the text features of the error correction text and the speech features of the problem speech according to the correlation between the text features and speech features, so as to obtain the problem fusion features;

[0092] The feature fusion here is actually achieved through the feature fusion network in the feature extraction model. Specifically, text features and image features are input into the feature fusion network in the feature extraction model. The feature fusion network can compare and fuse the text features and speech features based on the correlation between them to obtain the problem fusion features.

[0093] It is worth noting that the feature fusion network here can be understood as an attention mechanism. It can capture the relationship between text features and speech features, and use this relationship to add feature weights that are beneficial to implicit error correction (error tolerance) during the feature fusion process, while weakening the weights of irrelevant features. This allows the downstream question answering task to output the correct answer without being affected by transcription errors.

[0094] Before the error correction text and the question speech are input into the feature extraction model, the feature extraction model can be pre-trained using sample question pairs and the semantic relationship between the two sample question speech in the sample question pair. Here, the semantic relationship represents the similarity or difference in the semantics of the sample question statements contained in the two sample question speech, that is, whether the semantics are the same or different.

[0095] The training process of the feature extraction model specifically includes: First, collecting a large number of sample question speech to form positive sample question pairs and negative sample question pairs; wherein, the two sample question speech in the positive sample question pair have the same semantics, and the two sample question speech in the negative sample question pair have different semantics; then, the initial feature extraction model can be trained based on the positive sample question pairs and negative sample question pairs to obtain the trained feature extraction model.

[0096] In this embodiment of the invention, by training the model using positive and negative sample question pairs, the model can fully learn the proximity relationship between the sample question fusion features of two sample question speech under different semantic conditions. When the semantics are the same, the training makes the similarity between the sample question fusion features of the two sample question speech output by the model as high as possible; correspondingly, when the semantics are different, the similarity between the sample question fusion features of the two sample question speech output by the model is made as low as possible, which helps to improve the accuracy of speech question answering.

[0097] Based on the above embodiments, Figure 2 This is a flowchart illustrating the model training process provided by the present invention, as shown below. Figure 2 As shown, the feature extraction model is trained based on the following steps:

[0098] Step 210: Based on the initial feature extraction model, determine the sample question fusion features of the sample question speech;

[0099] Step 220: Select two sample question voices with the same semantics from the sample question voices as positive sample question pairs; select two sample question voices with different semantics from each sample question voice as negative sample question pairs.

[0100] Step 230: Based on the feature similarity between the sample question fusion features of the two sample question speech in the positive sample question pair and the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair, the initial feature extraction model is iterated to obtain the feature extraction model.

[0101] Specifically, the training process of a feature extraction model may include the following steps:

[0102] First, in step 210, an initial feature extraction model needs to be determined. This initial feature extraction model includes an initial text feature extraction network, an initial speech feature extraction network, and an initial feature fusion network. These three networks in the initial feature extraction model can determine the sample question fusion features of the sample question speech. Specifically, the initial text feature extraction network extracts features from the sample correction text to obtain sample text features. Here, the sample correction text is determined based on the sample transcribed text of the sample question speech. The initial speech feature extraction network extracts features from the sample question speech to obtain sample speech features. Then, in the initial feature fusion network, the correlation between the sample text features and the sample speech features is used as a benchmark to fuse the two, thereby obtaining the sample question fusion features.

[0103] Then, step 220 is executed to determine the semantic relationship between the sample question speech. Here, the semantic relationship represents the semantic similarity and difference of the sample question statements contained in the two. It can be measured by the similarity between features or it can be pre-annotated. This embodiment of the invention does not specifically limit this. Then, based on whether the semantics between the sample question speech are the same, two sample question speech with the same semantics can be selected from the sample question speech as positive sample question pairs. That is, positive sample question pairs are formed by two sample question speech with the same semantics. Correspondingly, two sample question speech with different semantics can be selected from the sample question speech as negative sample question pairs. That is, negative sample question pairs are formed by two sample question speech with different semantics.

[0104] Subsequently, step 330 is executed to determine the feature similarity between the sample question fusion features of the two sample question speech in the positive sample question pair output by the initial feature extraction model, and the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair. Based on these two feature similarities, the loss of the model is determined, and the parameters of the initial feature extraction model are iterated according to the loss to obtain the feature extraction model. This process is essentially adjusting the parameters of the initial feature extraction model so that it can fully learn the mapping relationship between the sample question speech and the sample question fusion features in the positive and negative sample question pairs. Thus, in the application process, the question fusion features corresponding to the error correction text and the question speech can be output based on this mapping relationship.

[0105] It is worth noting that the feature similarity here can be expressed as cosine similarity, Euclidean distance, Minkowski distance, etc.; and preferably, in the embodiments of the present invention, the feature similarity can be the semantic similarity between features measured by cosine similarity.

[0106] It should be noted that in the above process, the initial feature extraction model is trained with the feature similarity between the sample question fusion features of the two sample question speech pieces in a positive sample question pair and the feature similarity between the sample question fusion features of the two sample question speech pieces in a negative sample question pair as the training target. This allows the model to fully learn the proximity relationship between the sample question fusion features of the two sample question speech pieces under different semantic conditions. In other words, the initial feature extraction model can determine the feature similarity between the sample question fusion features of the sample question speech pieces based on the semantic similarities and differences between them. The aim is to maximize the similarity between the sample question fusion features of the two sample question speech pieces output by the model when the semantics are the same, and minimize the similarity between the sample question fusion features of the two sample question speech pieces output by the model when the semantics are different.

[0107] Furthermore, when training the model based on feature similarity, if the feature similarity between the sample question fusion features of the two sample question speech in the output positive sample question pair is high, and the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair is low, then the loss of the initial feature extraction model can be determined to be small. Conversely, if the feature similarity between the sample question fusion features of the two sample question speech in the output positive sample question pair is low, and / or the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair is high, then the loss of the initial feature extraction model can be determined to be large.

[0108] In this embodiment of the invention, the model is trained by using the feature similarity between the sample question fusion features of two sample question speech in a positive and negative sample question pair. This not only ensures the performance of the model, but also enables the model to fully learn the semantic relationship between the two sample question speech under different sample combinations during the training process. This provides crucial assistance for fault tolerance in the application process and for improving the accuracy of speech question answering.

[0109] Based on the above embodiments, the loss function of the above training process can be expressed as the following formula:

[0110]

[0111] In the formula, Loss(x,y,l;w) represents the loss function during training, x and y represent the two sample question speech in the sample question pair, l represents the semantic relationship between x and y, that is, the semantic similarity or difference between the two, 1 indicates that the two have the same semantics, -1 indicates that the two have different semantics, w is the parameter of the initial feature extraction model, and to ensure the effectiveness of the loss function, the margin is set in the range of (-1,1), which is usually 0 by default.

[0112] Based on the above embodiments, the feature extraction model includes a speech feature extraction network, a text feature extraction network, and a feature fusion network;

[0113] Among them, the speech feature extraction network is used to extract features from the problematic speech to obtain the speech features of the problematic speech;

[0114] Text feature extraction networks are used to extract features from error-corrected text to obtain its text features.

[0115] Feature fusion networks are used to fuse text features and speech features based on the correlation between them, resulting in question fusion features.

[0116] Specifically, in the above process, the feature extraction model used for feature extraction and feature fusion includes three networks: a text feature extraction network, a speech feature extraction network, and a feature fusion network. The text feature extraction network and the speech feature extraction network run in parallel, and the text feature extraction network and the speech feature extraction network are simultaneously connected serially to the feature fusion network. In other words, the outputs of the text feature extraction network and the speech feature extraction network are the inputs of the feature fusion network.

[0117] The text feature extraction network takes as input the error correction text of the problematic speech and extracts features from the error correction text to represent the problematic sentences, thereby obtaining the text features of the error correction text.

[0118] The input to the speech feature extraction network is the question speech, which can be used to extract features from the question speech, extracting rich features related to the question statement contained in the speech waveform and speech spectrum, thereby obtaining the speech features of the question statement.

[0119] The feature extraction network takes the text features of the error-correcting text and the speech features of the question speech as input. It can capture the relationship between the text features and the speech features to obtain the correlation between the text features and the speech features. This correlation can then be applied to fuse the two features. In the process of fusion, the network focuses on the relevant features that are helpful for error-tolerant processing and ignores other features, thereby reducing the interference of transcription errors on subsequent question-answering tasks.

[0120] Based on the above embodiments, the feature fusion network includes a feature weighting layer and a feature fusion layer, with residual connections between the feature weighting layer and the feature fusion layer;

[0121] The feature weighting layer is used to determine the weights of speech features based on the correlation between text features and speech features, and to weight the speech features based on the weights to obtain weighted speech features.

[0122] The feature fusion layer is used to determine the first speech feature based on weighted speech features and text features, and to fuse the first speech feature and text features to obtain the question fusion feature.

[0123] Specifically, the feature fusion network includes a feature weighting layer and a feature fusion layer, connected by a residual connection. This means that the input and output of the feature weighting layer are used together as the input to the feature fusion layer. The feature fusion network here can be viewed as an attention mechanism. The feature fusion process based on the attention mechanism can be divided into two steps: feature weighting and feature fusion.

[0124] The feature weighting layer is used to implement feature weighting. It can determine the weight of speech features based on the correlation between text features and speech features, and then weight the speech features according to the weight to obtain weighted speech features.

[0125] The feature fusion layer is used to achieve feature fusion. It can superimpose speech features on the weighted speech features output by the feature weighting layer to obtain the first speech feature. Then, the first speech feature and the text feature can be fused to obtain the question fusion feature.

[0126] In this embodiment of the invention, the residual connection relationship between the feature weighting layer and the feature fusion layer can accelerate the training speed of the feature extraction model, that is, accelerate the model convergence.

[0127] Based on the above embodiments, Figure 3This is a framework example diagram of the feature fusion process provided by the present invention, such as... Figure 3 As shown, applying a self-attention mechanism to text features can learn the correlation between the segmentation features of each word in the error-correcting text, while applying a self-attention mechanism to speech features can capture the dependencies between the speech frame features of each speech frame in the problem speech. This process can be represented by the following formula:

[0128]

[0129] In the formula, Q, K, and V are matrices generated based on text features and speech features, and K... T d is the transpose of K. k This indicates the dimension of matrix K.

[0130] The text features and speech features obtained after processing by the self-attention mechanism can be fused by the attention mechanism to achieve efficient fusion of the two features, thereby generating the final question fusion feature.

[0131] The attention mechanism here consists of two fully connected layers with a residual connection between them. Before passing through the other fully connected layer, the speech features need to be weighted using weights to obtain weighted speech features. A residual connection is then introduced on this basis to accelerate model convergence. Specifically, for the input speech features Z1 and text features Z2, the output question fusion feature X can be obtained, and its calculation formula is shown below:

[0132] a=σ(W2δ(W1Z1))

[0133] δ=a*Z1+Z1

[0134] X = S * Z2

[0135] In the formula, W1 and W2 are the parameters of the fully connected layer, δ is the activation function (The Rectified Linear Unit, ReLU), σ is the sigmoid activation function, a is the weight of the speech feature, Z1 is the speech feature, Z2 is the text feature, S is the first speech feature, and X is the question fusion feature.

[0136] Based on the above embodiments, step 120 includes:

[0137] Based on the text correction model, the transcribed text of the problematic speech is corrected to obtain the corrected text.

[0138] The text correction model is trained on the basis of the statistical language model by applying sample text and sample correction text of the sample text.

[0139] Specifically, in step 120, the process of performing text correction based on the transcribed text of the problematic speech to obtain the corrected text can be implemented using a text correction model. The specific process may include the following steps:

[0140] First, the transcribed text can be input into the text correction model. Then, the text correction model corrects the input transcribed text to correct superficial transcription errors such as homophones, similar words, and fixed collocations. Finally, the corrected text is output by the text correction model.

[0141] Before inputting the transcribed text into the text correction model, the text correction model can be pre-trained using sample text and its sample correction text. The training process of the text correction model includes the following steps: First, collect a large number of sample texts and determine the sample correction texts of the sample texts; then, the initial text correction model can be trained based on the sample texts and their sample correction texts to obtain the text correction model.

[0142] In this embodiment of the invention, the initial text correction model can be a statistical language model, such as an n-gram. In other words, the explicit error correction process for the transcribed text of the samples is actually implemented in a statistical language model. When performing text correction, the statistical language model n-gram typically uses bi-grams or tri-grams to model sentences.

[0143] Taking the sample question "What fertilizer should be applied to wheat in spring?" from the sample transcribed text as an example, the text correction process based on statistical language models is explained:

[0144] The statistical language model first segments the transcribed text of the sample, dividing the sample question statement "What fertilizer should be applied to wheat in spring" into "wheat / spring / should / is / what / fertilizer". Then, the word list obtained from the segmentation can be divided into a list of bigram pairs and a list of triplets. The list of bigram pairs is "[wheat spring][spring should][should be][what][what fertilizer]", and the list of triplets is "[wheat spring should][spring should be][should be][what][what fertilizer]". After that, the scores of the bigram list and the scores of the triplets in the sample question statement can be calculated sequentially using bi-gram and tri-gram models (log probability). The bi-gram model calculates 5 scores, and the tri-gram model calculates 4 scores. The scores are filled out n-1 times from the left and right ends of the score, and the average score is calculated using a sliding window of size n. This results in a score with the shape (6, 2), where n is the order of the n-gram model. Averaging along the column direction yields a vector of shape (6, 1), where each value corresponds to the average language log probability of the current position.

[0145] When the average language log probability is lower than a preset threshold, the current position is determined to be incorrect. For the incorrect word, homophones can be searched as replacement words. All replacement words are replaced in the corresponding position and PPL (perplexity) is calculated. Then, the PPL is sorted, and the replacement word with the lowest perplexity is selected as the corrected word for that position.

[0146] Taking bi-gram as an example, the PPL calculation formula can be expressed as:

[0147]

[0148] In the formula, M represents the sample question statement, w i Let represent the i-th word in the sample question statement, k represent the number of words in the sample question statement, and n represent the order of the n-gram model.

[0149] Based on the above embodiments, the initial feature extraction model includes an initial speech feature extraction network, an initial text feature extraction network, and an initial feature fusion network;

[0150] Among them, the initial speech feature extraction network is used to extract features from the sample question speech to obtain sample speech features;

[0151] The initial text feature extraction network is used to extract features from the sample error correction text of the sample question speech to obtain sample text features. The sample error correction text is obtained by correcting text errors based on the sample transcribed text of the sample question speech.

[0152] The initial feature fusion network is used to fuse sample text features and sample speech features based on the correlation between them, to obtain sample question fusion features;

[0153] The initial speech feature extraction network is built on the basis of the speech pre-trained model, and the initial text feature extraction network is built on the basis of the language model.

[0154] Specifically, the initial feature extraction model also includes three networks: the initial speech feature extraction network, the initial text feature extraction network, and the initial feature fusion network.

[0155] The initial speech feature extraction network takes sample question speech as input and extracts features from the input sample question speech to obtain sample speech features.

[0156] The input to the initial text feature extraction network is the sample error correction text of the sample question speech. It can extract features from the input sample error correction text to obtain sample text features. Here, the sample error correction text is obtained by performing text correction on the sample transcribed text obtained by transcribing the sample question speech.

[0157] The input to the initial feature fusion network is the sample speech features output by the initial speech feature extraction network and the sample text features output by the initial text feature extraction network. It can use the correlation between the sample text features and the sample speech features to fuse the sample text features and the sample speech features, thereby obtaining the sample question fusion features.

[0158] The initial speech feature extraction network is built on the basis of a speech pre-training model, such as the general speech pre-training model WavLM. In this embodiment of the invention, the WavLM speech pre-training model is used as the initial speech feature extraction network, which can extract the rich semantic information contained in the sample question speech. Furthermore, through the Masked Speech Denoising and Prediction Framework in its pre-training process, the model can have a strong semantic representation ability for key speech positions in the sample question speech.

[0159] The distinctive feature of WavLM is that it abandons the extraction of low-level speech descriptors, directly uses speech waveforms as model input, and uses masked speech denoising to model discrete pseudo-labels. In addition, WavLM's Transformer encoder uses gated relative position bias, which introduces the relative position of speech frames into attention calculation, thereby enabling better modeling of local speech information.

[0160] The initial text feature extraction network is built on the basis of a language model, such as BERT (Bidirectional Encoder Representation from Transformers). In this embodiment of the invention, BERT is used as the initial text feature extraction network, which can extract features from the sample error correction text, that is, it can extract features that can represent the sample problem statement from the sample error correction text. In this process, the key information for representing the sample problem statement mainly includes the following three points: first, the semantic information of each word in the sample problem statement; second, the position information of each word in the sample problem statement; and third, the interaction relationship between each word. These three types of information are input into the encoder of the transformer in BERT for feature extraction. The encoder consists of a multi-head self-attention layer and a feed forward network, which can learn the interaction relationship between the input information, thereby generating the word segmentation features of each word in the sample problem statement.

[0161] During the transfer training of BERT, a large amount of sample question-and-answer data from vertical domains can be used to fine-tune BERT with masks. During training, the token in each training sequence is randomly replaced with a mask token with a 15% probability, and then the original word at the mask position is predicted. This makes the model sensitive to words at the mask position in downstream question-and-answer tasks (i.e., transcribed words that cannot be corrected by text correction models), and thus a feature-level error correction mechanism can be introduced when generating features.

[0162] Based on the above embodiments, Figure 4 This is a general framework diagram of the voice question-answering method provided by the present invention, as shown below. Figure 4 As shown, the method includes:

[0163] First, we need to identify the problematic audio.

[0164] Subsequently, text correction can be performed on the transcribed text of the problematic speech to obtain the corrected text. Specifically, text correction can be performed on the transcribed text of the problematic speech based on the text correction model to obtain the corrected text. The text correction model here is trained on the basis of a statistical language model by applying sample text and sample corrected text of the sample text.

[0165] Subsequently, features can be extracted from the error correction text and the question speech separately to obtain text features and speech features. Based on the correlation between the text features and speech features, the text features and speech features are fused to obtain the question fusion features. Specifically, based on the feature extraction model, features can be extracted from the error correction text and the question speech separately to obtain text features and speech features. Based on the correlation between the text features and speech features, the text features and speech features are fused to obtain the question fusion features. The feature extraction model is trained based on sample question pairs and whether the semantics of the two sample question speech in the sample question pair are the same.

[0166] The feature extraction model includes a speech feature extraction network, a text feature extraction network, and a feature fusion network. The speech feature extraction network is used to extract features from the problematic speech to obtain the speech features of the problematic speech. The text feature extraction network is used to extract features from the error correction text to obtain the text features of the error correction text. The feature fusion network is used to fuse the text features and speech features based on the correlation between the text features and speech features to obtain the problem fusion features.

[0167] Furthermore, the feature fusion network includes a feature weighting layer and a feature fusion layer, with residual connections between the feature weighting layer and the feature fusion layer; wherein, the feature weighting layer is used to determine the weights of the speech features based on the correlation between the text features and the speech features, and to weight the speech features based on the weights to obtain weighted speech features; the feature fusion layer is used to determine the first speech feature based on the weighted speech features and the speech features, and to fuse the first speech feature and the text features to obtain the question fusion feature.

[0168] The feature extraction model includes the following steps: Based on the initial feature extraction model, determine the sample question fusion features of the sample question speech; Select two sample question speech with the same semantics from the sample question speech as positive sample question pairs, and select two sample question speech with different semantics from each sample question speech as negative sample question pairs; Based on the feature similarity between the sample question fusion features of the two sample question speech in the positive sample question pair and the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair, iterate the parameters of the initial feature extraction model to obtain the feature extraction model.

[0169] The initial feature extraction model includes an initial speech feature extraction network, an initial text feature extraction network, and an initial feature fusion network. The initial speech feature extraction network is used to extract features from the sample question speech to obtain sample speech features. The initial text feature extraction network is used to extract features from the sample error correction text of the sample question speech to obtain sample text features. The sample error correction text is obtained by correcting text errors based on the sample transcription text of the sample question speech. The initial feature fusion network is used to fuse the sample text features and sample speech features based on the correlation between them to obtain sample question fusion features. The initial speech feature extraction network is built on the basis of the speech pre-trained model, and the initial text feature extraction network is built on the basis of the language model.

[0170] After that, the question fusion features can be matched with the candidate question fusion features of each candidate question, and the answer corresponding to the successfully matched candidate question can be determined as the answer to the question speech.

[0171] The method provided in this invention performs text correction on the transcribed text of the question speech to obtain corrected text, and extracts features from both the corrected text and the question speech to obtain text features and speech features respectively. The features are then fused using the correlation between the two to obtain question fusion features. These question fusion features are matched with the candidate question fusion features of each candidate question, and the answer corresponding to the successfully matched candidate question is determined as the answer to the question speech. Through explicit error correction of the transcribed text and fault tolerance processing at the feature level during feature fusion, the negative impact of transcription errors in the transcribed text on downstream question-answering tasks can be minimized. This overcomes the shortcomings of traditional solutions, such as the inability to correct long-distance dependency errors with semantic relationships and poor performance in vertical domains. By explicitly correcting and tolerating transcription errors, the accuracy of speech question answering can be greatly improved.

[0172] The voice question-and-answer device provided by the present invention is described below. The voice question-and-answer device described below can be referred to in correspondence with the voice question-and-answer method described above.

[0173] Figure 5 This is a schematic diagram of the structure of the voice question-and-answer device provided by the present invention, as shown below. Figure 5 As shown, the device includes:

[0174] The speech determination unit 510 is used to determine the problematic speech;

[0175] The text correction unit 520 is used to perform text correction based on the transcribed text of the problematic speech to obtain the corrected text;

[0176] The feature fusion unit 530 is used to extract features from the error correction text and the problem speech respectively to obtain text features and speech features, and to fuse the text features and speech features based on the correlation between the text features and the speech features to obtain problem fusion features;

[0177] The answer determination unit 540 is used to match the question fusion feature with the candidate question fusion feature of each candidate question, and determine the answer corresponding to the successfully matched candidate question as the answer to the question speech.

[0178] The speech question-answering device provided by this invention performs text correction on the transcribed text of the question speech to obtain corrected text, and extracts features from both the corrected text and the question speech to obtain text features and speech features respectively. These features are then fused using the correlation between them to obtain question fusion features. The question fusion features are matched with the candidate question fusion features of each candidate question, and the answer corresponding to the successfully matched candidate question is determined as the answer to the question speech. Through explicit error correction of the transcribed text and fault tolerance processing at the feature level during feature fusion, the negative impact of transcription errors in the transcribed text on downstream question-answering tasks can be minimized. This overcomes the shortcomings of traditional solutions, such as the inability to correct long-distance dependency errors with semantic relationships and poor speech question-answering performance in vertical domains. By explicitly correcting and tolerating transcription errors, the accuracy of speech question answering can be greatly improved.

[0179] Based on the above embodiments, the feature fusion unit 530 is used for

[0180] Based on the feature extraction model, features are extracted from the error correction text and the question speech respectively to obtain text features and speech features. Based on the correlation between the text features and the speech features, the text features and the speech features are fused to obtain the question fusion features.

[0181] The feature extraction model is trained based on sample question pairs and whether the semantics of the two sample question voices in the sample question pair are the same.

[0182] Based on the above embodiments, the device further includes a model training unit, used for:

[0183] Based on the initial feature extraction model, the sample question fusion features of the sample question speech are determined;

[0184] From the sample question speech, two sample question speech with the same semantics are selected as positive sample question pairs, and from the sample question speech, two sample question speech with different semantics are selected as negative sample question pairs.

[0185] Based on the feature similarity between the sample question fusion features of the two sample question speech in the positive sample question pair and the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair, the initial feature extraction model is iterated to obtain the feature extraction model.

[0186] Based on the above embodiments, the feature extraction model includes a speech feature extraction network, a text feature extraction network, and a feature fusion network;

[0187] The speech feature extraction network is used to extract features from the problematic speech to obtain the speech features of the problematic speech;

[0188] The text feature extraction network is used to extract features from the error-corrected text to obtain the text features of the error-corrected text;

[0189] The feature fusion network is used to fuse the text features and the speech features based on the correlation between them to obtain question fusion features.

[0190] Based on the above embodiments, the feature fusion network includes a feature weighting layer and a feature fusion layer, and the feature weighting layer and the feature fusion layer are residually connected;

[0191] The feature weighting layer is used to determine the weight of the speech feature based on the correlation between the text feature and the speech feature, and to weight the speech feature based on the weight to obtain weighted speech features;

[0192] The feature fusion layer is used to determine a first speech feature based on the weighted speech features and the speech features, and to fuse the first speech feature and the text features to obtain the question fusion feature.

[0193] Based on the above embodiments, the text correction unit 520 is used for:

[0194] Based on the text correction model, the transcribed text of the problematic speech is corrected to obtain the corrected text.

[0195] The text correction model is trained using sample text and sample correction texts of the sample text, based on a statistical language model.

[0196] Based on the above embodiments, the initial feature extraction model includes an initial speech feature extraction network, an initial text feature extraction network, and an initial feature fusion network;

[0197] The initial speech feature extraction network is used to extract features from the sample question speech to obtain sample speech features;

[0198] The initial text feature extraction network is used to extract features from the sample error correction text of the sample problematic speech to obtain sample text features. The sample error correction text is obtained by correcting text errors based on the sample transcribed text of the sample problematic speech.

[0199] The initial feature fusion network is used to fuse the sample text features and the sample speech features based on the correlation between them, to obtain sample question fusion features;

[0200] The initial speech feature extraction network is built on the basis of the speech pre-trained model, and the initial text feature extraction network is built on the basis of the language model.

[0201] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a voice question-and-answer method. This method includes: determining the question speech; performing text correction based on the transcribed text of the question speech to obtain corrected text; extracting features from the corrected text and the question speech respectively to obtain text features and speech features; fusing the text features and speech features based on the correlation between the text features and the speech features to obtain a question fusion feature; matching the question fusion feature with the candidate question fusion features of each candidate question; and determining the answer corresponding to the successfully matched candidate question as the answer to the question speech.

[0202] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the voice question-answering method provided by the above methods, the method comprising: determining a question voice; performing text correction based on the transcribed text of the question voice to obtain a corrected text; extracting features from the corrected text and the question voice respectively to obtain text features and voice features; fusing the text features and the voice features based on the correlation between the text features and the voice features to obtain a question fusion feature; matching the question fusion feature with the candidate question fusion features of each candidate question, and determining the answer corresponding to the successfully matched candidate question as the answer to the question voice.

[0204] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the voice question-answering method provided by the above methods. The method includes: determining a question voice; performing text correction based on the transcribed text of the question voice to obtain a corrected text; extracting features from the corrected text and the question voice respectively to obtain text features and voice features; fusing the text features and the voice features based on the correlation between the text features and the voice features to obtain a question fusion feature; matching the question fusion feature with the candidate question fusion features of each candidate question; and determining the answer corresponding to the successfully matched candidate question as the answer to the question voice.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice question-and-answer method, characterized in that, include: Identify the problematic audio; Text correction is performed on the transcribed text of the problematic speech to obtain the corrected text; Feature extraction is performed on the error-corrected text and the question speech to obtain text features and speech features, respectively. Based on the correlation between the text features and the speech features, the text features and the speech features are fused to obtain question fusion features. In the feature fusion process, the semantic information contained in the speech features can compensate for the semantic information lost in the transcribed text due to transcription errors. Speech question answering based on the fused question features can minimize the impact of transcription errors on speech question answering. The question fusion feature is matched with the candidate question fusion feature of each candidate question, and the answer corresponding to the successfully matched candidate question is determined as the answer to the question speech.

2. The voice question-and-answer method according to claim 1, characterized in that, The steps involve extracting features from the corrected text and the problematic speech respectively to obtain text features and speech features, and then fusing the text features and speech features based on the correlation between them to obtain a problem fusion feature, including: Based on the feature extraction model, features are extracted from the error correction text and the question speech respectively to obtain text features and speech features. Based on the correlation between the text features and the speech features, the text features and the speech features are fused to obtain the question fusion features. The feature extraction model is trained based on sample question pairs and whether the semantics of the two sample question voices in the sample question pair are the same.

3. The voice question-and-answer method according to claim 2, characterized in that, The feature extraction model is trained based on the following steps: Based on the initial feature extraction model, the sample question fusion features of the sample question speech are determined; From the sample question speech, two sample question speech with the same semantics are selected as positive sample question pairs, and from the sample question speech, two sample question speech with different semantics are selected as negative sample question pairs. Based on the feature similarity between the sample question fusion features of the two sample question speech in the positive sample question pair and the feature similarity between the sample question fusion features of the two sample question speech in the negative sample question pair, the initial feature extraction model is iterated to obtain the feature extraction model.

4. The voice question-and-answer method according to claim 2 or 3, characterized in that, The feature extraction model includes a speech feature extraction network, a text feature extraction network, and a feature fusion network; The speech feature extraction network is used to extract features from the problematic speech to obtain the speech features of the problematic speech; The text feature extraction network is used to extract features from the error-corrected text to obtain the text features of the error-corrected text; The feature fusion network is used to fuse the text features and the speech features based on the correlation between them to obtain question fusion features.

5. The voice question-and-answer method according to claim 4, characterized in that, The feature fusion network includes a feature weighting layer and a feature fusion layer, and the feature weighting layer and the feature fusion layer are residually connected; The feature weighting layer is used to determine the weight of the speech feature based on the correlation between the text feature and the speech feature, and to weight the speech feature based on the weight to obtain weighted speech features; The feature fusion layer is used to determine a first speech feature based on the weighted speech features and the speech features, and to fuse the first speech feature and the text features to obtain the question fusion feature.

6. The voice question-and-answer method according to any one of claims 1 to 3, characterized in that, The text correction based on the transcribed text of the problematic speech to obtain the corrected text includes: Based on the text correction model, the transcribed text of the problematic speech is corrected to obtain the corrected text. The text correction model is trained using sample text and sample correction texts of the sample text, based on a statistical language model.

7. The voice question-and-answer method according to claim 3, characterized in that, The initial feature extraction model includes an initial speech feature extraction network, an initial text feature extraction network, and an initial feature fusion network; The initial speech feature extraction network is used to extract features from the sample question speech to obtain sample speech features; The initial text feature extraction network is used to extract features from the sample error correction text of the sample problematic speech to obtain sample text features. The sample error correction text is obtained by correcting text errors based on the sample transcribed text of the sample problematic speech. The initial feature fusion network is used to fuse the sample text features and the sample speech features based on the correlation between them, to obtain sample question fusion features; The initial speech feature extraction network is built on the basis of the speech pre-trained model, and the initial text feature extraction network is built on the basis of the language model.

8. A voice question-and-answer device, characterized in that, include: The speech determination unit is used to determine the problematic speech. The text correction unit is used to perform text correction based on the transcribed text of the problematic speech to obtain the corrected text; The feature fusion unit is used to extract features from the error-corrected text and the question speech respectively to obtain text features and speech features. Based on the correlation between the text features and the speech features, the text features and the speech features are fused to obtain question fusion features. In the feature fusion process, the semantic information contained in the speech features can compensate for the semantic information lost in the transcribed text due to transcription errors. The speech question answering based on the fused question features can minimize the impact of transcription errors on speech question answering. The answer determination unit is used to match the question fusion feature with the candidate question fusion feature of each candidate question, and determine the answer corresponding to the successfully matched candidate question as the answer to the question speech.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the voice question-and-answer method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the voice question-and-answer method as described in any one of claims 1 to 7.

Images