Question-and-answer oral proficiency testing methods and related devices, electronic equipment, and storage media

Abstract

This application discloses a question-and-answer spoken language detection method and related devices, electronic devices, and storage media. The method includes: recognizing the spoken response of a target subject to a test question to obtain a first phoneme sequence; selecting a second phoneme sequence as a target phoneme sequence based on the matching results of the first phoneme sequence with each of the second phoneme sequences; obtaining an editing operation sequence that transforms the target phoneme sequence into the first phoneme sequence, and obtaining a key identifier sequence for the target phoneme sequence; wherein the key identifier sequence contains a first identifier for each phoneme in the target phoneme sequence, and the first identifier represents whether the phoneme belongs to a keyword marked in the reference answer to which the target phoneme sequence belongs; and performing detection based on the spoken response, the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain the spoken language detection result of the target subject. This scheme can improve the accuracy of question-and-answer spoken language detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a question-and-answer spoken language detection method and related devices, electronic devices, and storage media. Background Technology

[0002] Oral language assessment is an important application of AI (Artificial Intelligence) technology in the field of language education. Typically, oral assessments evaluate a participant's spoken language proficiency by analyzing their oral responses to test questions.

[0003] However, existing technologies, such as the Ordinate automatic scoring technology used in Phone Pass SET-10, the SpeechRater automatic scoring system, and multi-feature fusion automatic scoring systems, do not achieve ideal accuracy in spoken language detection. Therefore, improving the accuracy of spoken language detection in question-and-answer sessions has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a question-and-answer spoken language detection method and related devices, electronic devices, and storage media, which can improve the accuracy of question-and-answer spoken language detection.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a question-and-answer spoken language detection method, comprising: recognizing the spoken response of a target object to a test question to obtain a first phoneme sequence; wherein the test question has several reference answers, and each reference answer corresponds to a second phoneme sequence; selecting a second phoneme sequence as a target phoneme sequence based on the matching results of the first phoneme sequence with each of the second phoneme sequences; obtaining an editing operation sequence for transforming the target phoneme sequence into the first phoneme sequence, and obtaining a key identifier sequence for the target phoneme sequence; wherein the key identifier sequence contains a first identifier for each phoneme in the target phoneme sequence, and the first identifier represents whether the phoneme belongs to a keyword marked in the reference answer to which the target phoneme sequence belongs; and performing detection based on the spoken response, the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain the spoken language detection result of the target object.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a question-and-answer spoken language detection device, comprising: a recognition module, a matching module, an acquisition module, and a detection module. The recognition module is used to recognize the speech of a target object's response to a test question, obtaining a first phoneme sequence; wherein the test question has several reference answers, and each reference answer corresponds to a second phoneme sequence; the matching module is used to select a second phoneme sequence as a target phoneme sequence based on the matching results of the first phoneme sequence with each of the second phoneme sequences; the acquisition module is used to acquire an editing operation sequence that transforms the target phoneme sequence into the first phoneme sequence, and to acquire a key identifier sequence of the target phoneme sequence; wherein the key identifier sequence contains a first identifier for each phoneme in the target phoneme sequence, and the first identifier represents whether the phoneme belongs to a keyword marked in the reference answer to which the target phoneme sequence belongs; the detection module is used to perform detection based on the response speech, the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain the spoken language detection result of the target object.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device including a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the question-and-answer spoken language detection method of the first aspect described above.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the question-and-answer spoken language detection method of the first aspect described above.

[0009] The above scheme identifies the target subject's speech in response to a test question, obtaining a first phoneme sequence. The test question has several reference answers, each with a corresponding second phoneme sequence. Based on the matching results of the first phoneme sequence with each of the second phoneme sequences, the second phoneme sequence is selected as the target phoneme sequence. Then, the editing operation sequence for transforming the target phoneme sequence into the first phoneme sequence is obtained, along with a key identifier sequence for the target phoneme sequence. This key identifier sequence contains the first identifier of each phoneme in the target phoneme sequence, indicating whether the phoneme belongs to a keyword marked in the reference answer to which the target phoneme sequence belongs. Detection based on response speech, target phoneme sequences, editing operation sequences, and key identifier sequences yields the spoken language detection results for the target subject. In question-and-answer spoken language detection, the use of more granular sequence information at the phoneme level, compared to directly using the word level, improves detection granularity. Furthermore, the introduction of editing operation sequences, target phoneme sequences, and key identifier sequences, compared to simply concatenating the actual answer with the reference answer, helps to capture the differences between the actual answer and the reference answer, as well as key vocabulary in the reference answer, providing more detailed auxiliary information for question-and-answer spoken language detection. Therefore, it can improve the accuracy of question-and-answer spoken language detection. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the oral communication detection method for question-and-answer questions in this application;

[0011] Figure 2 This is a schematic diagram of an embodiment of the preprocessing process;

[0012] Figure 3 This is a schematic diagram of the framework of an embodiment of the end-to-end model;

[0013] Figure 4 This is a schematic diagram of the framework of an embodiment of an encoding network;

[0014] Figure 5 This is a schematic diagram of the framework of one embodiment of the preprocessing module;

[0015] Figure 6 This is a schematic diagram of the framework of an embodiment of a decoding network;

[0016] Figure 7 This is a schematic diagram of the framework of an embodiment of a prediction network;

[0017] Figure 8 This is a schematic diagram of the framework of an embodiment of the oral communication testing device of this application;

[0018] Figure 9 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0019] Figure 10 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0023] Please see Figure 1 , Figure 1This is a flowchart illustrating an embodiment of the oral communication detection method for question-and-answer questions of this application. Specifically, it may include the following steps:

[0024] Step S11: Identify the target object's speech in response to the test question to obtain the first phoneme sequence.

[0025] In this embodiment of the disclosure, the test question has several reference answers, and each reference answer corresponds to a second phoneme sequence. It should be noted that, for ease of detection, the test question can be set as a closed-ended question to reduce the obstacles to evaluation caused by the unconstrained responses of test subjects when the test question is set as an open-ended question. For example, closed-ended questions can be such as "Where can I buy flowers?" or "Where can I buy flowers?", etc., and are not limited thereto.

[0026] In one implementation scenario, test questions can have several pre-set reference answers, such as one, two, three, or more; there is no limitation here. For example, to adapt to various testing situations as much as possible, test questions can have as many pre-set reference answers as possible. Taking the test question "Where can I buy flowers?" as an example, pre-set reference answers include, but are not limited to, the following: "You can buy flowers in the flower shop," "In the flower shop," etc., without limitation here. Other situations can be deduced similarly, and will not be listed further here.

[0027] In one implementation scenario, for ease of detection, each reference answer can be pre-set with a corresponding phoneme sequence. To distinguish it from the phoneme sequence of the answer speech, in this embodiment, the phoneme sequence of the answer speech is referred to as the first phoneme sequence, and the phoneme sequence corresponding to the reference answer is referred to as the second phoneme sequence. For example, taking the reference answer "Inthe flower shop" as an example, its second phoneme sequence can be represented as: ih n dh ax fl aw ax sh aa p, where ih, n, dh, ax, f, ..., p represent each phoneme in the second phoneme sequence. The process of obtaining the second phoneme sequence can be found in the technical details of phoneme annotation, and will not be elaborated here.

[0028] In one implementation scenario, to improve the convenience and accuracy of recognizing the first phoneme sequence, a speech recognition model can be pre-trained. This model can then be used to recognize the response speech and obtain the first phoneme sequence. The speech recognition model can include, but is not limited to, recurrent neural networks and long short-term memory networks. Specifically, several sample speech samples can be collected in advance, and their sample phoneme sequences can be labeled. For example, the sample speech "In the flower shop" can be labeled with the sample phoneme sequence "ih n dh ax fl aw ax sh aa p". Based on this, the speech recognition model can be used to recognize the sample speech, obtaining a predicted phoneme sequence. Then, based on the difference between the sample phoneme sequence and the predicted phoneme sequence, the network parameters of the speech recognition model can be adjusted. For methods of measuring these differences, please refer to loss functions such as cross-entropy; for methods of adjusting these parameters, please refer to optimization methods such as gradient descent. These details will not be elaborated here. Taking the test question "Where can I buy flowers?" as an example, the answer is "In the kate shop". By recognizing this answer, we can obtain the first phoneme sequence "ih n dh axk ey t sh aa p". Other cases can be deduced similarly, and will not be listed here.

[0029] Step S12: Based on the matching results of the first phoneme sequence with each of the second phoneme sequences, select the second phoneme sequence as the target phoneme sequence.

[0030] In one implementation scenario, the edit distance between the first and second phoneme sequences can be used as the matching result corresponding to the second phoneme sequence, and the second phoneme sequence corresponding to the minimum edit distance can be used as the target phoneme sequence. This method, by obtaining the edit distance between the first and second phoneme sequences, selects the target phoneme sequence based on the edit distance. This approach can select the target phoneme sequence from the second phoneme sequence that is closest to the first phoneme sequence as much as possible, thus improving the accuracy of determining the target phoneme sequence.

[0031] In a specific implementation scenario, edit distance, also known as Lewinstein distance, is a quantitative measure of the difference between two strings (e.g., English strings). It measures the minimum number of operations required to transform one string into another. For example, transforming the string "love" into "lolpe" requires two operations: inserting "l" and changing "v" to "p". Therefore, the edit distance between "love" and "lolpe" is 2. Other cases can be deduced similarly, and will not be listed here.

[0032] In a specific implementation scenario, taking the test question "Where can I buy flowers?" as an example, the reference answers include "You can buy flowers in the flower shop" and "In the flower shop," and the target's response is "In the Kate shop." By using the above method, we can determine that the edit distance between the first phoneme sequence of the response "In the Kate shop" and the second phoneme sequence of the reference answer "In the flower shop" is the smallest. Therefore, the second phoneme sequence of the reference answer "In the flower shop," "ih n dh axf l aw ax sh aa p," can be used as the target phoneme sequence. Other cases can be deduced similarly, and will not be listed here.

[0033] In another implementation scenario, unlike the aforementioned matching method using edit distance, matching methods such as BF (Brute Force) and RK (Robin-Karp) can also be used, which are limited here. For the specific process of matching using the above methods, please refer to the technical details of matching methods such as BF (Brute Force) and RK (Robin-Karp), which will not be elaborated here. Based on this, the matching degree between the first phoneme sequence and the second phoneme sequence can be obtained as the matching result corresponding to the second phoneme sequence. Therefore, the second phoneme sequence with the highest matching degree can be selected as the target phoneme sequence.

[0034] Step S13: Obtain the editing operation sequence for transforming the target phoneme sequence into the first phoneme sequence, and obtain the key identifier sequence of the target phoneme sequence.

[0035] In this embodiment, the key identifier sequence includes a first identifier for each phoneme in the target phoneme sequence, and the first identifier indicates whether the phoneme belongs to the keyword marked in the reference answer to which the target phoneme sequence belongs. For example, the number "0" can represent that the phoneme does not belong to the keyword marked in the reference answer to which the target phoneme sequence belongs, and the number "1" can represent that the phoneme belongs to the keyword marked in the reference answer to which the target phoneme sequence belongs. For example, taking the test question "Where can I buy flowers?" as an example, and the second phoneme sequence of the reference answer "In the flower shop" "ih n dhax fl aw ax sh aa p" as the target phoneme sequence, where the marked keyword is "shop" and the corresponding phoneme is "sh aa p", its key identifier sequence can be represented as: "0 0 0 0 0 0 0 0 1 1 1". Other cases can be deduced similarly, and will not be listed here. Of course, in practical applications, it is not limited to using numbers to represent whether a phoneme belongs to the keyword marked in the reference answer to which the target phoneme sequence belongs.

[0036] In one implementation scenario, the editing operation sequence can specifically include a second identifier for each phoneme in the target phoneme sequence, with different second identifiers representing different editing operations. For example, different numbers can also be used to represent different editing operations. For instance, the number "0" can represent the editing operation "skip," the number "1" can represent the editing operation "insert," the number "2" can represent the editing operation "delete," and the number "3" can represent the editing operation "replace." Of course, in practical applications, it is not limited to using numbers to represent editing operations. That is to say, the editing operation sequence can specifically include: transforming the target phoneme sequence into a first phoneme sequence, and the editing operations that need to be performed on each phoneme in the target phoneme sequence.

[0037] In one implementation scenario, taking the test question "Where can I buy flowers?" as an example, as mentioned before, the target phoneme sequence is "ih n dh ax fl aw ax sh aa p", and when the answer is "In the kate shop", the first phoneme sequence is "ih n dh ax k ey t sh aa p". Therefore, each phoneme in the target phoneme sequence needs to perform the following editing operations: skip, skip, skip, skip, replace, replace, replace, delete, skip, skip, skip. Thus, the editing operation sequence can be represented as "0 0 0 0 3 3 3 2 0 0 0". Other cases can be deduced similarly, and will not be listed here.

[0038] It should be noted that, through the above steps, based on the test questions and answers, a series of preprocessing steps can be performed to obtain the editing operation sequence, target phoneme sequence, and key identifier sequence required for subsequent detection.

[0039] In one implementation scenario, please refer to the following: Figure 2 And Table 1, Figure 2 This is a schematic diagram of one embodiment of the preprocessing process, and Table 1 is a schematic table of one embodiment of the preprocessing process. For example... Figure 2 As shown in Table 1, taking students as the target audience as an example, we can obtain the student's response to the test question (i.e., the aforementioned "response voice"), such as "in the kate shop," and use an automatic speech recognition system to recognize the response voice to obtain the student's answer phoneme sequence (i.e., the aforementioned "first phoneme sequence"), such as "ih n dh ax k ey t sh aa p." Simultaneously, we can convert each reference answer to the test question into a phoneme sequence (i.e., the aforementioned "second phoneme sequence"), and match the phoneme sequence with the minimum edit distance to the student's answer phoneme sequence. The reference answer to which this phoneme sequence belongs is then taken as the best matching reference answer, such as the aforementioned reference answer "in the flower shop." Based on this, on the one hand, its phoneme sequence (i.e., the aforementioned target phoneme sequence) can be calculated, such as the editing operation sequence for converting “ihndh ax fl aw ax sh aa p” into the phoneme sequence of the student's answer, such as “00 0 0 3 3 32 0 0 0”. On the other hand, based on its marked keyword information, such as “shop”, the segment sequence (i.e., the aforementioned key identifier sequence) can be obtained, such as “0 0 0 0 0 0 0 0 1 1 1”.

[0040] Table 1. Schematic diagram of an embodiment of the preprocessing procedure.

[0041] Reference answer for minimum edit distance in the flower shop Answer voice in the Kate Shop First phoneme sequence ih n dh ax k ey t sh aa p Target phoneme sequence ih n dh ax fl aw ax sh aa p Edit operation sequence 0 0 0 0 3 3 3 2 0 0 0 Key identifier sequence 0 0 0 0 0 0 0 0 1 1 1

[0042] It should be noted that the above example is merely one possible scenario in practical application and does not limit other possible scenarios. When testing other problems, the same logic can be applied to determine the target phoneme sequence, editing operation sequence, and key identifier sequence.

[0043] Step S14: Detect the spoken language of the target object based on the response speech, target phoneme sequence, edit operation sequence and key identifier sequence.

[0044] In one implementation scenario, several acoustic features can be extracted based on the response speech. Then, based on the target phoneme sequence, editing operation sequence, and key identifier sequence, a first feature sequence is extracted. This first feature sequence contains the first feature representation of each phoneme in the target phoneme sequence, which includes at least the following information: the standard pronunciation of the phoneme, its importance, and whether the target subject pronounced the phoneme. Further, a cross-attention mechanism can be used to process the acoustic features and the first feature sequence to obtain a second feature sequence. This second feature sequence contains the second feature representation of each phoneme in the target phoneme sequence, which includes at least the following information: the target subject's oral response level regarding the phonemes. Finally, a scoring detection is performed based on the second feature sequence to obtain the oral detection result. This approach, through the cross-attention mechanism, processes the acoustic features and the feature sequence of the phoneme sequence at the phoneme level, making it easier to align the phoneme sequence with the audio information. This helps to further improve the fusion of acoustic and semantic information, thereby enhancing the accuracy of question-and-answer oral detection.

[0045] In a specific implementation scenario, to further enhance the fusion of acoustic and semantic information, when extracting acoustic features, feature extraction can first be performed based on the response speech to obtain first acoustic features of several audio frames. These first acoustic features can include, but are not limited to, MFCC, FBank, etc., and are not specified here. Based on this, convolution can be performed on the first acoustic features of the several audio frames to obtain second acoustic features. That is, the first acoustic features are downsampled through convolution to establish the contextual information of the audio frames. Further, feature encoding can be performed based on the second acoustic features to obtain several acoustic features. Feature encoding can be implemented using encoding networks such as Transformer, and is not limited here. For example, feature extraction can first be performed on the response speech to obtain N first acoustic features (i.e., acoustic features representing N audio frames), then convolution can be performed on the N first acoustic features to obtain L second acoustic features, where L is less than N, and then the L second acoustic features can be encoded to obtain L acoustic features. The above method extracts features based on the response speech to obtain the first acoustic features of several audio frames. Then, it performs convolution based on the first acoustic features of several audio frames to obtain the second acoustic features. Finally, it performs feature encoding based on the second acoustic features to obtain several acoustic features. This method can establish contextual information during the acoustic feature extraction process, which helps to improve the fusion of acoustic and semantic information.

[0046] In a specific implementation scenario, after obtaining the target phoneme sequence, the editing operation sequence, and the key identifier sequence, the three can be fused to obtain the sequence to be processed. The sequence to be processed contains each phoneme in the target phoneme sequence and its first identifier and second identifier. Based on this, the first feature representation of each phoneme in the target phoneme sequence can be extracted through operations such as convolution, thereby obtaining the first feature sequence.

[0047] In a specific implementation scenario, after obtaining L acoustic features and a first feature sequence, both can be processed using a cross-attention mechanism to obtain a second feature sequence. It should be noted that the first and second feature sequences have the same sequence length, both equal to the sequence length of the target phoneme sequence, i.e., the number of phonemes M contained in the target phoneme sequence. Specifically, several acoustic features can be transformed using the key matrix K of the cross-attention mechanism to obtain key features, and several acoustic features can be transformed using the value matrix V of the cross-attention mechanism to obtain value features. The first feature sequence can be transformed using the query matrix Q of the cross-attention mechanism to obtain a query feature sequence. Based on this, an attention matrix of size L*M can be obtained between the key features and the query feature sequence, where the element in the i-th row and j-th column represents the correlation between the i-th acoustic feature and the j-th phoneme. Further, the value feature V can be applied to the attention matrix to obtain a second feature sequence of length M. It should be noted that the specific meanings of the key matrix K, value matrix V, query matrix Q, and attention matrix can be found in the technical details of the cross-attention mechanism, and will not be elaborated here. The above method transforms several acoustic features based on the key and value matrices of the cross-attention mechanism to obtain key features and value features, respectively. Then, it transforms the first feature sequence based on the query matrix of the cross-attention mechanism to obtain the query feature sequence, thereby obtaining the attention matrix between the key features and the query feature sequence. Finally, the value features are applied to the attention matrix to obtain the second feature sequence. This approach effectively models the correlation between acoustic features and the first feature sequence, contributing to the improvement of the fusion of acoustic and semantic information.

[0048] In a specific implementation scenario, after obtaining the second feature sequence, a score mapping can be performed based on the second feature sequence to obtain the score value of each phoneme in the target phoneme sequence. Then, based on the score values ​​of each phoneme in the target phoneme sequence, numerical statistics are performed to obtain the spoken language score of the target object as the spoken language detection result. It should be noted that the score mapping can be implemented using neural networks such as feedforward networks, which are not limited here. Furthermore, numerical statistics can include, but are not limited to: taking the maximum value, taking the minimum value, taking the average value, taking the median value, etc., which are not limited here. For example, taking the average value as the numerical statistics, after obtaining the score values ​​of each phoneme in the target phoneme sequence, these score values ​​can be averaged to obtain the spoken language score of the target object. More precisely, by performing numerical statistics on the score values ​​of each phoneme in the target phoneme sequence, the spoken language score of the target object on the test question is obtained. Other methods can be used similarly to determine the spoken language score, which will not be listed here. The above method maps scores based on the second feature sequence to obtain the score value of each phoneme in the target phoneme sequence. Then, it performs numerical statistics based on the score values ​​of each phoneme in the target phoneme sequence to obtain the spoken language score of the target object as the spoken language detection result. This helps to perform spoken language detection at the phoneme level and determine the spoken language score, thereby improving the granularity of question-and-answer spoken language detection.

[0049] In another implementation scenario, unlike the aforementioned methods, to further improve accuracy, an end-to-end model can be used to detect the response speech, target phoneme sequence, edit operation sequence, and key identifier sequence to obtain spoken language detection results. It should be noted that the end-to-end model can include, but is not limited to, Encoder-Decoder architectures, etc. This approach, by using an end-to-end model to detect the response speech, target phoneme sequence, edit operation sequence, and key identifier sequence to obtain spoken language detection results, can further reduce the cumulative errors that may occur during spoken language detection compared to step-by-step execution, thus helping to further improve the accuracy of question-and-answer spoken language detection.

[0050] In a specific implementation scenario, the end-to-end model may include a coding network; please refer to relevant documentation. Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of the framework of an embodiment of the end-to-end model. Figure 4This is a schematic diagram of a framework for an embodiment of the encoding network. As shown, the encoding network may include a first convolutional network and a first Transformer network connected in sequence. The first convolutional network is used to convolve the first acoustic features extracted from the response speech to obtain second acoustic features. The first Transformer network is used to encode the second acoustic features to obtain several acoustic features. As shown, the first convolutional network may include M convolutional blocks (convolutional layers and pooling layers as shown in the figure), and the first Transformer network may include N Transformer structures. As shown, the Transformer structures may include sequentially connected multi-head sub-attention, summation and layer normalization, feedforward network, and summation and layer normalization. Specifically, the first convolutional network may include, but is not limited to, VGGNet. By adding convolutional operations before the attention mechanism, high-frequency speech acoustic information can be effectively downsampled to extract low-frequency acoustic feature information, thereby helping to improve model efficiency. In addition, the contextual dependency information of the sequence information is modeled through multi-head sub-attention in the Transformer structure, thereby combining with other parts of the Transformer structure to extract the final acoustic features. It should be noted that, in order to reduce the latency and computational load caused by self-attention, truncated self-attention can be used instead of ordinary self-attention. For details, please refer to the technical details of Transformer; they will not be elaborated upon here.

[0051] In a specific implementation scenario, the end-to-end model may also include a preprocessing module; please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the preprocessing module. As shown in the figure, for each phoneme in the target phoneme sequence, the embedding representation of the phoneme in the target phoneme sequence, the embedding representation of the phoneme in the edit operation sequence, and the embedding representation of the phoneme in the key identifier sequence can be fused to obtain the final embedding representation of the phoneme. Then, the embedding representations of each phoneme in the target phoneme sequence can be used as the input of the decoding network.

[0052] In a specific implementation scenario, the end-to-end model may include a decoding network; please refer to [link / reference needed]. Figure 6 , Figure 6This is a schematic diagram of a framework for an embodiment of the decoding network. As shown, the decoding network may include a second convolutional network and a second Transformer network connected in sequence. The second convolutional network is used to extract features from the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain a first feature sequence. The second Transformer network is used to process several acoustic features and the first feature sequence to obtain a second feature sequence. As shown, the second convolutional network may include M convolutional blocks (convolutional layers and pooling layers as shown in the figure), and the second Transformer network may include N Transformer structures. As shown in the figure, the Transformer structures may include sequentially connected multi-head sub-attention, summation and layer normalization, feedforward network, and summation and layer normalization. Specifically, the first convolutional network may include, but is not limited to, VGGNet. In addition, the second Transformer network has a cross-attention network between the layer specification (“addition and layer specification” in the figure) and the feedforward layer (“feedforward network” in the figure). The output of the encoding network is sent to the cross-attention network, so that several acoustic features and the first feature sequence are processed by the cross-attention network to obtain the second feature sequence. For the specific processing process, please refer to the technical details of the cross-attention network, which will not be repeated here.

[0053] In a specific implementation scenario, an end-to-end model may include a prediction network; please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of a framework for an embodiment of a prediction network. The prediction network is used to perform score detection on a second feature sequence to obtain spoken language detection results. As shown in the figure, the prediction network may include components such as a feedforward network and normalization (e.g., sigmoid). Specifically, the feedforward network can map scores to the second feature sequence to obtain score values ​​for each phoneme in the target phoneme sequence. Further, to eliminate the influence of dimensions between indicators, these score values ​​can be normalized using normalization (e.g., sigmoid). Based on this, numerical statistics can be performed on the normalized score values ​​(e.g., taking the average, minimum, or maximum value) to obtain the spoken language score of the target object.

[0054] The above scheme identifies the target subject's speech in response to a test question, obtaining a first phoneme sequence. The test question has several reference answers, each with a corresponding second phoneme sequence. Based on the matching results of the first phoneme sequence with each of the second phoneme sequences, the second phoneme sequence is selected as the target phoneme sequence. Then, the editing operation sequence for transforming the target phoneme sequence into the first phoneme sequence is obtained, along with a key identifier sequence for the target phoneme sequence. This key identifier sequence contains the first identifier of each phoneme in the target phoneme sequence, indicating whether the phoneme belongs to a keyword marked in the reference answer to which the target phoneme sequence belongs. Detection based on response speech, target phoneme sequences, editing operation sequences, and key identifier sequences yields the spoken language detection results for the target subject. In question-and-answer spoken language detection, the use of more granular sequence information at the phoneme level, compared to directly using the word level, improves detection granularity. Furthermore, the introduction of editing operation sequences, target phoneme sequences, and key identifier sequences, compared to simply concatenating the actual answer with the reference answer, helps to capture the differences between the actual answer and the reference answer, as well as key vocabulary in the reference answer, providing more detailed auxiliary information for question-and-answer spoken language detection. Therefore, it can improve the accuracy of question-and-answer spoken language detection.

[0055] Please see Figure 8 , Figure 8 This is a schematic diagram of the framework of an embodiment of the question-and-answer spoken language detection device 80 of this application. The question-and-answer spoken language detection device 80 includes: an identification module 81, a matching module 82, an acquisition module 83, and a detection module 84. The identification module 81 is used to identify the speech of the target object's answer to a test question and obtain a first phoneme sequence; wherein, the test question has several reference answers, and each reference answer corresponds to a second phoneme sequence; the matching module 82 is used to select the second phoneme sequence as the target phoneme sequence based on the matching results of the first phoneme sequence with each of the second phoneme sequences; the acquisition module 83 is used to acquire the editing operation sequence of the target phoneme sequence to transform into the first phoneme sequence, and acquire the key identifier sequence of the target phoneme sequence; wherein, the key identifier sequence contains the first identifier of each phoneme in the target phoneme sequence, and the first identifier represents whether the phoneme belongs to the keyword marked in the reference answer to which the target phoneme sequence belongs; the detection module 84 is used to perform detection based on the answer speech, the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain the spoken language detection result of the target object.

[0056] The above-described scheme, in the process of question-and-answer oral test, improves the granularity of detection by using more fine-grained sequence information at the phoneme level compared to directly using the word level. Furthermore, by introducing editing operation sequences, target phoneme sequences, and key identifier sequences, it helps to capture the differences between the actual answer and the reference answer, as well as the key vocabulary in the reference answer, compared to simply concatenating the actual answer with the reference answer. This provides more detailed auxiliary information for question-and-answer oral test. Therefore, it can improve the accuracy of question-and-answer oral test.

[0057] In some disclosed embodiments, the edit operation sequence includes a second identifier for each phoneme in the target phoneme sequence, and different second identifiers represent different edit operations.

[0058] In some disclosed embodiments, the matching module 82 includes an edit distance calculation submodule, which is used to take the edit distance between the first phoneme sequence and the second phoneme sequence as the matching result corresponding to the second phoneme sequence; the matching module 82 includes a phoneme sequence selection submodule, which is used to take the second phoneme sequence corresponding to the minimum edit distance as the target phoneme sequence.

[0059] In some disclosed embodiments, the detection module 84 includes a feature sequence extraction submodule, used to extract several acoustic features based on the response speech, and to extract a first feature sequence based on the target phoneme sequence, the editing operation sequence, and the key identifier sequence; wherein, the first feature sequence contains a first feature representation of each phoneme in the target phoneme sequence, and the first feature representation of the phoneme contains at least the following feature information: the standard pronunciation of the phoneme, its importance, and whether the target object pronounces the phoneme; the detection module 84 includes a cross-attention processing submodule, used to process several acoustic features and the first feature sequence based on a cross-attention mechanism to obtain a second feature sequence; wherein, the second feature sequence contains a second feature representation of each phoneme in the target phoneme sequence, and the second feature representation of the phoneme contains at least the following feature information: the target object's oral response level on the phoneme; the detection module 84 includes a scoring detection submodule, used to perform scoring detection based on the second feature sequence to obtain oral detection results.

[0060] In some disclosed embodiments, the cross-attention processing submodule includes a feature transformation unit, used to transform several acoustic features based on the key matrix and value matrix of the cross-attention mechanism to obtain key features and value features, and to transform the first feature sequence based on the query matrix of the cross-attention mechanism to obtain a query feature sequence; the cross-attention processing submodule includes a matrix acquisition unit, used to acquire the attention matrix between the key features and the query feature sequence; the cross-attention processing submodule includes a matrix application unit, used to apply the value features to the attention matrix to obtain a second feature sequence.

[0061] In some disclosed embodiments, the scoring detection submodule includes a score mapping unit for performing score mapping based on the second feature sequence to obtain the score value of each phoneme in the target phoneme sequence; the scoring detection submodule includes a numerical statistics unit for performing numerical statistics based on the score value of each phoneme in the target phoneme sequence to obtain the spoken language score of the target object as the spoken language detection result.

[0062] In some disclosed embodiments, the feature sequence extraction submodule includes an acoustic feature extraction unit for extracting features based on the response speech to obtain first acoustic features of several audio frames; the feature sequence extraction submodule includes an acoustic feature convolution unit for convolution based on the first acoustic features of several audio frames to obtain second acoustic features; and the feature sequence extraction submodule includes a feature encoding unit for encoding features based on the second acoustic features to obtain several acoustic features.

[0063] In some publicly disclosed embodiments, spoken language detection results are obtained by an end-to-end model detecting the response speech, target phoneme sequence, edit operation sequence, and key identifier sequence.

[0064] In some disclosed embodiments, the spoken language detection result is obtained by an end-to-end model, which includes an encoding network, a decoding network, and a prediction network. The encoding network includes a first convolutional network and a first Transformer network connected in sequence. The decoding network includes a second convolutional network and a second Transformer network connected in sequence, and the second Transformer network has a cross-attention network between the layer specification and the feedforward layer of the Transformer structure. The first convolutional network is used to convolve the first acoustic features extracted from the response speech to obtain the second acoustic features. The first Transformer network is used to encode the second acoustic features to obtain several acoustic features. The second convolutional network is used to extract features from the target phoneme sequence, the edit operation sequence, and the key identifier sequence to obtain the first feature sequence. The second Transformer network is used to process the several acoustic features and the first feature sequence to obtain the second feature sequence. The prediction network is used to score and detect the second feature sequence to obtain the spoken language detection result.

[0065] Please see Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the electronic device 90 of this application. The electronic device 90 includes a memory 91 and a processor 92 coupled to each other. The memory 91 stores program instructions, and the processor 92 is used to execute the program instructions to implement the steps in any of the above-described embodiments of the question-and-answer spoken language detection method. Specifically, the electronic device 90 may include, but is not limited to, desktop computers, laptops, servers, mobile phones, tablet computers, etc., and is not limited thereto.

[0066] Specifically, processor 92 controls itself and memory 91 to implement the steps in any of the above-described question-and-answer spoken language detection method embodiments. Processor 92 can also be referred to as a CPU (Central Processing Unit). Processor 92 may be an integrated circuit chip with signal processing capabilities. Processor 92 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 92 can be implemented using integrated circuit chips.

[0067] In the above scheme, the electronic device 90 can implement the steps in any of the above-described question-and-answer oral language detection method embodiments. Specifically, during the question-and-answer oral language detection process, the use of more granular sequence information at the phoneme level, compared to directly using the word level, improves the detection granularity. Furthermore, the introduction of editing operation sequences, target phoneme sequences, and key identifier sequences, compared to directly concatenating the actual answer with the reference answer, helps to capture the differences between the actual answer and the reference answer, as well as the key vocabulary in the reference answer, thus providing more detailed auxiliary information for question-and-answer oral language detection. Therefore, the accuracy of question-and-answer oral language detection can be improved.

[0068] Please see Figure 10 , Figure 10 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 100 of this application. The computer-readable storage medium 100 stores program instructions 101 that can be executed by a processor. The program instructions 101 are used to implement the steps in any of the above embodiments of the question-and-answer spoken language detection method.

[0069] The above-described scheme, implemented by the computer-readable storage medium 100, allows for the execution of the steps in any of the above-described question-and-answer oral communication detection method embodiments. Specifically, during the question-and-answer oral communication detection process, the use of more granular sequence information at the phoneme level, compared to directly using the word level, improves the detection granularity. Furthermore, the introduction of editing operation sequences, target phoneme sequences, and key identifier sequences, compared to directly concatenating the actual answer with the reference answer, helps capture the differences between the actual answer and the reference answer, as well as key vocabulary in the reference answer, thus providing more detailed auxiliary information for question-and-answer oral communication detection. Therefore, the accuracy of question-and-answer oral communication detection can be improved.

[0070] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0071] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for detecting a question-answer speech, characterized by, include: The target object's speech in response to a test question is identified to obtain a first phoneme sequence; wherein, the test question has several reference answers, and each reference answer corresponds to a second phoneme sequence; Based on the matching results of the first phoneme sequence with each of the second phoneme sequences, the second phoneme sequence is selected as the target phoneme sequence. Obtain the editing operation sequence that transforms the target phoneme sequence into the first phoneme sequence, and obtain the key identifier sequence of the target phoneme sequence; wherein, the key identifier sequence contains the first identifier of each phoneme in the target phoneme sequence, and the first identifier represents whether the phoneme belongs to the keyword marked in the reference answer to which the target phoneme sequence belongs; Based on the response speech, the target phoneme sequence, the editing operation sequence, and the key identifier sequence, the spoken language detection result of the target object is obtained.

2. The method according to claim 1, characterized in that, The editing operation sequence includes a second identifier for each phoneme in the target phoneme sequence, and different second identifiers represent different editing operations.

3. The method according to claim 1 or 2, characterized in that, The step of selecting a second phoneme sequence as the target phoneme sequence based on the matching results of the first phoneme sequence with each of the second phoneme sequences includes: The edit distance between the first phoneme sequence and the second phoneme sequence is used as the matching result corresponding to the second phoneme sequence; The second phoneme sequence corresponding to the minimum edit distance is taken as the target phoneme sequence.

4. The method according to claim 1, characterized in that, The detection based on the response speech, the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain the spoken language detection result of the target object includes: Based on the response speech, several acoustic features are extracted, and based on the target phoneme sequence, the editing operation sequence, and the key identifier sequence, a first feature sequence is extracted; wherein, the first feature sequence contains the first feature representation of each phoneme in the target phoneme sequence, and the first feature representation of the phoneme contains at least the following feature information: the standard pronunciation of the phoneme, its importance, and whether the target object pronounces the phoneme; The acoustic features and the first feature sequence are processed based on a cross-attention mechanism to obtain a second feature sequence; wherein, the second feature sequence contains the second feature representation of each phoneme in the target phoneme sequence, and the second feature representation of the phoneme contains at least the following feature information: the oral response level of the target object on the phoneme; The oral detection result is obtained by performing a scoring detection based on the second feature sequence.

5. The method according to claim 4, characterized in that, The second feature sequence is obtained by processing the several acoustic features and the first feature sequence based on the cross-attention mechanism, including: The key matrix and value matrix based on the cross-attention mechanism are used to transform the several acoustic features to obtain key features and value features, respectively. The query matrix based on the cross-attention mechanism is used to transform the first feature sequence to obtain the query feature sequence. Obtain the attention matrix between the key features and the query feature sequence; The value features are applied to the attention matrix to obtain the second feature sequence.

6. The method according to claim 4, characterized in that, The step of scoring and detecting based on the second feature sequence to obtain the spoken language detection result includes: Based on the second feature sequence, a score mapping is performed to obtain the score value of each phoneme in the target phoneme sequence; Numerical statistics are performed on the score values ​​of each phoneme in the target phoneme sequence to obtain the spoken language score of the target object as the spoken language detection result.

7. The method according to claim 4, characterized in that, Based on the response speech, several acoustic features are extracted, including: Based on the spoken response, feature extraction is performed to obtain the first acoustic features of several audio frames. The second acoustic feature is obtained by convolving the first acoustic features based on the aforementioned audio frames; Based on the second acoustic feature, feature encoding is performed to obtain the plurality of acoustic features.

8. The method according to claim 1, characterized in that, The spoken language detection result is obtained by an end-to-end model detecting the response speech, the target phoneme sequence, the editing operation sequence, and the key identifier sequence.

9. The method according to claim 8, characterized in that, The spoken language detection result is obtained by an end-to-end model, which includes an encoding network, a decoding network, and a prediction network. The encoding network includes a first convolutional network and a first Transformer network connected in sequence. The decoding network includes a second convolutional network and a second Transformer network connected in sequence. The second Transformer network has a cross-attention network between the layer specification and the feedforward layer of the Transformer structure. The encoding network outputs to the cross-attention network. Wherein, the first convolutional network is used to convolve the first acoustic features extracted from the response speech to obtain the second acoustic features; the first Transformer network is used to encode the second acoustic features to obtain several acoustic features; the second convolutional network is used to extract features from the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain the first feature sequence; the second Transformer network is used to process the several acoustic features and the first feature sequence to obtain the second feature sequence; and the prediction network is used to score and detect the second feature sequence to obtain the spoken language detection result.

10. A question-and-answer oral language detection device, characterized in that, include: The recognition module is used to recognize the target object's speech in response to a test question and obtain a first phoneme sequence; wherein, the test question has several reference answers, and each reference answer corresponds to a second phoneme sequence; The matching module is used to select the second phoneme sequence as the target phoneme sequence based on the matching results of the first phoneme sequence with each of the second phoneme sequences; The acquisition module is used to acquire the editing operation sequence of transforming the target phoneme sequence into the first phoneme sequence, and to acquire the key identifier sequence of the target phoneme sequence; wherein, the key identifier sequence contains the first identifier of each phoneme in the target phoneme sequence, and the first identifier indicates whether the phoneme belongs to the keywords marked in the reference answer to which the target phoneme sequence belongs; The detection module is used to perform detection based on the response speech, the target phoneme sequence, the editing operation sequence, and the key identifier sequence to obtain the spoken language detection result of the target object.

11. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores program instructions and the processor executes the program instructions to implement the question-and-answer spoken language detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the question-and-answer spoken language detection method according to any one of claims 1 to 9.

Images