Audio detection method, apparatus, device, storage medium, and product

Abstract

The application discloses an audio detection method, device, equipment, storage medium and product, and belongs to the computer technical field. The method comprises the following steps: obtaining an answering audio corresponding to a first question; inputting the answering audio into an audio detection model corresponding to the first question to perform audio detection processing, and obtaining detection information corresponding to the answering audio. In the technical scheme provided by the embodiment of the application, sample answering audio corresponding to a second question similar to the first question and a label corresponding to the sample answering audio are used as model training data corresponding to the first question, so that automatic acquisition of the first question training data and completion of corresponding model training can be realized, audio detection of the first question answering audio can be realized through the trained audio detection model, and the detection cost and detection complexity of detecting the first question answering audio are effectively reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio detection method, apparatus, device, storage medium, and product. Background Technology

[0002] Oral exams measure a language learner's ability to communicate orally using the relevant language. Each oral exam consists of multiple test papers, which are assigned to different students to complete.

[0003] In this technology, for each exam question, a selection of students' audio responses are sent to experts for annotation. The oral exam engine then trains a detection model for each question based on the annotated data, thus constructing a separate detection model for each question and scoring the students' audio responses for each question. For supplementary exam papers in multiple sets of tests, which often have fewer participants, these papers are not annotated by experts and are scored manually.

[0004] Among related technologies, detecting student answer audio is costly and complex. Summary of the Invention

[0005] This application provides an audio detection method, apparatus, device, storage medium, and product that can reduce the detection cost and complexity of audio responses.

[0006] According to one aspect of the embodiments of this application, an audio detection method is provided, the method comprising:

[0007] Get the audio of the answer to the first question;

[0008] The audio of the answer is input into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the audio of the answer;

[0009] The audio detection model is a machine learning model trained on the sample answer audio corresponding to the second question and the label information corresponding to the sample answer audio. The question similarity between the second question and the first question meets the preset question similarity conditions.

[0010] In some possible designs, the training process of the audio detection model includes:

[0011] If the set of audio responses to the first question does not meet the preset audio quantity condition, obtain the sample audio responses and the tag information;

[0012] Based on the sample answer audio and the label information, the audio detection model corresponding to the first question is transferred to train, and the audio detection model is obtained.

[0013] In some possible designs, the training process of the audio detection model also includes:

[0014] Obtain the target question set corresponding to the first question;

[0015] Obtain the question information corresponding to the first question, and the question information corresponding to each question in the target question set;

[0016] The second question is obtained by performing similarity matching between the question information corresponding to the first question and the question information corresponding to each of the above questions.

[0017] In some possible designs, the step of performing similarity matching processing based on the question information corresponding to the first question and the question information corresponding to each of the other questions to obtain the second question includes:

[0018] Based on the question information corresponding to the first question and the question information corresponding to each of the questions, clustering is performed on the first question and the questions in the target question set to obtain the target clustering result corresponding to the first question. The target clustering result includes the first question and the second question.

[0019] In some possible designs, the step of performing transfer training on the audio detection model corresponding to the first question based on the sample answer audio and the label information to obtain the audio detection model includes:

[0020] The sample response audio is processed by feature extraction to obtain the sample response audio features corresponding to the sample response audio;

[0021] Obtain the second question rule features corresponding to the second question; the second question rule represents the answer requirement information corresponding to the second question.

[0022] The second question rule features and the sample answer audio features are subjected to feature fusion processing to obtain the sample fusion features corresponding to the sample answer audio;

[0023] The sample fusion features are input into the audio detection model to be trained for audio detection processing to obtain the sample detection information corresponding to the sample response audio.

[0024] Based on the sample detection information and the label information, the parameters of the audio detection model to be trained are adjusted to obtain the audio detection model.

[0025] In some possible designs, the sample fusion features include a target number of feature dimensions. After performing feature fusion processing on the second question rule features and the sample answer audio features to obtain the sample fusion features corresponding to the sample answer audio, the process further includes:

[0026] Iterate through the target's number of feature dimensions;

[0027] When traversing through the target dimension among the target number of feature dimensions, determine the feature value of the sample fusion feature corresponding to the target dimension;

[0028] The feature values ​​are standardized to obtain the transformed feature values ​​of the sample fusion features corresponding to the target dimension.

[0029] Based on the transformed feature values ​​corresponding to each feature dimension of the sample fusion features, the transformed sample fusion features corresponding to the sample response audio are obtained;

[0030] The step of inputting the sample fusion features into the audio detection model to be trained for audio detection processing to obtain sample detection information corresponding to the sample response audio includes:

[0031] The transformed sample fusion features are input into the audio detection model to be trained for audio detection processing to obtain the sample detection information.

[0032] In some possible designs, the standardization of the feature values ​​to obtain the transformed feature values ​​of the sample fusion features corresponding to the target dimension includes:

[0033] Determine the mean and standard deviation of the features corresponding to the eigenvalues;

[0034] Based on the mean and standard deviation of the features, the feature values ​​are standardized to obtain the transformed feature values.

[0035] In some possible designs, the method further includes:

[0036] The response audio is subjected to feature extraction processing to obtain the response audio features corresponding to the response audio;

[0037] Obtain the first question rule feature corresponding to the first question, where the first question rule feature represents the answer requirement information corresponding to the first question;

[0038] The first question rule features and the answer audio features are subjected to feature fusion processing to obtain the fused features corresponding to the answer audio;

[0039] The step of inputting the answer audio into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the answer audio includes:

[0040] The fusion features are input into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information.

[0041] According to one aspect of the embodiments of this application, an audio detection apparatus is provided, the apparatus comprising:

[0042] The audio acquisition module is used to acquire the audio of the answer to the first question;

[0043] The audio detection module is used to input the answer audio into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the answer audio;

[0044] The audio detection model is a machine learning model trained on the sample answer audio corresponding to the second question and the label information corresponding to the sample answer audio. The question similarity between the second question and the first question meets the preset question similarity conditions.

[0045] In some possible designs, the audio detection device includes a training device corresponding to the audio detection model, the training device comprising:

[0046] The sample acquisition module is used to acquire the sample answer audio and the tag information when the set of answer audio corresponding to the first question does not meet the preset audio quantity condition;

[0047] The model training module is used to perform transfer training on the audio detection model to be trained corresponding to the first question based on the sample answer audio and the label information, so as to obtain the audio detection model.

[0048] In some possible designs, the training device further includes:

[0049] The question set acquisition module is used to acquire the target question set corresponding to the first question;

[0050] The question information acquisition module is used to acquire the question information corresponding to the first question, as well as the question information corresponding to each question in the target question set;

[0051] The similar question determination module is used to perform similarity matching processing based on the question information corresponding to the first question and the question information corresponding to each of the questions to obtain the second question.

[0052] In some possible designs, the similar question determination module includes:

[0053] The question clustering unit is used to perform clustering processing on the first question and the questions in the target question set based on the question information corresponding to the first question and the question information corresponding to each of the questions, so as to obtain the target clustering result corresponding to the first question, wherein the target clustering result includes the first question and the second question.

[0054] In some possible designs, the model training module includes:

[0055] An audio feature extraction unit is used to perform feature extraction processing on the sample response audio to obtain the sample response audio features corresponding to the sample response audio;

[0056] The question feature acquisition unit is used to acquire the second question rule features corresponding to the second question, whereby the second question rule represents the answer requirement information corresponding to the second question.

[0057] The feature fusion determination unit is used to perform feature fusion processing on the second question rule features and the sample answer audio features to obtain the sample fusion features corresponding to the sample answer audio.

[0058] The fusion feature detection unit is used to input the sample fusion features into the audio detection model to be trained for audio detection processing, and obtain the sample detection information corresponding to the sample response audio.

[0059] The model parameter adjustment unit is used to adjust the parameters of the audio detection model to be trained based on the sample detection information and the label information, so as to obtain the audio detection model.

[0060] In some possible designs, the sample fusion features include a target number of feature dimensions, and the model training module further includes:

[0061] A feature dimension traversal unit is used to traverse the target number of feature dimensions;

[0062] The feature value determination unit is used to determine the feature value of the sample fusion feature corresponding to the target dimension when traversing the target number of feature dimensions;

[0063] The feature value transformation unit is used to standardize the feature values ​​to obtain the transformed feature values ​​of the sample fusion features corresponding to the target dimension.

[0064] The fusion feature transformation unit is used to obtain the transformed sample fusion feature corresponding to the sample response audio based on the transformed feature value corresponding to each feature dimension of the sample fusion feature.

[0065] The fusion feature detection unit is also used to input the transformed sample fusion features into the audio detection model to be trained for audio detection processing to obtain the sample detection information.

[0066] In some possible designs, the eigenvalue transformation unit includes:

[0067] The eigenvalue parameter determination subunit is used to determine the feature mean and standard deviation corresponding to the eigenvalue;

[0068] The eigenvalue transformation subunit is used to standardize the eigenvalues ​​based on the eigenvalue mean and the standard deviation to obtain the transformed eigenvalues.

[0069] In some possible designs, the audio detection device further includes:

[0070] An audio feature extraction unit is used to perform feature extraction processing on the response audio to obtain the response audio features corresponding to the response audio;

[0071] The question feature acquisition unit is used to acquire the first question rule feature corresponding to the first question, wherein the first question rule feature represents the answer requirement information corresponding to the first question;

[0072] The feature fusion determination unit is used to perform feature fusion processing on the first question rule features and the answer audio features to obtain the fusion features corresponding to the answer audio;

[0073] The audio detection module is specifically used to input the fused features into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information.

[0074] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described audio detection method.

[0075] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, the storage medium storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by a processor to implement the above-described audio detection method.

[0076] According to one aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the above-described audio detection method.

[0077] The technical solution provided in this application can bring the following beneficial effects:

[0078] Using the sample audio of the second question, which is similar to the first question, and its corresponding labels as the model training data for the first question, the training data for the first question can be automatically acquired and the corresponding model training can be completed. The trained audio detection model can then be used to detect the audio of the answer to the first question, effectively reducing the detection cost and complexity of the audio of the answer to the first question. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 This is a schematic diagram of an application runtime environment provided in one embodiment of this application;

[0081] Figure 2 An exemplary schematic diagram of an audio detection interface is shown;

[0082] Figure 3 An example is shown in the overall flowchart for detecting audio responses in an oral exam;

[0083] Figure 4 This is a flowchart of an audio detection method provided in one embodiment of this application. Figure 1 ;

[0084] Figure 5 An exemplary diagram illustrates the technical architecture of an audio detection scheme for oral exams.

[0085] Figure 6 This is a flowchart of an audio detection method provided in one embodiment of this application. Figure 2 ;

[0086] Figure 7 This is a flowchart of a training method for an audio detection model provided in one embodiment of this application. Figure 1 ;

[0087] Figure 8 This is a flowchart of a training method for an audio detection model provided in one embodiment of this application. Figure 2 ;

[0088] Figure 9 This example illustrates a flowchart of a process for filtering labeled data by identifying similar questions of the same type.

[0089] Figure 10 This is a flowchart of a training method for an audio detection model provided in one embodiment of this application. Figure 3 ;

[0090] Figure 11 This is a flowchart of a training method for an audio detection model provided in one embodiment of this application. Figure 4 ;

[0091] Figure 12(a) illustrates a statistical diagram of test results based on rapid response questions;

[0092] Figure 12(b) illustrates, for example, a statistical diagram of test results based on picture description questions;

[0093] Figure 12(c) illustrates, for example, a statistical diagram of test results based on oral composition topics;

[0094] Figure 13 This is a block diagram of an audio detection device provided in one embodiment of this application;

[0095] Figure 14 This is a structural block diagram of a computer device provided in one embodiment of this application. Figure 1 ;

[0096] Figure 15 This is a structural block diagram of a computer device provided in another embodiment of this application. Figure 2 . Detailed Implementation

[0097] The audio detection method provided in this application involves artificial intelligence technology, which will be briefly described below to facilitate understanding by those skilled in the art.

[0098] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0099] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0100] Key technologies in speech technology include Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech emerging as one of the most promising methods.

[0101] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0102] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0103] With the research and advancement of artificial intelligence (AI) technology, it is being studied and applied in numerous fields, such as smart education, smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. For instance, in smart education, AI technology can be used to automatically detect student answers to exam questions, enabling automated assessment. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role.

[0104] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0105] Please refer to Figure 1 This diagram illustrates an application runtime environment provided in one embodiment of this application. The application runtime environment may include: terminal 10 and server 20.

[0106] Terminal 10 includes, but is not limited to, electronic devices such as mobile phones, computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, game consoles, e-book readers, multimedia playback devices, and wearable devices. Application clients can be installed on terminal 10.

[0107] In this embodiment, the application described above can be any application capable of providing audio detection services. Typically, this application is an educational application. Of course, other types of applications besides educational applications can also provide audio detection services. For example, exam applications, video applications, news applications, social applications, interactive entertainment applications, browser applications, shopping applications, content sharing applications, virtual reality (VR) applications, augmented reality (AR) applications, etc., are not limited in this embodiment. Furthermore, different applications may contain different audio content and have different functions, which can be pre-configured according to actual needs, and are not limited in this embodiment. Optionally, the terminal 10 runs a client of the aforementioned application.

[0108] In one example, such as Figure 2 As shown, it exemplifies a schematic diagram of an audio detection interface. Figure 2 The audio detection interface 20 shown can be the client interface corresponding to the aforementioned application. This audio detection interface 20 can display the question 21 and a recording option 22. In response to the selection command for recording option 22, the device can start recording the answer audio and update recording option 22 to recording end option 23 in the audio detection interface 20. After the respondent finishes answering the question, they can select recording end option 23, and the device, in response to the selection command for recording end option 23, can end the recording of the answer audio, obtaining the complete answer audio. After detecting the answer audio, the audio detection interface 20 will display score display information 24, including the score obtained by the respondent for answering question 21.

[0109] Server 20 is used to provide background services for the client of the application in terminal 10. For example, server 20 can be the background server of the aforementioned application. For instance, terminal 10 can send the audio of the answer to the target question to server 20, server 20 determines the audio detection model corresponding to the target question, and performs detection processing on the answer audio based on the audio detection model to generate a detection score, and then sends the detection score generated by the detection processing to the terminal.

[0110] The audio detection method provided in this application is typically used in oral communication scenarios, and can be applied to open-ended expression questions in oral exams. Based on the audio detection method provided in this application, the audio of the answers to the open-ended expression questions can be detected and scored.

[0111] In one example, such as Figure 3 As shown, it exemplifies an overall flowchart for detecting audio responses in an oral exam. Figure 3 In the process, test taker 31, such as a student, answers the exam questions displayed on client 32. Client 32 records the test taker's answers, obtaining the audio of the answer, and sends the audio along with the questions to server 33. Server 33 determines the appropriate detection model based on the received questions. Specifically, server 33 queries the question database to determine whether the question belongs to a test paper with many or few test takers, and sends the audio of the answer to the corresponding model. If the question belongs to a test paper with many test takers (more than a preset threshold), the audio of the answer is processed by the question model 34 corresponding to that question to obtain an evaluation score. In this case, question model 34 is a machine learning model trained based on the labeled data corresponding to the question itself. If the question belongs to a test paper with few test takers (less than or equal to a preset threshold), the audio of the answer is processed by the question transfer model 35 corresponding to that question to obtain an evaluation score. In this case, the question transfer model 35 uses calibration data from other questions of the same type and similar to the question in the same exam (these questions often have a large number of respondents and are sampled and labeled by experts) to perform question type transfer learning, resulting in a more accurate detection model. This enables scoring of questions lacking or without calibration data. After obtaining the above evaluation score, the server 33 sends it to the client 32, which then displays it to the respondent 31.

[0112] Figure 1 The server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the server 20 can simultaneously provide background services for applications in multiple terminals 10.

[0113] Optionally, terminal 10 and server 20 can communicate with each other via network 30. Terminal 10 and server 20 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0114] Please refer to Figure 4 It illustrates the flow of an audio detection method provided in one embodiment of this application. Figure 1This method can be applied to computer devices, which refer to electronic devices with data computing and processing capabilities. For example, the entity executing each step can be... Figure 1 The application runtime environment shown is a server 20 or a terminal 10. The method may include the following steps (410-420).

[0115] Step 410: Obtain the audio of the answer to the first question.

[0116] Optionally, the set of answer audios corresponding to the first question does not meet the preset audio quantity condition. Optionally, the preset audio quantity condition includes, but is not limited to: the number of answer audios in the answer audio set is greater than a first quantity threshold; the number of answer audios with corresponding label information in the answer audio set is greater than a second quantity threshold. The first quantity threshold represents the minimum number of answer audios set for model training. The second quantity threshold represents the minimum number of answer audios with corresponding label information required for model training.

[0117] In practical application scenarios, such as oral exams, the aforementioned first question includes, but is not limited to: exam questions in the first test paper, questions without corresponding labeled data, and questions with labeled data less than or equal to a preset labeled data threshold. Optionally, the aforementioned first test paper includes, but is not limited to: test papers with fewer than or equal to a preset number of test takers, test papers that have not been released, and test papers with no labeled data or labeled data less than or equal to a preset labeled data threshold. The aforementioned labeled data refers to the audio responses with corresponding labeled scores. The aforementioned labeled scores can be determined by oral exam review experts based on the quality of the corresponding audio responses.

[0118] The audio recording above is the audio of the respondent's answer to the first question. The device can record audio during the respondent's answering process to obtain the audio recording above. For specific interaction procedures, please refer to [link / reference]. Figure 2 .

[0119] The audio detection method provided in this application embodiment can be executed by a terminal or a server. When executed by a terminal, the aforementioned response audio can be audio recorded by a recording device connected to the terminal or a recording device located inside the terminal, and the terminal can directly retrieve and use it from local storage. When executed by a server, the aforementioned response audio can be sent by the terminal, that is, the server receives the response audio sent by the terminal.

[0120] Step 420: Input the answer audio into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the answer audio.

[0121] Optionally, the audio detection model is a machine learning model trained based on the sample answer audio corresponding to the second question and the label information corresponding to the sample answer audio. Optionally, the question similarity between the second question and the first question meets a preset question similarity condition.

[0122] Optionally, if the set of answer audios corresponding to the first question does not include answer audios with corresponding label information, such as if the first question is a question without corresponding labeled data, the training data corresponding to the above audio detection model includes sample answer audios corresponding to the second question and label information corresponding to the sample answer audios. The above audio detection model can be obtained by training the preset machine learning model based on the sample answer audios corresponding to the second question and the label information corresponding to the sample answer audios.

[0123] Optionally, if the number of answer audios with corresponding label information in the answer audio set corresponding to the first question is less than or equal to the aforementioned second quantity threshold, such as if the first question is a question with less or equal to a preset label data threshold (a question with few label data), the training data corresponding to the aforementioned audio detection model may only include the label data corresponding to the second question, or it may include both the label data corresponding to the first question and the label data corresponding to the second question. Optionally, the label data corresponding to the second question includes the sample answer audio corresponding to the second question and the label information corresponding to that sample answer audio. Optionally, the label data corresponding to the first question includes the answer audios with corresponding label information in the answer audio set corresponding to the first question, and the label information corresponding to these answer audios. Accordingly, the aforementioned audio detection model may be a machine learning model trained based on the label data corresponding to the second question, or it may be a machine learning model trained based on fused label data. The aforementioned fused label data is the label data obtained by fusing the label data corresponding to the first question and the second question respectively, including the label data corresponding to the first question and the second question respectively.

[0124] Optionally, the preset question similarity conditions include, but are not limited to: the question similarity between the second question and the first question is greater than or equal to a preset similarity threshold, and the second question and the first question correspond to the same cluster result.

[0125] In one possible implementation, question similarity is measured from the perspectives of question difficulty (e.g., easy, medium, hard) and question assessment rules (semantic assessment, pronunciation assessment). Optionally, for each question, the question attribute information corresponding to each question on preset question attributes is obtained, and the feature vector corresponding to each question is determined based on the question information corresponding to each question on the preset attributes.

[0126] Optionally, the aforementioned preset attributes include the aforementioned question difficulty attribute, question assessment rule attribute, question scoring rule attribute, etc.; correspondingly, the question attribute information includes the question difficulty information (such as difficulty level) corresponding to the aforementioned question difficulty attribute, the question assessment rule information corresponding to the question assessment rule attribute, and the question scoring rule information corresponding to the question scoring rule attribute.

[0127] There are several ways to determine the similarity between two questions. One possible method is to obtain the feature vectors corresponding to the first question and the candidate questions; determine the feature distance between the feature vectors of the first question and the candidate questions; and then determine the question similarity between the first question and the candidate questions based on this feature distance. Here, the feature distance is negatively correlated with question similarity; the closer the feature distance, the higher the question similarity. Optionally, the second question is the candidate question whose corresponding question similarity is greater than or equal to a preset similarity threshold.

[0128] Another possible approach is to perform clustering based on the feature vectors corresponding to the first question and the candidate questions, obtaining various clustering results; correspondingly, the second question is the question that belongs to the same cluster as the first question. The above clustering can employ the k-means algorithm or other clustering algorithms. Optionally, the above candidate questions are questions of the same type as the first question.

[0129] Optionally, the second question mentioned above is a question in the second test paper that is similar to the first question mentioned above. Optionally, the second question mentioned above is a question in the second test paper that has the same question type as the first question mentioned above and whose question similarity is greater than or equal to a preset similarity threshold. Optionally, the second test paper includes, but is not limited to: test papers with more than a preset threshold for the number of test takers, test papers that have already been released, and test papers with more than a preset threshold for labeled data. The labeled data mentioned above refers to the audio of the answers corresponding to labeled scores.

[0130] Optionally, the sample answer audio for the second question refers to the labeled answer audio within the audio samples for the second question. The label information includes annotation content, such as expert scores and labeled scores. The labeled data for the second question includes the sample answer audio and its corresponding label information. This means that the audio detection model for the first question is a machine learning model trained using the labeled data for the second question as training data through transfer learning.

[0131] In one example, such as Figure 5The diagram illustrates the technical architecture of an audio response detection scheme for oral exams. In an oral exam, a single exam may have multiple test papers, which can be categorized into multi-person and multi-person test papers based on the number of test takers and a threshold. For multi-person test papers, audio recordings of student responses for each question are selected and submitted to experts for annotation (calibration process). The oral exam engine trains on each question based on the calibration data (calibration training), constructing question models for different questions. For multi-person test papers, experts generally do not annotate. For questions lacking calibration data in multi-person test papers, similar questions can be identified from the same question type in multi-person test papers. Annotated data for these similar questions can then be extracted from the annotation data of the same question type stored in the expert annotation database. Based on this annotation data, transfer learning is performed on these partially or uncalibrated questions to obtain a question type transfer model, enabling scoring of partially or uncalibrated questions. The oral exam scenario described above can be expanded from the same exam setting to a regional exam setting. Considering that the exams and assessment points in a region are almost identical, existing exam questions in a certain region can be filtered, and the labeled data of the filtered questions can be used to learn the question type transfer algorithm for similar questions of the same question type in the new exam.

[0132] In an exemplary embodiment, such as Figure 6 As shown, the above audio detection method may further include the following steps (430-450). Figure 6 The flowchart of an audio detection method provided in one embodiment of this application is shown. Figure 2 .

[0133] Step 430: Perform feature extraction processing on the answer audio to obtain the answer audio features corresponding to the answer audio.

[0134] Optionally, the audio of the response is subjected to automatic speech recognition processing to obtain the audio text corresponding to the audio of the response; the audio text is subjected to text feature extraction processing to obtain the text features corresponding to the audio text; the audio of the response is subjected to audio feature extraction processing to obtain the audio features corresponding to the audio of the response; and the text features and audio features are fused (e.g., spliced) to obtain the audio features of the response.

[0135] In an exemplary embodiment, the process of performing text feature extraction on the above-mentioned audio text to obtain the text features corresponding to the above-mentioned audio text is as follows:

[0136] Optionally, semantic feature extraction processing is performed on the audio text to obtain the semantic features corresponding to the audio text. Optionally, the semantic features include the topic features and TF-IDF (Term Frequency–Inverse Document Frequency) features corresponding to the audio text. In one possible implementation, the audio text is input into some pre-trained models, such as BERT (Bidirectional Encoder Representation from Transformers) or word2vec (word to vector) to obtain the semantic representation data corresponding to each corpus unit (e.g., word) in each sentence of the audio text. For sentences in the audio text, the sentences can be represented as [H(word1), H(word2), ..., H(wordN)]. Wherein, H(word1) represents the semantic representation vector corresponding to the first word in the sentence (i.e., the semantic representation data mentioned above), H(word2) represents the semantic representation vector corresponding to the second word in the sentence, and H(wordN) represents the semantic representation vector corresponding to the Nth word in the sentence. N is the total number of words in the sentence, and N is greater than or equal to 1. Optionally, the semantic representation vectors corresponding to the above N words can be averaged to obtain the semantic representation vector H corresponding to the sentence. Optionally, H = (H(word1) + H(word2) + ... + H(wordN)) / N.

[0137] Optionally, pragmatic feature extraction processing is performed on the aforementioned audio text to obtain the pragmatic features corresponding to the audio text. Optionally, the pragmatic features include word diversity data, sentence diversity data, and grammatical accuracy data corresponding to the audio text. The aforementioned word diversity data represents the diversity of words in the audio text, the aforementioned sentence diversity data represents the diversity of sentence structures in the audio text, and the aforementioned grammatical accuracy data represents the grammatical accuracy of each sentence in the audio text. Optionally, the aforementioned grammatical accuracy data is feature data obtained by performing grammatical accuracy detection processing based on a preset grammar detection language model.

[0138] Optionally, keyword feature extraction is performed on the audio text to obtain the keyword features corresponding to the audio text. Optionally, keywords from the standard answer to the first question and keywords from the audio text are extracted, and the keywords from the standard answer and the keywords from the audio text are compared to obtain the keyword features corresponding to the audio text. Optionally, the keyword features represent the degree of matching between the keywords in the audio text and the standard answer. Optionally, the keyword features include data such as precision and recall corresponding to the keywords in the audio text.

[0139] Optionally, the above audio text is subjected to fluency feature extraction processing to obtain the fluency features corresponding to the audio text. The fluency features represent the fluency of the response determined based on the audio text. Optionally, the fluency features include statistical information on the proportion of non-fluency components in the audio text. The aforementioned non-fluency components refer to components that disrupt the coherence of sentences, such as pauses, reduplicated words, and interjections.

[0140] Optionally, based on at least one of the above semantic features, pragmatic features, keyword features, and fluency features, the text features corresponding to the above audio text are determined.

[0141] In an exemplary embodiment, the process of performing audio feature extraction on the response audio to obtain the audio features corresponding to the aforementioned response audio is as follows:

[0142] Optionally, feature extraction based on pronunciation accuracy is performed on the response audio to obtain pronunciation accuracy features corresponding to the response audio. The pronunciation accuracy features characterize the pronunciation accuracy corresponding to the response audio. Optionally, the pronunciation accuracy features include the confidence level of each audio unit in the response audio obtained based on speech recognition relative to each phoneme, or each word, or each sentence.

[0143] Optionally, the response audio is processed by feature extraction based on pronunciation fluency to obtain the pronunciation fluency features corresponding to the response audio. The pronunciation fluency features characterize the pronunciation fluency corresponding to the response audio. Optionally, the pronunciation fluency features include speech rate features during the pronunciation process and features based on duration statistics (such as the average duration of pronunciation segments and the average pause duration between pronunciation segments). The above speech rate features can be calculated according to the number of phonemes or words read per second, that is, the speech rate features are determined based on the number of phonemes or words appearing within a preset time. Optionally, the speech rate features can be determined according to the following formula (1).

[0144]

[0145] Where, N phone The number of phonemes pronounced within time t.

[0146] Optionally, the response audio is subjected to feature extraction based on pronunciation prosodicity to obtain the pronunciation prosodicity features corresponding to the response audio. The pronunciation prosodicity features represent the pronunciation prosodicity corresponding to the response audio. Optionally, the pronunciation prosodicity features include detection data on pronunciation rhythm, detection data on word stress in sentences (used to evaluate the correctness of word stress pronunciation), detection data on sentence boundary intonation, etc.

[0147] Optionally, the audio features corresponding to the answer audio can be obtained based on at least one of the pronunciation accuracy features, pronunciation fluency features, and pronunciation prosody features.

[0148] Step 440: Obtain the first question rule features corresponding to the first question.

[0149] Optionally, the first question rule feature represents the answer requirement information corresponding to the first question. Optionally, the answer requirement information includes information such as the minimum total number of statements and the number of answer points. Optionally, the above answer requirement information is requirement information associated with the question's scoring points. Optionally, the answer requirement information corresponding to the first question is obtained; based on the above question requirement information, the above first question rule feature is generated.

[0150] In oral exams, questions may have specific rules to assess the test taker's speaking ability. For example, there might be a requirement to answer with at least three sentences (total sentence count requirement), or a score of 0 if the number of answer points is less than a certain threshold. These rules are compiled into corresponding features and used to train the detection model. This allows the model to be applied in conjunction with these question-specific features, improving its robustness.

[0151] Step 450: Perform feature fusion processing on the first question rule features and the answer audio features to obtain the fused features corresponding to the answer audio.

[0152] Optionally, the aforementioned audio features of the response are features generated by fusing the text features corresponding to the audio text and the audio features corresponding to the response audio. In this case, fusing the aforementioned first question rule features with the response audio features is equivalent to fusing the text features corresponding to the audio text, the audio features corresponding to the response audio, and the first rule features.

[0153] Accordingly, such as Figure 6 As shown, step 420 above can be replaced by step 421 below.

[0154] Step 421: Input the fused features into the audio detection model corresponding to the first question for audio detection processing to obtain detection information.

[0155] By fusing the features described above, the fused features can simultaneously include text modality features, audio modality features, and question requirement information. Detection based on these fused features yields robust detection results.

[0156] Optionally, the above detection information includes the detection score, i.e., the score corresponding to the audio of the response.

[0157] In an exemplary embodiment, such as Figure 7 As shown, the training process of the above audio detection model may include the following steps (710-720). Figure 7 The flowchart illustrates a training method for an audio detection model provided in one embodiment of this application. Figure 1 .

[0158] Step 710: If the set of audio responses to the first question does not meet the preset audio quantity condition, obtain sample audio responses and tag information.

[0159] The preset audio quantity conditions mentioned above have already been explained in the previous text and will not be repeated here. The device can determine whether transfer training is needed by judging whether the set of answer audios corresponding to the question meets the preset audio quantity conditions. It can accurately and automatically realize the transfer training of the question detection model, thereby improving the model training efficiency and audio detection efficiency.

[0160] In an exemplary embodiment, such as Figure 8 As shown, the training process of the above audio detection model also includes the following steps (730-750). Figure 8 The flowchart illustrates a training method for an audio detection model provided in one embodiment of this application. Figure 2 .

[0161] Step 730: Obtain the target question set corresponding to the first question.

[0162] Optionally, the target question set includes at least one candidate question with the same question type as the first question. Optionally, the question type information corresponding to the first question is determined; questions in the question set corresponding to the above question type information are determined as candidate questions, and the target question set is obtained based on the above candidate questions. Optionally, the question type corresponding to the first question includes free expression question types. Optionally, free expression question types include sub-question types such as picture description, oral composition, and rapid response.

[0163] Step 740: Obtain the question information corresponding to the first question, and the question information corresponding to each question in the target question set.

[0164] Optionally, the question information corresponding to the first question includes the question feature vector corresponding to the first question, and the question information corresponding to each question in the target question set includes the question feature vector corresponding to each question.

[0165] Optionally, the above-mentioned question feature vector is determined based on at least one of the following: question difficulty information, examination rule information, scoring rule information, etc.

[0166] Step 750: Perform similarity matching processing based on the question information corresponding to the first question and the question information corresponding to each question to obtain the second question.

[0167] In one possible implementation, the similarity determination process is performed between the feature vector of the first question and the feature vector of each question to obtain the question similarity between the first question and each question; based on the question similarity between the first question and each question, the second question in the target question set that is similar to the first question can be determined.

[0168] In another possible implementation, based on the question information corresponding to the first question and the question information corresponding to each question, clustering is performed on the first question and the questions in the target question set to obtain the target clustering result corresponding to the first question. The target clustering result includes the first question and the second question. Optionally, based on the question feature vector corresponding to the first question and the question feature vector corresponding to each question, clustering is performed on the first question and the questions in the target question set to obtain at least one clustering result. The clustering result containing the first question is found in the at least one clustering result, and the clustering result containing the first question is determined as the target clustering result. Optionally, the other questions in the target clustering result besides the first question are determined as the second questions. Optionally, the feature distances between the other questions in the target clustering result besides the first question and the first question are determined. The other questions in the target clustering result besides the first question are sorted in ascending order according to the feature distances, and the first preset number of questions in the sorted result are determined as the second questions.

[0169] In one example, such as Figure 9 As shown, this example illustrates a flowchart of a process for filtering labeled data by identifying similar questions of the same question type. To achieve better transferability, target questions belonging to the target question type in a few-person test are compared with different questions of the same target question type in a multiple-person test. Questions of the same and similar question type to the target questions without labeled data in the few-person test are selected from the multiple-person test. The labeled data corresponding to these similar questions in the expert annotation database is then used as the labeled data for the target questions in the few-person test, supplementing the target question annotation data. This similarity can be calculated from the perspectives of question difficulty (easy, medium, hard) and question assessment rules (semantic assessment, pronunciation assessment), constructing relevant feature vectors for each question. The vectors corresponding to different questions of the same question type are then clustered together with the vector of the target question. Clustering analysis can use the k-means algorithm or other clustering algorithms to obtain similar questions corresponding to the target questions in the few-person test. These similar question data are then aggregated to construct the expert-annotated data for the target questions in the few-person test.

[0170] Step 720: Based on the sample answer audio and label information, perform transfer training on the audio detection model to be trained corresponding to the first question to obtain the audio detection model.

[0171] Optionally, the audio detection model to be trained can be a pre-defined machine learning model to be trained.

[0172] In one possible implementation, the audio set of answers corresponding to the first question does not include audio answers with corresponding label information. In this case, the audio detection model to be trained for the first question can be transferred and trained based on the sample audio answers and label information of the second question.

[0173] In another possible implementation, if the number of audio responses with corresponding labels in the audio set corresponding to the first question is less than or equal to the second threshold, it indicates that the amount of labeled data corresponding to the first question itself is relatively small. Therefore, in addition to using its own limited labeled data, it is also necessary to perform transfer training on the audio detection model to be trained for the first question based on the sample audio responses and label information corresponding to the second question.

[0174] In an exemplary embodiment, such as Figure 10 As shown, the implementation process corresponding to step 720 above may include the following steps (721-725). Figure 10 The flowchart illustrates a training method for an audio detection model provided in one embodiment of this application. Figure 3 .

[0175] Step 721: Perform feature extraction processing on the sample response audio to obtain the sample response audio features corresponding to the sample response audio.

[0176] The feature extraction process for the sample answer audio is similar to that for the answer audio of the first question, and will not be repeated here.

[0177] Step 722: Obtain the rule features of the second question corresponding to the second question.

[0178] Optionally, the second question rule represents the answering requirements information corresponding to the second question.

[0179] The process of obtaining the rule features for the second question is similar to that for the first question, and will not be repeated here.

[0180] Step 723: Perform feature fusion processing on the feature features of the second question rules and the feature features of the sample answer audio to obtain the sample fusion feature corresponding to the sample answer audio.

[0181] Optionally, the feature of the second question rule and the feature of the sample answer audio are concatenated to obtain the sample fusion feature corresponding to the sample answer audio. Optionally, the sample fusion feature is a sample fusion feature vector.

[0182] Optionally, the above sample fusion features include the target number of feature dimensions. Each sample response audio corresponds to a sample fusion feature vector with the target number of feature dimensions.

[0183] Step 724: Input the sample fusion features into the audio detection model to be trained for audio detection processing to obtain the sample detection information corresponding to the sample response audio.

[0184] In an exemplary embodiment, such as Figure 11 As shown, after step 723 above, the following steps (726-729) may also be included. Figure 11 The flowchart illustrates a training method for an audio detection model provided in one embodiment of this application. Figure 4 .

[0185] Step 726: Traverse the target number of feature dimensions.

[0186] Step 727: When traversing through the target dimension among the target number of feature dimensions, determine the feature value corresponding to the sample fusion feature in the target dimension.

[0187] For example, the target dimension is the i-th dimension among the target number of feature dimensions mentioned above, and the feature value of the sample fusion feature in the target dimension is the feature value of the sample fusion feature corresponding to the i-th dimension of all sample answer audios.

[0188] Step 728: Standardize the feature values ​​to obtain the transformed feature values ​​of the sample fusion features in the target dimension.

[0189] Optionally, the mean and standard deviation of the feature values ​​are determined; based on the mean and standard deviation, the feature values ​​are standardized to obtain the transformed feature values.

[0190] Optionally, after determining the feature values ​​corresponding to the fusion features of all samples in the i-th dimension, the feature mean and standard deviation corresponding to these feature values ​​can be calculated. Then, each feature value is subtracted from the feature mean, and the result of the subtraction is divided by the standard deviation. The ratio of the result of the subtraction to the standard deviation is used as the transformed feature value.

[0191] Specifically, some question rules and features are filtered. Since multiple similar questions were selected from the same question type in the previous steps, the feature distribution of each question may not be exactly the same. Therefore, feature transformation is performed on the features of each selected question. Generally, feature standardization can be used, that is, the feature mean and standard deviation are calculated for each dimension of the extracted features of each question. Optionally, the feature mean can be determined by the following formula (2), and the standard deviation can be determined by the following formula (3).

[0192]

[0193]

[0194] Where N is the total number of samples in the question, the superscript k denotes the k-th sample, i denotes the i-th feature dimension, N is greater than or equal to 1, k is an integer greater than or equal to 1 and less than or equal to N, and μ i σ represents the mean of the features corresponding to the i-th feature dimension. i This represents the standard deviation corresponding to the i-th feature dimension.

[0195] The feature values ​​of each dimension of each question are standardized based on the mean and standard deviation of each feature dimension. The feature value corresponding to the fused feature of all samples for each question in the i-th feature dimension can be represented as a vector [f]. i 1 ,f i 2 ,…f i N ,], where N is the total number of samples in the question, the superscript indicates the nth sample, and i represents the i-th feature dimension. Optionally, the transformed feature values ​​can be determined according to the following formula (4).

[0196]

[0197] Among them, f i μ represents the eigenvalue corresponding to the i-th feature dimension. i σ represents the mean of the features corresponding to the i-th feature dimension. i f represents the standard deviation corresponding to the i-th feature dimension. i ′ represents the transformed feature value corresponding to the i-th feature dimension.

[0198] Through the above standardization process, the feature distributions of feature values ​​in the fused features that incorporate various modalities can be aligned, thereby improving model accuracy. The technical solution provided in this application achieves better transfer capability by performing feature filtering and feature transformation on relevant features.

[0199] Step 729: Based on the transformed feature values ​​corresponding to each feature dimension of the sample fusion features, obtain the transformed sample fusion features corresponding to the sample response audio.

[0200] Accordingly, such as Figure 11 As shown, step 724 above can be replaced by step 7241.

[0201] Step 7241: Input the transformed sample fusion features into the audio detection model to be trained for audio detection processing to obtain sample detection information.

[0202] Optionally, the above sample detection information includes the detection score corresponding to the audio of the sample response.

[0203] Step 725: Based on the sample detection information and label information, the parameters of the audio detection model to be trained are adjusted to obtain the audio detection model.

[0204] Optionally, the label information includes the labeled score corresponding to the sample response audio. The model loss information can be determined based on the detection score and the labeled score. Based on the above model loss information, the parameters of the audio detection model to be trained can be adjusted and iteratively trained until the model loss information meets the preset loss conditions, such as being less than the loss threshold, or reaching the maximum number of iterations, thereby obtaining the trained audio detection model.

[0205] The audio detection model to be trained can be a traditional machine learning model, such as SVM (Support Vector Machine) or XGB (a tree-based model), or a neural network-based model, such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Deep Neural Network (DNN). Optionally, the input to the audio detection model on the application side is the fused features corresponding to the audio response to be tested, and the output is the actual evaluation score.

[0206] In summary, the technical solution provided in this application uses the sample audio of the second question, which is similar to the first question, and its corresponding tags as the model training data for the first question. This enables the automatic acquisition of the training data for the first question and the completion of the corresponding model training. The trained audio detection model can then be used to detect the audio of the answer to the first question, effectively reducing the detection cost and complexity of detecting the audio of the answer to the first question.

[0207] The beneficial effects of the technical solution provided in this application embodiment are illustrated below using relevant experimental data. In this experiment, Pearson Correlation Coefficient (PCC) and consistency rate (i.e., the proportion of predicted scores and manual labels less than a certain threshold) are used to test the effectiveness. Specifically, the results measured by the technical solution provided in this application embodiment are compared with the results measured by a general question type model (practice system) to demonstrate the beneficial effects of the technical solution provided in this application embodiment. The aforementioned Pearson Correlation Coefficient is used to measure the correlation (linear correlation) between two variables, and its value ranges between -1 and 1.

[0208] In this experiment, the test items included rapid response questions, oral composition questions, and picture description questions. Figure 12(a) exemplarily shows a statistical diagram of the test results based on the rapid response questions; Figure 12(b) exemplarily shows a statistical diagram of the test results based on the picture description questions; Figure 12(c) exemplarily shows a statistical diagram of the test results based on the oral composition questions. Figures 12(a) to 12(c) In the figures, the question type transfer calibration (2 questions) and question type transfer calibration (4 questions) correspond to the technical solutions provided in the embodiments of this application. In addition to comparing the question transfer model in this application with the general question model of the practice system, the inventors of this case also compared the calibration effects of data with two selected questions and data with four selected questions. In Figure 12(a), the test index data corresponding to question type transfer calibration (4 questions) is slightly higher than that corresponding to question type transfer calibration (2 questions), while the test index data corresponding to the practice system is lower, indicating a poorer effect. Figures 12(b) and 12(c) show the same situation. The experimental results show that the technical solution provided in the embodiments of this application has a significant improvement in effect compared to the general question model used in the practice system. Furthermore, as the number of selected questions increases, the overall system indicators begin to stabilize. Therefore, when the number of selected questions reaches a certain level, no more question data is needed.

[0209] The following are embodiments of the apparatus of this application, which can be used to execute embodiments of the method of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method of this application.

[0210] Please refer to Figure 13 This diagram illustrates a block diagram of an audio detection apparatus according to an embodiment of this application. The apparatus has the function of implementing the above-described audio detection method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus 1300 may include: an audio acquisition module 1310 and an audio detection module 1320.

[0211] The audio acquisition module 1310 is used to acquire the audio of the answer to the first question.

[0212] The audio detection module 1320 is used to input the answer audio into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the answer audio;

[0213] The audio detection model is a machine learning model trained on the sample answer audio corresponding to the second question and the label information corresponding to the sample answer audio. The question similarity between the second question and the first question meets the preset question similarity conditions.

[0214] In some possible designs, the audio detection device includes a training device corresponding to the audio detection model, the training device comprising:

[0215] The sample acquisition module is used to acquire the sample answer audio and the tag information when the set of answer audio corresponding to the first question does not meet the preset audio quantity condition;

[0216] The model training module is used to perform transfer training on the audio detection model to be trained corresponding to the first question based on the sample answer audio and the label information, so as to obtain the audio detection model.

[0217] In some possible designs, the training device further includes:

[0218] The question set acquisition module is used to acquire the target question set corresponding to the first question;

[0219] The question information acquisition module is used to acquire the question information corresponding to the first question, as well as the question information corresponding to each question in the target question set;

[0220] The similar question determination module is used to perform similarity matching processing based on the question information corresponding to the first question and the question information corresponding to each of the questions to obtain the second question.

[0221] In some possible designs, the similar question determination module includes:

[0222] The question clustering unit is used to perform clustering processing on the first question and the questions in the target question set based on the question information corresponding to the first question and the question information corresponding to each of the questions, so as to obtain the target clustering result corresponding to the first question, wherein the target clustering result includes the first question and the second question.

[0223] In some possible designs, the model training module includes:

[0224] An audio feature extraction unit is used to perform feature extraction processing on the sample response audio to obtain the sample response audio features corresponding to the sample response audio;

[0225] The question feature acquisition unit is used to acquire the second question rule features corresponding to the second question, wherein the second question rule represents the answer requirement information corresponding to the second question;

[0226] The feature fusion determination unit is used to perform feature fusion processing on the second question rule features and the sample answer audio features to obtain the sample fusion features corresponding to the sample answer audio.

[0227] The fusion feature detection unit is used to input the sample fusion features into the audio detection model to be trained for audio detection processing, and obtain the sample detection information corresponding to the sample response audio.

[0228] The model parameter adjustment unit is used to adjust the parameters of the audio detection model to be trained based on the sample detection information and the label information, so as to obtain the audio detection model.

[0229] In some possible designs, the sample fusion features include a target number of feature dimensions, and the model training module further includes:

[0230] A feature dimension traversal unit is used to traverse the target number of feature dimensions;

[0231] The feature value determination unit is used to determine the feature value of the sample fusion feature corresponding to the target dimension when traversing the target number of feature dimensions;

[0232] The feature value transformation unit is used to standardize the feature values ​​to obtain the transformed feature values ​​of the sample fusion features corresponding to the target dimension.

[0233] The fusion feature transformation unit is used to obtain the transformed sample fusion feature corresponding to the sample response audio based on the transformed feature value corresponding to each feature dimension of the sample fusion feature.

[0234] The fusion feature detection unit is also used to input the transformed sample fusion features into the audio detection model to be trained for audio detection processing to obtain the sample detection information.

[0235] In some possible designs, the eigenvalue transformation unit includes:

[0236] The eigenvalue parameter determination subunit is used to determine the eigenvalue's mean and standard deviation;

[0237] The eigenvalue transformation subunit is used to standardize the eigenvalues ​​based on the eigenvalue mean and the standard deviation to obtain the transformed eigenvalues.

[0238] In some possible designs, the audio detection device further includes:

[0239] An audio feature extraction unit is used to perform feature extraction processing on the response audio to obtain the response audio features corresponding to the response audio;

[0240] The question feature acquisition unit is used to acquire the first question rule feature corresponding to the first question, wherein the first question rule feature represents the answer requirement information corresponding to the first question;

[0241] The feature fusion determination unit is used to perform feature fusion processing on the first question rule features and the answer audio features to obtain the fusion features corresponding to the answer audio.

[0242] The audio detection module is specifically used to input the fused features into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information.

[0243] In summary, the technical solution provided in this application uses the sample audio of the second question, which is similar to the first question, and its corresponding tags as the model training data for the first question. This enables the automatic acquisition of the training data for the first question and the completion of the corresponding model training. The trained audio detection model can then be used to detect the audio of the answer to the first question, effectively reducing the detection cost and complexity of detecting the audio of the answer to the first question.

[0244] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0245] Please refer to Figure 14 It illustrates the structural block of a computer device provided in one embodiment of this application. Figure 1 The computer device can be a terminal. This computer device is used to implement the audio detection method provided in the above embodiments. Specifically:

[0246] Typically, computer device 1400 includes a processor 1401 and a memory 1402.

[0247] Processor 1401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0248] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1402 is used to store at least one instruction, at least one program, code set, or instruction set, configured to be executed by one or more processors to implement the audio detection method described above.

[0249] In some embodiments, the computer device 1400 may also optionally include a peripheral device interface 1403 and at least one peripheral device. The processor 1401, memory 1402, and peripheral device interface 1403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1404, a touch display screen 1405, a camera assembly 1406, an audio circuit 1407, a positioning assembly 1408, and a power supply 1409.

[0250] Those skilled in the art will understand that Figure 14The structure shown does not constitute a limitation on the computer device 1400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0251] Please refer to Figure 15 It illustrates the structural block of a computer device provided in another embodiment of this application. Figure 2 The computer device can be a server for performing the aforementioned audio detection method. Specifically:

[0252] Computer device 1500 includes a central processing unit (CPU) 1501, a system memory 1504 including random access memory (RAM) 1502 and read-only memory (ROM) 1503, and a system bus 1505 connecting the system memory 1504 and the CPU 1501. Computer device 1500 also includes a basic input / output system (I / O system) 1506 that facilitates information transfer between various devices within the computer, and a mass storage device 1507 for storing the operating system 1513, application programs 1514, and other program modules 1515.

[0253] The basic input / output system 1506 includes a display 1508 for displaying information and an input device 1509 for user input, such as a mouse or keyboard. Both the display 1508 and the input device 1509 are connected to the central processing unit 1501 via an input / output controller 1510 connected to the system bus 1505. The basic input / output system 1506 may also include the input / output controller 1510 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1510 also provides output to a display screen, printer, or other types of output devices.

[0254] Mass storage device 1507 is connected to central processing unit 1501 via a mass storage controller (not shown) connected to system bus 1505. Mass storage device 1507 and its associated computer-readable media provide non-volatile storage for computer device 1500. That is, mass storage device 1507 may include computer-readable media (not shown) such as hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

[0255] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1504 and mass storage device 1507 described above can be collectively referred to as memory.

[0256] According to various embodiments of this application, the computer device 1500 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1500 can be connected to the network 1512 via the network interface unit 1511 connected to the system bus 1505, or the network interface unit 1511 can be used to connect to other types of networks or remote computer systems (not shown).

[0257] The memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the above-described audio detection method.

[0258] In an exemplary embodiment, a computer-readable storage medium is also provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set, when executed by a processor, implements the above-described audio detection method.

[0259] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0260] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned audio detection method.

[0261] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0262] In addition, in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0263] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An audio detection method, characterized in that, The method includes: Get the audio of the answer to the first question; The audio of the answer is input into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the audio of the answer; The audio detection model is a machine learning model trained on the audio detection model to be trained based on the sample detection information corresponding to the sample answer audio of the second question and the label information corresponding to the sample answer audio. The sample detection information is the detection information obtained by inputting the second question rule features corresponding to the second question and the sample answer audio features corresponding to the sample answer audio into the audio detection model to be trained for audio detection processing. The second question rule represents the answer requirement information corresponding to the second question. The sample answer audio features are obtained based on at least one of the pronunciation accuracy features, pronunciation fluency features, and pronunciation prosody features of the sample answer audio. The question similarity between the second question and the first question meets the preset question similarity conditions.

2. The method according to claim 1, characterized in that, The training process of the audio detection model includes: If the set of audio responses to the first question does not meet the preset audio quantity condition, obtain the sample audio responses and the tag information; Based on the sample answer audio and the label information, the audio detection model corresponding to the first question is transferred to train, and the audio detection model is obtained.

3. The method according to claim 2, characterized in that, The training process of the audio detection model also includes: Obtain the target question set corresponding to the first question; Obtain the question information corresponding to the first question, and the question information corresponding to each question in the target question set; The second question is obtained by performing similarity matching between the question information corresponding to the first question and the question information corresponding to each of the above questions.

4. The method according to claim 3, characterized in that, The step of performing similarity matching processing between the question information corresponding to the first question and the question information corresponding to each of the other questions to obtain the second question includes: Based on the question information corresponding to the first question and the question information corresponding to each of the questions, clustering is performed on the first question and the questions in the target question set to obtain the target clustering result corresponding to the first question. The target clustering result includes the first question and the second question.

5. The method according to any one of claims 2 to 4, characterized in that, The step of performing transfer training on the audio detection model corresponding to the first question based on the sample answer audio and the label information to obtain the audio detection model includes: The sample response audio is processed by feature extraction to obtain the sample response audio features corresponding to the sample response audio; Obtain the second question rule features corresponding to the second question; the second question rule represents the answer requirement information corresponding to the second question. The second question rule features and the sample answer audio features are subjected to feature fusion processing to obtain the sample fusion features corresponding to the sample answer audio; The sample fusion features are input into the audio detection model to be trained for audio detection processing to obtain the sample detection information corresponding to the sample response audio. Based on the sample detection information and the label information, the parameters of the audio detection model to be trained are adjusted to obtain the audio detection model.

6. The method according to claim 5, characterized in that, The sample fusion features include a target number of feature dimensions. After performing feature fusion processing on the second question rule features and the sample answer audio features to obtain the sample fusion features corresponding to the sample answer audio, the process further includes: Iterate through the target's number of feature dimensions; When traversing through the target dimension among the target number of feature dimensions, determine the feature value of the sample fusion feature corresponding to the target dimension; The feature values ​​are standardized to obtain the transformed feature values ​​of the sample fusion features corresponding to the target dimension. Based on the transformed feature values ​​corresponding to each feature dimension of the sample fusion features, the transformed sample fusion features corresponding to the sample response audio are obtained; The step of inputting the sample fusion features into the audio detection model to be trained for audio detection processing to obtain sample detection information corresponding to the sample response audio includes: The transformed sample fusion features are input into the audio detection model to be trained for audio detection processing to obtain the sample detection information.

7. The method according to claim 6, characterized in that, The standardization process for the feature values ​​to obtain the transformed feature values ​​of the sample fusion features corresponding to the target dimension includes: Determine the mean and standard deviation of the features corresponding to the eigenvalues; Based on the mean and standard deviation of the features, the feature values ​​are standardized to obtain the transformed feature values.

8. The method according to claim 1, characterized in that, The method further includes: The response audio is subjected to feature extraction processing to obtain the response audio features corresponding to the response audio; Obtain the first question rule feature corresponding to the first question, where the first question rule feature represents the answer requirement information corresponding to the first question; The first question rule features and the answer audio features are subjected to feature fusion processing to obtain the fused features corresponding to the answer audio; The step of inputting the answer audio into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the answer audio includes: The fusion features are input into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information.

9. An audio detection device, characterized in that, The device includes: The audio acquisition module is used to acquire the audio of the answer to the first question; The audio detection module is used to input the answer audio into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information corresponding to the answer audio; The audio detection model is a machine learning model trained on the audio detection model to be trained based on the sample detection information corresponding to the sample answer audio of the second question and the label information corresponding to the sample answer audio. The sample detection information is the detection information obtained by inputting the second question rule features corresponding to the second question and the sample answer audio features corresponding to the sample answer audio into the audio detection model to be trained for audio detection processing. The second question rule represents the answer requirement information corresponding to the second question. The sample answer audio features are obtained based on at least one of the pronunciation accuracy features, pronunciation fluency features, and pronunciation prosody features of the sample answer audio. The question similarity between the second question and the first question meets the preset question similarity conditions.

10. The apparatus according to claim 9, characterized in that, The audio detection device includes a training device corresponding to the audio detection model, and the training device includes: The sample acquisition module is used to acquire the sample answer audio and the tag information when the set of answer audio corresponding to the first question does not meet the preset audio quantity condition; The model training module is used to perform transfer training on the audio detection model to be trained corresponding to the first question based on the sample answer audio and the label information, so as to obtain the audio detection model.

11. The apparatus according to claim 10, characterized in that, The training device further includes: The question set acquisition module is used to acquire the target question set corresponding to the first question; The question information acquisition module is used to acquire the question information corresponding to the first question, as well as the question information corresponding to each question in the target question set; The similar question determination module is used to perform similarity matching processing based on the question information corresponding to the first question and the question information corresponding to each of the questions to obtain the second question.

12. The apparatus according to claim 11, characterized in that, The similar question determination module includes: The question clustering unit is used to perform clustering processing on the first question and the questions in the target question set based on the question information corresponding to the first question and the question information corresponding to each of the questions, so as to obtain the target clustering result corresponding to the first question, wherein the target clustering result includes the first question and the second question.

13. The apparatus according to any one of claims 10 to 12, characterized in that, The model training module includes: An audio feature extraction unit is used to perform feature extraction processing on the sample response audio to obtain the sample response audio features corresponding to the sample response audio; The question feature acquisition unit is used to acquire the second question rule features corresponding to the second question, wherein the second question rule represents the answer requirement information corresponding to the second question; The feature fusion determination unit is used to perform feature fusion processing on the second question rule features and the sample answer audio features to obtain the sample fusion features corresponding to the sample answer audio. The fusion feature detection unit is used to input the sample fusion features into the audio detection model to be trained for audio detection processing, and obtain the sample detection information corresponding to the sample response audio. The model parameter adjustment unit is used to adjust the parameters of the audio detection model to be trained based on the sample detection information and the label information, so as to obtain the audio detection model.

14. The apparatus according to claim 13, characterized in that, The sample fusion features include a target number of feature dimensions, and the model training module further includes: A feature dimension traversal unit is used to traverse the target number of feature dimensions; The feature value determination unit is used to determine the feature value of the sample fusion feature corresponding to the target dimension when traversing the target number of feature dimensions; The feature value transformation unit is used to standardize the feature values ​​to obtain the transformed feature values ​​of the sample fusion features corresponding to the target dimension. The fusion feature transformation unit is used to obtain the transformed sample fusion feature corresponding to the sample response audio based on the transformed feature value corresponding to each feature dimension of the sample fusion feature. The fusion feature detection unit is also used to input the transformed sample fusion features into the audio detection model to be trained for audio detection processing to obtain the sample detection information.

15. The apparatus according to claim 14, characterized in that, The eigenvalue transformation unit includes: The eigenvalue parameter determination subunit is used to determine the eigenvalue's mean and standard deviation; The eigenvalue transformation subunit is used to standardize the eigenvalues ​​based on the eigenvalue mean and the standard deviation to obtain the transformed eigenvalues.

16. The apparatus according to claim 9, characterized in that, The device further includes: An audio feature extraction unit is used to perform feature extraction processing on the response audio to obtain the response audio features corresponding to the response audio; The question feature acquisition unit is used to acquire the first question rule feature corresponding to the first question, wherein the first question rule feature represents the answer requirement information corresponding to the first question; The feature fusion determination unit is used to perform feature fusion processing on the first question rule features and the answer audio features to obtain the fusion features corresponding to the answer audio. The audio detection module is further configured to: input the fusion features into the audio detection model corresponding to the first question for audio detection processing to obtain the detection information.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the audio detection method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the audio detection method as described in any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the audio detection method as described in any one of claims 1 to 8.

Images