Audio answer scoring method, device, equipment, storage medium and product

Abstract

The present application provides an audio answer scoring method, apparatus, device, storage medium and product, the method comprising: obtaining a target answer audio and a first task instruction, the first task instruction comprising description information of the scoring task, question information corresponding to the target answer audio and a scoring rule; inputting the target answer audio and the first task instruction into a pre-trained multimodal large language model, the multimodal large language model outputting a scoring result for the target answer audio; the multimodal large language model is obtained by executing a first training task and a second training task, the first training task being used to score the first sample answer audio based on a transcribed text of the first sample answer audio, the second training task being used to determine a score for the first sample answer audio by processing the first sample answer audio, the scoring result of the first training task being used as a scoring label for the second training task.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to an audio answer scoring method, device, equipment, storage medium and product. Background Art

[0002] In education, assessment or evaluation scenarios, it is a common requirement to grade answers in audio format. With the update of educational concepts, the answers that need to be graded have gradually transitioned from fixed text reading to free expression, making the grading work more complicated.

[0003] At present, the scoring method for answer audio usually adopts a cascade solution, that is, voice recognition is performed first, and then text scoring is performed based on the voice recognition results. Due to the errors and limitations of voice recognition technology itself, this cascade solution often produces cascade errors due to voice recognition errors, thereby affecting the scoring effect of the answer audio. This high reliance on voice recognition accuracy limits the optimization space of the scoring method, making it a great challenge to improve the scoring effect. Summary of the invention

[0004] In view of this, the embodiments of the present application are committed to providing an audio answer scoring method, device, equipment, storage medium and product, which can score the target answer audio more accurately and quickly.

[0005] According to a first aspect of an embodiment of the present application, a method for scoring an audio answer is provided, the method comprising:

[0006] Obtaining a target answer audio and a first task instruction, wherein the first task instruction includes description information of a scoring task, question information corresponding to the target answer audio, and a scoring rule;

[0007] Inputting the target answer audio and the first task instruction into a pre-trained multimodal large language model, and allowing the multimodal large language model to determine a scoring result for the target answer audio according to the target answer audio and the first task instruction;

[0008] Among them, the multimodal large language model is obtained by executing a first training task and a second training task, the first training task is used to score the first sample answer audio based on the transcribed text of the first sample answer audio, and the second training task is used to determine the score of the first sample answer audio by processing the first sample answer audio, and the scoring result of the first training task is used as the scoring label of the second training task.

[0009] Optionally, the multimodal large language model includes an acoustic encoding module, a modality adaptation module, and a pre-trained large language model, wherein:

[0010] The acoustic encoding module is used to encode the target answer audio to obtain high-dimensional acoustic features of the target answer audio;

[0011] The modality adaptation module is used to convert the high-dimensional acoustic features into the text semantic space adopted by the large language model to obtain the final features of the target answer audio;

[0012] The large language model is used to determine a scoring result for the target answer audio based on the final features and the first task instruction.

[0013] Optionally, the multimodal large language model further includes a compression module, and the compression module is used to compress the high-dimensional acoustic feature to obtain a compressed feature corresponding to the high-dimensional acoustic feature;

[0014] The modal adaptation module is specifically used to convert the compressed features corresponding to the high-dimensional acoustic features into the text semantic space adopted by the large language model to obtain the final features of the target answer audio.

[0015] Optionally, the first training task and the second training task are used to optimize parameters of the modality adaptation module in the multimodal large language model.

[0016] Optionally, the first training task includes:

[0017] Inputting the transcribed text of the first sample answer audio and the second sample task instruction into a pre-built multimodal large language model, the multimodal large language model determining a first scoring result for the first sample answer audio according to the transcribed text of the first sample answer audio and the second sample task instruction, wherein the second sample task instruction instructs to perform a scoring task including at least one scoring dimension;

[0018] The second training task includes:

[0019] Inputting the first sample answer audio and the second sample task instruction into the pre-built multimodal large language model, and the multimodal large language model determines a second scoring result for the first sample answer audio according to the first sample answer audio and the second sample task instruction;

[0020] Determine a first loss function value according to the first scoring result and the second scoring result;

[0021] According to the first loss function value, parameters of the modality adaptation module in the pre-built multimodal large language model are optimized.

[0022] Optionally, the training task of the multimodal large language model further includes a third training task and a fourth training task, wherein the third training task includes:

[0023] Inputting the annotated transcribed text of the first sample audio into a pre-built multimodal large language model, and allowing the multimodal large language model to generate a first result based on the annotated transcribed text of the first sample audio;

[0024] The fourth training task includes:

[0025] Inputting the first sample audio into the pre-built multimodal large language model, and allowing the multimodal large language model to generate a second result based on the first sample audio;

[0026] Determine a second loss function value according to the first result and the second result;

[0027] According to the second loss function value, parameters of the modality adaptation module in the pre-built multimodal large language model are optimized.

[0028] Optionally, after performing parameter optimization on the modality adaptation module based on the third training task and the fourth training task, performing parameter optimization on the modality adaptation module based on the first training task and the second training task.

[0029] Optionally, the training task of the multimodal large language model further includes a fifth training task, and the fifth training task includes:

[0030] Inputting the second sample answer audio and the first sample task instruction into a pre-built multimodal large language model, the multimodal large language model determining a scoring result for the second sample answer audio according to the second sample answer audio and the first sample task instruction, wherein the first sample task instruction includes description information of the scoring task, question information corresponding to the second sample answer audio, and scoring rules;

[0031] Determine a third loss function value according to a scoring result of the multimodal large language model on the second sample answer audio and a labeled scoring result corresponding to the second sample answer audio;

[0032] According to the third loss function value, parameters of the large language model in the pre-constructed multimodal large language model are optimized.

[0033] According to a second aspect of an embodiment of the present application, there is provided an audio answer scoring device, comprising:

[0034] The first unit is used to obtain a target answer audio and a first task instruction, wherein the first task instruction includes description information of a scoring task, question information corresponding to the target answer audio, and a scoring rule;

[0035] The second unit is used to input the target answer audio and the first task instruction into a pre-trained multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio according to the target answer audio and the first task instruction;

[0036] Among them, the multimodal large language model is obtained by executing a first training task and a second training task, the first training task is used to score the first sample answer audio based on the transcribed text of the first sample answer audio, and the second training task is used to determine the score of the first sample answer audio by processing the first sample answer audio, and the scoring result of the first training task is used as the scoring label of the second training task.

[0037] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a memory and a processor;

[0038] The memory is connected to the processor and is used to store programs;

[0039] The processor is used to implement the audio answer scoring method as described in any one of the first aspects of the embodiments of the present application by running the program in the memory.

[0040] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the audio answer scoring method as described in any one of the first aspects of the embodiments of the present application is implemented.

[0041] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement an audio answer scoring method as described in any one of the first aspects of the embodiments of the present application.

[0042] The audio answer scoring method provided in the embodiment of the present application first obtains a target answer audio and a first task instruction, wherein the first task instruction includes description information of the scoring task, question information corresponding to the target answer audio, and a scoring rule; then the target answer audio and the first task instruction are input into a pre-trained multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio according to the target answer audio and the first task instruction; wherein the multimodal large language model is obtained by executing a first training task and a second training task, wherein the first training task is used to score the first sample answer audio based on a transcribed text of the first sample answer audio, and the second training task is used to score the first sample answer audio, and the scoring result of the first training task is used as a scoring label for the second training task.

[0043] The audio answer scoring method provided in the present application, on the one hand, uses a multimodal large language model to directly determine the scoring result of the target answer audio according to the target answer audio and the first task instruction, which can avoid the cascade error caused by the "speech recognition-text scoring" cascade scheme and the loss of non-text information of the target answer audio. Moreover, relying on the powerful understanding and analysis capabilities of the multimodal large language model, it can perform in-depth evaluation based on the target answer audio and the first task instruction, so as to score the target answer audio more accurately; on the other hand, using the first training task and the second training task to train the multimodal large language model not only strengthens the speech-text modality mapping capability of the multimodal large language model, but also ensures that the mapped semantic modality is highly consistent with the scoring task, so that the multimodal large language model can perform better in the scoring task, thereby improving the accuracy of the audio answer scoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A flowchart of an audio answer scoring method provided in an embodiment of the present application.

[0046] Figure 2 A schematic diagram of the structure of a multimodal large language model provided in an embodiment of the present application.

[0047] Figure 3 A schematic diagram of the structure of another multimodal large language model provided in an embodiment of the present application.

[0048] Figure 4 A schematic diagram of the structure of an audio answer scoring device provided in an embodiment of the present application.

[0049] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solution of the embodiment of the present application is applicable to various scenarios where the answer audio needs to be scored, including but not limited to various fields such as online oral examinations, oral ability assessments, language learning assessments, voice skills assessments, interview records, etc. In these scenarios, the answer audio of the respondent needs to be scored to measure the quality of the answer and evaluate the individual performance of the respondent.

[0051] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] Before introducing this application solution, the relevant technologies are first introduced:

[0054] In education, assessment or evaluation scenarios, it is a common requirement to grade answers in audio format. With the update of educational concepts, the answers that need to be graded have gradually transitioned from fixed text reading to free expression, making the grading work more complicated.

[0055] At present, the scoring method for answer audio usually adopts a cascade solution, that is, voice recognition is performed first, and then text scoring is performed based on the voice recognition results. Due to the errors and limitations of voice recognition technology itself, this cascade solution often produces cascade errors due to voice recognition errors, thereby affecting the scoring effect of the answer audio. This high reliance on voice recognition accuracy limits the optimization space of the scoring method, making it a great challenge to improve the scoring effect.

[0056] In view of this, the embodiments of the present application are committed to providing an audio answer scoring method, device, equipment, storage medium and product, which can score the target answer audio more accurately and quickly, and are described in detail one by one in the following embodiments.

[0057] Exemplary Methods

[0058] Figure 1 A flowchart of an audio answer scoring method provided in an embodiment of the present application. Figure 1 As shown, the audio answer scoring method provided in this embodiment includes steps S101-S102:

[0059] S101. Obtain a target answer audio and a first task instruction, wherein the first task instruction includes description information of a scoring task, question information corresponding to the target answer audio, and scoring rules.

[0060] The target answer audio can be understood as a specific audio file that needs to be scored, which contains one or more individuals' oral answers, statements or performances to a specific question. It is the object of scoring and is used to evaluate the respondent's answer performance.

[0061] The target answer audio may be a recording of a user's oral answer to a question, or may be audio generated by some speech synthesis technology, which usually includes speech features such as the content, intonation, and speaking speed of the answer.

[0062] The target answer audio may come from various scenarios where voice answers need to be scored, such as online language tests, oral tests, voice competitions, etc. In these scenarios, the respondent submits the answer content in the form of voice, and the voice data submitted by the respondent is the target answer audio.

[0063] This application does not limit the method of obtaining the target answer audio, which is determined according to the actual application scenario and needs. For example, the audio file can be downloaded or received from the online form or platform submitted by the respondent, the respondent's voice can be recorded in real time through a dedicated audio acquisition device or software, and it can also be retrieved and extracted from a stored audio database.

[0064] The first task instruction is a set of specific guidance information, which is used to guide the subject who receives the task instruction on how to score the target answer audio, and provides clear guidance and constraints for the execution subject. Optionally, the first task instruction is text information.

[0065] The first task instruction is used to trigger the execution of a scoring task, and the scoring task is used to determine a scoring result for the target answer audio based on the target answer audio and the first task instruction.

[0066] The first task instruction includes description information of the scoring task, question information corresponding to the target answer audio, and scoring rules.

[0067] The descriptive information of the scoring task can be understood as text content that explains or defines the scoring task itself in detail, usually including one or more of the nature of the task, the purpose of the task, the scope of evaluation that may be involved, etc. The descriptive information of the scoring task can make the object performing the scoring task clear about the scoring task and help it understand the context and intention of the scoring. For example, in an English speaking test, the descriptive information of the scoring task can be "This scoring task aims to evaluate the English speaking ability of the candidates. Through this scoring, we hope to be able to objectively and comprehensively reflect the English speaking level of the candidates. Please score the provided answer audio according to the provided question information and scoring rules, and output the scoring results."

[0068] The topic information corresponding to the target answer audio can be understood as a description of specific topic information directly related to the target answer audio content, which refers to the specific content and / or reference answer of the question that guides the respondent to give an oral answer or statement. Optionally, the topic information includes the question and / or the reference answer.

[0069] The topic information corresponding to the target answer audio provides the scorer with background information for understanding and evaluating the target answer audio. By understanding the topic information, the scorer can judge whether the respondent's answer is accurate and complete, and whether it is consistent with the topic requirements or reference answers. For example, in an English oral test, if the topic information is "Please talk about your views on environmental protection", then the answer audio should be the respondent's oral expression of the topic. When evaluating this audio, the scorer will refer to the topic information to judge whether the respondent's answer revolves around environmental protection, whether he expresses clear opinions and reasons, etc.

[0070] The scoring rules specify the criteria for scoring the answer quality of the answer audio. The scoring rules are determined according to the actual application scenario and are intended to score the answer audio through one or more of the set conditions, algorithms or parameters.

[0071] Optionally, the scoring rule includes at least one scoring dimension, which is set according to the evaluation target and / or the characteristics of the answer audio, and each dimension reflects the answer quality of the target answer audio from a different perspective. For example, in an English oral test, the scoring dimensions may include language fluency, pronunciation accuracy, grammatical correctness, content coherence and logic, language application ability, etc.

[0072] Furthermore, the scoring rules include weights and / or specific scoring criteria corresponding to each scoring dimension.

[0073] Optionally, each scoring dimension includes specific scoring criteria, which describe in detail how to evaluate or score the answer audio under the scoring dimension, provide clear guidance for the scorer, and make the scoring process more objective and accurate. Taking language fluency as an example, the scoring criteria may include "1. Appropriate pauses: The respondent should have appropriate pauses when expressing in order to distinguish sentence components, emphasize key information or give the audience time to think. The scoring point is that the pauses are natural and reasonable, and do not affect the coherence and expression of meaning of the sentence; 2. Repetition control: The respondent should avoid unnecessary repetitions when expressing, such as repeating the same words, phrases or sentences, to keep the expression concise and clear. The scoring point is no redundant repetition, and the information expression is efficient and direct; 3. Natural sentence transitions: The transitions between sentences should be smooth and natural, avoiding abrupt jumps or logical breaks. The scoring point is that the sentences are smoothly connected, the logical relationship is clear, and the audience can smoothly understand the overall meaning."

[0074] Optionally, in order to more intuitively reflect the performance of the respondent, the scoring rules will also specify the score level. The score level usually corresponds to a specific score range, and the evaluation reasons corresponding to each level are given. Exemplarily, the evaluation scores can be divided into five levels, such as excellent (90-100 points), good (80-89 points), medium (70-79 points), pass (60-69 points) and fail (below 60 points). For each score level, specific evaluation reasons will be given. For example, in the "excellent" level, the respondent may be required to perform well in all scoring dimensions, with fluent language, accurate pronunciation, correct grammar, rich content and clear logic. In the "fail" level, it may mean that the respondent has obvious deficiencies in multiple dimensions and cannot meet the basic expression requirements.

[0075] Optionally, the scoring rules only include concise core requirements or guidelines, for example, the scoring rule is "give an overall impression score to the candidate's answer from the dimensions of content, grammar, fluency and coherence". The multimodal large language model, with its powerful language understanding, reasoning and generation capabilities, can comprehensively analyze and score the target answer audio based on these concise rules to ensure the accuracy and objectivity of the scoring. This design simplifies the process of formulating scoring rules and makes full use of the advanced capabilities of the multimodal large language model.

[0076] Optionally, the first task instruction also includes question type information, which can be understood as a specific description of the question type to which the answer audio belongs. It defines the content and form range of the answer audio, as well as the aspects that should be focused on when scoring. For example, in an oral English test, question type information may include "self-introduction", "topic statement", "viewpoint discussion", etc. Each question type has its specific scoring points and emphasis. By providing question type information, the multimodal large language model can be further guided to more accurately understand and analyze the target answer audio, thereby giving a scoring result that is more in line with the requirements of the question.

[0077] As an optional implementation manner, obtaining the first task instruction includes: obtaining the first task instruction in response to a user input of the first task instruction.

[0078] As an optional implementation method, obtaining the first task instruction includes: obtaining the description information of the scoring task, the question information corresponding to the target answer audio, and the scoring rules, and generating the first task instruction according to a preset instruction format based on the description information of the scoring task, the question information corresponding to the target answer audio, and the scoring rules.

[0079] In this implementation, optionally, in response to the user's input operation, the description information of the scoring task, the topic information corresponding to the target answer audio, and the scoring rules are obtained. Optionally, according to the actual application scenario, the description information of the appropriate scoring task is selected from the preset task information library, and / or the appropriate scoring rules are selected from the preset rule library, and / or the appropriate topic information is selected from the topic information library. Optionally, through the machine learning model, the description information of the scoring task and / or the scoring rules are automatically generated according to historical scoring cases or scoring experience.

[0080] S102. Input the target answer audio and the first task instruction into a pre-trained multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio based on the target answer audio and the first task instruction.

[0081] Specifically, after obtaining the target answer audio and the first task instruction, the target answer audio and the first task instruction are input into a pre-trained multimodal large language model, the pre-trained multimodal large language model executes the first task instruction, scores the target answer audio according to the input content, and outputs the scoring result of the target answer audio.

[0082] The scoring result is a quantitative evaluation of the target answer audio, which is a conclusion drawn after a comprehensive evaluation of the target answer audio, reflecting the answer quality of the target answer audio. Its specific content and form of expression can be determined according to actual application scenarios and needs.

[0083] Optionally, the scoring result can be a specific score value, such as a score between 0 and 100, which is calculated according to the scoring rules. Optionally, the scoring result can be presented in the form of grades, such as A, B, C, D or excellent, good, medium, pass, fail, etc. The grade assessment is usually based on the score range, and each grade corresponds to a score segment, which is convenient for quickly understanding the answer quality of the target answer audio. Optionally, the scoring result may also include a detailed descriptive analysis, indicating the specific performance of the respondent in each scoring dimension. For example, in the language expression dimension, the advantages and disadvantages of the respondent in terms of fluency, clarity, accuracy of wording, etc. may be pointed out.

[0084] The multimodal large language model is an advanced generative deep neural network model that not only has powerful natural language processing capabilities, but can also process and understand data in multiple modalities including text, audio, images, etc. By integrating information from different modalities, it can more comprehensively understand and analyze complex data, thereby providing richer and more accurate language output.

[0085] The pre-trained multimodal large language model may be any type of multimodal large language model in the prior art that is capable of processing at least two modal data, text and audio.

[0086] As an optional implementation, the multimodal large language model includes an acoustic encoding module, a modal adaptation module and a pre-trained large language model, wherein the acoustic encoding module is used to encode the target answer audio to obtain the high-dimensional acoustic features of the target answer audio; the modal adaptation module is used to convert the high-dimensional acoustic features to the text semantic space adopted by the large language model to obtain the final features of the target answer audio; the large language model is used to determine the scoring result of the target answer audio according to the final features and the first task instruction. This will be explained in the subsequent content and will not be described in detail here.

[0087] Furthermore, the multimodal large language model is obtained by executing a first training task and a second training task, wherein the first training task is used to score the first sample answer audio based on the transcribed text of the first sample answer audio, and the second training task is used to determine the score of the first sample answer audio by processing the first sample answer audio, and the scoring result of the first training task is used as the scoring label of the second training task.

[0088] The first sample answer audio may be any type of answer audio. Optionally, the first sample answer audio may be obtained through a variety of channels, including but not limited to public data sets, field recordings, and the like.

[0089] Optionally, the transcribed text of the first sample answer audio may be a manually annotated transcribed text, or a transcribed text obtained by speech transcription of the first sample answer audio using automatic speech recognition (ASR) technology or other related technical means, which can reflect the audio content of the first sample answer audio.

[0090] Optionally, the scoring of the first sample answer audio in the first training task and the second training task can be understood as scoring the first sample answer audio including at least one scoring dimension.

[0091] Optionally, the scoring of the first sample answer audio in the first training task and the second training task can be understood as performing at least one of the scoring-related tasks on the first sample answer audio, such as grammar correction, composition scoring, off-topic detection, context coherence evaluation and other scoring-related tasks. The scoring result of the first sample answer audio can be understood as the execution result of the scoring-related tasks performed on the first sample answer audio.

[0092] In the first training task, the multimodal large language model first receives the transcribed text of the first sample answer audio as input, and then scores the original first sample answer audio according to the transcribed text of the first sample answer audio to obtain a scoring result for the first sample answer audio, which is obtained based on the transcribed text of the first sample answer audio.

[0093] In the second training task, the multimodal large language model directly receives the first sample answer audio as input instead of transcribing text, and then processes and analyzes the first sample answer audio and scores the first sample answer audio according to the first sample answer audio itself, thereby obtaining a scoring result for the first sample answer audio, which is based on the first sample answer audio itself.

[0094] During the training process, the scoring result of the first training task (the scoring result obtained based on the transcribed text of the first sample answer audio) is used as the scoring label of the second training task, which means that the scoring result generated by the first training task is regarded as a supervisory signal in the second training task, which is used to guide the learning of the multimodal large language model in the second training task.

[0095] Through this training method, the multimodal large language model can learn how to directly generate a scoring result that is consistent with the scoring based on the transcribed text based on the answer audio, which helps to improve the multimodal large language model's ability to score the answer audio. Specifically, on the one hand, it can effectively improve the multimodal large language model's speech-text modality mapping ability, and on the other hand, it can enable the multimodal large language model to more accurately understand and analyze the answer audio, and convert it into a text semantic modality that matches the scoring task, thereby improving the accuracy and efficiency of subsequent scoring tasks, giving the model the ability to accurately locate and evaluate the answer audio in a complex semantic space, laying a solid foundation for scoring work.

[0096] The audio answer scoring method provided in the present application, on the one hand, uses a multimodal large language model to directly determine the scoring result of the target answer audio according to the target answer audio and the first task instruction, which can avoid the cascade error caused by the "speech recognition-text scoring" cascade scheme and the loss of non-text information of the target answer audio. Moreover, relying on the powerful understanding and analysis capabilities of the multimodal large language model, it can perform in-depth evaluation based on the target answer audio and the first task instruction, so as to score the target answer audio more accurately; on the other hand, using the first training task and the second training task to train the multimodal large language model not only strengthens the speech-text modality mapping capability of the multimodal large language model, but also ensures that the mapped semantic modality is highly consistent with the scoring task, so that the multimodal large language model can perform better in the scoring task, thereby improving the accuracy of the audio answer scoring.

[0097] Figure 2 A schematic diagram of the structure of a multimodal large language model provided in an embodiment of the present application. Figure 2 As shown, as an optional implementation, the multimodal large language model includes an acoustic encoding module 210, a modal adaptation module 220, and a pre-trained large language model 230. Among them, the acoustic encoding module 210 is used to encode the target answer audio to obtain the high-dimensional acoustic features of the target answer audio; the modal adaptation module 220 is used to convert the high-dimensional acoustic features to the text semantic space adopted by the large language model to obtain the final features of the target answer audio; the pre-trained large language model 230 is used to determine the scoring result of the target answer audio according to the final features and the first task instruction.

[0098] Next, the acoustic coding module 210 is described in detail.

[0099] The acoustic encoding module 210 is used to encode the target answer audio to obtain high-dimensional acoustic features of the target answer audio.

[0100] Optionally, the acoustic encoding module 210 encodes the target answer audio, including: extracting the acoustic features of the target answer audio, converting the original audio data into numerical features that can be understood and processed by the model, and then further encoding the acoustic features of the target answer audio, converting and abstracting the acoustic features of the target answer audio, and then obtaining high-dimensional acoustic features of the target answer audio.

[0101] Optionally, after the target answer audio is input into the multimodal large language model, the original audio data is converted into numerical features that can be understood and processed by the model, namely, acoustic features, and then the acoustic features are input into the acoustic encoding module 210. In this case, the acoustic encoding module 210 encodes the target answer audio, including: further encoding the acoustic features corresponding to the target answer audio, converting and abstracting the acoustic features of the target answer audio, and obtaining high-dimensional acoustic features of the target answer audio.

[0102] The acoustic features can be understood as numerical features extracted directly from the original target answer audio, including basic audio parameters and information, such as spectrum, energy, zero-crossing rate, etc. The dimension of acoustic features is relatively low. Although it contains basic audio information, it is not enough to fully reflect the characteristics and changes of the target answer audio. It is more suitable for simple audio processing tasks, such as basic speech recognition or audio classification.

[0103] Optionally, feature extraction methods such as Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC) are used to extract acoustic features of the target answer audio from the original target answer audio.

[0104] The high-dimensional acoustic features are higher-dimensional acoustic feature representations obtained by further encoding, converting, and abstracting the acoustic features by the acoustic encoding module 210 based on the acoustic features.

[0105] Specifically, the acoustic coding module 210 performs convolution downsampling on the acoustic features of the target answer audio, and then obtains the high-dimensional acoustic features of the target answer audio through a multi-layer self-attention encoding mechanism. In the convolution downsampling process, a series of convolution layers are used to reduce the spatial dimension of the features while retaining important information; the self-attention encoding mechanism allows the model to take into account the information of all other frames when processing the current frame, thereby better capturing the contextual relationship in the speech signal.

[0106] The high-dimensional acoustic features obtained by the acoustic coding module 210 can capture the subtle changes and complex patterns in the target answer audio, and contain rich information in the target answer audio, including not only the local information of the target answer audio, but also the global context information of the target answer audio, and are more accurate and comprehensive in representing various aspects such as speech content and speech events, and numerically describe various aspects of the audio. High-dimensional acoustic features can capture key information in speech signals and provide strong support for subsequent processing and analysis.

[0107] Optionally, a pre-trained speech recognition model is used to construct the acoustic coding module 210, and specifically, the coding module in the pre-trained speech recognition model can be used. On the one hand, the training goal of the speech recognition model is to accurately convert speech into text, so the output of its coding module carries features that characterize the content information of the speech, which makes it easier for the subsequent modal adaptation module 220 to perform modal alignment, and the functions of each module remain independent of each other; on the other hand, the hidden layer output of the speech recognition model is a high-dimensional feature representation after multi-layer neural network processing. These features not only contain the acoustic characteristics of the speech, but also incorporate the model's understanding of the speech content. In addition, since these features have been generated by a well-trained speech recognition model, in the subsequent training process, the model parameters of the acoustic coding module 210 can be selected to be "frozen" (i.e., not updated), which can speed up the training convergence of the entire multi-modal large language model.

[0108] Next, the modality adaptation module 220 is described in detail.

[0109] The modality adaptation module 220 is used to convert the high-dimensional acoustic features into the text semantic space adopted by the large language model to obtain the final features of the target answer audio.

[0110] The input of the modality adaptation module 220 is the high-dimensional acoustic features output by the acoustic encoding module 210. The modality adaptation module 220 projects the input high-dimensional acoustic features to the text semantic space adopted by the embedding layer of the large language model 230, and obtains a feature representation in which each token dimension is consistent with the input dimension of the large language model, that is, the final features of the target answer audio. The final features of the target answer audio are already in the text semantic space adopted by the large language model 230 and can be directly understood and processed by the large language model 230. In this way, the large language model 230 can process inputs of both speech and text modalities at the same time.

[0111] The final features of the target answer audio are obtained by modal transformation of high-dimensional acoustic features. Therefore, the final features of the target answer audio also contain rich information in the target answer audio and can capture key information in all aspects of the speech signal.

[0112] Optionally, the modality adaptation module 220 adopts a network with a mainstream multi-layer self-attention encoding mechanism in speech recognition, such as Transformer and Conformer, etc., which is not limited in this application.

[0113] Optionally, the modality adaptation module 220 may use a series of feature mapping techniques to map high-dimensional acoustic features into text semantic space, such as linear transformation, nonlinear activation function, etc. Furthermore, in order to achieve effective modality conversion, the modality adaptation module 220 may use cross-modality learning methods, such as contrastive learning, multimodal fusion, etc., to learn the correspondence between acoustic features and text features.

[0114] Specifically, in this embodiment, a modality adaptation module 220 is pre-trained, and the trained modality adaptation module 220 can convert the high-dimensional acoustic features output by the acoustic encoding module 210 into feature representations in the text semantic space adopted by the large language model 230, thereby achieving cross-modal understanding and processing. As for the training process of the modality adaptation module 220, it will be described in subsequent embodiments and will not be described in detail here.

[0115] Next, the pre-trained large language model 230 is described in detail.

[0116] The pre-trained large language model 230 is used to determine the scoring result of the target answer audio according to the final features of the target answer audio and the first task instruction.

[0117] Large language model (LLM) refers to a generative deep neural network model based on the Transformer structure, which has powerful natural language processing capabilities. It should be noted that the embodiments of the present application can adopt any type of large language model in the prior art, such as GPT4, Llama, Qwen, iFlytek Spark, etc., and the present application does not limit this.

[0118] As an optional implementation method, the large language model 230 is a large language model obtained by fine-tuning any large language model in the prior art using a data set consisting of multiple groups of answer audio final features-first task instructions-labeled scoring results as training samples. After receiving the final features of the answer audio and the first task instructions, the fine-tuned large language model can perform the scoring task based on the guidance of the first task instructions, that is, according to the final features of the input answer audio and the first task instructions, determine the scoring result of the answer audio corresponding to the final features.

[0119] Specifically, the first task instruction input to the multimodal large language model and the final features of the target answer audio output by the modal adaptation module 220 are input to the pre-trained large language model 230. Relying on the excellent natural language processing capabilities of the pre-trained large language model, on the one hand, the large language model 230 can fully and accurately understand the final features of the target answer audio in the semantic space, and because the final features contain the multi-dimensional, comprehensive and rich feature representation of the target answer audio, it captures the key information in the target answer audio and covers the rich details of the target answer audio. Therefore, the large language model 230 can accurately capture the key information and potential contextual features in the target answer audio, and has a comprehensive and profound understanding of the target answer audio; on the other hand, the large language model 230 can deeply understand the first task instruction input therein, including but not limited to the description information of the scoring task, the title information and the scoring rules. Based on a comprehensive and profound understanding of the target answer audio and the first task instruction, the large language model 230 can accurately score the target answer audio and accurately generate a scoring result for the target answer audio.

[0120] Furthermore, on the one hand, in some scenarios, the audio sequence length of the target answer audio may be relatively long. For example, in the scenario of answering long-formula questions, the speech length of long-formula questions is usually about 1 to 3 minutes. Based on the conventional feature extraction configuration of 25ms frame length and 10ms frame shift, 3 minutes of speech has a total of 18,000 frames. For example, when the speech coding module is downsampled by 4 times, the high-dimensional acoustic features output by the speech coding module also have 4,500 frames. It is unrealistic to directly use all of these frames for acoustic representation or model input. Since the computational complexity and memory usage of the large language model will increase significantly with the increase of input length, the large language model usually has a limit on the length of input data. In actual applications, the sequence length of the final feature of the target answer audio and the sequence length of the first task instruction will exceed the input length limit of the large language model, resulting in the large language model being unable to process or the length of the first task instruction being limited.

[0121] On the other hand, there may be redundant information in the target answer audio, for example, the information between consecutive frames may be highly similar or repeated, some frames are background noise or silence, etc. Not every frame makes an important contribution to the final scoring result, and the redundant information in the target answer audio will increase the complexity of processing. Therefore, when processing the target answer audio, you can consider compressing the high-dimensional acoustic features of the target answer audio and only retain the frames containing important information in the high-dimensional acoustic features to meet the input length limit of the large language model, reduce the amount of calculation and improve processing efficiency.

[0122] Figure 3 A schematic diagram of the structure of another multimodal large language model provided in the embodiment of the application. Figure 3As shown, the multimodal large language model may further include a compression module 240. The compression module 240 is used to compress the high-dimensional acoustic features to obtain compressed features corresponding to the high-dimensional acoustic features.

[0123] The input of the compression module 240 is the high-dimensional acoustic features output by the acoustic encoding module 210. The compression module 240 compresses the input high-dimensional acoustic features and outputs the compressed features corresponding to the high-dimensional acoustic features of the target answer audio after compression processing.

[0124] Optionally, the compression module 240 adopts a CTC compressor, compresses the high-dimensional acoustic features using the blank tag output by CTC (Connectionist Temporal Classification), selects representative frames from the long audio signal of the high-dimensional acoustic features output by the acoustic coding module 210 for acoustic representation, generates and outputs compressed features of the target answer audio, thereby reducing the length of subsequent inputs and matching the audio and text durations to the same scale.

[0125] CTC is a loss function for sequence modeling that allows the model to insert any number of blank tokens into the output sequence, which indicate that the model did not output any valid characters at that point in time.

[0126] Optionally, audio compression is performed based on the blank tag, including: taking out the hidden layer output corresponding to the frame where the blank tag is located, thereby reducing the number of frames to be processed while retaining important information in the audio signal.

[0127] Optionally, audio compression is performed based on blank tags, including: for each group of frames between blank tags, average pooling is performed according to the time dimension, the average value of the frames between each group of blank tags is calculated, and this average value is used as a representative representation of the time period. The average value is considered to be able to represent the overall information of this group of audio frames. In this way, the number of frames to be processed can be further reduced while retaining important acoustic information, thereby achieving compression of high-dimensional acoustic features.

[0128] In this implementation, the modality adaptation module 220 is specifically used to convert the compressed features corresponding to the high-dimensional acoustic features into the text semantic space adopted by the large language model 230 to obtain the final features of the target answer audio.

[0129] Specifically, after the acoustic encoding module 210 outputs high-dimensional acoustic features, the high-dimensional acoustic features are input into the compression module 240. The compression module 240 compresses the input high-dimensional acoustic features and outputs compressed features. The compressed features are then input into the modal adaptation module 220. The modal adaptation module 220 converts the input compressed features into the text semantic space adopted by the large language model 230, outputs the final features of the target answer audio, and then inputs the final features of the target answer audio into the large language model 230 for subsequent processing.

[0130] In this implementation, optionally, the acoustic coding module 210 adopts a pre-trained speech recognition model with an audio compression branch (such as a CTC speech recognition branch). It can be a speech recognition model with an Encoder-Only structure that only uses audio compression (such as CTC) as the objective function, or it can be a speech recognition model in which the Encoder in the Encoder-Decoder structure adds an audio compression branch (such as a CTC branch). As for the specific model structure, this application does not limit it, and mainstream speech recognition model structures can be adopted, such as Transformer, Conformer, Whisper, etc.

[0131] As an optional implementation, the first training task and the second training task are used to optimize the parameters of the modal adaptation module in the multimodal large language model. During the execution of the first training task and the second training task, the parameters of the remaining modules in the multimodal large language model except the modal adaptation module are frozen, and only the parameters of the modal adaptation module are updated.

[0132] In the prior art, the following method is usually used to optimize the parameters of the modal adaptation module: the transcribed text of the sample audio is input into the multimodal large language model, and the multimodal large language model performs autoregression processing to generate a first processing result, the sample audio is input into the multimodal large language model, and the multimodal large language model performs autoregression to generate a second processing result, and then the parameters of the modal adaptation module are optimized according to the difference between the first processing result and the second processing result. If this method is directly used to optimize the parameters of the modal adaptation module in this application, the following technical problems exist:

[0133] On the one hand, the training task is not suitable: the training label used in this training method is the output of the transcribed text based on the sample audio after autoregression processing by the multimodal large language model. Because the transcribed text itself does not have any task attributes, the output of the multimodal large language model lacks actual physical meaning and is not suitable for the downstream scoring task. There is a mismatch between the training goal and the actual application scenario. On the other hand, the semantic space is not suitable: the sample audio used in this training method is quite different from the speech answer data of the scoring task in the semantic space. The high-dimensional acoustic features sent to the modal adaptation module and the transcribed text features of the sample audio sent to the large language model are not in the same semantic space as the actual downstream scoring task. After the high-dimensional acoustic features of the speech modality are sent to the modal adaptation module, the output of its semantic modality is not compatible with the scoring task in the semantic space; on the other hand, the input of the large language model is not compatible: in this training method, the input of the large language model is only the text modality output after the high-dimensional acoustic features of the speech modality are converted by the modal adaptation module, while during the subsequent downstream scoring task training, the input of the large language model also includes the task instruction, a pure text modality. According to the back-propagation mechanism of the neural network model training, the gradient is calculated layer by layer from back to front during training, and the error will be back-propagated from the large language model to the modal adaptation module to update the model parameters of the modal adaptation module. Therefore, the incompatibility of the large language model input will affect the training of the modal adaptation module.

[0134] Based on the above problems, the present application proposes a technical idea of ​​training the modal adaptation module using a first training task and a second training task. As an optional implementation method, the first training task includes: inputting the transcribed text of the first sample answer audio and the second sample task instruction into a pre-built multimodal large language model, and the multimodal large language model determines the first scoring result of the first sample answer audio based on the transcribed text of the first sample answer audio and the second sample task instruction, and the second sample task instruction instructs the execution of a scoring task including at least one scoring dimension.

[0135] The second training task includes: inputting the first sample answer audio and the second sample task instruction into the pre-constructed multimodal large language model, and the multimodal large language model determines a second scoring result for the first sample answer audio based on the first sample answer audio and the second sample task instruction; determining a first loss function value based on the first scoring result and the second scoring result; and performing parameter optimization on the modal adaptation module in the pre-constructed multimodal large language model based on the first loss function value.

[0136] Optionally, the second sample task instruction can be understood as instruction information instructing the multimodal large language model to perform a scoring task, and the scoring task includes at least one scoring dimension. The scoring task can be understood as a scoring task in the true sense, or as a scoring-related task, such as grammatical error correction, composition scoring, off-topic detection, context coherence evaluation, and other scoring-related tasks.

[0137] Specifically, the transcribed text of the first sample answer audio and the second sample task instruction are input into a pre-built multimodal large language model. Both are directly sent to the large language model, and the large language model determines and outputs a first scoring result for the first sample answer audio based on the transcribed text of the first sample answer audio and the second sample task instruction.

[0138] The first sample answer audio and the second sample task instruction are input into a pre-built multimodal large language model, and the first sample answer audio is sent to the acoustic coding module and the modal adaptation module in sequence to obtain the final features of the first sample answer audio, and then the final features of the first sample answer audio and the second sample task instruction are sent to the large language model, and the large language model determines and outputs a second scoring result for the first sample answer audio based on the final features of the first sample answer audio and the second sample task instruction.

[0139] The first scoring result and the second scoring result are compared to determine the difference between the two, and then determine the first loss function value. The first loss function value reflects the performance difference of the multimodal large language model when performing the same scoring task based on text modal answer data and voice modal answer data, that is, the performance difference when performing the same scoring task based on different modal answer data. This difference is mainly caused by the modal adaptation module executing the step of "converting the high-dimensional acoustic features to the text semantic space adopted by the large language model to obtain the final features of the target answer audio". Therefore, in the process of executing the first training task and the second training task, except for the modal adaptation module, the remaining module parameters of the multimodal large language model are frozen, and the first loss function value is used to optimize the parameters of the modal adaptation module.

[0140] In this implementation, on the one hand, the training sample includes a second sample task instruction, which is used to instruct the multimodal large language model to perform the scoring task during the training process. The output of the multimodal large language model is the scoring result of the sample answer audio, which is adapted to the subsequent scoring task; on the other hand, the first answer audio is used as the training sample, and the output of the modality adaptation module in the semantic mode is adapted to the scoring task in the semantic space; on the other hand, the training sample includes the first sample answer audio in the voice mode, and also includes the second task instruction in the text mode, which is adapted to the input of the large language model in the actual application. Therefore, the training method adopted in this embodiment can not only improve the modal mapping ability of the voice-text mode of the modality adaptation module, but also ensure that the mapped semantic mode is highly consistent with the scoring task, so that the multimodal large language model can play a more superior performance in the scoring task, thereby improving the accuracy of the audio answer scoring.

[0141] Furthermore, considering the data volume and text coverage of the first sample answer audio, an embodiment of the present application provides another training process for the modal adaptation module. As an optional implementation, in addition to the first training task and the second training task, the training task of the multimodal large language model also includes a third training task and a fourth training task. In this implementation, the modal adaptation module is optimized by combining the first training task and the second training task, and the modal adaptation module is optimized by combining the third training task and the fourth training task.

[0142] Among them, the third training task includes: inputting the annotated transcribed text of the first sample audio into a pre-built multimodal large language model, and the multimodal large language model generates a first result based on the annotated transcribed text of the first sample audio.

[0143] The fourth training task includes: inputting the first sample audio into the pre-constructed multimodal large language model, and the multimodal large language model generating a second result based on the first sample audio; determining a second loss function value according to the first result and the second result; and optimizing parameters of the modal adaptation module in the pre-constructed multimodal large language model according to the second loss function value.

[0144] The first sample audio may be obtained through various channels, including but not limited to public data sets, field recordings, etc. Optionally, the first sample audio may be any type of audio file containing language expressions, and may cover a variety of scenarios and contexts, not limited to answer audio.

[0145] The annotated transcribed text of the first sample audio can be understood as the manually annotated correct transcribed text of the first sample audio, which can accurately reflect the audio content of the first sample audio.

[0146] The modality adaptation module is trained by using the third training task and the fourth training task, so that the trained modality adaptation module has preliminary speech-text modality alignment capability.

[0147] Simply relying on the first and second training tasks may be limited by the data volume and text coverage of the first sample answer audio, while the first sample audio used in the third and fourth training tasks has a larger data volume and more comprehensive text coverage. This implementation method can provide more diverse training data for the model by introducing the third and fourth training tasks, which not only increases the data volume, but also provides audio samples in different scenarios and contexts, which helps the model to better generalize to unseen data, and achieves a balance between data volume, text coverage, and adaptability of the scoring task during the training process. This comprehensive optimization strategy helps the modal adaptation module to perform well in many aspects.

[0148] Optionally, the modality adaptation module is trained using a combination of the first training task and the second training task, or a combination of the third training task and the fourth training task. A simultaneous mixed training method, a staged training method, or a progressive optimization method may be used.

[0149] As an optional implementation method, a staged training method is adopted: after optimizing the parameters of the modal adaptation module based on the third training task and the fourth training task, the parameters of the modal adaptation module are optimized based on the first training task and the second training task.

[0150] Specifically, the training is carried out in two stages. The first stage includes optimizing the parameters of the modal adaptation module using the third training task and the fourth training task, and using the first sample audio to perform preliminary alignment of the speech-text modality. This setting is because the first sample audio data volume is large and the text coverage is wide, and the modal adaptation module after the first stage training has better versatility. The second stage includes optimizing the parameters of the modal adaptation module using the first training task and the second training task, and using the first sample answer audio and the second sample task instructions to perform further speech-text modality adaptation training on the modal adaptation module. The modal adaptation module after the second stage training is more suitable for downstream scoring tasks.

[0151] Optionally, during actual training, the first identifier (such as <bos>’s embedding), which is used to prompt the large language model to start outputting.

[0152] As an optional implementation method, the training task of the multimodal large language model also includes a fifth training task, and the fifth training task includes: inputting the second sample answer audio and the first sample task instruction into a pre-constructed multimodal large language model, and the multimodal large language model determines the scoring result of the second sample answer audio according to the second sample answer audio and the first sample task instruction, the first sample task instruction including description information of the scoring task, question information corresponding to the second sample answer audio, and scoring rules; determining a third loss function value according to the scoring result of the multimodal large language model for the second sample answer audio and the labeled scoring result corresponding to the second sample answer audio; and optimizing the parameters of the large language model in the pre-constructed multimodal large language model according to the third loss function value.

[0153] The second sample answer audio may be obtained through a variety of channels, including but not limited to public data sets, field recordings, etc. Optionally, the second sample answer audio covers a variety of voice answering scenarios.

[0154] The annotated scoring result can be understood as the correct scoring result of the manually annotated second sample answer audio. Optionally, the annotated scoring result is determined based on the sample scoring rule.

[0155] After fine-tuning the large language model in the fifth training task, the large language model can be equipped with the ability to score the answer audio. The parameters of the large language model are optimized according to the third loss function value, so that the scoring ability of the large language model for the answer audio can meet the actual needs. In this process, the parameters of other modules in the multimodal large language model except the large language model remain unchanged.

[0156] Exemplary Devices

[0157] Corresponding to the above-mentioned audio answer scoring method, the embodiment of the present application also provides an audio answer scoring device. Figure 4 is a structural diagram of an audio answer scoring device provided in an embodiment of the present application, such as Figure 4 As shown, the audio answer scoring device provided in the embodiment of the present application includes:

[0158] The first unit 410 is used to obtain a target answer audio and a first task instruction, where the first task instruction includes description information of a scoring task, question information corresponding to the target answer audio, and a scoring rule;

[0159] The second unit 420 is used to input the target answer audio and the first task instruction into a pre-trained multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio according to the target answer audio and the first task instruction;

[0160] Among them, the multimodal large language model is obtained by executing a first training task and a second training task, the first training task is used to score the first sample answer audio based on the transcribed text of the first sample answer audio, and the second training task is used to determine the score of the first sample answer audio by processing the first sample answer audio, and the scoring result of the first training task is used as the scoring label of the second training task.

[0161] The audio answer scoring device provided in the present application, on the one hand, uses a multimodal large language model to directly determine the scoring result of the target answer audio according to the target answer audio and the first task instruction, which can avoid the cascade error caused by the "speech recognition-text scoring" cascade scheme and the loss of non-text information of the target answer audio. Moreover, relying on the powerful understanding and analysis capabilities of the multimodal large language model, it can perform in-depth evaluation based on the target answer audio and the first task instruction, so as to score the target answer audio more accurately; on the other hand, using the first training task and the second training task to train the multimodal large language model not only strengthens the speech-text modality mapping capability of the multimodal large language model, but also ensures that the mapped semantic modality is highly consistent with the scoring task, so that the multimodal large language model can perform better in the scoring task, thereby improving the accuracy of the audio answer scoring.

[0162] Optionally, the multimodal large language model includes an acoustic encoding module, a modality adaptation module, and a pre-trained large language model, wherein:

[0163] The acoustic encoding module is used to encode the target answer audio to obtain high-dimensional acoustic features of the target answer audio;

[0164] The modality adaptation module is used to convert the high-dimensional acoustic features into the text semantic space adopted by the large language model to obtain the final features of the target answer audio;

[0165] The large language model is used to determine a scoring result for the target answer audio based on the final features and the first task instruction.

[0166] Optionally, the multimodal large language model further includes a compression module, and the compression module is used to compress the high-dimensional acoustic feature to obtain a compressed feature corresponding to the high-dimensional acoustic feature;

[0167] The modal adaptation module is specifically used to convert the compressed features corresponding to the high-dimensional acoustic features into the text semantic space adopted by the large language model to obtain the final features of the target answer audio.

[0168] Optionally, the first training task and the second training task are used to optimize parameters of the modality adaptation module in the multimodal large language model.

[0169] Optionally, the first training task includes:

[0170] Inputting the transcribed text of the first sample answer audio and the second sample task instruction into a pre-built multimodal large language model, the multimodal large language model determining a first scoring result for the first sample answer audio according to the transcribed text of the first sample answer audio and the second sample task instruction, wherein the second sample task instruction instructs to perform a scoring task including at least one scoring dimension;

[0171] The second training task includes:

[0172] Inputting the first sample answer audio and the second sample task instruction into the pre-built multimodal large language model, and the multimodal large language model determines a second scoring result for the first sample answer audio according to the first sample answer audio and the second sample task instruction;

[0173] Determine a first loss function value according to the first scoring result and the second scoring result;

[0174] According to the first loss function value, parameters of the modality adaptation module in the pre-built multimodal large language model are optimized.

[0175] Optionally, the training task of the multimodal large language model further includes a third training task and a fourth training task, wherein the third training task includes:

[0176] Inputting the annotated transcribed text of the first sample audio into a pre-built multimodal large language model, and allowing the multimodal large language model to generate a first result based on the annotated transcribed text of the first sample audio;

[0177] The fourth training task includes:

[0178] Inputting the first sample audio into the pre-built multimodal large language model, and allowing the multimodal large language model to generate a second result based on the first sample audio;

[0179] Determine a second loss function value according to the first result and the second result;

[0180] According to the second loss function value, parameters of the modality adaptation module in the pre-built multimodal large language model are optimized.

[0181] Optionally, after performing parameter optimization on the modality adaptation module based on the third training task and the fourth training task, performing parameter optimization on the modality adaptation module based on the first training task and the second training task.

[0182] Optionally, the training task of the multimodal large language model further includes a fifth training task, and the fifth training task includes:

[0183] Inputting the second sample answer audio and the first sample task instruction into a pre-built multimodal large language model, the multimodal large language model determining a scoring result for the second sample answer audio according to the second sample answer audio and the first sample task instruction, wherein the first sample task instruction includes description information of the scoring task, question information corresponding to the second sample answer audio, and scoring rules;

[0184] Determine a third loss function value according to a scoring result of the multimodal large language model on the second sample answer audio and a labeled scoring result corresponding to the second sample answer audio;

[0185] According to the third loss function value, parameters of the large language model in the pre-constructed multimodal large language model are optimized.

[0186] The audio answer scoring device provided in this embodiment belongs to the same application concept as the audio answer scoring method provided in the above embodiments of this application, and can execute the audio answer scoring method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the audio answer scoring method. For technical details not fully described in this embodiment, please refer to the specific processing content of the audio answer scoring method provided in the above embodiments of this application, and will not be repeated here.

[0187] The functions implemented by the above first unit 410 and the second unit 420 may be implemented by the same or different processors, respectively, and the embodiment of the present application is not limited thereto.

[0188] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0189] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0190] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0191] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0192] Exemplary Electronic Devices

[0193] Another embodiment of the present application also provides an electronic device, see Figure 5 As shown, the device includes:

[0194] Memory 200 and processor 210;

[0195] The memory 200 is connected to the processor 210 and is used to store programs;

[0196] The processor 210 is used to implement the audio answer scoring method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0197] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0198] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0199] A bus may include a pathway that transfers information between components of a computer system.

[0200] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0201] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0202] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0203] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0204] Output device 240 may include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0205] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0206] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any audio answer scoring method provided in the above embodiments of the present application.

[0207] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the audio answer scoring method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the embodiment of the audio answer scoring method mentioned above.

[0208] Exemplary computer program products and storage media

[0209] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the audio answer scoring method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0210] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0211] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the audio answer scoring method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0212] S101. Obtain a target answer audio and a first task instruction, wherein the first task instruction includes description information of a scoring task, question information corresponding to the target answer audio, and scoring rules.

[0213] S102, inputting the target answer audio and the first task instruction into a pre-trained multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio according to the target answer audio and the first task instruction;

[0214] Among them, the multimodal large language model is obtained by executing a first training task and a second training task, the first training task is used to score the first sample answer audio based on the transcribed text of the first sample answer audio, and the second training task is used to determine the score of the first sample answer audio by processing the first sample answer audio, and the scoring result of the first training task is used as the scoring label of the second training task.

[0215] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0216] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0217] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0218] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0219] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0220] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0222] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0223] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0224] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0225] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / bos>

Claims

1. An audio answer scoring method, characterized in that: The method comprises: Obtaining a target answer audio and a first task instruction, wherein the first task instruction includes description information of a scoring task, question information corresponding to the target answer audio, and a scoring rule; Inputting the target answer audio and the first task instruction into a pre-trained multimodal large language model, and allowing the multimodal large language model to determine a scoring result for the target answer audio according to the target answer audio and the first task instruction; The multimodal large language model includes a modality adaptation module, which converts the encoding features of the target answer audio into a text semantic space adopted by the multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio based on the encoding features converted to the text semantic space and the first task instruction; The multimodal large language model is obtained by performing a first training task and a second training task to optimize parameters of the modal adaptation module, wherein the first training task is used to score the first sample answer audio based on the transcribed text of the first sample answer audio, and the second training task is used to determine the score of the first sample answer audio by processing the first sample answer audio, and the scoring result of the first training task is used as the scoring label of the second training task.

2. The method according to claim 1, characterized in that: The multimodal large language model also includes an acoustic encoding module and a pre-trained large language model, wherein: The acoustic encoding module is used to encode the target answer audio to obtain high-dimensional acoustic features of the target answer audio; The modality adaptation module is used to convert the high-dimensional acoustic features into the text semantic space adopted by the large language model to obtain the final features of the target answer audio; The large language model is used to determine a scoring result for the target answer audio based on the final features and the first task instruction.

3. The method according to claim 2, characterized in that The multimodal large language model further includes a compression module, wherein the compression module is used to compress the high-dimensional acoustic feature to obtain a compressed feature corresponding to the high-dimensional acoustic feature; The modal adaptation module is specifically used to convert the compressed features corresponding to the high-dimensional acoustic features into the text semantic space adopted by the large language model to obtain the final features of the target answer audio.

4. The method according to any one of claims 1 to 3, characterized in that The first training task includes: Inputting the transcribed text of the first sample answer audio and the second sample task instruction into a pre-built multimodal large language model, the multimodal large language model determining a first scoring result for the first sample answer audio according to the transcribed text of the first sample answer audio and the second sample task instruction, wherein the second sample task instruction instructs to perform a scoring task including at least one scoring dimension; The second training task includes: Inputting the first sample answer audio and the second sample task instruction into the pre-built multimodal large language model, and the multimodal large language model determines a second scoring result for the first sample answer audio according to the first sample answer audio and the second sample task instruction; Determine a first loss function value according to the first scoring result and the second scoring result; According to the first loss function value, parameters of the modality adaptation module in the pre-built multimodal large language model are optimized.

5. The method according to any one of claims 1 to 3, characterized in that: The training task of the multimodal large language model also includes a third training task and a fourth training task, wherein the third training task includes: Inputting the annotated transcribed text of the first sample audio into a pre-built multimodal large language model, and generating a first result based on the annotated transcribed text of the first sample audio by the multimodal large language model; The fourth training task includes: Inputting the first sample audio into the pre-built multimodal large language model, and allowing the multimodal large language model to generate a second result based on the first sample audio; Determine a second loss function value according to the first result and the second result; According to the second loss function value, parameters of the modality adaptation module in the pre-built multimodal large language model are optimized.

6. The method according to claim 5, characterized in that After the parameters of the modality adaptation module are optimized based on the third training task and the fourth training task, the parameters of the modality adaptation module are optimized based on the first training task and the second training task.

7. The method according to claim 2 or 3, characterized in that: The training task of the multimodal large language model also includes a fifth training task, and the fifth training task includes: Inputting the second sample answer audio and the first sample task instruction into a pre-built multimodal large language model, the multimodal large language model determining a scoring result for the second sample answer audio according to the second sample answer audio and the first sample task instruction, wherein the first sample task instruction includes description information of the scoring task, question information corresponding to the second sample answer audio, and scoring rules; Determine a third loss function value according to a scoring result of the multimodal large language model on the second sample answer audio and a labeled scoring result corresponding to the second sample answer audio; According to the third loss function value, parameters of the large language model in the pre-constructed multimodal large language model are optimized.

8. An audio answer scoring device, characterized in that: include: The first unit is used to obtain a target answer audio and a first task instruction, wherein the first task instruction includes description information of a scoring task, question information corresponding to the target answer audio, and a scoring rule; The second unit is used to input the target answer audio and the first task instruction into a pre-trained multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio according to the target answer audio and the first task instruction; The multimodal large language model includes a modality adaptation module, which converts the encoding features of the target answer audio into a text semantic space adopted by the multimodal large language model, and the multimodal large language model determines a scoring result for the target answer audio based on the encoding features converted to the text semantic space and the first task instruction; The multimodal large language model is obtained by performing a first training task and a second training task to optimize parameters of the modal adaptation module, wherein the first training task is used to score the first sample answer audio based on the transcribed text of the first sample answer audio, and the second training task is used to determine the score of the first sample answer audio by processing the first sample answer audio, and the scoring result of the first training task is used as the scoring label of the second training task.

9. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the audio answer scoring method as described in any one of claims 1-7 by running the program in the memory.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the audio answer scoring method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to implement the audio answer scoring method as described in any one of claims 1 to 7.

Images