A cheating detection method, device, equipment, storage medium and product
By a combination of semantic spatial features of audio and large language model, the problem of insufficient cheat detection accuracy and applicability in audio answering scenarios in the prior art is solved, and the accurate identification and adaptability of cheating behavior is achieved.
Patent Information
- Application Number
- CN202411154397.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The prior art has limited accuracy and applicability of cheat detection in audio answer scenarios, making it difficult to adapt to complex and diverse answer scenarios and cheating methods, which affects the accuracy of cheat detection and educational equity.
By obtaining the semantic spatial feature representation of the target answer audio, and using a pre-trained large language model combined with cheat judgment rules, cheat detection is performed to adapt to different answer scenarios and cheating methods.
Accurate detection of cheating behavior is achieved, the adaptability and robustness of cheating detection is improved, and the fairness of online answers and educational equity is ensured.
Smart Images

Figure CN118658467B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and in particular to a cheating detection method, device, equipment, storage medium and product. Background Art
[0002] With the continuous advancement of educational technology, new educational models such as online exams and remote answering are becoming increasingly popular. However, this increase in convenience is also accompanied by an increase in cheating, especially in audio answering scenarios, where cheating methods emerge in an endless stream, posing a severe challenge to educational equity. In order to maintain the fairness of online answering, strict cheating detection must be performed on the answering audio.
[0003] Traditional cheating detection methods mostly rely on manual monitoring or simple audio feature comparison. Manual monitoring methods are inefficient, and although audio feature comparison methods can detect cheating behavior to a certain extent, their detection accuracy and applicability are limited. They are difficult to adapt to complex and diverse answering scenarios and cheating methods. This not only affects the accuracy of cheating detection, but also makes it difficult to effectively maintain the fairness of online answering and educational equity. Summary of the invention
[0004] In view of this, the embodiments of the present application are committed to providing a cheating detection method, device, equipment, storage medium and product, which can not only accurately identify cheating behavior in the target answer audio, thereby accurately detecting cheating behavior, but also adapt to different answer scenarios and cheating methods, thereby improving the adaptability and robustness of cheating detection.
[0005] According to a first aspect of an embodiment of the present application, a cheating detection method is provided, comprising:
[0006] Obtaining a target answer audio and first indication information, wherein the first indication information includes task information of a cheating detection task and a cheating judgment rule; the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result;
[0007] Determining a first feature representation of the target answer audio in a semantic space;
[0008] The first feature representation and the first indication information are input into a pre-trained large language model, and the large language model determines a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0009] Optionally, the first indication information also includes question information, the question information includes the question and / or reference answer, and the cheating judgment rule includes a rule for making a cheating judgment when the target answer audio is correct.
[0010] Optionally, determining a first feature representation of the target answer audio in a semantic space includes:
[0011] Encoding the target answer audio to obtain a second feature representation of the target answer audio in the speech space;
[0012] The second feature representation is converted into a semantic space to obtain a first feature representation of the target answer audio in the semantic space.
[0013] Optionally, determining a first feature representation of the target answer audio in a semantic space, and inputting the first feature representation and the first indication information into a pre-trained large language model, and determining, by the large language model, a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information, includes:
[0014] The target answer audio and the first indication information are input into a pre-trained multimodal large language model so that the multimodal large language model determines a first feature representation of the target answer audio in a semantic space, and determines a cheating detection result corresponding to the target answer audio based on the first feature representation and the first indication information.
[0015] Optionally, the multimodal large language model includes a feature extraction module, a modality conversion module and a pre-trained large language model, wherein:
[0016] The feature extraction module is used to encode the target answer audio to obtain a second feature representation of the target answer audio in the speech space;
[0017] The modality conversion module is used to convert the second feature representation into a semantic space to obtain a first feature representation of the target answer audio in the semantic space;
[0018] The large language model is used to determine a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0019] Optionally, the training process of the multimodal large language model includes:
[0020] Obtaining a first sample audio and an annotated transcription of the first sample audio;
[0021] Inputting the first sample audio into a pre-built multimodal large language model, and generating a first result based on the first sample audio by the pre-built multimodal large language model; and inputting the annotated transcribed text into the pre-built multimodal large language model, and generating a second result based on the annotated transcribed text by the pre-built multimodal large language model;
[0022] Determine a first loss function value according to the first result and the second result;
[0023] According to the first loss function value, parameters of the modality conversion module in the pre-built multimodal large language model are optimized.
[0024] Optionally, the training process of the multimodal large language model includes:
[0025] Acquire multiple groups of training samples and label texts corresponding to each group of training samples, each group of training samples includes a second sample audio and second sample indication information corresponding to the second sample audio, the second sample indication information of each group of training samples indicates any one of speech processing tasks, and the second sample indication information of each group of training samples indicates different speech processing tasks, and the speech processing tasks include at least one of speech recognition, voiceprint recognition, sound event detection, and spoken language comprehension;
[0026] Inputting the plurality of groups of training samples into a pre-built multimodal large language model, and having the multimodal large language model perform speech processing on the second sample audio of each group of training samples according to the second sample indication information in the group of training samples, to generate a processing result corresponding to each group of training samples;
[0027] Determine the loss function value corresponding to each group of training samples according to the processing results corresponding to each group of training samples and the label text corresponding to each group of training samples;
[0028] Determine a second loss function value according to the loss function value corresponding to each group of training samples;
[0029] According to the second loss function value, parameters of the feature extraction module in the pre-built multimodal large language model are optimized.
[0030] Optionally, the training process of the multimodal large language model includes:
[0031] Obtaining a sample answer audio, a sample first indication information, and a marked cheating result corresponding to the sample answer audio;
[0032] Inputting the sample answer audio and the sample first indication information into a pre-built multimodal large language model to obtain a cheating detection result corresponding to the sample answer audio;
[0033] Determine a third loss function value according to the cheating detection result corresponding to the sample answer audio and the labeled cheating result of the sample answer audio;
[0034] According to the third loss function value, parameters of the large language model in the pre-constructed multimodal large language model are optimized.
[0035] According to a second aspect of an embodiment of the present application, a cheating detection device is provided, comprising:
[0036] The first unit is used to obtain a target answer audio and first indication information, wherein the first indication information includes task information of a cheating detection task and a cheating judgment rule; the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result;
[0037] A second unit is used to determine a first feature representation of the target answer audio in a semantic space;
[0038] The third unit is used to input the first feature representation and the first indication information into a pre-trained large language model, and the large language model determines a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0039] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a memory and a processor;
[0040] The memory is connected to the processor and is used to store programs;
[0041] The processor is used to implement the cheating detection method as described in any one of the first aspects of the embodiments of the present application by running the program in the memory.
[0042] According to a fourth aspect of the embodiments of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the cheating detection method as described in any one of the first aspects of the embodiments of the present application is implemented.
[0043] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the cheating detection method as described in any one of the first aspects of the embodiments of the present application.
[0044] This application first determines the first feature representation of the target answer audio in the semantic space, which contains detailed feature information of the target answer audio. Then, by using the powerful semantic understanding and analysis capabilities of the large language model, by analyzing the first feature representation of the target answer audio in the semantic space, it can capture the key information and potential contextual features in the target answer audio, such as the specific audio content, the sound events contained, and the emotions expressed, etc., and combine this information with the cheating judgment rules in the input indication information. The large language model can accurately identify cheating behavior in the target answer audio based on a full understanding of the relevant information of the target answer audio and the cheating judgment rules, thereby achieving accurate detection of cheating behavior. In addition, this application can flexibly adjust the cheating judgment rules according to different answering scenarios and cheating methods, so that it can adapt to different answering scenarios and cheating methods, and improve the adaptability and robustness of cheating detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0046] Figure 1 A flowchart of a cheating detection method provided in an embodiment of the present application.
[0047] Figure 2 A flowchart of a cheating detection method provided in another embodiment of the present application.
[0048] Figure 3 A structural example diagram of a multimodal large language model provided in an embodiment of the present application.
[0049] Figure 4 A flowchart of a model training method provided in an embodiment of the present application.
[0050] Figure 5 A flowchart of a model training method according to another embodiment of the present application.
[0051] Figure 6 A flowchart of a model training method according to another embodiment of the present application.
[0052] Figure 7 A schematic diagram of the structure of a cheating detection device provided in an embodiment of the present application.
[0053] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] The technical solution of the embodiment of the present application is suitable for use in various scenarios where cheating detection is required for answering audio, such as online oral examinations, oral ability assessments, voice competitions, remote answering and other scenarios. In these scenarios, in order to maintain the fairness of the answers and the accuracy of the assessment, it is necessary to detect cheating on the answering audio to determine whether the answerer has cheated when answering. The technical solution of the embodiment of the present application can accurately identify cheating behavior in the target answering audio, thereby achieving accurate detection of cheating behavior, and can also adapt to different answering scenarios and cheating methods, improving the adaptability and robustness of cheating detection.
[0055] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.
[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. The described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0057] Before introducing this application solution, the relevant technologies are first introduced:
[0058] With the continuous advancement of educational technology, new educational models such as online exams and remote answering are becoming increasingly popular. However, this increase in convenience is also accompanied by an increase in cheating, especially in audio answering scenarios, where cheating methods emerge in an endless stream, posing a severe challenge to educational equity. In order to maintain the fairness of online answering, strict cheating detection must be performed on the answering audio.
[0059] Traditional cheating detection methods mostly rely on manual monitoring or simple audio feature comparison. Manual monitoring methods are inefficient, and although audio feature comparison methods can detect cheating behavior to a certain extent, their detection accuracy and applicability are limited. They are difficult to adapt to complex and diverse answering scenarios and cheating methods. This not only affects the accuracy of cheating detection, but also makes it difficult to effectively maintain the fairness of online answering and educational equity.
[0060] In view of this, the embodiments of the present application are committed to providing a cheating detection method, device, equipment, storage medium and product, which can not only accurately identify cheating behavior in the target answer audio, thereby accurately detecting cheating behavior, but also adapt to different answer scenarios and cheating methods, and improve the adaptability and robustness of cheating detection, which are described in detail one by one in the following embodiments.
[0061] Exemplary Methods
[0062] Figure 1 A schematic diagram of a cheating detection method provided in an embodiment of the present application. Figure 1 As shown, the cheating detection method provided in this embodiment includes steps S101-S103:
[0063] S101. Obtain target answer audio and first indication information, where the first indication information includes task information of a cheating detection task and a cheating judgment rule, and the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result.
[0064] The target answer audio can be understood as the answer audio file that needs to be detected for cheating. The target answer audio can come from various scenarios that require evaluation of voice answers, such as online language tests, oral tests, voice competitions, etc. In these scenarios, the answerer submits the answer content in the form of voice, and the voice data submitted by the answerer is the target answer audio. The system needs to perform cheating detection on the target answer audio to determine whether the answerer has cheated during the answering process.
[0065] This application does not limit the method of obtaining the target answer audio, which is determined according to the actual application scenario and needs. For example, the audio file can be downloaded or received from the online form or platform submitted by the respondent, the respondent's voice can be recorded in real time through a dedicated audio acquisition device or software, and it can also be retrieved and extracted from a stored audio database.
[0066] The first indication information may be understood as indication information for guiding the object receiving the information to output the cheating detection result. Optionally, the first indication information is text information.
[0067] The first indication information includes task information of the cheating detection task and cheating judgment rules.
[0068] Among them, the task information of the cheating detection task can be understood as the task description information of the cheating detection task, so that the object receiving the information can clearly understand and execute the cheating detection task, perform cheating detection on the target answer audio, and determine the cheating detection result. The task information of the cheating detection task may include the execution identity, task background, detection target, detection accuracy, etc., which provides guidance and direction for cheating detection. This application does not limit the specific content and format of the task information of the cheating detection task, which is determined according to the actual application scenario and needs. For example, the task information of the cheating detection task can be "Your task is to determine whether there is cheating behavior in the provided answer audio according to the provided cheating judgment rules, and output the cheating detection result."
[0069] The cheating judgment rules can be understood as rules and standards for judging whether there is cheating in the answer audio and / or what kind of cheating exists. The cheating judgment rules are determined according to actual needs and goals, and are intended to determine whether there is cheating in the answer audio and / or what kind of cheating exists through set conditions, algorithms, parameters and other information. This application does not limit the specific content and format of the task information of the cheating detection task, which is determined according to the actual application scenario and needs.
[0070] Optionally, the cheating detection rule includes a cheating behavior definition, which defines multiple cheating behaviors. As long as one of the cheating behaviors is met, it can be determined that the target answer audio has a cheating suspicion. Exemplarily, a definition of a cheating behavior can be "the main answerer does not speak, the answer of others is recorded, and the answer of others is correct."
[0071] As an optional implementation manner, obtaining the first indication information includes: obtaining the first indication information in response to a user input of the first indication information.
[0072] As an optional implementation method, obtaining the first indication information includes: obtaining task information and cheating judgment rules of the cheating detection task, and generating the first indication information according to a preset indication information format based on the task information and cheating detection rules of the cheating detection task.
[0073] Optionally, in response to the user's input operation, the task information and / or cheating judgment rule of the cheating judgment task is obtained. Optionally, according to the actual application scenario, the task information of the cheating detection task suitable for the task is selected from the preset task information library and / or the cheating judgment rule is selected from the preset rule library. Optionally, through the machine learning model, based on historical cheating cases and detection experience, the task information and / or cheating judgment rule of the cheating detection task is automatically generated.
[0074] S102: Determine a first feature representation of the target answer audio in a semantic space.
[0075] After obtaining the target answer audio, it is necessary to perform feature extraction and modal conversion processing on the target answer audio to convert the target answer audio into a format that can be understood and processed by the large language model.
[0076] The first feature representation can be understood as a set of feature vectors extracted from the target answer audio that can reflect the key information of the target answer audio, and is a comprehensive feature representation in the semantic space. The first feature representation not only includes features that can characterize the surface information of the target answer audio (such as volume, sound intensity, spectrum, etc.), but also includes features that can characterize the deep information of the target answer audio (such as specific audio content, sound events, emotional expression, etc.). It characterizes the target answer audio from multiple different dimensions, contains rich details of the target answer audio, and can more comprehensively reflect the essential attributes of the target answer audio.
[0077] As an optional implementation, step S102 "determining a first feature representation of the target answer audio in the semantic space" includes steps A1-A2:
[0078] A1. Encode the target answer audio to obtain a second feature representation of the target answer audio in the speech space.
[0079] Optionally, after obtaining the target answer audio, the target answer audio is encoded and processed to convert the original speech waveform in the target answer audio into a high-dimensional feature vector, i.e., the second feature representation. The second feature representation is a comprehensive feature representation that includes multi-dimensional features of the target answer audio, including features that can characterize the surface information of the target answer audio (such as volume, sound intensity, spectrum, etc.), as well as features that can characterize the deep information of the target answer audio (such as specific audio content, timing information, pause information, emotional information, etc.). The second feature representation captures the key information in the target answer audio, covers the rich details of the target answer audio, and can more comprehensively reflect the essential attributes of the target answer audio.
[0080] Optionally, the target answer audio is encoded, including: performing convolution downsampling on the target answer audio to reduce the dimension of the data while retaining important local features in the target answer audio; and then further processing the convolution downsampled data using a multi-layer self-attention encoding mechanism to capture the temporal dependencies and contextual information in the speech signal through the self-attention mechanism, thereby generating a high-dimensional feature representation, i.e., the second feature representation.
[0081] Optionally, the target answer audio is encoded using a pre-trained acoustic encoder. The trained acoustic encoder can encode the target answer audio and extract multi-dimensional comprehensive features of the target answer audio.
[0082] Optionally, the acoustic encoder can be implemented based on a deep RNN (Recurrent Neural Network) or Transformer (transformer) structure. Optionally, the acoustic encoder can be an acoustic model of a speech recognizer, such as Whisper's Encoder, or a self-supervised model pre-trained with massive unsupervised speech, such as wav2vec2.0, huBERT, wavLM, etc.
[0083] A2. Convert the second feature representation into a semantic space to obtain a first feature representation of the target answer audio in the semantic space.
[0084] In step A1, a second feature representation of the target answer audio in the speech space is obtained. Because the large language model receives and processes input in text mode, in order for the large language model to understand and process the feature representation of the target answer audio, the second feature representation needs to be mapped to a semantic space that the large language model can process (for example, the feature space used by the embedding layer of the large language model).
[0085] Specifically, after obtaining the second feature representation of the target answer audio in the speech space, the second feature representation is processed, and the second feature representation in the speech space is mapped to the semantic space that can be understood by the large language model to obtain the first feature representation of the target answer audio in the semantic space.
[0086] Optionally, a pre-trained modality adapter is used to project the second feature representation from the speech space to the semantic space. The trained modality adapter can perform conversion from the speech modality to the semantic modality.
[0087] Optionally, the modal adapter can be implemented using a linear projection layer. By pre-training the weight parameters of the linear projection layer, the modal adapter after training can convert the second feature representation of the target answer audio in the speech space into the first feature representation of the target answer audio in the semantic space. In addition to the linear projection layer, other methods such as kernel functions can also be used to implement the modal adapter.
[0088] As an optional implementation method, step S102 "determining the first feature representation of the target answer audio in the semantic space" includes: inputting the target answer audio into a trained feature representation generation model, and the feature representation generation model can output the first feature representation of the target answer audio in the semantic space.
[0089] Specifically, a feature representation generation model is pre-trained in the embodiment of the present application. Optionally, the feature representation generation model can be obtained by performing feature representation generation training on a first training sample, wherein the first training sample includes sample audio and a first feature representation label corresponding to the sample audio. The feature representation generation training is performed so that the feature representation generation model can determine the first feature representation of the sample audio in the first training sample in the semantic space, and the goal is that the loss function determined based on the first feature representation output by the model and the first feature representation label meets preset requirements. The trained feature representation generation model can generate a first feature representation of the input audio in the speech space.
[0090] Optionally, the feature representation generation model can be implemented based on a variety of deep learning neural network structures, such as CNN (Convolutional Neural Networks) network structure, RNN (Recurrent Neural Networks) network structure, CRNN (Convolutional Recurrent Neural Network) network structure, etc., which is not limited in this application.
[0091] S103: input the first feature representation and the first indication information into a pre-trained large language model, and the large language model determines a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0092] The cheating detection result can be understood as the task execution result of the large language model performing the cheating detection task on the target answer audio, which can characterize the cheating situation of the target answer audio. The specific content and form of the cheating detection result are determined according to the actual application scenario and requirements, and this application does not limit this.
[0093] Optionally, the cheating detection result includes a binary classification result of "cheating exists" or "cheating does not exist". Optionally, in addition to determining whether cheating occurs, the large language model can further identify the specific type of cheating, and the cheating detection result includes the type of cheating. Optionally, the large language model can evaluate the severity of cheating, for example, by using a scoring system (such as 1-10 points or in the form of a percentage) to indicate the severity of cheating, and the cheating detection result includes the severity of cheating. Optionally, the cheating detection result can also include a detailed cheating detection report, including the specific time point of cheating, a detailed description of the cheating type, and possible suggestions for improvement.
[0094] A large language model (LLM) refers to a generative deep neural network model based on a Transformer structure, which has a powerful natural language processing capability. It should be noted that any large language model can be used in the embodiments of the present application.
[0095] As an optional implementation manner, the pre-trained large language model may be any type of large language model in the prior art.
[0096] As an optional implementation, the pre-trained large language model can be a large language model obtained by training any large language model in the prior art using a data set consisting of multiple sets of first feature representations, first indication information, and labeled cheating detection results as training samples. After receiving the input first feature representation and first indication information, the fine-tuned large language model can perform the cheating detection task based on the guidance of the first indication information.
[0097] Specifically, after obtaining the first feature representation of the target answer audio in the semantic space and the first indication information guiding the execution of the cheating detection task through steps S101 and S102, the first feature representation of the target answer audio in the semantic space and the first indication information guiding the execution of the cheating detection task are input into the pre-trained large language model. The large language model will execute the cheating detection task based on the guidance of the first indication information, that is, determine the cheating detection result corresponding to the target answer audio according to the input first feature representation and the first indication information.
[0098] In this way, relying on the excellent natural language processing capabilities of the pre-trained large language model, on the one hand, the large language model can fully and accurately understand the first feature representation of the target answer audio in the semantic space. Because the first feature representation contains the multi-dimensional features of the target answer audio, it captures the key information in the target answer audio and covers the rich details of the target answer audio. Therefore, the large language model can accurately capture the key information and potential contextual features in the target answer audio, such as the specific audio content, the sound events contained, and the emotions expressed, and has a comprehensive and profound understanding of the target answer audio; on the other hand, the large language model can deeply understand the cheating judgment rules in the input first indication information. Based on a comprehensive and profound understanding of the target answer audio and the cheating judgment rules, the large language model can accurately detect cheating on the target answer audio and accurately generate cheating detection results corresponding to the target answer audio.
[0099] Moreover, this solution is applicable to any answer audio and cheating judgment rule. It only needs to determine the target answer audio and its corresponding cheating judgment rule to achieve cheating detection with the help of the super power of the large language model, which has high universality. For example, the first feature representation of the target answer audio and the task information of the cheating detection task in the first indication information are used as the Prompt of the large language model to ensure that the Prompt covers the comprehensive information of the target answer audio, and the cheating judgment rule is set as a parameter of the Prompt, so as to filter different cheating behaviors according to different scenarios and improve the flexibility of the cheating detection system.
[0100] This application first determines the first feature representation of the target answer audio in the semantic space, and the first feature representation contains detailed feature information of the target answer audio in multiple dimensions. Then, by using the powerful semantic understanding and analysis capabilities of the large language model, by analyzing the first feature representation of the target answer audio in the semantic space, it can capture the key information and potential contextual features in the target answer audio, such as the specific audio content, the sound events contained, and the emotions expressed, etc., and combine this information with the cheating judgment rules in the input indication information. The large language model can accurately identify cheating behavior in the target answer audio based on a full understanding of the relevant information of the target answer audio and the cheating judgment rules, thereby achieving accurate detection of cheating behavior. In addition, this application can flexibly adjust the cheating judgment rules according to different answering scenarios and cheating methods, so that it can adapt to different answering scenarios and cheating methods, and improve the adaptability and robustness of cheating detection.
[0101] As an optional implementation method, in actual application scenarios, the cheating detection method provided in the embodiment of the present application can be used in a human-machine coupling manner. For example, the system detects data suspected of cheating and whose machine score is higher than a certain threshold, and then manually confirms whether there is cheating behavior. Conclusion.
[0102] As an optional implementation manner, the first indication information not only includes task information of the cheating detection task and basic cheating judgment rules, but also further includes question information, and the question information includes the question and / or reference answer.
[0103] The introduction of question information provides the large language model with richer background knowledge, which helps the large language model to more accurately grasp the key content of the target answer audio, and provides more context and reference for cheating detection, thereby improving the accuracy and reliability of detection.
[0104] Specifically, first, the question information provides the large language model with richer background knowledge. In the cheating detection task, the question information serves as a guide and framework for answering, providing the large language model with an important basis for judging whether the answer is compliant or deviates from the topic. Through the question information, the large language model can have a deeper understanding of the background and context of the answer audio, so as to more accurately identify the key information and potential problems. Secondly, the introduction of question information enables the large language model to more accurately understand the relevant content of the target answer audio. The provision of question information establishes a clear reference system for the large language model, enabling it to more accurately analyze the correlation between the content in the answer audio and the question, thereby more effectively identifying cheating behavior.
[0105] Furthermore, in some scenarios, when the answer content of the answer audio is incorrect, the impact of cheating on the final result (such as whether the respondent passes the assessment, whether he is hired, etc.) becomes secondary. Specifically, if the respondent's answer is wrong, then even if the respondent obtains these answer information through cheating, it will not effectively improve his performance. In this case, the wrong answer itself has determined the unfavorable result, and the cheating behavior is only an additional improper behavior on this basis, which will not change the core result, and the impact of cheating on the final evaluation result is limited. However, if the respondent obtains the correct answer result through cheating, it will seriously undermine the fairness and impartiality of the assessment or evaluation. Therefore, in these scenarios, when the target answer audio is wrong, the standard for cheating judgment can be relatively relaxed, or even no cheating judgment can be made, but the main energy can be used to detect cheating on the correct answer audio, so as to allocate and utilize limited resources more reasonably and improve the efficiency of cheating detection.
[0106] In the above scenario, the cheating judgment rule may include a rule for making a cheating judgment when the target answer audio is correct, and the rule specifies the rules and standards for judging whether there is cheating behavior and / or what kind of cheating behavior exists in the answer audio when the target answer audio is correct. Optionally, the rule specifies that no cheating judgment is made when the target answer audio is wrong.
[0107] Optionally, the rule specifies various definitions of cheating behaviors that may exist when the target answer audio is answered correctly.
[0108] Exemplarily, the first instruction information is "I expect you to play the role of a teacher. This audio clip comes from a student's answer in an oral test, and the question information corresponding to the question is xxx. Please combine the question information and the audio to determine whether there is cheating in the student's answer. If so, output 1, otherwise output 0. Cheating behavior is defined as follows, and there is suspicion of cheating if any one of the following is met: A. The main answerer does not speak, but records the sound of others' answers, and the answers of others are correct; B. The main answerer speaks first, gives an incorrect answer, then records the sound of others' answers, and the answer is correct, and the main answerer then changes the answer to the correct one; C. The main answerer does not speak at first, but records the sound of others' answers, and the answer is correct, and the main answerer then answers and the answer is correct."
[0109] Figure 2 This is a flow chart of a cheating detection method provided by another embodiment of the present application. Figure 2 As shown, the cheating detection method provided in the embodiment includes steps S201-S202:
[0110] S201. Obtain target answer audio and first indication information, where the first indication information includes task information of a cheating detection task and a cheating judgment rule; the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result.
[0111] Step S201 corresponds to the content of step S101 in the aforementioned embodiment. The specific content of step S201 can refer to step S101 and will not be described in detail here.
[0112] S202. Input the target answer audio and the first indication information into a pre-trained multimodal large language model, so that the multimodal large language model determines a first feature representation of the target answer audio in a semantic space, and determines a cheating detection result corresponding to the target answer audio based on the first feature representation and the first indication information.
[0113] The Multimodal Large Language Model (MLLM) is a model that further introduces multimodal information processing capabilities based on LLM. It can process and understand data input from different modalities, such as text, images, audio, and video, and learn the correlation between different modalities through joint training, thereby providing richer and more accurate language output.
[0114] As an optional implementation, the multimodal large language model may adopt any type of large language model in the prior art that has bimodal processing capabilities of text modality and audio modality.
[0115] As an optional implementation method, the pre-trained multimodal large language model can be a fine-tuned multimodal large language model obtained by training any multimodal large language model with dual-modal processing capabilities of text modality and audio modality in the prior art using a data set consisting of multiple groups of answer audio-first indication information-labeled cheating detection results as training samples.
[0116] After receiving the input answer audio and the first indication information, the pre-trained large language model can determine the first feature representation of the target answer audio in the semantic space, and determine and output the cheating detection result corresponding to the target answer audio based on the first feature representation and the first indication information.
[0117] As an optional implementation, Figure 3 As shown, the multimodal large language model includes a feature extraction module 310 , a modality conversion module 320 and a pre-trained large language model 330 .
[0118] The feature extraction module 310 is used to encode the target answer audio to obtain a second feature representation of the target answer audio in the speech space.
[0119] Specifically, in response to inputting the target answer audio and the first indication information into the multimodal large language model, the feature extraction module 310 encodes the target answer audio input into the multimodal large language model to obtain a second feature representation of the target answer audio, and inputs the second feature representation into the modal conversion module 320. As for the specific content of "encoding the target answer audio to obtain a second feature representation of the target answer audio in the speech space", reference can be made to the content of step A1 in the aforementioned embodiment, which will not be elaborated here.
[0120] Optionally, the feature extraction module 310 may use an acoustic encoder. The acoustic encoder may be implemented based on a deep RNN (Recurrent Neural Network) or Transformer (transformation network) structure. Optionally, the acoustic encoder may be an acoustic model of a speech recognizer, such as Whisper's Encoder, or a self-supervised model pre-trained with massive unsupervised speech, such as wav2vec2.0, huBERT, wavLM, etc.
[0121] The modality conversion module 320 is used to convert the second feature representation into a semantic space to obtain a first feature representation of the target answer audio in the semantic space.
[0122] Specifically, in response to the input of the second feature representation of the target answer audio, the modal conversion module 320 performs spatial mapping on the input second feature representation, mapping it from the speech space to the semantic space, and obtains the first feature representation of the target answer audio in the semantic space. As for the specific content of "converting the second feature representation to the semantic space to obtain the first feature representation of the target answer audio in the semantic space", reference can be made to the content of step A2 in the aforementioned embodiment, which will not be repeated here.
[0123] Optionally, the modality conversion module 320 may use a modality adapter. The modality adapter may be implemented using a linear projection layer or other methods such as a kernel function.
[0124] The large language model 330 is used to determine a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0125] Specifically, after determining the first feature representation, the first feature representation and the first indication information are input into the large language model 330, and the large language model 330 determines the cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information. As for the specific content of "determining the cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information", the content of step S103 in the aforementioned embodiment can be referred to, and no further description is given here.
[0126] Furthermore, after constructing the multimodal large language model, the parameters of each module in the multimodal large model need to be initialized. Exemplarily, the parameters of the feature extraction module 310 are initialized using the existing speech recognition model parameters or the open source Whisper Encoder, the parameters of the large language model 330 are initialized using the parameters of the open source large models such as Llama3 and Qwen or the parameters of the iFlytek Spark large model, and the parameters of the modal conversion module 320 are randomly initialized. The multimodal large language model structure obtained by the initialization is universal.
[0127] Figure 4 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the training process includes steps S401-S404:
[0128] S401: Obtain a first sample audio and an annotated transcribed text of the first sample audio.
[0129] Optionally, the first sample audio may be obtained through a variety of channels, including but not limited to public data sets, field recordings, etc. Optionally, the first sample audio covers a variety of scenarios and contexts.
[0130] The annotated transcribed text of the first sample audio is a correct transcribed text of the first sample audio and can accurately reflect the audio content of the first sample audio.
[0131] S402: input the first sample audio into a pre-built multimodal large language model, and the pre-built multimodal large language model generates a first result based on the first sample audio; and input the annotated transcribed text into the pre-built multimodal large language model, and the pre-built multimodal large language model generates a second result based on the annotated transcribed text.
[0132] Specifically, the first sample audio is input into a pre-built multimodal large language model. The first sample audio is first encoded by a feature extraction module and converted into its second feature representation in the speech space. It is then converted to the semantic space by a modal conversion module and converted into its first feature representation in the semantic space. It is then input into the large language model, and the large language model generates a first result by autoregression.
[0133] The annotated transcribed text of the first sample audio is input into the pre-built multimodal large language model without being processed by the feature extraction module and the modality conversion module. The annotated transcribed text of the first sample audio is directly input into the large language model, and the large language model is autoregressively generated to generate a second result. The second result is used as the target label of supervised learning in the training of this embodiment.
[0134] S403: Determine a first loss function value according to the first result and the second result.
[0135] The first result and the second result are compared to determine the difference between the two, and a first loss function value is determined, where the first loss function value reflects the difference and error in the multimodal large language model when processing the first sample audio and the annotated transcribed text of the first sample audio.
[0136] The specific type of the first loss function value can be selected according to different application scenarios, and this application does not limit this. For example, a cross entropy loss function can be used to measure the difference between the predicted distribution and the actual distribution.
[0137] S404: Optimize parameters of the modal conversion module in the pre-built multimodal large language model according to the first loss function value.
[0138] In this embodiment, the feature extraction module and the large language model are initialized with pre-trained module parameters, and have basic feature extraction capabilities and text processing capabilities respectively, while the parameters of the modal conversion module are randomly initialized, and the modal conversion capability is poor. It can be seen that the differences and errors in the multimodal large language model when processing the first sample audio and the annotated transcribed text of the first sample audio are mainly caused by the modal conversion module. The main optimization object of the training process of this embodiment is concentrated on the modal conversion module.
[0139] The parameters of the modal conversion module are optimized according to the value of the first loss function, and the parameters of the feature extraction module and the large language model remain unchanged. The ability of the modal conversion module to convert audio features into semantic features is improved, so that the ability of the modal conversion module meets actual needs.
[0140] After the training process of this embodiment, the multimodal large language model after training can align the two modal inputs of speech and text into the same semantic space, so that the large language model module can simultaneously understand and process signals from two different modalities.
[0141] Figure 5 A flow chart of another model training method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the training process includes steps S501-S505:
[0142] S501. Obtain multiple groups of training samples and label texts corresponding to each group of training samples, each group of training samples includes a second sample audio and second sample indication information corresponding to the second sample audio, the second sample indication information of each group of training samples indicates any one of the speech processing tasks, and the speech processing tasks indicated by the second sample indication information of each group of training samples are different, and the speech processing tasks include at least one of speech recognition, voiceprint recognition, sound event detection, and spoken language comprehension.
[0143] In this embodiment, the feature extraction module is initialized with parameters of an acoustic model such as a speech recognizer, and has basic feature extraction capabilities, but does not have true audio understanding capabilities, that is, it does not have the ability to obtain a second feature representation that can characterize multi-dimensional key information of the audio and cover the rich details of the audio. Therefore, it is necessary to fine-tune the feature extraction module of the multimodal large model so that it has the ability to generate a second feature representation of the input audio.
[0144] The second sample audio may be obtained through various channels, including but not limited to public datasets, field recordings, etc. Optionally, the second sample audio covers a variety of scenarios and contexts.
[0145] Optionally, the speech processing task may also include speech translation, emotion recognition and other tasks, which are determined according to actual medical scenarios and needs.
[0146] Specifically, each group of training data includes training samples (second sample audio, second sample indication information) and label text, wherein the second sample indication information can be understood as indication information for guiding the object receiving the information to perform a corresponding speech processing task, and the speech processing tasks indicated by the second sample indication information of each group of training samples are not the same, which means that training data needs to be prepared separately for each speech processing task.
[0147] Optionally, the second sample indication information can be understood as a prompt customized for each speech processing task, for example, the prompt for a speech recognition task can be "Please transcribe the content of this speech", the prompt for a speech translation task can be "Please transcribe the content of this speech into Chinese / English", the prompt for an emotion recognition task can be "What is the speaker's emotion in this speech", etc. In order to ensure the generalization of the model, multiple prompts are designed for each speech processing task, and each set of training data randomly uses one of the prompts for the corresponding speech processing task.
[0148] S502: Input the multiple groups of training samples into a pre-built multimodal large language model, and the multimodal large language model performs speech processing on the second sample audio of each group of training samples according to the second sample indication information in the group of training samples, to generate processing results corresponding to each group of training samples.
[0149] Specifically, multiple groups of training samples are input into a pre-built multimodal large language model, which performs specific speech processing on the corresponding second sample audio according to the second sample indication information in each group of training samples. This means that the multimodal large language model will perform multiple speech processing tasks at the same time and generate different processing results for different types of speech processing tasks. In this process, the feature extraction module of the multimodal large language model will extract the speech features required by different types of speech processing tasks, which usually involves multi-dimensional feature extraction.
[0150] S503: Determine the loss function value corresponding to each group of training samples according to the processing results corresponding to each group of training samples and the label texts corresponding to each group of training samples.
[0151] S504: Determine a second loss function value according to the loss function value corresponding to each group of training samples.
[0152] Specifically, after determining the loss function value of each group of training samples, the loss function value of each group of training samples is comprehensively calculated to determine the second loss function value of the entire model. The comprehensive calculation can be achieved by taking the average value, weighted average value or other aggregation methods, which are determined according to actual conditions.
[0153] S505. Optimize parameters of the feature extraction module in the pre-built multimodal large language model according to the second loss function value.
[0154] After joint training of multiple speech tasks, the feature extraction module can have the ability of multi-dimensional and comprehensive feature extraction. The parameters of the feature extraction module are optimized according to the value of the second loss function. The parameters of the modal conversion module and the large language model remain fixed, which can make the capabilities of the feature extraction module meet actual needs.
[0155] Optionally, in order to speed up training time and save hardware resources, Lora training can be used as the training method.
[0156] Figure 6 A flow chart of another model training method provided in an embodiment of the present application is as follows: Figure 6 As shown, the training process includes steps S601-S604:
[0157] S601, obtaining a sample answer audio, a sample first indication information, and a marked cheating result corresponding to the sample answer audio.
[0158] The sample answer audio can be obtained through various channels, including but not limited to public data sets, field recordings, etc. Optionally, the sample answer audio covers a variety of voice answer scenarios. The labeled cheating results can correctly reflect the cheating detection results corresponding to the sample answer audio.
[0159] S602: Input the sample answer audio and the sample first indication information into a pre-built multimodal large language model to obtain a cheating detection result corresponding to the sample answer audio.
[0160] S603. Determine a third loss function value according to the cheating detection result corresponding to the sample answer audio and the marked cheating result of the sample answer audio.
[0161] S604: Optimize parameters of the large language model in the pre-built multimodal large language model according to the third loss function value.
[0162] After fine-tuning the large language model through cheating detection training, the large language model can be equipped with cheating detection capabilities. The parameters of the large language model are optimized according to the value of the third loss function. The parameters of the modal conversion module and the feature extraction module remain unchanged, so that the cheating detection capabilities of the large language model can meet actual needs. Optionally, in order to speed up training time and save hardware resources, Lora training can be used as the training method.
[0163] As an optional implementation, first Figure 4 In the speech-text modality alignment training in the embodiment shown, according to the first loss function value, the modality conversion module in the multimodal large language model is optimized in parameters, so that the multimodal large model has the ability to align the speech-text modality; and then Figure 5 In the embodiment shown, the voice-text supervised training is performed, and according to the second loss function value, the parameters of the feature extraction module in the multimodal large language model are optimized, so that the multimodal large model has real audio understanding ability; finally, Figure 6 The cheating detection training in the illustrated embodiment optimizes the parameters of the large language model in the multimodal large language model according to the third loss function value, and fine-tunes the large language model through the cheating detection task, so that the multimodal large language model has the cheating detection capability of the answer audio.
[0164] Exemplary Devices
[0165] Corresponding to the above-mentioned cheating detection method, the embodiment of the present application also provides a cheating detection device. Figure 7 is a schematic diagram of the structure of a cheating detection device provided in an embodiment of the present application, such as Figure 7 As shown, the cheating detection device provided in the embodiment of the present application includes:
[0166] The first unit 701 is used to obtain a target answer audio and first indication information, where the first indication information includes task information of a cheating detection task and a cheating judgment rule, and the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result;
[0167] The second unit 702 is used to determine a first feature representation of the target answer audio in a semantic space;
[0168] The third unit 703 is used to input the first feature representation and the first indication information into a pre-trained large language model, and the large language model determines the cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0169] This application first determines the first feature representation of the target answer audio in the semantic space, and the first feature representation contains detailed feature information of the target answer audio in multiple dimensions. Then, by using the powerful semantic understanding and analysis capabilities of the large language model, by analyzing the first feature representation of the target answer audio in the semantic space, it can capture the key information and potential contextual features in the target answer audio, such as the specific audio content, the sound events contained, and the emotions expressed, etc., and combine this information with the cheating judgment rules in the input indication information. The large language model can accurately identify cheating behavior in the target answer audio based on a full understanding of the relevant information of the target answer audio and the cheating judgment rules, thereby achieving accurate detection of cheating behavior. In addition, this application can flexibly adjust the cheating judgment rules according to different answering scenarios and cheating methods, so that it can adapt to different answering scenarios and cheating methods, and improve the adaptability and robustness of cheating detection.
[0170] As an optional implementation, the first indication information also includes question information, the question information includes the question and / or reference answer, and the cheating judgment rule includes a rule for making a cheating judgment when the target answer audio is correct.
[0171] As an optional implementation, the second unit 702 may be specifically configured to:
[0172] Encoding the target answer audio to obtain a second feature representation of the target answer audio in the speech space;
[0173] The second feature representation is converted into a semantic space to obtain a first feature representation of the target answer audio in the semantic space.
[0174] As an optional implementation, the second unit 702 and the third unit 703 may be combined into one unit, which may be specifically used for:
[0175] The target answer audio and the first indication information are input into a pre-trained multimodal large language model so that the multimodal large language model determines a first feature representation of the target answer audio in a semantic space, and determines a cheating detection result corresponding to the target answer audio based on the first feature representation and the first indication information.
[0176] As an optional implementation, the multimodal large language model includes a feature extraction module, a modality conversion module and a pre-trained large language model, wherein:
[0177] The feature extraction module is used to encode the target answer audio to obtain a second feature representation of the target answer audio in the speech space;
[0178] The modality conversion module is used to convert the second feature representation into a semantic space to obtain a first feature representation of the target answer audio in the semantic space;
[0179] The large language model is used to determine a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0180] As an optional implementation, the training process of the multimodal large language model includes:
[0181] Obtaining a first sample audio and an annotated transcription of the first sample audio;
[0182] Inputting the first sample audio into a pre-built multimodal large language model, and generating a first result based on the first sample audio by the pre-built multimodal large language model; and inputting the annotated transcribed text into the pre-built multimodal large language model, and generating a second result based on the annotated transcribed text by the pre-built multimodal large language model;
[0183] Determine a first loss function value according to the first result and the second result;
[0184] According to the first loss function value, parameters of the modality conversion module in the pre-built multimodal large language model are optimized.
[0185] As an optional implementation, the training process of the multimodal large language model includes:
[0186] Acquire multiple groups of training samples and label texts corresponding to each group of training samples, each group of training samples includes a second sample audio and second sample indication information corresponding to the second sample audio, the second sample indication information of each group of training samples indicates any one of speech processing tasks, and the second sample indication information of each group of training samples indicates different speech processing tasks, and the speech processing tasks include at least one of speech recognition, voiceprint recognition, sound event detection, and spoken language comprehension;
[0187] Inputting the plurality of groups of training samples into a pre-built multimodal large language model, and having the multimodal large language model perform speech processing on the second sample audio of each group of training samples according to the second sample indication information in the group of training samples, to generate a processing result corresponding to each group of training samples;
[0188] Determine the loss function value corresponding to each group of training samples according to the processing results corresponding to each group of training samples and the label text corresponding to each group of training samples;
[0189] Determine a second loss function value according to the loss function value corresponding to each group of training samples;
[0190] According to the second loss function value, parameters of the feature extraction module in the pre-built multimodal large language model are optimized.
[0191] As an optional implementation, the training process of the multimodal large language model includes:
[0192] Obtaining a sample answer audio, a sample first indication information, and a marked cheating result corresponding to the sample answer audio;
[0193] Inputting the sample answer audio and the sample first indication information into a pre-built multimodal large language model to obtain a cheating detection result corresponding to the sample answer audio;
[0194] Determine a third loss function value according to the cheating detection result corresponding to the sample answer audio and the labeled cheating result of the sample answer audio;
[0195] According to the third loss function value, parameters of the large language model in the pre-constructed multimodal large language model are optimized.
[0196] The cheating detection device provided in this embodiment belongs to the same application concept as the cheating detection method provided in the above embodiments of this application, and can execute the cheating detection method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the cheating detection method. For technical details not fully described in this embodiment, please refer to the specific processing content of the cheating detection method provided in the above embodiments of this application, and will not be repeated here.
[0197] The functions implemented by the above first unit 701, the second unit 702 and the third unit 703 can be implemented by the same or different processors respectively, which is not limited in the embodiment of the present application.
[0198] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.
[0199] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.
[0200] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.
[0201] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.
[0202] Exemplary Electronic Devices
[0203] Another embodiment of the present application also provides an electronic device, see Figure 8 As shown, the device includes:
[0204] Memory 200 and processor 210;
[0205] The memory 200 is connected to the processor 210 and is used to store programs;
[0206] The processor 210 is configured to implement the cheating detection method disclosed in any of the above embodiments by running the program stored in the memory 200 .
[0207] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .
[0208] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.
[0209] A bus may include a pathway that transfers information between components of a computer system.
[0210] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0211] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.
[0212] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.
[0213] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.
[0214] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.
[0215] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.
[0216] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any cheating detection method provided in the above embodiments of the present application.
[0217] The embodiment of the present application further provides a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the cheating detection method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the embodiment of the cheating detection method.
[0218] Exemplary computer program products and storage media
[0219] In addition to the above-mentioned method and device, an embodiment of the present application may also be a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the steps of the cheating detection method according to various embodiments of the present application described in any of the above-mentioned embodiments of this specification.
[0220] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0221] In addition, the embodiment of the present application may also be a storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the cheating detection method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:
[0222] S101. Obtain target answer audio and first indication information, where the first indication information includes task information of a cheating detection task and a cheating judgment rule; the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result.
[0223] S102: Determine a first feature representation of the target answer audio in a semantic space.
[0224] S103: input the first feature representation and the first indication information into a pre-trained large language model, and the large language model determines a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
[0225] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0226] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0227] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.
[0228] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.
[0229] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0230] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0231] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.
[0232] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0233] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0234] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0235] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A cheating detection method, It is characterized in that include: Obtaining a target answer audio and first indication information, wherein the first indication information includes task information of a cheating detection task and a cheating judgment rule, wherein the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result; Determining a first feature representation of the target answer audio in a semantic space; The first feature representation and the first indication information are input into a pre-trained large language model, and the large language model determines a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
2. The method according to claim 1, It is characterized in that The first indication information also includes question information, the question information includes a question and / or a reference answer, and the cheating judgment rule includes a rule for making a cheating judgment when the target answer audio is correct.
3. The method according to claim 1, It is characterized in that The determining of a first feature representation of the target answer audio in a semantic space includes: Encoding the target answer audio to obtain a second feature representation of the target answer audio in the speech space; The second feature representation is converted into a semantic space to obtain a first feature representation of the target answer audio in the semantic space.
4. The method according to any one of claims 1 to 3, It is characterized in that The determining of a first feature representation of the target answer audio in a semantic space, and inputting the first feature representation and the first indication information into a pre-trained large language model, wherein the large language model determines a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information, comprises: The target answer audio and the first indication information are input into a pre-trained multimodal large language model so that the multimodal large language model determines a first feature representation of the target answer audio in a semantic space, and determines a cheating detection result corresponding to the target answer audio based on the first feature representation and the first indication information.
5. The method according to claim 4, It is characterized in that The multimodal large language model includes a feature extraction module, a modality conversion module and a pre-trained large language model, wherein: The feature extraction module is used to encode the target answer audio to obtain a second feature representation of the target answer audio in the speech space; The modality conversion module is used to convert the second feature representation into a semantic space to obtain a first feature representation of the target answer audio in the semantic space; The large language model is used to determine a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
6. The method according to claim 5, It is characterized in that The training process of the multimodal large language model includes: Obtaining a first sample audio and an annotated transcription of the first sample audio; Inputting the first sample audio into a pre-built multimodal large language model, and generating a first result based on the first sample audio by the pre-built multimodal large language model; and inputting the annotated transcribed text into the pre-built multimodal large language model, and generating a second result based on the annotated transcribed text by the pre-built multimodal large language model; Determine a first loss function value according to the first result and the second result; According to the first loss function value, parameters of the modal conversion module of the pre-built multimodal large language model are optimized.
7. The method according to claim 5, It is characterized in that The training process of the multimodal large language model includes: Acquire multiple groups of training samples and label texts corresponding to each group of training samples, each group of training samples includes a second sample audio and second sample indication information corresponding to the second sample audio, the second sample indication information of each group of training samples indicates any one of speech processing tasks, and the second sample indication information of each group of training samples indicates different speech processing tasks, and the speech processing tasks include at least one of speech recognition, voiceprint recognition, sound event detection, and spoken language comprehension; Inputting the plurality of groups of training samples into a pre-built multimodal large language model, and having the multimodal large language model perform speech processing on the second sample audio of each group of training samples according to the second sample indication information in the group of training samples, to generate a processing result corresponding to each group of training samples; Determine the loss function value corresponding to each group of training samples according to the processing results corresponding to each group of training samples and the label text corresponding to each group of training samples; Determine a second loss function value according to the loss function value corresponding to each group of training samples; According to the second loss function value, parameters of the feature extraction module of the pre-built multimodal large language model are optimized.
8. The method according to claim 5, It is characterized in that The training process of the multimodal large language model includes: Obtaining a sample answer audio, a sample first indication information, and a marked cheating result corresponding to the sample answer audio; Inputting the sample answer audio and the sample first indication information into a pre-built multimodal large language model to obtain a cheating detection result corresponding to the sample answer audio; Determine a third loss function value according to the cheating detection result corresponding to the sample answer audio and the labeled cheating result of the sample answer audio; According to the third loss function value, parameters of the large language model in the pre-constructed multimodal large language model are optimized.
9. A cheating detection device, It is characterized in that include: A first unit is used to obtain a target answer audio and first indication information, wherein the first indication information includes task information of a cheating detection task and a cheating judgment rule, and the cheating detection task is used to perform cheating detection on the target answer audio and determine a cheating detection result; A second unit is used to determine a first feature representation of the target answer audio in a semantic space; The third unit is used to input the first feature representation and the first indication information into a pre-trained large language model, and the large language model determines a cheating detection result corresponding to the target answer audio according to the first feature representation and the first indication information.
10. An electronic device, It is characterized in that including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the cheating detection method according to any one of claims 1 to 8 by running the program in the memory.
11. A storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the cheating detection method according to any one of claims 1 to 8 is implemented.
12. A computer program product, It is characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to implement the cheating detection method according to any one of claims 1 to 8.
