Classroom evaluation and model training method, device and equipment

Abstract

The invention discloses a classroom evaluation and model training method, device and equipment. The method comprises the following steps: acquiring video data of a to-be-evaluated classroom; performing voice recognition on the video data to obtain a plurality of target text segments; inputting the plurality of target text segments and the target video images corresponding to the plurality of target text segments into a classroom evaluation model to obtain target text feature vectors and target image feature vectors corresponding to the plurality of target text segments, and outputting a target evaluation result of the to-be-evaluated classroom based on a target fusion feature vector obtained by fusing the target text feature vector and the target image feature vector. According to the method, a machine learning model can be used for replacing manpower to carry out classroom observation evaluation, the labor cost is reduced, the feedback speed is increased, information of different modes can be fully considered, and the method has high evaluation effectiveness.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a classroom evaluation and model training method, device and equipment. Background Art

[0002] Classroom observation is a widely used educational research method, usually conducted by researchers or professional teachers to systematically observe, record, and analyze teaching and learning behaviors in the classroom to gain insights into educational practice. The goal of classroom observation is to provide an in-depth understanding of classroom dynamics and obtain effective classroom evaluations, thereby providing educators with feedback on teaching effectiveness, student engagement, teaching strategies, and classroom environment. Traditional classroom observation and evaluation methods mainly involve observers entering the classroom or observing classroom videos, and using pre-designed observation tools such as scales or checklists to systematically record specific teaching and learning behaviors. Since traditional classroom observation and evaluation methods require in-depth analysis by professionals, and different schools or subjects may use different evaluation criteria, there are problems such as inconsistent evaluation criteria, high costs, and long feedback times. Summary of the invention

[0003] The embodiments of the present application provide a classroom evaluation and model training method, device and equipment, which can use a machine learning model to replace manual classroom observation and evaluation, reduce labor costs, speed up feedback, and fully consider information from different modalities, with high evaluation effectiveness.

[0004] In a first aspect, an embodiment of the present application provides a classroom evaluation method, the method comprising:

[0005] Obtain video data of the class to be evaluated;

[0006] Performing speech recognition on the video data to obtain multiple target text segments;

[0007] Input the above-mentioned multiple target text segments and the target video images corresponding to the above-mentioned multiple target text segments into the classroom evaluation model, obtain the target text feature vectors and the target image feature vectors corresponding to the above-mentioned multiple target text segments, and output the target evaluation result of the above-mentioned classroom to be evaluated based on the target fused feature vector obtained by fusion of the above-mentioned target text feature vector and the above-mentioned target image feature vector;

[0008] Among them, the above-mentioned classroom evaluation model is obtained by training based on multiple classroom videos with known evaluation results.

[0009] In a possible implementation, the classroom evaluation model includes a large language model and an image encoder, and the multiple target text segments and the target video images corresponding to the multiple target text segments are input into the classroom evaluation model to obtain the target text feature vectors and the target image feature vectors corresponding to the multiple target text segments, including:

[0010] Inputting the plurality of target text segments into the large language model respectively to generate target text feature vectors corresponding to the plurality of target text segments respectively;

[0011] The target video images corresponding to the above-mentioned multiple target text segments are respectively input into the above-mentioned image encoder to generate the target image feature vectors corresponding to the above-mentioned multiple target text segments.

[0012] In a possible implementation, the fused feature vector based on the fusion of the target text feature vector and the target image feature vector outputs the target evaluation result of the class to be evaluated, including:

[0013] Inputting the target text feature vectors and the target image feature vectors corresponding to the target text segments into the linear layer of the classroom evaluation model to obtain multiple target fusion feature vectors;

[0014] The above-mentioned multiple target fusion vectors are input into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the target evaluation results of the above-mentioned classroom to be evaluated.

[0015] In a second aspect, an embodiment of the present application provides a classroom evaluation model training method, the method comprising:

[0016] Get multiple first-classroom videos with known evaluation results;

[0017] Performing speech recognition on the first classroom video to obtain a plurality of first text segments;

[0018] Inputting a target second text segment among the plurality of second text segments and a second video image corresponding to the target second text segment into an initial evaluation model, obtaining a second text feature vector and a second image feature vector corresponding to the target second text segment, and outputting a second prediction result of the target second text segment based on a second fused feature vector fused with the second image feature vector; the second prediction result represents a predicted second text segment corresponding to a next sequence of the target second text segment;

[0019] The classroom evaluation model is updated based on the evaluation results corresponding to each of the multiple first classroom videos and the first prediction results.

[0020] In a possible implementation, before inputting the plurality of first text segments and the first video images corresponding to the plurality of first text segments into the classroom evaluation model, the method further includes:

[0021] Get multiple second classroom videos;

[0022] Performing voice recognition on the second classroom video to obtain a plurality of second text segments; the second text segments are arranged in a preset order;

[0023] Inputting a target second text segment among the plurality of second text segments and a second video image corresponding to the target second text segment into an initial evaluation model, and outputting a second prediction result of the target second text segment based on a second fused feature vector fused with the second image feature vector; the second prediction result represents a predicted second text segment corresponding to a next sequence of the target second text segment;

[0024] Based on the second text segment corresponding to the next sequence of the target second text segment and the predicted second text segment, the initial evaluation model is updated to obtain the classroom evaluation model.

[0025] In a possible implementation, the classroom evaluation model includes a large language model and an image encoder, and the inputting of the plurality of first text segments and the first video images corresponding to the plurality of first text segments into the classroom evaluation model to obtain the first text feature vectors and the first image feature vectors corresponding to the plurality of first text segments includes:

[0026] Inputting the plurality of first text segments into the large language model respectively to generate first text feature vectors corresponding to the plurality of first text segments respectively;

[0027] The first video images corresponding to the first text segments are respectively input into the image encoder to generate first image feature vectors corresponding to the first text segments.

[0028] In a possible implementation, the first fused feature vector obtained by fusing the first text feature vector and the first image feature vector is output as a first prediction result of the first classroom video, including:

[0029] Inputting the first text feature vectors and the first image feature vectors corresponding to each of the plurality of first text segments into the linear layer of the classroom evaluation model to obtain a plurality of first fused feature vectors;

[0030] The above-mentioned multiple first fusion vectors are input into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the first prediction result of the above-mentioned first classroom video.

[0031] In a third aspect, an embodiment of the present application provides a classroom evaluation device, the device comprising:

[0032] The first acquisition module is used to acquire video data of the class to be evaluated;

[0033] A first recognition module is used to perform speech recognition on the video data to obtain multiple target text segments;

[0034] An evaluation module, used for inputting the above-mentioned multiple target text segments and the target video images corresponding to the above-mentioned multiple target text segments into a classroom evaluation model, obtaining the target text feature vectors and the target image feature vectors corresponding to the above-mentioned multiple target text segments, and outputting the target evaluation result of the above-mentioned classroom to be evaluated based on the target fused feature vector obtained by fusion of the above-mentioned target text feature vector and the above-mentioned target image feature vector;

[0035] Among them, the above-mentioned classroom evaluation model is obtained by training based on multiple classroom videos with known evaluation results.

[0036] In a possible implementation, the classroom evaluation model includes a large language model and an image encoder, and the evaluation module includes:

[0037] A first evaluation unit, configured to input the plurality of target text segments into the large language model respectively, and generate target text feature vectors corresponding to the plurality of target text segments respectively;

[0038] The second evaluation unit is used to input the target video images corresponding to the above-mentioned multiple target text segments into the above-mentioned image encoder respectively, and generate the target image feature vectors corresponding to the above-mentioned multiple target text segments respectively.

[0039] In a possible implementation, the evaluation module includes:

[0040] A first fusion unit is used to input the target text feature vectors and the target image feature vectors corresponding to the multiple target text segments into the linear layer of the classroom evaluation model to obtain multiple target fusion feature vectors;

[0041] The first output unit is used to input the above-mentioned multiple target fusion vectors into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the target evaluation result of the above-mentioned classroom to be evaluated.

[0042] In a fourth aspect, an embodiment of the present application provides a classroom evaluation model training device, the device comprising:

[0043] A second acquisition module is used to acquire a plurality of first classroom videos with known evaluation results;

[0044] A second recognition module is used to perform voice recognition on the first classroom video to obtain a plurality of first text segments;

[0045] A first training module is used to input the above-mentioned multiple first text segments and the first video images corresponding to each of the above-mentioned multiple first text segments into the classroom evaluation model, obtain the first text feature vectors and the first image feature vectors corresponding to each of the above-mentioned multiple first text segments, and output the first prediction result of the above-mentioned first classroom video based on the first fused feature vector obtained by fusion of the above-mentioned first text feature vector and the above-mentioned first image feature vector; the above-mentioned first prediction result represents the prediction evaluation result of the above-mentioned first classroom video;

[0046] The second training module is used to update the classroom evaluation model based on the evaluation results corresponding to each of the multiple first classroom videos and the first prediction results.

[0047] In a possible implementation, the device further includes:

[0048] A third acquisition module is used to acquire multiple second classroom videos;

[0049] A third recognition module is used to perform voice recognition on the second classroom video to obtain a plurality of second text segments; the second text segments are arranged in a preset order;

[0050] The third training module is used to input a target second text segment among the multiple second text segments and a second video image corresponding to the target second text segment into an initial evaluation model, obtain a second text feature vector and a second image feature vector corresponding to the target second text segment, and output a second prediction result of the target second text segment based on a second fused feature vector fused with the second image feature vector; the second prediction result represents a predicted second text segment corresponding to a next sequence of the target second text segment;

[0051] The fourth training module is used to update the initial evaluation model to obtain the classroom evaluation model based on the second text segment corresponding to the next sequence of the target second text segment and the predicted second text segment.

[0052] In a possible implementation, the classroom evaluation model includes a large language model and an image encoder, and the first training module includes:

[0053] A first training unit, configured to input the plurality of first text segments into the large language model respectively, and generate first text feature vectors corresponding to the plurality of first text segments respectively;

[0054] The second training unit is used to input the first video images corresponding to the multiple first text segments into the image encoder respectively, and generate the first image feature vectors corresponding to the multiple first text segments respectively.

[0055] In a possible implementation, the first training module includes:

[0056] A second fusion unit is used to input the first text feature vectors and the first image feature vectors corresponding to each of the plurality of first text segments into a linear layer of the classroom evaluation model to obtain a plurality of first fused feature vectors;

[0057] The second output unit is used to input the above-mentioned multiple first fusion vectors into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the first prediction result of the above-mentioned first classroom video.

[0058] In a fifth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above method steps.

[0059] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above method steps.

[0060] In one or more embodiments of the present application, video data of a class to be evaluated is obtained; speech recognition is performed on the video data to obtain multiple target text segments; the multiple target text segments and the target video images corresponding to the multiple target text segments are input into a classroom evaluation model to obtain target text feature vectors and target image feature vectors corresponding to the multiple target text segments, and based on the target fusion feature vector after the fusion of the target text feature vector and the target image feature vector, the target evaluation result of the class to be evaluated is output; wherein the classroom evaluation model is obtained by training classroom videos based on multiple known evaluation results. This method can use a machine learning model to replace manual classroom observation and evaluation, which reduces labor costs, speeds up feedback, and can fully consider information from different modalities, with high evaluation effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A system architecture diagram of a classroom evaluation system provided in an embodiment of the present application;

[0063] Figure 2 A flow chart of a classroom evaluation method provided in an embodiment of the present application;

[0064] Figure 3 An overall architecture diagram of a classroom evaluation model provided in an embodiment of the present application;

[0065] Figure 4 A flow chart of a classroom evaluation model training method provided in an embodiment of the present application;

[0066] Figure 5 A pre-training stage architecture diagram of a classroom evaluation model provided in an embodiment of the present application;

[0067] Figure 6 An SFT phase architecture diagram of a classroom evaluation model provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of the structure of a classroom evaluation device provided in an embodiment of the present application;

[0069] Figure 8 A schematic diagram of the structure of a classroom evaluation model training device provided in an embodiment of the present application;

[0070] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0072] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0073] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the video data of the classroom to be evaluated involved in this specification are all obtained with full authorization.

[0074] The present application is described in detail below with reference to specific embodiments.

[0075] Please refer to Figure 1 , Figure 1 The following is a schematic diagram of the architecture of a classroom evaluation system provided by an exemplary embodiment of the present application. Figure 1 As shown, the classroom evaluation system may include a terminal 110 and a server 120 .

[0076] The terminal 110 is connected to the server 120 via the network 130. Optionally, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto. The terminal 110 installs and runs an application that supports classroom evaluation. The network 130 can be a wireless network or a wired network.

[0077] The server 120 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The server 120 provides background services for applications running on the terminal 110. For example, the server 120 provides corresponding services for applications running on the terminal 110. In the embodiment of the present application, the server 120 provides background services for applications running on the terminal that support classroom evaluation, such as running or training classroom evaluation models.

[0078] Those skilled in the art will appreciate that the number of the above-mentioned terminals 110 and servers 120 may be more or less. For example, there may be only one terminal 110 and server 120, or there may be dozens or hundreds of terminals 110 and servers 120, or a greater number. In this case, the above-mentioned system architecture may also include other terminals and servers. The embodiment of the present application does not limit the number of terminals and device types.

[0079] The classroom evaluation method or classroom evaluation model training method involved in one or more embodiments of the present application can be implemented by relying on a computer program and can be run on a classroom evaluation device or a classroom evaluation model training device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application. The execution subject corresponding to the classroom evaluation method can be the above-mentioned terminal 110, and the server 120 can be responsible for running or training the classroom evaluation model, and sending the model or model call interface to the terminal 110; the execution subject corresponding to the classroom evaluation method can also be the server 120, and the terminal 110 can be responsible for uploading the video data of the classroom to be evaluated to the server 120. It is determined specifically based on the actual application environment, and the embodiments of the present application are not limited to this.

[0080] Next, combine Figure 1 , introduces the classroom evaluation method provided by the embodiment of this specification. For details, please refer to Figure 2 , which is a flow chart of a classroom evaluation method provided by an exemplary embodiment of the present application. Figure 2 As shown, with the execution subject being the server, the classroom evaluation method includes the following steps:

[0081] S201, obtaining video data of the class to be evaluated.

[0082] Specifically, the classroom to be evaluated in the embodiment of the present application is a classroom that needs to be observed and evaluated in the classroom, and the place where the educational or teaching activities are carried out includes but is not limited to classrooms, conference rooms in schools, and online education platforms. The classroom to be evaluated can be an education or teaching unit (such as an online course), which usually includes teacher lectures, student participation, questions and answers, discussions, etc., reflecting the students' learning status and the teacher's teaching methods. By evaluating and analyzing the classroom to be evaluated, educators can understand students' learning situation and feedback, improve teaching methods and strategies, and thus improve teaching quality and effectiveness.

[0083] The video data in the embodiment of the present application may be a recording of the class to be evaluated, that is, after recording the class to be evaluated, the video data (including audio data) obtained from the recording is collected for subsequent processing and analysis to obtain an evaluation result of the class to be evaluated.

[0084] S202, performing speech recognition on the video data to obtain multiple target text segments.

[0085] Specifically, the embodiment of the present application performs speech recognition and text conversion on the audio data in the video data to convert the speech content in the audio data into text to obtain target text data.

[0086] Speech recognition can be achieved using ASR technology (Automatic Speech Recognition). ASR is an automatic speech recognition system based on artificial intelligence and speech signal processing technology, which can convert human speech into computer-readable text or commands to achieve human-computer interaction and natural language processing. In some embodiments, models such as CTC (Connectionist Temporal Classification), AED (Audio Event Detection), and RNN-T (Recurrent Neural Network Transducer) can be used to perform speech recognition on audio data to obtain corresponding text data.

[0087] It should be noted that before performing speech recognition on audio data, the audio data needs to be processed to improve the quality of the speech signal and reduce the influence of noise, aliasing and other factors, so as to make the subsequent speech recognition more accurate.

[0088] After speech recognition, the target text data needs to be segmented into multiple target text segments. Each target text segment is continuous and can find the corresponding video content in the video data. Segmentation can be based on classroom conversation content or time, for example, a continuous conversation content can be divided into a target text segment.

[0089] S203, input multiple target text segments and target video images corresponding to each of the target text segments into the classroom evaluation model, obtain target text feature vectors and target image feature vectors corresponding to each of the target text segments, and output the target evaluation result of the classroom to be evaluated based on the target fused feature vector formed by fusing the target text feature vector and the target image feature vector.

[0090] Specifically, Figure 3As shown, it is an overall architecture diagram of a classroom evaluation model provided by an embodiment of the present application. It can be seen from the architecture diagram that this embodiment splits the classroom video into two dimensions, namely text and image. The text dimension represents the auditory information in the classroom video, and the image dimension represents the visual information in the classroom video. Auditory information can be converted into text data by ASR technology to extract important information and details. Visual information can be intercepted and processed by intercepting the number of frames to capture the video screen, and the visual information in the video is converted into a picture format, so as to analyze and process the visual content of the classroom video. Thus, it is possible to use data from two modes to represent the information carried by the classroom video itself. After the data from the two modes are fused with each other, the classroom evaluation model can analyze the content of the classroom video more comprehensively and objectively, so that the classroom observation evaluation results output by it have higher validity.

[0091] In this embodiment, the target text segment and the target video image are matched with each other and belong to a video segment together. The video segment in the video data can be determined according to the target text segment, and then multiple pictures are intercepted from the video segment as the target video image. When the target text segment and the target video image are input into the classroom evaluation model, the target text feature vector will be extracted for the target text segment, and the target image feature vector will be extracted for the target video image, and then the target text feature vector and the target image feature vector corresponding to the same target text segment will be fused with each other to obtain a target fusion feature vector. Since the video data is divided into multiple target text segments, multiple target fusion feature vectors are also obtained. Finally, based on all the target fusion feature vectors, the final evaluation result is generated, and the evaluation result is the target evaluation result of the classroom to be evaluated. This process is to use multimodal data (text and image) for classroom observation and evaluation, which can overcome the limitations of single modal information, provide more comprehensive and richer classroom video information, and help generate more accurate and comprehensive classroom observation and evaluation.

[0092] In some embodiments, the classroom evaluation model includes a large language model and an image encoder. The above-mentioned inputting multiple target text segments and target video images corresponding to each of the multiple target text segments into the classroom evaluation model to obtain target text feature vectors and target image feature vectors corresponding to each of the multiple target text segments includes:

[0093] Inputting multiple target text segments into the large language model respectively to generate target text feature vectors corresponding to the multiple target text segments;

[0094] The target video images corresponding to the multiple target text segments are respectively input into the image encoder to generate the target image feature vectors corresponding to the multiple target text segments.

[0095] Specifically, in this embodiment, the classroom evaluation model uses a large language model and an image encoder to extract information in two dimensions, text and image. Large Language Model (LLM) is a natural language processing technology based on deep learning. It analyzes a large corpus and learns the intrinsic structure and rules of language, so as to generate high-quality natural language text. In the classroom evaluation model, the large language model can accept a target text segment as input and generate a target text feature vector, which can capture the semantic, grammatical and contextual information in the text.

[0096] An image encoder is a deep learning model used to convert image data into a feature vector. In the classroom evaluation model, the image encoder accepts a target video image as input and generates a target image feature vector that captures the visual information in the image, such as classroom atmosphere, student performance, and the use of teaching aids.

[0097] In some embodiments, the above-mentioned fused feature vector based on the fusion of the target text feature vector and the target image feature vector outputs the target evaluation result of the class to be evaluated, including:

[0098] Input the target text feature vectors and target image feature vectors corresponding to the multiple target text segments into the linear layer of the classroom evaluation model to obtain multiple target fusion feature vectors;

[0099] Multiple target fusion vectors are input into the encoder-decoder network of the classroom evaluation model to obtain the target evaluation results of the classroom to be evaluated.

[0100] Specifically, this embodiment merges the feature vectors of the two dimensions of text and image through a linear layer (fully connected layer) to generate a fused feature vector. This fused feature vector contains both text information and image information, which can more comprehensively reflect the information of the classroom video. Among them, the target text feature vector and the target image feature vector input to the linear layer for fusion should match each other and belong to the same target text segment.

[0101] Finally, the obtained multiple target fusion vectors are input into the encoder-decoder network of the classroom evaluation model to obtain the target evaluation results of the classroom to be evaluated. The encoder-decoder network (Encoder-Decoder) is a deep learning model, which is often used for sequence-to-sequence learning problems. In the classroom evaluation model, the encoder first converts the input multiple target fusion feature vectors into a hidden layer representation (also called encoder output), and then the decoder uses this hidden layer representation to generate the final evaluation result, which is the target evaluation result of the classroom to be evaluated.

[0102] It should be noted that the classroom evaluation model in the embodiment of the present application is trained based on multiple classroom videos with known evaluation results, so that it can evaluate classroom videos more accurately and objectively and generate evaluation results.

[0103] In one or more embodiments of the present application, video data of a class to be evaluated is obtained; speech recognition is performed on the video data to obtain multiple target text segments; the multiple target text segments and the target video images corresponding to the multiple target text segments are input into a classroom evaluation model to obtain target text feature vectors and target image feature vectors corresponding to the multiple target text segments, and based on the target fusion feature vector after the fusion of the target text feature vector and the target image feature vector, the target evaluation result of the class to be evaluated is output; wherein the classroom evaluation model is obtained by training classroom videos based on multiple known evaluation results. This method can use a machine learning model to replace manual classroom observation and evaluation, which reduces labor costs, speeds up feedback, and can fully consider information from different modalities, with high evaluation effectiveness.

[0104] Next, combine Figure 1 , taking the server as the execution subject, introduces the classroom evaluation model training method provided by the embodiment of the present application. Figure 4 As shown in FIG. 1 , the classroom evaluation model training method includes the following steps:

[0105] S401, obtaining a plurality of first classroom videos with known evaluation results.

[0106] Specifically, before conducting classroom evaluation, the classroom evaluation model needs to be trained first. Classroom video data with evaluation results, namely first classroom videos, can be obtained from existing data sets. These data have been labeled or evaluated and are associated with past classroom observation and evaluation content. Training the classroom evaluation model with these first classroom videos with known classroom observation and evaluation allows the model to learn and analyze the features in the classroom videos to effectively evaluate and predict new data and generate accurate and effective evaluation results.

[0107] S402, performing voice recognition on the first classroom video to obtain a plurality of first text segments.

[0108] Specifically, the ASR technology is used to convert the audio data in the classroom video into text data, and divide it into multiple text segments, namely, first text segments. Each first text segment is continuous and can find the corresponding video content in the first classroom video. The segmentation can be based on the classroom conversation content or time, for example, a continuous conversation content is divided into a first text segment.

[0109] S403, inputting multiple first text segments and the first video images corresponding to each of the multiple first text segments into the classroom evaluation model, obtaining first text feature vectors and first image feature vectors corresponding to each of the multiple first text segments, and outputting a first prediction result of the first classroom video based on a first fused feature vector formed by fusing the first text feature vector and the first image feature vector.

[0110] Specifically, the first text segment and the first video image match each other and belong to a video segment together. The video segment in the first classroom video can be determined based on the first text segment, and then multiple pictures can be captured from the video segment as the first video image.

[0111] When the first text segment and the first video image are input into the classroom evaluation model, the first text feature vector is extracted for the first text segment, and the first image feature vector is extracted for the first video image. Then, the first text feature vector and the first image feature vector corresponding to the same first text segment are fused to obtain a first fused feature vector. Since the first classroom video is divided into multiple first text segments, multiple first fused feature vectors are also obtained. Finally, based on all the first fused feature vectors, the first prediction result of the first classroom video is generated, and the first prediction result represents the predicted evaluation result (evaluation report) of the first classroom video.

[0112] In some embodiments, the classroom evaluation model includes a large language model and an image encoder, and the plurality of first text segments and the first video images corresponding to the plurality of first text segments are input into the classroom evaluation model to obtain the first text feature vectors and the first image feature vectors corresponding to the plurality of first text segments, respectively, including:

[0113] Inputting the plurality of first text segments into the large language model respectively, and generating first text feature vectors corresponding to the plurality of first text segments respectively;

[0114] The first video images corresponding to the plurality of first text segments are respectively input into an image encoder to generate first image feature vectors corresponding to the plurality of first text segments.

[0115] Specifically, a large language model and an image encoder can be used to extract features from the text and image dimensions, and convert them into feature vectors. These feature vectors can more comprehensively reflect the information of the classroom video, which helps to evaluate and understand the classroom video more accurately and comprehensively.

[0116] In some embodiments, the first prediction result of the first classroom video is output based on the first fused feature vector obtained by fusing the first text feature vector and the first image feature vector, including:

[0117] Inputting first text feature vectors and first image feature vectors corresponding to each of the plurality of first text segments into a linear layer of a classroom evaluation model to obtain a plurality of first fused feature vectors;

[0118] The multiple first fusion vectors are input into the encoder-decoder network of the classroom evaluation model to obtain a first prediction result of the first classroom video.

[0119] Specifically, the linear layer and encoder-decoder network in the above embodiment can be one or more models and technologies, which can be selected and adjusted according to actual needs. At the same time, the generation method of the first fused feature vector can also be adjusted according to actual needs, for example, different fusion methods, optimization algorithms, etc. can be used.

[0120] S404: Update the classroom evaluation model based on the evaluation results corresponding to each of the plurality of first classroom videos and the first prediction results.

[0121] Specifically, optimization algorithms such as back propagation and gradient descent can be used to continuously adjust the parameters of the model so that the model can predict the evaluation results of unknown classroom videos more and more accurately. The back propagation algorithm can calculate the contribution of each parameter to the prediction result and adjust the parameter value accordingly. The gradient descent algorithm can update the parameters according to the difference between the predicted result and the actual result, gradually reducing the difference.

[0122] Repeat the above steps until the difference between the model's predicted results and the actual results is small enough, or the preset number of training times is reached. This process can be completed through iterative training. Each iteration will update the model's parameters so that the model gradually approaches the optimal solution. By continuously updating and optimizing the model, the classroom evaluation model based on multimodal data can be made more accurate and reliable, and better serve classroom observation and evaluation.

[0123] It should be noted that the training process of the above-mentioned classroom evaluation model corresponds to the prediction process of the classroom evaluation model, and has the same model structure and input and output types, so that the trained classroom evaluation model can predict new classroom video data and generate accurate and effective evaluation results.

[0124] In some embodiments, before inputting the plurality of first text segments and the first video images corresponding to the plurality of first text segments into the classroom evaluation model, the method further includes:

[0125] Get multiple second classroom videos;

[0126] Performing speech recognition on the second classroom video to obtain a plurality of second text segments; the second text segments are arranged in a preset order;

[0127] Inputting a target second text segment among the multiple second text segments and a second video image corresponding to the target second text segment into an initial evaluation model, obtaining a second text feature vector and a second image feature vector corresponding to the target second text segment, and outputting a second prediction result of the target second text segment based on a second fused feature vector fused with the second text feature vector and the second image feature vector; the second prediction result represents a predicted second text segment corresponding to a next sequence of the target second text segment;

[0128] Based on the second text segment corresponding to the next sequence of the target second text segment and the predicted second text segment, the initial evaluation model is updated to obtain the classroom evaluation model.

[0129] Specifically, the above-mentioned classroom evaluation model is a pretrained model. The above-mentioned training process belongs to the supervised fine-tuning (SFT) stage. The pre-trained classroom evaluation model can learn the common features in the data from large-scale data. Therefore, in the supervised fine-tuning stage, it can achieve better generalization effect and improve the output accuracy and effectiveness of the model.

[0130] In this embodiment, it is the pre-training step of the initial evaluation model. The second classroom video can be classroom video data obtained from an existing data set. The second classroom video has a large number scale and does not require data annotation. In the pre-training stage, the extraction of the fused feature vector is consistent with the supervised fine-tuning stage. It is necessary to first split the second classroom video into data of two dimensions, text and image, and then input the initial evaluation model to extract the second text feature vector for the second text segment and the second image feature vector for the second video image, and then fuse the second text feature vector and the second image feature vector corresponding to the same second text segment to obtain a second fused feature vector.

[0131] In the pre-training stage, unlike the supervised fine-tuning stage, the pre-training stage is based on the second fusion feature vector corresponding to a second text segment to predict the next sequence of second text segments to achieve the effect of self-supervision. Therefore, the second text segments need to be arranged in a preset order, which can be arranged in the order of video playback time. After the initial evaluation model is pre-trained, the above-mentioned classroom evaluation model can be obtained, and then the classroom evaluation model is supervised and fine-tuned using high-precision (labeled) data to make the classroom evaluation model meet the classroom observation and evaluation needs.

[0132] For example, Figure 5 As shown, the training steps of the pre-training phase of the classroom evaluation model include:

[0133] S501, converting the audio of the classroom observation video into multiple ASR text segments, usually one ASR segment is a continuous conversation in the classroom.

[0134] S502, take one of the ASR text segments, find the video segment occupied by the original video corresponding to the segment, and capture M pictures from the segment, where M can be any natural number, usually greater than 5.

[0135] S503, input an ASR text segment into a pre-trained general large language model to extract the text vector of the ASR text segment, and the large language model may be LLaMA, GPT-3, etc. Among them, since the large language model has a large number of parameters, during the pre-training process, the large language model may be fine-tuned using LoRA or Adapter, or the parameters may be directly frozen.

[0136] S504, using an image encoder to process the image, inputting M images to the image encoder, and outputting M image feature vectors, so that the image encoder learns the ability of images to represent texts. The image encoder can use a VIT (Vision-and-Language Integrated Transformer) or a CNN (Convolutional Neural Network) type network.

[0137] S505, a final vector is generated by using a linear layer to obtain the text vector obtained after the text processing in S503 and the image vector obtained after the image processing in S504.

[0138] S506: Use the Encoder-Decoder layer to output the next sequence of ASR text segments.

[0139] For example, Figure 6 As shown, the steps of the SFT stage of the classroom evaluation model include:

[0140] S601, converting the audio of the classroom observation video into multiple ASR text segments, usually one ASR segment is a continuous conversation in the classroom.

[0141] S602, take one of the ASR text segments, find the video segment occupied by the original video corresponding to the segment, and capture M pictures from the segment, where M can be any natural number, usually greater than 5.

[0142] S603, input an ASR text segment into a pre-trained large language model, extract the text vector of the ASR text segment, the large language model uses the model obtained in the above pre-training step, and the frozen parameters are no longer trained.

[0143] S604, use an image encoder to process the image, input M images to the image encoder, and output M image feature vectors. The image encoder uses the pre-trained Vit model, allowing parameters to be fine-tuned.

[0144] S605, the text vector obtained after the text processing in S603 and the image vector obtained after the image processing in S604 are used to generate a final vector using a linear layer, and the final vector is a sentence vector. The linear layer uses the parameters of the pre-training stage as initial values, allowing fine-tuning.

[0145] S606, all sentence vectors are taken as input, and after passing through the Encoder-Decoder layer, the classroom evaluation of the current class is finally output.

[0146] Please refer to the following Figure 7 , Figure 7 The following is a schematic diagram of a classroom evaluation device provided by an exemplary embodiment of the present application. Figure 7 As shown, the classroom evaluation device 700 includes:

[0147] A first acquisition module 710 is used to acquire video data of the class to be evaluated;

[0148] A first recognition module 720 is used to perform speech recognition on the video data to obtain multiple target text segments;

[0149] Evaluation module 730, used for inputting the above-mentioned multiple target text segments and the target video images corresponding to the above-mentioned multiple target text segments into the classroom evaluation model, obtaining the target text feature vectors and the target image feature vectors corresponding to the above-mentioned multiple target text segments, and outputting the target evaluation result of the above-mentioned classroom to be evaluated based on the target fused feature vector obtained by fusion of the above-mentioned target text feature vector and the above-mentioned target image feature vector;

[0150] Among them, the above-mentioned classroom evaluation model is obtained by training based on multiple classroom videos with known evaluation results.

[0151] In some possible embodiments, the classroom evaluation model includes a large language model and an image encoder, and the evaluation module 730 includes:

[0152] A first evaluation unit, configured to input the plurality of target text segments into the large language model respectively, and generate target text feature vectors corresponding to the plurality of target text segments respectively;

[0153] The second evaluation unit is used to input the target video images corresponding to the above-mentioned multiple target text segments into the above-mentioned image encoder respectively, and generate the target image feature vectors corresponding to the above-mentioned multiple target text segments respectively.

[0154] In some possible embodiments, the evaluation module 730 includes:

[0155] A first fusion unit is used to input the target text feature vectors and the target image feature vectors corresponding to the multiple target text segments into the linear layer of the classroom evaluation model to obtain multiple target fusion feature vectors;

[0156] The first output unit is used to input the above-mentioned multiple target fusion vectors into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the target evaluation result of the above-mentioned classroom to be evaluated.

[0157] The division of each module in the above-mentioned classroom evaluation device is only for illustration. In other embodiments, the classroom evaluation device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned classroom evaluation device. The implementation of each module in the classroom evaluation device provided in the embodiment of this specification can be in the form of a computer program. The computer program can be run on a terminal or a server. The program module composed of the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, all or part of the steps of the classroom evaluation method described in the embodiment of this specification are implemented.

[0158] Please refer to the following Figure 8 , Figure 8 The following is a schematic diagram of a classroom evaluation model training device provided by an exemplary embodiment of the present application. Figure 8 As shown, the training device 800 includes:

[0159] A second acquisition module 810 is used to acquire a plurality of first classroom videos with known evaluation results;

[0160] A second recognition module 820 is used to perform speech recognition on the first classroom video to obtain a plurality of first text segments;

[0161] The first training module 830 is used to input the above-mentioned multiple first text segments and the first video images corresponding to each of the above-mentioned multiple first text segments into the classroom evaluation model, obtain the first text feature vectors and the first image feature vectors corresponding to each of the above-mentioned multiple first text segments, and output the first prediction result of the above-mentioned first classroom video based on the first fused feature vector obtained by fusion of the above-mentioned first text feature vector and the above-mentioned first image feature vector; the above-mentioned first prediction result represents the prediction evaluation result of the above-mentioned first classroom video;

[0162] The second training module 840 is used to update the classroom evaluation model based on the evaluation results corresponding to each of the multiple first classroom videos and the first prediction results.

[0163] In some possible embodiments, the device 800 further includes:

[0164] A third acquisition module is used to acquire multiple second classroom videos;

[0165] A third recognition module is used to perform voice recognition on the second classroom video to obtain a plurality of second text segments; the second text segments are arranged in a preset order;

[0166] The third training module is used to input a target second text segment among the multiple second text segments and a second video image corresponding to the target second text segment into an initial evaluation model, obtain a second text feature vector and a second image feature vector corresponding to the target second text segment, and output a second prediction result of the target second text segment based on a second fused feature vector fused with the second image feature vector; the second prediction result represents a predicted second text segment corresponding to a next sequence of the target second text segment;

[0167] The fourth training module is used to update the initial evaluation model to obtain the classroom evaluation model based on the second text segment corresponding to the next sequence of the target second text segment and the predicted second text segment.

[0168] In some possible embodiments, the classroom evaluation model includes a large language model and an image encoder, and the first training module 830 includes:

[0169] A first training unit, configured to input the plurality of first text segments into the large language model respectively, and generate first text feature vectors corresponding to the plurality of first text segments respectively;

[0170] The second training unit is used to input the first video images corresponding to the multiple first text segments into the image encoder respectively, and generate the first image feature vectors corresponding to the multiple first text segments respectively.

[0171] In some possible embodiments, the first training module 830 includes:

[0172] A second fusion unit is used to input the first text feature vectors and the first image feature vectors corresponding to each of the plurality of first text segments into a linear layer of the classroom evaluation model to obtain a plurality of first fused feature vectors;

[0173] The second output unit is used to input the above-mentioned multiple first fusion vectors into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the first prediction result of the above-mentioned first classroom video.

[0174] The division of each module in the above-mentioned classroom evaluation model training device is only for illustration. In other embodiments, the classroom evaluation model training device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned classroom evaluation model training device. The implementation of each module in the classroom evaluation model training device provided in the embodiments of this specification can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, all or part of the steps of the classroom evaluation model training method described in the embodiments of this specification are implemented.

[0175] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of this specification. Fig. 9 As shown, the electronic device 900 may include: at least one processor 910, at least one communication bus 920, a user interface 930, at least one network interface 940, and a memory 950. The communication bus 920 may be used to realize the connection and communication of the above components.

[0176] The user interface 930 may include a display screen (Display) and a camera (Camera), and the optional user interface may also include a standard wired interface and a wireless interface.

[0177] The network interface 940 may optionally include a Bluetooth module, a Near Field Communication (NFC) module, a Wireless Fidelity (Wi-Fi) module, and the like.

[0178] Among them, the processor 910 may include one or more processing cores. The processor 910 uses various interfaces and lines to connect various parts within the entire electronic device 900, and executes various functions and processes data of the routing electronic device 900 by running or executing instructions, programs, code sets or instruction sets stored in the memory 950, and calling data stored in the memory 950. Optionally, the processor 910 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 910 can integrate one or more combinations of a processor (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 910, but may be implemented separately through a chip.

[0179] The memory 950 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 950 includes a non-transitory computer-readable medium. The memory 950 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 950 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as an acquisition function, an evaluation function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 950 may optionally be at least one storage device located away from the aforementioned processor 910. As Fig. 9 As shown, the memory 950 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.

[0180] Specifically, the processor 910 may be used to call the program instructions stored in the memory 950, and specifically perform the following operations:

[0181] Obtain video data of the class to be evaluated;

[0182] Performing speech recognition on the video data to obtain multiple target text segments;

[0183] Input the above-mentioned multiple target text segments and the target video images corresponding to the above-mentioned multiple target text segments into the classroom evaluation model, obtain the target text feature vectors and the target image feature vectors corresponding to the above-mentioned multiple target text segments, and output the target evaluation result of the above-mentioned classroom to be evaluated based on the target fused feature vector obtained by fusion of the above-mentioned target text feature vector and the above-mentioned target image feature vector;

[0184] Among them, the above-mentioned classroom evaluation model is obtained by training based on multiple classroom videos with known evaluation results.

[0185] In some possible embodiments, the classroom evaluation model includes a large language model and an image encoder. When the processor 910 inputs a plurality of target text segments and target video images corresponding to the plurality of target text segments into the classroom evaluation model to obtain target text feature vectors and target image feature vectors corresponding to the plurality of target text segments, the processor 910 is specifically used to execute:

[0186] Inputting the plurality of target text segments into the large language model respectively to generate target text feature vectors corresponding to the plurality of target text segments respectively;

[0187] The target video images corresponding to the above-mentioned multiple target text segments are respectively input into the above-mentioned image encoder to generate the target image feature vectors corresponding to the above-mentioned multiple target text segments.

[0188] In some possible embodiments, when the processor 910 executes the fused feature vector obtained by fusion of the target text feature vector and the target image feature vector to output the target evaluation result of the class to be evaluated, it is specifically used to execute:

[0189] Inputting the target text feature vectors and the target image feature vectors corresponding to the target text segments into the linear layer of the classroom evaluation model to obtain multiple target fusion feature vectors;

[0190] The above-mentioned multiple target fusion vectors are input into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the target evaluation results of the above-mentioned classroom to be evaluated.

[0191] In some possible embodiments, the electronic device 900 may be the aforementioned classroom evaluation model training device, and the processor 910 may further specifically execute:

[0192] Get multiple first-classroom videos with known evaluation results;

[0193] Performing speech recognition on the first classroom video to obtain a plurality of first text segments;

[0194] Input the above-mentioned multiple first text segments and the first video images corresponding to each of the above-mentioned multiple first text segments into the classroom evaluation model, obtain the first text feature vectors and the first image feature vectors corresponding to each of the above-mentioned multiple first text segments, and output the first prediction result of the above-mentioned first classroom video based on the first fused feature vector obtained by fusion of the above-mentioned first text feature vector and the above-mentioned first image feature vector; the above-mentioned first prediction result represents the prediction evaluation result of the above-mentioned first classroom video;

[0195] The classroom evaluation model is updated based on the evaluation results corresponding to each of the multiple first classroom videos and the first prediction results.

[0196] In some possible embodiments, before inputting the plurality of first text segments and the first video images corresponding to the plurality of first text segments into the classroom evaluation model, the processor 910 further specifically executes:

[0197] Get multiple second classroom videos;

[0198] Performing voice recognition on the second classroom video to obtain a plurality of second text segments; the second text segments are arranged in a preset order;

[0199] Inputting a target second text segment among the plurality of second text segments and a second video image corresponding to the target second text segment into an initial evaluation model, obtaining a second text feature vector and a second image feature vector corresponding to the target second text segment, and outputting a second prediction result of the target second text segment based on a second fused feature vector fused with the second image feature vector; the second prediction result represents a predicted second text segment corresponding to a next sequence of the target second text segment;

[0200] Based on the second text segment corresponding to the next sequence of the target second text segment and the predicted second text segment, the initial evaluation model is updated to obtain the classroom evaluation model.

[0201] In some possible embodiments, the classroom evaluation model includes a large language model and an image encoder. When the processor 910 inputs the plurality of first text segments and the first video images corresponding to the plurality of first text segments into the classroom evaluation model to obtain the first text feature vectors and the first image feature vectors corresponding to the plurality of first text segments, the processor 910 is specifically used to execute:

[0202] Inputting the plurality of first text segments into the large language model respectively to generate first text feature vectors corresponding to the plurality of first text segments respectively;

[0203] The first video images corresponding to the first text segments are respectively input into the image encoder to generate first image feature vectors corresponding to the first text segments.

[0204] In some possible embodiments, when the processor 910 outputs a first prediction result of the first classroom video based on a first fused feature vector fused from the first text feature vector and the first image feature vector, the processor 910 is specifically configured to execute:

[0205] Inputting the first text feature vectors and the first image feature vectors corresponding to each of the plurality of first text segments into the linear layer of the classroom evaluation model to obtain a plurality of first fused feature vectors;

[0206] The above-mentioned multiple first fusion vectors are input into the encoder-decoder network of the above-mentioned classroom evaluation model to obtain the first prediction result of the above-mentioned first classroom video.

[0207] The embodiment of the present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in the above embodiment. If the components of the above classroom evaluation device or classroom evaluation model training device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium.

[0208] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above-mentioned computer program product includes one or more computer instructions. When the above-mentioned computer program instructions are loaded and executed on a computer, the above-mentioned process or function according to the embodiment of the present application is generated in whole or in part. The above-mentioned computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium or transmitted by the above-mentioned computer-readable storage medium. The above-mentioned computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The above-mentioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The above-mentioned available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, digital versatile discs (DVD)), or semiconductor media (for example, solid state disks (SSD)), etc.

[0209] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes. In the absence of conflict, the technical features in this embodiment and the implementation scheme can be combined arbitrarily.

[0210] The embodiments described above are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the protection scope determined by the claims.

[0211] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims and the specification can be performed in an order different from the order in the embodiments recorded in the specification and still achieve the desired results. In addition, the process depicted in the drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A classroom evaluation method, characterized in that: The method comprises: Obtain video data of the class to be evaluated; Performing speech recognition on the video data to obtain multiple target text segments; Input the multiple target text segments and the target video images corresponding to the multiple target text segments into the classroom evaluation model, obtain the target text feature vectors and the target image feature vectors corresponding to the multiple target text segments, and output the target evaluation result of the classroom to be evaluated based on the target fused feature vector obtained by fusion of the target text feature vector and the target image feature vector; The classroom evaluation model is obtained by training based on multiple classroom videos with known evaluation results.

2. The method according to claim 1, characterized in that: The classroom evaluation model includes a large language model and an image encoder, and the multiple target text segments and the target video images corresponding to the multiple target text segments are input into the classroom evaluation model to obtain the target text feature vectors and the target image feature vectors corresponding to the multiple target text segments, including: Inputting the multiple target text segments into the large language model respectively, and generating target text feature vectors corresponding to the multiple target text segments respectively; The target video images corresponding to the multiple target text segments are respectively input into the image encoder to generate target image feature vectors corresponding to the multiple target text segments.

3. The method according to claim 1, characterized in that The outputting of the target evaluation result of the class to be evaluated based on the fused feature vector obtained by fusion of the target text feature vector and the target image feature vector comprises: Inputting the target text feature vectors and the target image feature vectors corresponding to each of the multiple target text segments into the linear layer of the classroom evaluation model to obtain multiple target fusion feature vectors; The multiple target fusion vectors are input into the encoder-decoder network of the classroom evaluation model to obtain the target evaluation result of the classroom to be evaluated.

4. A classroom evaluation model training method, characterized in that: The method comprises: Get multiple first-classroom videos with known evaluation results; Performing speech recognition on the first classroom video to obtain a plurality of first text segments; Input the multiple first text segments and the first video images corresponding to each of the multiple first text segments into the classroom evaluation model, obtain the first text feature vectors and the first image feature vectors corresponding to each of the multiple first text segments, and output the first prediction result of the first classroom video based on the first fused feature vector obtained by fusing the first text feature vector and the first image feature vector; the first prediction result represents the prediction evaluation result of the first classroom video; The classroom evaluation model is updated based on the evaluation results corresponding to each of the multiple first classroom videos and the first prediction result.

5. The method according to claim 4, characterized in that Before inputting the plurality of first text segments and the first video images respectively corresponding to the plurality of first text segments into the classroom evaluation model, the method further includes: Get multiple second classroom videos; Performing speech recognition on the second classroom video to obtain a plurality of second text segments; the second text segments are arranged in a preset order; Inputting a target second text segment among the multiple second text segments and a second video image corresponding to the target second text segment into an initial evaluation model, obtaining a second text feature vector and a second image feature vector corresponding to the target second text segment, and outputting a second prediction result of the target second text segment based on a second fused feature vector fused with the second image feature vector; the second prediction result represents a predicted second text segment corresponding to a next sequence of the target second text segment; Based on the second text segment corresponding to the next sequence of the target second text segment and the predicted second text segment, the initial evaluation model is updated to obtain the classroom evaluation model.

6. The method according to claim 4, characterized in that The classroom evaluation model includes a large language model and an image encoder, and the inputting of the plurality of first text segments and the first video images corresponding to the plurality of first text segments into the classroom evaluation model to obtain the first text feature vectors and the first image feature vectors corresponding to the plurality of first text segments includes: Inputting the plurality of first text segments into the large language model respectively, and generating first text feature vectors corresponding to the plurality of first text segments respectively; The first video images corresponding to each of the plurality of first text segments are respectively input into the image encoder to generate first image feature vectors corresponding to each of the plurality of first text segments.

7. The method according to claim 4, characterized in that The outputting a first prediction result of the first classroom video based on a first fused feature vector obtained by fusing the first text feature vector and the first image feature vector comprises: Inputting the first text feature vectors and the first image feature vectors corresponding to each of the plurality of first text segments into the linear layer of the classroom evaluation model to obtain a plurality of first fused feature vectors; The multiple first fusion vectors are input into the encoder-decoder network of the classroom evaluation model to obtain a first prediction result of the first classroom video.

8. A classroom evaluation device, characterized in that: The device comprises: The first acquisition module is used to acquire video data of the class to be evaluated; A first recognition module, configured to perform speech recognition on the video data to obtain a plurality of target text segments; An evaluation module, used for inputting the multiple target text segments and the target video images corresponding to the multiple target text segments into a classroom evaluation model, obtaining target text feature vectors and target image feature vectors corresponding to the multiple target text segments, and outputting a target evaluation result of the classroom to be evaluated based on a target fused feature vector obtained by fusion of the target text feature vector and the target image feature vector; The classroom evaluation model is obtained by training based on multiple classroom videos with known evaluation results.

9. A classroom evaluation model training device, characterized in that: The device comprises: A second acquisition module is used to acquire a plurality of first classroom videos with known evaluation results; A second recognition module is used to perform speech recognition on the first classroom video to obtain a plurality of first text segments; A first training module is used to input the multiple first text segments and the first video images corresponding to the multiple first text segments into a classroom evaluation model, obtain the first text feature vectors and the first image feature vectors corresponding to the multiple first text segments, and output a first prediction result of the first classroom video based on a first fused feature vector obtained by fusing the first text feature vector and the first image feature vector; the first prediction result represents a predicted evaluation result of the first classroom video; The second training module is used to update the classroom evaluation model based on the evaluation results corresponding to each of the multiple first classroom videos and the first prediction results.

10. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 7.

Images