Method, system and device for generating wonderful classroom scene information and storage medium

Abstract

The embodiment of the invention relates to the technical field of teaching management, in particular to a classroom wonderful scene information generation method, which comprises the following steps: acquiring classroom observation data; according to the classroom observation data, a classroom live-recording manuscript and an effective teaching behavior event are determined, the classroom live-recording manuscript is a manuscript obtained by converting voice data collected in a classroom into text content, and the effective teaching behavior event is a behavior event corresponding to an effective teaching behavior; and generating wonderful classroom scene information according to the effective teaching behavior event and the classroom live record manuscript. Wherein the wonderful classroom scene information is mainly obtained according to the effective teaching behavior events, so that the influence of behavior events irrelevant to the effective teaching behavior events is avoided, the recognition accuracy of the wonderful classroom scene information is improved, and the application effect of the wonderful classroom scene information is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and in particular to a method, system, device and storage medium for generating classroom wonderful scene information. Background Art

[0002] Classroom wonderful scene information refers to the key information generated by classroom participants during classroom teaching. Analyzing the classroom teaching process based on classroom wonderful scene information helps improve the effect of classroom teaching. At present, the relevant technology mainly combines ASR technology and computer vision technology when generating classroom wonderful scene information. Through AI, the start and end time of wonderful events are identified and the video is captured to obtain classroom wonderful scene information.

[0003] However, there are many complex factors in real classroom teaching scenarios, and the participants in the classroom will have a variety of behavioral events, which can easily lead to inaccurate information about some wonderful classroom scenes and fail to reflect the key situations of classroom teaching. For example, students may make a variety of unconscious movements in the classroom, which may be saved and used to generate wonderful classroom scene information, but this type of information cannot be used to judge the effectiveness of classroom teaching, affecting the accuracy of identifying wonderful classroom scene information. Summary of the invention

[0004] One purpose of an embodiment of the present application is to provide a method for generating classroom wonderful scene information to solve the technical problem of low recognition accuracy of classroom wonderful scene information.

[0005] In a first aspect, an embodiment of the present application provides a method for generating classroom wonderful scene information, comprising:

[0006] Obtain classroom observation data;

[0007] Determining classroom transcripts and effective teaching behavior events according to the classroom observation data, wherein the classroom transcripts are transcripts converted from voice data collected in the classroom into text content, and the effective teaching behavior events are behavior events corresponding to effective teaching behaviors;

[0008] The classroom wonderful scene information is generated according to the effective teaching behavior events and the classroom transcript.

[0009] In combination with the first aspect, in a possible implementation, generating classroom wonderful scene information according to the effective teaching behavior event and the classroom transcript includes:

[0010] A sentence segment corresponding to the effective teaching behavior event is intercepted from the classroom transcript as a wonderful sentence segment, the time range of the wonderful sentence segment is equal to the effective time range of the effective teaching behavior event, and the effective time range is a time range with a preset time length floating up and down around the time point of the effective teaching behavior event;

[0011] Generate classroom wonderful scene information based on the wonderful sentence fragments.

[0012] In combination with the first aspect, in a possible implementation, the extracting the sentence segments corresponding to the effective teaching behavior events from the classroom transcript as the wonderful sentence segments includes:

[0013] The classroom transcript, the effective teaching behavior events and the effective time range are input into a preset teaching behavior model to obtain wonderful sentence fragments.

[0014] In combination with the first aspect, in a possible implementation manner, the classroom wonderful scene information includes wonderful summary information about the wonderful sentence fragment, and generating the classroom wonderful scene information according to the wonderful sentence fragment includes:

[0015] The wonderful summary information is input into a preset teaching behavior macro model to obtain the wonderful summary information.

[0016] In combination with the first aspect, in a possible implementation, the classroom observation data includes video data collected in the classroom, the classroom wonderful scene information includes a wonderful video segment with the wonderful sentence segment, and generating the classroom wonderful scene information according to the wonderful sentence segment includes:

[0017] Determine the start time and end time of the wonderful sentence fragment;

[0018] Determine the wonderful video segment corresponding to the start time and end time of the wonderful sentence segment on the time axis of the video data.

[0019] In combination with the first aspect, in a possible implementation, the classroom wonderful scene information further includes a wonderful cover, and the generating of the classroom wonderful scene information according to the wonderful sentence fragments further includes:

[0020] Extracting key frames satisfying preset constraints from the wonderful video clips as target key frames;

[0021] Obtaining wonderful summary information related to the wonderful sentence fragment;

[0022] A wonderful cover is generated according to the target key frame and the wonderful summary information.

[0023] In combination with the first aspect, in a possible implementation manner, the wonderful video segment includes a plurality of key frames arranged in a frame sequence, and extracting a key frame satisfying a preset constraint condition as a target key frame from the wonderful video segment includes:

[0024] If the frame sequence of the key frame is the specified frame sequence, it is determined that the key frame meets the preset constraint condition, and the key frame is selected as the target key frame.

[0025] In conjunction with the first aspect, in a possible implementation manner, the classroom observation data includes audio data collected in the classroom, and determining the classroom transcript according to the classroom observation data includes:

[0026] According to a preset speech recognition model, the audio data is converted into a classroom transcript.

[0027] In combination with the first aspect, in a possible implementation, the classroom observation data includes audio data, video data and classroom teaching information collected in the classroom, the effective teaching behavior event is configured with an effective time range, and determining the effective teaching behavior event according to the classroom observation data includes:

[0028] Performing multimodal analysis based on the audio data, video data and classroom teaching information to obtain teaching behavior events and behavior occurrence time points;

[0029] The teaching behavior event and the time point of occurrence of the behavior are input into a preset state machine to obtain a valid teaching behavior event and a valid time range.

[0030] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0031] Acquire auxiliary association information, where the auxiliary association information is used to assist in generating the classroom wonderful scene information;

[0032] Generate packaging information from the auxiliary association information;

[0033] The block chain operation is performed on the packaged information to obtain a block.

[0034] In the second aspect, an embodiment of the present application also proposes a system for generating classroom wonderful scene information, including a data acquisition device and a terminal device, the terminal device includes a memory and a processor, the processor is used to execute one or more computer programs stored in the memory, so that the terminal device controls the data acquisition device to implement the method described above.

[0035] In a third aspect, the embodiment of the present application further proposes a device for generating classroom wonderful scene information, including:

[0036] Data acquisition module, used to obtain classroom observation data;

[0037] A data confirmation module, used to determine classroom transcripts and effective teaching behavior events according to the classroom observation data, wherein the classroom transcripts are transcripts converted from voice data collected in the classroom into text content, and the effective teaching behavior events are behavior events corresponding to effective teaching behaviors;

[0038] The data generation module is used to generate classroom wonderful scene information based on the effective teaching behavior events and the classroom transcript.

[0039] In a fourth aspect, an embodiment of the present application further proposes a computer device, comprising a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory, and when the processor executes the one or more computer programs, the computer device implements the method described above.

[0040] In a fifth aspect, an embodiment of the present application further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described above.

[0041] The embodiments of the present application can achieve the following technical effects:

[0042] Based on the method proposed in the embodiment of the present application, when generating classroom wonderful scene information, classroom observation data is obtained; classroom transcripts and effective teaching behavior events are determined based on the classroom observation data, where the classroom transcripts are transcripts converted from voice data collected in the classroom into text content, and effective teaching behavior events are behavior events corresponding to effective teaching behaviors; classroom wonderful scene information is generated based on effective teaching behavior events and classroom transcripts. Among them, since the classroom wonderful scene information is mainly obtained based on effective teaching behavior events, the influence of behavior events unrelated to effective teaching behavior events is avoided, the recognition accuracy of the classroom wonderful scene information is improved, and thus the application effect of the classroom wonderful scene information is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1A schematic diagram of a system for generating wonderful classroom scene information provided in an embodiment of the present application;

[0045] Figure 2 A flow chart of a method for generating classroom wonderful scene information provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a device for generating wonderful classroom scene information provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0049] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in this application do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0050] In order to facilitate the explanation of this application, the application scenarios involved in the embodiments of this application are first introduced. Figure 1 , Figure 1 The figure shows a system 10 for generating information on wonderful classroom scenes, including a collection device 11, a server 12 and a terminal device 13. The server 12 is respectively connected to the collection device 11 and the terminal device 13 for communication, so as to receive the classroom observation data collected by the collection device 11 and perform analysis and processing. The results of the analysis and processing are sent to the terminal device 13 for presentation.

[0051] Among them, the communication connection between the server 12 and the collection device 11 and the terminal device 13 can be in the form of wired communication connection or wireless communication connection. The wired communication connection includes various communication connections that use tangible media such as metal wires and optical fibers to transmit information; the wireless communication connection includes 5G communication, Zig-Bee communication, Bluetooth communication, wireless broadband (WiFi), ultra-wideband (UWB) and near-field communication (NFC), etc.

[0052] As a feasible implementation, the collection device 11 is set in a classroom where classroom teaching is carried out, and is equipped with a communication module and a sensor component. The communication module is used to communicate with the server 12, and the sensor component is used to collect and record classroom observation data, mainly including audio and video data generated by teachers and students during classroom teaching. For example, the sensor component may include a microphone and / or a camera, the microphone is used to collect audio data during teaching, and the camera is used to collect video data during teaching.

[0053] In some embodiments, the server 12 has a certain data processing capability, and is provided with a memory and a processor. The memory stores program instructions for processing classroom observation data. When the program instructions are executed by the processor, the server 12 executes a corresponding method, specifically including: obtaining classroom observation data; determining classroom transcripts and effective teaching behavior events based on the classroom observation data, wherein the classroom transcript is a transcript converted from voice data collected in the classroom into text content, and the effective teaching behavior event is a behavior event corresponding to an effective teaching behavior; generating classroom wonderful scene information based on the effective teaching behavior events and the classroom transcript.

[0054] In some embodiments, the server 12 sends the generated classroom wonderful scene information to the terminal device 13 for presentation to the user. The terminal device 13 is provided with a display device. The user watches the classroom wonderful scene information through the display device to understand the classroom teaching situation and improve the effect of classroom teaching.

[0055] In some embodiments, when generating classroom wonderful scene information, the collection device 11, the server 12 and the terminal device 13 package the required auxiliary related information to generate packaged information, and perform a blockchain on-chain operation on the packaged information, thereby setting the collection device 11, the server 12 and the terminal device 13 as blocks to build a blockchain. Among them, each block performs smart contract management on device information based on consensus storage credentials, and encrypts and uploads various generated data according to the distributed ID service contract, thereby improving the data's tamper-proof capability and reliability.

[0056] As a feasible implementation method, when the collection device 11 sends classroom observation data to the server 12, it generates unstructured data hash code information and performs blockchain on-chain operations. When the server 12 sends classroom wonderful scene information to the terminal device 13, it generates unstructured data point hash code and performs blockchain on-chain operations. The user uses the terminal device 13 to authenticate and log in. After the authentication is passed, the hash code of the user login will be calculated and the blockchain on-chain operation will be performed.

[0057] Based on the above application scenarios, an embodiment of the present application proposes a method for generating classroom wonderful scene information.

[0058] For details, please refer to Figure 2 ,like Figure 2 As shown, the method for generating classroom wonderful scene information includes:

[0059] Step S10, obtaining classroom observation data;

[0060] Among them, classroom observation data refers to relevant data that can be used to generate information about wonderful classroom scenes. Information about wonderful classroom scenes refers to key information jointly generated by classroom participants during a classroom teaching process. Depending on the key information evaluation criteria, there may be multiple types of key information in the same classroom teaching. For example, if the evaluation criteria for key information is the interaction effect between teachers and students, then in this classroom teaching, the scene with a good interaction effect between teachers and students (the interaction effect can be determined according to pre-set rules) is the corresponding key information.

[0061] It should be noted that there are many ways to generate classroom observation data. All data related to key information evaluation can be used as classroom observation data. Figure 1 Taking the system shown as an example, the data collected by the collection device in the classroom can be used as a kind of classroom observation data, and the teaching resource information pre-set by the user can also be stored in the server as another kind of classroom observation data. Multiple classroom observation data can be stored in corresponding storage media or in the cloud for easy analysis and processing.

[0062] In some embodiments, classroom observation data mainly includes audio and video data generated by teachers and students during classroom teaching and recorded by acquisition devices in the classroom. Subsequent analysis and processing of this audio and video data can be used to understand the actual situation of classroom teaching and generate information about wonderful classroom scenes.

[0063] In other embodiments, classroom observation data may also include teaching resource data related to classroom teaching, such as courseware, lesson plans used by teachers in classroom teaching, and exercises prepared for students, etc. This part of data can be processed into a form that can be understood by computers through resource integration, which is conducive to more accurate generation of wonderful classroom scene information.

[0064] Step S20, determining a classroom transcript and an effective teaching behavior event according to the classroom observation data, wherein the classroom transcript is a transcript converted from voice data collected in the classroom into text content, and the effective teaching behavior event is a behavior event corresponding to an effective teaching behavior;

[0065] Among them, the classroom transcript referred to in this embodiment is obtained by performing voice recognition conversion on the voice data in the classroom observation data, and its essence is a file that records the semantics of the corresponding voice data; effective teaching behavior events refer to behavior events corresponding to effective teaching behaviors pre-set by the user. It is easy to understand that a classroom teaching includes multiple behavior events, such as the teacher's movement on the podium, blackboard writing, students unconsciously raising their hands, discussions or speeches, etc., but under normal circumstances, most behavior events are invalid teaching behavior events that are not related to effective teaching behaviors. Therefore, it is necessary to determine the effective teaching behavior events so that in the subsequent steps, the classroom wonderful scene information based on the effective behavior events can be determined.

[0066] In some embodiments, when determining an effective teaching behavior event, first determine all teaching behavior events based on classroom observation data, then compare each teaching behavior event with a preset effective teaching behavior, and if it is determined that a certain teaching behavior event corresponds to an effective teaching behavior, then the teaching behavior event can be determined to be an effective teaching behavior event. It is easy to understand that the setting of effective teaching behavior can be achieved by setting multiple consecutive action behaviors. For example, a student answering a question raised by a teacher is considered an effective teaching behavior, and a consecutive action of "student raising hand-standing-sitting down" can be set for the effective teaching behavior. When a certain teaching behavior event corresponds to the consecutive action of determining a student raising hand-standing-sitting down, it is considered that the student has answered the question raised by the teacher and an effective teaching behavior has been generated.

[0067] Step S30, generating classroom wonderful scene information according to the effective teaching behavior events and the classroom transcript.

[0068] In this embodiment, the information on wonderful classroom scenes mainly includes wonderful classroom moments extracted from classroom observation data, as well as auxiliary information related to the wonderful classroom moments. For example, this auxiliary information may include a summary or description of the wonderful classroom scenes, such as "students raise their hands to answer questions", "students discuss and interact in groups", etc. The summary or description can reflect the characteristics of the wonderful classroom scenes and help users understand the content of the wonderful classroom moments.

[0069] Furthermore, in the above embodiment, generating classroom wonderful scene information based on effective teaching behavior events and classroom transcripts includes: extracting sentence fragments corresponding to effective teaching behavior events in the classroom transcripts as wonderful sentence fragments, the time range of the wonderful sentence fragments is equal to the effective time range of the effective teaching behavior events, and the effective time range is a time range that fluctuates up and down for a preset time length with the time point of occurrence of the effective teaching behavior event as the center; generating classroom wonderful scene information based on the wonderful sentence fragments.

[0070] Among them, the classroom transcript includes the participants of classroom teaching, mainly the dialogue text between teachers and students. The dialogue text includes multiple sentence fragments, and the multiple sentence fragments can be divided into two categories, namely, wonderful sentence fragments and non-wonderful sentence fragments. It is easy to understand that in an actual classroom scenario, each sentence fragment can correspond to a behavioral event. For example, in a certain sentence fragment, the teacher's speech content is: Please raise your hand to answer the question, and the student's speech content is the answer to the question. Then this text corresponds to the behavioral event of the teacher asking questions-students raising their hands to answer. If the preset effective teaching behavior events include the event of the teacher asking questions-students raising their hands to answer, then this part of the sentence fragment can be determined as a wonderful sentence fragment.

[0071] In this embodiment, the effective time range refers to the context time of the corresponding effective teaching behavior event. It can be understood that the classroom transcript includes the content and events of each sentence fragment. Therefore, after determining the wonderful sentence fragment, the time range corresponding to the wonderful sentence fragment can be determined according to the range of the wonderful sentence fragment, that is, the time point when the behavior of the effective teaching behavior event occurs is determined, and the time point is fluctuated up and down by a certain length of time according to the time point, which expands the context time of the original effective teaching behavior event, that is, determines the effective time range.

[0072] Specifically, the time range of the preset duration refers to the time range in which the context time of the effective teaching behavior can be counted. It is easy to understand that for an effective teaching behavior event, improving the completeness of the corresponding classroom wonderful scene information will help users better analyze the event, thereby making more reasonable classroom teaching analysis and suggestions. For example, assuming that the effective teaching behavior event is "teacher asks questions-students raise their hands to answer", the start time point and the end time point of the behavior are (a, b), and a and b are the coordinates of the time axis of classroom teaching. If the preset floating duration is f, then the corresponding effective time range can be determined to be (a±f, b±f). Compared with (a, b), the time range of (a±f, b±f) can also include context information such as the type of questions asked by the teacher and whether the student answers the questions correctly, so that the effective time range corresponding to the effective behavior event is more complete.

[0073] In some embodiments, extracting sentence segments corresponding to effective teaching behavior events on the classroom transcript as wonderful sentence segments includes: inputting the classroom transcript, effective teaching behavior events and effective time range into a preset teaching behavior model to obtain wonderful sentence segments.

[0074] It should be noted that the specific form of the teaching behavior model referred to in the present embodiment can be a LLM (Large Language Module, i.e., large language model) model. The LLM model refers to a text generation model based on deep learning and natural language processing, and its basic principle is to use a neural network model to learn the statistical laws and semantic information of the input text to generate a text that meets the user's expectations. In the present embodiment, the user's expectations mainly refer to identifying some wonderful sentence fragments in the multiple sentence fragments contained in the classroom transcript.

[0075] In this embodiment, the semantic analysis capability of the LLM model for text is mainly utilized. The LLM model has undergone a lot of training before use, so it can perform semantic analysis on the input text data and identify the wonderful sentence fragments corresponding to the effective teaching behavior events.

[0076] In some embodiments, classroom wonderful scene information includes wonderful summary information of wonderful sentence fragments. Generating classroom wonderful scene information according to the wonderful sentence fragments includes: inputting the wonderful sentence fragments into a preset teaching behavior model to obtain the wonderful summary information.

[0077] Among them, the wonderful summary information refers to the summary information that can reflect the main features of the wonderful sentence fragment. Since the essence of the wonderful sentence fragment is the text corresponding to the effective behavior event, the wonderful summary information also represents the information of the effective behavior event. In addition, the wonderful summary information can also include information summarizing the wonderful sentence fragment, such as judging the content of the wonderful summary information. For example, if the wonderful language text is "Teacher asks questions - students raise their hands to answer", the wonderful summary information can also include whether the students have correctly answered the teacher's questions, etc.

[0078] It is understandable that the teaching behavior model needs to undergo necessary training before use so that it can output wonderful summary information based on the wonderful sentence fragments input by the user. Since the specific form of the teaching behavior model is the LLM model, its training process can refer to the training process of the LLM model. The basic principle is to perform supervised training through a pre-created training data set, which includes a large amount of text data, so that the LLM model can learn the language structure and contextual features of the wonderful summary information.

[0079] Furthermore, in the above embodiment, the classroom observation data includes video data collected in the classroom, the classroom wonderful scene information includes wonderful video clips with wonderful sentence fragments, and generating the classroom wonderful scene information based on the wonderful sentence fragments includes: determining the start time and end time of the wonderful sentence fragments; determining the wonderful video fragments corresponding to the start time and end time of the wonderful sentence fragments on the timeline of the video data.

[0080] It can be understood that since the video data is collected during classroom teaching, the video data and the classroom transcript correspond to each other. Based on this understanding, for various behavioral events included in the classroom transcript, the corresponding video content can be determined in the video data, that is, for the wonderful sentence fragments associated with the effective behavioral events, the corresponding wonderful video fragments can be determined in the video data.

[0081] More specifically, since the video data is collected in real time by the collection device in the classroom, the final video data should be a video stream with multiple event nodes. Correspondingly, since the wonderful sentence segments are intercepted from the classroom transcript, each wonderful sentence segment corresponds to a start time and an end time. According to the start time and the end time, the video segments are intercepted in the video stream, and the wonderful video segments corresponding to each wonderful sentence segment can be obtained.

[0082] In some embodiments, classroom wonderful scene information also includes a wonderful cover, and generating classroom wonderful scene information based on wonderful sentence fragments also includes: traversing key frames that meet preset constraints in the wonderful video fragments as target key frames; obtaining wonderful summary information related to the wonderful sentence fragments; and generating a wonderful cover based on the target key frames and the wonderful summary information.

[0083] Among them, the wonderful cover refers to a video cover used to characterize the characteristics of the wonderful video clip, and the video cover includes a frame image belonging to the wonderful video clip, that is, a target key frame, which is extracted from all video frames in the wonderful video clip according to the constraints pre-set by the user, and represents the most critical content in the wonderful video clip. After obtaining the target key frame, combined with the wonderful summary information in the above embodiment, a video cover can be set for the wonderful video clip.

[0084] As a feasible implementation method, the constraints pre-set by the user can be the basis for analyzing the characteristics of the wonderful video from any dimension. For example, the user believes that the student's performance in class is the most critical, that is, the student's performance is the constraint condition. Then, for a certain wonderful video clip, the video frame most relevant to the student's behavioral event can be set as the target key frame.

[0085] In some embodiments, the wonderful video clip includes multiple key frames arranged in a frame order. When traversing the target key frame, the target key frame is determined by judging the frame order of each key frame. If the frame order of the key frame is a specified frame order, it is determined that the key frame meets the preset constraint condition, and the key frame is selected as the target key frame. Among them, the specified frame order can be a frame order corresponding to the effective teaching behavior event represented by the wonderful sentence segment corresponding to the wonderful video clip. The process of traversing the target key frame can be achieved by various computer vision technologies, which will not be repeated in this embodiment.

[0086] Furthermore, in the above embodiment, the classroom observation data includes audio data collected in the classroom, and determining the classroom transcript based on the classroom observation data includes: converting the audio data into the classroom transcript based on a preset speech recognition model.

[0087] Among them, the specific form of the speech recognition model can be a model that adopts ASR (Automatic Speech Recognition) technology. Its basic principle is to convert the original speech signal into semantically consistent text, such as the classroom transcript in this embodiment, by analyzing the spectrum, sound characteristics and contextual information of the speech signal.

[0088] Furthermore, in the above embodiment, the classroom observation data includes audio data, video data and classroom teaching information collected in the classroom, and the effective teaching behavior events are configured with an effective time range. Determining the effective teaching behavior events based on the classroom observation data includes: performing multimodal analysis based on the audio data, video data and classroom teaching information to obtain the teaching behavior events and the time points when the behaviors occur; inputting the teaching behavior events and the time points when the behaviors occur into a preset state machine to obtain the effective teaching behavior events and the effective time range.

[0089] Among them, multimodal analysis includes multi-dimensional analysis of audio and video and teaching materials through computer vision, speech recognition processing, NLP algorithm, etc. The results of the multi-dimensional analysis include all teaching behavior events generated by all participants in the classroom during a classroom teaching process, which can be divided into effective teaching behavior events and invalid teaching behavior events. The classification rules of effective teaching behavior events and invalid teaching behavior events are pre-set in the state machine. Therefore, effective teaching behavior events and corresponding effective time ranges can be determined based on all teaching behavior events and the time points of the behavior occurrence.

[0090] In some embodiments, the process of generating classroom wonderful scene information also includes: obtaining auxiliary related information, the auxiliary related information is used to assist in generating classroom wonderful scene information; generating packaged information from the auxiliary related information; performing a blockchain on-chain operation on the packaged information to obtain a block.

[0091] Among them, the auxiliary associated information mainly refers to the credential information of each node device when executing all the above embodiments or the data credential information generated during the algorithm execution process, such as device fingerprint credentials, IOT service version credentials, audio and video real-time interactive service version credentials, IOT device credential information, Hashcold credentials of classroom observation data, algorithm engine version credentials, input video credentials, output unstructured information credentials, observation service version credentials, etc. After being packaged, this information is uploaded to the chain to enable all node devices to provide blockchain services and improve data tamper-proofing and reliability.

[0092] It should be noted that, in each of the above-mentioned embodiments, there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present application, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0093] As another aspect of the embodiment of the present application, the embodiment of the present application provides a device for generating classroom wonderful scene information. The device for generating classroom wonderful scene information can be a software module, and the software module includes a number of instructions, which are stored in a memory, and the processor can access the memory and call the instructions for execution to complete the method for generating classroom wonderful scene information described in each of the above embodiments.

[0094] In some embodiments, the generation device of classroom wonderful scene information can also be constructed by hardware devices. For example, the generation device of classroom wonderful scene information can be constructed by one or more chips, and each chip can work in coordination with each other to complete the generation device of classroom wonderful scene information described in each of the above embodiments. For another example, the generation device of classroom wonderful scene information can also be constructed by various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), single-chip microcomputers, ARM (AcornRISC Machine) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination of these components.

[0095] For details, please refer to Figure 3 ,like Figure 3 The figure shows a schematic diagram of the structure of a device for generating wonderful classroom scene information, comprising:

[0096] A data acquisition module 310 is used to acquire classroom observation data;

[0097] A data determination module 320 is used to determine classroom transcripts and effective teaching behavior events according to the classroom observation data, wherein the classroom transcripts are transcripts converted from voice data collected in the classroom into text content, and the effective teaching behavior events are behavior events corresponding to effective teaching behaviors;

[0098] The data generation module 330 is used to generate classroom wonderful scene information based on the effective teaching behavior events and the classroom transcript.

[0099] In a possible implementation, the data generation module 330, when used to generate classroom wonderful scene information according to the effective teaching behavior event and the classroom transcript, is specifically used to:

[0100] A sentence segment corresponding to the effective teaching behavior event is intercepted from the classroom transcript as a wonderful sentence segment, the time range of the wonderful sentence segment is equal to the effective time range of the effective teaching behavior event, and the effective time range is a time range with a preset time length floating up and down around the time point of the effective teaching behavior event;

[0101] Generate classroom wonderful scene information based on the wonderful sentence fragments.

[0102] In a possible implementation, the data generation module 330, when used to intercept the sentence fragment corresponding to the effective teaching behavior event as a wonderful sentence fragment, is specifically used to:

[0103] The classroom transcript, the effective teaching behavior events and the effective time range are input into a preset teaching behavior model to obtain wonderful sentence fragments.

[0104] In a possible implementation, the classroom wonderful scene information includes wonderful summary information about the wonderful sentence fragments, and the data generation module 330, when used to generate the classroom wonderful scene information according to the wonderful sentence fragments, is specifically used to:

[0105] The wonderful summary information is input into a preset teaching behavior macro model to obtain the wonderful summary information.

[0106] In a possible implementation, the classroom observation data includes video data collected in the classroom, the classroom wonderful scene information includes wonderful video clips with the wonderful sentence fragments, and the data generation module 330, when used to generate the classroom wonderful scene information according to the wonderful sentence fragments, is specifically used to:

[0107] Determine the start time and end time of the wonderful sentence fragment;

[0108] Determine the wonderful video segment corresponding to the start time and end time of the wonderful sentence segment on the time axis of the video data.

[0109] In a possible implementation, the classroom wonderful scene information also includes a wonderful cover, and the data generation module 330, when used to generate the classroom wonderful scene information according to the wonderful sentence fragment, is specifically used to:

[0110] Extracting key frames satisfying preset constraints from the wonderful video clips as target key frames;

[0111] Obtaining wonderful summary information related to the wonderful sentence fragment;

[0112] A wonderful cover is generated according to the target key frame and the wonderful summary information.

[0113] In a possible implementation, the wonderful video clip includes multiple key frames arranged in frame order, and the data generation module 330, when used to extract key frames satisfying preset constraints from the wonderful video clip as target key frames, is specifically used to:

[0114] If the frame sequence of the key frame is the specified frame sequence, it is determined that the key frame meets the preset constraint condition, and the key frame is selected as the target key frame.

[0115] In a possible implementation, the classroom observation data includes audio data collected in the classroom, and the data determination module 320, when used to determine the classroom transcript according to the classroom observation data, is specifically used to:

[0116] According to a preset speech recognition model, the audio data is converted into a classroom transcript.

[0117] In a possible implementation, the classroom observation data includes audio data, video data and classroom teaching information collected in the classroom, the effective teaching behavior event is configured with an effective time range, and the data determination module 320, when used to determine the effective teaching behavior event according to the classroom observation data, is specifically used to:

[0118] Performing multimodal analysis based on the audio data, video data and classroom teaching information to obtain teaching behavior events and behavior occurrence time points;

[0119] The teaching behavior event and the time point of occurrence of the behavior are input into a preset state machine to obtain a valid teaching behavior event and a valid time range.

[0120] In a possible implementation, the data generating module 330 is further configured to:

[0121] Acquire auxiliary association information, where the auxiliary association information is used to assist in generating the classroom wonderful scene information;

[0122] Generate packaging information from the auxiliary association information;

[0123] The block chain operation is performed on the packaged information to obtain a block.

[0124] It should be noted that the above-mentioned device for generating classroom wonderful scene information can execute the method for generating classroom wonderful scene information provided in the implementation mode of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in the implementation mode of the device for generating classroom wonderful scene information, please refer to the method for generating classroom wonderful scene information provided in the implementation mode of this application.

[0125] See also Figure 4 , Figure 4 4 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device 40 includes one or more processors 41 and a memory 42. The memory 42 is connected to the one or more processors 41, for example, connected to the processor 41 via a bus.

[0126] The processor 41 is configured to support the computer device to perform the corresponding functions in the method in the above method embodiment. The processor 41 can be a central processing unit 41 (CPU), a network processor 41 (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0127] The memory 42 is used to store program codes, etc. The memory 42 may include a volatile memory 42 (VM), such as a random access memory 42 (RAM); the memory 42 may also include a non-volatile memory 42 (NVM), such as a read-only memory 42 (ROM), a flash memory 42, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 42 may also include a combination of the above-mentioned types of memory 42.

[0128] The memory 42 can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method for generating classroom wonderful scene information in the embodiment of the present application. The processor 41 executes various functional applications and data processing of the method for generating classroom wonderful scene information and the device for generating classroom wonderful scene information by running the non-volatile software programs, instructions and modules stored in the memory 42, that is, realizes the functions of each module or unit of the method for generating classroom wonderful scene information and the device for generating classroom wonderful scene information provided in the above method embodiment.

[0129] The memory 42 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function. The data storage area may store data created according to the use of a device for generating classroom wonderful scene information, etc. In some embodiments, the memory 42 may optionally include a memory 42 remotely arranged relative to the processor 41, and these remote memories 42 may be connected to the device for generating classroom wonderful scene information via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The one or more modules are stored in the memory 42. When executed by the one or more processors 41, the device for generating the wonderful classroom scene information in any of the above-mentioned method embodiments is executed. For example, the method steps described in the above-mentioned method embodiments are executed to realize the functions of the modules described in the above-mentioned device embodiments.

[0131] An embodiment of the present application also provides a system for generating classroom wonderful scene information, including a data acquisition device and a terminal device, the terminal device including a memory and a processor, the processor is used to execute one or more computer programs stored in the memory, so that the terminal device controls the data acquisition device to implement the method described in the above embodiment.

[0132] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0133] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0134] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for generating classroom wonderful scene information, characterized in that: include: Obtain classroom observation data; Determining classroom transcripts and effective teaching behavior events according to the classroom observation data, wherein the classroom transcripts are transcripts converted from voice data collected in the classroom into text content, and the effective teaching behavior events are behavior events corresponding to effective teaching behaviors; The classroom wonderful scene information is generated according to the effective teaching behavior events and the classroom transcript.

2. The generation method according to claim 1, characterized in that: The generating of classroom wonderful scene information according to the effective teaching behavior event and the classroom transcript includes: A sentence segment corresponding to the effective teaching behavior event is intercepted from the classroom transcript as a wonderful sentence segment, the time range of the wonderful sentence segment is equal to the effective time range of the effective teaching behavior event, and the effective time range is a time range with a preset time length floating up and down around the time point of the effective teaching behavior event; Generate classroom wonderful scene information based on the wonderful sentence fragments.

3. The generation method according to claim 2, characterized in that: The sentence segments corresponding to the effective teaching behavior events are intercepted from the classroom transcript as wonderful sentence segments, including: The classroom transcript, the effective teaching behavior events and the effective time range are input into a preset teaching behavior macro model to obtain the wonderful sentence fragments.

4. The generation method according to claim 2, characterized in that: The classroom wonderful scene information includes wonderful summary information of the wonderful sentence fragments, and generating the classroom wonderful scene information according to the wonderful sentence fragments includes: The wonderful sentence fragments are input into a preset teaching behavior model to obtain wonderful summary information.

5. The generation method according to claim 2, characterized in that: The classroom observation data includes video data collected in the classroom, the classroom wonderful scene information includes wonderful video segments corresponding to the wonderful sentence segments, and the generation of classroom wonderful scene information according to the wonderful sentence segments includes: Determine the start time and end time of the wonderful sentence fragment; Determine the wonderful video segment corresponding to the start time and end time of the wonderful sentence segment on the time axis of the video data.

6. The generation method according to claim 5, characterized in that: The classroom wonderful scene information also includes a wonderful cover, and the generation of classroom wonderful scene information according to the wonderful sentence fragments also includes: Traversing the key frames satisfying preset constraints in the wonderful video clip as target key frames; Obtaining wonderful summary information related to the wonderful sentence fragment; A wonderful cover is generated according to the target key frame and the wonderful summary information.

7. The generation method according to claim 6, characterized in that: The wonderful video clip includes a plurality of key frames arranged in a frame order, and traversing the key frames satisfying preset constraints in the wonderful video clip as target key frames includes: If the frame sequence of the key frame is the specified frame sequence, it is determined that the key frame meets the preset constraint condition, and the key frame is selected as the target key frame.

8. The generation method according to any one of claims 1 to 7, characterized in that: The classroom observation data includes audio data collected in the classroom, and determining the classroom transcript according to the classroom observation data includes: According to a preset speech recognition model, the audio data is converted into a classroom transcript.

9. The generation method according to any one of claims 1 to 7, characterized in that: The classroom observation data includes audio data, video data and classroom teaching information collected in the classroom, the effective teaching behavior event is configured with an effective time range, and the effective teaching behavior event is determined according to the classroom observation data. The event includes: Performing multimodal analysis based on the audio data, video data and classroom teaching information to obtain teaching behavior events and behavior occurrence time points; The teaching behavior event and the time point of occurrence of the behavior are input into a preset state machine to obtain a valid teaching behavior event and a valid time range.

10. The generation method according to any one of claims 1 to 7, characterized in that: Also includes: Acquire auxiliary association information, where the auxiliary association information is used to assist in generating the classroom wonderful scene information; Generate packaging information from the auxiliary association information; The block chain operation is performed on the packaged information to obtain a block.

11. A system for generating classroom wonderful scene information, characterized in that: The generation system includes a data acquisition device and a terminal device, the terminal device includes a memory and a processor, the processor is used to execute one or more computer programs stored in the memory, so that the terminal device controls the data acquisition device to implement the method according to any one of claims 1-10.

12. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the computer device implements the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 10.

Images