Live broadcast teaching AI analysis method and device based on multiple modes, equipment and medium

Abstract

The invention relates to a multimodal-based live broadcast teaching AI analysis method, device and equipment and a medium, and is applied to the technical field of data processing, and the method comprises the steps: obtaining historical physiological parameter data collected by a detection device, historical image parameter data collected by an environment camera device, and historical teaching audio data of teaching personnel; analyzing and processing the historical physiological parameter data to generate a physiological state label; analyzing and processing the historical teaching audio data to generate prompt keywords; constructing an attention model based on the historical image parameter data; constructing an ability evaluation model based on the historical physiological parameter data, the historical image data, the physiological state tag and the prompt keyword; acquiring current physiological parameters and current image parameters of the target student; and generating a live teaching scheme of the target student based on the attention model, the ability evaluation model, the current physiological parameters and the current image parameters. The method and the device have the effect of efficiently improving the training quality of the anchor.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, in particular to a live teaching AI analysis method, device and equipment based on multi-modal and medium. BACKGROUND

[0002] In recent years, the deep integration of e-commerce and live broadcast economy has given birth to the emerging industry of e-commerce live broadcast. As the core role directly affecting sales conversion and user experience, the professional ability of e-commerce anchors is crucial. However, the cultivation of anchors currently mainly relies on traditional "master- apprentice" type of on-the-job learning or short-term concentrated training. It is difficult to impart the core skills of excellent anchors such as verbal ability, camera expressiveness, interactive rhythm, emotional mobilization and product display professionalism to new anchors in a short period of time.

[0003] Currently, some technical means have attempted to monitor and analyze personnel performance, but the existing methods have significant limitations when applied to anchor training scenarios and rely on a large amount of manual data. In addition, existing technical solutions mostly stop at single-dimensional data collection or simple quantification, and fail to establish a deep connection with the professional skills of e-commerce anchors. For example, the system may monitor the "heart rate rise" or "appearance of a specific gesture" of the anchor, but cannot further judge whether it corresponds to "whether the product explanation is in place", "whether the audience interaction is effective" or "whether the emotional expression is appropriate" and other professional dimensions, thus failing to provide improvement suggestions with teaching guidance significance.

[0004] Therefore, there is an urgent need for a new technical solution that can automatically analyze the professional skill performance of anchors without affecting their normal performance, ultimately generating objective, accurate and operable improvement suggestions, thereby efficiently improving the training quality of anchors. SUMMARY

[0005] In order to efficiently improve the training quality of anchors, the present application provides a live teaching AI analysis method, device and equipment based on multi-modal.

[0006] In the first aspect, the present application provides a live teaching AI analysis method based on multi-modal, which adopts the following technical solution:

[0007] A live teaching AI analysis method based on multi-modal, comprising:

[0008] acquiring historical physiological parameter data collected by a detection device, historical image parameter data collected by an environmental camera device and historical teaching audio data of a teaching personnel;

[0009] analyzing and processing the historical physiological parameter data to generate a physiological state label;

[0010] The historical teaching audio data is analyzed and processed to generate a prompt keyword;

[0011] A focus model is constructed based on the historical image parameter data;

[0012] An ability evaluation model is constructed based on the historical physiological parameter data, the historical image data, the physiological state label and the prompt keyword;

[0013] Current physiological parameter and current image parameter of a target student are acquired;

[0014] A live teaching scheme of the target student is generated based on the focus model, the ability evaluation model, the current physiological parameter and the current image parameter.

[0015] By using the above technical solution, the physiological data of the student is collected by the detection device, the image data of the student is collected by the environmental camera device, the historical physiological parameter data and the historical image parameter data generated historically are obtained respectively, the audio data generated by the on-site teaching of the teaching personnel is collected to obtain the historical teaching audio data, the physiological state label used to express the emotion and physiological state of the student is generated using the historical physiological parameter data, the prompt keyword used to prompt the student to adjust is generated using the historical teaching audio data, and the focus model and the ability evaluation model are created, the live teaching scheme is automatically formulated using the created focus model and ability evaluation model and the current physiological parameter and current image parameter of the target student, and the training guidance is performed using the live teaching scheme for the target student, so that the training quality of the host is efficiently improved.

[0016] Optionally, the analyzing and processing the physiological parameter data to generate the physiological state label comprises:

[0017] A target label category and a parameter type corresponding to the label category are acquired;

[0018] The physiological parameter data is analyzed and extracted based on the parameter type to generate to-be-generated data;

[0019] The to-be-generated data is judged based on a preset judgment rule and the target label category to generate a judgment result;

[0020] The physiological state label is generated based on the target label category and the judgment result.

[0021] Optionally, the analyzing and processing the historical teaching audio data to generate the prompt keyword comprises:

[0022] The historical teaching audio data is processed by speech-to-text to generate teaching text data;

[0023] Obtain live-streaming teaching standards;

[0024] Based on the live teaching standards, keywords are extracted from the teaching text data to generate live teaching keywords;

[0025] Based on the live teaching keywords, audio is extracted from the teaching audio data to generate the extraction result;

[0026] The extracted results are bound to the live teaching keywords to generate prompt keywords.

[0027] Optionally, constructing the attention model based on the historical image parameter data includes:

[0028] Obtain the installation location of the environmental camera device corresponding to the historical image parameter data;

[0029] The shooting angle of the historical image parameter data is determined based on the installation location;

[0030] Based on the shooting angle and the historical image parameter data, a preset encoder is trained with change attention to generate a visual encoder.

[0031] Based on the preset historical image parameter data, the preset segmentation model is trained with target contour annotation to generate the main character contour segmentation model.

[0032] An attention model is generated based on the visual encoder and the protagonist contour segmentation model.

[0033] Optionally, the ability assessment model constructed based on the historical physiological parameter data, the historical image data, the physiological state labels, and the prompting keywords includes:

[0034] The historical image data is separated and extracted to generate historical video data and historical audio data;

[0035] A behavioral ability assessment model based on visual performance is constructed based on the historical video data, the historical physiological parameters, the physiological state labels, and the prompt keywords.

[0036] An expressive ability assessment model based on audio performance is constructed based on the historical audio data, the historical physiological parameters, the physiological state labels, and the prompt keywords.

[0037] Optionally, generating the live-streaming teaching plan for the target learner based on the attention model, the ability assessment model, the current physiological parameters, and the current image parameters includes:

[0038] Obtain the calculation percentage;

[0039] annotating the target student in the current image parameter based on the attention model, to generate student video data;

[0040] evaluating the student video data based on the ability evaluation model and the calculation proportion, to generate an ability evaluation result;

[0041] generating a live teaching scheme based on the ability evaluation result.

[0042] Optionally, after the live teaching scheme of the target student is generated based on the attention model, the ability evaluation model, the current physiological parameter and the current image parameter, the method further includes:

[0043] obtaining a receiving device of the target student;

[0044] sending the teaching scheme to the receiving device.

[0045] In a second aspect, the application provides a live teaching AI analysis device based on multi-modal, which adopts the following technical scheme:

[0046] A live teaching AI analysis device based on multi-modal, comprising:

[0047] a related data obtaining module, configured to obtain historical physiological parameter data collected by a detection device, historical image parameter data collected by an environmental camera device and historical teaching audio data of a teaching staff;

[0048] a physiological label generating module, configured to analyze and process the historical physiological parameter data, to generate a physiological state label;

[0049] a prompt vocabulary generating module, configured to analyze and process the historical teaching audio data, to generate a prompt keyword;

[0050] an attention model constructing module, configured to construct an attention model based on the historical image parameter data;

[0051] an evaluation model constructing module, configured to construct an ability evaluation model based on the historical physiological parameter data, the historical image data, the physiological state label and the prompt keyword;

[0052] a student parameter obtaining module, configured to obtain current physiological parameter and current image parameter of a target student;

[0053] a teaching scheme generating module, configured to generate a live teaching scheme of the target student based on the attention model, the ability evaluation model, the current physiological parameter and the current image parameter.

[0054] By adopting the technical scheme, the physiological data of the trainee is collected by the detection device, the image data of the trainee is collected by the environment camera device, the historical physiological parameter data and the historical image parameter data generated historically are obtained respectively, the audio data generated by the on-site teaching of the teaching personnel is collected, the historical teaching audio data is obtained, the physiological state label for expressing the emotion and the physiological state of the trainee is generated by using the historical physiological parameter data, the prompt keyword for prompting the trainee to adjust is generated by using the historical teaching audio data, the creation of the attention model and the ability evaluation model is performed, the teaching scheme is automatically formulated by using the created attention model and the ability evaluation model and the current physiological parameter and the current image parameter of the target trainee, the live teaching scheme for the target trainee is used for training guidance, and the training quality of the host is efficiently improved.

[0055] In a third aspect, the present application provides an electronic device, which adopts the following technical scheme:

[0056] An electronic device, comprising a processor coupled with a memory;

[0057] The processor is configured to execute a computer program stored in the memory, so that the electronic device executes the computer program of the multi-modal based live teaching AI analysis method according to any one of the first aspect.

[0058] In a fourth aspect, the present application provides a computer readable storage medium, which adopts the following technical scheme:

[0059] A computer readable storage medium, which stores a computer program capable of being loaded and executed by a processor to execute the multi-modal based live teaching AI analysis method according to any one of the first aspect.

[0060] In summary, the present application has at least one of the following beneficial technical effects:

[0061] By adopting the technical scheme, the physiological data of the trainee is collected by the detection device, the image data of the trainee is collected by the environment camera device, the historical physiological parameter data and the historical image parameter data generated historically are obtained respectively, the audio data generated by the on-site teaching of the teaching personnel is collected, the historical teaching audio data is obtained, the physiological state label for expressing the emotion and the physiological state of the trainee is generated by using the historical physiological parameter data, the prompt keyword for prompting the trainee to adjust is generated by using the historical teaching audio data, the creation of the attention model and the ability evaluation model is performed, the teaching scheme is automatically formulated by using the created attention model and the ability evaluation model and the current physiological parameter and the current image parameter of the target trainee, the live teaching scheme for the target trainee is used for training guidance, and the training quality of the host is efficiently improved. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a flowchart of a multi-modal based live teaching AI analysis method provided by an embodiment of the present application.

[0063] Figure 2 is a structural block diagram of a multi-modal based live teaching AI analysis device provided by an embodiment of the present application.

[0064] Figure 3 is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application will be further described in detail below with reference to the accompanying drawings.

[0066] An embodiment of the present application provides a multi-modal based live teaching AI analysis method, which can be executed by an electronic device. The electronic device can be a server or a terminal device. The server can be a physical server, a server cluster composed of multiple physical servers or a distributed system, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a desktop computer, etc., but is not limited thereto.

[0067] Figure 1 is a flowchart of a multi-modal based live teaching AI analysis method provided by an embodiment of the present application.

[0068] As shown in Figure 1 , the main process of the method is described as follows (steps S101-S107):

[0069] In step S101, historical physiological parameter data collected by a detection device, historical image parameter data collected by an environmental camera device and historical teaching audio data of a teaching staff are acquired.

[0070] In the embodiment, the detection device is set as an ear return or a headset. In addition to the function of the detection device itself, the detection device can hear the supervision and guidance of the teaching staff. The detection device can also monitor the heart rate and blood oxygen of the human body by using a PPG. A 3D acceleration sensor capable of sensing the movement of the human head is arranged in the detection device. The detection device also includes a 3D acceleration sensor arranged on the torso for monitoring movement. The environmental camera device is a multi-angle camera, which is used to shoot and record the performance video of the anchor. The environmental camera device can also be used to switch different angles during live broadcast. Generally, the camera is arranged at three positions of left, right and center. The historical teaching audio data is data generated by the teaching staff during the previous live teaching. When the teaching staff gives on-site guidance, the guidance content is recorded by audio recording or video recording, so as to obtain the historical teaching audio data.

[0071] Step S102, the historical physiological parameter data is analyzed and processed to generate a physiological state label.

[0072] For step S102, the target label category and the parameter type corresponding to the label category are obtained; the physiological parameter data is analyzed based on the parameter type to generate to-be-generated data; the to-be-generated data is determined based on the preset determination rule and the target label category to generate a determination result; and the physiological state label is generated based on the target label category and the determination result.

[0073] In the embodiment, the target label category mainly includes emotional state and mental saturation degree, and other label categories can be added according to actual needs. According to the heart rate obtained by PPG, HRV (heart rate variability) can be obtained, and the emotional state of the student, such as excitement, tension, depression, confidence, fatigue, etc., can be analyzed by combining blood oxygen and 3D acceleration sensor, and the mental saturation degree of the student, such as head / torso absolute / relative activity amount, reflecting the amplitude of the body, can also be analyzed. Among them, excitement: fast heart rate and HRV is large; tension: fast heart rate and HRV is large and small; depression: slow heart rate and HRV is small; confidence: medium heart rate and HRV is large; fatigue: slow heart rate and HRV is small, and blood oxygen is low. Different thresholds are set for heart rate and HRV to distinguish fast / medium / slow and large / medium / small, and the emotional state can be multiple, such as fatigue and depression may exist at the same time.

[0074] Among them, the mental saturation degree Sp is a numerical value, which can also be quantified as a grade, and is calculated according to multiple motion amplitudes. For example, the present method adopts an acceleration sensor to obtain a comprehensive acceleration, specifically: A=sqrt(ax2+ay2+az2), wherein ax, ay, and az are acceleration values of three axes, and sqrt is a square root. Among them, Ah is the head comprehensive acceleration, and At is the torso comprehensive acceleration. The physiological state label can be obtained by comparison and calculation based on the above information.

[0075] Step S103, the historical teaching audio data is analyzed and processed to generate a prompt keyword.

[0076] For step S103, the historical teaching audio data is processed by speech-to-text to generate teaching text data; live teaching standards are obtained; the teaching text data is processed by keyword extraction based on the live teaching standards to generate live teaching keywords; the audio is intercepted in the teaching audio data based on the live teaching keywords to generate an interception result; and the interception result is bound with the live teaching keywords to generate a prompt keyword.

[0077] In the embodiment, the historical teaching audio data is processed by speech-to-text technology to obtain teaching text data. The set live teaching standard is used to extract keywords from the teaching text data, and a large number of keywords related to the live state can be obtained, such as: small action amplitude, loud voice, passion, slow speech speed, and relaxed head. After obtaining the live teaching keywords, there are cases where the teaching staff's voice needs to be used directly, so the teaching audio data is intercepted to obtain audio data consistent with the live teaching keywords, that is, the interception result, and the interception result is bound to the corresponding live teaching keywords to obtain prompt keywords.

[0078] In step S104, the attention model is constructed based on the historical image parameter data.

[0079] For step S104, the installation position of the environment camera corresponding to the historical image parameter data is obtained; the shooting angle of the historical image parameter data is determined based on the installation position; the preset encoder is trained for attention change based on the shooting angle and the historical image parameter data to generate a visual encoder; the preset segmentation model is trained for target contour labeling based on the preset historical image parameter data to generate a main character contour segmentation model; and the attention model is generated based on the visual encoder and the main character contour segmentation model.

[0080] In the embodiment, the general environment camera is set to three, and the installation positions are left, middle and right three sides, so that the shooting angle of the corresponding image in the historical image parameter data is determined according to the installation position, and the visual encoder is generated according to the obtained shooting angle and the historical image parameter data, that is, the mutual learning between each camera, from simple to complex, such as predicting the middle camera from the left and right sides at the beginning, training the preset encoder VIT encode backbone, so as to pay attention to the change of different angles from the features, that is, to construct a recognition encoder, which is where c is the color channel, w is the picture width, and h is the picture height. Then, the main character contour segmentation model is constructed, that is, each camera learns how to pay attention to the most important “main character”, which is the target student in the room for live broadcast, and the existing AI such as SAM2, that is, Segment Anything Model 2, is used for distillation learning to obtain the main character contour segmentation model, which is Wherein, Acsy, Acsx respectively represent the areas of the two contour maps, represents the intersection, represents the union, Ccsy, Ccsx respectively represent the barycenters of the two contour maps, Ed is the Euclidean distance, k is the weight, generally 0.1-0.3, the visual compiler is bound and combined with the main character contour segmentation model to obtain an attention model, through the attention model, the target student to be analyzed can be quickly locked and the target student can be accurately divided in the environment, so as to analyze and confirm the expression and action.

[0081] In step S105, an ability evaluation model is constructed based on historical physiological parameter data, historical image data, physiological state labels and prompt keywords.

[0082] For step S105, the historical image data is separated and extracted to generate historical video data and historical audio data; a behavior ability evaluation model based on visual performance is constructed based on historical video data, historical physiological parameters, physiological state labels and prompt keywords; an expression ability evaluation model based on audio performance is constructed based on historical audio data, historical physiological parameters, physiological state labels and prompt keywords.

[0083] In this embodiment, two different direction ability evaluation models are created respectively, the historical image data is separated into historical audio data and historical video data, the historical video data, the historical physiological parameters, the physiological state labels and the prompt keywords are used to learn the grasp of performance ability, the physiological state labels and the prompt keywords are used as supervision signals to train a model composed of CNN+MaxPool and ViT, the model maps video features to specific performance evaluation to obtain a behavior ability evaluation model, the purpose of setting CNN+maxpool in the model is to reduce the large parameter vector generated by VIT, specifically, Wherein, crossE is the cross-entropy loss function, and m is the adjustable weight, generally 0.5-1.5. The expression ability evaluation model is constructed in a similar way to the behavior ability evaluation model, and a audio processing model is trained using the physiological state labels and the prompt keywords as supervision signals, the model analyzes the text after speech recognition (compares the script to check errors), speech speed uniformity, reasonable pause and stop, pronunciation problems (such as swallowing words, virtual pronunciation), etc., specifically, Wherein, a, β are adjustable weights, generally 0.1-0.2. When used, the results output by the two ability models are all guiding labels, i.e. prompt keywords, such as relax, speak in a lighter tone, pay attention to expression management, etc.

[0084] In step S106, the current physiological parameters and current image parameters of the target student are obtained.

[0085] In this embodiment, after the creation of the complete model is completed, the model is used to automatically analyze the live situation of the current target student, and the same detection device is used to collect and the environmental camera is used to collect parameters to obtain the current physiological parameters and the current image parameters. The image parameters are image data composed of audio and video.

[0086] In step S107, the live teaching scheme of the target student is generated based on the attention model, the ability evaluation model, the current physiological parameters and the current image parameters.

[0087] For step S107, the calculation proportion is obtained; the target student in the current image parameters is highlighted based on the attention model to generate student video data; the student video data is evaluated based on the ability evaluation model and the calculation proportion to generate the ability evaluation result; and the live teaching scheme is generated based on the ability evaluation result.

[0088] In this embodiment, when the live teaching scheme is generated, different results can be set according to actual needs to set different proportions, i.e., the calculation proportion. First, the attention model is used to highlight the current image parameters to identify and label the target student to obtain student video data. Then, the two obtained ability evaluation models are used to evaluate the student video data to generate prompt keywords for guiding the target student. Then, the calculation proportion is used to proportionally screen the obtained prompt keywords to generate the ability evaluation result, which is used as the live teaching scheme.

[0089] In this embodiment, the receiving device of the target student is obtained; and the teaching scheme is sent to the receiving device.

[0090] After the live teaching scheme is automatically generated, the target student needs to be guided for teaching by using the live teaching scheme. The receiving device of the target student needs to be determined, and the corresponding live teaching scheme is sent to the receiving device of the corresponding target student. The teaching guidance can be performed in the form of voice broadcast.

[0091] Figure 2 A structure block diagram of a live teaching AI analysis device 200 based on multiple modalities provided for the application embodiment.

[0092] As shown in Figure 2 The live teaching AI analysis device 200 based on multiple modalities mainly includes:

[0093] The related data acquisition module 201 is used to acquire the historical physiological parameter data collected by the detection device, the historical image parameter data collected by the environmental camera and the historical teaching audio data of the teaching personnel.

[0094] The physiological label generation module 202 is used to analyze and process the historical physiological parameter data to generate a physiological state label.

[0095] The prompt word generation module 203 is configured to analyze and process the historical teaching audio data, and generate prompt keywords;

[0096] The attention model construction module 204 is configured to construct an attention model based on historical image parameter data;

[0097] The evaluation model construction module 205 is configured to construct an ability evaluation model based on historical physiological parameter data, historical image data, physiological state labels and prompt keywords;

[0098] The student parameter acquisition module 206 is configured to acquire current physiological parameters and current image parameters of a target student;

[0099] The teaching scheme generation module 207 is configured to generate a live teaching scheme for the target student based on the attention model, the ability evaluation model, the current physiological parameters and the current image parameters.

[0100] As an optional implementation of the embodiment, the physiological label generation module 202 is specifically configured to acquire a target label category and a parameter type corresponding to the label category; perform data extraction analysis on the physiological parameter data based on the parameter type, to generate to-be-generated data; perform data determination on the to-be-generated data based on a preset determination rule and the target label category, to generate a determination result; and generate a physiological state label based on the target label category and the determination result.

[0101] As an optional implementation of the embodiment, the prompt word generation module 203 is specifically configured to perform speech-to-text processing on the historical teaching audio data, to generate teaching text data; acquire a live teaching standard; perform keyword extraction on the teaching text data based on the live teaching standard, to generate live teaching keywords; perform audio interception in the teaching audio data based on the live teaching keywords, to generate an interception result; and bind the interception result with the live teaching keywords, to generate prompt keywords.

[0102] As an optional implementation of the embodiment, the attention model construction module 204 is specifically configured to acquire an installation position of an environmental camera device corresponding to the historical image parameter data; determine a shooting angle of the historical image parameter data based on the installation position; perform change attention training on a preset encoder based on the shooting angle and the historical image parameter data, to generate a visual encoder; perform target contour annotation training on a preset segmentation model based on preset historical image parameter data, to generate a main character contour segmentation model; and generate the attention model based on the visual encoder and the main character contour segmentation model.

[0103] As an optional implementation of the embodiment, the evaluation model construction module 205 is specifically configured to perform separate extraction processing on the historical image data to generate historical video data and historical audio data; construct a behavior ability evaluation model based on visual performance based on the historical video data, the historical physiological parameters, the physiological state label and the prompt keyword; and construct an expression ability evaluation model based on audio performance based on the historical audio data, the historical physiological parameters, the physiological state label and the prompt keyword.

[0104] As an optional implementation of the embodiment, the teaching scheme generation module 207 is specifically configured to acquire a calculation proportion; perform key labeling on the target student in the current image parameter based on the attention model to generate student video data; perform evaluation on the student video data based on the ability evaluation model and the calculation proportion to generate an ability evaluation result; and generate a live teaching scheme based on the ability evaluation result.

[0105] As an optional implementation of the embodiment, the live teaching AI analysis device 200 based on multi-modalities further comprises:

[0106] The receiving device acquisition module is configured to acquire a receiving device of the target student.

[0107] The teaching scheme sending module is configured to send the teaching scheme to the receiving device.

[0108] In one example, the modules in any of the apparatuses above can be one or more integrated circuits configured to implement the methods above, for example, one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0109] For another example, when the modules in the apparatuses can be implemented in the form of a processing element scheduler, the processing element can be a general purpose processor, for example, a central processing unit (CPU) or other processor that can invoke programs. For another example, these modules can be integrated together to be implemented in the form of a system-on-a-chip (SOC).

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the apparatuses and modules described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0111] Figure 3 A structural block diagram of an electronic device 300 is provided for embodiments of the present application.

[0112] As shown in Figure 3 , the electronic device 300 includes a processor 301 and a memory 302, and can further include one or more of an information input / output (I / O) interface 303, a communication component 304, and a communication bus 305.

[0113] The processor 301 is configured to control overall operations of the electronic device 300 to complete all or part of the steps of the multi-modal based live teaching AI analysis method described above. The memory 302 is configured to store various types of data to support operations of the electronic device 300, which can include, for example, instructions for any application or method operating on the electronic device 300, and application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0114] The I / O interface 303 provides an interface between the processor 301 and other interface modules, which can be a keyboard, a mouse, a button, and the like. These buttons can be virtual buttons or physical buttons. The communication component 304 is configured to perform wired or wireless communication between the electronic device 300 and other devices. The wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them, so the corresponding communication component 304 can include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0115] The electronic device 300 can be implemented with one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for performing the multi-modal based live teaching AI analysis method according to the embodiments described above.

[0116] The communication bus 305 can include a path for transmitting information between the above-described components. The communication bus 305 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus 305 can be divided into an address bus, a data bus, a control bus, and the like.

[0117] The electronic device 300 can include, but is not limited to, a mobile terminal of a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet PC), a PMP (Portable Multimedia Player), a car terminal (for example, a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like, and can also be a server or the like.

[0118] The present application also provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the multi-modal based live teaching AI analysis method described above.

[0119] The computer readable storage medium can include a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, various media capable of storing program codes.

[0120] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0121] The description above only concerns preferred embodiments of the application and the explanation of the principles of the technology used. The person skilled in the art should understand that the scope of the application involved in the application is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by any combinations of the technical features described above or their equivalent features without departing from the above-mentioned application concept. For example, the technical solutions formed by the mutual replacement of the above-mentioned features and the technical features applied in the application (but not limited to) having similar functions.

Claims

1. A multimodal AI analysis method for live-streaming teaching, characterized in that, include: Acquire historical physiological parameter data collected by the detection device, historical image parameter data collected by the environmental camera device, and historical teaching audio data from the teaching staff; The historical physiological parameter data is analyzed and processed to generate physiological state labels; The historical teaching audio data is analyzed and processed to generate prompt keywords; An attention model is constructed based on the historical image parameter data; A capability assessment model is constructed based on the historical physiological parameter data, the historical image data, the physiological state labels, and the prompt keywords; Obtain the target student's current physiological parameters and current image parameters; The live teaching plan for the target student is generated based on the attention model, the ability assessment model, the current physiological parameters, and the current image parameters.

2. The method according to claim 1, characterized in that, The step of analyzing and processing the physiological parameter data to generate physiological state labels includes: Obtain the target tag type and the parameter type corresponding to the tag type; Based on the parameter type, the physiological parameter data is extracted and analyzed to generate data to be generated; Based on preset judgment rules and the target label type, the data to be generated is judged, and a judgment result is generated. Physiological state labels are generated based on the target label type and the determination result.

3. The method according to claim 1, characterized in that, The process of analyzing and processing the historical teaching audio data to generate prompt keywords includes: The historical teaching audio data is processed to convert speech to text, generating teaching text data; Obtain live-streaming teaching standards; Based on the live teaching standards, keywords are extracted from the teaching text data to generate live teaching keywords; Based on the live teaching keywords, audio is extracted from the teaching audio data to generate the extraction results; The extracted results are bound to the live teaching keywords to generate prompt keywords.

4. The method according to claim 1, characterized in that, The construction of the attention model based on the historical image parameter data includes: Obtain the installation location of the environmental camera device corresponding to the historical image parameter data; The shooting angle of the historical image parameter data is determined based on the installation location; Based on the shooting angle and the historical image parameter data, a preset encoder is trained with change attention to generate a visual encoder. Based on the preset historical image parameter data, the preset segmentation model is trained with target contour annotation to generate the main character contour segmentation model. An attention model is generated based on the visual encoder and the protagonist contour segmentation model.

5. The method according to claim 1, characterized in that, The capability assessment model based on the historical physiological parameter data, the historical image data, the physiological state labels, and the prompting keywords includes: The historical image data is separated and extracted to generate historical video data and historical audio data; A behavioral ability assessment model based on visual performance is constructed based on the historical video data, the historical physiological parameters, the physiological state labels, and the prompt keywords. An expressive ability assessment model based on audio performance is constructed based on the historical audio data, the historical physiological parameters, the physiological state labels, and the prompt keywords.

6. The method according to claim 1, characterized in that, The process of generating a live-streaming teaching plan for the target learner based on the attention model, the ability assessment model, the current physiological parameters, and the current image parameters includes: Obtain the calculation percentage; Based on the attention model, the target student in the current image parameters is highlighted to generate student video data; The student's video data is evaluated based on the aforementioned competency assessment model and the calculated percentage, and competency assessment results are generated. A live-streaming teaching plan is generated based on the results of the competency assessment.

7. The method according to claim 1, characterized in that, After generating the live-streaming teaching plan for the target learner based on the attention model, the ability assessment model, the current physiological parameters, and the current image parameters, the method further includes: Obtain the receiving device of the target student; The teaching plan is sent to the receiving device.

8. A multimodal live-streaming teaching AI analysis device, characterized in that, include: The relevant data acquisition module is used to acquire historical physiological parameter data collected by the detection device, historical image parameter data collected by the environmental camera device, and historical teaching audio data of the teaching staff; The physiological label generation module is used to analyze and process the historical physiological parameter data to generate physiological state labels; The prompt vocabulary generation module is used to analyze and process the historical teaching audio data to generate prompt keywords; The attention model construction module is used to construct an attention model based on the historical image parameter data. The evaluation model construction module is used to construct a capability evaluation model based on the historical physiological parameter data, the historical image data, the physiological state labels, and the prompt keywords; The student parameter acquisition module is used to acquire the current physiological parameters and current image parameters of the target student. The teaching plan generation module is used to generate a live teaching plan for the target student based on the attention model, the ability assessment model, the current physiological parameters, and the current image parameters.

9. An electronic device, characterized in that, Includes a processor, which is coupled to a memory; The processor is configured to execute a computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It includes a computer program or instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images