A teaching interaction method, device, equipment and storage medium

Abstract

The application relates to the technical field of intelligent teaching, in particular to a teaching interaction method and device, equipment and a storage medium, the method comprising the following steps: collecting first image information of a teacher, second image information of students and heart rate data of the students in real time; preprocessing first image sequences, second image sequences and heart rate sequences; inputting a pre-trained gesture recognition neural network to perform recognition, and outputting a gesture type of the teacher and a gesture type of the students; obtaining a heart rate change rate corresponding to the gesture type of the students at each moment; and issuing first prompt information and second prompt information in response to the gesture type of the teacher, the gesture type of the students and the corresponding heart rate change rate. According to the method, the gesture recognition result of the students is associated with the heart rate data for correlation analysis, the learning state of the students can be more accurately evaluated, the teacher can better understand the emotional state of the students, teaching strategies can be timely adjusted, and personalized guidance and support can be provided.

Description

Technical Field

[0001] This application relates to the field of intelligent teaching technology, and in particular to a teaching interaction method, device, equipment and storage medium. Background Technology

[0002] In traditional teaching methods, interaction between teachers and students relies primarily on language and written expression. However, this approach suffers from low communication efficiency and limited comprehension. Even though one-on-one tutoring can provide personalized and customized teaching services to meet the diverse needs of students, teachers often struggle to understand students' actual learning status and emotional feedback during the process. Teachers may overlook students' learning needs or fail to promptly detect their learning progress, making it difficult to guarantee teaching effectiveness and provide the most appropriate guidance and support in certain areas. Summary of the Invention

[0003] This application provides a teaching interaction method, apparatus, device, and storage medium to address the shortcomings of related technologies where teacher-student interaction relies primarily on language and written expression, resulting in low communication efficiency and limited comprehension. The technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a teaching interaction method, including:

[0005] Real-time acquisition of the teacher's first image information, the student's second image information, and the student's heart rate data;

[0006] The teacher's first image sequence is obtained by combining the time sequence of the first image information, the student's second image sequence is obtained by combining the time sequence of the second image information, and the student's heart rate sequence is obtained by combining the time sequence of the heart rate data; the first image sequence, the second image sequence, and the heart rate sequence are preprocessed.

[0007] The preprocessed first and second gesture sequences are input into a pre-trained gesture recognition neural network for recognition, and the teacher's gesture type at each moment of the first gesture sequence and the student's gesture type at each moment of the second gesture sequence are output; the heart rate change rate corresponding to the student's gesture type at each moment is obtained.

[0008] Based on a pre-set teaching instruction library, a first prompt message is issued in response to the teacher's gesture type at each time, and a second prompt message is issued in response to the student's gesture type and the corresponding heart rate change rate at each time.

[0009] In one alternative embodiment of the first aspect, preprocessing of the first image sequence and the second image sequence includes:

[0010] Calculate the difference value of each pixel in every two adjacent image frames, filter out pixels with difference values ​​less than the difference value threshold to generate a difference image, and superimpose the difference image with the original image to obtain the first processed image;

[0011] Based on the difference between the first processed image and the current image frame, foreground objects are detected, foreground objects are filtered out, and hand recognition is performed to extract gesture image information;

[0012] Feature extraction is performed on the gesture image information to obtain finger position features, palm shape features, and joint angle features;

[0013] The extracted finger position features, palm shape features, and joint angle features are normalized to obtain a second processed image;

[0014] After noise filtering and data dimensionality reduction, the third processed image is obtained.

[0015] One alternative to the first aspect includes:

[0016] After preprocessing the first image sequence and the second image sequence, the process includes:

[0017] Annotate the gesture information in the third processed image;

[0018] Label the gesture type of each gesture and generate a mapping relationship with the third processed image;

[0019] The third processed image labeled with gesture type is used as the training set for the gesture recognition neural network.

[0020] In one alternative to the first aspect, the preprocessing of the heart rate sequence includes:

[0021] The heart rate sequence was examined to remove abnormal data;

[0022] The data was standardized to obtain heart rate data with a standard normal distribution, meaning 0 and standard deviation 1.

[0023] In one alternative to the first aspect, training the gesture recognition neural network includes:

[0024] Gesture features are extracted from the input gesture image; model input for each feature is generated using multiple convolutional layers, and the resulting image features are processed to the same size to generate planar coordinates for each feature, thus obtaining the positional encoding.

[0025] Enhanced features are obtained based on the positional encoding process, and then decoder embedding is obtained;

[0026] The offset and weight are calculated, and regression is performed on each weight to obtain the 3D coordinates of the predicted gesture.

[0027] Normalized weights are calculated based on the 3D coordinates of the gesture;

[0028] The loss function is used to calculate the coordinates of the real gesture and the predicted 3D coordinates of the gesture.

[0029] The parameters of the gesture recognition neural network are adjusted based on the loss function.

[0030] In one alternative to the first aspect, obtaining the student's heart rate variability includes:

[0031] Obtain students' historical heart rate data and generate corresponding student heart rate baseline data;

[0032] Based on the aforementioned heart rate baseline data, the student's heart rate change rate is output.

[0033] In one alternative to the first aspect, issuing a second prompt message in response to the student's gesture type and the corresponding heart rate variability at each time point includes:

[0034] The heart rate change rate at the corresponding moment of the gesture is obtained. Based on the corresponding heart rate change rate, the emotional change data of the student when making the gesture is identified. Based on the emotional change data and the type of the student's gesture, the student's learning needs and corresponding learning status are determined. Based on the learning needs and the learning status, a second prompt message is sent to the teacher.

[0035] Secondly, embodiments of this application also provide a teaching interactive device, including:

[0036] The acquisition unit is used to collect the teacher's first image information, the student's second image information, and the student's heart rate data in real time.

[0037] The preprocessing unit is used to obtain a first image sequence of the teacher by combining the time sequence of the first image information, a second image sequence of the student by combining the time sequence of the second image information, and a heart rate sequence of the student by combining the time sequence of the heart rate data; and to preprocess the first image sequence, the second image sequence, and the heart rate sequence.

[0038] The first processing unit is configured to input the preprocessed first gesture sequence and second gesture sequence into a pre-trained gesture recognition neural network for recognition, and output the gesture type of the teacher at each moment of the first gesture sequence and the gesture type of the student at each moment of the second gesture sequence; it is also configured to obtain the heart rate change rate corresponding to the gesture type of the student at each moment.

[0039] The second processing unit is used to issue a first prompt message in response to the teacher's gesture type at each time, based on a preset teaching instruction library, and to issue a second prompt message in response to the student's gesture type and the corresponding heart rate change rate at each time.

[0040] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the method as described in any of the first aspects of this application.

[0041] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method described in any one of the first aspects of this application.

[0042] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following:

[0043] (1) By using a gesture recognition model, the gestures of teachers and students can be accurately recognized, which can enable teachers and students to have more intuitive and natural teaching interactions. This will greatly enhance the sense of participation and interactivity in teaching and promote students' active participation and in-depth understanding.

[0044] (3) Correlate the student's gesture recognition results with the student's heart rate data. During the learning process, the captured student's gestures may be actions that the student makes unintentionally and are unrelated to the course content. However, the heart rate signal can detect the student's psychological changes. Therefore, combining gestures and heart rate data can more accurately assess the student's learning status. Teachers can better understand the student's emotional state, adjust teaching strategies in a timely manner, and provide personalized guidance and support, thereby more effectively promoting the student's learning outcomes. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating a teaching interaction method according to an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of the structure of the gesture recognition neural network provided in the embodiments of this application;

[0048] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0051] It should be noted that the terms "first" and "second" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than those described or illustrated herein.

[0052] Please refer to the following. Figure 1 This application provides an exemplary embodiment of a teaching interaction method, characterized in that it includes:

[0053] The S101 collects the teacher's first image information, the student's second image information, and the student's heart rate data in real time.

[0054] Specifically, image data of teachers and students is collected using a depth camera, and heart rate data is obtained by collecting heart rate data from watches worn by students that have heart rate sensors.

[0055] Specifically, the student heart rate monitoring includes: measuring the R-peak of the pre-processed heart rate signal using methods such as thresholding, waveform morphology, and correlation; then calculating the heart rate (number of heartbeats per minute) based on the position of the R-peaks. The RR interval (the time interval between two adjacent R-peaks) is typically used to calculate the heart rate, which is obtained by converting the RR interval to heart rate per minute. Finally, the heart rhythm is analyzed, including measuring and evaluating various time and frequency domain parameters, such as RR interval variability and spectral analysis. These parameters can be correlated with the student's gesture recognition results to assess the student's learning status.

[0056] S102 Combines the time sequence of the first image information to obtain the first image sequence of the teacher, combines the time sequence of the second image information to obtain the second image sequence of the student, and combines the time sequence of the heart rate data to obtain the heart rate sequence of the student; preprocesses the first image sequence, the second image sequence, and the heart rate sequence.

[0057] Specifically, preprocessing is performed on the first image sequence and the second image sequence, including:

[0058] Dynamic gestures are extracted by calculating the difference between adjacent frames using the frame difference method. First, the images I1 and I2 of the two read frames are converted into grayscale images G1 and G2. Then, the two grayscale images are subtracted to obtain the difference image. The grayscale value of the current frame is R, the grayscale value of the previous frame is L, and the difference value is D. The difference of each pixel can be calculated using the following formula:

[0059] D = RL

[0060] Thresholding is performed on the difference image D, ignoring differences smaller than the threshold f and retaining only those larger than the threshold to filter out areas of invariance in the learning scene. Dilation is then performed on the thresholded difference image ID to better detect object motion. Finally, the processed difference image ID is overlaid with the original image I to obtain the first processed image of the visualization result, which serves as the input for subsequent operation steps.

[0061] Furthermore, the video image sequence captured by the depth camera is used as the background, and the difference between the background image and the current frame is used to detect the foreground object, thereby extracting the gesture; methods such as skin color segmentation and morphological operations can be used to detect and track the hand in the image to obtain gesture image information.

[0062] Based on the difference between the first processed image and the current image frame, foreground objects are detected, foreground objects are filtered out, and hand recognition is performed to extract gesture image information.

[0063] Feature extraction is performed on the gesture image information to extract the hand region and obtain the finger position features f. p Hand shape characteristics p s and joint angle features j a .

[0064] The extracted finger position features, palm shape features, and joint angle features are normalized to obtain a second processed image, thereby mapping it to a uniform range to eliminate differences caused by different human body structures and movements. The specific operation is as follows:

[0065] By iterating through all extracted gesture data points, the minimum value (min) and maximum value (max) are found. For each gesture data point, the original value is I0, and the normalized value a is calculated using the following formula:

[0066] a = (I0 - min) / (max - min)

[0067] Optionally, the gesture data can be smoothed using filtering algorithms or other signal processing methods to reduce noise and oscillations, thereby reducing the convergence difficulty of the neural network.

[0068] Optionally, principal component analysis (PCA) can be used to reduce the dimensionality of the feature data to improve processing efficiency. After noise filtering and data dimensionality reduction, a third-processed image is obtained.

[0069] The pre-processed third image is input into the pre-trained gesture recognition neural network.

[0070] Specifically, after preprocessing the heart rate sequence, the process includes:

[0071] Data quality check: Check whether the collected student heart rate data has missing values, outliers, or values ​​that are outside the normal range, and exclude outliers and erroneous data;

[0072] Data smoothing: Heart rate data is smoothed using median filtering to reduce the impact of noise;

[0073] Data interpolation: If missing values ​​exist, linear interpolation can be used to fill them.

[0074] Data standardization: The student heart rate data is standardized and transformed into a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0075] Data downsampling: If the original heart rate data sampling frequency is too high, use average pooling or max pooling to reduce the sampling frequency to save storage space and computing resources;

[0076] Outlier Handling: For heart rate data points with abnormal fluctuations or mutations, outlier detection and processing are performed using statistical or model-based methods to obtain a heart rate sequence suitable for subsequent tasks: HR2 = hr1, hr2, ..., hr n .

[0077] S103 inputs the preprocessed first gesture sequence and second gesture sequence into a pre-trained gesture recognition neural network for recognition, outputs the teacher's gesture type at each moment of the first gesture sequence and the student's gesture type at each moment of the second gesture sequence; and obtains the heart rate change rate corresponding to the student's gesture type at each moment.

[0078] Specifically, such as Figure 2 As shown in this embodiment, the gesture recognition neural network includes feature fusion, multi-layer Transformer, and fully connected layers. Specifically, the multi-layer Transformer includes layer normalization, multi-head attention, and prefeedback; the fully connected layers are used to recognize the final gesture recognition results of teachers and students.

[0079] According to the above scheme, the feature fusion process is as follows: the acquired teacher video images are T2 = (t1, t2, ..., t n The collected student video images are S2=(s1,s2,…,s…). n The method takes n frames of video from the teacher and students as input, extracts key gesture features through convolution and pooling operations, and then fuses these features, represented as C = [c1, c2, ..., c]. n ].

[0080] S104, based on a preset teaching instruction library, issues a first prompt message in response to the teacher's gesture type at each time, and issues a second prompt message in response to the student's gesture type and the corresponding heart rate change rate at each time.

[0081] Specifically, both the first and second prompts are based on the recognition results of teacher and student gestures. Based on these recognition results, corresponding teaching suggestions are provided to the teacher; and based on changes in student heart rate, the learning status is comprehensively assessed, and corresponding teaching feedback is provided to the teacher. For example, if the teacher's gesture is verbal, indicating that the teacher is explaining key content or emphasizing a concept, the teacher can be advised to continue explaining and provide more examples or practical activities to deepen the student's understanding. If the teacher's gesture is directional, indicating that the teacher is asking the student a question or encouraging student interaction, the teacher can be advised to wait a while to give the student a chance to think and answer the question, or to provide prompts and guidance to help the student answer the question. If the student's gesture is hands supporting their chin, indicating that the student has difficulty understanding the knowledge points explained by the teacher or has questions, the teacher can be advised to proactively communicate with the student and provide additional explanations and demonstrations. If a student raises their hand, indicating that the student is interested in a certain content or has their own insights, the teacher can be advised to encourage the student to share and engage in discussion to promote the student's active learning and thinking abilities.

[0082] It's understandable to compare and analyze students' heart rate data with their learning performance. Two aspects can be observed:

[0083] Learning effectiveness: Pay attention to whether lower or higher heart rates are associated with learning effectiveness. For example, anxiety or stress may lead to decreased learning effectiveness, while excitement or focus may improve it.

[0084] Attention level: Observe whether changes in heart rate reflect the student's level of attention. A lower heart rate may indicate that the student lacks interest, while a higher heart rate may indicate that the student is highly focused.

[0085] For example, monitoring heart rate data can reflect changes in students' emotions. The principle is that during learning, students' heart rate may increase when they feel tense or excited, and decrease when they are relaxed or tired. Therefore, monitoring students' heart rate changes can provide a preliminary understanding of their mood and attention levels.

[0086] Specifically, in response to the student's gesture type and corresponding heart rate variability at each time point, a second prompt message is issued, which also includes:

[0087] The heart rate change rate at the corresponding moment of the gesture is obtained. Based on the corresponding heart rate change rate, the emotional change data of the student when making the gesture is identified. Based on the emotional change data and the type of the student's gesture, the student's learning needs and corresponding learning status are determined. Based on the learning needs and the learning status, a second prompt message is sent to the teacher.

[0088] That is, to comprehensively judge the student's learning status and emotions based on the student's heart rate when making gestures;

[0089] For example, if a student's gesture is to rest their chin on their hands and their heart rate is flat, it indicates that the student is relatively focused on learning. At this time, the student may have difficulty understanding or have questions about the knowledge points being explained by the teacher. If the student's heart rate is below the baseline, it indicates that the student may be inattentive, prompting the teacher to help the student refocus. If a student's gesture is to raise their hand and their heart rate is increased or flat, it indicates that the student is highly focused and has a strong learning attitude, prompting the teacher to provide further explanation. If the heart rate is decreased, it indicates that the gesture may have been unintentional. The above are merely examples of embodiments of the present invention and are not intended to limit the embodiments of the present invention. Judgment should be based on the individual student's state.

[0090] Therefore, obtaining the student's heart rate variability also includes:

[0091] Obtain students' historical heart rate data and generate corresponding student heart rate baseline data;

[0092] Based on the aforementioned heart rate baseline data, output the student's heart rate variability rate;

[0093] This allows for an adaptive assessment of each student's learning status based on their heart rate and heart rate variability.

[0094] Furthermore, teachers can provide personalized teaching feedback to students based on the first and second prompts. For example, the following measures can be taken:

[0095] Encouragement and support: If a student is anxious or stressed, provide encouragement and support to help them relax and build confidence.

[0096] Adjust teaching strategies: Adjust teaching strategies based on changes in student attention levels. For example, introduce more interactive and engaging elements in subsequent lessons to increase student interest and focus.

[0097] Personalized tutoring: Providing students with personalized tutoring and guidance based on the correlation between heart rate changes and learning outcomes.

[0098] This application embodiment can perform correlation analysis between students' gesture recognition results and students' heart rate data at different time periods, and feed the results back to the teacher, so that the teacher can make appropriate adjustments to the time and teaching methods of the content taught according to the students' learning status.

[0099] In one embodiment, training the gesture recognition neural network includes:

[0100] Specifically, a Generative Adversarial Network (GAN) can be selected as the backbone network to extract teacher and student features from the input teacher and student gesture images. Specifically, three output layers with downsampling rates of 8, 16, and 32 are used to obtain features. Simultaneously, three convolutional layers are used to generate the input to the Transformer model for each feature, and an additional convolutional layer is used to extract the final feature layer to preserve more spatial information. Then, convolutional layers and group normalization layers are used to process the obtained image features to the same size. After flattening and concatenation, the planar coordinates (x, y) of the generated feature F are added to the positional encoding (PE) P. xy :

[0101] P xy =PE(x,y);

[0102] Replace the self-attention mechanism in Transformer with a multi-scale deformation attention mechanism (MSDAM). For the self-attention mechanism, query Q1, key K1, and value V1 are the same as F, while query Q is ref(·), and the query includes an additional positional term P. xy :

[0103]

[0104] Then, Q1, K1, and V1 are sent to MSDAM to obtain enhanced features.

[0105] For the self-attention mechanism, the settings for query Q2 key K2 and value V2 in the Transformer decoder layer are similar to those in step 2.1.2, and the decoder embedding is D:

[0106]

[0107] To obtain the offset and weight Z h (a) Add two MLP layers to the output of the multi-layer Transformer model. Calculate the offset P through one MLP layer. h (k), then divided into Another MLP layer for each weight Z h (a) Perform regression. Finally, the predicted 3D coordinates of the gesture can be represented as:

[0108]

[0109] in, and The planar coordinates and depth coordinates of the gesture are indicated by C. i (k) and C d(k) indicates the in-plane coordinates and depth coordinates of the key point k. These are the normalized weights from keypoint k toward gesture h, which can be calculated using softmax:

[0110]

[0111] Loss function calculation: After obtaining the estimated 3D gesture coordinates, this invention uses the gesture estimation loss E1 to supervise the final output, and its formula is as follows:

[0112]

[0113] in, These are the real-plane coordinates and depth coordinates of the actual gesture. The parameter μ defaults to 0.5 to balance the loss between in-plane and depth offset estimation tasks. L τ (·) is a class of smooth given by the following formula. L1 Loss function:

[0114]

[0115] Among them, τ1 and τ2 are set to 1 and 5 respectively to better smooth the depth values.

[0116] To improve the model's generalization ability, this invention defines the loss E2 around keypoints as follows:

[0117]

[0118] Finally, when and When set to 5 and 1 respectively, the total loss function E is formulated as follows:

[0119]

[0120] According to the above scheme, the fully connected layer calculates the final recognition result.

[0121] The teacher's gestures are categorized as: pointing, verbal, and interactive; the student's gestures are categorized as: interactive, descriptive, and control.

[0122] Gesture features are extracted from the input gesture image; model input for each feature is generated using multiple convolutional layers, and the resulting image features are processed to the same size to generate planar coordinates for each feature, thus obtaining the positional encoding.

[0123] Enhanced features are obtained based on the positional encoding process, and then decoder embedding is obtained;

[0124] The offset and weight are calculated, and regression is performed on each weight to obtain the 3D coordinates of the predicted gesture.

[0125] Normalized weights are calculated based on the 3D coordinates of the gesture;

[0126] The loss function is used to calculate the coordinates of the real gesture and the predicted 3D coordinates of the gesture.

[0127] The parameters of the gesture recognition neural network are adjusted based on the loss function.

[0128] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0129] The interactive teaching device provided in this application includes:

[0130] The acquisition unit is used to collect the teacher's first image information, the student's second image information, and the student's heart rate data in real time.

[0131] The preprocessing unit is used to obtain a first image sequence of the teacher by combining the time sequence of the first image information, a second image sequence of the student by combining the time sequence of the second image information, and a heart rate sequence of the student by combining the time sequence of the heart rate data; and to preprocess the first image sequence, the second image sequence, and the heart rate sequence.

[0132] The first processing unit is configured to input the preprocessed first gesture sequence and second gesture sequence into a pre-trained gesture recognition neural network for recognition, and output the gesture type of the teacher at each moment of the first gesture sequence and the gesture type of the student at each moment of the second gesture sequence; it is also configured to obtain the heart rate change rate corresponding to the gesture type of the student at each moment.

[0133] The second processing unit is used to issue a first prompt message in response to the teacher's gesture type at each time, based on a preset teaching instruction library, and to issue a second prompt message in response to the student's gesture type and the corresponding heart rate change rate at each time.

[0134] It should be noted that the teaching interaction device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the teaching interaction calculation method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiments and the teaching interaction method embodiments belong to the same concept, and its implementation process can be found in the method embodiments, which will not be repeated here.

[0135] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0136] Please see Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0137] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302.

[0138] In this embodiment, the processor 301 is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor 301 may include one or more processing cores, such as a 4-core processor or an 8-core processor. The processor 301 can be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0139] Processor 301 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor used to process data in the standby state.

[0140] Memory 302 may include one or more computer-readable storage media, which may be non-transitory. Memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the non-transitory computer-readable storage media in memory 302 is used to store at least one instruction, which is executed by processor 301 to implement the method in the embodiments of this application.

[0141] In some embodiments, the electronic device 300 further includes a peripheral device interface 303 and at least one peripheral device. The processor 301, memory 302, and peripheral device interface 303 can be connected via a bus or signal line. Various peripheral devices can be connected to peripheral device interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral devices include: the display screen 304, camera 305, and audio circuitry 306. Peripheral device interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 301 and memory 302.

[0142] In some embodiments of this application, the processor 301, memory 302, and peripheral device interface 303 are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor 301, memory 302, and peripheral device interface 303 can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.

[0143] Display screen 304 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 304 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 301 for processing. In this case, display screen 304 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0144] In some embodiments of this application, there may be one display screen 304, disposed on the front panel of the electronic device 300; in other embodiments, there may be at least two display screens 304, disposed on different surfaces of the electronic device 300 or in a folded design; in still other embodiments, the display screen 304 may be a flexible display screen, disposed on a curved or folded surface of the electronic device 300. Furthermore, the display screen 304 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. The display screen 304 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0145] Camera 305 is used to capture images or videos. Optionally, camera 305 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device, and the rear-facing camera is located on the back of the electronic device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments of this application, camera 305 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0146] The audio circuit 306 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 301 for processing. For stereo acquisition or sound reduction purposes, there may be multiple microphones, which are respectively located in different parts of the electronic device 300. The microphone may also be an array microphone or an omnidirectional acquisition microphone.

[0147] Power supply 307 is used to supply power to various components in electronic device 300. Power supply 307 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 307 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0148] The electronic device structural block diagram shown in the embodiments of this application does not constitute a limitation on the electronic device 300. The electronic device 300 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0149] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A teaching interaction method, characterized in that, include: Real-time acquisition of the teacher's first image information, the student's second image information, and the student's heart rate data; The teacher's first image sequence is obtained by combining the time sequence of the first image information, the student's second image sequence is obtained by combining the time sequence of the second image information, and the student's heart rate sequence is obtained by combining the time sequence of the heart rate data. Preprocessing is performed on the first image sequence, the second image sequence, and the heart rate sequence; The preprocessing of the first image sequence and the second image sequence includes: Calculate the difference value of each pixel in every two adjacent image frames, filter out pixels with difference values ​​less than the difference value threshold to generate a difference image, and superimpose the difference image with the original image to obtain the first processed image; Based on the difference between the first processed image and the current image frame, foreground objects are detected, foreground objects are filtered out, and hand recognition is performed to extract gesture image information; Feature extraction is performed on the gesture image information to obtain finger position features, palm shape features, and joint angle features; The extracted finger position features, palm shape features, and joint angle features are normalized to obtain a second processed image; After noise filtering and data dimensionality reduction, the third-processed image is obtained; The third processed image, obtained by preprocessing the first image sequence and the second image sequence, is input into a pre-trained gesture recognition neural network for recognition, and the gesture type of the teacher and the gesture type of the student at the corresponding time of the third processed image are output; the heart rate change rate corresponding to the gesture type of the student at each time is obtained; Based on a pre-set teaching instruction library, a first prompt message is issued in response to the teacher's gesture type at each time, and a second prompt message is issued in response to the student's gesture type and the corresponding heart rate change rate at each time. The step of obtaining the heart rate change rate includes: Obtain students' historical heart rate data and generate corresponding student heart rate baseline data; Based on the aforementioned heart rate baseline data, output the student's heart rate variability rate; The step of issuing a second prompt message in response to the student's gesture type and corresponding heart rate variability at each time point includes: The heart rate change rate at the corresponding moment of the gesture is obtained. Based on the corresponding heart rate change rate, the emotional change data of the student when making the gesture is identified. Based on the emotional change data and the type of the student's gesture, the student's learning needs and corresponding learning status are determined. Based on the learning needs and the learning status, a second prompt message is sent to the teacher. The gesture recognition neural network includes feature fusion, multi-layer Transformer, and fully connected layers. The multi-layer Transformer includes layer normalization, multi-head attention, and forward feedback.

2. The method according to claim 1, characterized in that, After preprocessing the first image sequence and the second image sequence, the process includes: Annotate the gesture information in the third processed image; Label the gesture type of each gesture and generate a mapping relationship with the third processed image; The third processed image labeled with gesture type is used as the training set for the gesture recognition neural network.

3. The method according to claim 1, characterized in that, Preprocessing the heart rate sequence includes: The heart rate sequence was examined to remove abnormal data; The data was standardized to obtain heart rate data with a standard normal distribution, meaning 0 and standard deviation 1.

4. The method according to claim 1 or 2, characterized in that, Training the gesture recognition neural network includes: Gesture features are extracted from the input gesture image; model input for each feature is generated using multiple convolutional layers, and the resulting image features are processed to the same size to generate planar coordinates for each feature, thus obtaining the positional encoding. Enhanced features are obtained based on the positional encoding process, and then decoder embedding is obtained; The offset and weight are calculated, and regression is performed on each weight to obtain the 3D coordinates of the predicted gesture. Normalized weights are calculated based on the 3D coordinates of the gesture; The loss function is used to calculate the coordinates of the real gesture and the predicted 3D coordinates of the gesture. The parameters of the gesture recognition neural network are adjusted based on the loss function.

5. An apparatus based on the interactive teaching method according to any one of claims 1-4, characterized in that, The device includes: The acquisition unit is used to collect the teacher's first image information, the student's second image information, and the student's heart rate data in real time. A preprocessing unit is configured to obtain a first image sequence of the teacher by combining the temporal order of the first image information, a second image sequence of the student by combining the temporal order of the second image information, and a heart rate sequence of the student by combining the temporal order of the heart rate data; and to preprocess the first image sequence, the second image sequence, and the heart rate sequence. A first processing unit is configured to input a third processed image obtained from the preprocessing of the first and second image sequences into a pre-trained gesture recognition neural network for recognition, and output the gesture type of the teacher and the gesture type of the student at the corresponding time point of the third processed image; and to obtain the heart rate change rate corresponding to the gesture type of the student at each time point. The second processing unit is used to issue a first prompt message in response to the teacher's gesture type at each time, based on a preset teaching instruction library, and to issue a second prompt message in response to the student's gesture type and the corresponding heart rate change rate at each time.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images