Children education strategy generation method and system based on multi-modal perception, and medium

Abstract

The invention discloses a children education strategy generation method and system based on multi-modal perception, and a medium, and relates to the technical field of children education strategies and interaction. Comprising the following steps: acquiring first modal behavior data and second modal behavior data of a child in a teaching process; according to the first modal behavior data, a ResNet-18 model is adopted to recognize the facial emotion of the child, and a first emotion classification result is obtained; according to the second modal behavior data, adopting an SVM classifier to classify the voice emotion to obtain a second emotion classification result; based on a D-S evidence theory, fusing the first emotion classification result and the second emotion classification result to obtain a final emotion classification result; according to the final emotion classification result, the development stage, the age interval, the symbolization ability and the logical reasoning threshold value, a four-dimensional evaluation matrix is constructed, and a teaching strategy suitable for the current state of the child is generated. According to the invention, the teaching content is dynamically matched with the children cognitive development stage.

Description

Technical Field

[0001] This invention relates to the field of children's education strategies and interaction technology, specifically to a method, system, and medium for generating children's education strategies based on multimodal perception. Background Technology

[0002] Existing children's education strategies are mostly generated based on electronic early education machines, but they suffer from the following technical challenges:

[0003] (1) One-way interaction:

[0004] Traditional electronic early education machines can only execute preset instructions and cannot adjust their strategies according to the child's real-time state. This type of technology uses a "preset instruction library + fixed response logic" as its core architecture, and its specific implementation includes:

[0005] Command triggering mechanism: Preset functions are triggered by physical buttons (such as "story button" and "nursery rhyme button"), simple voice commands (such as "play ABC song") or touch screen clicks. The command recognition range is limited to a predefined vocabulary library and cannot recognize children's non-standard pronunciation or vague expression of needs.

[0006] Content output logic: Once the command is triggered, the device will output content in a fixed order or for a fixed duration. For example, after triggering the "Math Enlightenment" command, the device will play audio recordings of number recognition from 1 to 10 in sequence. During this time, the playback rhythm or content will not be adjusted regardless of whether the child is distracted or asks repeated questions.

[0007] Lack of status feedback: There is no child status awareness module. There is no camera to capture children's attention or microphone to collect children's real-time feedback. The only way to judge children's needs is by "whether to trigger the command again", which causes the interaction to remain in a one-way loop of "child triggers - device executes".

[0008] (2) Lack of emotional perception:

[0009] Existing solutions (such as patent US2018 / 035678) do not integrate the dual analysis of facial expressions and voice emotions;

[0010] Some existing technologies attempt to incorporate emotion perception capabilities, but are limited to single-modal analysis. Specifically:

[0011] Single-modal data acquisition: This involves collecting children's voice data solely through a microphone or capturing their facial images solely through a camera, without simultaneously acquiring and analyzing both types of data. For example, one solution judges emotion solely based on voice tone (e.g., a rising tone is considered excitement), but ignores scenarios where the voice and expression contradict each other, such as "a child laughing while saying 'It's too hard'."

[0012] The emotional judgment dimension is too narrow: The emotional classification model is built based on a single data dimension, such as dividing emotions into "happy" and "irritable" based only on the "speech speed" and "volume" in speech, or judging emotions based only on the "curve of the corners of the mouth" and "state of the eyebrows" on the face. This cannot cover the complex emotions of children (such as "curiosity", "confusion" and "shyness").

[0013] Mechanical emotional response: Even when an emotion is detected, the response is relatively fixed. For example, when "irritability" is detected, the current content is simply paused and soothing music is played, without adjusting subsequent interaction strategies based on the cause of the emotion (such as the content being too difficult or too easy).

[0014] (3) Inadequate cognitive fit:

[0015] The teaching content was not integrated with Piaget's four-stage theory of cognitive development.

[0016] Therefore, most of the existing technologies mentioned above are based on single-modal data, which has problems such as the inability to quantitatively assess children's emotional state and the failure to dynamically match teaching content with children's cognitive development stages. Summary of the Invention

[0017] The technical problem this invention aims to solve is that most existing technologies are based on single-modal data, which suffers from limitations such as the inability to quantitatively assess children's emotional states and the lack of dynamic matching of teaching content with children's cognitive development stages. This invention aims to provide a method, system, and medium for generating children's educational strategies based on multimodal perception. This invention collects multimodal behavioral data of children during the teaching process, employs a hybrid emotion computing model to fuse facial expressions and vocal emotional features, and dynamically generates teaching content delivery strategies based on cognitive development stages. This invention assesses children's emotional states based on multimodal data and dynamically matches teaching content with children's cognitive development stages.

[0018] This invention is achieved through the following technical solution:

[0019] In a first aspect, the present invention provides a method for generating children's educational strategies based on multimodal perception, the method comprising:

[0020] Acquire multimodal behavioral data of children during the teaching process. The multimodal behavioral data includes first-modal behavioral data and second-modal behavioral data. The first-modal behavioral data consists of facial expressions and body movements, while the second-modal behavioral data consists of speech data.

[0021] Based on the first modality of behavioral data, the ResNet-18 model was used to identify children's facial emotions, resulting in a first emotion classification result; based on the second modality of behavioral data, the SVM classifier was used to classify speech emotions, resulting in a second emotion classification result.

[0022] Based on the DS evidence theory, the results of the first emotion classification and the second emotion classification are fused to obtain the final emotion classification result;

[0023] Based on the final emotion classification results, developmental stage, age range, symbolic ability, and logical reasoning threshold, a four-dimensional assessment matrix is ​​constructed; based on the assessment matrix, teaching strategies suitable for the child's current state are dynamically generated.

[0024] Furthermore, the steps for acquiring the first modality behavioral data are as follows:

[0025] The system captures children's facial expressions using RGB images from an RGB-D camera; based on these expressions, it obtains 3D facial information through depth images; based on this 3D information, it uses a facial key point detection algorithm to locate facial key points; and based on changes in the position of these key points, it identifies basic expressions, including happiness, confusion, and irritability.

[0026] The system uses depth images from an RGB-D camera to capture children's body movements; based on these movements, a skeletal tracking algorithm is used to identify key skeletal points in the children; and by analyzing the changes in these key skeletal points over time, the system identifies body movements, including waving and jumping.

[0027] Furthermore, the steps for acquiring the second modality behavioral data are as follows:

[0028] The system simultaneously collects ambient sound signals using a microphone array, including children's voices and environmental noise.

[0029] Beamforming technology is used to weight and combine multiple sound signals collected by a microphone array to obtain a speech signal.

[0030] The speech signal is preprocessed and speech features are extracted. The preprocessing includes denoising, framing, windowing, and Fourier transform.

[0031] Furthermore, based on the DS evidence theory, the results of the first and second emotion classifications are fused to obtain the final emotion classification result, including:

[0032] Based on the first emotion classification result and the second emotion classification result, calculate the trust level and likelihood of the first emotion classification result and the second emotion classification result respectively;

[0033] Based on the combined trust level of the DS evidence theory, the trust level and likelihood of the two modalities are fused to obtain the final emotion classification result.

[0034] Furthermore, the expression for combined trust level is:

[0035] ;

[0036] In the formula, and , respectively, are the trust functions for facial emotion recognition and voice emotion analysis in children; A is the emotion category, i.e., the final emotion classification result; B and C are the emotion recognition results under the two modalities, i.e., B is the first emotion classification result and C is the second emotion classification result.

[0037] Furthermore, based on the final emotion classification results, developmental stage, age range, and a combination of symbolic ability and logical reasoning threshold, a four-dimensional assessment matrix is ​​constructed, including:

[0038] The final emotion classification results are used as the emotion positivity index and listed as the first dimension of the evaluation matrix; the emotion positivity index reflects the child's emotional state during the current teaching process.

[0039] Based on Piaget's theory of cognitive development, children's cognitive development is divided into four stages, and the developmental stage is listed as the second dimension of the assessment matrix; the four stages of cognitive development include the sensorimotor stage, the preoperational stage, the concrete operational stage, and the formal operational stage.

[0040] The child's age range is listed as the third dimension of the assessment matrix; the age range corresponds to the developmental stage.

[0041] Symbolic ability and logical reasoning threshold are listed as the fourth dimension of the assessment matrix; symbolic ability and logical reasoning threshold reflect the specific ability level of children at their current cognitive development stage.

[0042] A four-dimensional assessment matrix is ​​constructed by combining the final emotion classification results, developmental stage, age range, symbolic ability, and logical reasoning threshold.

[0043] Furthermore, the method also includes:

[0044] The evaluation matrix is ​​optimized based on reinforcement learning, and the weight coefficients of each evaluation matrix are adjusted.

[0045] The reward function for reinforcement learning is: , As an emotional positivity index, To focus on study time, Total study time To accumulate the number of correct feedbacks or the number of correct operations / answers, This is the weighting coefficient for the positive emotional index. The weighting coefficient for the proportion of time spent on focus. The weighting coefficients for correct feedback.

[0046] Secondly, the present invention provides a children's education strategy generation system based on multimodal perception, the system comprising:

[0047] The multimodal data acquisition unit is used to acquire children's multimodal behavioral data during the teaching process. The multimodal behavioral data includes first modal behavioral data and second modal behavioral data. The first modal behavioral data is facial expression and body movement data, and the second modal behavioral data is voice data.

[0048] The emotion sub-classification unit is used to identify children's facial emotions using the ResNet-18 model based on the first modality of behavioral data, and obtain the first emotion classification result; and to classify speech emotions using the SVM classifier based on the second modality of behavioral data, and obtain the second emotion classification result.

[0049] The emotion outcome fusion unit is used to fuse the first emotion classification result and the second emotion classification result based on the DS evidence theory to obtain the final emotion classification result;

[0050] The teaching strategy generation unit is used to construct a four-dimensional assessment matrix based on the final emotion classification results, developmental stage, age range, symbolic ability, and logical reasoning threshold; and dynamically generate teaching strategies suitable for the child's current state based on the assessment matrix.

[0051] Furthermore, the multimodal data acquisition unit includes:

[0052] The first modality acquisition subunit is used to capture children's facial expressions based on RGB images from an RGB-D camera; obtain 3D facial information through depth images based on facial expressions; locate facial key points using a facial key point detection algorithm based on the 3D facial information; identify basic expressions based on changes in the position of facial key points, including happiness, confusion, and irritability; capture children's body movements based on depth images from an RGB-D camera; identify children's skeletal key points using a skeleton tracking algorithm based on body movements; analyze changes in skeletal key points over time to identify body movements, including waving and jumping.

[0053] The second modality acquisition subunit is used to simultaneously acquire sound signals from the environment based on a microphone array. The sound signals include children's voice and environmental noise. The multiple sound signals acquired by the microphone array are weighted and combined using beamforming technology to obtain a speech signal. The speech signal is preprocessed and speech features are extracted. The preprocessing includes denoising, framing, windowing, and Fourier transform.

[0054] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating children's educational strategies based on multimodal perception.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] This invention relates to a method, system, and medium for generating children's educational strategies based on multimodal perception. It collects multimodal behavioral data of children during the teaching process, employs a hybrid emotion computing model to integrate facial expressions and vocal emotional features, and dynamically generates teaching content delivery strategies based on cognitive development stages. This invention assesses children's emotional states based on multimodal data and dynamically matches teaching content with children's cognitive development stages to generate learning strategies adapted to different teaching processes. Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart of the children's education strategy generation method based on multimodal perception according to the present invention;

[0059] Figure 2 This is a block diagram of the children's education strategy generation system based on multimodal perception according to the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0061] Example 1

[0062] like Figure 1 As shown, the present invention provides a method for generating children's educational strategies based on multimodal perception, which includes:

[0063] Step 1: Obtain multimodal behavioral data of children during the teaching process. The multimodal behavioral data includes first-modal behavioral data and second-modal behavioral data. The first-modal behavioral data consists of facial expressions and body movements, and the second-modal behavioral data consists of speech data.

[0064] In this embodiment, step 1 involves acquiring facial expressions and body movements of children during the teaching process using an RGB-D camera. The startup and initialization of the RGB-D camera includes:

[0065] (1) Start the RGB-D camera and initialize the device parameters to ensure that the camera is in normal working condition;

[0066] (2) Set the camera sampling rate to 30 frames per second (fps) or higher to ensure the smoothness and accuracy of captured facial expressions and body movements.

[0067] In this embodiment, the steps for acquiring the first modal behavior data are as follows:

[0068] Children's facial expressions are captured using RGB images from an RGB-D camera; based on the facial expressions, three-dimensional facial information is obtained through depth images (D images) to improve the accuracy of expression recognition; based on the three-dimensional facial information, facial key point detection algorithms (such as Dlib, OpenFace, etc.) are used to locate facial key points (such as eyes, nose, mouth, etc.); basic expressions are identified based on the positional changes of facial key points, including happiness, confusion, and irritability.

[0069] The system captures children's body movements using depth images from an RGB-D camera; based on these movements, it employs skeletal tracking algorithms (such as OpenPose, Microsoft Kinect SDK, etc.) to identify key skeletal points (e.g., joint positions); and analyzes the changes in these key skeletal points over time to identify body movements, including waving and jumping.

[0070] In this embodiment, step 1 involves acquiring voice data using a beamforming microphone array; the initialization of the microphone array includes:

[0071] (1) Start the beamforming microphone array and initialize the device parameters to ensure that the microphone array is in normal working condition;

[0072] (2) Set the sampling rate and signal-to-noise ratio (SNR) parameters of the microphone array to ensure the quality of the acquired speech signal; the signal-to-noise ratio of the microphone array is ≥20dB.

[0073] In this embodiment, the steps for obtaining the second modal behavior data are as follows:

[0074] The system simultaneously collects ambient sound signals using a microphone array, including children's voices and environmental noise.

[0075] Beamforming technology is used to weight and combine multiple sound signals collected by a microphone array to obtain a speech signal, thereby enhancing children's speech signals and suppressing environmental noise.

[0076] The speech signal is preprocessed and speech features are extracted. The preprocessing includes denoising, framing, windowing and Fourier transform.

[0077] Among them, denoising (such as spectral subtraction, Wiener filtering, etc.) further improves the signal-to-noise ratio; preprocessing operations such as framing, windowing, and Fourier transform of the speech signal are performed to prepare for subsequent speech feature extraction.

[0078] Extract speech features (such as fundamental frequency, energy, speech rate, etc.) and acoustic features (such as Mel-frequency cepstral coefficients, MFCC).

[0079] The features extracted above are used for subsequent speech sentiment analysis and intent classification.

[0080] This invention uses multimodal behavioral data collected in step 1 (multimodal data acquisition stage) to ensure that the collected data is accurate and comprehensive, providing a reliable data foundation for subsequent emotion calculation in steps 2 and 3 and cognitive development stage determination in step 4.

[0081] Step 2: Based on the first modality behavior data, the ResNet-18 model is used to identify children's facial emotions to obtain the first emotion classification result; based on the second modality behavior data, the SVM classifier is used to classify speech emotions to obtain the second emotion classification result.

[0082] Step 3: Based on the DS evidence theory, the first emotion classification result and the second emotion classification result are fused to obtain the final emotion classification result;

[0083] In this embodiment, steps 2 and 3 are the emotion calculation stage. The ResNet-18 model used in this invention is fine-tuned on the FER-2013 dataset. The output of this model is the emotion probability distribution, that is, the first emotion classification result: pleasure (0.62) / confusion (0.25) / irritability (0.13). The SVM classifier used in this invention is an SVM classifier built based on Prosodic features (fundamental frequency, energy, speech rate).

[0084] In this embodiment, based on the DS evidence theory, the first emotion classification result and the second emotion classification result are fused to obtain the final emotion classification result, including:

[0085] Based on the first emotion classification result and the second emotion classification result, calculate the belief and likelihood of the first emotion classification result and the second emotion classification result respectively.

[0086] Based on the combined trust level of the DS evidence theory, the trust level and likelihood of the two modalities are fused to obtain the final emotion classification result.

[0087] Specifically, the expression for combined trust level is:

[0088] ;

[0089] In the formula, and , respectively, are the trust functions for facial emotion recognition and voice emotion analysis in children; A is the emotion category, i.e., the final emotion classification result; B and C are the emotion recognition results under the two modalities, i.e., B is the first emotion classification result and C is the second emotion classification result.

[0090] By using the ResNet-18 model and multimodal data fusion method described above, children's emotional states can be identified more accurately, providing reliable emotional information for the generation of subsequent teaching strategies.

[0091] Step 4: Based on the final emotion classification results, developmental stage, age range, symbolic ability, and logical reasoning threshold, construct a four-dimensional assessment matrix; based on the assessment matrix, dynamically generate teaching strategies suitable for the child's current state.

[0092] In this embodiment, step 4 is the determination of cognitive development stage. By combining the positive emotional index (0-1 standardized value) obtained in the previous step with three other dimensions (developmental stage, age range, symbolic ability, and logical reasoning threshold), a four-dimensional evaluation matrix is ​​constructed for dynamically generating teaching strategies. The specific implementation steps are as follows:

[0093] Step 41: Construct a four-dimensional assessment matrix based on the final emotion classification results, developmental stage, age range, symbolic ability, and logical reasoning threshold.

[0094] (1) The final emotion classification result is used as the positive emotion index (0-1 standardized value) and listed as the first dimension of the evaluation matrix; the positive emotion index reflects the emotional state of children in the current teaching process;

[0095] (2) Based on Piaget's theory of cognitive development, children's cognitive development is divided into four stages, and the developmental stage is listed as the second dimension of the assessment matrix. The four stages of cognitive development include the sensorimotor stage (1-2 years old), the preoperational stage (3-4 years old), the concrete operational stage (5-6 years old), and the formal operational stage (7 years old and above). Each cognitive stage corresponds to different cognitive abilities and learning characteristics.

[0096] (3) The age range of children is listed as the third dimension of the assessment matrix; the age range corresponds to the developmental stage;

[0097] (4) Symbolic ability and logical reasoning threshold are listed as the fourth dimension of the assessment matrix; symbolic ability and logical reasoning threshold reflect the child's specific ability level at the current cognitive development stage; among which:

[0098] Symbolic ability: A child's ability to use symbols (such as language and numbers) to represent things.

[0099] Logical reasoning threshold: The level of a child's ability to perform logical reasoning.

[0100] (5) Based on the final emotion classification results, development stage, age range, symbolic ability and logical reasoning threshold, a four-dimensional evaluation matrix is ​​constructed.

[0101] Each element in the assessment matrix represents a child's emotional state at a specific developmental stage, age range, symbolic ability, and logical reasoning threshold.

[0102] The constructed evaluation matrix is ​​shown in Table 1:

[0103] Table 1 Evaluation Matrix

[0104]

[0105] Step 42: Based on the assessment matrix, dynamically generate teaching strategies suitable for the child's current state.

[0106] For example, if the assessment matrix shows that the child is in the preoperational stage (3-4 years old), is 3 years old, has basic symbolic abilities, and has an emotional positivity index of 0.6, then the following teaching strategies can be generated:

[0107] (a) Select teaching content suitable for 3-year-old children with dual-condition judgment (such as simple classification games).

[0108] (b) In combination with the children’s emotional state, increase interaction and feedback appropriately to maintain their interest in learning.

[0109] (c) Help young children develop symbolic abilities through voice and visual cues (such as using pictures and simple language descriptions).

[0110] As a further implementation, the method also includes:

[0111] Step 5: Optimize the evaluation matrix based on reinforcement learning and adjust the weight coefficients of each component in the evaluation matrix;

[0112] Specifically, the evaluation matrix is ​​continuously optimized based on historical interaction data to improve the relevance and effectiveness of teaching strategies. Through machine learning algorithms (such as reinforcement learning), weight coefficients are adjusted (e.g., α=0.4, β=0.3, γ=0.3) to adapt to the needs and learning pace of different children.

[0113] The reward function for reinforcement learning is:

[0114]

[0115] In the formula, As an emotional positivity index, To focus on study time, Total study time To accumulate the number of correct feedbacks or the number of correct operations / answers, This is the weighting coefficient for the positive emotional index. The weighting coefficient for the proportion of time spent on focus. The weighting coefficients for correct feedback.

[0116] The above reward function is designed to comprehensively assess children's learning status through multimodal perception (such as visual and auditory perception).

[0117] Positive Emotion Index ( ): Reflects a child's emotional state during the learning process; positive emotions (such as interest and pleasure) can earn higher rewards.

[0118] Focus time ratio ): Measures how well a child maintains attention on learning tasks and encourages high levels of focus;

[0119] Logarithmic correct feedback ( Rewarding correct behaviors (such as answering questions correctly and interacting successfully) using a logarithmic approach can prevent an initial surge in correct responses from resulting in excessively high reward values, allowing the model to focus more on long-term stable performance.

[0120] By adjusting the weights α, β, and γ, the system can flexibly balance the influence of emotion, focus, and accuracy on the final strategy generation.

[0121] Through steps 1 to 5 of this invention, teaching strategies can be dynamically generated and optimized based on children's cognitive development stages and emotional states, thereby improving the effectiveness and adaptability of early childhood education.

[0122] In practical implementation, the following application scenarios are possible:

[0123] First, the language training scenario is as follows:

[0124] 1. A child says, "I want to hear a dinosaur story."

[0125] 2. System:

[0126] (21) The keyword "dinosaur" was detected (TF-IDF weight 0.85);

[0127] (23) Identify micro-expressions AU4 (frowning) + AU12 (turning up the corners of the mouth) → Confusion index 0.4;

[0128] (23) Activate auxiliary strategies:

[0129] Play a 3D dinosaur model and ask: "How many legs does a Tyrannosaurus Rex have?"

[0130] If there is no response within 5 seconds, the selection range will be automatically narrowed down (displaying a comparison image of 2 / 4 of the feet).

[0131] Second, scenarios for early math education

[0132] 1. A touchscreen display comparing the number of fruits;

[0133] 2. Repeated erasure of operation trajectory detected → Emotion index drops to 0.3;

[0134] 3. System:

[0135] (31) Switch to physical teaching aid linkage mode (connect building blocks via AR markers);

[0136] (32) Generate encouraging voice: "You have completed 80%, do you need the assistant's help?"

[0137] Example 2

[0138] like Figure 2 As shown, the difference between this embodiment and Embodiment 1 is that this embodiment provides a children's education strategy generation system based on multimodal perception, which corresponds one-to-one with the children's education strategy generation method based on multimodal perception in Embodiment 1; the system includes:

[0139] The multimodal data acquisition unit is used to acquire children's multimodal behavioral data during the teaching process. The multimodal behavioral data includes first modal behavioral data and second modal behavioral data. The first modal behavioral data is facial expression and body movement data, and the second modal behavioral data is voice data.

[0140] The emotion sub-classification unit is used to identify children's facial emotions using the ResNet-18 model based on the first modality of behavioral data, and obtain the first emotion classification result; and to classify speech emotions using the SVM classifier based on the second modality of behavioral data, and obtain the second emotion classification result.

[0141] The emotion outcome fusion unit is used to fuse the first emotion classification result and the second emotion classification result based on the DS evidence theory to obtain the final emotion classification result;

[0142] The teaching strategy generation unit is used to construct a four-dimensional assessment matrix based on the final emotion classification results, developmental stage, age range, symbolic ability, and logical reasoning threshold; and dynamically generate teaching strategies suitable for the child's current state based on the assessment matrix.

[0143] As a further implementation, the multimodal data acquisition unit includes:

[0144] The first modality acquisition subunit is used to capture children's facial expressions based on RGB images from an RGB-D camera; obtain 3D facial information through depth images based on facial expressions; locate facial key points using a facial key point detection algorithm based on the 3D facial information; identify basic expressions based on changes in the position of facial key points, including happiness, confusion, and irritability; capture children's body movements based on depth images from an RGB-D camera; identify children's skeletal key points using a skeleton tracking algorithm based on body movements; analyze changes in skeletal key points over time to identify body movements, including waving and jumping.

[0145] The second modality acquisition subunit is used to simultaneously acquire sound signals from the environment based on a microphone array. The sound signals include children's voice and environmental noise. The multiple sound signals acquired by the microphone array are weighted and combined using beamforming technology to obtain a speech signal. The speech signal is preprocessed and speech features are extracted. The preprocessing includes denoising, framing, windowing, and Fourier transform.

[0146] The execution process of each unit can be carried out according to the steps of the method for generating children's education strategies based on multimodal perception in Example 1, and will not be described in detail in this example.

[0147] Meanwhile, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for generating children's educational strategies based on multimodal perception.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a child education strategy based on multi-modal perception, characterized in that, The method comprises: acquiring multi-modal behavior data of children in a teaching process, the multi-modal behavior data comprising first modal behavior data and second modal behavior data, the first modal behavior data being facial expression and body movement data, and the second modal behavior data being speech data; according to the first modal behavior data, using a ResNet-18 model to recognize the facial emotion of the children to obtain a first emotion classification result; and according to the second modal behavior data, using an SVM classifier to classify the speech emotion to obtain a second emotion classification result; based on D-S evidence theory, fusing the first emotion classification result and the second emotion classification result to obtain a final emotion classification result; according to the final emotion classification result, a development stage, an age interval, and a combination of symbolic ability and logical reasoning threshold, constructing a four-dimensional evaluation matrix; and according to the evaluation matrix, dynamically generating a teaching strategy suitable for the current state of the children.

2. The multimodal perception based child education strategy generation method according to claim 1, wherein, The acquisition step of the first modal behavior data comprises: capturing the facial expression of the children based on an RGB image of an RGB-D camera; and obtaining facial three-dimensional information through a depth image according to the facial expression; positioning facial key points by using a facial key point detection algorithm according to the facial three-dimensional information; recognizing basic expressions according to the position changes of the facial key points; capturing the body movement of the children based on a depth image of the RGB-D camera; and recognizing the skeletal key points of the children by using a skeletal tracking algorithm according to the body movement; analyzing the changes of the skeletal key points over time to recognize the body movement. 3.The multi-modal perception based child education strategy generation method of claim 1, wherein, The acquisition step of the second modal behavior data comprises: simultaneously collecting sound signals in the environment based on a microphone array, the sound signals comprising the speech of the children and environmental noise; obtaining a speech signal by weighting and combining multiple sound signals collected by the microphone array through beamforming technology; preprocessing the speech signal and extracting speech features, the preprocessing comprising denoising, framing, windowing, and Fourier transform.

4. The multimodal perception based child education strategy generation method of claim 1, wherein, based on D-S evidence theory, fusing the first emotion classification result and the second emotion classification result to obtain a final emotion classification result, comprising: calculating the trust degree and the likelihood degree of the first emotion classification result and the second emotion classification result, respectively, according to the first emotion classification result and the second emotion classification result; fusing the trust degree and the likelihood degree under two modalities to obtain the final emotion classification result based on the combined trust degree of D-S evidence theory.

5. The multimodal perception based child education strategy generation method of claim 4, wherein, The expression of the combined trust degree is: ； In the formula, and respectively are the trust degree functions of the child facial emotion recognition and the speech emotion analysis; A is the emotion category, i.e. the final emotion classification result; B and C are the emotion recognition results under two modalities, i.e. B is the first emotion classification result and C is the second emotion classification result.

6. The multimodal perception based child education strategy generation method according to claim 1, wherein, according to the final emotion classification result, a development stage, an age interval, and a combination of symbolic ability and logical reasoning threshold, constructing a four-dimensional evaluation matrix, comprising: taking the final emotion classification result as an emotional positive index and listing it as the first-dimensional data of the evaluation matrix; the emotional positive index reflects the emotional state of the children in the current teaching process; based on Piaget's cognitive development theory, dividing the cognitive development of the children into four stages, and listing the development stage as the second-dimensional data of the evaluation matrix; the four stages of cognitive development comprise the perceptual-motor stage, the pre-operational stage, the concrete operational stage, and the formal operational stage; an age interval of the child is listed as the third dimension data of the evaluation matrix; the age interval corresponds to the development stage; symbolic ability and logical reasoning threshold are listed as the fourth dimension data of the evaluation matrix; the symbolic ability and logical reasoning threshold reflect the specific ability level of the child at the current cognitive development stage; a four-dimensional evaluation matrix is constructed according to the final emotion classification result, the development stage, the age interval and the symbolic ability and logical reasoning threshold.

7. The multimodal perception based child education strategy generation method according to claim 1, wherein, The method further comprises: optimizing the evaluation matrix based on reinforcement learning to adjust the weight coefficients of the evaluation matrix; The reward function of the reinforcement learning is: , is an emotional positive index, is a focused learning time, is a total learning time, is a cumulative correct feedback number or a cumulative correct operation / answer number, is a weight coefficient of the emotional positive index, is a weight coefficient of the focused time proportion, is a weight coefficient of the correct feedback.

8. A child education strategy generation system based on multi-modal perception, characterized in that, The system comprises: a multi-modal data acquisition unit configured to acquire multi-modal behavior data of the child during the teaching process, the multi-modal behavior data comprising first modal behavior data and second modal behavior data, the first modal behavior data being facial expression and body movement data, and the second modal behavior data being voice data; an emotion sub-classification unit configured to identify the facial emotion of the child according to the first modal behavior data using a ResNet-18 model to obtain a first emotion classification result, and classify the voice emotion according to the second modal behavior data using an SVM classifier to obtain a second emotion classification result; an emotion result fusion unit configured to fuse the first emotion classification result and the second emotion classification result based on D-S evidence theory to obtain a final emotion classification result; a teaching strategy generation unit configured to construct a four-dimensional evaluation matrix according to the final emotion classification result, the development stage, the age interval and the symbolic ability and logical reasoning threshold, and dynamically generate a teaching strategy suitable for the current state of the child according to the evaluation matrix.

9. The multi-modal perception based child education strategy generation system of claim 8, wherein, The multi-modal data acquisition unit comprises: a first modal acquisition sub-unit configured to capture the facial expression of the child based on the RGB image of the RGB-D camera, obtain the three-dimensional information of the face based on the facial expression, locate the facial key points using a facial key point detection algorithm based on the three-dimensional information of the face, identify the basic expression based on the position change of the facial key points, capture the body movement of the child based on the depth image of the RGB-D camera, identify the skeletal key points of the child using a skeletal tracking algorithm based on the body movement, and analyze the change of the skeletal key points over time to identify the body movement; a second modal acquisition sub-unit configured to simultaneously collect sound signals in the environment based on a microphone array, the sound signals comprising the voice of the child and environmental noise, combine the multiple sound signals collected by the microphone array using beamforming technology to obtain a voice signal, and pre-process the voice signal and extract voice features, the pre-processing comprising denoising, framing, windowing and Fourier transform.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program, when executed by a processor, implements the multi-modal perception-based child education strategy generation method of any one of claims 1 to 7.

Images