Hierarchical infant drawing inspiration system and method based on multi-modal behavior perception

Abstract

The application relates to the technical field of intelligent education and man-machine interaction, in particular to a hierarchical infant drawing inspiration system and method based on multi-modal behavior perception, which comprises a multi-modal perception evaluation module, a hierarchical decision engine and an adaptive interaction module. Behavior data is collected through a sensor array, and a concentration, emotion and skill evaluation vector is generated by using deep learning; the engine realizes accurate grade division and smooth transition based on a clustering algorithm and reinforcement learning, and matches an inspiration strategy in real time. The application can realize deep perception and adaptive guidance of an infant drawing state, an intelligent interaction closed loop is constructed through AR projection and multi-modal feedback, and the problems of rigid guidance strategy and insufficient individualized teaching ability in the prior art are effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of smart education and human-computer interaction technology, and in particular to a graded early childhood drawing inspiration system and method based on multimodal behavior perception. Background Technology

[0002] Painting, as an important medium for young children to express emotions, understand the world, and inspire creativity, occupies a pivotal position in early childhood education. Traditional early childhood painting instruction mainly relies on on-site guidance from teachers or parents. This model is not only extremely costly in terms of manpower, making it difficult to achieve large-scale implementation, but the guidance process is also often highly subjective. With the rapid development of smart education and human-computer interaction technologies, a series of digital teaching products have emerged on the market, such as instructional videos and painting applications, aiming to assist young children's painting learning through information technology.

[0003] However, existing digital art teaching technologies still have significant shortcomings in practical applications. First, existing products generally lack the ability to perceive children's status in real time, and most operate on a one-way information input model, failing to effectively capture children's real-time feedback during the drawing process. Due to this lack of perception, the system struggles to identify whether a child is attentively listening, cannot determine whether a child is deeply thinking or experiencing frustration when encountering drawing difficulties, and is unable to keenly perceive changes in the child's emotions. Consequently, the entire teaching process becomes merely a simple transmission of information, lacking necessary two-way interaction.

[0004] Secondly, existing teaching models often exhibit a one-size-fits-all approach. The teaching content and difficulty of these products are usually pre-set and fixed, failing to meet the individualized needs of children of different ages and developmental levels. For children with higher drawing skills, fixed content can easily lead to boredom, while for children with lower drawing skills, the pre-set difficulty is often too high. This imbalance between supply and demand makes it impossible for existing technologies to truly achieve individualized instruction.

[0005] Furthermore, existing interaction methods are too simplistic and lack flexibility. Current interaction methods mainly rely on pre-recorded voice or static image prompts, lacking an interactive mode that can intelligently adjust based on children's real-time drawing behavior. This lack of targeted interaction mechanisms results in insufficient immediacy of feedback, making it difficult to provide precise inspiration based on children's dynamic performance.

[0006] In summary, existing early childhood art education systems have shortcomings in terms of perception capabilities, teaching models, and depth of interaction. To improve the effectiveness of early childhood art education, the industry urgently needs an intelligent art inspiration system that can accurately perceive children's states and provide adaptive, personalized guidance. This system should deeply integrate multimodal behavioral perception with intelligent hierarchical decision-making to achieve real-time, dynamic, and intelligent guidance of children's painting process, thereby providing them with more efficient and personalized art learning tools. Summary of the Invention

[0007] This invention addresses the shortcomings of existing early childhood art education systems, such as limited perceptual dimensions, inability to capture real-time feedback on children's psychological and physiological states, rigid guidance strategies, and a lack of adaptive adjustment to individual developmental differences. It aims to provide a tiered early childhood art education system and method based on multimodal behavioral perception. This invention achieves comprehensive and in-depth perception of children's painting behavior, physiological characteristics, and emotional states by integrating a multi-source sensor array. Combined with a deep learning-based state quantification model and a reinforcement learning-based dynamic tiered decision engine, it constructs an intelligent closed-loop system capable of accurately identifying, scientifically grading, and implementing personalized interactive guidance based on children's real-time performance. This solves the technical problems of delayed teaching feedback, limited interaction modes, and the inability to achieve "teaching according to aptitude" in existing technologies.

[0008] This invention provides a hierarchical early childhood drawing inspiration system based on multimodal behavior perception, comprising a multimodal behavior perception and state assessment module, a hierarchical inspiration decision engine, and an adaptive interactive presentation module. The multimodal behavior perception and state assessment module is connected to a sensor array via a hardware interface. The sensor array includes a high-definition camera positioned above the drawing area, a pressure-sensitive drawing board embedded below the drawing support surface, a microphone array positioned in front of the child, and a posture sensor worn on the child's wrist.

[0009] The multimodal behavior perception and state assessment module performs data preprocessing operations, simultaneously acquiring facial expression image sequences, pen stroke pressure trajectory data, speech signals, and limb movement features of young children using a multi-threaded concurrency mechanism. Furthermore, the module includes a feature extraction unit for denoising, resampling, and time-axis alignment of the acquired raw data. Specifically, for facial expression images, a convolutional neural network is used to extract facial key point features and micro-expression vectors; for pen stroke pressure trajectories, high-pass filtering is used to extract the tremor frequency and pressure change gradient of the pen strokes; for speech signals, Mel-frequency cepstral coefficients are used to extract the intonation features of children's self-talk or questions during the drawing process; and for limb movements, a quaternion algorithm is used to calculate the acceleration and angular velocity of the wrist.

[0010] Based on the feature extraction results described above, the multimodal behavior perception and state assessment module generates a comprehensive state assessment vector using a multimodal fusion algorithm. Among them, the focus component The data is obtained by tracking the coordinates of the child's gaze point using a high-definition camera, calculating the percentage of time the gaze area falls within the canvas area, and combining this with a weighted sum of body sway frequencies output by a posture sensor. The weighting coefficients are dynamically adjusted based on a preset model of the child's age group. Emotional valence component. The acquisition method is as follows: the extracted facial micro-expression vectors and voice intonation features are input into a pre-trained bidirectional long short-term memory network, which outputs the child's current positive emotion score, which is distributed within the closed interval [0, 1]. Skill level component The method of obtaining the data is as follows: calculate the Hausdorff distance between the current pen stroke trajectory and the preset standard template, and make a comprehensive evaluation by combining the uniformity of pen stroke pressure and the time efficiency of completing the unit area composition.

[0011] The hierarchical heuristic decision engine communicates with the multimodal behavior perception and state assessment module to receive real-time updated comprehensive state assessment vectors. The engine internally constructs a capability-level clustering space optimized based on the K-means++ algorithm. This space pre-defines three cluster centers corresponding to the "exploration period," "imitation period," and "creation period." The hierarchical heuristic decision engine performs Euclidean distance calculation to measure real-time vectors. Distance from each cluster center .

[0012] Furthermore, the hierarchical heuristic decision engine includes a logical decision unit. When the minimum distance... When the drawing level is within a preset stable threshold range, the system automatically locks the child's current drawing level. Specifically, when the vector... The overlapping region of two cluster centers or all distances When all values ​​exceed a preset confidence threshold, the engine classifies the state as a "level transition period." For children in the level transition period, the decision engine invokes a built-in Markov decision process model, treating the current heuristic guidance strategy as the "action space" and the child's subsequent changes in attention and skill gains as the "reward function." It then uses a reinforcement learning algorithm to search online and determine the optimal heuristic strength and guidance modality.

[0013] The adaptive interactive presentation module drives the interactive output device to execute differentiated presentation logic based on the decision instructions output by the hierarchical heuristic decision engine. The interactive output device includes an augmented reality (AR) projection unit, a haptic feedback vibration unit, and an intelligent audio synthesis unit. Specifically, when a child is determined to be in the "exploration phase," the adaptive interactive presentation module activates the AR projection unit to overlay a high-contrast dynamic light and shadow outline onto the drawing paper, and the intelligent audio synthesis unit plays rhythmic onomatopoeic guidance to guide the child in completing the basic line drawing.

[0014] Furthermore, once toddlers enter the "imitation stage," the adaptive interactive presentation module implements a gradual fading guidance scheme. This module controls the transparency of the projected content to linearly decrease from 100% to 20% over time, prompting toddlers to shift from simple covering and copying to memory-based shape replication. Simultaneously, the tactile feedback vibration unit generates micro-feedback vibrations of different frequencies in real time based on the degree of pen stroke deviation, used to correct the toddler's pen grip and writing trajectory.

[0015] For children identified as being in the "creative stage," the adaptive interactive presentation module switches to an open-ended heuristic mode. This module no longer provides specific line templates but instead uses generative adversarial networks to generate complementary virtual background scenes in real-time based on the child's existing partial drawings. These background scenes are then integrated into the physical environment surrounding the canvas using projection technology. Furthermore, the system considers the child's emotional valence components... The fluctuations in the background music's tone and the color saturation of the virtual scene are dynamically adjusted to stimulate children's desire to express their emotions.

[0016] This invention also provides a graded method for inspiring early childhood drawing based on multimodal behavioral perception, characterized by the following steps: S1: The system initializes and starts the multi-source sensor array. The high-definition camera captures facial images and ambient light information of the child at a frame rate of no less than 60fps. The pressure-sensitive drawing board records the two-dimensional coordinates and pressure values ​​of the brushstrokes in real time at a sampling frequency of 100Hz. The microphone array and posture sensor are turned on simultaneously to obtain environmental voice and body posture data.

[0017] S2: Performs multimodal data synchronization and fusion processing. The system uses a unified clock source to label various sensor data, ensuring that the time deviation of different modal data is less than 10ms. Speech and tremor signals are processed through Fast Fourier Transform, and facial images are aligned and normalized using a spatial transformation network.

[0018] S3: Construct a three-dimensional state evaluation vector. The system calls a preset evaluation model to calculate the components representing focus. Components representing emotional states and the component representing the level of painting skills .in, The calculation combines the entropy of the eye-tracking trajectory with the stability of the head posture; The calculation is based on the rate of change of facial expression feature points and the fundamental frequency of speech. The calculation depends on the smoothness of the brushstrokes, the completeness of the closed shape, and the complexity of the color distribution.

[0019] S4: Execute intelligent hierarchical decision-making logic. The generated vector... Mapped to a three-dimensional feature space, the geometric distance between the vector and three preset standard cluster centers for the "exploration period," "imitation period," and "creation period" is calculated. If the vector falls within the neighborhood of a cluster center, the corresponding teaching level is determined; if the vector is in a region with ambiguous boundaries, a reinforcement learning mechanism is triggered to obtain real-time feedback from children through tentative interactive actions, and the grading results are adjusted accordingly.

[0020] S5: Implement adaptive heuristic-guided interaction. Based on the level determined in step S4, the system automatically retrieves the corresponding interaction parameters from the heuristic strategy library. During the exploration phase, a strong visual guidance strategy is implemented, using bright light spots to guide children's gaze; during the imitation phase, a weak intervention and assistance strategy is implemented, using tactile feedback to assist in the formation of muscle memory; during the creative phase, a divergent association strategy is implemented, using generative artificial intelligence technology to provide creative background support.

[0021] S6: Implement closed-loop feedback adjustment. The system continuously monitors the child's state vector after receiving guidance and inspiration. The changing trend of focus. A sustained decline or emotional valence If the behavior is negative, the system will automatically reduce the difficulty of guidance or change the interaction mode until the state vector returns to the preset ideal range.

[0022] Furthermore, in the multimodal behavior perception and state assessment module, the focus component... The computational model is implemented as follows: First, the system acquires the child's gaze offset vector within a unit time window, and assesses the degree of visual attention concentration by calculating the cumulative distribution function of this vector. Simultaneously, using acceleration data collected by a six-axis inertial measurement unit, the system extracts the energy spectrum of the child's body swaying in the low-frequency range (0.5Hz-2Hz), and uses the reciprocal of this energy spectrum as a measure of postural stability. Finally, a linear regression equation is used to fuse the visual attention score and the postural stability score, thereby obtaining a quantitative value of focus that reflects the child's true psychological engagement.

[0023] Furthermore, in the hierarchical heuristic decision engine, the reinforcement learning algorithm employs a deep Q-network (DQN) architecture. The input layer of this architecture receives the current state vector. Based on the characteristics of the current guidance strategy, the intermediate layer extracts deep associations between state-action pairs through a multilayer perceptron, while the output layer predicts the cumulative benefits that different guidance actions (such as increasing projection brightness, playing encouraging voices, and reducing pen stroke assistance) will bring over a period of time. In this way, when facing young children in critical developmental stages, the system can automatically find the interaction combination that best stimulates their potential.

[0024] Furthermore, the AR projection unit in the adaptive interactive presentation module has an automatic calibration function. The module acquires the pose information of the drawing paper in three-dimensional space in real time through a depth vision sensor, and uses an affine transformation matrix to map the virtual guide pattern to the precise position on the physical canvas in real time. Even if a child moves the drawing paper or changes their sitting posture during the drawing process, the system can recalculate the projection coordinates in a very short time, thereby ensuring the accuracy of the alignment between the guide information and the physical brushstrokes, and avoiding visual misalignment that could interfere with the child's work.

[0025] Furthermore, the intelligent audio synthesis unit is used not only to output commands but also to perform emotional feedback. The system has a built-in emotional speech synthesis engine that can perform emotional valence components. The system automatically adjusts the speech rate, pitch, and energy of the synthesized speech based on real-time data. When the system detects negative emotions caused by a child's creative block, it automatically switches to a gentle, encouraging tone; when it detects positive emotions generated by a child completing a complex composition, it switches to a cheerful, appreciative tone. Through auditory emotional compensation, the system further optimizes the child's painting experience environment.

[0026] Furthermore, the flexible pressure-sensitive array employs a high-resolution matrix layout, enabling it to capture the pressure distribution of the pen tip on the canvas with millimeter-level precision. By analyzing the power spectral density of the pressure trajectory, the system can distinguish between intentional sketching and unconscious doodling by the child. For unconscious doodling, the system will automatically lower the skill level component. The weight of the light and shadow is used to trigger the guidance mode of the exploration period, and to guide children back to the purposeful creative process through fun light and shadow interaction.

[0027] Furthermore, in the execution of the method described in this invention, the data alignment in step S2 employs a hardware-triggered synchronization mechanism. At the beginning of each acquisition cycle, the main control unit in the sensor group sends a synchronization pulse signal to all sub-sensors. Upon receiving the pulse, each sensor immediately archives its current sampling buffer and affixes a global timestamp. This hardware-level synchronization scheme eliminates random jitter caused by operating system scheduling, providing a microsecond-level timing foundation for the subsequent accurate correlation of multimodal features.

[0028] Furthermore, in step S5, for the adaptive inspirational guidance during the "creative period," the system automatically associates related logical elements (such as "bones" and "clouds") based on the object labels (such as "puppy" and "sun") drawn by the child, using a pre-built semantic knowledge graph. These elements appear as semi-transparent shadows in the blank areas of the canvas. This semantic association-based inspirational approach not only broadens children's creative thinking but also guides them to learn the logical relationships between objects, achieving simultaneous promotion of drawing skills improvement and cognitive development.

[0029] The beneficial effects of this invention are as follows: First, this invention achieves high-precision, multi-dimensional, real-time perception of children's drawing status. By integrating multiple sensors such as vision, touch, hearing, and posture, it overcomes the drawbacks of traditional teaching systems that involve one-way information transmission and lack of feedback. Second, this invention constructs a scientific hierarchical decision-making mechanism and a smooth transition strategy for different levels. By quantifying children's drawing abilities and status into vectors in three-dimensional space and using clustering algorithms for level classification, it achieves personalized customization of educational content. Third, this invention provides a highly adaptive and immersive interactive experience. The system no longer provides rigid, pre-set teaching processes but dynamically adjusts the presentation of interactive content based on children's real-time performance. From strong guidance in the exploration stage to gradual assistance in the imitation stage, and then to generative backgrounds in the creative stage, this progressive inspirational logic conforms to the laws of children's cognitive development. Fourth, this invention possesses efficient system response performance and good engineering scalability. Through an optimized multi-threaded concurrent processing mechanism and a lightweight deep learning inference model, the system can control the end-to-end perception and decision-making latency within 200ms, ensuring the immediacy of interactive feedback. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the embodiments of the present invention or the prior art will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 This is a block diagram of the overall architecture of a graded early childhood drawing inspiration system based on multimodal behavior perception, according to the present invention. Figure 2This is a flowchart illustrating the logical processing of the multimodal behavior perception and state assessment module in this invention. Figure 3 This is a schematic diagram of the hierarchical heuristic decision engine's level determination and decision logic in this invention. Figure 4 This is a schematic diagram illustrating the strategy execution of the adaptive interactive presentation module at different levels in this invention; Figure 5 This is a schematic diagram of the application interface of the present invention. Detailed Implementation

[0032] Combined with appendix Figure 1 To be continued Figure 5 This invention provides a detailed description of specific embodiments. The core architecture of this invention, a tiered early childhood drawing inspiration system based on multimodal behavior perception, constructs a complete closed-loop control circuit through the cascading collaboration of a multimodal behavior perception and state assessment module, a tiered inspirational decision engine, and an adaptive interactive presentation module. At the hardware level, the system integrates a high-definition camera, a pressure-sensitive drawing board, a microphone array, and a posture sensor to capture children's physiological and behavioral data in real time during the drawing process. At the output end, the system utilizes an augmented reality projection unit, a haptic feedback vibration unit, and an intelligent audio synthesis unit to implement differentiated inspirational guidance based on the decision results.

[0033] See attached document Figure 1 As shown, the multimodal behavior perception and state assessment module serves as the system's input core, connected to the sensor array via a high-speed bus. A high-definition camera is typically mounted directly above the child's drawing area, covering the entire drawing paper area and the child's face from a top-down angle. Its sampling frequency is set to at least 60 frames per second to capture facial expression image sequences. A pressure-sensitive drawing board is embedded beneath the surface, featuring 1024 levels of pressure sensitivity to record the two-dimensional coordinates and dynamic pressure values ​​of the brushstrokes in real time. A microphone array is positioned at the front of the system support, using beamforming technology to filter ambient noise and accurately pick up the child's voice signals. A posture sensor, either worn on a wristband or embedded in the end of the brush, acquires data on the acceleration and angular velocity of the child's hand movements.

[0034] See attached document Figure 2 After acquiring the raw data, the multimodal behavior perception and state assessment module performs a series of complex preprocessing and feature extraction operations. This is specifically tailored to the attention component. The calculation module first calls the eye-tracking algorithm of the high-definition camera to determine the child's position within a given time window. Total time spent looking into the canvas area At the same time, the number of times the viewer's gaze leaves the canvas area is counted. Combined with body posture stability data fed back by posture sensor 7 The system uses a weighted formula The focus level score is calculated. The weighting coefficients are... The values ​​can be 0.5, 0.3, and 0.2 respectively, to ensure that visual focus plays a dominant role in the assessment.

[0035] Regarding the valence of emotion The evaluation module, which focuses on multimodal behavior perception and state assessment, utilizes a convolutional neural network (CNN) to evaluate facial image sequences. Deep analysis was performed. The convolutional layer extracted micro-expression features such as upturned corners of the mouth and furrowed brows in young children using three 3x3 filters. After mapping through a fully connected layer, these features were activated by a sigmoid function. Output a continuous emotion score between 0 and 1. Specifically, It is possible Where Conv: a three-layer convolutional feature extraction function; δ: the ReLU activation function; : Fully connected weight matrix. When young children exhibit feelings of joy and surprise, The value approaches 1. When feelings of frustration or boredom arise, The value approaches 0.

[0036] Based on skill level components The system evaluates stroke trajectory by acquiring it through a pressure-sensitive drawing board. The module calculates the variance of the first derivative of the stroke speed sequence and takes its reciprocal as the stroke smoothness. Simultaneously, the current trajectory is compared with a preset reference shape by the system, and the average deviation value is calculated. Skill level components are expressed using a formula. Achieving quantification, where weighting coefficients Take 0.6, Take 0.4. Finally, module 1 will... , , Encapsulated as a comprehensive state evaluation vector .

[0037] See attached document Figure 3 The hierarchical heuristic decision engine receives vectors Then, the hierarchical mapping logic is executed. The engine internally has three pre-defined cluster centers representing the exploration phase, the imitation phase, and the creation phase. Let i ∈ {exploration period, imitation period, creation period}. The system calculates the Euclidean distance. To determine the current developmental stage of a child. If the minimum distance... Less than the preset threshold (For example If all All are greater than or equal to The engine determines that the child is in a transitional period between levels.

[0038] During the transition period between levels, the hierarchical heuristic decision engine initiates a strategy self-learning mechanism based on reinforcement learning. The system defines different guided interactive actions as an action space, and then tracks the child's subsequent attention span. Improvement or skill level Growth is defined as a reward function. By continuously experimenting with slight adjustments to variables such as projection brightness and voice prompt frequency, the engine searches for and determines the most suitable blended guidance strategy for a specific child. This dynamic adjustment mechanism eliminates the abrupt level transitions in traditional teaching, achieving a smooth transition in guidance intensity.

[0039] See attached document Figure 4 The adaptive interactive presentation module, based on the engine's decision instructions, calls upon hardware resources to implement personalized guidance. During the exploration phase, the system determines that the child is in a low-order cognitive exploration state, at which point their concentration level... With skill level Typically, the readings are in the lower range. The adaptive interactive presentation module drives the augmented reality projection unit to project simple geometric outlines with bright red borders onto the drawing paper. Simultaneously, the intelligent audio synthesis unit plays step-by-step instructions at a slow pace, guiding the child to complete basic line closures. If the pressure-sensitive drawing board detects a stroke deviation exceeding 0.3, the projection unit will provide real-time correction prompts by flashing a red halo.

[0040] As toddlers enter the imitation stage, the system recognizes that they possess a certain level of observational ability and pen control. The adaptive interactive presentation module executes a gradual fading guidance strategy, with the transparency of the demonstration animation projected by the augmented reality projection unit gradually decreasing from 100% to 30%, forcing the toddler to gradually reduce visual dependence. At this point, the haptic feedback vibration unit plays a crucial role, responding to changes in the pen stroke trajectory. When the vibration intensity exceeds 0.5, the vibrating unit drives the paintbrush to produce a weak high-frequency vibration, prompting the child to adjust the hand pressure through tactile feedback, rather than directly providing a visual answer.

[0041] During the creative phase, young children demonstrate a high level of focus and creative autonomy. The adaptive interactive presentation module stops providing specific outline templates and instead drives the augmented reality projection unit to generate dynamic, related scenes in the physical environment surrounding the canvas. For example, if a child draws a tree in the center of the canvas, the system automatically projects the silhouettes of birds, grass, or clouds around the edge of the paper. The intelligent audio synthesis unit then plays open-ended prompts, such as asking the child if they want to add fruit to the tree. In this way, the system shifts the focus from skills training to creative stimulation.

[0042] The present invention provides a graded method for inspiring children's drawing based on multimodal behavior perception, and the specific implementation steps are as follows.

[0043] S1: The system initiates a multi-source sensing process. A high-definition camera scans the child's face and drawing area at a frequency of 60Hz, while the pressure-sensitive drawing board simultaneously activates coordinate capture mode. The microphone array is in standby mode, monitoring the child's voice feedback in real time. The posture sensor begins recording hand movement vectors.

[0044] S2: Perform multimodal feature fusion. The multimodal behavior perception and state assessment module timestamps the acquired image, pressure, speech, and inertial data. It uses deep learning algorithms to extract facial keypoint motion features, pen pressure gradients, and body sway amplitude.

[0045] S3: Generate a comprehensive state assessment vector The system's background processor calculates the components representing focus based on a pre-set mathematical model. Components representing emotional valence and components representing skill level These three components, after normalization, constitute a dynamically updated three-dimensional evaluation vector. .

[0046] S4: Perform hierarchical decision-making. The hierarchical heuristic decision engine will use vectors... Input the hierarchical decision function. Calculate the vector... Euclidean distance to the cluster centers of each capability level in the feature space The system determines whether the child is in the exploration, imitation, or creation phase. If the determination is in the transition phase, the reinforcement learning model is triggered to generate a hybrid strategy.

[0047] S5: Implement adaptive interactive presentation. The adaptive interactive presentation module receives decision instructions and allocates hardware resources according to different levels. During the exploration phase, the projection unit projects a highlighted outline; during the imitation phase, the projection unit executes a fade-out animation and the vibration unit provides micro-feedback; during the creation phase, the projection unit 8 generates a virtual complementary scene and the audio unit provides thematic inspiration.

[0048] S6: Real-time monitoring and dynamic correction. The system continuously loops through steps S1 to S5. If the emotional valence component of the child is detected... If a continuous decrease occurs, the system determines that the current guidance difficulty is too high or the interaction method is causing discomfort, and thus automatically reduces the difficulty level or adjusts the tone of the audio synthesis unit until the state vector... Returning to the ideal range achieves a closed-loop feedback of perception.

[0049] In a real-world application scenario, such as a smart art class in a kindergarten, a child begins to try drawing the sun. In stage S1, the system detects that the child's grip on the pen is unstable and their gaze frequently wanders. In stages S2 and S3, the multimodal behavior perception and state assessment module calculates... It is 0.4. The value is 0.3, at which point the emotional component is... A score of 0.8 indicates that the child is interested but has limited ability. In stage S4, the hierarchical heuristic decision engine determines that the child is in the exploratory phase.

[0050] Upon entering stage S5, the augmented reality projection unit immediately projects a bright yellow circular phantom image in the center of the drawing paper, with a flashing effect to attract attention. The intelligent audio synthesis unit provides audio prompts for the child to trace along the circle. As the child completes the drawing of the circle, their skill component... Increased to 0.6, focus level Increased to 0.7. In the next iteration, the hierarchical heuristic decision engine discovers vectors through computation. It has approached the cluster center of the imitation period, so the strategy is automatically switched.

[0051] In the imitation phase mode, the projection unit no longer displays a static circle, but instead shows a dynamic process of coloring a sun, with the color fading over time. When a child colors outside the lines, the tactile feedback vibration unit produces a slight vibration, reminding the child to reduce the range of their strokes. This immediate non-visual feedback effectively helps children develop a sense of spatial control. The confidence a child displays after successfully completing their imitation sun drawing enhances their emotional impact. Maintaining a high level, and skill content It reached 0.85.

[0052] The system then guides the creative phase. The augmented reality projection unit projects an image of a rainbow and clouds around the sun, but the children are not required to replicate it. The intelligent audio synthesis unit asks the sun if it needs a friend. At this point, the children use their imagination to draw a moon next to the sun. After recognizing the moon image, the system uses a generative adversarial network to add a starry background to the projected image, greatly enhancing the fun and sense of accomplishment in drawing.

[0053] The system performance of this invention has been optimized, with the end-to-end perception and interaction latency controlled within 150 milliseconds. The multimodal behavior perception and state assessment module employs multi-threaded parallel computing technology to ensure that visual feature extraction and pen pressure analysis do not interfere with each other. The clustering model of the hierarchical heuristic decision engine has been lightweighted and can run smoothly on embedded processors. The projection calibration algorithm of the adaptive interactive presentation module utilizes the paper pose fed back by a high-definition camera to achieve sub-millimeter-level projection alignment accuracy through an affine transformation matrix.

[0054] In addition to recording coordinates, the pressure-sensitive drawing board also analyzes pressure gradients to identify whether a child's pen-holding posture is correct. If the pressure distribution shows an abnormal bias, the intelligent audio synthesis unit will play an animated reminder to correct the pen-holding posture. The posture sensor analyzes the frequency of hand tremors to filter out natural shaking caused by the child's immature physiological development, ensuring the skill level component is accurate. The calculation results can truly reflect the artist's artistic intent rather than physiological limitations.

[0055] This invention overcomes the shortcomings of traditional art education products by deeply integrating multimodal information. Through the scientific division of hierarchical decision functions, it solves the problem of a one-size-fits-all approach to teaching content. By leveraging the synergistic effect of AR projection, haptic feedback, and emotional audio, it changes the current situation of singular interaction methods. In a smart education scenario, the system achieves a complete intelligent closed loop, from perceiving the child's state to making scientific decisions on levels, and then to adaptively guiding the child's creation, significantly improving the scientific rigor and effectiveness of early childhood art education.

[0056] In summary, this invention achieves precise perception and intelligent guidance of children's drawing process by constructing a tiered early childhood drawing inspiration system based on multimodal behavioral perception. The system utilizes a sensor array to collect multidimensional behavioral data in real time, quantifies children's states through a deep learning model, and implements dynamic ability grading based on a clustering algorithm. Through a combination of augmented reality projection, tactile feedback, and intelligent audio interaction, the system can provide personalized inspiration strategies for children at different developmental stages, thereby improving drawing skills while maximizing the protection and stimulation of children's creativity and interest in drawing.

Claims

1. A graded early childhood drawing inspiration system based on multimodal behavioral perception, characterized in that, It includes a multimodal behavior perception and state assessment module, a hierarchical heuristic decision engine, and an adaptive interactive presentation module; The multimodal behavior perception and state assessment module is connected to a sensor group via a hardware interface. The sensor group includes a high-definition camera set above the painting area, a pressure-sensitive drawing board embedded below the painting support surface, a microphone array set in front of the child, and a posture sensor worn on the child's wrist. The multimodal behavior perception and state assessment module is used to synchronously collect facial expression image sequences, pen stroke pressure trajectory data, voice signals and limb movement features of young children using a multi-threaded concurrency mechanism, and generate a comprehensive state assessment vector S through a multimodal fusion algorithm. The comprehensive state assessment vector S consists of a focus component α, an emotional valence component β and a skill level component γ. The hierarchical heuristic decision engine is used to receive the real-time updated comprehensive state evaluation vector S, and map the comprehensive state evaluation vector S to a preset ability level clustering space, calculate the Euclidean distance Di between the comprehensive state evaluation vector S and each cluster center point in the space, so as to determine the child's current drawing level. The adaptive interactive presentation module is used to drive interactive output devices, including augmented reality projection unit, haptic feedback vibration unit and intelligent audio synthesis unit, to execute differentiated presentation logic according to the decision instructions output by the hierarchical heuristic decision engine.

2. The graded early childhood drawing inspiration system based on multimodal behavioral perception as described in claim 1, characterized in that, The focus component α is calculated by taking the gaze point coordinates tracked by the high-definition camera and calculating the percentage of time the gaze area falls within the canvas area, and then weighted and summed in combination with the body sway frequency output by the posture sensor; the emotional valence component β is obtained by inputting facial micro-expression vectors and voice tone features into a bidirectional long short-term memory network; the skill level component γ is obtained by calculating the Hausdorff distance between the current pen stroke trajectory and the preset standard template, and then comprehensively evaluating it in combination with the uniformity of pen stroke pressure and the time efficiency of completing the composition per unit area.

3. A graded early childhood drawing inspiration system based on multimodal behavioral perception as described in claim 2, characterized in that... Further specifying, the calculation model for the attention component α is as follows: obtaining the child's gaze offset vector within a unit time window, and assessing the degree of visual attention concentration by calculating the cumulative distribution function of the vector; using the acceleration data collected by the posture sensor, extracting the swaying energy spectrum of the child's body in the low-frequency range of 0.5Hz to 2Hz, and using the reciprocal of the energy spectrum as a measure of posture stability; and fusing the visual attention score and the posture stability score through a linear regression equation.

4. The hierarchical early childhood drawing inspiration system based on multimodal behavioral perception as described in claim 1, characterized in that, The hierarchical heuristic decision engine internally constructs an ability level clustering space optimized based on the K-means++ algorithm. This space is preset with three cluster centers corresponding to the exploration period, imitation period, and creative period. The hierarchical heuristic decision engine is equipped with a logical judgment unit. When the minimum distance min(Di) is within the preset stable threshold range, the system automatically locks the child's current drawing level.

5. A graded early childhood drawing inspiration system based on multimodal behavioral perception as described in claim 4, characterized in that... Further specifying, when the comprehensive state evaluation vector S is in the overlapping area of ​​two cluster centers or all distances Di exceed the preset confidence threshold, the hierarchical heuristic decision engine determines the state as a level transition period; for children in the level transition period, the hierarchical heuristic decision engine calls the built-in Markov decision process model and uses the reinforcement learning algorithm of the deep Q network architecture to search online and determine the optimal heuristic strength and guidance mode.

6. A graded early childhood drawing inspiration system based on multimodal behavioral perception as described in claim 1, characterized in that, When a child is identified as being in the exploration phase, the adaptive interactive presentation module activates the augmented reality projection unit to overlay dynamic light and shadow outlines on the drawing paper, and plays onomatopoeic guidance in conjunction with the intelligent audio synthesis unit; when the child enters the imitation phase, the adaptive interactive presentation module controls the transparency of the projected content to linearly decrease from 100% to 20% over time, while the tactile feedback vibration unit generates vibration feedback of different frequencies according to the deviation of the pen stroke.

7. A graded early childhood drawing inspiration system based on multimodal behavioral perception as described in claim 1, characterized in that, For children identified as being in the creative stage, the adaptive interactive presentation module uses a generative adversarial network to generate a virtual background scene in real time based on the partial patterns drawn by the child, and integrates it into the physical environment surrounding the canvas through the augmented reality projection unit; the intelligent audio synthesis unit automatically adjusts the speech rate, pitch and energy of the synthesized speech through the built-in emotional speech synthesis engine according to the real-time value of the emotional valence component β.

8. A graded method for inspiring early childhood drawing based on multimodal behavioral perception, employing a graded method for inspiring early childhood drawing based on multimodal behavioral perception as described in any one of claims 1 to 7, characterized in that, Includes the following steps: S1: The system initializes and starts the multi-source sensor array, captures facial images of children through a high-definition camera, records the two-dimensional coordinates and pressure values ​​of the brushstrokes using a pressure-sensitive drawing board, and simultaneously turns on the microphone array and posture sensor to acquire environmental voice and body posture data. S2: Performs multimodal data synchronization and fusion processing, uses a unified clock source to label various sensor data, processes speech and tremor signals through fast Fourier transform, and uses a spatial transformation network to align and normalize facial images. S3: Construct a three-dimensional state evaluation vector, calculate the component α representing focus, the component β representing emotional state, and the component γ representing painting skill level, and encapsulate them into a comprehensive state evaluation vector S. S4: Execute intelligent hierarchical decision-making logic, map the generated comprehensive state evaluation vector S to the three-dimensional feature space, calculate its geometric distance with the three preset standard cluster centers of the exploration period, imitation period and creation period, and determine the teaching level or trigger the reinforcement learning mechanism based on the distance; S5: Implement adaptive heuristic guidance interaction, retrieve the corresponding interaction parameters from the heuristic strategy library according to the determined level, and perform differentiated guidance through augmented reality projection unit, haptic feedback vibration unit and intelligent audio synthesis unit; S6: Implement closed-loop feedback adjustment, continuously monitor the changing trend of the comprehensive state assessment vector S of children after receiving inspiration and guidance, and automatically adjust the guidance difficulty or change the interaction mode according to the feedback values ​​of the attention component α or the emotional valence component β.

Images