A child learning state multi-modal recognition method, device and medium

Abstract

The application discloses a kind of children learning state multimodal recognition method, equipment and medium, it is related to artificial intelligence technical field, comprising: obtaining visual key point feature sequence and acoustic feature sequence;Align and segment, obtain windowed visual features and windowed acoustic features;Respectively obtain corresponding time sequence visual feature representation and time sequence acoustic feature representation;Visual gating vector is generated based on time sequence acoustic feature representation, and acoustic gating vector is generated based on time sequence visual feature representation;Modulated visual feature is obtained by using visual gating vector modulation time sequence visual feature representation, and modulated acoustic feature is obtained by using acoustic gating vector modulation time sequence acoustic feature representation;Cross-modal fusion feature is generated based on modulated visual feature and modulated acoustic feature, input state inference unit carries out state inference, and obtains state vector.The application can be realized on low power NPU platform, and children learning state multimodal recognition with real-time, privacy security and robustness.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, device and medium for multimodal recognition of children's learning states. Background Technology

[0002] Currently, children's learning aids are increasingly integrating emotion recognition and attention analysis functions, aiming to dynamically adjust learning content or interaction methods based on the child's state. Existing technologies mainly fall into two categories. One is cloud-based audio and video analysis, which uploads images captured by a camera and audio captured by a microphone to a cloud server and uses large-scale deep learning models for state analysis. The other is on-device-based coarse-grained judgment based on single-modal information, such as blink frequency, voice energy, or simple gesture features, combined with rules or lightweight models.

[0003] However, the inventors of this application have found that the above-mentioned solutions have significant limitations in practical applications. Cloud-based analysis methods are highly dependent on network transmission stability, resulting in high overall processing latency and difficulty in responding in real time to rapidly changing states during children's learning process. Furthermore, uploading children's audio and video data to the cloud for processing poses significant privacy risks and compliance concerns, limiting its widespread deployment in home and school settings. Furthermore, while edge-side single-modal solutions avoid network dependence, their ability to represent complex learning states is limited, making them susceptible to interference from factors such as lighting changes, environmental noise, and individual differences, resulting in recognition accuracy and robustness that fail to meet practical application requirements. Furthermore, embedded children's learning devices are generally equipped with only low-powered NPUs, making it difficult to support the direct deployment of existing multimodal deep models based on raw images and raw speech on the edge. Summary of the Invention

[0004] This invention provides a method, device, and medium for multimodal recognition of children's learning states. The technical problem it aims to solve is how to implement a multimodal recognition scheme for children's learning states that combines real-time performance, privacy security, and robustness on a low-computing-power NPU platform.

[0005] In a first aspect, embodiments of the present invention provide a method for multimodal recognition of children's learning states, comprising:

[0006] A visual keypoint feature sequence and an acoustic feature sequence are obtained. The visual keypoint feature sequence is extracted from the video stream captured by the camera of the children's learning device, and the acoustic feature sequence is extracted from the audio stream captured by the microphone of the children's learning device. The visual keypoint feature sequence and the acoustic feature sequence are aligned in the time dimension and segmented using a sliding window to obtain aligned windowed visual features and windowed acoustic features. A temporal visual feature representation and a temporal acoustic feature representation of the windowed visual features are obtained. A visual gating vector and an acoustic gating vector are generated based on the temporal acoustic feature representation. The temporal visual feature representation is modulated using the visual gating vector to obtain modulated visual features, and the temporal acoustic feature representation is modulated using the acoustic gating vector to obtain modulated acoustic features. The modulated visual features and the modulated acoustic features are concatenated and linearly fused to generate cross-modal fusion features. The cross-modal fusion features are input into a state inference unit for state inference, and a state vector representing the child's learning state is output.

[0007] Optionally, aligning the visual keypoint feature sequence and the acoustic feature sequence in the time dimension includes:

[0008] When the frame rate of the visual keypoint feature sequence is inconsistent with the frame rate of the acoustic feature sequence, the acoustic feature sequence is interpolated and resampled based on the timestamp of the visual keypoint feature sequence; or the visual keypoint feature sequence is interpolated and resampled based on the timestamp of the acoustic feature sequence; or the visual keypoint feature sequence and the acoustic feature sequence are respectively input into a preset visual sub-network and an acoustic sub-network for temporal encoding, and then the encoded visual features and acoustic features are aggregated and aligned based on the preset timestamp to obtain a temporally aligned visual keypoint feature sequence and acoustic feature sequence.

[0009] Optionally, the temporal visual feature representation H v Through formula H v = PWConv1D(σ(DWConv1D( E v )))Calculated, where E v = X v · W v + b v This is an embedding representation of visual features after linear projection. Xv For the windowed visual features, W v and b v Here, σ(·) represents the weight matrix and bias vector of the linear projection of visual features, respectively; DWConv1D(·) is a one-dimensional depthwise convolution operation along the time dimension; σ(·) is a non-linear activation function; and PWConv1D(·) is a one-dimensional pointwise convolution operation.

[0010] The temporal acoustic feature representation H a Through formula H a = PWConv1D(σ(DWConv1D( E a The calculation yielded, where E a = X a · W a + b a This is the embedded representation of acoustic features after linear projection. X a For the windowed acoustic features, W a and b a These are the weight matrix and bias vector for the linear projection of the acoustic features, respectively.

[0011] Optionally, the visual gating vector g v Through formula g v = σ( H a · W gv + b gv The calculation yielded, where H a This represents the temporal acoustic features. W gv and b gv These are the weight matrix and bias vector for acoustically guided visual gating, respectively, where σ(·) is the Sigmoid activation function; the acoustic gating vector g a Through formula g a = σ( H v · Wga + b ga The calculation yielded, where H v This is a representation of the temporal visual features. W ga and b ga These are the weight matrix and bias vector for visually guided acoustic gating, respectively.

[0012] Optionally, the modulated visual features Through formula The calculation yielded, where ⊙ represents element-wise multiplication; the modulated acoustic features Through formula Calculated;

[0013] The cross-modal fusion features H fuse Through formula The calculation yields the result, where Concat(·) is the feature concatenation operation. W f and b f These are the weight matrix and bias vector for the cross-modal fusion mapping, respectively.

[0014] Optionally, the method is executed on an NPU. When matrix multiplication operations are involved in the linear projection, linear fusion, state reasoning, and generation of the visual and acoustic gate vectors, if the size of the weight matrix involved in the operation exceeds the capacity of the NPU's on-chip cache, the method includes the following steps:

[0015] The weight matrix is ​​divided into multiple weight sub-blocks by columns, and the size of each weight sub-block does not exceed a preset cache capacity threshold;

[0016] Each weight sub-block is loaded sequentially into the on-chip cache of the NPU. The currently loaded weight sub-block and the input feature matrix are used to calculate a partial output result, and the partial output result is stored in external memory or the on-chip cache of the NPU.

[0017] After all weight sub-blocks have been calculated, the output results of all parts are concatenated to obtain the final output matrix.

[0018] Optionally, in the processes of linear projection, linear fusion, state reasoning, and generation of the visual gating vector and acoustic gating vector, when sequentially performed linear transformations, nonlinear activations, and normalization operations are involved, the method includes:

[0019] During the model compilation phase, the sequentially executed linear transformations, nonlinear activations, and normalization operations are merged into a single computation graph node.

[0020] During NPU runtime, the merged computation graph nodes complete all operations in a single data loading process and retain intermediate computation results in the NPU's on-chip cache without needing to write back to external memory.

[0021] Optionally, after outputting the state vector, the method further includes: obtaining the state vector of the current time window. s t and historical smooth state vector Using the exponential moving average formula The current smooth state vector is calculated. Where α is the smoothing factor; determine the current smoothing state vector. The value of the target dimension corresponding to the preset learning state type is checked against a preset threshold condition for the target dimension within N consecutive time windows, where N is greater than or equal to 3. If the condition is met, a learning intervention behavior corresponding to the dimension is triggered, including adjusting the difficulty of the learning content, providing voice prompts, or triggering a rest reminder. If the condition is not met, the state vector of the current time window is returned. s t The process continues by observing the state vectors of subsequent time windows.

[0022] Secondly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0023] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0024] This invention provides a method, device, and medium for multimodal recognition of children's learning states. The method includes: acquiring a visual keypoint feature sequence and an acoustic feature sequence, wherein the visual keypoint feature sequence is extracted from a video stream captured by a camera of a children's learning device, and the acoustic feature sequence is extracted from an audio stream captured by a microphone of the children's learning device; aligning the visual keypoint feature sequence and the acoustic feature sequence in a time dimension, and performing segmentation processing using a sliding window to obtain aligned windowed visual features and windowed acoustic features; and acquiring a temporal visual feature representation of the windowed visual features and a temporal acoustic feature table of the windowed acoustic features. The invention demonstrates the following: A visual gating vector is generated based on the temporal acoustic feature representation, and an acoustic gating vector is generated based on the temporal visual feature representation. The temporal visual feature representation is modulated using the visual gating vector to obtain modulated visual features, and the temporal acoustic feature representation is modulated using the acoustic gating vector to obtain modulated acoustic features. The modulated visual features and the modulated acoustic features are concatenated and linearly fused to generate cross-modal fusion features. The cross-modal fusion features are input to a state inference unit for state inference, and a state vector representing the child's learning state is output. This invention replaces the transmission and processing of original image and audio data by extracting visual key point feature sequences from video streams and acoustic feature sequences from audio streams as inputs. This significantly reduces the data dimensionality and computational load, avoids the risk of leakage of children's original audio and video data, and protects privacy from the source. Through alignment and sliding window segmentation, the temporal synergy of multimodal data is ensured, providing a structured input for real-time, continuous edge analysis. By separately acquiring temporal feature representations and modulating these representations using cross-generated visual and acoustic gating vectors, a dynamic and complementary enhancement of visual and auditory information is achieved. This effectively suppresses unimodal noise and focuses on state-related cues. Finally, the modulated features are concatenated, linearly fused, and state vectors are generated through state inference. This enables real-time and accurate quantitative identification of children's multi-dimensional learning states, such as attention and frustration, on embedded devices, providing a reliable basis for subsequent adaptive learning interventions. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a method for recognizing multimodal learning states in children according to an embodiment of the present invention.

[0027] Figure 2 This is another flowchart illustrating a method for multimodal recognition of children's learning states provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the state rule determination process of a multimodal recognition method for children's learning states provided in an embodiment of the present invention;

[0029] Figure 4 This is a flowchart illustrating the process of weight block loading, partial output calculation and splicing, and operator chain fusion execution in a multimodal recognition method for children's learning states provided in an embodiment of the present invention.

[0030] Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It should be understood that, when used in this specification and the appended claims, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0033] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0034] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0035] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0036] Please see Figure 1 and combined Figure 2 This invention provides a method for multimodal recognition of children's learning states, the method comprising:

[0037] S1, obtain the visual key point feature sequence and the acoustic feature sequence. The visual key point feature sequence is extracted from the video stream captured by the camera of the children's learning device, and the acoustic feature sequence is extracted from the audio stream captured by the microphone of the children's learning device.

[0038] In practice, a video stream containing the child's face or upper body is captured in real time using a camera integrated into the child's learning device. From each frame of the video stream, a set of coordinate data representing key points indicating human posture or facial expressions is extracted. These key points include, for example, the corners of the eyes, the corners of the mouth, and the shoulders, etc., which are not specifically limited in this invention. These key point coordinate data arranged in chronological order are combined to form a visual key point feature sequence.

[0039] Furthermore, an audio stream from the environment is acquired in real time via a microphone integrated into the same children's learning device. After preprocessing the audio stream, such as framing and windowing, acoustic features of each frame are extracted, such as low-dimensional features like Mel-frequency cepstral coefficients and zero-crossing rates, forming an acoustic feature sequence. This invention does not specifically limit the specific features extracted. The visual keypoint feature sequence and the acoustic feature sequence are two different modalities of raw feature data with temporal markers.

[0040] S2, align the visual key point feature sequence and the acoustic feature sequence in the time dimension, and perform segmentation processing through a sliding window to obtain the aligned windowed visual features and windowed acoustic features.

[0041] In practice, since the video frame rate and audio frame rate may be different, it is necessary to time-align the visual key point feature sequence and the acoustic feature sequence so that the visual features and acoustic features at the same timestamp correspond to the child's state at the same moment.

[0042] Furthermore, after alignment, a sliding window of fixed length and fixed step size is used to truncate the aligned bimodal feature sequence along the time axis, resulting in a series of temporally aligned windowed visual and acoustic features. Each window contains a segment of visual and acoustic data within a continuous time period.

[0043] In some preferred embodiments, aligning the visual keypoint feature sequence and the acoustic feature sequence in the time dimension includes: when the frame rate of the visual keypoint feature sequence is inconsistent with the frame rate of the acoustic feature sequence, interpolating and resampling the acoustic feature sequence based on the timestamp of the visual keypoint feature sequence; or interpolating and resampling the visual keypoint feature sequence based on the timestamp of the acoustic feature sequence; or inputting the visual keypoint feature sequence and the acoustic feature sequence into preset visual subnetworks and acoustic subnetworks respectively for temporal encoding, and then aggregating and aligning the encoded visual features and acoustic features based on preset timestamps to obtain a visual keypoint feature sequence and an acoustic feature sequence aligned in the time dimension.

[0044] In practice, a time alignment step is performed, with the input being the aforementioned visual keypoint feature sequence and acoustic feature sequence. The goal of alignment is to establish a strict correspondence between visual frames and acoustic frames on the time axis. Several implementation methods are available.

[0045] The first approach uses the timestamps of the visual keypoint feature sequence as a reference. This means mapping the timeline of the acoustic feature sequence to each timestamp of the visual feature sequence through interpolation and resampling. For example, for each visual frame's time point, two adjacent acoustic frames are found on the timeline of the acoustic feature sequence, and the acoustic feature values ​​at that moment are calculated through linear interpolation, thereby generating a new acoustic feature sequence as output that is completely consistent with the visual sequence's timestamps.

[0046] Furthermore, the second method is symmetrical to the first method, that is, using the timestamp of the acoustic feature sequence as a reference, the visual key point feature sequence is interpolated and resampled to output a visual feature sequence that is consistent with the timestamp of the acoustic sequence. This invention will not elaborate on this.

[0047] Furthermore, in the third approach, the visual keypoint feature sequence is first input into a pre-defined visual sub-network for temporal encoding to obtain encoded visual features; simultaneously, the acoustic feature sequence is input into a pre-defined acoustic sub-network for temporal encoding to obtain encoded acoustic features. The visual and acoustic sub-networks are used to extract high-level temporal representations of their respective modalities. Then, using a pre-defined unified timestamp sequence as a reference, aggregation and alignment operations are performed on the encoded visual and acoustic features, for example, by interpolation to map them onto this unified timeline. The final output is a visual keypoint feature sequence and an acoustic feature sequence that are strictly aligned in the temporal dimension.

[0048] Furthermore, after time alignment is completed, the aligned bimodal feature sequence is used as input to perform the segmentation process using a sliding window as described in the above embodiment, thereby obtaining windowed visual features and windowed acoustic features, and then continuing the subsequent process.

[0049] This embodiment provides a flexible and optional solution to the inherent time synchronization problem of multimodal data. Two interpolation-based resampling methods are computationally direct, quickly unifying feature sequences with different sampling rates onto a common time reference. This eliminates feature misalignment caused by hardware sampling differences, providing a time-consistent data foundation for subsequent windowing processing and fusion calculations, ensuring accurate analysis of the temporal correlation between visual behavior and speech events. Furthermore, a third method, using sub-network encoding followed by alignment, provides a higher level of time alignment capability. The sub-network first abstracts and smooths the original time sequences of each modality, aligning them in a more robust feature space. This partially compensates for potential time jitter or frame drops in the original feature sequences, and the preset unified time reference provides greater flexibility in system design, facilitating integration with downstream fixed-window mechanisms. These alignment strategies collectively ensure the effectiveness of multimodal information fusion, avoiding information confusion caused by time asynchrony, and are a crucial prerequisite for achieving accurate time-series state analysis on embedded devices.

[0050] S3, obtain the temporal visual feature representation of the windowed visual features and the temporal acoustic feature representation of the windowed acoustic features.

[0051] In the specific implementation, temporal feature extraction is performed on both windowed visual features and windowed acoustic features. For windowed visual features, a learnable linear projection layer is first used to map them to a higher-dimensional or more suitable feature space, obtaining an embedded representation of the visual features. Further, a one-dimensional deep convolution operation is performed on this embedded representation along the temporal dimension to capture the local dynamic patterns of the visual features in the temporal dimension, such as the trajectory of head movements. Further, a non-linear activation function is applied to the output of the deep convolution to introduce the non-linear expressive power of the model. Finally, a one-dimensional pointwise convolution operation is used to fuse and transform information from different feature channels, outputting the final temporal visual feature representation.

[0052] For windowed acoustic features, a completely independent but structurally identical processing flow is adopted: first, an acoustic embedding representation is obtained through linear projection, then a one-dimensional deep convolution is used to capture the short-term temporal pattern of the acoustic features (such as pitch changes), then a nonlinear activation function is used, and finally a one-dimensional pointwise convolution is used to output the temporal acoustic feature representation. This will not be elaborated further in this invention.

[0053] In some preferred embodiments, the temporal visual feature representation H v Through formula H v = PWConv1D(σ(DWConv1D( E v The calculation yielded, where E v = X v · W v + b v This is an embedding representation of visual features after linear projection. X v For the windowed visual features, W v and b v Here, σ and PWConv1D are the weight matrix and bias vector for the linear projection of visual features, respectively. DWConv1D(·) is a one-dimensional depthwise convolution operation along the time dimension, σ(·) is a non-linear activation function, and PWConv1D(·) is a one-dimensional pointwise convolution operation.

[0054] The temporal acoustic feature representation H a Through formula H a = PWConv1D(σ(DWConv1D( E a The calculation yielded, whereE a = X a · W a + b a This is the embedded representation of acoustic features after linear projection. X a For the windowed acoustic features, W a and b a These are the weight matrix and bias vector for the linear projection of the acoustic features, respectively.

[0055] In practice, the temporal visual feature representation is obtained. The input to this step is windowed visual features. The implementation involves a sequential computation process. First, linear projection: the windowed visual features are multiplied by a learnable weight matrix and a learnable bias vector is added. The purpose of this operation is to map the original keypoint coordinate features to a new, potentially higher-dimensional or more expressive feature space, obtaining an embedded representation of the visual features. Next, one-dimensional deep convolution: the above embedded representation is subjected to a one-dimensional deep convolution operation along the temporal dimension. The characteristic of deep convolution is that each convolutional kernel is responsible for processing only one input feature channel, sliding independently along the time axis, thus efficiently capturing the local temporal patterns and dependencies of each visual feature channel, such as the movement trajectory of a keypoint in consecutive frames. Next, nonlinear activation: a nonlinear activation function, such as a rectified linear unit function, is applied to the result of the one-dimensional deep convolution operation. This operation introduces nonlinear transformation capabilities, enabling the model to learn and express more complex feature patterns. Finally, one-dimensional pointwise convolution: the activated result is subjected to a one-dimensional pointwise convolution operation. Pointwise convolution uses a 1x1 convolution kernel. Its core function is to linearly combine and fuse information between different feature channels, and it can flexibly adjust the number of output feature channels, ultimately outputting a temporal visual feature representation.

[0056] Further, a temporal acoustic feature representation is obtained. The input to this step is a windowed acoustic feature. Its processing flow is completely symmetrical to the acquisition of the temporal visual feature representation, but the parameters are independent. Similarly, an acoustic embedding representation is obtained through linear projection, and then a one-dimensional depthwise convolution is used to capture the time-varying pattern of each acoustic feature dimension (such as different Mel frequency band energies). Then, a nonlinear activation function is used, and finally, a one-dimensional pointwise convolution is used to integrate the channel information to output the temporal acoustic feature representation. This process will not be described in detail in this invention.

[0057] Furthermore, the obtained temporal visual feature representation and temporal acoustic feature representation will be used as inputs for subsequent cross-modal interaction steps.

[0058] This embodiment employs a meticulously designed lightweight temporal network structure to extract representations rich in temporal context information from visual and acoustic window features. Linear projection, as a feature preprocessing step, transforms the data into a space more suitable for deep learning models. Furthermore, one-dimensional deep convolution is a key component, focusing on uncovering the local evolution patterns of each feature dimension along the time axis with extremely low parameter counts. For example, changes in the vertical coordinate of the brow can reflect nodding frequency, and changes in speech energy in specific frequency bands can reflect intonation fluctuations, which are crucial for understanding continuous behavioral and speech patterns. Non-linear activation functions break the limitations of linear models, enhancing their expressive power. Further, one-dimensional pointwise convolution plays a role in feature compression and integration, fusing the temporal patterns extracted from each channel by deep convolution across channels to generate a compact and information-rich temporal feature representation. This processing chain, which progresses from simple to complex, gradually extracts high-level, time-sensitive features from the original windowed data, providing a high-quality single-modal feature foundation for subsequent accurate cross-modal attention modulation. At the same time, its lightweight design also meets the deployment requirements of low-computing-power NPUs.

[0059] S4, generate a visual gating vector based on the temporal acoustic feature representation, and generate an acoustic gating vector based on the temporal visual feature representation.

[0060] In practice, cross-modal interaction is achieved. Based on temporal acoustic feature representation, a visual gating vector is generated through a linear transformation layer and a sigmoid activation function. Each element of this vector has a value between 0 and 1, determined by the acoustic features, and is used to evaluate the importance of each dimension of the visual features. Furthermore, based on the temporal visual feature representation, another linear transformation layer and a sigmoid activation function are used to generate an acoustic gating vector, which is then used to evaluate the importance of each dimension of the acoustic features. This enables intermodal information guidance.

[0061] In some preferred embodiments, the visual gating vector g v Through formula g v = σ( H a · W gv + b gv ) was calculated to obtain, where H a This represents the temporal acoustic features. W gv and b gv σ and σ(·) are the weight matrix and bias vector of the acoustically guided visual gating, respectively, and σ(·) is the Sigmoid activation function.

[0062] The acoustic gating vector g a Through formula g a = σ( H v · W ga + b ga The calculation yielded, where H v This is a representation of the temporal visual features. W ga and b ga These are the weight matrix and bias vector for visually guided acoustic gating, respectively.

[0063] In practice, a visual gating vector is generated. The input to this step is the temporal acoustic feature representation. Specifically, the temporal acoustic feature representation is input into a linear transformation layer, which is multiplied by a learnable weight matrix and then added to it with a learnable bias vector. The result of the linear transformation is then input into a sigmoid activation function. The sigmoid function maps each input value to a value between 0 and 1, and its output is the visual gating vector. The dimension of this vector corresponds to the temporal visual feature representation to be modulated, and the value at each position is a scalar weight between 0 and 1, dynamically determined by the current acoustic feature information.

[0064] Further, an acoustic gating vector is generated. The input to this step is a temporal visual feature representation. Specifically, the temporal visual feature representation is input into another independent linear transformation layer, and the result is passed through a sigmoid activation function to output an acoustic gating vector. The dimension of this vector corresponds to the temporal acoustic feature representation to be modulated, and the value at each position is also a weight dynamically generated from the current visual feature information.

[0065] Furthermore, the generated visual gating vector and acoustic gating vector will be used as control signal inputs for the modulation step.

[0066] This embodiment implements a lightweight and efficient cross-modal interaction mechanism. Its core is to generate feature selection weights (gating vectors) for the current modality based on information from another modality. A linear transform layer learns how to extract key information from the temporal features of the other modality to guide feature selection for the current modality. The Sigmoid activation function then transforms this guidance information into specific, interpretable attention weights. Values ​​closer to 1 indicate that the corresponding modal feature is more important and should be preserved or enhanced; values ​​closer to 0 indicate that the feature may be irrelevant or noise and should be suppressed. For example, when the system detects rapid breathing or a frustrated tone in acoustic features, the generated visual gating vector may assign high weights to facial key feature channels reflecting frowning or downturned corners of the mouth. Conversely, when visual features detect frequent blinking or wandering gaze in children, the generated acoustic gating vector may suppress background music channels while enhancing the weights of speech pauses or sighs. This design transforms the visual and acoustic modal analysis processes from isolated processes into a mutually guiding and reinforcing synergistic relationship, significantly enhancing the model's ability to focus on state-related cues in complex real-world scenarios.

[0067] S5, the temporal visual feature representation is modulated using the visual gating vector to obtain modulated visual features, and the temporal acoustic feature representation is modulated using the acoustic gating vector to obtain modulated acoustic features.

[0068] In practice, the generated visual gating vector is used to modulate the temporal visual feature representation. Specifically, the temporal visual feature representation is multiplied element-wise by the visual gating vector. Positions with high values ​​in the visual gating vector enhance the corresponding visual features, while positions with low values ​​suppress them, thus obtaining modulated visual features that emphasize the acoustically relevant parts. Furthermore, the temporal acoustic feature representation is multiplied element-wise by the acoustic gating vector to obtain the modulated acoustic features.

[0069] In some preferred embodiments, the modulated visual features Through formula The calculation yielded, where ⊙ represents element-wise multiplication; the modulated acoustic features Through formula Calculated.

[0070] In practice, visual feature modulation is performed. The inputs to this step are a temporal visual feature representation and a visual gating vector. The specific operation is element-wise multiplication, where each element in the temporal visual feature representation is multiplied by the corresponding element in the visual gating vector. Since the gating vector's value range is [0,1], this operation is equivalent to rescaling the visual features using a set of weights dynamically determined by acoustic information, thereby outputting modulated visual features. Feature values ​​are either enhanced or weakened.

[0071] Next, acoustic feature modulation is performed. The inputs to this step are the temporal acoustic feature representation and the acoustic gating vector. The specific operation is also element-wise multiplication, and the output is the modulated acoustic feature, which will not be described in detail in this invention.

[0072] S6, the modulated visual features and the modulated acoustic features are spliced ​​and linearly fused to generate cross-modal fusion features.

[0073] In practice, feature fusion is performed. The modulated visual features and modulated acoustic features are concatenated along the feature dimension to form a joint feature vector that incorporates bimodal information. Then, a linear fusion transformation is applied to this joint feature vector, which involves multiplying it by a learnable weight matrix and adding a bias vector. This maps the concatenated high-dimensional features to a vector with moderate dimensionality and more complete information fusion, i.e., cross-modal fused features.

[0074] In some preferred embodiments, the cross-modal fusion feature H fuse Through formula The calculation yields the result, where Concat(·) is the feature concatenation operation. W f and b f These are the weight matrix and bias vector for the cross-modal fusion mapping, respectively.

[0075] In practice, feature concatenation and linear fusion are performed. The inputs to this step are the modulated visual features and modulated acoustic features. First, a concatenation operation is performed, which connects the modulated visual and acoustic features along the feature dimension to form a longer-dimensional joint feature vector. This operation preserves all information from both modalities. Next, a linear fusion transformation is performed on the concatenated joint feature vector, which is multiplied by a learnable weight matrix and supplemented with a learnable bias vector. This linear transformation globally mixes and compresses the information from the two modalities, learning an optimal fusion method, and ultimately outputting a unified feature representation that integrates visual and acoustic information, i.e., a cross-modal fusion feature. This cross-modal fusion feature will serve as the input to the state inference step.

[0076] This embodiment completes the key steps from feature selection to information integration. The modulation operation achieves fine-grained, dynamic feature selection through element-wise multiplication. It directly applies the soft attention weights generated by the gating vector to the original features, making the computation efficient and intuitive. After modulation, features in each modality that are related to the information of the other modality and contribute significantly to state judgment are amplified, while irrelevant or contradictory features are attenuated. This is essentially a purification and enhancement of the original features based on cross-modal context. The concatenation operation simply merges the purified bimodal features in terms of dimension, ensuring that all information is preserved and providing raw materials for deep fusion. The subsequent linear fusion transformation is the core of information integration. Through the learned weight matrix, it automatically explores and establishes the intrinsic correlation and contribution ratio between visual and acoustic features, projecting the concatenated high-dimensional vector into a new, usually lower-dimensional but higher-information-density subspace. The features in this subspace, i.e., the cross-modal fusion features, are no longer a simple juxtaposition of visual and acoustic features, but rather a new, more discriminative representation generated by the collaborative selection and weighted combination of the two, thereby improving accuracy.

[0077] S7, input the cross-modal fusion feature into the state reasoning unit for state reasoning, and output a state vector to represent the child's learning state.

[0078] In practice, cross-modal fused features are input into the state inference unit. The state inference unit typically consists of one or more fully connected layers, whose function is to perform nonlinear transformations and mappings on the fused features. The output of the state inference unit is a fixed-dimensional state vector, where each dimension corresponds to a predefined learning state attribute, and its numerical value represents the strength of that state. For example, the three dimensions of a three-dimensional state vector can respectively quantify a child's level of focus, level of frustration, and level of fatigue.

[0079] This embodiment reduces computational load and privacy risks by extracting key points and acoustic features to replace the original audio and video, thus reducing data volume and privacy risks at the data source. Furthermore, time alignment and windowing ensure consistency of multimodal data across the analysis period. Furthermore, using temporal networks to extract visual and acoustic feature representations effectively captures the dynamic change patterns of each modality. Furthermore, cross-generating gating vectors enables attention guidance between modalities; auditory information guides which visual cues to focus on, and visual information guides which acoustic features to focus on, enhancing intermodal complementarity. Furthermore, modulating features using gating vectors essentially performs dynamic feature selection and enhancement, suppressing noise and highlighting state-related information. Furthermore, concatenating and linearly fusing the modulated bimodal features achieves deep information fusion at the feature level. Finally, the state inference unit maps the fused high-level abstract features into quantifiable state vectors, allowing abstract learning states such as distraction, frustration, and fatigue to be continuously and objectively measured, providing accurate and real-time decision-making basis for subsequent personalized learning interventions.

[0080] In some preferred embodiments, the state vector has a dimension K equal to 3, with the three dimensions representing the child's attention or focus level, frustration or negative experience level, and arousal or fatigue level, respectively.

[0081] In practice, the state vector is designed to have a fixed dimension K, where K equals 3. This means that the state vector is a three-dimensional real-valued vector, where each dimension independently and quantitatively represents a different psychological or physiological attribute of the child's learning state.

[0082] Specifically, the first dimension's value represents the child's level of attention or focus. A higher value generally indicates that the child is concentrating their cognitive resources on the learning task within the current time window, and that attention-related behaviors such as eye tracking and stable body posture are more pronounced; a lower value indicates that the child may be distracted or inattentive.

[0083] Furthermore, the second dimension's value is used to represent the degree of frustration or negative experience a child may have. The higher the value, the stronger the negative emotions such as confusion, irritability, and frustration that the child may experience during learning, which may be related to tasks that are too difficult or repeated mistakes; the lower the value, the more peaceful or positive the emotions are.

[0084] Furthermore, the third dimension represents the child's arousal level or fatigue level. A higher value may reflect a higher physiological arousal state, such as excitement; a consistently low value is usually associated with fatigue, drowsiness, or lethargy. This three-dimensional state vector constitutes a compact and comprehensive quantitative description of the child's learning state, serving as the input for the subsequent temporal smoothing module and the direct basis for ultimately triggering differentiated learning intervention behaviors.

[0085] This embodiment clearly defines the three-dimensional structure and semantics of the state vector, transforming the abstract learning state into concrete, measurable, and operable indicators. The attention dimension is directly related to the core efficiency of the learning process and is key to assessing cognitive engagement; the frustration dimension is related to emotional experience and is crucial for maintaining learning motivation and achieving positive education; the arousal dimension is related to physiological and energy state and is the foundation for ensuring sustainable learning. These three dimensions cover the three core aspects affecting learning outcomes—cognitive, emotional, and physiological—providing a relatively complete state profile. Furthermore, this clear dimensional division makes the state output highly interpretable, facilitating educational experts or system developers to set differentiated thresholds and design precise intervention strategies for each dimension. For example, the system can distinguish between attention decline due to fatigue and frustrated attention loss due to overly difficult content, thus taking different intervention measures such as suggesting rest or adjusting content strategies. This provides refined, multi-faceted state-aware input for achieving a truly personalized, adaptive learning support system, enabling the technology to better align with the complex realities of children's learning.

[0086] See Figure 3 This is a schematic diagram of the state rule determination process in one embodiment of the present invention. In some preferred embodiments, after outputting the state vector, the method further includes: obtaining the state vector of the current time window. s t and historical smooth state vector Using the exponential moving average formula The current smooth state vector is calculated. Where α is the smoothing factor; determine the current smoothing state vector. The value of the target dimension corresponding to the preset learning state type is checked against a preset threshold condition for the target dimension within N consecutive time windows, where N is greater than or equal to 3. If the condition is met, a learning intervention behavior corresponding to the dimension is triggered, including adjusting the difficulty of the learning content, providing voice prompts, or triggering a rest reminder. If the condition is not met, the state vector of the current time window is returned. s t The process continues by observing the state vectors of subsequent time windows.

[0087] In practice, after obtaining the state vector of the current time window, time-series smoothing and state confirmation processing are performed. The inputs to this step include: the state vector of the current time window, and the stored historical smoothed state vector obtained after the processing of the previous time window. Initially, the historical smoothed state vector can be initialized as a zero vector or the first window state vector.

[0088] Further, the current smoothed state vector is calculated. Specifically, this is done using the exponential moving average formula. The current smooth state vector is calculated. The smoothing factor is α. The current smoothed state vector is obtained by multiplying the current time window's state vector by the smoothing factor and then adding the difference between the historical smoothed state vector multiplied by the smoothing factor. The smoothing factor is a configurable parameter between 0 and 1, used to balance the weights of the current observation and the historical trend. In this embodiment, the preferred value range for α is [0.2, 0.5]. A larger smoothing factor makes the smoothed result more closely follow the current observation, while a smaller smoothing factor makes the result more stable and less sensitive to noise.

[0089] Further, a continuity check is performed. For the current smoothed state vector, a target dimension corresponding to a preset learning state type is selected. The value of this target dimension is checked to see if it meets a preset threshold condition across N consecutive time windows, including the current window. Here, N is an integer greater than or equal to 3. For example, if the preset state is "high frustration," the threshold condition for the second dimension of the corresponding state vector is "greater than 0.7." The check logic is to trace back the smoothed state vectors of the current and previous N-1 windows and check if the values ​​of the second dimension of all these vectors are greater than 0.7.

[0090] Furthermore, if the judgment result is satisfied, that is, if N consecutive windows meet the threshold condition, then the preset learning intervention behavior corresponding to this learning state type is triggered. The intervention behavior includes, but is not limited to: automatically adjusting the difficulty level of subsequent learning content, playing encouraging or prompting words through the device's speech synthesis module, or popping up a reminder interface suggesting a break on the device screen.

[0091] Furthermore, if the judgment result is not satisfied, no intervention is triggered. The system slides the window to the next time period, returns to the step of obtaining the state vector of the next time window, and continues to observe and smooth the subsequent states.

[0092] This embodiment, building upon instantaneous state recognition, adds a temporal-level smoothing and confirmation mechanism, significantly improving the stability of the system output and the rationality of decisions. State vectors obtained through direct inference are susceptible to fluctuations caused by instantaneous disturbances. Smoothing via exponential moving average effectively filters out random noise and instantaneous outliers, resulting in a smoother estimate that better reflects the slow trend of state changes, making the state curve smoother and more reliable. Furthermore, simple single-point threshold triggering is prone to false alarms due to accidental model misjudgments or brief actions by the child, leading to frequent and unnecessary interventions that interfere with the learning process. By introducing a continuity judgment rule, intervention is only triggered when the target state continuously exceeds the threshold for multiple consecutive time windows. This constitutes a state confirmation mechanism. For example, a child's accidental frown will not trigger a system response; only when the system continuously detects high frustration signals for multiple windows is it confirmed that the child is indeed in a persistent predicament, at which point triggering an intervention to reduce difficulty is appropriate and effective. This design significantly reduces the false alarm rate, making the system's interactive behavior more stable, intelligent, and human-like, enhancing children's trust and comfort when interacting with the learning device.

[0093] In some preferred embodiments, the method is executed on an NPU. When matrix multiplication is involved in the linear projection, linear fusion, state reasoning, and generation of the visual and acoustic gating vectors, if the size of the weight matrix involved in the calculation exceeds the capacity of the NPU's on-chip cache, the method includes the following steps: dividing the weight matrix into multiple weight sub-blocks by columns, with the size of each weight sub-block not exceeding a preset cache capacity threshold; sequentially loading each weight sub-block into the NPU's on-chip cache; calculating a partial output result using the currently loaded weight sub-block and the input feature matrix; and storing the partial output result in external memory or the NPU's on-chip cache; after all weight sub-blocks have been calculated, concatenating all partial output results to obtain the final output matrix.

[0094] For specific implementation details, please refer to... Figure 4 This is a schematic flowchart illustrating the weight block loading, partial output calculation and splicing, and operator chain fusion execution in one embodiment of the present invention. The method of the present invention is executed on an NPU. During the execution of the linear projection, the linear fusion, the state reasoning, and the generation of the visual gating vector and the acoustic gating vector, matrix multiplication operations of the form Y = X · W are involved, where X is the input feature matrix, W is the weight matrix, and Y is the output feature matrix.

[0095] When the size of the weight matrix W involved in the calculation exceeds the capacity of the NPU's on-chip cache, a weight-block loading calculation strategy is adopted. The specific process is as follows:

[0096] First, before performing the calculation, the weight matrix W is divided into multiple consecutive weight sub-blocks {W1, W2, ..., W...} along the column direction. m}, each weighted sub-block W i Its size is designed to ensure that it can be fully loaded into the NPU's on-chip cache, thus meeting the preset cache capacity threshold.

[0097] Furthermore, a loop processing method is used when the NPU performs calculations. Each weighted sub-block W is processed sequentially. i Loaded from external memory to the NPU's on-chip cache. For the currently loaded weighted sub-block W i The NPU performs matrix multiplication with the complete input feature matrix X to calculate a partial output result Y. i = X · W i The output result for this part is Y. i It can be temporarily stored in the NPU's on-chip cache, or written back to external memory according to the cache policy.

[0098] Furthermore, the process of loading weighted sub-blocks and calculating partial output results is repeated iteratively, i.e., when i = i+1, the next weighted sub-block W is loaded. i+1 Calculate Y i+1 = X · W i+1 Until all weighted sub-blocks {W1, W2, ..., W m All have been processed.

[0099] Finally, output all the calculated partial results {Y1, Y2, ..., Y}. m Following the original column order, these columns are concatenated along the output feature dimension to form the complete, final output matrix Y = Concat(Y1,Y2, ..., Y...). m ).

[0100] In this way, even if the model layer requires a large weight matrix W, it can be efficiently computed on an NPU with limited cache, avoiding frequent off-chip data transfer caused by an excessively large weight matrix and significantly improving computational efficiency.

[0101] This embodiment effectively overcomes the memory wall bottleneck encountered when deploying large models on embedded NPUs with limited computing resources by introducing a weight matrix block computation technique. The NPU's on-chip cache capacity is limited and cannot accommodate large weight matrices at once. Forcing computation would lead to frequent data swapping between slow external storage and the cache, causing significant latency and energy consumption. By dividing the large matrix into columns and ensuring that the size of each sub-block is adapted to the cache capacity, all weight data required for each core computation can reside in the cache. This greatly improves data reuse and minimizes dependence on external storage bandwidth. Furthermore, although the overall computational load remains unchanged, the overall computational efficiency and energy efficiency are significantly improved by reducing the overhead of significant off-chip data movement. This technology enables multimodal fusion layers and fully connected layers with strong expressive power and therefore many parameters to be deployed in low-computing-power, low-power embedded learning devices, thereby achieving more accurate and complex recognition of children's learning states on the edge without relying on the cloud.

[0102] In some preferred embodiments, during the linear projection, linear fusion, state reasoning, and generation of the visual gating vector and acoustic gating vector, when sequentially executed linear transformations, nonlinear activations, and normalization operations are involved, the method includes: during the model compilation phase, fusing the sequentially executed linear transformations, nonlinear activations, and normalization operations into a single computation graph node; during NPU runtime, the fused computation graph node completes all operations in a single data loading process and retains the intermediate computation results in the NPU's on-chip cache without writing back to external memory.

[0103] In specific implementation, during the execution of the linear projection, the linear fusion, the state reasoning, and the generation of the visual gating vector and the acoustic gating vector, when the linear transformation, nonlinear activation, and normalization operations are executed sequentially, the calculation mode follows a chain of operations such as Y = Norm(σ(X · W + b)), where X is the input feature matrix, W and b are the weight matrix and bias vector, σ(·) is the nonlinear activation function, Norm(·) is the normalization operation, and Y is the output feature matrix.

[0104] To optimize runtime efficiency on the NPU, operator fusion optimization is performed during the model compilation phase. The compiler analyzes the computation graph of the entire neural network, identifying sequentially executed linear transformations X · W + b, nonlinear activation functions σ(·), and normalization operations Norm(·). Then, the compiler merges these originally independent operations, which require sequential calls to multiple computation kernels at runtime, into a single combined computation graph node. This merged node contains all the computational instructions needed to complete the linear transformation, nonlinear activation, and normalization.

[0105] Furthermore, when the NPU actually runs inference tasks, it only needs to start the computation kernel once when it reaches this fused computation node. In one operation, the kernel continuously completes the following steps: starting from the input feature matrix X, it performs a linear transformation and adds the bias to obtain an intermediate result X · W + b; immediately applies a nonlinear activation function σ(·) to this intermediate result to obtain the activated result σ(X · W + b); then it performs a normalization operation Norm(·) until the final output Y is produced.

[0106] Throughout the process, the intermediate calculation results X · W + b generated by the linear transformation and the intermediate results σ(X · W + b) after activation do not need to be written back to external memory. Instead, they are directly stored in the NPU's on-chip cache or registers as input for the next operation (activation function or normalization). All intermediate data transfers are completed at high speed inside the chip.

[0107] By fusing operator chains, the number of kernel startups is reduced, and off-chip data transfer of intermediate results is eliminated, greatly improving data locality and computational efficiency, enabling complex multimodal neural networks to run smoothly on resource-constrained embedded NPUs.

[0108] In this embodiment, the computational model employs fixed-point quantization deployment, preferably INT8 weighted quantization and INT8 or INT16 activation quantization. The quantization parameters include scale and zero-point, and are fixed in the deployment file. Higher precision can be used as an optional implementation for precision-sensitive layers.

[0109] This embodiment significantly optimizes the inference latency and energy efficiency of the model on low-performance NPUs through operator fusion technology. Without fusion, each basic operation is an independent kernel. Kernel startup itself incurs overhead, and input / output data needs to be moved between on-chip cache and external storage before and after each kernel execution, resulting in a large amount of redundant data access. Furthermore, by fusion of multiple sequential operations into a composite kernel through the compiler, the number of kernel startups is fundamentally reduced. More importantly, it eliminates the read / write operations of intermediate results in off-chip memory, allowing data to remain in high-speed on-chip storage for as long as possible, ready for direct use by the next computation stage. This greatly improves data locality, reduces dependence on external storage bandwidth and access latency, and reduces power consumption caused by data movement. This optimization is crucial for tasks requiring real-time response, such as children's learning state recognition, enabling complex multimodal neural networks to run smoothly on resource-constrained embedded devices, meeting the stringent timing requirements for real-time analysis and intervention triggering.

[0110] Please see Figure 5 , Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0111] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0112] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a multimodal recognition method for children's learning states.

[0113] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0114] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a multimodal recognition method for children's learning states.

[0115] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0116] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a multimodal recognition method for children's learning states provided in any of the above method embodiments.

[0117] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0118] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0119] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of the multimodal recognition method for children's learning states provided in any of the above-described method embodiments.

[0120] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0121] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0122] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0123] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0126] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0127] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for multimodal recognition of children's learning states, characterized in that, include: A visual key point feature sequence and an acoustic feature sequence are obtained. The visual key point feature sequence is extracted from the video stream captured by the camera of the children's learning device, and the acoustic feature sequence is extracted from the audio stream captured by the microphone of the children's learning device. The visual key point feature sequence and the acoustic feature sequence are aligned in the time dimension, and segmented by a sliding window to obtain the aligned windowed visual features and windowed acoustic features. Obtain the temporal visual feature representation of the windowed visual features and the temporal acoustic feature representation of the windowed acoustic features; A visual gating vector is generated based on the temporal acoustic feature representation, and an acoustic gating vector is generated based on the temporal visual feature representation. The temporal visual feature representation is modulated using the visual gating vector to obtain modulated visual features, and the temporal acoustic feature representation is modulated using the acoustic gating vector to obtain modulated acoustic features. The modulated visual features and the modulated acoustic features are spliced ​​and linearly fused to generate cross-modal fusion features; The cross-modal fusion features are input into the state reasoning unit for state reasoning, and a state vector representing the child's learning state is output. The visual gating vector g v Through formula g v = σ( H a · W gv + b gv The calculation yielded, where H a This represents the temporal acoustic features. W gv and b gv These are the weight matrix and bias vector for acoustically guided visual gating, respectively, and σ(·) is the Sigmoid activation function; The acoustic gating vector g a Through formula g a = σ( H v · W ga + b ga The calculation yielded, where H v This is a representation of the temporal visual features. W ga and b ga These are the weight matrix and bias vector for visually guided acoustic gating, respectively.

2. The multimodal recognition method for children's learning states according to claim 1, characterized in that, Aligning the visual keypoint feature sequence and the acoustic feature sequence in the time dimension includes: When the frame rate of the visual keypoint feature sequence is inconsistent with the frame rate of the acoustic feature sequence, the acoustic feature sequence is interpolated and resampled based on the timestamp of the visual keypoint feature sequence; or the visual keypoint feature sequence is interpolated and resampled based on the timestamp of the acoustic feature sequence; or the visual keypoint feature sequence and the acoustic feature sequence are respectively input into a preset visual sub-network and an acoustic sub-network for temporal encoding, and then the encoded visual features and acoustic features are aggregated and aligned based on the preset timestamp to obtain a temporally aligned visual keypoint feature sequence and acoustic feature sequence.

3. The multimodal recognition method for children's learning states according to claim 1, characterized in that, The temporal visual feature representation H v Through formula H v = PWConv1D(σ(DWConv1D( E v The calculation yielded, where E v = X v · W v + b v This is an embedding representation of visual features after linear projection. X v For the windowed visual features, W v and b v Here, σ(·) represents the weight matrix and bias vector of the linear projection of visual features, respectively; DWConv1D(·) is a one-dimensional depthwise convolution operation along the time dimension; σ(·) is a non-linear activation function; and PWConv1D(·) is a one-dimensional pointwise convolution operation. The temporal acoustic feature representation H a Through formula H a = PWConv1D(σ(DWConv1D( E a The calculation yielded, where E a = X a · W a + b a This is the embedded representation of acoustic features after linear projection. X a For the windowed acoustic features, W a and b a These are the weight matrix and bias vector for the linear projection of the acoustic features, respectively.

4. The multimodal recognition method for children's learning states according to claim 1, characterized in that, The modulated visual features Through formula The calculation yielded, where ⊙ represents element-wise multiplication; the modulated acoustic features Through formula Calculated; The cross-modal fusion features H fuse Through formula The calculation yields the result, where Concat(·) is the feature concatenation operation. W f and b f These are the weight matrix and bias vector for the cross-modal fusion mapping, respectively.

5. The method according to claim 4, characterized in that, The method is executed on an NPU. When matrix multiplication operations are involved in linear projection, linear fusion, state reasoning, and the generation of the visual and acoustic gate vectors, if the size of the weight matrix involved in the operation exceeds the capacity of the NPU's on-chip cache, the method includes the following steps: The weight matrix is ​​divided into multiple weight sub-blocks by columns, and the size of each weight sub-block does not exceed a preset cache capacity threshold; Each weight sub-block is loaded sequentially into the on-chip cache of the NPU. The currently loaded weight sub-block and the input feature matrix are used to calculate a partial output result, and the partial output result is stored in external memory or the on-chip cache of the NPU. After all weight sub-blocks have been calculated, the output results of all parts are concatenated to obtain the final output matrix.

6. The method according to claim 5, characterized in that, In the processes of linear projection, linear fusion, state reasoning, and generation of the visual gating vector and acoustic gating vector, when sequential linear transformation, nonlinear activation, and normalization operations are involved, the method includes: During the model compilation phase, the sequentially executed linear transformations, nonlinear activations, and normalization operations are merged into a single computation graph node. During NPU runtime, the merged computation graph nodes complete all operations in a single data loading process and retain intermediate computation results in the NPU's on-chip cache without needing to write back to external memory.

7. The multimodal recognition method for children's learning states according to claim 1, characterized in that, After outputting the state vector, the method further includes: Get the state vector of the current time window and historical smooth state vector ; Using the exponential moving average formula The current smooth state vector is calculated. , where α is the smoothing factor; Determine the current smooth state vector The value of the target dimension corresponding to the preset learning state type is determined whether it meets the preset threshold condition corresponding to the target dimension within N consecutive time windows, where N is greater than or equal to 3. If the conditions are met, a learning intervention behavior corresponding to the dimension is triggered. The learning intervention behavior includes adjusting the difficulty of the learning content, providing voice prompts, or triggering a rest reminder. If the condition is not met, then return to the state vector obtained for the current time window. The process continues by observing the state vectors of subsequent time windows.

8. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images