Multi-element intelligent evaluation method and system for comprehensive development ability of infants

Abstract

The invention discloses a multivariate intelligent evaluation method and system for comprehensive development ability of infants, and belongs to the technical field of intelligent evaluation. The method comprises the following steps: S1, extracting a key frame sequence containing micro expressions and limb actions; s2, fusing into a spatio-temporal feature vector; s3, screening out a second-level key frame; and S4, mapping the behavior feature vector of the secondary key frame to a development plane coordinate system, outputting a standardized score of each capability dimension, and generating a multivariate intelligent evaluation report. According to the method, the spatial-temporal feature vector is input into the hierarchical Transform structure to obtain the behavior feature vector, and the secondary key frame is screened out, so that the most representative key frame can be focused on the basis of deep feature extraction, efficient screening and concentration of the infant behavior features are realized, and the accuracy of the infant behavior feature extraction is improved. The problems that in the prior art, feature dimensions are redundant, and key information is prone to being diluted are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligence assessment, and in particular to a method and system for assessing the comprehensive developmental abilities of infants and young children with multiple intelligences. Background Art

[0002] With society's increasing emphasis on early childhood education and children's all-round development, assessment of infants' and young children's comprehensive developmental abilities has become a crucial tool in education, medicine, and psychology. Accurately assessing infants' and young children's developmental abilities is crucial for early intervention, personalized educational planning, and medical decision-making. Traditional assessment methods rely primarily on parent questionnaires, teacher observations, and limited video analysis, but these methods have numerous limitations in practical application.

[0003] Traditional assessment methods, based on existing technologies, collect data from a single dimension, failing to fully capture infants' and young children's behaviors in different scenarios. This results in incomplete and inaccurate assessment results. Furthermore, existing technologies struggle to effectively integrate spatial and temporal features within videos, leading to redundant feature dimensions and the dilution of key information, thus impacting the accuracy and efficiency of assessments. Summary of the Invention

[0004] The embodiments of the present application provide a method and system for evaluating the comprehensive developmental abilities of infants and young children based on multiple intelligences, thereby solving the problems of redundant feature dimensions and easy dilution of key information in the prior art, and achieving efficient screening and concentration of infant and young children's behavioral characteristics.

[0005] The embodiment of the present application provides a method for evaluating the comprehensive developmental abilities of infants and young children with multiple intelligences, comprising the following steps: S1, using a multi-view camera device to acquire activity video data of infants and young children in different types of test activities and pre-processing the activity video data to extract key frame sequences containing micro-expressions and body movements;

[0006] S2, inputs the key frame sequence into a two-stream convolutional neural network, extracts the spatial features of hand features in the key frames and the dynamic features of expression changes between consecutive frames, and fuses the two into a spatiotemporal feature vector;

[0007] S3, input the spatiotemporal feature vector into the hierarchical Transformer structure to obtain the behavioral feature vector and filter out the secondary key frames;

[0008] S4 maps the behavioral feature vectors of the secondary key frames to the development plane coordinate system, outputs the standardized scores of each ability dimension, and generates a multiple intelligence assessment report.

[0009] Furthermore, the preprocessing of the active video data includes time alignment and illumination normalization:

[0010] Among them, timing alignment includes the following steps:

[0011] Optical flow field estimation is used to compensate for timing misalignment caused by device jitter, and motion video frames are filtered out from the set of video frames after illumination normalization.

[0012] The inter-frame similarity matching based on Hu moment is used for multi-view motion video frames to generate the affine transformation matrix of adjacent frames and obtain the aligned video frame sequence;

[0013] The aligned video frame sequence is input into the STIP detector to extract the spatiotemporal interest points containing hand-eye coordination actions and form a standardized key frame sequence.

[0014] Furthermore, the two-stream convolutional neural network includes a spatial stream network and a temporal stream network;

[0015] The spatial stream network is used to extract hand feature spatial features in key frames through depth-wise separable convolution kernels;

[0016] The temporal stream network is used to capture the dynamic features of expression changes between consecutive frames through 3D convolution kernels.

[0017] Furthermore, the hierarchical Transformer structure includes a bottom-level attention head and a top-level cross-fragment attention mechanism:

[0018] The underlying attention head operates on the query matrix, key matrix, and value matrix, combined with the local mask matrix to limit the attention range, process the short-term dependencies within a single activity segment, and extract short-term behavior features;

[0019] The top-level cross-segment attention mechanism is used to generate the query vector from the CLS tag of the previous activity segment, and the key-value pairs come from the global features of the current segment, establishing long-term associations across activity scenes and generating a behavior feature vector.

[0020] Furthermore, the local mask matrix is ​​constructed as follows:

[0021] For the input sequence, the window size is obtained according to the maximum local window size and the window scale coefficient;

[0022] When the position distance between two elements in the input sequence exceeds the window size, the corresponding position of the mask matrix is ​​set to negative infinity, otherwise it is set to zero.

[0023] Furthermore, the steps of screening the secondary key frames are:

[0024] The second-level key frames in the video frame sequence are extracted by judging whether the conditions of fully connected attention are met through the change rate of the action feature vector;

[0025] The calculation formula of the motion feature vector change rate is:

[0026]

[0027] Among them, a f,t represents the action feature vector at time t, τ a is the action change threshold;

[0028] If the change rate of the action feature vector is greater than the action change threshold, the fully connected attention condition is met and the corresponding video frame is recorded as a secondary key frame. Otherwise, the fully connected attention condition is not met and it is not recorded as a secondary key frame.

[0029] Furthermore, the step of obtaining the standardized score includes:

[0030] The spatiotemporal feature vectors are mapped to the development plane coordinate system through the projection matrix and activation function to obtain the coordinate representation of each ability dimension;

[0031] Extract the developmental trajectory point set from the standard developmental curve dataset, construct a contrast loss function, calculate the Mahalanobis distance between the positive samples and the standard features to find the minimum contrast loss function, and output the optimized projection matrix;

[0032] The behavioral feature vector is mapped to the development plane coordinate system according to the optimized projection matrix, and the Mahalanobis distance between the current development status and the standard milestone is calculated again. The Mahalanobis distance is normalized to generate the standardized score of each ability dimension.

[0033] Furthermore, the contrast loss function is:

[0034]

[0035] Among them, L d is the contrast loss function, D md is the Mahalanobis distance, N p is the number of positive samples, N q is the number of negative samples, μ d,i is the mean value of the standard development trajectory at the i-th development stage, μ d,j is the mean value of the standard development trajectory at the jth development stage, In order to meet the characteristics of positive samples in this development stage, the screening criteria are μ d,i The cosine similarity is greater than ρ s , ρ s is the similarity threshold, It is a negative sample characteristic that does not meet the current development stage.

[0036] Furthermore, the calculation formula of the development plane coordinates is:

[0037] (x d ,yd )=Φ d (F d )=W d,M ReLU(W d,p ·F d +b d )

[0038] Among them, F d is the space-time eigenvector, Φ d To develop the plane coordinate mapping function, the coordinate axis x d Corresponding standard development progress, y d Indicates the deviation from the standard curve. is the projection matrix, D p is the projection dimension, D f is the feature dimension, b d is the bias term in the projection process, W d,M To develop the matrix of plane coordinate mapping, M represents the mapping operation.

[0039] The embodiment of the present application provides a system for evaluating the comprehensive developmental abilities of infants and young children using multiple intelligences, including: a key frame extraction module, a spatiotemporal feature fusion module, a secondary key frame screening module, and an evaluation report generation module;

[0040] The key frame extraction module is used to obtain activity video data of infants and young children in different types of test activities through multi-view camera equipment and pre-process the activity video data to extract key frame sequences containing micro-expressions and body movements;

[0041] The spatiotemporal feature fusion module is used to input the key frame sequence into the two-stream convolutional neural network, extract the hand feature spatial features in the key frames and the dynamic features of expression changes between consecutive frames, and fuse the two into a spatiotemporal feature vector;

[0042] The secondary key frame screening module is used to input the spatiotemporal feature vector into the hierarchical Transformer structure to obtain the behavioral feature vector and screen out the secondary key frames;

[0043] The evaluation report generation module is used to map the behavioral feature vectors of the secondary key frames to the development plane coordinate system, output the standardized scores of each ability dimension, and generate a multiple intelligence evaluation report.

[0044] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0045] 1. By inputting the spatiotemporal feature vector into the hierarchical Transformer structure to obtain the behavioral feature vector and screening out the secondary key frames, it is possible to focus on the most representative key frames based on deep feature extraction, thereby achieving efficient screening and concentration of infant behavioral characteristics, effectively solving the problems of redundant feature dimensions and easy dilution of key information in existing technologies.

[0046] 2. By inputting the key frame sequence into a two-stream convolutional neural network, the spatial features of hand features and the dynamic features of expression changes between consecutive frames are extracted, and the two are fused into a spatiotemporal feature vector. This can simultaneously retain the spatial details in the key frames and the dynamic information in the time series, thereby achieving a refined representation of the behavioral characteristics of infants and young children.

[0047] 3. By using multi-view camera equipment to obtain video data of infants and young children in different types of test activities and pre-processing the video data to extract key frame sequences, it is possible to comprehensively and accurately capture the micro-expressions and body movement information of infants and young children in various scenarios, thereby realizing multi-dimensional collection of infants and young children's behavioral characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flow chart of a method for evaluating the comprehensive developmental abilities of infants and young children using multiple intelligences provided in an embodiment of the present application;

[0049] Figure 2 This is a structural diagram of a system for assessing the comprehensive developmental abilities of infants and young children, as provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application provide a method and system for assessing the comprehensive developmental abilities of infants and young children with multiple intelligences, thereby solving the problems of redundant feature dimensions and easy dilution of key information in the prior art. By inputting spatiotemporal feature vectors into a hierarchical Transformer structure to obtain behavioral feature vectors and screening out secondary key frames, it is possible to focus on the most representative key frames based on deep feature extraction, thereby achieving efficient screening and concentration of infant behavioral characteristics.

[0051] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0052] like Figure 1FIG2 is a flow chart of a method for evaluating the comprehensive developmental abilities of infants and young children according to an embodiment of the present application. The method is applied to a system for evaluating the comprehensive developmental abilities of infants and young children, and includes the following steps: S1, using a multi-view camera device to acquire activity video data of infants and young children in different types of test activities, and pre-processing the activity video data to extract key frame sequences containing micro-expressions and body movements;

[0053] S2, inputs the key frame sequence into a two-stream convolutional neural network, extracts the spatial features of hand features in the key frames and the dynamic features of expression changes between consecutive frames, and fuses the two into a spatiotemporal feature vector;

[0054] S3, input the spatiotemporal feature vector into the hierarchical Transformer structure to obtain the behavioral feature vector and filter out the secondary key frames;

[0055] S4 maps the behavioral feature vectors of the secondary key frames to the development plane coordinate system, outputs the standardized scores of each ability dimension, and generates a multiple intelligence assessment report.

[0056] Furthermore, the preprocessing of the active video data includes time alignment and illumination normalization:

[0057] Among them, timing alignment includes the following steps:

[0058] Optical flow field estimation is used to compensate for timing misalignment caused by device jitter, and motion video frames are filtered out from the set of video frames after illumination normalization.

[0059] Filter conditions: ‖optical flow ‖2>τ f ;

[0060] Among them, optical flow Refers to optical flow, τ f is the motion saliency threshold;

[0061] The inter-frame similarity matching based on Hu moment is used for multi-view motion video frames to generate the affine transformation matrix of adjacent frames and obtain the aligned video frame sequence;

[0062] The affine transformation matrix T t The calculation formula is:

[0063]

[0064] Among them, Hu k (·) represents the kth order Hu invariant moment calculation function, N h is the Hu moment feature dimension, I t Represents the video frame image at time t;

[0065] The aligned video frame sequence is input into the STIP detector to extract the spatiotemporal interest points containing hand-eye coordination actions and form a standardized key frame sequence;

[0066] The illumination normalization comprises the following steps:

[0067] Perform multi-scale Gaussian filtering on the original video frame to obtain the illumination component, and calculate the reflection component R according to the ratio of the original video frame and the illumination component. i , the illumination normalized video frame is I′=γ·R avg +(1-γ)·R i , where γ is the empirical coefficient, R avg is the average reflection component, and outputs a set of video frames after illumination normalization.

[0068] Furthermore, the two-stream convolutional neural network includes a spatial stream network and a temporal stream network;

[0069] The spatial stream network is used to extract hand feature spatial features in key frames through depth-wise separable convolution kernels;

[0070] The temporal stream network is used to capture the dynamic features of expression changes between consecutive frames through 3D convolution kernels.

[0071] Furthermore, the hierarchical Transformer structure includes a bottom-level attention head and a top-level cross-fragment attention mechanism:

[0072] The underlying attention head operates on the query matrix, key matrix, and value matrix, combined with the local mask matrix to limit the attention range, process the short-term dependencies within a single activity segment, and extract short-term behavior features;

[0073] The calculation formula of the underlying attention head is:

[0074]

[0075] Among them, M local is the local mask matrix that limits the attention range, d k,l represents the dimension of the underlying key vector, Q l ,K l ,V l represent the underlying query, key matrix, and value matrix respectively;

[0076] The top-level cross-segment attention mechanism is used to generate the query vector from the CLS tag of the previous activity segment, and the key-value pairs come from the global features of the current segment, establishing long-term associations across activity scenes and generating a behavior feature vector.

[0077] Furthermore, the local mask matrix is ​​constructed as follows:

[0078] For the input sequence, the window size is obtained according to the maximum local window size and the window scale coefficient;

[0079] The calculation formula of the window size is:

[0080] w m =min(L max ,η w L f )

[0081] Among them, L max is the maximum local window size, η w is the window scale factor, L f is the length of the input sequence;

[0082] When the position distance between two elements in the input sequence exceeds the window size, the corresponding position of the mask matrix is ​​set to negative infinity, otherwise it is set to zero;

[0083]

[0084] Among them, M local [i,j] is the corresponding position of the mask matrix between element i and element j.

[0085] Furthermore, the steps of screening the secondary key frames are:

[0086] The second-level key frames in the video frame sequence are extracted by judging whether the conditions of fully connected attention are met through the change rate of the action feature vector;

[0087] The calculation formula of the motion feature vector change rate is:

[0088]

[0089] Among them, a f,t represents the action feature vector at time t, τ a is the action change threshold;

[0090] If the change rate of the action feature vector is greater than the action change threshold, the fully connected attention condition is met and the corresponding video frame is recorded as a secondary key frame. Otherwise, the fully connected attention condition is not met and it is not recorded as a secondary key frame.

[0091] Furthermore, the step of obtaining the standardized score includes:

[0092] The spatiotemporal feature vectors are mapped to the development plane coordinate system through the projection matrix and activation function to obtain the coordinate representation of each ability dimension;

[0093] Extract the developmental trajectory point set from the standard developmental curve dataset, construct a contrast loss function, calculate the Mahalanobis distance between the positive samples and the standard features to find the minimum contrast loss function, and output the optimized projection matrix;

[0094] The behavioral feature vector is mapped to the developmental plane coordinate system according to the optimized projection matrix. The Mahalanobis distance between the current developmental status and the standard milestone is calculated again, and the Mahalanobis distance is normalized to generate the standardized score of each ability dimension.

[0095] The Mahalanobis distance D between the current development status and the standard milestone md The calculation formula is:

[0096]

[0097] Among them, D m To develop the number of assessment dimensions, represents the mth row and nth column element of the inverse covariance matrix, μ d,m is the mean of the standard development trajectory on the mth evaluation dimension, μ d,n is the mean of the standard development trajectory on the nth evaluation dimension, x d,m is the coordinate value of the mapped behavior feature vector on the mth evaluation dimension, x d,n is the coordinate value of the mapped behavior feature vector on the nth evaluation dimension.

[0098] Furthermore, the contrast loss function is:

[0099]

[0100] Among them, L d is the contrast loss function, D md is the Mahalanobis distance, N p is the number of positive samples, N q is the number of negative samples, μ d,i is the mean value of the standard development trajectory at the i-th development stage, μ d,j is the mean value of the standard development trajectory at the jth development stage, In order to meet the characteristics of positive samples in this development stage, the screening criteria are μ d,i The cosine similarity is greater than ρ s , ρ s is the similarity threshold, It is a negative sample characteristic that does not meet the current development stage. Represents the Mahalanobis distance between the positive sample and the standard feature. Represents the Mahalanobis distance between the negative sample and another standard feature.

[0101] Furthermore, the calculation formula of the development plane coordinates is:

[0102] (x d ,y d )=Φ d (F d )=W d,M ReLU(W d,p ·F d +b d )

[0103] Among them, F d is the space-time eigenvector, Φ d To develop a plane coordinate mapping function to transform the high-dimensional space-time feature vector F d Converted to two-dimensional development plane coordinates (x d ,y d ), coordinate axis x d Corresponding standard development progress, y d Indicates the deviation from the standard curve. is the projection matrix, D p is the projection dimension, D f is the feature dimension, b d is the bias term in the projection process, W d,M To develop the matrix of plane coordinate mapping, M represents the mapping operation.

[0104] like Figure 2 , which is a structural diagram of a system for evaluating the comprehensive developmental abilities of infants and young children provided in an embodiment of the present application, and includes: a key frame extraction module, a spatiotemporal feature fusion module, a secondary key frame screening module, and an evaluation report generation module;

[0105] The key frame extraction module is used to obtain activity video data of infants and young children in different types of test activities through multi-view camera equipment and pre-process the activity video data to extract key frame sequences containing micro-expressions and body movements;

[0106] The spatiotemporal feature fusion module is used to input the key frame sequence into the two-stream convolutional neural network, extract the hand feature spatial features in the key frames and the dynamic features of expression changes between consecutive frames, and fuse the two into a spatiotemporal feature vector;

[0107] The secondary key frame screening module is used to input the spatiotemporal feature vector into the hierarchical Transformer structure to obtain the behavioral feature vector and screen out the secondary key frames;

[0108] The evaluation report generation module is used to map the behavioral feature vectors of the secondary key frames to the development plane coordinate system, output the standardized scores of each ability dimension, and generate a multiple intelligence evaluation report.

[0109] In summary, the embodiment of the present application obtains a behavioral feature vector by inputting the spatiotemporal feature vector into a hierarchical Transformer structure, and screens out secondary key frames, so that it can focus on the most representative key frames based on deep feature extraction, thereby achieving efficient screening and concentration of infant behavioral characteristics.

[0110] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0114] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0115] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for evaluating the comprehensive developmental abilities of infants and young children based on their multiple intelligences, characterized by: The following steps are involved: S1, using multi-view camera equipment to obtain activity video data of infants and young children in different types of test activities and pre-process the activity video data to extract key frame sequences containing micro-expressions and body movements; S2, inputs the key frame sequence into a two-stream convolutional neural network, extracts the spatial features of hand features in the key frames and the dynamic features of expression changes between consecutive frames, and fuses the two into a spatiotemporal feature vector; S3, input the spatiotemporal feature vector into the hierarchical Transformer structure to obtain the behavioral feature vector and filter out the secondary key frames; S4 maps the behavioral feature vectors of the secondary key frames to the development plane coordinate system, outputs the standardized scores of each ability dimension, and generates a multiple intelligence assessment report.

2. A method for assessing the comprehensive developmental abilities of infants and young children according to claim 1, characterized in that: The preprocessing of the active video data includes time alignment and illumination normalization: Among them, timing alignment includes the following steps: Optical flow field estimation is used to compensate for timing misalignment caused by device jitter, and motion video frames are filtered out from the set of video frames after illumination normalization. The inter-frame similarity matching based on Hu moment is used for multi-view motion video frames to generate the affine transformation matrix of adjacent frames and obtain the aligned video frame sequence; The aligned video frame sequence is input into the STIP detector to extract the spatiotemporal interest points containing hand-eye coordination actions and form a standardized key frame sequence.

3. A method for assessing the comprehensive developmental abilities of infants and young children according to claim 1, characterized in that: The two-stream convolutional neural network includes a spatial stream network and a temporal stream network; The spatial stream network is used to extract hand feature spatial features in key frames through depth-wise separable convolution kernels; The temporal stream network is used to capture the dynamic features of expression changes between consecutive frames through 3D convolution kernels.

4. A method for assessing the comprehensive developmental abilities of infants and young children according to claim 1, characterized in that: The hierarchical Transformer structure consists of a bottom-level attention head and a top-level cross-fragment attention mechanism: The underlying attention head operates on the query matrix, key matrix, and value matrix, combined with the local mask matrix to limit the attention range, process the short-term dependencies within a single activity segment, and extract short-term behavior features; The top-level cross-segment attention mechanism is used to generate the query vector from the CLS tag of the previous activity segment, and the key-value pairs come from the global features of the current segment, establishing long-term associations across activity scenes and generating a behavior feature vector.

5. A method for assessing the comprehensive developmental abilities of infants and young children according to claim 4, characterized in that: The local mask matrix is ​​constructed as follows: For the input sequence, the window size is obtained according to the maximum local window size and the window scale coefficient; When the position distance between two elements in the input sequence exceeds the window size, the corresponding position of the mask matrix is ​​set to negative infinity, otherwise it is set to zero.

6. A method for evaluating the comprehensive developmental abilities of infants and young children according to claim 1, characterized in that: The steps of screening the secondary key frames are: The second-level key frames in the video frame sequence are extracted by judging whether the conditions of fully connected attention are met through the change rate of the action feature vector; The calculation formula of the motion feature vector change rate is: Among them, a f,t represents the action feature vector at time t, τ a is the action change threshold; If the change rate of the action feature vector is greater than the action change threshold, the fully connected attention condition is met and the corresponding video frame is recorded as a secondary key frame. Otherwise, the fully connected attention condition is not met and it is not recorded as a secondary key frame.

7. The method for evaluating the comprehensive developmental abilities of infants and young children according to claim 1, wherein: The steps of obtaining the standardized score include: The spatiotemporal feature vectors are mapped to the development plane coordinate system through the projection matrix and activation function to obtain the coordinate representation of each ability dimension; Extract the developmental trajectory point set from the standard developmental curve dataset, construct a contrast loss function, calculate the Mahalanobis distance between the positive samples and the standard features to find the minimum contrast loss function, and output the optimized projection matrix; The behavioral feature vector is mapped to the development plane coordinate system according to the optimized projection matrix, and the Mahalanobis distance between the current development status and the standard milestone is calculated again. The Mahalanobis distance is normalized to generate the standardized score of each ability dimension.

8. A method for evaluating the comprehensive developmental abilities of infants and young children according to claim 7, characterized in that: The contrast loss function is: Among them, L d is the contrast loss function, D md is the Mahalanobis distance, N p is the number of positive samples, N q is the number of negative samples, μ d,i is the mean value of the standard development trajectory at the i-th development stage, μ d,j is the mean value of the standard development trajectory at the jth development stage, In order to meet the characteristics of positive samples in this development stage, the screening criteria are μ d,i The cosine similarity is greater than ρ s , ρ s is the similarity threshold, It is a negative sample characteristic that does not meet the current development stage.

9. The method for evaluating the comprehensive developmental abilities of infants and young children according to claim 7, wherein: The calculation formula of the development plane coordinates is: (x d ,y d )=Φ d (F d )=In d,M ·ReLU(W d,p ·F d +b d ) Among them, F d is the space-time eigenvector, Φ d To develop the plane coordinate mapping function, the coordinate axis x d Corresponding standard development progress, y d Indicates the deviation from the standard curve. is the projection matrix, D p is the projection dimension, D f is the feature dimension, b d is the bias term in the projection process, W d,M To develop the matrix of plane coordinate mapping, M represents the mapping operation.

10. A system for assessing the comprehensive developmental abilities of infants and young children based on multiple intelligences, characterized by: include: Key frame extraction module, spatiotemporal feature fusion module, secondary key frame screening module, and evaluation report generation module; The key frame extraction module is used to obtain activity video data of infants and young children in different types of test activities through multi-view camera equipment and pre-process the activity video data to extract key frame sequences containing micro-expressions and body movements; The spatiotemporal feature fusion module is used to input the key frame sequence into the two-stream convolutional neural network, extract the hand feature spatial features in the key frames and the dynamic features of expression changes between consecutive frames, and fuse the two into a spatiotemporal feature vector; The secondary key frame screening module is used to input the spatiotemporal feature vector into the hierarchical Transformer structure to obtain the behavioral feature vector and screen out the secondary key frames; The evaluation report generation module is used to map the behavioral feature vectors of the secondary key frames to the development plane coordinate system, output the standardized scores of each ability dimension, and generate a multiple intelligence evaluation report.

Images