Quantitative analysis method and system for social behaviors of children and application

Abstract

The invention belongs to the technical field of behavior recognition and medical auxiliary diagnosis, and relates to a child social behavior quantitative analysis method and system and application. The analysis method comprises the steps that interactive videos and interactive audios in a natural state are collected, multi-modal social behavior data of children in the natural state are obtained from the interactive videos and the interactive audios, and the multi-modal social behavior data comprise face data, audio data, text data and posture data; constructing corresponding quantitative features based on the data and fusing the quantitative features to obtain fused quantitative features; and configuring a pre-trained large language model, and inputting the fused quantitative features into the large language model to obtain an ASD risk probability, a multi-dimensional quantitative score vector, a frame-level anomaly probability sequence and a microscopic anomaly positioning timestamp set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of behavior recognition and medical auxiliary diagnosis, and in particular to a child social behavior quantitative analysis method, system and application. BACKGROUND

[0002] Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by social communication impairment and stereotyped repetitive behavior, usually onset in early childhood, and has become an important public health problem affecting the growth of children. Current clinical screening mainly relies on manual observation and scale assessment (such as ADOS), but there are problems such as long time consumption, strong subjectivity, poor repeatability, etc., which are difficult to meet the needs of early large-scale screening.

[0003] ASD children often show multi-dimensional behavior characteristics such as gaze avoidance and language abnormalities in natural interaction, and can better reflect the real situation in an unstructured environment without instructions. However, the structured interview and task setting in the traditional evaluation process limit the natural expression of children, making it difficult to fully reflect their real social skills. And the existing artificial intelligence-based auxiliary diagnosis method still relies on task instruction-driven and mostly uses single modal, lacking ecological effectiveness, and difficult to capture multi-dimensional behavior coordination features, limiting the application in clinical and family scenarios.

[0004] Therefore, there is an urgent need for a method that can automatically collect multi-modal behavior data without interference in a natural interaction environment and objectively quantify children's social behavior, achieving efficient, accurate and interpretable ASD auxiliary screening. SUMMARY

[0005] The purpose of the present application is to design a child social behavior quantitative analysis method, system and application to address the problems of long time consumption, strong subjectivity, poor repeatability, and large environmental interference of existing child autism screening technology. It is suitable for collecting multi-modal behavior data of children in natural interaction scenarios and constructing structured feature representation to assist in identifying early behavior characteristics related to autism spectrum disorder, improving the automation and objectivity of screening. To achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a child social behavior quantitative analysis method, comprising the following steps: Collecting interaction videos and interaction audios in a natural state, obtaining multi-modal social behavior data of children in a natural state from the interaction videos and the interaction audios, the multi-modal social behavior data including facial data, audio data, text data and posture data; Identifying the target child's face based on the facial data, screening the main angle of view image, extracting the facial motion intensity feature, the facial symmetry feature and the emotional stability value feature, and constructing the facial quantitative feature; Extracting and obtaining an effective speech segment set of the target child from the audio data, and performing frame segmentation on each effective speech segment in the effective speech segment set with a preset window length and a preset frame shift, extracting frame-level fundamental frequency, loudness, and the first three formants, and constructing speech quantization features; Obtaining a timestamp set corresponding to the effective speech segment set of the target child, obtaining a transcription text corresponding to the time in the timestamp set, obtaining a word sequence and a sentence sequence of each text in the transcription text, extracting speech rate features, average sentence length features, syntax complexity features, and repetition rate features, and constructing text quantization features; Obtaining a two-dimensional motion trajectory of the target child based on the interactive video, calculating spatial edge proportion features, activity features, barycenter stability features, and upper limb activity features, and constructing posture quantization features; Fusing the facial quantization features, the speech quantization features, the text quantization features, and the posture quantization features to obtain fused quantization features; Configuring a pre-trained large language model, and inputting the fused quantization features into the large language model to obtain an ASD risk probability, a multi-dimensional quantization score vector, a frame-level anomaly probability sequence, and a micro-anomaly positioning timestamp set.

[0006] As a possible implementation manner, the interactive video is a multi-camera synchronous interactive video sequence; a plurality of multi-view images are obtained from the interactive video sequence to constitute the facial data; Identifying the face of the target child based on the facial data, specifically including: For each multi-view image, a reference embedding vector is generated by applying a face recognition model and a calibrated front-facing photo of the target child configured in the face recognition model; Each multi-view face region in each multi-view image is identified using a face detection model; and each multi-view face embedding vector of each frame is extracted by inputting each multi-view face region into the face recognition model; The cosine similarity between each multi-view face embedding vector of each frame and the reference embedding vector is calculated; The multi-view face region with a cosine similarity greater than or equal to a similarity threshold is determined as the target child, and a plurality of the multi-view face regions constitute an effective candidate multi-view image set.

[0007] As a possible implementation manner, the main multi-view image is screened, specifically including the following steps: The quality score of each effective candidate multi-view image in a preset time frame is calculated; A screening rule for the main multi-view image is constructed, specifically including: selecting the multi-view image with the highest quality score as the initial frame; For subsequent frames, under the premise of maintaining the main multi-view image of the current frame for a preset time, if the quality score of another multi-view image improves by more than a threshold, the multi-view image is switched, otherwise the current multi-view image is maintained; Based on the above filtering rules, a main-view frame sequence that is temporally continuous and has stable identity is obtained. The main view sequence was affinely normalized to standard pixel size to obtain a normalized main view frame sequence.

[0008] As one possible approach, facial motion intensity features, facial symmetry features, and emotional stability value features are extracted and facial quantification features are constructed, specifically including the following steps: The calculation of facial motion intensity features specifically includes: extracting key points from each frame in a normalized main view frame sequence to obtain a set of key points; determining N regions of the face and the key points corresponding to each region; calculating the geometric centroid of each region; calculating the Euclidean distance time series of the change of geometric centroid of each region in adjacent frames; and calculating the average geometric centroid movement distance of each region to obtain the facial motion intensity features. Facial feature symmetry features are obtained by calculating the Pearson correlation coefficient of the geometric centroids of the left and right facial regions along the horizontal x-axis within the first f frames; The emotion stability value feature is obtained by using a lightweight emotion recognition model to perform five-class emotion classification on a normalized first-person perspective frame sequence to obtain an emotion label sequence, and calculating the average value of the emotion labels based on the indicator function to obtain the emotion stability value feature. We construct an overall facial feature vector and a temporal feature matrix, which include facial motion intensity features, facial symmetry features, and emotional stability value features in each region, to form facial quantitative features.

[0009] As one possible implementation, extracting and obtaining a set of effective speech segments from the audio data for the target child specifically includes: Noise reduction, temporal reconstruction, and beamforming are performed on audio data to obtain enhanced audio data; Extracting speaker embedding vectors from enhanced audio signals; Calculate the cosine similarity between the voiceprint embedding vector and the pre-configured target child's voiceprint vector; Speech segments corresponding to voiceprint embedding vectors with cosine similarity greater than or equal to the threshold are identified as valid speech segments. Consecutive adjacent valid speech segments are merged to obtain a set of valid speech segments.

[0010] As one possible implementation, the fundamental frequency, loudness, and top three formants at the frame level are extracted and used to construct speech quantization features, specifically including: The YIN algorithm is used to extract the frame-level fundamental frequency, and zeros are padded to the undetected frame-level fundamental frequency to obtain the fundamental frequency of the frame-level fundamental frequency; Calculate frame-level loudness under ISO 226 standard A-weighting; Based on 12th-order LPC analysis, pole frequencies are extracted to obtain the first three resonance peaks. The frequency range of the first three resonance peaks is limited, and zeros are padded for those that do not conform to the range. The sequence of the first three resonance peaks is then output. The average value of the feature sequences of frame-level fundamental frequency, frame-level loudness, and the top three formants of the frame is calculated in the time dimension to obtain the overall speech feature vector. At the same time, a speech temporal feature sequence is constructed, and the speech quantization feature is composed of the overall speech feature vector and the speech temporal feature matrix.

[0011] As one possible implementation, text quantization features include a text overall feature vector and a text temporal feature matrix; wherein, the text overall feature vector is composed of the average values ​​of word sequences and sentence sequences extracted during the entire interaction process, as well as speech rate features, average sentence length features, syntactic complexity features, and repetition rate features; The text temporal feature matrix consists of the speech rate sequence, average sentence length sequence, syntactic complexity sequence, and repetition rate sequence of K speech segments.

[0012] As one possible approach, a wide-angle camera is placed at the top center of the interactive space to simultaneously capture the entire global interactive video sequence, while eight cameras are placed around the top center to simultaneously capture the entire multi-view video sequence. The two-dimensional motion trajectory is obtained as follows: the YOLOv5 detection model is used to obtain the center coordinates of the bounding box of the target child in each frame from the entire global interactive video sequence, and the two-dimensional motion trajectory is formed by the center coordinates of the bounding box of the target child in each frame. The spatial edge proportion characteristic is obtained by calculating the proportion of target children staying in the edge area; The activity feature is calculated as follows: the overall activity feature is obtained by calculating the displacement of consecutive frames, and the average value of the overall activity feature is obtained. The calculation method for the center of gravity stability feature is as follows: obtain a temporally continuous and identity-stable main view frame sequence using the filtering rules of the main view image; extract N body key points from each frame to form a key point set; define the center of gravity stability region and its corresponding key points; calculate the center of gravity of each frame for each center of gravity stability region; calculate the inter-frame center of gravity offset based on the center of gravity of each frame; and obtain the average value of the inter-frame center of gravity offset to obtain the center of gravity stability feature. The calculation method for upper limb activity features is as follows: define the upper limb activity region and its corresponding key points; calculate the geometric centroid of the upper limb activity region; calculate the distance difference between the geometric centroids of the upper limb activity regions between frames; and calculate the average value of the distance difference to obtain the upper limb activity features.

[0013] Secondly, the present invention provides a quantitative analysis system for children's social behavior, comprising: a multimodal behavior data acquisition module, which acquires interactive videos and interactive audio in a natural state, and obtains multimodal social behavior data of children in a natural state from the interactive videos and interactive audio, wherein the multimodal social behavior data includes facial data, audio data, text data and posture data; The behavioral pattern feature quantization module includes a facial feature quantization unit, a speech feature quantization unit, a text feature quantization unit, and a posture feature quantization unit. The facial feature quantization unit is used to identify the target child's face based on facial data, filter first-person perspective images, extract facial motion intensity features, facial symmetry features, and emotional stability value features, and construct facial quantization features. The speech feature quantization unit is used to extract and obtain a set of effective speech segments from the audio data of the target child, divide each effective speech segment in the set into frames with a preset window length and preset frame shift, extract the frame-level fundamental frequency, loudness, and top three formants, and construct speech quantization features. The text feature quantization unit is used to obtain the timestamp set corresponding to the set of effective speech segments of the target child, obtain the transcribed text at the corresponding time in the timestamp set, obtain the word sequence and sentence sequence of each segment in the transcribed text, extract speech rate features, average sentence length features, syntactic complexity features, and repetition rate features, and construct text quantization features. The posture feature quantization unit is used to obtain the two-dimensional motion trajectory of the target child based on interactive video, calculate spatial edge proportion features, activity features, center of gravity stability features, and upper limb activity features, and construct posture quantization features. The multimodal feature fusion module fuses facial quantization features, speech quantization features, text quantization features, and pose quantization features to obtain fused quantization features; The large model evaluation module configures a pre-trained large language model and inputs the fused quantized features into the large language model to obtain ASD risk probability, multi-dimensional quantized scoring vector, frame-level anomaly probability sequence, and micro-anomaly localization timestamp set.

[0014] Thirdly, this invention provides an application of a method for quantitative analysis of children's social behavior, which is used for autism screening.

[0015] Compared with existing technologies, it has the following beneficial effects: 1. The method, system and application for quantitative analysis of children's social behavior provided by this invention do not require instruction or scripted data collection. They directly assess children's spontaneous behavior in natural scenarios, resulting in high ecological effectiveness. They are non-contact, automated and have low human dependence, making them suitable for promotion and application in family, medical and educational scenarios.

[0016] 2. The method and system for quantitative analysis of children's social behavior provided by this invention cover multimodal features of voice, face, posture and language text, so as to achieve comprehensive characterization of behavior and accurate identification of autism; 3. The method and system for quantitative analysis of children's social behavior provided by this invention collects data non-contactly through fixed sensors, combines multimodal feature fusion and large model evaluation, and outputs structured screening scores and interpretive behavioral indicators; it supports the output of dimensional behavioral scores, risk levels and abnormal segment prompts, and has good interpretability. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of the method for quantitative analysis of children's social behavior in an embodiment of the present invention; Figure 2 This is a layout diagram of the data acquisition system in an embodiment of the present invention; Figure 3 This is a block diagram of the module structure of the children's social behavior quantitative analysis system in an embodiment of the present invention. Detailed Implementation

[0018] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0019] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0020] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0021] To overcome the problems of existing autism screening methods, such as reliance on structured tasks, subjective human judgment, and limited behavioral dimensions, this invention proposes a method, system, and application for quantitative analysis of children's social behavior. It collects video and audio data in a natural environment without interference, and through multimodal behavioral modeling and intelligent assessment, achieves quantitative analysis and anomaly identification of children's social behavior. Specific implementation methods are as follows: Firstly, this invention provides a method for quantitative analysis of children's social behavior, see [link to relevant documentation]. Figure 1 It includes the following steps: Collect interactive videos and audio in natural states, and obtain multimodal social behavior data of children in natural states from the interactive videos and audio. Multimodal social behavior data includes facial data, audio data, text data and posture data; As one possible implementation, interactive video is an interactive video sequence captured synchronously by multiple cameras; For example, in a rectangular room of suitable size and with a sound-absorbing environment, data on children's social behavior in their natural state are collected by using multi-angle cameras and microphone arrays; For example, see Figure 2 It is equipped with 9 synchronized cameras, including a wide-angle camera installed in the center of the ceiling to record the child's overall activity trajectory, and high-definition cameras distributed in the four corners of the room and the middle of the wall to capture the child's facial expressions and body movements; each camera has a resolution of 1920×1080, a frame rate of ≥60Hz, supports low light enhancement, outputs AVI format, has a color depth of 24-bit RGB, and is equipped with a zoom lens.

[0022] Video capture achieves frame-level synchronization (error ≤ 5ms) via the IEEE 1588 protocol, and is transmitted in real time via Gigabit Ethernet or USB 3.0 interface, with breakpoint resume and integrity verification functions; Audio acquisition was accomplished using a directional microphone array arranged diagonally across the room, with the microphones facing the interaction area. The microphones employed a cardioid or supercardioid pickup pattern, with a signal-to-noise ratio ≥70dB, sensitivity -38 dBV / Pa, sampling rate 48kHz, bit depth 24-bit, and PCM or AAC encoding. The audio was simultaneously labeled with the video for subsequent multimodal alignment.

[0023] The aforementioned system supports continuous acquisition of 20–60 minutes of natural interactive data. The acquired data is stored on a local SSD with a capacity of ≥8TB, organized by session number, meeting the needs of long-term, high-frequency data acquisition. It also features RAID backup to ensure data security and reliability, minimizing the risk of data loss even in the event of hardware failure or other unforeseen circumstances. Furthermore, the system supports remote monitoring and anomaly alarm functions, enabling real-time monitoring of various anomalies during the acquisition process, such as frame drops, audio interruptions, and image obstruction, and promptly issuing alarms to alert staff and ensure the smooth operation of the acquisition process.

[0024] Before data collection, children must complete a one-time registration calibration, including the collection of facial images and voice samples, for subsequent facial recognition, posture tracking, and voice binding, enabling continuous tracking of individual children. During the collection process, no task instructions or prompts are given; the guide accompanies the children in a natural way, allowing them to exhibit authentic social behaviors in an unrestrained and natural state. This results in the collection of high-quality, high-value behavioral data, providing strong support for subsequent research and analysis of children's social behaviors.

[0025] Multiple frames of multi-view images are acquired from an interactive video sequence to form facial data; For example, receiving a video sequence simultaneously captured by multiple cameras: ; in, For camera indexing; For video frame index, the video sampling frame rate is Frames per second.

[0026] Facial recognition of target children based on facial data, specifically including: For each frame of multi-view image, a face recognition model and a calibrated frontal photo of the target child configured in the face recognition model are applied to generate a reference embedding vector; For example, for each frame of multi-view images, a reference embedding vector is first generated based on a calibrated frontal photo using a pre-trained face recognition model (such as ArcFace). .

[0027] The face detection model is used to identify the face regions in each viewpoint of each frame of multi-view images; and the face regions in each viewpoint are input into the face recognition model to extract the face embedding vector of each frame and each viewpoint. For example, face detection tools (such as OpenFace) are used to identify different viewpoints. The face region is input into the face recognition model, and the face embedding vector for each frame and viewpoint is extracted. .

[0028] Calculate the cosine similarity between the face embedding vector and the reference embedding vector for each frame and each viewpoint; For example, the formula for calculating cosine similarity is as follows: ; The facial regions with a cosine similarity greater than or equal to the similarity threshold are identified as the target child, and multiple facial regions with this viewpoint constitute a set of effective candidate viewpoint images. For example, if the following conditions are met: ; The facial region from that perspective is then identified as the target child. For example, the similarity threshold can be set to .

[0029] The process involves identifying target children's faces based on facial data, filtering first-person view images, extracting facial motion intensity features, facial symmetry features, and emotional stability value features, and constructing facial quantitative features. As one possible approach, filtering images from the main viewpoint involves the following steps: Calculate the quality score of each valid candidate viewpoint image within a preset time frame. For example, for all effective candidate viewpoints at time frame t, calculate the image quality score: ; in, Image sharpness index (based on Laplacian operator variance to evaluate image sharpness); Face occlusion rate (based on the proportion of missing key points); The head pose offset is calculated using the Euclidean distances between pitch, yaw, roll, and the frontal pose. The weighting parameters are set sequentially as follows: , , .

[0030] The selection rules for the main viewpoint images are constructed, specifically including: selecting the viewpoint with the highest quality score in the initial frame; In subsequent frames, if the quality score of another viewpoint improves beyond a threshold while maintaining the preset time of the current frame's main viewpoint, the viewpoint is switched; otherwise, the current viewpoint is maintained. For example, the preset time is seconds, corresponding to the number of frames ; For example, the threshold is set as follows: ; Specifically, ; in, The best viewing angle for scoring the image quality of the current frame. This is the start time of the current main perspective.

[0031] Based on the above filtering rules, a main-view frame sequence that is temporally continuous and has stable identity is obtained. For example, the first-person perspective frame sequence is represented as follows: ; The main view sequence was affinely normalized to standard pixel size to obtain a normalized main view frame sequence. For example, each frame of the main view face image is affinely normalized to a standard size of 256×256 pixels to generate a normalized sequence. .

[0032] As one possible approach, facial motion intensity features, facial symmetry features, and emotional stability value features are extracted and facial quantification features are constructed, specifically including the following steps: Calculating facial motion intensity features specifically includes: Extract key points from each frame in the normalized main view frame sequence to obtain a set of key points; For example, based on the Dlib 68-point facial landmark model, the keypoint set for each frame is extracted as follows: ; Identify N regions of the face and the key points corresponding to each region; For example, the face is divided into four regions, and the key point index is defined as follows: upper area Points 17 to 26; Lower region Points 48 to 59; Left area : Point set {0,1,2,3,17,18,19,20,21,36,37,38,39,40,41,48,60}; Right region : Point set {13,14,15,16,22,23,24,25,26,42,43,44,45,46,47,54,64}.

[0033] Calculate the geometric centroid of each region; For example, the geometric centroid of each region is calculated as follows: ; in, Corresponding key point set , Indicates the number of key points.

[0034] Calculate the Euclidean distance time series of the geometric centroid changes between adjacent frames in each region; For example, the Euclidean distance time series is represented as follows: , .

[0035] The facial motion intensity characteristics are obtained by calculating the average geometric centroid movement distance of each region. For example, the average geometric centroid movement distance for each region is represented as follows: .

[0036] Facial feature symmetry features are obtained by calculating the Pearson correlation coefficient of the geometric centroids of the left and right facial regions along the horizontal x-axis within the first f frames; For example, facial symmetry features are obtained by calculating the Pearson correlation coefficient between the centroids of the left and right facial regions along the horizontal x-axis within the first second (f frames): ; in, For frame f, , This represents the mean value within the window.

[0037] The emotion stability value feature is obtained by using a lightweight emotion recognition model to perform five-class emotion classification on a normalized first-person perspective frame sequence to obtain an emotion label sequence, and calculating the average value of the emotion labels based on the indicator function to obtain the emotion stability value feature. For example, lightweight emotion recognition models can be selected from OpenVINO, MobileFaceNet, Noldus, or DeepEmotion.

[0038] For example, the emotion stability value feature is applied to the normalized sequence using a lightweight emotion recognition model. Above, perform a five-category classification of emotions to obtain an emotion label sequence: ; ; in, It is an indicator function; it is 1 if the preceding and following emotion labels are different, and 0 otherwise.

[0039] Construct an overall facial feature vector and a temporal facial feature matrix, including facial motion intensity features, facial symmetry features, and emotional stability value features of each region, to form facial quantitative features; For example, the overall facial feature vector is represented as follows: ; Facial temporal features are represented as follows: ; To ensure timing alignment, any shortfall in length is padded with zeros.

[0040] Extract and obtain a set of effective speech segments for the target child from the audio data. Divide each effective speech segment in the set into frames with a preset window length and a preset frame shift. Extract the frame-level fundamental frequency, loudness, and the top three formants and construct speech quantization features. As one possible implementation, extracting and obtaining a set of effective speech segments from the audio data for the target child specifically includes: Noise reduction, temporal reconstruction, and beamforming are performed on audio data to obtain enhanced audio data; For example, a dual-microphone array is used to acquire interactive audio data, and multi-channel speech enhancement is performed to improve the signal-to-noise ratio, including spectral subtraction noise reduction and GCC-PHAT beamforming; Spectral subtraction, subtracting the signal from each channel Short-time Fourier transform (STFT) was performed with a window length of 25 ms and a frame shift of 10 ms to calculate the noise power spectrum of the silent segment: ; in, This represents the number of silent frames. No. Frame corresponding spectrum; Calculate the filter function: ; Enhanced spectrum for each channel Reconstructing the time-domain signal using inverse STFT; The inter-microphone delay was estimated using the GCC-PHAT beamforming method and then superimposed to obtain a single-channel enhanced signal. ; Extracting speaker embedding vectors from enhanced audio signals; For example, on the enhanced signal, a pre-trained ECAPA-TDNN model is used to extract the voiceprint embedding vector for a length of 1 second and a step size of 0.5 seconds. ; Calculate the cosine similarity between the voiceprint embedding vector and the pre-configured target child's voiceprint vector; For example, the voiceprint embedding vector is compared with the pre-registered target child's voiceprint vector. Calculate cosine similarity: ; Speech segments corresponding to voiceprint embedding vectors with cosine similarity greater than or equal to the threshold are identified as valid speech segments. Consecutive adjacent valid speech segments are merged to obtain a set of valid speech segments. For example, the cosine similarity threshold can be set as follows: ; If similarity This period was determined to be the time when the target child was speaking; Consecutive adjacent valid time segments are merged, and short segments with a duration of less than 300 ms are removed to obtain a set of valid speech segments: .

[0041] As one possible implementation, the fundamental frequency, loudness, and top three formants at the frame level are extracted and used to construct speech quantization features, specifically including: The YIN algorithm is used to extract the frame-level fundamental frequency, and zeros are padded to the undetected frame-level fundamental frequency to obtain the fundamental frequency of the frame-level fundamental frequency; For example, for each valid speech segment The enhanced signal was framed with a window length of 25 ms and a frame shift of 10 ms to extract key acoustic features; The YIN algorithm (sampling rate 48 kHz, search range 50–500 Hz, confidence threshold 0.8) is used to extract the frame-level fundamental frequency. Undetected frame fundamental frequencies are padded with zeros, resulting in: ; in, For the fundamental frequency; Calculate frame-level loudness under ISO 226 standard A-weighting; For example, calculate the frame-level loudness under ISO 226 standard A-weighting and output it in dB: ; in, Loudness; Based on 12th-order LPC analysis, pole frequencies are extracted to obtain the first three resonance peaks. The frequency range of the first three resonance peaks is limited, and zeros are padded for those that do not conform to the range. The sequence of the first three resonance peaks is then output. For example, pole frequencies are extracted based on 12th-order LPC analysis to obtain the top three resonance peaks. The frequency range is limited to: ; Pad with zeros if the value does not meet the range, output sequence: ; The average value of the feature sequences of frame-level fundamental frequency, frame-level loudness, and frame-level top three formants is calculated in the time dimension to obtain the overall speech feature vector. At the same time, a speech temporal feature sequence is constructed, and the speech quantization feature is composed of the overall speech feature vector and the speech temporal feature matrix. For example, the overall feature vector of speech is expressed as: ; The speech temporal feature sequence is expressed as: ; The overall speech feature vector and the speech temporal feature matrix constitute the speech quantization features.

[0042] Get the set of timestamps corresponding to the set of valid speech segments of the target child. The transcribed text at the corresponding time in the timestamp set is obtained, the word sequence and sentence sequence of each segment of the transcribed text are obtained, and speech rate features, average sentence length features, syntactic complexity features and repetition rate features are extracted and text quantification features are constructed. For example, the expression for the set of timestamps for the target child's speech segments is: ; The corresponding transcribed text expression at that time is: ; The text was obtained through transcription by an automatic speech recognition model and preprocessed, including removing meaningless pause words (such as "um", "ah", etc.) and automatically segmenting sentences while retaining punctuation. For each paragraph of text; For each text segment The word sequence is obtained using Chinese word segmentation (HanLP / jieba): ; in, For word count; Simultaneously, use periods, question marks, exclamation marks, etc., to break down the sentences and obtain the sentence sequence: in, The number of sentences.

[0043] As one possible implementation, text quantization features include a text overall feature vector and a text temporal feature matrix; wherein, the text overall feature vector is composed of the average values ​​of word sequences and sentence sequences extracted during the entire interaction process, as well as speech rate features, average sentence length features, syntactic complexity features, and repetition rate features; For example, speech rate features quantify the fluency of children's language output in natural interaction scenarios, calculated as follows: ; The average sentence length characteristic represents children's sentence organization ability, and is calculated as follows: ; Syntactic complexity features are based on dependency parsing, which counts the number of clauses in each text segment. Number of times the modifying relationship appears Number of times parallel relationships appear And calculate: ; The repetition rate feature is defined as follows: The vocabulary set after removing stop words The total number of effective words is recorded as Count the number of words that appear more than once. And calculate: ; The overall feature vector is defined as the average of all features throughout the entire interaction process, forming the overall language structured feature vector: ; The text temporal feature matrix consists of the speech rate sequence, average sentence length sequence, syntactic complexity sequence, and repetition rate sequence of K speech segments; For example, the time series feature matrix Output the speech rate sequence for each of the K corresponding speech segments. Average sentence length sequence Syntactic complexity sequence Repetition rate sequence If no valid transcribed content is available, zeros are automatically added to ensure sequence alignment.

[0044] Two-dimensional motion trajectories of target children are obtained based on interactive videos. Spatial edge proportion features, activity features, center of gravity stability features, upper limb activity features are calculated, and posture quantification features are constructed. For example, the two-dimensional motion trajectory is denoted as ; The pose quantization feature is denoted as ; Corresponding to spatial edge proportion characteristics, activity characteristics, center of gravity stability characteristics, and upper limb activity characteristics; The attitude temporal feature matrix is ​​denoted as Missing frames are padded with zeros to ensure timing alignment.

[0045] As one possible approach, a wide-angle camera is placed at the top center of the interactive space to simultaneously capture the entire global interactive video sequence, while eight cameras are placed around the top center to simultaneously capture the entire multi-view video sequence. For example, via the top wide-angle camera Synchronous acquisition of the entire interactive video sequence And by eight surrounding cameras Synchronous acquisition of video sequences .

[0046] The two-dimensional motion trajectory is obtained as follows: the center coordinates of the bounding box of the target child in each frame are obtained from the entire global interactive video sequence using the YOLOv5 detection model, and the two-dimensional motion trajectory is constructed from the center coordinates of the bounding box of the target child in each frame; this is used to visualize the spatial movement pattern of the child.

[0047] For example, the center coordinates can be denoted as: ; The two-dimensional motion trajectory is expressed as: ; The spatial edge proportion characteristic is obtained by calculating the proportion of target children staying in the edge area; For example, let the room resolution be... Define the edge width: ; If the child's center coordinates fall within the edge region: ; Then calculate the spatial edge proportion characteristics: ; The activity feature is calculated as follows: the overall activity feature is obtained by calculating the displacement of consecutive frames, and the average value of the overall activity feature is obtained. For example, the overall activity feature is obtained by calculating the shifts between consecutive frames: ; The activity feature is obtained by averaging the overall activity features: ; Multiplying by the frame rate f converts pixels / frame to pixels / second.

[0048] The calculation method for the center of gravity stability feature is as follows: obtain a temporally continuous and identity-stable main view frame sequence using the filtering rules of the main view image; extract N body key points from each frame to form a key point set; define the center of gravity stability region and its corresponding key points; calculate the center of gravity of each frame for each center of gravity stability region; calculate the inter-frame center of gravity offset based on the center of gravity of each frame; and obtain the average value of the inter-frame center of gravity offset to obtain the center of gravity stability feature. For example, in the feature extraction of the surrounding 8 cameras, an automatic main view selection mechanism consistent with facial data recognition is automatically adopted (based on YOLOv5 + OSNet, threshold 0.6, hold time 4 s, switching gain threshold 0.06) to ensure stable and continuous feature calculation; After selecting the main viewpoint, use OpenPose to extract a set of 18 body keypoints per frame: ; The set of key points for centroid stability is defined as follows: ; Calculate the centroid of each frame: ; Inter-frame centroid shift: ; Calculate the average value of the inter-frame centroid offset to obtain the centroid stability characteristics: ; Multiplying by the frame rate f converts the center of gravity displacement of each frame from "pixels / frame" to "pixels / second", supporting numerical comparisons at different video frame rates.

[0049] The calculation method for upper limb activity characteristics is as follows: Define the upper limb activity area and its corresponding key points; For example, the active area of ​​the upper limb and its corresponding key points are defined as follows: ; Calculate the geometric centroid of the active area of ​​the upper limb; For example, the geometric centroid is represented as follows: ; Calculate the distance difference between the geometric centroids of active upper limb regions between frames; For example, the formula for calculating the distance difference is as follows: Calculate the average distance difference to obtain upper limb activity characteristics; For example, the upper limb activity characteristic is expressed as: .

[0050] The fused quantization features are obtained by fusing facial quantization features, speech quantization features, text quantization features, and pose quantization features; For example, facial quantization features, speech quantization features, text quantization features, and pose quantization features are concatenated into vectors in sequence, as shown in the following expression: ; Fusion quantization feature vector This allows for the large-scale model to perform overall risk assessment.

[0051] Temporal feature fusion, using a video frame rate of f = 60Hz as a unified reference time base, performs frame-level alignment on the temporal features of the four quantization features to obtain: ; For speech quantization features and text quantization features, frame matching is required to fill the corresponding frame with the mean value of the feature value in the corresponding window, and fill the uncovered frame with zeros to maintain time alignment. Temporal feature fusion matrix This provides time for large models to perform dynamic behavior pattern recognition and anomaly localization.

[0052] Standardization is performed before fusion to ensure uniformity of feature dimensions across different modalities and stable training. ; in and These are the mean and standard deviation of the training set, respectively, while retaining the original dimensional range for interpretability of the results.

[0053] Configure a pre-trained large language model and input the fused quantized features into the large language model to obtain ASD risk probability, multi-dimensional quantized scoring vector, frame-level anomaly probability sequence and micro-anomaly localization timestamp set.

[0054] For example, the open-source LLaMA-3-8B large language model was selected as the core architecture for fine-tuning. This model is based on the transformer encoder-decoder architecture, with a parameter scale of 8B, which can fully explore the multimodal comprehensive behavioral characteristics of children under natural interaction to assist in ASD risk assessment.

[0055] For fine-tuning, a dataset of 500 labeled ASD cases from top-tier hospitals was used as training data. The AdamW optimizer was employed, with a learning rate set to [value missing]. The batch size is 8, the training epochs are 20, the classification prediction uses BCE Loss, the regression scoring uses Smooth L1 Loss, and the anomaly detection uses Focal Loss (γ=2) to deal with the class imbalance problem during anomaly detection.

[0056] For the temporal feature matrix, a 4-layer transformer encoder (hidden=1024, head=8) structure is adopted and learnable positional encoding is used to preserve frame-level temporal context information.

[0057] In the overall risk prediction and quantitative scoring section, Input the fine-tuned feedforward prediction network and calculate the ASD risk probability: ; in Use the Sigmoid activation function; , To train the optimized weights and bias parameters, the prediction results are... A higher value indicates a higher risk of childhood ASD. It also outputs a multi-dimensional quantitative score: ; in, To train the optimized weights and bias parameters. These scores correspond to scores for emotion expression, motor coordination, language organization, and activity level, respectively, and are used to quantify children's performance in multimodal behavioral dimensions in natural interaction scenarios, facilitating intuitive clinical interpretation.

[0058] In the microscopic anomaly detection and localization section, the temporal fusion feature matrix is ​​used. Input a temporal transformer encoder, output a frame-level embedding representation. And the frame-level anomaly probability is calculated through the prediction layer: ; in, Optimize the parameters after training. Set a threshold. ,when The system determines if a micro-behavioral anomaly exists in the current frame. It then merges time periods where consecutive frames are marked as anomalies (marked as 1) to generate the micro-anomaly localization result. ; It is used to directly output the periods of suspicious abnormal behavior during video playback, making it easier for doctors to accurately view and further interpret the data.

[0059] The final output is the probability of ASD risk. Multi-dimensional quantitative scoring vector Frame-level anomaly probability sequence and the set of timestamps for locating micro-anomalies It maintains index synchronization and alignment with video and other modal results, enabling static overall assessment of ASD risk in children and dynamic micro-behavioral detection and localization. It facilitates non-invasive quantitative screening and long-term follow-up monitoring in natural interactive scenarios, providing support for rapid and objective clinical screening of ASD risk in children and long-term quantitative tracking, and has practical clinical application value.

[0060] Secondly, this invention provides a quantitative analysis system for children's social behavior, see [link to relevant documentation]. Figure 3 It includes: a multimodal behavior data acquisition module, which collects interactive videos and audios in a natural state, and obtains multimodal social behavior data of children in a natural state from the interactive videos and audios. The multimodal social behavior data includes facial data, audio data, text data and posture data; For example, the data acquisition module includes multiple directional cameras positioned around the perimeter of the data acquisition room and a wide-angle camera at the center of the ceiling. Two microphone arrays in different locations are also set up to record children's facial expressions, body movements, and voice signals during natural interactions with parents or teachers. The entire data acquisition process is free from paradigm restrictions or task guidance, ensuring ecological adaptability and high authenticity. The system also features an identity verification function to ensure the accuracy of subsequent individual feature extraction and continuous analysis.

[0061] The behavioral pattern feature quantization module includes a facial feature quantization unit, a speech feature quantization unit, a text feature quantization unit, and a posture feature quantization unit; Among them, the facial feature quantization unit is used to identify the target child's face based on facial data, filter the main view image, extract facial motion intensity features, facial symmetry features, and emotional stability value features, and construct facial quantification features; For example, the facial feature quantization unit acquires video sequences through multiple cameras, automatically identifies the target child's face and filters the main view images, extracts quantitative features that reflect the intensity of facial movement, symmetry and emotional stability, and uses them to describe the child's facial expressions and emotional regulation ability, providing input for multimodal behavior analysis.

[0062] The speech feature quantization unit is used to extract and obtain the set of effective speech segments of the target child from the audio data. Each effective speech segment in the set of effective speech segments is divided into frames with a preset window length and a preset frame shift. The frame-level fundamental frequency, loudness and the top three formants are extracted and the speech quantization features are constructed. For example, the speech feature quantization unit achieves audio noise reduction and target speech separation through a multi-channel microphone array, extracting acoustic features such as fundamental frequency, loudness, and formants of children's pronunciation to characterize language rhythm, tone control, and language expression characteristics.

[0063] For example, the speech feature quantization unit uses a dual-microphone array to collect speech with a high signal-to-noise ratio, and combines voiceprint-assisted segmentation and standardized feature extraction technology to automatically locate the target child's speech in a natural interaction scene and extract key prosodic features, providing structured input for multimodal fusion and ASD behavior analysis; The speech feature quantization unit enables automatic recognition, localization, and simultaneous extraction of multi-dimensional acoustic features of target children's speech in natural interaction scenarios, providing standardized input for multimodal fusion and behavior analysis.

[0064] The text feature quantization unit is used to obtain the timestamp set corresponding to the effective speech segment set of the target child, obtain the transcribed text at the corresponding time in the timestamp set, obtain the word sequence and sentence sequence of each segment of the transcribed text, extract speech rate features, average sentence length features, syntactic complexity features and repetition rate features, and construct text quantization features; For example, the text feature quantization unit extracts language content features (speech rate, average sentence length, sentence complexity, repetition rate) of the target child through speech transcription and speaker separation to characterize their language organization and semantic expression abilities.

[0065] For example, the text feature quantification unit extracts the language organization features of the target child from the transcribed text, reflecting their language fluency, syntactic complexity, sentence organization ability and expression repetition tendency in natural social scenarios, providing language modality input for multimodal fusion analysis and behavior screening.

[0066] The posture feature quantization unit is used to obtain the two-dimensional motion trajectory of the target child based on interactive video, calculate the spatial edge proportion feature, activity feature, center of gravity stability feature, upper limb activity feature, and construct posture quantization feature; For example, the posture feature quantification unit integrates multi-view data and extracts quantitative features of children's motor behavior and spatial activities (motor trajectory, spatial edge activity tendency, overall activity level, center of gravity stability, and upper limb activity level) through skeleton posture estimation and automatic main view selection mechanism. These features are used to characterize children's motor coordination, spatial behavior patterns, and motor performance.

[0067] The multimodal feature fusion module fuses facial quantization features, speech quantization features, text quantization features, and pose quantization features to obtain fused quantization features; For example, the overall features extracted by the facial, speech, text and pose feature quantization units are uniformly time-aligned and standardized for fusion with temporal features. Through uniform frame rate alignment, Z-score standardization and structured splicing, a multimodal overall fusion feature vector and a temporal fusion feature matrix are obtained to support subsequent intelligent screening and evaluation.

[0068] The large model evaluation module is configured with a pre-trained large language model. The fused quantized features are input into the large language model to obtain the ASD risk probability, multi-dimensional quantized scoring vector, frame-level anomaly probability sequence and micro-anomaly localization timestamp set. For example, the large model evaluation module receives a multimodal overall fusion feature vector and a temporal fusion feature matrix. It performs ASD risk assessment and anomaly detection through a large language model combined with a temporal Transformer. The fine-tuned large model performs global risk prediction and multi-dimensional scoring on the overall fusion feature vector, and performs frame-level anomaly detection and micro-anomaly localization on the temporal fusion feature matrix. The output includes the child's ASD risk probability, quantitative scores of key dimensions (emotional expression, motor coordination, language organization, and activity level), frame-level anomaly probability sequence, and set of abnormal time periods, providing support for intelligent screening and clinical quantitative assessment.

[0069] Thirdly, this invention provides an application of a quantitative analysis method for children's social behavior, which is used for autism screening.

[0070] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the description of the drawings, in carrying out the claimed invention. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several of the functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0071] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for quantitative analysis of children's social behavior, characterized in that, Includes the following steps: Collect interactive videos and audio in natural states, and obtain multimodal social behavior data of children in natural states from the interactive videos and audio. Multimodal social behavior data includes facial data, audio data, text data and posture data. The process involves identifying target children's faces based on facial data, filtering first-person view images, extracting facial motion intensity features, facial symmetry features, and emotional stability value features, and constructing facial quantitative features. Extract and obtain a set of effective speech segments for the target child from the audio data. Divide each effective speech segment in the set of effective speech segments into frames with a preset window length and a preset frame shift. Extract the frame-level fundamental frequency, loudness, and the top three formants and construct speech quantization features. Obtain the timestamp set corresponding to the effective speech segments of the target child, obtain the transcribed text at the corresponding time in the timestamp set, obtain the word sequence and sentence sequence of each segment in the transcribed text, extract speech rate features, average sentence length features, syntactic complexity features and repetition rate features, and construct text quantification features; Two-dimensional motion trajectories of target children are obtained based on interactive videos. Spatial edge proportion features, activity features, center of gravity stability features, upper limb activity features are calculated, and posture quantification features are constructed. The fused quantization features are obtained by fusing facial quantization features, speech quantization features, text quantization features, and pose quantization features; Configure a pre-trained large language model and input the fused quantized features into the large language model to obtain ASD risk probability, multi-dimensional quantized scoring vector, frame-level anomaly probability sequence and micro-anomaly localization timestamp set.

2. The method for quantitative analysis of children's social behavior according to claim 1, characterized in that, Interactive video is a sequence of interactive videos captured simultaneously by multiple cameras; multiple frames of multi-view images are obtained from the interactive video sequence to form facial data; Facial recognition of target children based on facial data, specifically including: For each frame of multi-view image, a face recognition model and a calibrated frontal photo of the target child configured in the face recognition model are applied to generate a reference embedding vector; The face detection model is used to identify the face regions in each viewpoint of each frame of multi-view images; and the face regions in each viewpoint are input into the face recognition model to extract the face embedding vector of each frame and each viewpoint. Calculate the cosine similarity between the face embedding vector and the reference embedding vector for each frame and each viewpoint; The facial regions with a cosine similarity greater than or equal to the similarity threshold from that viewpoint are identified as the target child, and multiple facial regions from that viewpoint constitute a set of valid candidate viewpoint images.

3. The method for quantitative analysis of children's social behavior according to claim 2, characterized in that, Selecting images from the main perspective involves the following steps: Calculate the quality score of each valid candidate viewpoint image within a preset time frame. The selection rules for the main viewpoint images are constructed, specifically including: selecting the viewpoint with the highest quality score in the initial frame; In subsequent frames, if the quality score of another viewpoint improves beyond a threshold while maintaining the preset time of the current frame's main viewpoint, the viewpoint is switched; otherwise, the current viewpoint is maintained. Based on the above filtering rules, a main-view frame sequence that is temporally continuous and has stable identity is obtained. The main view sequence was affinely normalized to standard pixel size to obtain a normalized main view frame sequence.

4. The method for quantitative analysis of children's social behavior according to claim 3, characterized in that, The process of extracting facial motion intensity features, facial symmetry features, and emotional stability value features, and constructing facial quantitative features, specifically includes the following steps: The calculation of facial motion intensity features specifically includes: extracting key points from each frame in a normalized main view frame sequence to obtain a set of key points; determining N regions of the face and the key points corresponding to each region; calculating the geometric centroid of each region; calculating the Euclidean distance time series of the change of geometric centroid of each region in adjacent frames; and calculating the average geometric centroid movement distance of each region to obtain the facial motion intensity features. Facial feature symmetry features are obtained by calculating the Pearson correlation coefficient of the geometric centroids of the left and right facial regions along the horizontal x-axis within the first f frames; The emotion stability value feature is obtained by using a lightweight emotion recognition model to perform five-class emotion classification on a normalized first-person perspective frame sequence to obtain an emotion label sequence, and calculating the average value of the emotion labels based on the indicator function to obtain the emotion stability value feature. We construct an overall facial feature vector and a temporal feature matrix, which include facial motion intensity features, facial symmetry features, and emotional stability value features in each region, to form facial quantitative features.

5. The method for quantitative analysis of children's social behavior according to claim 1, characterized in that, Extracting and obtaining a set of effective speech segments from the audio data for the target child, specifically including: Noise reduction, temporal reconstruction, and beamforming are performed on audio data to obtain enhanced audio data; Extracting speaker embedding vectors from enhanced audio signals; Calculate the cosine similarity between the voiceprint embedding vector and the pre-configured target child voiceprint vector; Speech segments corresponding to voiceprint embedding vectors with cosine similarity greater than or equal to a threshold are identified as valid speech segments. Consecutive adjacent valid speech segments are merged to obtain a set of valid speech segments.

6. The method for quantitative analysis of children's social behavior according to claim 5, characterized in that, Extracting frame-level fundamental frequency, loudness, and the top three formants, and constructing speech quantization features, specifically including: The YIN algorithm is used to extract the frame-level fundamental frequency, and zeros are padded to the undetected frame-level fundamental frequency to obtain the fundamental frequency of the frame-level fundamental frequency; Calculate frame-level loudness under ISO 226 standard A-weighting; Based on 12th-order LPC analysis, pole frequencies are extracted to obtain the first three resonance peaks. The frequency range of the first three resonance peaks is limited, and zeros are padded for those that do not conform to the range. The sequence of the first three resonance peaks is then output. The average value of the feature sequences of frame-level fundamental frequency, frame-level loudness, and the top three formants of the frame is calculated in the time dimension to obtain the overall speech feature vector. At the same time, a speech temporal feature sequence is constructed, and the speech quantization feature is composed of the overall speech feature vector and the speech temporal feature matrix.

7. The method for quantitative analysis of children's social behavior according to claim 1, characterized in that, The text quantification features include the overall text feature vector and the text temporal feature matrix; among them, the overall text feature vector is composed of the average values ​​of the word sequence and sentence sequence extracted during the entire interaction process, as well as the speech rate feature, average sentence length feature, syntactic complexity feature, and repetition rate feature. The text temporal feature matrix consists of the speech rate sequence, average sentence length sequence, syntactic complexity sequence, and repetition rate sequence of K speech segments.

8. The method for quantitative analysis of children's social behavior according to claim 1, characterized in that, A wide-angle camera is placed at the top center of the interactive space to simultaneously capture the entire interactive video sequence. At the same time, eight cameras are placed around the top center to simultaneously capture the entire multi-view video sequence. The two-dimensional motion trajectory is obtained as follows: the YOLOv5 detection model is used to obtain the center coordinates of the bounding box of the target child in each frame from the entire global interactive video sequence, and the two-dimensional motion trajectory is formed by the center coordinates of the bounding box of the target child in each frame. The spatial edge proportion characteristic is obtained by calculating the proportion of target children staying in the edge area; The activity feature is calculated as follows: the overall activity feature is obtained by calculating the displacement of consecutive frames, and the average value of the overall activity feature is obtained. The calculation method for the center of gravity stability feature is as follows: obtain a temporally continuous and identity-stable main view frame sequence using the filtering rules of the main view image; extract N body key points from each frame to form a key point set; define the center of gravity stability region and its corresponding key points; calculate the center of gravity of each frame for each center of gravity stability region; calculate the inter-frame center of gravity offset based on the center of gravity of each frame; and obtain the average value of the inter-frame center of gravity offset to obtain the center of gravity stability feature. The calculation method for upper limb activity features is as follows: define the upper limb activity region and its corresponding key points; calculate the geometric centroid of the upper limb activity region; calculate the distance difference between the geometric centroids of the upper limb activity regions between frames; and calculate the average value of the distance difference to obtain the upper limb activity features.

9. A quantitative analysis system for children's social behavior, characterized in that, include: The multimodal behavior data acquisition module collects interactive videos and audios in a natural state, and obtains multimodal social behavior data of children in a natural state from the interactive videos and audios. The multimodal social behavior data includes facial data, audio data, text data and posture data. The behavioral pattern feature quantization module includes a facial feature quantization unit, a speech feature quantization unit, a text feature quantization unit, and a posture feature quantization unit. The facial feature quantization unit is used to identify the target child's face based on facial data, filter first-person perspective images, extract facial motion intensity features, facial symmetry features, and emotional stability value features, and construct facial quantization features. The speech feature quantization unit is used to extract and obtain a set of effective speech segments from the audio data of the target child, divide each effective speech segment in the set into frames with a preset window length and preset frame shift, extract the frame-level fundamental frequency, loudness, and top three formants, and construct speech quantization features. The text feature quantization unit is used to obtain the timestamp set corresponding to the set of effective speech segments of the target child, obtain the transcribed text at the corresponding time in the timestamp set, obtain the word sequence and sentence sequence of each segment in the transcribed text, extract speech rate features, average sentence length features, syntactic complexity features, and repetition rate features, and construct text quantization features. The posture feature quantization unit is used to obtain the two-dimensional motion trajectory of the target child based on interactive video, calculate spatial edge proportion features, activity features, center of gravity stability features, and upper limb activity features, and construct posture quantization features. The multimodal feature fusion module fuses facial quantization features, speech quantization features, text quantization features, and pose quantization features to obtain fused quantization features; The large model evaluation module configures a pre-trained large language model and inputs the fused quantized features into the large language model to obtain ASD risk probability, multi-dimensional quantized scoring vector, frame-level anomaly probability sequence, and micro-anomaly localization timestamp set.

10. An application of a quantitative analysis method for children's social behavior, characterized in that, A quantitative analysis method for children's social behavior is used for autism screening.

Images