ASD early auxiliary diagnosis model construction method based on HPE and multi-source and multi-mode asynchronous data fusion

Abstract

The invention discloses an ASD early auxiliary diagnosis model construction method based on HPE and multi-source and multi-mode asynchronous data fusion. The method comprises the steps that multi-source and multi-mode asynchronous data are acquired and enhanced through gamma correction and CHALE; extracting different modal features, including Pose / AU / Gaze features, rPPG signals and human body posture features; constructing a cross-modal fusion module to map the modal features to a unified dimension, calculating attention weights of all time steps of the modal features, generating aligned cross-modal features, and finally performing residual fusion-graph convolution operation on the original modal features and the aligned cross-modal features to output global fusion features; and inputting the global fusion features into a full-connection classification network, calculating a prediction probability by using a Sigmoid function, and carrying out auxiliary diagnosis to complete construction of an ASD early auxiliary diagnosis model. According to the method, an asynchronous alignment mechanism of cross-modal time attention is introduced after modal coding, and behavioral signals and physiological signals are fused, so that the model discrimination capability is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, computer vision and medical auxiliary diagnosis, and particularly relates to an ASD early auxiliary diagnosis model construction method based on HPE and multi-source multi-modal asynchronous data fusion. BACKGROUND

[0002] Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by social communication impairment, narrow interests and stereotyped behavior. Early intervention has a significant effect on improving the language, social and cognitive abilities of patients. However, traditional diagnosis relies on clinical experts through behavior observation, questionnaire and interview, which has the disadvantages of strong subjectivity, long time consumption and difficulty in standardization.

[0003] With the development of computer vision and deep learning, automatic diagnosis using video, speech, facial expression and physiological signals has become a research hotspot. However, the existing methods have the following problems: It is difficult to synchronize the collection of multi-modal data, and there are time step differences and missing data among different modalities; Single modality (such as only using facial expression or speech) cannot fully reflect the social interaction behavior characteristics of children; Therefore, there is an urgent need for a model method that integrates human pose recognition and multi-source multi-modal asynchronous data fusion to improve the objectivity, accuracy and practicality of early ASD auxiliary diagnosis. SUMMARY

[0004] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide an ASD early auxiliary diagnosis model construction method based on HPE and multi-source multi-modal asynchronous data fusion.

[0005] In order to achieve the above purpose, the following technical solutions are adopted: In one aspect of the present application, an ASD early auxiliary diagnosis model construction method based on HPE and multi-source multi-modal asynchronous data fusion is provided, comprising the following steps: Obtain a multi-source multi-modal asynchronous data set and enhance it through adaptive local contrast enhancement and nonlinear gamma correction; the multi-source multi-modal asynchronous data includes video data recorded by different types of devices; Extract the modal features in the multi-source multi-modal asynchronous data set obtained after data enhancement; the modal features include head pose features, facial action unit features, gaze direction features, rPPG signals and human pose features; A modal encoder module is constructed to map the modal features to a unified dimension; a cross-modal time attention alignment module is constructed to calculate the attention weights of all time steps of the modal features and generate aligned cross-modal features; a cross-modal fusion module is constructed to perform residual fusion on the original modal features and the aligned cross-modal features and further fuse through a graph convolution operation to output global fusion features; The global fusion features are input into a fully connected classification network, and a Sigmoid function is used to calculate the prediction probability and perform auxiliary diagnosis, thereby completing construction of an ASD early auxiliary diagnosis model.

[0006] As a preferred technical solution, the nonlinear gamma correction is specifically: A gamma correction lookup table is constructed: gamma parameters are set, and for input pixel values i ∈[0,255], the mapped pixel values are calculated according to the formula .

[0007] As a preferred technical solution, the adaptive local contrast enhancement is specifically: The input video frame is converted from an RGB color space to an LAB color space, in which the L channel independently represents luminance information, and the A / B channels represent chrominance information, respectively. The CLAHE operation is only implemented on the L channel: a contrast limit threshold is set, the image is divided into a plurality of regions using a local grid division strategy, and a histogram is independently calculated and equalized in each region; The processed L channel and the unprocessed A / B chroma channels are recombined and converted back to the RGB color space.

[0008] As a preferred technical solution, the head pose feature, the facial action unit feature, and the gaze direction feature are obtained by feature extraction on the face region, specifically: The head pose feature is obtained by a three-dimensional face key point detection and fitting method, which obtains the pose parameters of the head in a three-dimensional space, including a yaw angle, a pitch angle, and a roll angle. The facial action unit feature is obtained by detecting and quantifying the intensity value of the facial action unit, which includes eyebrow lifting, frown, and mouth up. The gaze direction feature is obtained by an eye tracking method, which locates the eye key points and estimates the gaze direction vector to obtain the gaze point position and angle information.

[0009] As a preferred technical solution, the rPPG The signal is obtained by the following steps: The face region is detected using MTCNN, and the skin pixels are segmented by U-Net to reserve the blood flow sensitive region; the frame sequence after reserving the blood flow sensitive region is normalized and time-aligned, and then down-sampling and up-sampling in the time dimension are performed, specifically: ; wherein, HR is the original heart rate, interval , stride and padding are sampling parameters; A 5-layer 3D convolutional encoder is used to extract multi-scale spatio-temporal features; A 3D deconvolutional decoder is used to recover the time sequence information and generate rPPG signals; A multi-scale encoding-decoding network based on three-dimensional convolution is constructed, including an encoder and a decoder; the encoder includes several layers of three-dimensional convolution and pooling operations, and a residual structure is set; the decoder gradually recovers the time dimension through three-dimensional deconvolution, and generates a rPPG waveform sequence consistent with the length of the frame sequence through global average pooling and one-dimensional convolution smoothing; In the multi-scale encoding-decoding network, the following is introduced: frame-by-frame spatial-channel attention (3D-S / T): for the spatial features of each frame, the significant regions related to rPPG are enhanced; global time sequence-space attention (3D-S-T): for the spatio-temporal features of the entire video, the periodic pulse fluctuation pattern is highlighted; According to the rPPG waveform sequence, the power spectral density analysis method is used to calculate the heart rate: HR = argmaxPSD ( f )×60; wherein, PSD ( f ) is the power spectral density of the rPPG waveform sequence, and the frequency corresponding to the peak value f is converted into the heart rate value per minute.

[0010] As a preferred technical solution, the human posture features are extracted by the following steps: The input video frame sequence is subjected to human key point detection by HRNet, and the two-dimensional coordinates of each human key point are output P 2D ( t )={( x i ( t ), y i ( t ))| i =1,..., K}, t =1,..., T ; wherein,K Indicates the number of key points; Two-dimensional joint sequence Input a temporally convolutional 3D pose estimation network to obtain 3D joint coordinates. P 3D ( t )= f θ ({ P 2D ( t - k ),..., P 2D ( t + k )}),in, f θ (•) represents a 3D pose regression network, 2 k +1 represents the receptive field length of the temporal convolution; the output 3D human skeleton is... P 3D ( t )={( X i ( t ), Y i ( t ), Z i ( t ))| i =1,..., K},in,( X i , Y i , Z i ) indicates the first i The coordinates of each joint in three-dimensional space; Based on the sequence of the human 3D skeleton, calculate human posture features: velocity features Acceleration characteristics and attitude stability index ,in, , These are the keypoint vectors for the head and torso, respectively.

[0011] As a preferred technical solution, the modal encoder module specifically comprises: ; in, Mode-specific encoder functions; , F m These are the modal features before and after mapping, respectively; It is the set of real numbers;T m For the first m The length of the time dimension of each modality; d For the unified dimension after mapping; face This represents head posture features, facial motion unit features, and gaze direction features. rPPG express rPPG Signal, pose3D It indicates the characteristics of human posture.

[0012] As a preferred technical solution, the cross-modal temporal attention alignment module specifically comprises: For modes m The i Each time step represents , and modality n All time steps Calculate attention weights And generate aligned cross-modal features. ;in, exp It is an exponential function. W Q m , W K n and W V n For learnable projection matrices, j’ For summation index.

[0013] As a preferred technical solution, the cross-modal fusion module specifically comprises: The original modal features Aligned cross-modal features Perform residual fusion: ; Constructing modal diagrams G =( V , E ), where the node set V ={ face , rPPG , pose3D}, edge set E To represent the semantic correlation between modalities, further fusion is achieved through graph convolution operations: ; in, For modal adjacency matrix, For degree matrix, H (L) For the first l Layer node features σ It is a non-linear activation function; The global fusion feature is represented as: ; wherein, Pool t Indicates a pooling operation in the time dimension.

[0014] As a preferred technical solution, the global fusion feature is input into a fully connected classification network, adopts Sigmoid function to calculate the prediction probability, and performs auxiliary diagnosis to complete the construction of the ASD early auxiliary diagnosis model, specifically: The global fusion feature F fusion is input into a fully connected classification network to calculate the prediction probability z = W c F fusion b c ; wherein, W c is a weight matrix, b c is a bias vector; The Sigmoid function is used to calculate the prediction probability ; wherein, σ (•) is a Sigmoid function, exp is an exponential function; Indicates the probability of the model predicting that the individual has ASD; According to the set threshold auxiliary diagnosis is performed: .

[0015] Compared with the prior art, the present application has the following advantages and beneficial effects: (1) The present application introduces an asynchronous alignment mechanism of cross-modal time attention (Cross-Modal Time Attention) after modal encoding. This mechanism adaptively learns the corresponding relationship between different modal time steps in the query-key-value (Q / K / V) mechanism, realizes alignment without the need to interpolate all modalities to the same frequency, and avoids information distortion and high-frequency information loss caused by interpolation. In addition, it can naturally handle the differences in time steps, frame loss or short-term occlusion between modalities, thereby significantly improving the robustness under actual acquisition conditions.

[0016] (2) The present application extracts behavioral signals (facial expressions: AU , pose , gaze ; three-dimensional posture movements) and physiological signals (heart rate: rPPG extracted from HRV ​​Information is fused, and the semantic complementarity between modalities is used to improve the model's discriminative ability. This enhances the ability to discriminate complex behaviors and reduces the risk of misjudgment / missed judgment caused by single modality. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method for constructing an early auxiliary diagnosis model for ASD based on HPE and multi-source multimodal asynchronous data fusion according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the process for feature extraction of the face region in an embodiment of the present invention; Figure 3 This is a schematic diagram of the rPPG signal extraction process in an embodiment of the present invention; Figure 4 This is a schematic diagram of the human posture feature extraction process in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0019] Example: like Figure 1 As shown, this embodiment provides a method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion, including the following steps: S1. Obtain and preprocess a multi-source, multi-modal asynchronous dataset; the preprocessing includes enhancing the multi-source, multi-modal asynchronous dataset using adaptive local contrast enhancement and nonlinear gamma correction.

[0020] Specifically, step S1 includes the following steps: S1.1 Multi-source multimodal asynchronous data acquisition.

[0021] Multi-source, multimodal, asynchronous data mainly refers to video data recorded from camera devices (mobile phone cameras, laptop cameras) of different manufacturers and models. Head pose features, facial motion unit features, and gaze direction features are extracted based on this video data. rPPG Signals and human posture characteristics. The data source is mainly videos recorded by parents using their own devices on the software.

[0022] S1.2 Preprocessing of multi-source, multi-modal asynchronous data, including adaptive local contrast enhancement and nonlinear gamma correction.

[0023] In the video-based multi-modal feature extraction task, the original video often causes the key area details to be blurred due to uneven illumination, insufficient contrast or limited dynamic range, which directly affects the detection accuracy and feature stability of subsequent face analysis algorithms such as OpenFace. Traditional enhancement methods only optimize the brightness or global contrast, and it is difficult to balance local detail enhancement and overall visual quality at the same time.

[0024] S1.2.1, global brightness distribution optimization (gamma correction stage).

[0025] In this stage, the global brightness distribution of the frame image is further adjusted through nonlinear gamma correction, aiming to solve the problem of dark area detail loss or bright area overexposure caused by poor original lighting conditions. Specifically, a gamma correction lookup table (LUT) is constructed: set the gamma parameter (gamma=1.3), and for the input pixel value i ∈[0,255], the mapped pixel value is calculated according to the formula Through pre-generating the lookup table, the nonlinear transformation of the whole image pixel can be efficiently completed, and finally the enhanced frame image with more reasonable global brightness distribution is output.

[0026] S1.2.2, adaptive local contrast enhancement (CLAHE stage).

[0027] On the basis of the first stage of global brightness distribution optimization, the contrast limited adaptive histogram equalization (CLAHE) technology is used to enhance the details of the key areas (such as the face) in a targeted manner. First, the input video frame is converted from the RGB color space to the LAB color space. The L channel of the color space independently represents the brightness information, and the A / B channels represent the chrominance information, respectively. By separating the brightness and chrominance components, the influence of chrominance distortion on the enhancement effect can be avoided. Then, the CLAHE operation is only performed on the L channel. The clipLimit threshold is set to 3.0 to suppress the risk of noise amplification, and the tileGridSize is set to (8, 8) to divide the image into multiple small regions. In each region, the histogram is calculated and equalized independently. This design makes the brightness enhancement effect concentrated in the local dark or over-bright areas (such as the face shadow and highlight reflection area), avoiding the overall over-enhancement problem caused by traditional global histogram equalization. After processing, the enhanced L channel is combined with the original untreated A / B chrominance channels, and converted back to the RGB color space to obtain the preliminary optimized frame image, which has significantly improved local contrast and stable color naturalness. This stage cooperates with the previous gamma correction processing. Gamma correction optimizes the overall picture's light and shade level, while CLAHE solves the problem of insufficient contrast in local areas. Both of them improve the visual quality and feature recognizability of the video frame.

[0028] S2, performing feature extraction on the preprocessed multi-source multi-modal asynchronous data set to obtain different modal features; the modal features include head pose features, facial action unit features, gaze direction features, rPPG signal and human posture features; Specifically, step S2 includes the following steps: S2.1, Pose / AU / Gaze feature extraction.

[0029] As shown in the figure, for the input video data, feature extraction is performed on the face region. Preferably, the open source face behavior analysis tool OpenFace can be used for feature extraction. Figure 2

[0030] Specifically, it includes: Head pose features ( Pose ): three-dimensional face key point detection and fitting method, obtain the head pose parameters in three-dimensional space, including yaw angle, pitch angle and roll angle, used to represent the head orientation and motion state of the learner in the video process; facial action unit features ( Action Units , AU ​): Detect and quantify the intensity values ​​of multiple facial action units, including eyebrow raising, eyebrow lowering, and mouth corner raising, to reflect the learner's facial expression changes and emotional state; preferably, the intensity values ​​(such as AU01 eyebrow raising, AU04 eyebrow lowering, AU12 mouth corner raising, etc.) can be detected using OpenFace's facial expression analysis module. gaze direction characteristics ( Gaze ): Locate key eye points and estimate gaze direction vectors to obtain the learner's gaze point position and angle information, which can be used to reflect their visual attention distribution; preferably, an eye-tracking method based on OpenFace can be used.

[0031] By extracting multi-dimensional temporal features, learners' posture, expression, and gaze behavior can be comprehensively represented at the facial level, providing basic data support for subsequent multimodal feature fusion and learning focus recognition.

[0032] S2.2, rPPG Signal extraction.

[0033] rPPG Signal extraction employs a remote optical volumetric mapping method based on short-time face video. remote photoplethysmography , rPPG A signal extraction method is proposed to reconstruct heart rate-related physiological signals from changes in light intensity on facial skin. This method combines a 3D convolutional network, multi-scale feature extraction, and a spatiotemporal attention mechanism to achieve stable heart rate estimation within a short time window (approximately 5 seconds). Figure 3 As shown, the specific steps are as follows: S2.2.1 Face region detection and preprocessing.

[0034] MTCNN was used to detect facial regions, and skin pixels were segmented using U-Net to preserve blood flow-sensitive areas. The frame sequence after preserving the blood flow-sensitive areas was normalized to 100×100 resolution and temporally aligned.

[0035] S2.2.2 Data augmentation processing.

[0036] To address the issue of uneven heart rate sample distribution, the video frame sequence is downsampled and upsampled in the temporal dimension: Downsampling shortens the signal period, causing the heart rate value to increase proportionally. Upsampling extends the signal period through interpolation, causing the heart rate value to decrease proportionally.

[0037] Its mathematical expression is: ; in, HR The original heart rate, interval, stride , padding are the sampling parameters, respectively. This method extends the frequency range of training samples and improves the robustness of the model to different heart rate intervals.

[0038] A 5-layer 3D convolutional encoder (ConvB_1-5) is used to extract multi-scale spatio-temporal features.

[0039] A 3D deconvolutional decoder (DConv_1-3) restores the temporal information and generates the rPPG signal.

[0040] S2.2.3, Spatio-temporal feature extraction network.

[0041] A multi-scale encoding-decoding network based on 3D convolution is constructed: The encoder part is composed of multiple layers of 3D convolution and pooling operations, which gradually extracts spatio-temporal features of different scales; The residual structure is used to pass key physiological features during downsampling and reduce feature loss; The decoder gradually restores the time dimension through 3D deconvolution, and finally generates a rPPG signal (waveform sequence) consistent with the video length through global average pooling and one-dimensional convolution smoothing.

[0042] S2.2.4, Spatio-temporal fusion attention mechanism.

[0043] Two types of attention mechanisms are introduced in the feature extraction process of the multi-scale encoding-decoding network: Frame-by-frame spatial-channel attention (3D-S / T): For the spatial features of each frame, the significant regions related to rPPG are enhanced; Global temporal-spatial attention (3D-S-T): For the spatio-temporal features of the entire video, the periodic pulse fluctuation pattern is highlighted.

[0044] S2.2.5, rPPG Signal reconstruction and heart rate calculation.

[0045] The temporal features output by the network are subjected to global pooling and convolution smoothing operations to obtain a continuous rPPG signal. Then the power spectral density (PSD) analysis method is used to calculate the heart rate: ; where PSD ( f ) is the power spectral density of the rPPG signal, and the frequency corresponding to the peak f is converted to the heart rate value per minute.

[0046] S2.3, Human posture feature extraction.

[0047] As Figure 4 shown, the extraction of human pose features adopts a depth-based video human pose feature extraction method to obtain two-dimensional and three-dimensional skeleton coordinates from the input video sequence, and generate time-sequenced pose features. The specific steps include: S2.3.1, two-dimensional key point detection.

[0048] The input video frame sequence is subjected to human key point detection. A high-resolution network (HRNet) is adopted as the basic structure to maintain high-resolution feature representation throughout the process and repeatedly fuse between multi-scale features. The output is the two-dimensional coordinates of each key point of the human body: P 2D ( t )={( x i ( t ), y i ( t ))| i =1,..., K}, t =1,..., T ; wherein K represents the number of key points (such as 17 COCO joint nodes).

[0049] S2.3.2, three-dimensional pose reconstruction The two-dimensional joint node sequence is input into a three-dimensional pose estimation network based on time convolution (VideoPose3D) to obtain three-dimensional joint node coordinates through one-dimensional time convolution to capture the time sequence correlation across frames: ; wherein f θ (•) represents a three-dimensional pose regression network, 2 k +1 is the receptive field length of time convolution.

[0050] The three-dimensional skeleton of the human body output by the network is: P 3D ( t )={( X i ( t ), Y i ( t ), Z i ( t ))|​i =1,..., K}; wherein, X i , Y i , Z i denotes the coordinates of the j-th joint in the three-dimensional space. i

[0051] S2.3.3, Temporal pose feature construction.

[0052] Based on the three-dimensional skeleton sequence, further calculate the dynamic feature vector, that is, the human body pose feature to be extracted: Velocity feature: ; Acceleration feature: ; Pose stability index: ; wherein, , are the key point vectors of the head and the torso respectively.

[0053] Through the above method, the three-dimensional skeleton dynamic features of the learner can be automatically extracted from the two-dimensional video sequence, and the velocity, acceleration and pose stability indexes are combined to construct complete human motion temporal features, which provide reliable dynamic behavior signals for subsequent multi-modal feature fusion and learning concentration analysis.

[0054] S3, input each modal feature: Construct a modal encoder module for mapping the modal features to a unified dimension; Construct a cross-modal time attention alignment module for calculating the attention weights of all time steps of the modal features to generate aligned cross-modal features; Construct a cross-modal fusion module for residual fusion of the original modal features and the aligned cross-modal features, and further fusion through graph convolution operation to output global fusion features.

[0055] Specifically, in step S3, the asynchronous fusion step of the extracted features is: S3.1, modal encoder module.

[0056] To realize cross-modal alignment and fusion, the system sets a modal encoder module to map the above-mentioned modal features to a unified dimension d : ; wherein,​ is a modal-specific encoder function; , F m are mapped and unmapped modal features respectively; is a real number set; T m is the length of the time dimension of the m th modal; d is the unified dimension after mapping; face denotes head pose features, facial action unit features, and gaze direction features, rPPG denotes rPPG signals, pose3D denotes human pose features.

[0057] S3.2, Cross-modal temporal attention alignment module.

[0058] To solve the inconsistency of different modalities in sampling rate and time scale, the system sets an asynchronous alignment module based on cross-modal temporal attention. The method is as follows: For the m th time step representation i of the modal , calculate the attention weight with all time steps n of the modal : ; where exp is an exponential function, j’ is a summation index.

[0059] And generate the aligned cross-modal representation: ; where W Q m , W K n and W V n is a learnable projection matrix. This method does not need to force the sampling rate to be aligned, but automatically learns the corresponding relationship between different modal time steps through the attention mechanism, thereby realizing soft alignment.

[0060] S3.3, Cross-modal fusion module.

[0061] After completing the asynchronous alignment, the system performs cross-modal feature interaction and fusion for each modality, which includes: S3.3.1, Cross-modal feature interaction.

[0062] The original features of each modality aligned cross-modal features Residual fusion is performed: ; S3.3.2, modal-level graph convolution fusion.

[0063] Constructing modal graph G =( V , E ), where the node set V ={ face , rPPG , pose3D} and the edge set E represent the semantic correlation between modalities. Further fusion is performed by graph convolution operation: ; wherein is the modal adjacency matrix, is the degree matrix, H (L) is the node feature of the l layer, σ is a nonlinear activation function.

[0064] S3.4, global fusion feature output.

[0065] After cross-modal interaction and graph convolution fusion, the system obtains a unified fusion representation: ; wherein Pool t represents a pooling operation in the time dimension.

[0066] S4, constructing an output part.

[0067] Specifically, step S4 includes the following steps: S4.1, fusion feature input.

[0068] The global fusion feature F fusion is input to a fully connected classification network to calculate its prediction probability in different categories: z = W c • F fusion + b c ; wherein W c is a weight matrix, b c is a bias vector; S4.2, Sigmoid classification.

[0069] use Sigmoid The function calculates the predicted probability: ; Among them, among them, σ (•)for Sigmoid function; This indicates the probability that the model predicts an individual has ASD.

[0070] S4.3 Classification and determination.

[0071] Based on the set threshold A final diagnosis was made: ; In another embodiment of this application, an early auxiliary diagnosis model for ASD based on HPE and multi-source multimodal asynchronous data fusion is also provided, including a data acquisition module, a data augmentation module, a feature extraction module, a feature fusion module, and an output module; The data acquisition module is used to acquire video data recorded by different models of devices; The data augmentation module is used to preprocess multi-source, multi-modal asynchronous data, including adaptive local contrast enhancement and nonlinear gamma correction. The feature extraction module is used to extract features from the preprocessed multi-source multimodal asynchronous dataset to obtain different modal features; the modal features include head pose features, facial action unit features, gaze direction features, etc. rPPG Signals and human posture characteristics; The feature fusion module includes a modal encoder module, a cross-modal temporal attention alignment module, and a cross-modal fusion module. The modal encoder module maps the modal features to a unified dimension. The cross-modal temporal attention alignment module calculates the attention weights of the modal features at all time steps to generate aligned cross-modal features. The cross-modal fusion module performs residual fusion between the original modal features and the aligned cross-modal features, and further fuses them through graph convolution operations to output global fused features.

[0072] The output module is used to input the globally fused features into a fully connected classification network, employing... Sigmoid The function calculates the predicted probability, performs auxiliary diagnosis, and outputs the results. It should be noted that the system provided in the above embodiment is only an example illustrating the division of the functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0073] In another embodiment of the present application, an ASD early auxiliary diagnosis method based on HPE and multi-source multi-modal asynchronous data fusion is also provided, and the ASD early auxiliary diagnosis is performed by using the ASD early auxiliary diagnosis model based on HPE and multi-source multi-modal asynchronous data fusion in the above embodiment.

[0074] It should be understood that parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized in hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0075] The above-described embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above-described embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion, characterized in that, Includes the following steps: A multi-source, multi-modal, asynchronous dataset is acquired and enhanced using adaptive local contrast enhancement and nonlinear gamma correction; the multi-source, multi-modal, asynchronous data includes video data recorded by devices of different models. Modal features are extracted from the multi-source, multimodal, asynchronous dataset obtained after data augmentation; the modal features include head pose features, facial action unit features, gaze direction features, rPPG signals, and human pose features; A modal encoder module is constructed to map the modal features to a unified dimension; a cross-modal temporal attention alignment module is constructed to calculate the attention weights of the modal features at all time steps and generate aligned cross-modal features. A cross-modal fusion module is constructed to perform residual fusion of the original modal features and the aligned cross-modal features, and further fuse them through graph convolution operations to output global fused features; The globally fused features are input into a fully connected classification network, and the predicted probability is calculated using the Sigmoid function. This process is then used to assist in diagnosis, thus completing the construction of an early auxiliary diagnosis model for ASD.

2. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion as described in claim 1, characterized in that, The nonlinear gamma correction specifically refers to: Constructing a gamma correction lookup table: Setting gamma parameters for input pixel values i ∈[0,255], according to the formula Calculate the mapped pixel values.

3. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion as described in claim 1, characterized in that, The adaptive local contrast enhancement specifically refers to: The input video frame is converted from the RGB color space to the LAB color space. In the LAB color space, the L channel independently represents the luminance information, and the A / B channels respectively represent the chrominance information. The CLAHE operation is limited to the L channel only: a contrast limit threshold is set, the image is divided into several regions using a local grid division strategy, and the histogram is calculated and equalized independently in each region. The processed L channel is then merged back with the unprocessed A / B chroma channels and converted back to the RGB color space.

4. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion as described in claim 1, characterized in that, Head pose features, facial action unit features, and gaze direction features are obtained through feature extraction from the face region, specifically: By using a 3D facial key point detection and fitting method, the head's pose parameters in 3D space, including yaw angle, pitch angle and roll angle, are obtained to obtain head pose features. Facial motion unit features are obtained by detecting and quantifying the intensity values ​​of facial motion units; the facial motion units include eyebrow raising, eyebrow lowering, and corner of mouth raising. By locating key eye points and estimating the gaze direction vector using eye-tracking methods, the position and angle information of the gaze point are obtained as gaze direction features.

5. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion according to claim 1, characterized in that, The rPPG signal is extracted through the following steps: MTCNN was used to detect facial regions, and skin pixels were segmented using U-Net to preserve blood flow-sensitive areas. The frame sequences after preserving blood flow-sensitive areas were normalized and temporally aligned, followed by temporal downsampling and upsampling, specifically: ;in, HR The original heart rate, interval , stride and padding These are the sampling parameters; A 5-layer 3D convolutional encoder was used to extract multi-scale spatiotemporal features; A 3D deconvolution decoder is used to recover timing information and generate... rPPG Signal; A multi-scale encoder-decoder network based on 3D convolution is constructed, including an encoder and a decoder. The encoder includes several layers of 3D convolution and pooling operations, and a residual structure is set. The decoder gradually recovers the temporal dimension through 3D deconvolution, and then smooths it through global average pooling and one-dimensional convolution to generate a sequence with the same length as the frame sequence. rPPG waveform sequence; Introducing frame-by-frame spatial-channel attention (3D-S / T) into multi-scale encoder-decoder networks: This enhances the spatial features of each frame and... rPPG Relevant salient regions; Global Temporal-Spatial Attention (3D-ST): Highlighting periodic pulse fluctuation patterns based on the spatiotemporal characteristics of the entire video; according to rPPG Heart rate was calculated using power spectral density analysis of the waveform sequence. HR = argmaxPSD ( f )×60; where, PSD ( f )for rPPG Power spectral density of waveform sequence, frequency corresponding to peak value f Convert to heart rate per minute.

6. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion as described in claim 1, characterized in that, The human posture features are extracted through the following steps: Using HRNet to process the input video frame sequence Perform human body key point detection and output two-dimensional coordinates of each key point of the human body. P 2D ( t )={( x i ( t ), y i ( t ))| i =1,..., K }, t =1,..., T ; in, K Indicates the number of key points; Two-dimensional joint sequence Input a temporally convolutional 3D pose estimation network to obtain 3D joint coordinates. P 3D ( t )= f θ ({ P 2D ( t - k ),..., P 2D ( t + k )}),in, f θ (•) represents a 3D pose regression network, 2 k +1 represents the receptive field length of the temporal convolution; the output 3D human skeleton is... P 3D ( t )={( X i ( t ), Y i ( t ), Z i ( t ))| i =1,..., K },in,( X i , Y i , Z i ) indicates the first i The coordinates of each joint in three-dimensional space; Based on the sequence of the human 3D skeleton, calculate human posture features: velocity features Acceleration characteristics and attitude stability index ,in, , These are the keypoint vectors for the head and torso, respectively.

7. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion as described in claim 1, characterized in that, The modal encoder module is specifically: ； in, Mode-specific encoder functions; , F m These are the modal features before and after mapping, respectively; It is the set of real numbers; T m For the first m The length of the time dimension of each modality; d For the unified dimension after mapping; face This represents head posture features, facial motion unit features, and gaze direction features. rPPG express rPPG Signal, pose3D It indicates the characteristics of human posture.

8. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion according to claim 1, characterized in that, The cross-modal temporal attention alignment module is specifically as follows: For modes m The i Each time step represents , and modality n All time steps Calculate attention weights And generate aligned cross-modal features. ;in, exp It is an exponential function. W Q m , W K n and W V n For learnable projection matrices, j’ For summation index.

9. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion according to claim 1, characterized in that, The cross-modal fusion module is specifically as follows: The original modal features Aligned cross-modal features Perform residual fusion: ； Constructing modal diagrams G =( V , E ), where the node set V ={ face , rPPG , pose3D }, edge set E To represent the semantic correlation between modalities, graph convolution operations are used for further fusion: ； in, For modal adjacency matrix, For degree matrix, H (L) For the first l Layer node features σ It is a non-linear activation function; The global fusion feature is represented as: ;in, Pool t This represents pooling operations over time.

10. The method for constructing an early auxiliary diagnostic model for ASD based on HPE and multi-source multimodal asynchronous data fusion according to claim 1, characterized in that, The process involves inputting globally fused features into a fully connected classification network, calculating the predicted probability using the Sigmoid function, and performing auxiliary diagnosis to complete the construction of an early ASD auxiliary diagnosis model. Specifically: Global fusion features F fusion The input is fed into a fully connected classification network to calculate the predicted probabilities for different categories. z = W c • F fusion + b c ; in, W c This is the weight matrix. b c It is the bias vector; use Sigmoid Function to calculate predicted probability ;in, σ (•)for Sigmoid function, exp It is an exponential function; This indicates the probability that the model predicts an individual has ASD; Based on the set threshold Perform auxiliary diagnosis: 。

Images