Neural development disorder screening method and system based on facial image

Abstract

The invention discloses a neural development disorder screening method and system based on a face image, and the method comprises the steps: obtaining a to-be-detected face image, and extracting image texture features and face key point structure features from the face image; fusing the two features through an attention mechanism which takes the facial key point structure features as query and takes the image texture features as key values to obtain fused feature representation; and performing classification processing on the fusion feature representation through a classification strategy adaptive to long-tail data distribution to obtain a neurodevelopment disorder category corresponding to the face image. According to the method, the problem that the recognition capability is poor when the types of the neurodevelopment disorders are recognized through a computer vision technology is solved, and accurate and automatic screening of various types of neurodevelopment disorders including various rare diseases under a unified framework is achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and system for screening neurodevelopmental disorders based on facial images. Background Technology

[0002] Neurodevelopmental disorders are a group of diseases that appear early in brain development, with complex clinical manifestations. Diagnosis relies on developmental assessment scales, behavioral observation, parent interviews, and the experience of specialist physicians, which has limitations such as high subjectivity, low efficiency, and difficulty in large-scale implementation. With the development of computer vision technology, using facial images for early screening of neurodevelopmental disorders has become a new research direction.

[0003] Current mainstream facial recognition technologies for neurodevelopmental disorders primarily rely on convolutional neural networks (CNNs) to extract image texture information. For example, single CNNs such as ResNet, MobileNet, Xception, and Inception are used to encode facial images. Some studies combine facial landmark information: first, a landmark detector is used to obtain coordinates, then geometric features, such as eye distance ratio, are calculated manually, and finally, these features are simply concatenated and fused with the texture features from the CNN.

[0004] However, the phenotypic differences in neurodevelopmental disorders are often very subtle, making it difficult for a single convolutional neural network structure to simultaneously capture both local geometric features and global morphological features. Furthermore, due to the severe shortage of samples for some diseases, the models are susceptible to long-tailed distributions, resulting in poor performance in recognizing a few categories. In addition, the interpretability of the aforementioned approaches largely relies on general heatmap tools, whose interpretable results are not stable enough, making it difficult to explain the model's decision-making basis to clinicians and affecting its usability in real-world scenarios. Summary of the Invention

[0005] This application provides a method and system for screening neurodevelopmental disorders based on facial images, which solves the problem of poor recognition ability when using computer vision technology to identify neurodevelopmental disorders. It achieves high-precision automated screening of various neurodevelopmental disorders and can also accurately screen rare diseases with scarce samples.

[0006] This application provides a method and system for screening neurodevelopmental disorders based on facial images. The method for screening neurodevelopmental disorders based on facial images includes: Step S1: Obtain the facial image to be detected, and extract image texture features and facial key point structural features from the facial image; Step S2: The two features are fused using an attention mechanism that uses the facial key point structural features as the query and the image texture features as the key to obtain a fused feature representation. Step S3: Classify the fused feature representation using a classification strategy adapted to the long-tail data distribution to obtain the neurodevelopmental disorder category corresponding to the facial image.

[0007] Optionally, the step of extracting image texture features in step S1 includes: The facial image is input in parallel into at least two pre-trained convolutional neural networks with different architectures, and forward propagation calculations are performed to output initial depth feature maps. Global average pooling and global max pooling are performed on each of the initial depth feature maps to obtain two spatial description feature maps. The two spatial description feature maps are concatenated and a spatial attention weight map is generated through convolution operation; The corresponding initial depth feature map is weighted using the spatial attention weight map to obtain an enhanced feature map; Each of the enhanced feature maps is uniformly scaled or projected to have the same number of channels and spatial size to obtain multiple standard feature maps. All standard feature maps are then stitched together along the channel dimension to obtain the final image texture features.

[0008] Optionally, the step of extracting facial key point structural features in step S1 includes: Using a key point localization algorithm with fixed parameters, key point features in the facial image are extracted to generate coordinate data of facial anatomical key points. The coordinate data is encoded into a geometric structure feature map; wherein the spatial size of the geometric structure feature map matches the spatial size of any feature map included in the image texture feature extraction process.

[0009] Optionally, step S2 includes: The structural features of the facial key points are linearly projected into a query vector; The image texture features are linearly projected into key vectors and value vectors, respectively. Based on the query vector, key vector, and value vector, attention calculation and fusion are performed to obtain a fused feature representation.

[0010] Optionally, the step of performing attention calculation and fusion based on the query vector, key vector, and value vector to obtain the fused feature representation includes: Global attention fusion features are obtained by calculating based on the complete set of query vectors, key vectors, and value vectors. A local attention mask is generated based on the spatial location of the facial key points. The calculation of the key vector and value vector is constrained within the local spatial neighborhood corresponding to each key point. The local attention fusion feature is obtained based on the query vector and the constrained key vector and value vector. The global attention fusion feature and the local attention fusion feature are fused together to obtain a fused feature representation.

[0011] Optionally, after the step of fusing the global attention fusion feature with the local attention fusion feature to obtain the fused feature representation, the method further includes: Self-attention calculation is performed on the fused feature representation to obtain the correlation weights between different elements in the fused feature representation; Based on the relevance weights, the elements in the fused feature representation are reweighted, and the fused feature representation is updated.

[0012] Optionally, step S3 includes: The updated fusion feature representation is linearly transformed to obtain the original discriminant values ​​corresponding to each neurodevelopmental disorder category; The original discriminant value is normalized to an index, and the result is adjusted by combining the prior distribution of the neurodevelopmental disorder category to obtain the category prediction probability. Based on the predicted probability of the category, the neurodevelopmental disorder category corresponding to the facial image is determined.

[0013] Optionally, after the step of determining the neurodevelopmental disorder category corresponding to the facial image based on the predicted probability of the category, the method further includes: Obtain the attention weights calculated by the attention mechanism during the fusion process; Attention weights are mapped back to the corresponding spatial locations of the facial images to generate a visual heatmap; wherein the region of interest represented by the visual heatmap is associated with the spatial distribution of facial anatomical structures defined by the facial key point structural features.

[0014] Furthermore, to achieve the above objectives, embodiments of the present invention also provide a neurodevelopmental disorder screening system, the system comprising: The feature extraction module is used to receive facial images and output image texture features and facial key point structural features; A global-local cross-fusion encoder, connected to the feature extraction module, is configured to: use the facial key point structural features as the query and the image texture features as the key, fuse the two through a cross-attention mechanism, and output a fused feature representation; The long-tail robust classification module, connected to the global-local cross-fusion encoder, is configured to classify the fused feature representation based on a classification strategy adapted to the long-tail data distribution and output the predicted results of the neurodevelopmental disorder category.

[0015] Optionally, the system further includes an interpretability output module that generates a visual heatmap associated with the facial key point structural features based on the attention weights generated by the global-local cross-fusion encoder.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By fusing image texture features with facial key point structural features, a unified feature space with both texture understanding and structural modeling capabilities is constructed. This addresses the problem that related technologies, due to their simple feature fusion methods, are insufficient in capturing subtle, structural facial phenotypic differences, thereby improving the ability to express complex phenotypic differences. Furthermore, by modeling long-range dependencies through a global-local cross-fusion encoder, the separability of subtle structural differences between different disease categories is improved, enhancing the classification accuracy of multi-class tasks.

[0017] By combining the category prior distribution with the category prediction probability adjustment strategy to adapt to the long-tailed data distribution, the problem of decreased performance in identifying rare diseases caused by the long-tailed distribution of categories in real clinical data is solved. This enables the unified model to automatically focus on rare categories and improve the detection performance of diseases with small sample sizes.

[0018] Because attention weights are generated during the deep fusion of image texture features and facial key point structural features, a visual explanation can be generated based on this attention mechanism. This solves the problems of opaque decision-making processes and low credibility of category screening results, and provides traceable decision-making basis for automated screening, thereby improving the credibility of classification results. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the framework of the facial image-based neurodevelopmental disorder screening method of this application; Figure 2 This is a flowchart illustrating the neurodevelopmental disorder screening method based on facial images proposed in this application. Figure 3 This is a general framework diagram of the facial image-based neurodevelopmental disorder screening method of this application; Figure 4 This is a schematic diagram of the framework of the neurodevelopmental disorder screening system of this application; Figure 5 This is a schematic diagram of the terminal structure of the hardware operating environment involved in one embodiment of this application. Detailed Implementation

[0020] To address the issue of poor recognition capabilities when using computer vision technology to identify neurodevelopmental disorder categories, this application provides a neurodevelopmental disorder screening method based on facial images. It employs an attention mechanism that uses facial keypoint features as queries and image texture features as keys to perform cross-modal interaction and alignment of image texture features and facial keypoint structural features. Furthermore, it introduces a classification strategy adapted to long-tailed distributions, fusing category priors to correct category prediction probabilities. This enables automatic screening of multiple neurodevelopmental disorder types while improving the ability to identify rare types with scarce samples.

[0021] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0022] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0023] Example 1 In this embodiment, a method for screening neurodevelopmental disorders based on facial images is provided.

[0024] Reference Figure 1-2 The facial image-based neurodevelopmental disorder screening method of this embodiment includes the following steps: Step S1: Obtain the facial image to be detected, and extract image texture features and facial key point structural features from the facial image; In this embodiment, the facial image to be detected is an RGB or grayscale digital image containing a clear, frontal face. The image texture features are the overall facial shape, local texture, and deep semantic representation extracted through various pre-trained convolutional neural networks. The facial key point structural features are key point features reflecting differences in facial structure extracted by a facial key point detection network with frozen weights, including interocular distance, facial feature shape, facial contour, and other key geometric indicators.

[0025] As an optional implementation, when extracting image texture features, facial images are input in parallel into at least two pre-trained convolutional neural networks with different architectures. Forward propagation is performed on each network, and initial depth feature maps are output. Global average pooling and global max pooling are then performed on each initial depth feature map to obtain two spatial description feature maps. These two spatial description feature maps are concatenated, and a spatial attention weight map is generated through convolution. The corresponding initial depth feature maps are then weighted using this spatial attention weight map to obtain enhanced feature maps. All enhanced feature maps are uniformly scaled or projected to have the same number of channels and spatial dimensions to obtain multiple standard feature maps. All standard feature maps are then concatenated along the channel dimension to obtain the final image texture features. By extracting initial features in parallel using multiple convolutional neural networks, feature diversity is broadened. The spatial attention mechanism enables autonomous focusing on key regions of disease phenotypes. Finally, standardized aggregation forms a unified texture feature representation that is rich in information and has strong discriminative power.

[0026] For example, such as Figure 3 As shown, features can be extracted using multiple convolutional neural networks, such as Xception, IR50, MobileNetV2, and ResNet50. This embodiment loads two pre-trained convolutional neural networks in parallel, such as ResNet50 for extracting deep, semantically rich features and MobileNetV2 for extracting lightweight, efficient features. The facial image is input into these two networks respectively, and feature maps are extracted from specific intermediate layers of each network. Assume that the ResNet50 outputs an initial 256×7×7 depth feature map. Output a 128×7×7 initial depth feature map from MobileNetV2. .

[0027] To improve the ability of features to focus on disease-related facial regions, methods such as Figure 3 The Spatial Attention Module (SAM) in the code applies a spatial attention mechanism to each initial deep feature map: [the SAM applies spatial attention to the feature map]. Global average pooling and global max pooling operations are performed along the channel dimension to obtain two 2D spatial description feature maps with height H and width W, representing the average and most salient responses of the features at each spatial location, respectively. The two spatial description feature maps are concatenated along the channel dimension to form a 2-channel feature map. This concatenated feature map is then processed by a lightweight convolutional layer to generate a single-channel spatial attention weight map of size H×W. Each element in this weight map has a value between 0 and 1, with larger values ​​indicating greater importance at the corresponding spatial location. The generated spatial attention weight map is then compared with the corresponding initial depth feature map. Perform element-wise multiplication, that is, use the spatial attention weight map to... Spatial dimension reweighting is performed to highlight the feature responses of important regions and suppress facial features or interfering regions, thereby obtaining an enhanced feature map. For the initial depth feature map Perform the same process to obtain .

[0028] Since feature maps from different convolutional neural networks may have different numbers of channels, it is necessary to uniformly scale or project each enhanced feature map to have the same number of channels C and spatial dimensions H'×W', resulting in multiple standard feature maps. All standard feature maps are then concatenated along the channel dimension to form a fused feature tensor of dimensions (N×C)×H'×W', where N is the number of convolutional neural networks used. This fused feature tensor is the final image texture feature, which integrates multi-source, multi-view visual information and undergoes selective enhancement with spatial attention.

[0029] As another optional implementation, when extracting facial key point structural features, a key point localization algorithm with fixed parameters is used to extract key point features from the facial image, generating coordinate data of facial anatomical key points, and encoding the coordinate data into a geometric structural feature map. The spatial dimensions of the geometric structural feature map match the spatial dimensions of any feature map included in the image texture feature extraction process. By using a key point localization algorithm with fixed parameters, the stability of geometric information is ensured, avoiding bias during end-to-end training. Encoding the coordinate data into aligned feature maps transforms discrete facial structural information into a data format that can be deeply and directly fused with continuous texture features.

[0030] For example, such as Figure 3 As shown, MobileFaceNet is used to detect facial keypoints, including the locations of key regions such as the eyes, nose, and mouth. MobileFaceNet receives a facial image and performs forward propagation, ultimately outputting a set of predefined two-dimensional coordinate data for facial keypoints. For example, it can output the (x,y) coordinates of 98 keypoints, which precisely correspond to important landmark locations such as the corners of the eyes, the tip of the nose, the corners of the mouth, and the facial contours. This coordinate data is a sparse, discrete set of points, which needs to be encoded into a dense, gridded feature representation, i.e., a geometric feature map, so that it can interact with dense image texture features in depth space.

[0031] Optionally, the process of encoding the coordinate data can be implemented using a lightweight encoding network. Based on the coordinate data, a Gaussian heatmap or one-hot response is generated at the location corresponding to each keypoint on a spatial grid of the target feature map, such as a 7×7 grid, forming a sparse initial structural feature map. This sparse feature map is then smoothed or transformed through one or more convolutional layers to generate a dense geometric structural feature map. The spatial size of this geometric structural feature map matches the spatial size of any intermediate feature map included in the aforementioned image texture feature extraction process. This geometric structural feature map is the facial keypoint structural feature.

[0032] Step S2: The two features are fused using an attention mechanism that uses the facial key point structural features as the query and the image texture features as the key to obtain a fused feature representation. In this embodiment, the fused feature representation is a deep representation reconstructed from dynamic texture features, guided by stable structural features, and possesses cross-modal semantic associations. By using an attention mechanism that queries structural features, the correlation between structural features and global texture features is calculated, enabling the feature fusion to accurately capture subtle texture anomalies that are easily overlooked near local facial regions defined by keypoints.

[0033] As an optional implementation, facial keypoint structural features are linearly projected into a query vector Q, and image texture features are linearly projected into a key vector K and a value vector V, respectively. Attention is then calculated and fused based on the query vector, key vector, and value vector to obtain a fused feature representation.

[0034] For example, when performing fusion based on Q, K, and V, including global attention fusion and local attention fusion, it can be done through... Figure 3 The Global-Local Cross-Fusion Transformer Encoder (GLCFTE) implementation shown is configured with a global and local multi-head self-attention (MSA) mechanism, which can fuse facial keypoint features and image texture features, and collaboratively merge two-stream features at the global and local levels, combining local features with broader facial features.

[0035] In the global fusion branch, the attention weight matrix is ​​calculated. ,in Let A be the dimension of the key vector. Using this weight A, the value vector V is weighted and aggregated to obtain the global attention fusion feature: This feature captures contextual information about the overall facial structure.

[0036] In the local fusion branch, a local attention mask is generated for each facial keypoint based on its spatial coordinates. This mask has a value of 1 only within a pre-defined neighborhood around the keypoint in the feature map space, and 0 in the remaining areas. This local mask is then applied to the key vector K and value vector V. Specifically, K and V at each location are multiplied by the local masks for all corresponding keypoints, ensuring that the query for each keypoint... When calculating attention, only the key and value vectors within its local neighborhood are visible. The local attention fusion feature is obtained by calculating the local attention based on the constrained key and value vectors and the query vector. .

[0037] The global attention fusion features from the global path output are fused with the local attention fusion features from the local path output. The fusion method can be summation, concatenation, or adaptive weighting through a gating network. The fused result is a more refined fusion feature representation.

[0038] Optionally, after completing global-local fusion and obtaining the fused feature representation, feature enhancement can be performed to further optimize the internal structure of the fused features and improve their discriminative power. The fused feature representation is then input into a Multi-features Interaction Network (MFI) for attention computation, and the fused feature representation is simultaneously linearly projected to generate a query for self-attention computation. ,key AND value Perform self-attention operation to calculate the query. AND key The similarity is calculated and normalized to obtain the self-attention weight matrix. This weight matrix indicates the inherent semantic relationships and importance between different spatial locations or feature channels within the fused feature representation. This self-attention weight matrix is ​​then used to adjust the original fused feature representation or value. The system reweights features to dynamically strengthen those elements that contribute most to the final classification, while weakening redundant or distracting information. Finally, to maintain training stability and preserve the integrity of the original information, the reweighted features are fused with the fused feature representation via residual connections. This fused feature representation is then updated and optimized to obtain the enhanced features. These enhanced features improve the analysis of subtle changes in facial morphology, texture, and other disease indicators.

[0039] Step S3: Classify the fused feature representation using a classification strategy adapted to the long-tail data distribution to obtain the neurodevelopmental disorder category corresponding to the facial image.

[0040] In this embodiment, the classification strategy adapted to long-tail data distribution refers to a training and optimization method designed to address the extremely uneven distribution of samples across different categories in the training data, i.e., a "long-tail distribution" (with many samples in the head category and very few samples in the tail category). Its core principle is to maintain a similar recognition sensitivity for the tail category (such as rare diseases) as for the head category (which has abundant samples), thereby avoiding the omission of rare diseases due to data bias.

[0041] As an optional implementation, the fused feature representation after feature enhancement and update undergoes a linear transformation to obtain the original discriminant value corresponding to each neurodevelopmental disorder category. The original discriminant value is then normalized to an index, and the result is adjusted based on the prior distribution of the neurodevelopmental disorder categories to obtain the category prediction probability. Based on the category prediction probability, the neurodevelopmental disorder category corresponding to the facial image to be detected is determined. The prior distribution refers to the sample proportion distribution of each category in the long-tailed data.

[0042] For example, the final fused feature representation from step S2 is input to a pre-defined classifier. This classifier incorporates a correction mechanism adapted to long-tailed data distributions. Specifically, the classifier performs a linear transformation on the input fused feature representation to obtain raw discriminant values ​​corresponding to eight predefined neurodevelopmental disorder categories (Angelman syndrome, autism spectrum disorder, De Lange syndrome, Down syndrome, Fragile X syndrome, KBG syndrome, Smith-Limley-Oppitz syndrome, Williams syndrome, etc.). A probability correction calculation is then performed: the classifier calculates a normalization index on the raw discriminant values ​​based on a pre-stored parameter set reflecting the prior distribution of sample quantity for each category, incorporating the prior distribution information during this calculation, and directly outputs a set of category prediction probabilities corrected for class imbalance. This correction process is equivalent to performing a Balanced Softmax calculation, increasing the likelihood of higher prediction probabilities for categories with fewer samples. Finally, based on this set of corrected category prediction probabilities, the category with the highest probability value is selected as the neurodevelopmental disorder category corresponding to the facial image and output.

[0043] As an alternative implementation, after determining the neurodevelopmental disorder category corresponding to the facial image, a visualization can be generated to provide an intuitive basis for classification decisions. First, during the global-local cross-fusion process in step S2, attention weights generated by the attention mechanism are extracted. These attention weights quantify the degree of attention each facial keypoint pays to each spatial feature in the image texture feature map during feature fusion. Then, the attention weights are mapped back to the corresponding spatial location of the facial image to generate a visualization heatmap; wherein the area of ​​interest represented by the visualization heatmap is associated with the spatial distribution of facial anatomical structures defined by the structural features of the facial keypoints. In this visualization heatmap, warmer-colored areas indicate higher attention weights given by the model when making classification decisions, i.e., a greater feature contribution from that area.

[0044] It should be noted that the high-attention areas represented by the visualized heatmap are spatially significantly correlated with the facial structures defined by the facial key point structural features extracted in step S1. For example, if the diagnosis is "Down syndrome," the generated heatmap may highly focus on the corner of the eye area; if the diagnosis is "De Lange syndrome," the heatmap may significantly focus on the brow ridge and eyelash area.

[0045] In this embodiment, a deep fusion mechanism is constructed using facial key point structures as a stable guide and image texture as a dynamic response. An optimized classification strategy adapted to long-tail data distribution is also employed, addressing three core shortcomings of related technologies: weak feature fusion capabilities, sensitivity to imbalanced data, and opaque decision-making processes. This significantly improves the screening accuracy and robustness for various neurodevelopmental disorders, especially rare diseases with scarce samples, achieving accurate identification of complex and subtle facial phenotypes. Furthermore, the attention mechanism during feature fusion automatically generates visual heatmaps highly correlated with key anatomical regions, making the final decision-making basis intuitive and traceable, greatly enhancing the clinical credibility of the results.

[0046] Based on the same inventive concept, this application also provides a system corresponding to the method in Embodiment 1, as shown in Embodiment 2.

[0047] Example 2 Based on Embodiment 1, another embodiment of this application is proposed, with reference to... Figure 4 A neurodevelopmental disorder screening system includes at least: a feature extraction module, a global-local cross-fusion encoder, and a long-tailed robust classification module.

[0048] The feature extraction module is used to receive facial images and output image texture features and facial key point structural features; In this embodiment, the feature extraction module is the input end of the system, responsible for receiving the standardized facial image to be detected and extracting two complementary features in parallel.

[0049] Optionally, the feature extraction module includes an image texture feature extraction submodule. This module is responsible for extracting deep features containing facial visual semantics, including a multi-source coding unit and a spatial attention unit. The multi-source coding unit integrates at least two pre-trained convolutional neural networks with different architectures, such as ResNet50 and MobileNetV2. These convolutional neural networks work in parallel to extract multiple initial deep feature maps from the input facial image to capture diverse visual patterns. The spatial attention unit is connected to the multi-source coding unit and processes each initial deep feature, performing global average pooling and global max pooling. After fusion, the features are passed through convolutional layers to generate a spatial attention weight map. Finally, this weight map is used to reweight the original feature map, outputting the enhanced image texture features.

[0050] Optionally, the feature extraction module also includes a facial landmark structure feature extraction submodule. This module is responsible for extracting accurate facial geometric structure information. A pre-trained facial landmark detection model, Mobile FaceNet, with fixed parameters is used. Its weights remain unchanged during training to ensure a stable and objective geometric benchmark.

[0051] Optionally, the feature extraction module may also include a structural feature encoding submodule (not shown in the illustration), used to encode discrete facial keypoint coordinate data into dense geometric structural feature maps aligned with the image texture feature space size. This transforms the structural information into a data format suitable for subsequent deep fusion.

[0052] A global-local cross-fusion encoder, connected to the feature extraction module, is configured to: use the facial key point structural features as the query and the image texture features as the key, fuse the two through a cross-attention mechanism, and output a fused feature representation; In this embodiment, the global-local cross-fusion encoder is connected to the feature extraction module and receives the image texture features and facial key point structural features output by the module.

[0053] Optionally, the global-local cross-fusion encoder is configured to use the facial keypoint structural features as the query vector and the image texture features as the key and value vectors, respectively, and perform computation through a cross-attention mechanism. This fusion is accomplished through collaborative global and local dual-path attention, ultimately outputting a fused feature representation containing cross-modal information.

[0054] The long-tail robust classification module, connected to the global-local cross-fusion encoder, is configured to classify the fused feature representation based on a classification strategy adapted to the long-tail data distribution and output the predicted results of the neurodevelopmental disorder category.

[0055] In this embodiment, the long-tail robust classification module employs an optimization strategy adapted to the long-tail data distribution. The fused feature representation is processed, and the calculation of class probabilities is corrected based on the pre-stored class prior distribution to improve the sensitivity of identifying classes with few samples, and finally outputs the predicted class of neurodevelopmental disorder.

[0056] In addition, the neurodevelopmental disorder screening system may also include an interpretable output module, which generates a visual heatmap related to facial key point regions based on the attention weights generated by the global-local cross-fusion encoder, providing an intuitive visual basis for predicting classification probabilities.

[0057] In this embodiment, through modular design, multi-source feature extraction, deep fusion, classification decision and interpretable output are integrated into an organic whole. This not only achieves high-precision automated screening of multiple diseases, but also ensures the transparency of the decision-making process through an endogenous attention mechanism, and ensures the fairness of the screening results through a long-tail optimization strategy.

[0058] Since the system described in Embodiment 2 of this application is a system used to implement the method of Embodiment 1 of this application, those skilled in the art can understand the specific structure and variations of the system based on the method described in Embodiment 1 of this application, and therefore will not be described again here. All systems used in the method of Embodiment 1 of this application fall within the scope of protection of this application.

[0059] Example 3 In this application embodiment, a neurodevelopmental disorder screening device based on facial images is proposed.

[0060] Reference Figure 5 , Figure 5 This is a schematic diagram of the terminal structure of the hardware operating environment involved in one embodiment of this application.

[0061] like Figure 5As shown, the control terminal may include: a processor 1001, such as a CPU, a network interface 1003, a memory 1004, and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The network interface 1003 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1004 may be high-speed RAM or stable non-volatile memory, such as disk storage. Alternatively, the memory 1004 may be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art will understand that Figure 5 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0063] like Figure 5 As shown, the memory 1004, which serves as a computer storage medium, may include an operating system, a network communication module, and a facial image-based neurodevelopmental disorder screening program.

[0064] exist Figure 5 In the hardware structure of the facial image-based neurodevelopmental disorder screening device shown, the processor 1001 can call the facial image-based neurodevelopmental disorder screening program stored in the memory 1004 and perform the following operations: Step S1: Obtain the facial image to be detected, and extract image texture features and facial key point structural features from the facial image; Step S2: The two features are fused using an attention mechanism that uses the facial key point structural features as the query and the image texture features as the key to obtain a fused feature representation. Step S3: Classify the fused feature representation using a classification strategy adapted to the long-tail data distribution to obtain the neurodevelopmental disorder category corresponding to the facial image.

[0065] Optionally, the processor 1001 may invoke a facial image-based neurodevelopmental disorder screening program stored in the memory 1004, and further perform the following operations: The facial image is input in parallel into at least two pre-trained convolutional neural networks with different architectures, and forward propagation calculations are performed to output initial depth feature maps. Global average pooling and global max pooling are performed on each of the initial depth feature maps to obtain two spatial description feature maps. The two spatial description feature maps are concatenated and a spatial attention weight map is generated through convolution operation; The corresponding initial depth feature map is weighted using the spatial attention weight map to obtain an enhanced feature map; Each of the enhanced feature maps is uniformly scaled or projected to have the same number of channels and spatial size to obtain multiple standard feature maps. All standard feature maps are then stitched together along the channel dimension to obtain the final image texture features.

[0066] Optionally, the processor 1001 may invoke a facial image-based neurodevelopmental disorder screening program stored in the memory 1004, and further perform the following operations: Using a key point localization algorithm with fixed parameters, key point features in the facial image are extracted to generate coordinate data of facial anatomical key points. The coordinate data is encoded into a geometric structure feature map; wherein the spatial size of the geometric structure feature map matches the spatial size of any feature map included in the image texture feature extraction process.

[0067] Optionally, the processor 1001 may invoke a facial image-based neurodevelopmental disorder screening program stored in the memory 1004, and further perform the following operations: The structural features of the facial key points are linearly projected into a query vector; The image texture features are linearly projected into key vectors and value vectors, respectively. Based on the query vector, key vector, and value vector, attention calculation and fusion are performed to obtain a fused feature representation.

[0068] Optionally, the processor 1001 may invoke a facial image-based neurodevelopmental disorder screening program stored in the memory 1004, and further perform the following operations: Global attention fusion features are obtained by calculating based on the complete set of query vectors, key vectors, and value vectors. A local attention mask is generated based on the spatial location of the facial key points. The calculation of the key vector and value vector is constrained within the local spatial neighborhood corresponding to each key point. The local attention fusion feature is obtained based on the query vector and the constrained key vector and value vector. The global attention fusion feature and the local attention fusion feature are fused together to obtain a fused feature representation.

[0069] Optionally, the processor 1001 may invoke a facial image-based neurodevelopmental disorder screening program stored in the memory 1004, and further perform the following operations: Self-attention calculation is performed on the fused feature representation to obtain the correlation weights between different elements in the fused feature representation; Based on the relevance weights, the elements in the fused feature representation are reweighted, and the fused feature representation is updated.

[0070] Optionally, the processor 1001 may invoke a facial image-based neurodevelopmental disorder screening program stored in the memory 1004, and further perform the following operations: The updated fusion feature representation is linearly transformed to obtain the original discriminant values ​​corresponding to each neurodevelopmental disorder category; The original discriminant value is normalized to an index, and the result is adjusted by combining the prior distribution of the neurodevelopmental disorder category to obtain the category prediction probability. Based on the predicted probability of the category, the neurodevelopmental disorder category corresponding to the facial image is determined.

[0071] Optionally, the processor 1001 may invoke a facial image-based neurodevelopmental disorder screening program stored in the memory 1004, and further perform the following operations: Obtain the attention weights calculated by the attention mechanism during the fusion process; Attention weights are mapped back to the corresponding spatial locations of the facial images to generate a visual heatmap; wherein the region of interest represented by the visual heatmap is associated with the spatial distribution of facial anatomical structures defined by the facial key point structural features.

[0072] Furthermore, to achieve the above objectives, embodiments of the present invention also provide a neurodevelopmental disorder screening system, comprising: The feature extraction module is used to receive facial images and output image texture features and facial key point structural features; A global-local cross-fusion encoder, connected to the feature extraction module, is configured to: use the facial key point structural features as the query and the image texture features as the key, fuse the two through a cross-attention mechanism, and output a fused feature representation; The long-tail robust classification module, connected to the global-local cross-fusion encoder, is configured to classify the fused feature representation based on a classification strategy adapted to the long-tail data distribution and output the predicted results of the neurodevelopmental disorder category.

[0073] Optionally, the system further includes an interpretability output module that generates a visual heatmap associated with the facial key point structural features based on the attention weights generated by the global-local cross-fusion encoder.

[0074] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, third, etc., does not indicate any order. These words can be interpreted as names.

[0079] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0080] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A method for screening neurodevelopmental disorders based on facial images, characterized in that, The method includes: Step S1: Obtain the facial image to be detected, and extract image texture features and facial key point structural features from the facial image; Step S2: The two features are fused using an attention mechanism that uses the facial key point structural features as the query and the image texture features as the key to obtain a fused feature representation. Step S3: Classify the fused feature representation using a classification strategy adapted to the long-tail data distribution to obtain the neurodevelopmental disorder category corresponding to the facial image.

2. The method as described in claim 1, characterized in that, The step of extracting image texture features in step S1 includes: The facial image is input in parallel into at least two pre-trained convolutional neural networks with different architectures, and forward propagation calculations are performed to output initial depth feature maps. Global average pooling and global max pooling are performed on each of the initial depth feature maps to obtain two spatial description feature maps. The two spatial description feature maps are concatenated and a spatial attention weight map is generated through convolution operation; The corresponding initial depth feature map is weighted using the spatial attention weight map to obtain an enhanced feature map; Each of the enhanced feature maps is uniformly scaled or projected to have the same number of channels and spatial size to obtain multiple standard feature maps. All standard feature maps are then stitched together along the channel dimension to obtain the final image texture features.

3. The method as described in claim 1 or 2, characterized in that, The step of extracting facial key point structural features in step S1 includes: Using a key point localization algorithm with fixed parameters, key point features in the facial image are extracted to generate coordinate data of facial anatomical key points. The coordinate data is encoded into a geometric structure feature map; wherein the spatial size of the geometric structure feature map matches the spatial size of any feature map included in the image texture feature extraction process.

4. The method according to any one of claims 1 to 3, characterized in that, Step S2 includes: The structural features of the facial key points are linearly projected into a query vector; The image texture features are linearly projected into key vectors and value vectors, respectively. Based on the query vector, key vector, and value vector, attention calculation and fusion are performed to obtain a fused feature representation.

5. The method as described in claim 4, characterized in that, The step of performing attention calculation and fusion based on the query vector, key vector, and value vector to obtain the fused feature representation includes: Global attention fusion features are obtained by calculating based on the complete set of query vectors, key vectors, and value vectors. A local attention mask is generated based on the spatial location of the facial key points. The calculation of the key vector and value vector is constrained within the local spatial neighborhood corresponding to each key point. The local attention fusion feature is obtained based on the query vector and the constrained key vector and value vector. The global attention fusion feature and the local attention fusion feature are fused together to obtain a fused feature representation.

6. The method as described in claim 5, characterized in that, After the step of fusing the global attention fusion feature with the local attention fusion feature to obtain the fused feature representation, the method further includes: Self-attention calculation is performed on the fused feature representation to obtain the correlation weights between different elements in the fused feature representation; Based on the relevance weights, the elements in the fused feature representation are reweighted, and the fused feature representation is updated.

7. The method as described in claim 6, characterized in that, Step S3 includes: The updated fusion feature representation is linearly transformed to obtain the original discriminant values ​​corresponding to each neurodevelopmental disorder category; The original discriminant value is normalized to an index, and the result is adjusted by combining the prior distribution of the neurodevelopmental disorder category to obtain the category prediction probability. Based on the predicted probability of the category, the neurodevelopmental disorder category corresponding to the facial image is determined.

8. The method as described in claim 7, characterized in that, After the step of determining the neurodevelopmental disorder category corresponding to the facial image based on the predicted probability of the category, the method further includes: Obtain the attention weights calculated by the attention mechanism during the fusion process; Attention weights are mapped back to the corresponding spatial locations of the facial images to generate a visual heatmap; wherein the region of interest represented by the visual heatmap is associated with the spatial distribution of facial anatomical structures defined by the facial key point structural features.

9. A neurodevelopmental disorder screening system, characterized in that, The system includes: The feature extraction module is used to receive facial images and output image texture features and facial key point structural features; A global-local cross-fusion encoder, connected to the feature extraction module, is configured to: use the facial key point structural features as the query and the image texture features as the key, fuse the two through a cross-attention mechanism, and output a fused feature representation; The long-tail robust classification module, connected to the global-local cross-fusion encoder, is configured to classify the fused feature representation based on a classification strategy adapted to the long-tail data distribution and output the predicted results of the neurodevelopmental disorder category.

10. The system as described in claim 9, characterized in that, The system also includes an interpretable output module that generates a visual heatmap associated with the facial key point structural features based on the attention weights generated by the global-local cross-fusion encoder.

Images