Autism spectrum disorder classification method and device, electronic equipment and storage medium

Abstract

The invention provides an autism spectrum disorder classification method and device, electronic equipment and a storage medium, and relates to the field of image processing. The method comprises the following steps: acquiring a target image; the target image is an image comprising a target object; obtaining a fusion feature by capturing an internal relationship among a plurality of local regions in the target image and an interaction relationship between the local regions and a global region in the target image; classifying autism spectrum disorders according to the fusion features to obtain a classification result; the classification result is used for indicating the autism spectrum disorder type to which the target object belongs. According to the method and the apparatus, the problem of low classification accuracy of autism spectrum disorders in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of image processing. Specifically, this application relates to a classification method, apparatus, electronic device, and storage medium for autism spectrum disorder. Background Art

[0002] Autism spectrum disorder (ASD) encompasses a wide range of neuropsychiatric disorders, characterized by varying degrees of impairment in social communication, interaction, and academic functioning. People with ASD typically exhibit different behavioral, communication, and learning patterns compared to their peers. The Autism and Developmental Disabilities Monitoring Network (ADDM) of the Centers for Disease Control and Prevention (CDC) in the United States reported that approximately 1 in every 36 children exhibits symptoms related to ASD. Nevertheless, based on CDC data over the past 15 years, the average age of ASD diagnosis is still approximately 4 years old, although parents and caregivers often notice abnormalities as early as 2 years old.

[0003] Due to the developmental plasticity of the brain, early detection and diagnosis are crucial for implementing timely treatment for children presenting with ASD symptoms. Research has shown that early intervention before the age of two can improve cognitive outcomes, enhance the overall quality of life of children, and improve social integration compared to late intervention.

[0004] Research has shown that ASD is mainly caused by genetic and environmental factors. However, its pathogenesis remains unclear, and there are no accurate biomarkers. In addition, due to the high heterogeneity of its phenotypes and etiologies, early diagnosis of ASD is difficult, which is not conducive to early intervention and treatment. Therefore, although the medical community has made significant progress in the in-depth study of ASD, there is still a need for efficient and accurate screening methods.

[0005] As can be seen from the above, how to improve the accuracy of the classification of autism spectrum disorder remains to be solved. Summary of the Invention

[0006] This application provides a classification method, apparatus, electronic device, and storage medium for autism spectrum disorder, which can solve the problem of low accuracy in the classification of autism spectrum disorder in the related art. The technical solutions are as follows:

[0007] According to one aspect of this application, a classification method for autism spectrum disorder includes: obtaining a target image; the target image is an image including a target object; by capturing the internal relationships between several local regions in the target image and the interaction relationships between the local regions and the global region in the target image, a fusion feature is obtained; according to the fusion feature, classification of autism spectrum disorder is performed to obtain a classification result; the classification result is used to indicate the type of autism spectrum disorder to which the target object belongs.

[0008] According to one aspect of the present application, a classification device for autism spectrum disorder includes: an acquisition module configured to acquire a target image; the target image is an image including a target object; a processing module configured to obtain a fusion feature by capturing internal relationships between a plurality of local regions in the target image and interaction relationships between local regions and a global region in the target image; a classification module configured to perform classification of autism spectrum disorder based on the fusion feature to obtain a classification result; the classification result is used to indicate the type of autism spectrum disorder to which the target object belongs.

[0009] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the classification method for autism spectrum disorder as described above is implemented.

[0010] According to one aspect of the present application, a storage medium stores a computer program, and when the computer program is executed by one or more processors, the classification method for autism spectrum disorder as described above is implemented.

[0011] According to one aspect of the present application, a computer program product includes a computer program, and when the computer program is executed by one or more processors, the classification method for autism spectrum disorder as described above is implemented.

[0012] The beneficial effects brought by the technical solution provided by the present application are as follows:

[0013] In the above technical solution, by capturing the internal relationships between a plurality of local regions in the target image and the interaction relationships between local regions and the global region in the target image, a fusion feature is obtained, which can effectively combine the local region and the global region of the target image of the target object, and achieve high-precision and high-efficiency classification of ASD; at the same time, the fusion feature can effectively adapt to different facial poses of different target objects, avoiding the phenomena of intra-class differences and inter-class similarities. In addition, through automated target image analysis, this method can quickly and effectively assist in the classification of ASD, provide interpretable decision support for medical professionals, thereby improving the early intervention and treatment of ASD patients, and thus can effectively solve the problem of low accuracy in the classification of autism spectrum disorder in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for description in the embodiments of the present application. Obviously, the following drawings are only some embodiments of the present application, and those skilled in the art can obtain other drawings without creative efforts based on these drawings.

[0015] Figure 1 is a schematic diagram of the implementation environment related to the present application;

[0016] Figure 2 is a hardware structure diagram of an electronic device shown according to an exemplary embodiment;

[0017] Figure 3 is a flowchart of a classification method for autism spectrum disorder shown according to an exemplary embodiment;

[0018] Figure 4 is Figure 3 a flowchart of step 330 in the corresponding embodiment in one embodiment;

[0019] Figure 5 is Figure 4 a flowchart of the steps before step 331 in the corresponding embodiment in one embodiment;

[0020] Figure 6 is Figure 4 a flowchart of step 335 in the corresponding embodiment in one embodiment;

[0021] Figure 7 is Figure 3 a flowchart of the training process of the autism classification model in the corresponding embodiment in one embodiment;

[0022] Figures 8a to 8d is a specific implementation schematic diagram of a classification method for autism spectrum disorder in an application scenario;

[0023] Figure 9 is a structural block diagram of a classification device for autism spectrum disorder shown according to an exemplary embodiment;

[0024] Figure 10 is a structural block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0025] The embodiments of the present application are described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described by referring to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as limiting the present application.

[0026] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0027] The following is an introduction and explanation of several terms involved in this application:

[0028] Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that affects social interaction, communication, and behavior.

[0029] Typically Developing (TD) children refer to children whose developmental trajectories in aspects such as neurodevelopment, behavior, language, social interaction, and cognitive ability conform to or are close to the age norm.

[0030] As mentioned above, the pathogenesis of ASD is still unclear, and there are no accurate biomarkers. In addition, due to the high heterogeneity of its phenotypes and etiologies, early diagnosis of ASD is difficult, which is not conducive to early intervention and treatment.

[0031] Currently, the classification of ASD mainly relies on observation and the experience of clinicians. This method takes a long time, is prone to misdiagnosis, and has a strong dependence on professional medical staff. Specifically, the above method includes observation and interview, relying on behavioral symptoms and the observation of parents or caregivers, and finally a doctor makes a diagnosis. Currently, commonly used reliable and standardized autism diagnosis tools include the Autism Diagnostic Observation Schedule (ADOS), the Autism Diagnostic Interview - Revised (ADI - R), and the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM - 5). Although clinicians use standardized ASD detection tools, a significant drawback of this method is that it takes a lot of time to evaluate and interpret scores. These methods are also easily affected by biases related to the abilities and skills of doctors. In addition, parents or caregivers may not always provide accurate data or complete questionnaires, which may affect the reliability of the diagnosis.

[0032] Currently, deep learning methods can also be used to classify ASD. However, existing deep learning methods usually only focus on global facial features, ignoring the importance of local features and facial landmark features, as well as the complex relationships between them, which limits the model's ability to capture complex facial features and relationships. In addition, existing deep learning methods fail to effectively handle the intra-class differences and inter-class similarities of samples and are sensitive to pose changes in facial images, which limits the robustness and accuracy of the model in practical applications.

[0033] As can be seen from the above, there are still defects in the related technology, such as the low accuracy of classifying autism spectrum disorder.

[0034] Therefore, the classification method for autism spectrum disorder provided in this application can effectively improve the accuracy of classifying autism spectrum disorder. Correspondingly, the classification method for autism spectrum disorder is applicable to a classification device for autism spectrum disorder, and the classification device for autism spectrum disorder can be deployed on an electronic device, which can be a computer device configured with a von Neumann architecture. For example, the computer device includes a desktop computer, a laptop computer, a server, etc.

[0035] To make the purpose, technical solution, and advantages of this application clearer, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings.

[0036] Figure 1 It is a schematic diagram of an implementation environment involved in a classification method for autism spectrum disorder. It should be noted that this implementation environment is only an example adapted to this invention and should not be considered as providing any limitation to the scope of use of this invention.

[0037] This implementation environment includes a collection end 110 and a server end 130.

[0038] Specifically, the collection end 110, which can also be regarded as an image collection device, includes, but is not limited to, electronic devices with a shooting function such as cameras, cameras, video recorders, etc. For example, the collection end 110 is a camera.

[0039] The server end 130 can be an electronic device such as a desktop computer, a laptop computer, a server, etc., or a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. Among them, the server end 130 is used to provide background services. For example, the background services include, but are not limited to, classification services for autism spectrum disorder, etc.

[0040] A network communication connection is established in advance between the server end 130 and the collection end 110 by means of wired or wireless, etc., and data transmission between the server end 130 and the collection end 110 is realized through this network communication connection. The transmitted data includes, but is not limited to: target images, etc.

[0041] In an application scenario, through the interaction between the acquisition terminal 110 and the server terminal 130, the acquisition terminal 110 captures and acquires a target image for a target object, and uploads the target image to the server terminal 130 to request the server terminal 130 to provide a classification service for autism spectrum disorder.

[0042] For the server terminal 130, after receiving the target image of the target object uploaded by the acquisition terminal 110, it calls the classification service for autism spectrum disorder. By capturing the internal relationships between several local regions in the target image and the interaction relationships between the local regions and the global region in the target image, a fusion feature is obtained, and classification for autism spectrum disorder is performed based on the fusion feature to obtain a classification result; the classification result is used to indicate the type of autism spectrum disorder to which the target object belongs, so as to solve the problem of low accuracy in the classification of autism spectrum disorder existing in the related art.

[0043] Please refer to Figure 2 , Figure 2 which is a hardware structure diagram of an electronic device shown according to an exemplary embodiment. This electronic device is applicable to Figure 1 the server terminal 130 in the implementation environment shown.

[0044] It should be noted that this electronic device is only an example adapted to this application and cannot be considered as providing any limitation to the scope of use of this application. This electronic device cannot be interpreted as requiring dependence on or necessarily having Figure 2 one or more components in the exemplary electronic device 200 shown.

[0045] The hardware structure of the electronic device 200 may vary greatly due to different configurations or performances. As Figure 2 shown, the electronic device 200 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0046] Specifically, the power supply 210 is used to provide working voltage for each hardware device on the electronic device 200.

[0047] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, for Figure 1 the interaction between the acquisition terminal 110 and the server terminal 130 in the implementation environment shown.

[0048] Of course, in the remaining examples adapted to this application, the interface 230 may further include at least one serial-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. AsFigure 2 As shown, specific limitations are not constituted hereby.

[0049] The memory 250, as a carrier for resource storage, can be a read-only memory, a random access memory, a magnetic disk, an optical disc, etc. The resources stored thereon include an operating system 251, application programs 253, data 255, etc., and the storage method can be transient storage or permanent storage.

[0050] Among them, the operating system 251 is used to manage and control each hardware device and application program 253 on the electronic device 200, so as to realize the operation and processing of the massive data 255 in the memory 250 by the central processing unit 270. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0051] The application program 253 is a computer program formed by computer-readable instructions that complete at least one specific task based on the operating system 251. It can include at least one module ( Figure 2 not shown), and each module can respectively contain corresponding computer-readable instructions. For example, the classification device for autism spectrum disorder can be regarded as an application program 253 deployed on the electronic device 200.

[0052] The data 255 can be photos, pictures, etc. stored in the magnetic disk, and can also be a target image, a classification result, etc., and is stored in the memory 250.

[0053] The central processing unit 270 can include one or more than one processors, and is set to communicate with the memory 250 through at least one communication bus, so as to read the computer program stored in the memory 250, and further realize the operation and processing of the massive data 255 in the memory 250. For example, the classification method for autism spectrum disorder is completed in the form of reading the application program 253 stored in the memory 250 by the central processing unit 270.

[0054] In addition, the present application can also be implemented by a hardware circuit or a combination of a hardware circuit and software. Therefore, the implementation of the present application is not limited to any specific hardware circuit, software, and the combination of both.

[0055] Please refer to Figure 3 , the embodiment of the present application provides a classification method for autism spectrum disorder. This method is applicable to an electronic device. For example, this electronic device can be Figure 1 the server 130 in the shown implementation environment, and the hardware structure of this electronic device can be as Figure 2 shown.

[0056] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an example of an electronic device for illustration, but this is not a specific limitation thereto.

[0057] As Figure 3 shown, the method may include the following steps:

[0058] Step 310, obtain a target image.

[0059] Among them, the target image is an image including a target object. For example, the target image is a facial image of the target object, and the target object refers to any person. For example, it may be a patient with ASD, which is not limited herein.

[0060] The target image is obtained by an image acquisition device shooting and collecting the target object. Among them, the image acquisition device may be an electronic device with an image acquisition function, such as a camera, a smart phone equipped with a camera, and so on.

[0061] Regarding the acquisition of the target image, the target image may be derived from an image shot and collected by the image acquisition device in real time, or may be a target image shot and collected by the image acquisition device during a historical time period pre-stored in the electronic device. Then, for the electronic device, after the image acquisition device shoots and collects the target image, the target image can be processed in real time, or can be pre-stored and then processed. For example, the target image is processed when the CPU of the electronic device is low, or the target image is processed according to the instructions of the staff. Thus, the identification of autism spectrum disorder in this embodiment can be directed to the target image obtained in real time, or can be directed to the target image obtained during the historical time period, which is not specifically limited herein.

[0062] Step 330, obtain a fusion feature by capturing the internal relationships between several local regions in the target image and the interaction relationships between the local regions and the global region in the target image.

[0063] First of all, it should be noted that since the current classification of ASD mainly relies on observation and clinicians' experience, this method is time-consuming, prone to misdiagnosis, and highly dependent on professional medical staff. Therefore, the inventors expect to find an ASD classification method that does not rely on humans. The inventors found that facial features can be used as effective indicators for ASD classification. However, there are problems of intra-class inconsistency and inter-class similarity in classifying ASD using facial features. Specifically, intra-class inconsistency means that the samples within the same category (such as ASD or TD) vary greatly. That is, for children in the same autism category, their facial features may be very different. For example, one child in the ASD group may have a blank expression, another child may have wandering eyes, and some children may look very calm. Inter-class similarity means that the samples from different categories look similar. That is, ASD and TD children show very similar facial features in some scenarios. For example, a mildly autistic child may be almost indistinguishable from a TD child when smiling.

[0064] Then, to solve the problems of intra-class inconsistency and inter-class similarity in ASD classification, the inventors further found that by capturing the internal relationships between several local regions in the target image and the interaction relationships between the local regions and the global region in the target image, a fused feature can be obtained. The fused feature can focus on both the global facial features and the local facial features and facial landmark features.

[0065] Among them, the fused feature is a comprehensive feature representation of the internal relationships between local regions and the interaction relationships between local regions and the global region.

[0066] Regarding the internal relationships between local regions, it can refer to the semantic coordination between facial sub-regions such as the eyes, nose, and mouth of the target object. For example, whether the eyes and the corners of the mouth are coordinated, and whether the left and right eyes are symmetrical.

[0067] Regarding the interaction relationships between local regions and the global region, it can refer to the role of each local region in the overall facial context, such as whether the eyes are prominent, and the semantic role of the corners of the mouth in the overall expression.

[0068] It can be understood that the internal relationships between local regions help to extract more consistent representative features within the same category, thereby alleviating the problem of intra-class inconsistency; while the interaction relationships between local and global regions help to capture the semantic differences of facial features in the overall expression, thus enhancing the model's discriminative ability for inter-class similar samples. Through the construction of such fused features, the model can more accurately perceive and understand the subtle differences in ASD-related facial features, improving the classification effect.

[0069] In a possible implementation, the first target feature can be obtained by performing facial landmark detection on the target image; image features can be obtained by performing image feature extraction on the target image, and in combination with the spatial attention mechanism, a significant facial region related to ASD can be recognized from the entire face or specific key local regions in the image features to obtain the second target feature; then, by using the global-local cross-fusion relationship Transfomer encoder and the multi-feature interaction (Multi-features interaction Network, MFI) network, feature cross-fusion and multi-feature interaction are performed on the first target feature and the second target feature, and thus the fused feature can be obtained.

[0070] Step 350, classify the autism spectrum disorder based on the fused feature to obtain a classification result.

[0071] Among them, the classification result is used to indicate the type of autism spectrum disorder to which the target object belongs. For example, if the target object is a child, then the classification result can indicate whether the target object belongs to ASD or TD.

[0072] In a possible implementation, the fused feature is input into one or more fully connected layers (Fully Connected Layer, FC) for feature compression and mapping. Specifically, the fused feature first undergoes a flattening operation and then is input into the fully connected neural network structure, and high-level semantic information is extracted through a series of linear transformations and non-linear activation functions. The output of the last fully connected layer is a classification probability vector, which is used to indicate the autism spectrum disorder category to which the target object belongs.

[0073] For example, if a binary classification structure is adopted, the output is a vector containing two dimensions, corresponding to the typical development (TD) and autism spectrum disorder (ASD) categories respectively, and is converted into a probability form through the Softmax function, and the category corresponding to the maximum probability is taken as the final classification result.

[0074] Through the above process, by capturing the internal relationships between several local regions in the target image and the interaction relationships between the local regions and the global region in the target image, the fused feature is obtained, which can effectively combine the local region and the global region of the target image of the target object and achieve high-precision and high-efficiency classification of ASD; at the same time, the fused feature can effectively adapt to different facial poses of different target objects, avoiding the phenomena of intra-class differences and inter-class similarities. In addition, through automated target image analysis, this method can quickly and effectively assist in the classification of ASD, provide interpretable decision support for medical professionals, and thus improve the early intervention and treatment of ASD patients.

[0075] Please refer to Figure 4, in an exemplary embodiment, step 330 may include the following steps:

[0076] Step 331, obtain the first target feature and the second target feature of the target image.

[0077] Among them, the first target feature is the feature corresponding to the facial landmark points of the target object, and the second target feature is the spatial attention feature of the target object.

[0078] In a possible implementation, obtaining the first target feature may include: performing facial landmark detection on the target image to obtain the first target feature.

[0079] Regarding the facial landmark points, it may refer to the positions of key regions such as the eyes, nose, and mouth of the target object. It should be understood that due to the typical abnormalities in facial expressions, social interactions, and emotional responses of ASD, there are significant differences in the structural distribution and dynamic behavior of its facial landmark points compared with TD children (such as rigid expressions, gaze drift, changes in facial symmetry, etc.). Based on this, facial landmark points can not only reflect facial morphological features but also be used to assist in identifying ASD.

[0080] Regarding facial landmark detection, it can be implemented using MobileFaceNet. MobileFaceNet is a lightweight facial recognition network that can effectively identify facial landmark points in the target image.

[0081] Through the above process, it is possible to extract the feature corresponding to the facial landmark points of the target object based on the target image. The facial landmark points can assist in identifying ASD. As the position information of the key regions, the facial landmark points can reflect the basic morphology of the target object's facial structure and can also reveal behavioral features such as emotional expression and social response.

[0082] In a possible implementation, as Figure 5 shown, obtaining the second target feature may include the following steps:

[0083] Step 410, use at least one convolutional neural network to perform image feature extraction on the target image respectively to obtain at least one image feature of the target image.

[0084] Among them, each image feature corresponds to a different convolutional neural network respectively.

[0085] First of all, it should be noted that due to the different network structures of different convolutional neural networks, the understanding and feature extraction focus of the target image are different. Therefore, using multiple different convolutional neural networks to perform image feature extraction on the target image respectively can improve the expression ability and generalization ability of the model for facial images.

[0086] In a possible implementation, multiple pre-trained convolutional neural networks such as Xception, IR50, MobileNetV2, and ResNet50 are respectively used as image backbone networks to extract image features from the target image, obtaining corresponding image features.

[0087] Specifically, Xception can better capture local textures and edge changes in the target image; IR50 can better fuse deep structural features (identity, pose); MobileNetV2 can better extract key facial regions; ResNet50 can better capture global structural features. Based on this, using multiple CNNs can extract more discriminative features from different perspectives / dimensions, including texture changes, facial structures, local details, and expression styles.

[0088] Furthermore, in the ASD classification task, the facial expressions, angles, and lighting of the target object may vary. A single CNN may have unstable performance in some cases, while the features extracted by multiple CNNs complement each other, helping the model adapt to various complex situations and be able to adapt to target images of the target object in multiple poses.

[0089] Step 430, use the spatial attention mechanism to identify the features related to autism spectrum disorder in each image feature, obtaining the second target feature.

[0090] First of all, the spatial attention mechanism is an attention mechanism in deep learning that can find the most useful position regions in the spatial dimension of image features, that is, identify the features related to autism spectrum disorder in each image feature.

[0091] It can be understood that in the identification of ASD, parts such as the eyes, corners of the mouth, eyebrows, and facial muscle movements of the target object are more likely to reflect certain characteristics of ASD. Therefore, the goal of the spatial attention mechanism is to identify the significant facial regions related to ASD from the entire face or specific key local regions.

[0092] In a possible implementation, use the spatial attention module to respectively identify the features related to autism spectrum disorder in each image feature, obtaining corresponding spatial attention features.

[0093] For example, Xception corresponds to A image features, IR50 corresponds to B image features, MobileNetV2 corresponds to C image features, and ResNet50 corresponds to D image features. Then, by using the spatial attention module to identify the features related to autism spectrum disorder in A image features, B image features, C image features, and D image features respectively, A spatial attention feature, B spatial attention feature, C spatial attention feature, and D spatial attention feature can be obtained. Thus, the second target feature is obtained based on the A spatial attention feature, B spatial attention feature, C spatial attention feature, and D spatial attention feature.

[0094] Under the action of the above embodiments, a variety of pre-trained CNN models are introduced to extract diverse structural features, texture features, and key region information from the target images of the target object respectively. This multi-model joint extraction method improves the richness and discriminability of feature representation and enhances the robustness of the model under complex facial expressions, poses, and lighting changes; the spatial attention mechanism is used to identify the features related to autism spectrum disorder in each image feature respectively to focus on the key facial regions related to ASD, such as eyes, eyebrows, corners of the mouth, etc., so as to screen out more discriminative local information from the image features (global) to form the second target feature, providing a more refined regional representation for subsequent feature fusion and classification.

[0095] Step 333, perform cross-fusion of global features and local features on the first target feature and the second target feature to obtain intermediate features.

[0096] First of all, it should be noted that the first target feature is the local structural feature extracted based on facial landmark points, which can reflect the positions and local detail information of the key regions such as eyes, nose, and mouth in the face of the target object; the second target feature is the global image feature extracted from different convolutional neural networks, and the spatial attention feature obtained after being processed by the spatial attention mechanism, which can reflect the significant regions related to the identification of autism spectrum disorder in the facial region of the target object.

[0097] In a possible implementation, the global and local multi-attention mechanism is used to fuse the first target feature and the second target feature to achieve the coordinated cross-fusion of the first target feature and the first target feature at the global and local levels, and obtain intermediate features.

[0098] It should be noted that in order to achieve the cross - fusion between the first target feature and the second target feature in global features and local features, a Global - Local Cross - Fusion Relation Transformer Encoder (GLCFRTE) can be used to implement the multi - attention mechanism for global and local. This encoder includes multiple Multi - head Self - Attention (MSA) structures, which can fuse the first target feature and the second target feature, and synergistically merge the two - stream features at the global and local levels, combining detailed local features with broader facial features.

[0099] Specifically, GLCFRTE can capture the internal correlation relationships between local regions in the first target feature, the context relationships between significant regions in the second target feature, and the cross - correlation relationships between the first target feature and the second target feature.

[0100] Step 335: Based on the intermediate features, perform multi - feature interaction fusion to obtain fused features.

[0101] First of all, it should be noted that multi - feature interaction fusion can further enhance the discriminative ability of intermediate features for the autism spectrum disorder category.

[0102] In a possible implementation, as Figure 6 shown, step 335 may include the following steps:

[0103] Step 3351: Use the multi - feature interaction mechanism to calculate the attention weights for the intermediate features.

[0104] Among them, the multi - feature interaction mechanism can be implemented through a Multi - Features Interaction Network (MFI).

[0105] The attention weights can be generated in parallel using the multi - head attention mechanism to capture the key feature expressions of the fused features, thereby enhancing the sensitivity and expressiveness of the model to the differences in different facial features. Specifically, the attention mechanism in MFI is used to perform weighted evaluation on each feature embedding of the intermediate features to generate attention weights indicating the importance of each feature dimension or region.

[0106] Step 3353: According to the attention weights, dynamically select and embed the features related to autism spectrum disorder recognition in the intermediate features to obtain fused features.

[0107] Then, according to the attention weights, the feature subset most relevant to the autism spectrum disorder recognition task can be dynamically selected from the intermediate features and weighted combined to obtain the final fused features.

[0108] Through the above cooperation, by calculating the attention weights, the feature embeddings most useful for ASD recognition are dynamically selected, and effective information exchange and integration between different features are allowed, enhancing the model's analysis ability for subtle changes in facial morphology, texture, and other ASD indicators, and improving the robustness when facing different facial expressions and conditions.

[0109] Through the cooperation of the above embodiments, through the cross-fusion of global features and local features, the fusion of the first target feature and the second target feature is achieved, effectively modeling the context dependence and interaction correlation between different regions of the face, highlighting the semantic connections between key parts, thereby reducing the differences between samples within the same category (intra-class inconsistency), and enhancing the ability to distinguish subtle differences between different category samples (alleviating inter-class similarity), and then effectively improving the accuracy and robustness of ASD recognition, significantly improving the ASD classification performance of the model under complex facial features.

[0110] Please refer to Figure 7 , in an exemplary embodiment, the classification method for autism spectrum disorder is implemented through an autism classification model; the autism classification model is a machine learning model obtained through training and having the ability to classify the autism spectrum disorder of a target object based on a target image.

[0111] The training process of the autism classification model may include the following steps:

[0112] Step 610, obtain sample data.

[0113] Among them, the sample data includes sample images and sample results; the sample results are used to indicate the autism spectrum disorder type of the sample object.

[0114] In a specific embodiment, the sample data is sourced from the Autism Facial Image Dataset (KAFID) released by Kaggle. The original dataset contained 3014 images, and 78 duplicate images were excluded to ensure data quality.

[0115] Step 630, perform generalization processing on the training images using image enhancement techniques to obtain multiple training images.

[0116] It can be understood that to improve the generalization ability of the model, image enhancement processing can be performed on the sample images. Specifically, it includes operations such as random grayscale, color jitter, and random rotation, which are not limited herein.

[0117] Step 650, configure the initial autism classification model for training, and input each training image into the initially configured autism classification model for classification to obtain training results.

[0118] Among them, the training results are used to indicate the autism spectrum disorder type of the sample object predicted by the initial autism classification model.

[0119] Configuring the initial autism classification model for training specifically includes the following parameter settings: setting the size of the input target image, setting the training batch size, setting the learning rate, setting the training cycle, setting the loss function, setting the optimizer, setting the evaluation metrics, etc., which are not specifically limited here.

[0120] In a possible implementation, the input size of the training images is uniformly 224×224 to ensure the consistency of the model input; the training batch size is set to 32 to balance the training efficiency and video memory management; the learning rate is set to 5×10^-5 to ensure that the model converges at a stable speed; the training cycle is 100 rounds to fully cover the training data distribution; the loss function uses the Label Smoothing Cross Entropy loss function to alleviate overfitting and improve the model's discrimination ability for boundary samples; the optimizer uses the Adam optimizer, which has the ability of adaptive learning rate and is suitable for the training of complex neural network structures; the evaluation metrics include Accuracy, Recall, Precision, F1-score and AUC (Area Under Curve) to comprehensively measure the model performance.

[0121] Step 670, calculate the target loss function based on the sample results and the training results, and optimize the initial autism classification model based on the target loss function and the Adam optimizer until the classification effect indicates that the trained autism classification model is obtained.

[0122] Among them, the target loss function can be the Label Smoothing Cross Entropy loss function.

[0123] Specifically, compare the training results output by the model with the sample results, and calculate the error based on the Label Smoothing Cross Entropy loss function, that is, the target loss function. Subsequently, based on this target loss function and the configured Adam optimizer, the model parameters are iteratively updated.

[0124] During the training process, the model continuously adjusts the parameters to minimize the loss function until the training converges, and the trained autism classification model with the ability to classify autism spectrum disorders is obtained.

[0125] It should be noted that the ten-fold cross-validation method can also be used to evaluate the robustness of the training process, that is, the data set is divided into 10 subsets. Each time, one subset is used as the validation set, and the remaining 9 subsets are used as the training set, repeating 10 times, so as to reduce the dependence of model training on sample division and improve the reliability of the evaluation results.

[0126] Under the action of the above embodiments, the generalization ability of the autism classification model is improved through image enhancement. By using a suitable target loss function and optimizer, the classification accuracy of the model is improved during the training process, avoiding the overfitting problem, enhancing the practicability and reliability of the model. Finally, the obtained autism classification model can accurately and stably classify whether the target object belongs to ASD.

[0127] Figures 8a to 8d It is a schematic diagram of the specific implementation of the classification of autism spectrum disorder in an application scenario.

[0128] First of all, it should be noted that the overall framework of the MFMLCF model (autism classification model) is as Figure 8a shown. The following is a detailed introduction to the network structure and its feature processing of the MFMLCF model:

[0129] 1. Input layer: The MFMLCF model accepts RGB face images (target images in the classification method of autism spectrum disorder) with a standard size of (224×224 pixels) as input.

[0130] 2. Facial landmark detection: MobileFaceNet is used to detect facial landmark points, which include the positions of key areas such as eyes, nose, and mouth. The detected landmark points are encoded as a feature vector Lf of 256×7×7 (the first target feature in the classification method of autism spectrum disorder) for subsequent fusion and analysis.

[0131] 3. Image feature extraction: Pre-trained convolutional neural networks such as Xception, IR50, MobileNetV2, and ResNet50 are used as the image backbone network to extract image feature maps of size 256×7×7 (image features in the classification method of autism spectrum disorder) from the input image, which are used for feature fusion in subsequent steps.

[0132] 4. Spatial attention module: The spatial attention mechanism aims to extract key information from the input spatial dimension and particularly focus on regions relevant to the task at hand. For face images, this technique enables the MFMLCF model to identify significant facial regions related to ASD from the entire face or specific key local regions. In the present invention, the feature map extracted from the image backbone network is input into the spatial attention module to obtain the spatial attention feature Isf (the second target feature in the classification method of autism spectrum disorder).

[0133] 5. Global-Local Cross-Fusion Relationship Transformer Encoder: This module has global and local multi-head self-attention layers, which can fuse the facial feature point features Lf and the spatial attention features Isf, and synergistically merge the two-stream features at the global and local levels, combining detailed local features with broader facial features to obtain intermediate features. Among them, the MFMLCF model architecture includes four GLCFRT encoders, and the depth of each encoder is 8.

[0134] 6. Multi-Feature Interaction Network (MFI): The intermediate feature token embeddings generated after global-local cross-fusion are input into the MFI network for attention calculation to further enhance feature fusion. The MFI network dynamically selects the feature embeddings that are most useful for ASD recognition by calculating attention weights, and allows the MFMLCF model to effectively communicate and integrate information between different feature sets, enhancing the MFMLCF model's analysis ability for subtle changes in facial morphology, texture, and other ASD metrics, and improving the robustness of the MFMLCF model in the face of different facial expressions and conditions.

[0135] 7. Output layer: A series of comprehensive features generated by MFI are fed into the fully connected layer for the classification of ASD and TD, thereby obtaining the classification results.

[0136] Regarding the training process of the MFMLCF model, the dataset used in this application scenario is the Kaggle Autism Facial Image Dataset, which contains 3014 RGB facial images of children aged 2 to 14. The dataset is divided into two categories: ASD children's facial images and TD children's facial images. To maintain data quality, we excluded 78 duplicate images identified from the original dataset.

[0137] Estimate the skills of the deep learning model using ten - fold cross - validation. All input facial images are resized to a unified size (224×224) to ensure the consistency of model input. In the training stage, image augmentation techniques such as random grayscale conversion, color jittering, and random rotation are adopted to enhance the generalization ability of the model. Configure the training batch size to 32, which helps to balance memory usage and training efficiency. Set the learning rate to 5×10^-5, which is a commonly used learning rate range in deep learning and helps the model to converge stably during training. The model is trained for 100 epochs, which is usually sufficient for the model to achieve good performance on the given dataset. Use the standard label - smoothed cross - entropy loss function, which helps to improve the discriminative ability of the model for different classes and reduce overfitting. Select the Adam optimizer. Evaluation metrics include Accuracy, Recall, Precision, F1 - score, and AUC (Area Under the Curve), etc. These metrics can comprehensively reflect the classification effect of the MFMLCF model.

[0138] It should be noted that during the training process, the weights of the image feature extraction backbone network are updated according to new data, while the weights of the facial landmark detector remain unchanged to ensure the stability of facial landmark point features.

[0139] Furthermore, the MFMLCF model is compared with various leading models on the KAFID dataset using single training - test splits and 10 - fold cross - validation respectively. As shown in Tables 1 and 2, the MFMLCF model outperforms other state - of - the - art (SOTA) methods in several key metrics. Among them, in the single training - test split validation classification experiment, the accuracy, precision, recall, F1 - score, and area under the curve (AUC) scores reach 96.79%, 97.12%, 96.43%, 96.77%, and 99.49% respectively.

[0140] Table 1 Comparison of experimental results using single training - test split on the KAFID dataset

[0141]

[0142]

[0143] Table 2 Comparison of experimental results using 10 - fold cross - validation on the KAFID dataset

[0144]

[0145] To verify the generalization ability of the MFMLCF model, two other datasets were used for testing, namely the Chinese Social Media Autism Children Dataset (CSMACD) and the Bangladesh ASD Children Facial Expression Dataset (BACFED). The former is a video dataset containing 182 Chinese autistic children aged 6 months to 15 years in natural conditions, collected by the inventors themselves, mainly from major Chinese mainstream social media. After screening and screenshotting, 114 photos were finally obtained, including 86 boys and 28 girls. The latter was created by Fakir et al. in 2023 for studying autism-related facial expressions. The children in this dataset are aged between 6 and 14 years old, and it contains 148 photos of 63 autistic children in Bangladesh. One neutral-expression facial photo was selected for each child, and a total of 63 photos were obtained, including 41 boys and 22 girls. For the class balance between ASD and TD, 114 facial pictures of Asian children and 63 facial pictures of Indian children were respectively selected from a large public face dataset as TD children's facial photos in this application scenario, and the gender balance was maintained. As shown in Table 3, the MFMLCF model achieved good results on these two datasets.

[0146] Table 3 Comparison of the experimental results of MACFUT on the KAFID, CSMACD, and BACFED datasets

[0147]

[0148] To better evaluate the effectiveness of the MFMLCF model, a detailed analysis of its performance in different categories was conducted. Three datasets were used as benchmarks to fine-tune several baseline models including Xception, IR50, MobileNetV2, ResNet50, and MobileFaceNet. In this application scenario, the performance of these baseline models was compared with that of MFMLCF through category-specific evaluations. According to Figure 8b the confusion matrix shown, MFMLCF provided satisfactory results, showing the highest accuracy in the ASD and TD categories.

[0149] In addition, this application scenario also evaluated the sensitivity of MFMLCF to the pose and gender of images. The results showed that on the pose change dataset, MFMLCF had stronger classification performance compared with the baseline models, and the results on the gender balance dataset proved that the MFMLCF model could effectively identify and utilize the unique facial features inherent in each gender.

[0150] To demonstrate the ability of the MFMLCF model in learning effective features for ASD detection, this application scenario uses the T-SNE toolkit to visualize high-dimensional ASD-related facial features from the baseline model and the MFMLCF model. The visualization results are as Figure 8c shown, demonstrating the superior ability of the MFMLCF model in extracting discriminative and invariant features, and confirming its effectiveness in reducing intra-class differences and inter-class similarities.

[0151] To demonstrate the effectiveness of the MFMLCF model in identifying ASD-related facial features, this application scenario adopts attention visualization technology. Figure 8d The input images, detected facial landmark images, and corresponding attention maps of a set of facial images are shown. The attention maps highlight the parts that the model considers to be the most discriminative when making classifications. In the images of children with ASD, the attention is concentrated on the forehead (frontal bone prominence), cheekbones, nose, parotid gland, and philtrum areas. While in the images of TD children, the attention is concentrated on the eyebrow arches, glabella, eyes, radix nasi, and chin areas. This finding not only emphasizes the potential of facial features as biomarkers for differentiating ASD and TD at an early stage, but also opens up a way for its practical application in clinical practice, contributing a new perspective to the ongoing discussion on ASD pathology and care.

[0152] In this application scenario, a complete classification method for autism spectrum disorder (ASD) based on children's facial images is constructed. This method introduces a multi-feature multi-level cross fusion transformer network (Multi-feature Multi-level Cross Fusion Transformer, abbreviated as MFMLCF), which fuses facial key point features and image-level features, and combines multi-level feature representation and multiple attention mechanisms, thus significantly improving the model's adaptability to different facial poses and enhancing its robustness to complex facial expressions and perspective changes.

[0153] Specifically, the MFMLCF model uses a spatial attention mechanism to highlight the key facial regions related to ASD, and further models the multi-level associations between local regions through global and local attention mechanisms, enhancing the information interaction and expression ability between features. On the publicly available dataset of facial images of autistic children (KAFID), this model achieves a classification accuracy of 96.786%, and its performance is significantly better than the existing technical solutions.

[0154] In addition, this application scenario also fully considers the influence of gender and racial differences in ASD recognition. By introducing gender- and race-specific facial feature learning mechanisms, the recognition accuracy of the model for diverse populations is further improved. In the model design, interpretable artificial intelligence methods are adopted, making the discrimination process of the model more transparent and traceable. This not only enhances clinicians' understanding and trust in the diagnostic results but also provides strong support for subsequent medical research and clinical practice.

[0155] It should be understood that although the steps in the flowchart of the accompanying drawings are shown sequentially in the direction of the arrows, these steps do not necessarily have to be executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time but can be executed at different times. Their execution order does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0156] The following is an apparatus embodiment of the present application, which can be used to execute the classification method of autism spectrum disorder involved in the present application. For the details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the classification method of autism spectrum disorder involved in the present application.

[0157] Please refer to Figure 9 , in the embodiment of the present application, a classification apparatus 900 for autism spectrum disorder is provided, including but not limited to: an acquisition module 910, a processing module 930, and a classification module 950.

[0158] Among them, the acquisition module 910 is used to acquire a target image; the target image is an image including a target object.

[0159] The processing module 930 is used to obtain a fusion feature by capturing the internal relationships between several local regions in the target image and the interaction relationships between the local regions and the global region in the target image.

[0160] The classification module 950 is used to classify the autism spectrum disorder based on the fusion feature to obtain a classification result; the classification result is used to indicate the type of autism spectrum disorder to which the target object belongs.

[0161] It should be noted that when the classification device for autism spectrum disorder provided in the above embodiments classifies autism spectrum disorder, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the classification device for autism spectrum disorder will be divided into different functional modules to complete all or part of the functions described above.

[0162] In addition, the classification device for autism spectrum disorder provided in the above embodiments and the embodiments of the classification method for autism spectrum disorder belong to the same concept. The specific manners in which each module performs operations have been described in detail in the method embodiments, and will not be repeated here.

[0163] Please refer to Figure 10 , in the embodiments of the present application, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc.

[0164] In Figure 10 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0165] Among them, the data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0166] Optionally, the electronic device 4000 may further include a transceiver 4004. The transceiver 4004 can be used for data interaction between the electronic device and other electronic devices, such as data sending and / or data receiving, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation to the embodiments of the present application.

[0167] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of this application. The processor 4001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0168] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other medium that can be used to carry or store computer programs in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited thereto.

[0169] A computer program is stored on the memory 4003, and the processor 4001 can read the computer program stored in the memory 4003 through the communication bus 4002.

[0170] The computer program is executed by one or more processors 4001 to implement the classification method for autism spectrum disorder in the above embodiments.

[0171] In addition, an embodiment of the present application provides a storage medium on which a computer program is stored, and the computer program is executed by one or more processors to implement the classification method for autism spectrum disorder as described above.

[0172] In an embodiment of the present application, a computer program product is provided, including a computer program, which is executed by one or more processors to implement the classification method for autism spectrum disorder as described above.

[0173] Compared with the related art, this solution captures the internal relationships between several local regions in the target image and the interaction relationships between the local regions and the global region in the target image to obtain a fused feature, which can effectively combine the local region and the global region of the target image of the target object, and achieve high-precision and high-efficiency classification of ASD. At the same time, the fused feature can effectively adapt to different facial poses of different target objects, avoiding the phenomena of intra-class differences and inter-class similarities. In addition, through automated analysis of the target image, this method can quickly and effectively assist in the classification of ASD, providing interpretable decision support for medical professionals, thereby improving the early intervention and treatment of ASD patients, and effectively solving the problem of low accuracy in the classification of autism spectrum disorder existing in the related art.

[0174] The above description is only part of the embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for classifying autism spectrum disorders, characterized in that: include: Acquire a target image; the target image is an image including a target object; Obtaining fusion features by capturing the internal relationship between a plurality of local regions in the target image and the interactive relationship between the local region and the global region in the target image; Autism spectrum disorder is classified according to the fusion features to obtain a classification result; the classification result is used to indicate the type of autism spectrum disorder to which the target object belongs.

2. The method according to claim 1, wherein The fusion feature is obtained by capturing the internal relationship between a plurality of local regions in the target image and the interactive relationship between the local region and the global region in the target image, including: Acquire a first target feature and a second target feature of the target image; Cross-fusing the first target feature and the second target feature with global features and local features to obtain an intermediate feature; Based on the intermediate features, multiple features are interactively fused to obtain the fused features.

3. The method according to claim 2, wherein Before acquiring the first target feature and the second target feature of the target image, the method further includes: Facial landmark detection is performed on the target image to obtain the first target feature; the first target feature is a feature corresponding to the facial landmark of the target object.

4. The method according to claim 2, wherein Before acquiring the first target feature and the second target feature of the target image, the method further includes: Using at least one convolutional neural network to extract image features from the target image, respectively, to obtain at least one image feature of the target image; each image feature corresponds to a different convolutional neural network; A spatial attention mechanism is used to identify features related to autism spectrum disorder in each of the image features to obtain the second target feature.

5. The method according to claim 2, wherein The cross-fusion of global features and local features of the first target feature and the second target feature to obtain an intermediate feature includes: The first target feature and the second target feature are fused by utilizing a global and local multi-attention mechanism to achieve coordinated cross-fusion of the first target feature and the second target feature at the global and local levels to obtain the intermediate feature.

6. The method according to claim 2, wherein The interactive fusion of multiple features based on the intermediate features to obtain the fused features includes: Using a multi-feature interaction mechanism, attention is calculated on the intermediate features to obtain attention weights; According to the attention weights, features related to autism spectrum disorder recognition in the intermediate features are dynamically selected and embedded to obtain fused features.

7. The method according to any one of claims 1 to 6, wherein: The autism spectrum disorder classification method is implemented by an autism classification model; the autism classification model is a machine learning model that is trained and has the ability to classify the autism spectrum disorder of the target subject based on the target image; The training process of the autism classification model includes: Acquire sample data; the sample data includes a sample image and a sample result; the sample result is used to indicate the type of autism spectrum disorder of the sample subject; Performing generalization processing on the training image using image enhancement technology to obtain multiple training images; Performing training configuration for an initial autism classification model, and inputting each of the training images into the initial autism classification model that has completed the training configuration for classification, to obtain a training result; The target loss function is calculated based on the sample results and the training results, and the initial autism classification model is optimized based on the target loss function and the Adam optimizer until the classification effect indicates that the training of the autism classification model is completed.

8. A classification device for autism spectrum disorder, characterized in that: include: An acquisition module, configured to acquire a target image; the target image is an image including a target object; a processing module, configured to obtain fusion features by capturing internal relationships between a plurality of local regions in the target image and interactive relationships between a local region and a global region in the target image; A classification module is used to classify autism spectrum disorder according to the fusion features to obtain a classification result; the classification result is used to indicate the type of autism spectrum disorder to which the target object belongs.

9. An electronic device comprising at least one processor and at least one memory, wherein: The memory stores a computer program, wherein when the computer program is executed by the processor, the method for classifying autism spectrum disorders according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by one or more processors, the computer program implements the method for classifying autism spectrum disorders according to any one of claims 1 to 7.

Images