An autism child attention prediction method based on a Mamba-Unet structure

Abstract

The application discloses an autism children attention prediction method based on a Mamba-Unet structure, and comprises the following steps: firstly, a multi-modal data set for autism children attention prediction is constructed; secondly, an autism children attention prediction model based on the Mamba-Unet structure is constructed and trained; and finally, a non-typical visual attention prediction is performed on a to-be-detected image by using the trained model to obtain a prediction result. The model comprises a pretreatment module, an encoder part, a bottleneck part, a decoder part and an output part which are connected in sequence, wherein an efficient adaptive visual state space block, a frequency domain attention module, a conditional visual state space block and a conditional attention fusion module are arranged; and the output of the encoder part is further transmitted to the decoder part through a jump connection. The method can significantly improve the accuracy of autism spectrum disorder non-typical visual attention prediction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology in computer science, specifically relating to an attention prediction method for autistic children based on the Mamba-Unet structure. Background Technology

[0002] Autism spectrum disorder (ASD) is a complex neurodevelopmental disorder characterized by deficits in social communication, difficulties in social interaction, and restrictive, repetitive behaviors. ASD presents with a variety of symptoms, and there are no definitive biomarkers for definitive diagnosis. Individuals with ASD often struggle with social participation and maintaining attention on specific tasks. Recent epidemiological studies estimate that approximately 1 in 50 to 68 children have ASD, highlighting its prevalence.

[0003] A striking feature of autism spectrum disorder (ASD) is atypical visual attention, which differs from that of typical developmental individuals (TDs). This difference is attributed to the unique sensory perception and detail-oriented cognitive preferences of individuals with ASD. Significant differences exist in visual attention patterns between TDs and TDs; while TDs primarily focus on central objects in an image, they often tend to focus on background elements or less prominent areas such as trees, lights, or the sky. Eye-tracking technology has become a valuable tool for studying these visual attention patterns. Eye-tracking involves capturing, recording, and analyzing eye movements, or gaze points, which are specific locations in a visual scene where an individual's gaze is focused. In the context of ASD, eye-tracking technology has gained significant attention because consistent observation of gaze abnormalities is a hallmark of the disorder. Many psychological studies have utilized eye-tracking technology to analyze gaze patterns in response to verbal or visual stimuli, providing insights into the underlying mechanisms of ASD. These findings not only deepen our understanding of ASD but also offer promising pathways for early diagnosis and intervention, which are crucial for improving long-term prognosis. Furthermore, eye-tracking technology has practical significance for the study of atypical visual attention, including the development of personalized learning materials and the creation of adaptive computer interfaces tailored for individuals with autism. In addition, computer-aided techniques based on atypical visual saliency prediction (i.e., atypical visual attention prediction) have shown promise as diagnostic tools for autism spectrum disorders.

[0004] The visual saliency prediction task (i.e., the visual attention prediction task) aims to estimate the probability of each pixel in an image being gazed upon, thereby predicting human visual attention bias. In recent years, deep learning models have made significant progress in predicting human visual attention, primarily due to the widespread application of large-scale annotated datasets. Existing visual attention prediction methods generally employ deep learning-based feature extraction architectures. Among these, the encoder-decoder framework, represented by U-Net, is widely used, with its encoder typically utilizing a pre-trained object recognition convolutional neural network (CNN) to extract latent features. Transformer-based models, on the other hand, have achieved state-of-the-art predictive performance by effectively capturing long-range spatial dependencies. However, these visual attention prediction methods, which are highly effective in typical populations, have made limited progress in predicting the visual attention of individuals with autism. This is fundamentally due to the lack of high-quality eye-tracking data specifically for this group and the insufficient adaptability of existing deep learning models to predict the visual attention patterns of individuals with autism. Summary of the Invention

[0005] The purpose of this invention is to provide a method for predicting the attention of children with autism based on the Mamba-Unet architecture, thereby addressing the problem of insufficient accuracy in existing methods for predicting the attention of children with autism. This method effectively improves the accuracy of predicting the unique visual attention patterns of individuals with autism spectrum disorder (ASD) by integrating the powerful long-range dependency modeling capabilities of the state-space model in the Mamba deep learning architecture.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for predicting attention in autistic children based on the Mamba-Unet architecture includes the following steps:

[0008] S1: Construct a multimodal dataset for attention prediction in children with autism, and divide the multimodal dataset into a training set, a test set, and a validation set;

[0009] S2: Construct an attention prediction model for autistic children based on the Mamba-Unet structure. The attention prediction model for autistic children based on the Mamba-Unet structure includes a preprocessing module, an encoder part, a bottleneck part, a decoder part, and an output part connected in sequence. The encoder part and the decoder part each include three stages connected in sequence. The outputs of the three stages of the encoder part are also propagated to the three stages of the decoder part through skip connections.

[0010] The preprocessing module is used to reduce the dimensionality of the input image X;

[0011] The encoder section is used to extract features from the feature map. Each stage of the encoder section includes a sequentially connected Efficient Adaptive Visual State Space Block (EA-VSS Block) and Patch Merging downsampling operation. The Efficient Adaptive Visual State Space Block (EA-VSS Block) is used to efficiently capture the global spatial dependencies of the feature map and provide key context modeling capabilities. The Patch Merging downsampling operation is used to halve the spatial resolution and double the number of channels to generate a hierarchical feature map.

[0012] The bottleneck section is used to perform deeper subsampling of the feature map to capture deep features; the bottleneck section includes an efficient adaptive visual state space block (EA-VSS Block), a frequency domain attention module (FDA), and a conditional visual state space block (C-VSS Block). The frequency domain attention module (FDA) is used to achieve enhanced feature modeling with minimal computational overhead while preserving key feature information; the conditional visual state space block (C-VSS Block) is used to fuse global context and optimize spatial dependencies.

[0013] The decoder section is used to perform upsampling processing to obtain fused features. Each stage of the decoder section includes a Conditional Attention Fusion Module (CAF), a Patch Expanding upsampling operation, and a Conditional Visual State Space Block (C-VSS Block). The Conditional Attention Fusion Module (CAF) is used to generate dynamic fusion weights to achieve weighted fusion of multi-scale features. The Patch Expanding upsampling operation is used to expand the channels and reconstruct the spatial dimensions of the feature map, achieving progressive upsampling of spatial resolution and enhancing the fusion capability of local and global features.

[0014] The output section is used to output the prediction results;

[0015] S3: Train the autism children's attention prediction model based on the Mamba-Unet structure using the data in the training set, adjust the model parameters using the data in the validation set, and determine whether overfitting occurs. During the model training and parameter tuning process, a hybrid loss function is used to guide the optimization of model parameters. After training, test the network performance of the model using the data in the test set.

[0016] S4: Using a trained autism children's attention prediction model based on the Mamba-Unet structure, atypical visual attention prediction for autism spectrum disorder is performed on the image to be tested, and the corresponding binary saliency map is obtained. Based on the binary saliency map and the original input image, a predicted heat map is generated to visualize the visual attention area of ​​individuals with autism spectrum disorder.

[0017] Furthermore, the preprocessing module includes an image patching module and a linear projection module for dimensionality reduction of the input image.

[0018] Furthermore, the output of the first stage of the encoder is propagated to the third stage of the decoder through skip connections, the output of the second stage of the encoder is propagated to the second stage of the decoder through skip connections, and the output of the third stage of the encoder is propagated to the first stage of the decoder through skip connections. Each stage contains two skip connection methods. The first skip connection method is to propagate the feature map processed by the efficient adaptive visual state space block (EA-VSS Block) in the encoder to the conditional attention fusion module (CAF) in the decoder. The second skip connection method is to first propagate the feature map processed by the patch merging downsampling operation to the frequency domain attention module (FDA) for further optimization, and then propagate the optimized feature map to the conditional attention fusion module (CAF) in the decoder.

[0019] Furthermore, the High Efficiency Adaptive Visual State Space Block (EA-VSS Block) includes a High Efficiency Adaptive Module (EA) and two Visual State Space Blocks (VSS Blocks); the High Efficiency Adaptive Module (EA) is used for deep feature extraction, prioritizing high-entropy pixels containing more salient information to enhance feature representation; the Visual State Space Blocks (VSS Blocks) are used for feature extraction, applying bidirectional scanning along the horizontal and vertical axes to enhance the spatial dependencies of the captured feature map by integrating local and global dependencies;

[0020] The efficient adaptive visual state space block (EA-VSS Block) in the encoder contains two parallel paths. One path involves two visual state space blocks (VSS Blocks) directly extracting features from the input feature map and propagating them to the subsequent layers of the encoder, i.e., the patch merging downsampling operation. The other path involves the efficient adaptive module (EA) extracting deep features from the input feature map and then propagating them to the conditional attention fusion (CAF) module in the decoder through skip connections.

[0021] The High Efficiency Adaptive Visual State Space Block (EA-VSS Block) in the bottleneck section first processes the input feature map by the High Efficiency Adaptive Module (EA). The output of the High Efficiency Adaptive Module (EA) is then processed by the two Visual State Space Blocks (VSS Blocks) and the output feature map is then propagated to the subsequent layer of the bottleneck section, namely the Frequency Domain Attention Module (FDA).

[0022] Furthermore, the efficient adaptive module (EA) includes a dual-branch feature extraction structure, which comprises a parallel efficient grouped feature branch and an adaptive branch. The efficient grouped feature branch includes grouped convolutions and performs dimension mapping through batch normalization (BN), SiLU activation function, and rearrangement operation. The adaptive branch includes average pooling (AP), 1×1 convolution, dimension mapping using rearrangement operation, and normalization using Softmax function. Then, the efficient adaptive module (EA) performs element-wise multiplication of the output feature maps of the efficient grouped feature branch and the adaptive branch, performs one-dimensional summation on the last layer of the obtained feature map tensor, and finally performs bilinear interpolation upsampling to obtain the output feature map of the efficient adaptive module (EA).

[0023] Furthermore, the Visual State Space Block (VSS Block) includes a main path and a branch path. The main path includes a LayerNorm, a Linear Transformation Operation, a Depthwise Separable Convolution (DWConv), a 2D State Space Modeling Module (SS2D), a LayerNorm, and a Linear Transformation Operation, connected sequentially. The branch path has one Linear Transformation Operation. The output of the first LayerNorm on the main path is processed by the Linear Transformation Operation on the branch path and then multiplied element-wise with the output of the second LayerNorm on the main path. The result of the element-wise multiplication is then input into the second Linear Transformation Operation on the main path for processing. Simultaneously, the output features of the second Linear Transformation Operation on the main path are added to the initial input features of the Visual State Space Block (VSSS Block) through a residual connection to obtain the final output of the Visual State Space Block (VSS Block).

[0024] Furthermore, the frequency domain attention module (FDA) first divides the input feature map into chunks along the channel dimension. Divided into three subsets along the channel dimension , and Each subset is sequentially processed through a 1×1 convolution and a 3×3 depthwise convolution for feature extraction and spatial information modeling, respectively, to obtain feature representations. , and Subsequently, the rearrangement operator is used. Representing features respectively and feature representation Perform a rearrangement operation to obtain the rearranged feature representation. and feature representation And respectively, the features after rearrangement are represented. and feature representation A Fast Fourier Transform (FFT) is applied and projected into the frequency domain to extract frequency domain features and enhance the model's ability to model global structural information. Then, the attention map F is obtained by performing element-wise multiplication on the result of the FFT in the frequency domain. AB And attention map F AB The inverse operation involves performing a two-dimensional inverse Fourier transform (IFFT) to obtain the attention map A. Then, the attention map A is normalized using a layer normalization layer (LayerNorm) before being compared with the feature representation. Element-wise multiplication yields the attention-weighted feature map. Finally, an attention-weighted feature map is processed by a 1×1 convolution. After processing, it is then connected to the input feature map X via residual connections. F Element-wise addition is performed to refine the feature map, resulting in an output feature map. .

[0025] Furthermore, the Conditional Visual State Space Block (C-VSS Block) includes a central main path, a parallel feedback path, and a right branch. The central main path includes a layer normalization layer (LayerNorm), an affine transformation operation (Shift & Scale), a linear transformation operation (Linear), a depthwise separable convolution (DWConv), a SiLU activation function, a two-dimensional state space modeling module (SS2D), a scaling function (Scale), a layer normalization layer (LayerNorm), an affine transformation operation (Shift & Scale), and a linear transformation operation (Linear) connected in sequence. The parallel feedback path propagates category feedback information to the two-dimensional state space modeling module (SS2D) and the two affine transformation operations (Shift & Scale) in the central main path through the MLP module. The right branch includes a linear transformation operation (Linear) and a SiLU activation function connected in sequence. The output of the first affine transformation operation (Shift & Scale) in the central main path passes through the right branch and is then combined with the output of the second affine transformation operation (Shift & Scale) in the central main path. The output of the &Scale operation is multiplied element-wise and then propagated to the second linear transformation operation (Linear) on the main path. At the same time, the output of the second linear transformation operation (Linear) on the main path is added element-wise to the input feature map of the Conditional Visual State Space Block (C-VSS Block) through the residual connection to obtain the final output of the Conditional Visual State Space Block (C-VSS Block).

[0026] Furthermore, in the parallel feedback path of the Conditional Visual State Space Block (C-VSS Block), the category feedback information is extracted by the MLP module based on the data category information contained in the predefined label T; label T contains learnable tokens, and the dimension of label T is D; the tokens are fed into the MLP module for regressing the modulation parameter scaling factor. Translation factor The MLP module is represented as ): → ,in ) indicates an MLP module. For input features, Indicates model parameters, This indicates that the input features are a real matrix of size 4 rows and D columns. The output is a 4x5 real matrix Y, where each row of Y corresponds to an input token, and five key parameters are output: Select the row corresponding to input label T from matrix Y. , This represents the z-th row vector in matrix Y, which is the modulation parameter row corresponding to the input label T. Let it be a vector S, i.e., S = [ , , , , The vector S is the set of modulation parameters corresponding to the input marker T; each parameter in vector S is used to adjust the computational behavior at different locations in the model, wherein... The modulation operation is located in the second affine transformation operation (Shift & Scale) in the middle main path. Used to control the intensity of features in the 2D state-space modeling module (SS2D), The modulation operation is the first affine transformation operation (Shift & Scale) located in the middle main path; the specific formula for adjusting the computational behavior at different positions in the model according to the parameters in vector S is expressed as follows:

[0027] ,

[0028] in, This represents a feature map modulated by dynamic parameters. This represents the feature map before modulation; It is a scaling factor, including , , ; It is the translation factor, including , ; This indicates element-wise multiplication.

[0029] Furthermore, in the decoder section, the first and second stages each consist of a Conditional Attention Fusion Module (CAF), a Patch Expanding upsampling operation, and two Conditional Visual State Space Blocks (C-VSS Blocks) connected in sequence. The third stage consists of a Conditional Attention Fusion Module (CAF), a Patch Expanding upsampling operation, and a Conditional Visual State Space Block (C-VSS Block) connected in sequence. The Conditional Attention Fusion Module (CAF) receives the feature maps passed from the two skip connections and the output of the previous Conditional Visual State Space Block (C-VSS Block) as input, and generates dynamic fusion weights based on the condition information extracted from the Conditional Visual State Space Block (C-VSS Block), thereby realizing the weighted fusion of multi-scale features through dynamic fusion weights.

[0030] Furthermore, the Conditional Attention Fusion (CAF) module includes bilinear interpolation, an adaptive feature weighting branch, and a multi-scale feature interaction branch. The adaptive feature weighting branch includes a conditional weight generation module, which comprises a 1×1 convolution, a linear transformation operation, a sigmoid activation function, a shape transformation (View), and a dimension expansion operation connected in sequence. The multi-scale feature interaction branch includes a shape transformation (View), a multi-head attention mechanism (MHA), and a shape transformation (View). The Conditional Attention Fusion (CAF) module first uses bilinear interpolation to process the three feature maps received from the three different modules. , , The spatial dimensions are adjusted to obtain three feature maps with adjusted dimensions;

[0031] Meanwhile, in the adaptive feature weighting branch, the conditional weight generation module calculates the fusion weights based on the conditional information extracted from the Conditional Visual State Space Block (C-VSS Block). Then, the three resized feature maps are respectively fused with the weights. Element-wise multiplication is performed to obtain three conditionally modulated weighted feature maps; then, the three conditionally modulated weighted feature maps are summed element-wise to obtain the aggregated representation. ;

[0032] The multi-scale feature interaction branch first performs shape transformation (View) on the three scaled feature maps, rearranging the tensor dimensions of the feature map representations, and then inputs them into the multi-head attention mechanism (MHA) for feature interaction; then, it performs shape transformation (View) on the output feature map of the MHA, rearranging the tensor dimensions of the feature map representations again, to obtain the attention-enhanced feature representation. Finally, the aggregation representation will be... and feature representation By adding elements one by one, we obtain the fused feature map. .

[0033] Furthermore, the output section includes a final patch expanding upsampling operation and a linear projection layer, which are used to upsample the feature map to restore the feature map to the resolution of the original input image, thereby outputting the final saliency prediction map.

[0034] Furthermore, the formula for calculating the hybrid loss function is as follows:

[0035] ,

[0036] in, Represents the mixed loss function. A true saliency plot, s represents the true fixation map, and s is the predicted saliency map generated by the model. The weighting coefficients representing the KL divergence. Denotes the KL divergence function. The weighting coefficient represents the correlation coefficient. Represents the correlation coefficient function. Weighting coefficients representing similarity Represents the similarity function. The weighting coefficients represent the significance values ​​of the normalized scan path. This represents the saliency function of the normalized scan path. The weighting coefficients represent the mean squared error. This represents the mean square error function.

[0037] The beneficial effects of this invention are as follows:

[0038] (1) This invention constructs an attention prediction model for children with autism based on the deep learning architecture Mamba. It utilizes the linear computational complexity of the deep learning architecture Mamba to effectively capture remote visual information, which significantly improves the accuracy of atypical visual attention prediction for autism spectrum disorder.

[0039] (2) This invention constructs an efficient adaptive visual state space block (EA-VSS Block), which achieves adaptive downsampling and dynamic feature extraction during multi-scale sampling, enhancing the model's ability to focus on the target region. In addition, the frequency domain attention module (FDA) is integrated into the skip connection to calculate scaled dot product attention in the frequency domain, thereby reducing spatial and temporal complexity while preserving key features.

[0040] (3) The Conditional Attention Fusion (CAF) module constructed in this invention aims to combine the state-space conditional reflexes of the deep learning architecture Mamba with multi-head attention, retain rich attention information during the upsampling process, and improve feature fusion. It achieves state-of-the-art performance in the application of the autism spectrum disorder eye-tracking dataset.

[0041] (4) This invention utilizes the long-range dependency modeling capability of the state-space model in the Mamba deep learning architecture, introduces an encoder and decoder architecture based on the U-Net structure, and proposes an attention prediction model for autistic children based on the Mamba-Unet structure. It also introduces a frequency-domain attention mechanism to achieve computational efficiency and context-aware saliency prediction. Extensive experiments demonstrate that this model achieves state-of-the-art performance on specific benchmarks for autism spectrum disorder, confirming its effectiveness in predicting atypical visual attention. This invention applies the state-space model to atypical visual attention prediction, promoting the evolution of computer-aided intelligent diagnostic technology for autism spectrum disorder towards interpretable clinical decision support systems. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a structural diagram of the attention prediction model for autistic children based on the Mamba-Unet structure in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the structure of the High-Efficiency Adaptive Visual State Space Block (EA-VSS Block) in an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the structure of the Visual State Space Block (VSS Block) in an embodiment of the present invention.

[0046] Figure 4 This is a schematic diagram of the structure of the Conditional Visual State Space Block (C-VSS Block) in an embodiment of the present invention.

[0047] Figure 5 This is a schematic diagram of the structure of the high-efficiency adaptive module (EA) in an embodiment of the present invention.

[0048] Figure 6 This is a schematic diagram of the frequency domain attention module (FDA) in an embodiment of the present invention.

[0049] Figure 7 This is a schematic diagram of the Conditional Attention Fusion (CAF) module in an embodiment of the present invention.

[0050] Figure 8 This is a visualization showing the comparison between the method of this invention and nine other existing methods for predicting attention in children with autism.

[0051] Figure 9 This is a visualization of the results of the attention prediction method for autistic children based on the Mamba-Unet structure in an embodiment of the present invention, along with a visualization of the actual labeled image. Detailed Implementation

[0052] To better understand the above-described objects, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention; however, the invention may be practiced in other ways different from those described herein, and therefore, the invention is not limited to the specific embodiments disclosed below.

[0053] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art described herein. The terms “first,” “second,” and similar terms used herein do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “a” or “one,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “connected,” “linked,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “up,” “down,” “left,” and “right,” etc., are used only to indicate relative positional relationships, which change accordingly when the absolute position of the described object changes.

[0054] like Figure 1 As shown, the present invention provides a method for predicting the attention of autistic children based on the Mamba-Unet structure, comprising the following steps:

[0055] Step S1: Construct a multimodal dataset for attention prediction in children with autism, and divide the multimodal dataset into a training set, a test set, and a validation set. In this embodiment of the invention, the Saliency4ASD dataset, which is widely used in the field of attention prediction in children with autism, is used, and the data in the Saliency4ASD dataset is divided into a training set, a test set, and a validation set in a ratio of 8:1:1.

[0056] The Saliency4ASD dataset contains 300 high-resolution images, each accompanied by an eye-tracking image of a subject with autism spectrum disorder (ASD) and a corresponding ground-truth saliency map. The images in the Saliency4ASD dataset cover a variety of salient object types, including animals, people, and natural scenes, making it suitable for evaluating the performance of visual attention prediction methods for ASD individuals in complex scenarios.

[0057] Step S2: Construct an attention prediction model for autistic children based on the Mamba-Unet structure. (The Mamba-Unet-based attention prediction model for autistic children is described below.) Figure 1 As shown, the model adopts a U-Net structure, which includes a preprocessing module, an encoder section, a bottleneck section, a decoder section, an output section, and multiple skip connections.

[0058] The preprocessing module includes an image patching module and a linear embedding module for dimensionality reduction of the input image. The preprocessing module first performs dimensionality reduction on the input image X using the image patching module and the linear embedding module, reducing the size of the input image X to [value missing]. The dimensionality-reduced feature map Xs is obtained, and then Xs is input into the encoder part for feature extraction; where, R represents the set of real numbers, H represents the height of the image, W represents the width of the image, and C represents the number of color channels in the image.

[0059] The encoder consists of three stages, each comprising a sequentially connected Efficient Adaptive Visual State Space Block (EA-VSS Block) and a Patch Merging downsampling operation. The EA-VSS Block efficiently captures the global spatial dependencies of the feature maps, providing key context modeling capabilities. The Patch Merging downsampling operation halves the spatial resolution and doubles the number of channels to generate hierarchical feature maps. The three stages of the encoder output sizes of [size missing]. , , The feature map is obtained from the encoder. The three stages of the encoder are connected sequentially, and the output feature map of the encoder is input to the bottleneck part for deeper downsampling and efficient feature map capture processing.

[0060] Simultaneously, the outputs of the three stages of the encoder are propagated to the three stages of the decoder via skip connections. Specifically, the output of the first stage of the encoder is propagated to the third stage of the decoder, the output of the second stage of the encoder is propagated to the second stage of the decoder, and the output of the third stage of the encoder is propagated to the first stage of the decoder. The U-Net structure employs two skip connection methods: the first method propagates the feature map processed by the efficient adaptive module (EA) within the efficient adaptive visual state space block (EA-VSS Block) of the encoder to the conditional attention fusion module (CAF) of the decoder; the second method propagates the feature map processed by the patch merging downsampling operation to the frequency domain attention module (FDA) for further optimization, and then propagates the optimized feature map to the conditional attention fusion module (CAF) of the decoder.

[0061] The bottleneck section comprises an efficient adaptive visual state space block (EA-VSS Block), a frequency domain attention module (FDA), and two conditional visual state space blocks (C-VSS Blocks). The frequency domain attention module (FDA) is used to achieve enhanced feature modeling with minimal computational overhead while preserving key feature information. The conditional visual state space blocks (C-VSS Blocks) are used to effectively utilize visual cues specific to the visual patterns of autism spectrum disorder, extract conditional information from the feature map, fuse global context, and optimize spatial dependencies. The efficient adaptive visual state space block (EA-VSS Block), the frequency domain attention module (FDA), and the conditional visual state space blocks (C-VSS Blocks) all output a size of [size missing]. The feature map of the bottleneck section is input to the decoder section for upsampling.

[0062] The decoder consists of three stages. The first and second stages each consist of a Conditional Attention Fusion (CAF) module, a Patch Expanding upsampling operation, and two Conditional Visual State Space Blocks (C-VSSBlocks) connected sequentially. The third stage also consists of a CAF module, a Patch Expanding upsampling operation, and a C-VSSBlock connected sequentially. The CAF module receives feature maps from two skip connections and the output of the previous C-VSSBlock as input. It generates dynamic fusion weights based on the condition information extracted from the C-VSSBlock, thereby achieving weighted fusion of multi-scale features, improving fusion efficiency while preserving the detail accuracy of saliency prediction. The Patch Expanding upsampling operation consists of a Linear Layer and a rearrangement operator. It consists of a layer normalization layer (LayerNorm) used to expand the channels and reconstruct the spatial dimensions of the feature map, achieving progressive upsampling of spatial resolution and enhancing the fusion capability of local and global features. The three stages of the decoder are connected sequentially, and the output size of the three stages of the decoder is respectively... , , The feature map is then used. Finally, the output of the decoder is fed back into the output section for upsampling.

[0063] The output section includes a final patch expanding upsampling operation and a linear projection layer to upsample the feature map, restoring it to the resolution of the original input image X and generating the prediction result. The final patch expanding upsampling operation also consists of a linear layer and a rearrangement operator. The decoder consists of a LayerNorm normalization layer, but its scaling factor is greater than that of the Patch Expanding operation in the decoder section. The Final Patch Expanding operation is used to restore the size of the feature map output by the decoder section to the same resolution (i.e., size) as the original input image X through a four-fold spatial reconstruction, and is used to generate the final saliency prediction map. In this embodiment, the scaling factor of the Patch Expanding operation in the decoder section is set to 2 times (dim_scale=2); the scaling factor of the Final Patch Expanding operation in the output section is set to 4 times (dim_scale=4).

[0064] In this embodiment of the invention, the structure of the High-Efficiency Adaptive Visual State Space Block (EA-VSS Block) is as follows: Figure 2 As shown. The Efficient Adaptive Visual State Space Block (EA-VSS Block) includes an Efficient Adaptive Module (EA) and two Visual State Space Blocks (VSS Blocks). The Efficient Adaptive Module (EA) is used for deep feature extraction, prioritizing high-entropy pixels containing more salient information to enhance feature representation. The Visual State Space Blocks (VSS Blocks) are used for feature extraction, applying bidirectional scanning along the horizontal and vertical axes to enhance the spatial dependencies of the captured feature map by integrating local and global dependencies. The encoder part of the model has three Efficient Adaptive Visual State Space Blocks (EA-VSSS Blocks), corresponding to Layer numbers 0, 1, and 2, respectively. In the three Efficient Adaptive Visual State Space Blocks (EA-VSS Blocks) of the encoder part, the input feature map is divided into two parallel paths for operation. One path is directly processed by the two Visual State Space Blocks (VSS Blocks) and propagated to the subsequent layers of the encoder part, i.e., the Patch Merging operation. The other path is used for deep feature extraction by the Efficient Adaptive Module (EA) and then propagated to the Conditional Attention Fusion (CAF) module of the decoder part through skip connections. The bottleneck part of the model contains an efficient adaptive visual state space block (EA-VSS Block), corresponding to layer number 3. In the efficient adaptive visual state space block (EA-VSS Block) of the bottleneck part, the efficient adaptive module (EA) first processes the input feature map. The output of the efficient adaptive module (EA) enters two visual state space modules (VSS Block) for processing, and then the output feature map is propagated to the subsequent layer of the bottleneck part, namely the frequency domain attention module (FDA).

[0065] The efficient adaptive module (EA) used in this embodiment of the invention can reduce parameter complexity and computational cost while retaining rich semantic information. The structure of the efficient adaptive module (EA) is as follows: Figure 5 As shown, this module employs a dual-branch feature extraction structure and introduces a group convolution mechanism within the branches. By dividing the input features into N subgroups and performing convolution operations on each group, an effective mapping from input features to output features is achieved. Compared to traditional fully connected convolution operations, the group convolution can significantly reduce the number of parameters while maintaining feature expressive power. Its total number of parameters is approximately 1 / N of that of traditional convolution, where N is the number of groups in the group convolution, thereby effectively improving the computational efficiency of the network and reducing redundancy.

[0066] like Figure 5 As shown, one branch of the efficient adaptive module (EA) is the efficient grouped feature branch. First, the feature map Xs input to the EA is subjected to grouped convolution, dividing the channels of feature map Xs into non-overlapping segments. Each segment contains C / 16 channels and is processed independently. The resulting feature map undergoes dimension mapping. The dimension mapping is first normalized by a batch normalization layer (BN, i.e., BatchNorm) and activated by the SiLU activation function, and then a rearrangement operator is applied. By reorganizing the spatial and channel dimensions, the feature map is ultimately mapped to the new space, completing the dimensional mapping of the feature map and obtaining the feature map. This process can be specifically represented as follows:

[0067] ,

[0068] in, This represents the feature map output by the efficient grouping feature branch. ; This indicates that a rearrangement operation is performed using the rearrangement operator. This represents the SiLU activation function. Indicates the batch normalization layer. This indicates grouped convolution.

[0069] Another branch of the efficient adaptive module (EA) is the adaptive branch. To enhance spatial adaptability, the adaptive branch first applies average pooling (AP) to the feature map Xs input to the efficient adaptive module (EA), then performs feature extraction through 1×1 convolution; and then uses a rearrangement operator. The feature map is rearranged to map its dimensions. After dimension mapping, the Softmax function is used to normalize the feature map along the last dimension to obtain the final feature map. This process can be specifically represented as follows:

[0070] ,

[0071] in, The feature map representing the output of the adaptive branch. ; This indicates the average pooling operation. Represents a 1×1 convolution; This indicates that the Softmax function is applied along the last dimension to ensure normalized spatial weights.

[0072] Then, the efficient adaptive module (EA) will... and feature map Element-wise multiplication is performed, followed by one-dimensional summation of the last layer of the obtained feature map tensor. Finally, bilinear interpolation upsampling is applied to restore the feature map to the size of the input feature map Xs, thus obtaining the feature map. This process can be specifically represented as follows:

[0073] ,

[0074] in, This represents the output feature map of the efficient adaptive module. Interp represents bilinear interpolation. This indicates that the tensor is summed along the last layer of one dimension. This means that the height and width of the input feature map will both be doubled. This represents element-wise multiplication; this design enhances spatial adaptability while maintaining computational efficiency by leveraging grouped convolution processing and shared attention across spatial segments.

[0075] In this embodiment of the invention, the structure of the Visual State Space Block (VSS Block) is as follows: Figure 3As shown. The Visual State Space Block (VSS Block) includes a main path and a branch path. The main path includes a LayerNorm, a Linear Transformation Operation, a Depthwise Separable Convolution (DWConv), a 2D State Space Modeling Module (SS2D), a LayerNorm, and a Linear Transformation Operation, connected sequentially. A branch path includes one Linear Transformation Operation. The output of the first LayerNorm on the main path is processed by the Linear Transformation Operation on the branch path and then multiplied element-wise with the output of the second LayerNorm on the main path. The result of this element-wise multiplication is then input into the second Linear Transformation Operation on the main path for further processing. Simultaneously, the output features of the second Linear Transformation Operation on the main path are added to the initial input features of the Visual State Space Block (VSS Block) through a residual connection to obtain the final output of the Visual State Space Block (VSS Block).

[0076] The feature processing process of the Visual State Space Block (VSS Block) is as follows: First, the input feature map F of the Visual State Space Block is normalized by a LayerNorm, and the feature map is mapped to a high-dimensional space by a linear transformation operation. Then, a depthwise separable convolution (DWConv) is introduced to extract local spatial features. Next, the two-dimensional state space modeling module (SS2D) captures long-distance spatial dependencies, and is normalized again by a LayerNorm. The output of the first normalization layer (LayerNorm) is processed by a linear transformation operation on the branch. The result is then multiplied element-wise with the output features of the second normalization layer (LayerNorm) on the main branch to achieve gated modulation. The result after element-wise multiplication is then restored to feature dimension by the second linear transformation operation on the main branch. Finally, the output features of the second linear transformation operation (Linear) are residually concatenated with the initial input features to enhance context modeling capability and gradient propagation stability, thereby outputting a feature representation with rich spatial modeling capabilities.

[0077] The core of the Visual State Space Block (VSS Block) is the Two-Dimensional State Space Modeling Module (SS2D). The SS2D uses a cross-scanning strategy to expand the input image into unidirectional sequences along the four diagonal directions (top left to bottom right, top right to bottom left, bottom left to top right, and bottom right to top left), and then inputs these sequences into the Selective State Space Model (SSM) contained within the SS2D for modeling. Each direction's scan sequence is processed independently and then recombined into a two-dimensional feature map through weighted fusion, enabling each pixel to integrate multi-directional global contextual information. This design, through the dynamic parameter mechanism of the Selective State Space Model (SSM), adaptively captures local details and long-range spatial dependencies, significantly enhancing the modeling capability of the Mamba deep learning architecture based on the Selective State Space Model (SSM) for complex visual patterns. The specific feature processing method of the Visual State Space Block (VSS Block) is as follows:

[0078] ,

[0079] ,

[0080] ,

[0081] in, This represents the output feature of the first normalized layer, and F represents the input feature map of the Visual State Space Module (VSSBlock). , This indicates the first normalized layer; This represents the output features of the second normalized layer. This indicates the second normalization layer. This represents a two-dimensional state-space modeling module. This represents depthwise separable convolution. Represents a linear transformation operation; This represents the output feature map of the visual state space module. This indicates element-wise multiplication.

[0082] The structure of the frequency domain attention module (FDA) in this embodiment of the invention is as follows: Figure 6 As shown, the Frequency Domain Attention Module (FDA) utilizes frequency domain transformation to achieve enhanced feature modeling with minimal computational overhead, while simultaneously preserving key feature information. The input feature map of the Frequency Domain Attention Module (FDA) is defined. The width and height of the spatial dimension are The number of channels is First, the input feature map is divided into chunks along the channel dimension. Divided into three subsets along the channel dimension: , and Each subset is sequentially processed through a 1×1 convolution and a 3×3 depthwise convolution for feature extraction and spatial information modeling, respectively, to obtain feature representations. , and While maintaining computational efficiency, more refined multi-scale salient features are extracted. Subsequently, a rearrangement operator is used. Representing features respectively and feature representation Perform a rearrangement operation to obtain the rearranged feature representation. and feature representation And respectively, the features after rearrangement are represented. and feature representation A Fast Fourier Transform (FFT) is applied and projected into the frequency domain to extract frequency domain features and enhance the model's ability to model global structural information. Then, the attention map F is obtained by performing element-wise multiplication on the result of the FFT in the frequency domain. AB Then, for the attention map F AB Performing the inverse operation involves executing the two-dimensional inverse Fourier transform (IFFT) to obtain attention map A:

[0083] ,

[0084] ,

[0085] in, This represents the Fast Fourier Transform, also known as FFT(·). represents the Fourier transform of the two-dimensional inverse matrix, and * represents the conjugate transpose; This indicates element-wise multiplication;

[0086] Then, attention map A is normalized using a LayerNorm to obtain the attention map. And pay attention to the diagram With feature representation Element-wise multiplication yields the attention-weighted feature map. :

[0087] ,

[0088] Finally, an attention-weighted feature map is processed by a 1×1 convolution. After processing, it is then connected to the input feature map X via residual connections. F Element-wise addition is performed to refine the feature map, resulting in an output feature map. :

[0089] ,

[0090] By utilizing frequency domain attention, the aforementioned frequency domain attention module (FDA) significantly reduces computational complexity while maintaining the ability to model long-range dependencies.

[0091] The Conditional Visual State Space Block (C-VSS Block) in this embodiment of the invention introduces a class feedback mechanism to enhance semantic perception capabilities, such as... Figure 4 As shown, the Conditional Visual State Space Block (C-VSS Block) includes a central main path, a parallel feedback path, and a right branch. The central main path includes a layer normalization layer (LayerNorm), an affine transformation operation (Shift & Scale), a linear transformation operation (Linear), a depthwise separable convolution (DWConv), a SiLU activation function, a 2D state space modeling module (SS2D), a scaling function (Scale), a layer normalization layer (LayerNorm), an affine transformation operation (Shift & Scale), and a linear transformation operation (Linear) connected in sequence. The parallel feedback path propagates category feedback information to the 2D state space modeling module (SS2D) and the two affine transformation operations (Shift & Scale) in the central main path through the MLP module. The right branch includes a linear transformation operation (Linear) and a SiLU activation function connected in sequence. The output of the first affine transformation operation (Shift & Scale) in the central main path passes through the right branch and is then combined with the output of the second affine transformation operation (Shift & Scale) in the central main path. The output of the &Scale operation is multiplied element-wise and then propagated to the second linear transformation operation (Linear) on the main path. At the same time, the output of the second linear transformation operation (Linear) on the main path is added element-wise to the input feature map of the Conditional Visual State Space Block (C-VSS Block) through the residual connection to obtain the final output of the Conditional Visual State Space Block (C-VSS Block).

[0092] The feature processing of the Conditional Visual State Space Block (C-VSS Block) is as follows: First, the main path performs layer normalization and affine transformation (Shift & Scale) operations on the input feature map of the C-VSS Block to normalize the feature distribution. Next, a linear transformation is performed to map the channels of the feature map, which is then fed into a depthwise separable convolution (DWConv) to extract local spatial structure information. Afterward, a SiLU activation function is used for nonlinear transformation, and the result is fed into the 2D State Space Modeling Module (SS2D) for processing to capture global spatial dependencies through the state transition process. Subsequently, the output features of the SS2D are scaled and modulated, and layer normalization and affine transformation (Shift & Scale) operations are performed again. In the parallel feedback path, the category feedback information is extracted by the MLP module (i.e., the multilayer perceptron) based on the data category information contained in the predefined label T, and propagated to the SS2D and the two affine transformation operations (Shift & Scale) in the main path. The output of the first affine transformation operation (Shift & Scale) in the main central path is processed by the linear transformation operation (Linear) and SiLU activation function in the right branch. This processed output is then interactively modulated with the feature output of the second affine transformation operation (Shift & Scale) in the main central path through element-wise multiplication, achieving semantic enhancement. Finally, the element-wise multiplied feature is processed by the linear transformation operation (Linear) and added element-wise with the input feature map through residual connections. The output is a high-quality feature that integrates category semantic feedback and spatial modeling capabilities. The MLP module modulates the feature map using a specific input label T. The adaptive mechanism of the MLP module effectively utilizes visual cues unique to the visual patterns of autism spectrum disorder, thereby improving the accuracy of atypical visual attention prediction. The modulation of the feature map is achieved through dynamic parameters (scaling factor) obtained by the MLP module. Translation factor Implement dynamic scaling and translation operations:

[0093] ,

[0094] in, This represents a feature map modulated by dynamic parameters. This represents the feature map before modulation; It is a scaling factor, including , , ; It is the translation factor, including , ; This represents element-wise multiplication. To capture specific features of the content, a label T is defined, containing learnable tokens, and the dimension of label T is D. The tokens are fed into an MLP module consisting of K hidden layers, where the dimension of each layer of the MLP module is... , , ..., , used for regression modulation parameter scaling factor Translation factor This MLP module can be represented as ): → ,in ) indicates an MLP module. For input features, Indicates model parameters, This indicates that the input features are a real matrix of size 4 rows and D columns. The output is a 4x5 real matrix Y, where each row of Y corresponds to an input token, and five key parameters are output: Select the row corresponding to input label T from matrix Y. , This represents the z-th row vector in matrix Y, which is the modulation parameter row corresponding to the input label T. Let it be a vector S, i.e., S = [ , , , , The vector S is the set of modulation parameters corresponding to the input marker T; each parameter in vector S is used to adjust the computational behavior at different locations in the model, wherein: The modulation operation is located in the second affine transformation operation (Shift & Scale) in the middle main path. Used to control the intensity of features in the 2D state-space modeling module (SS2D), The modulation operation is the first affine transformation operation (Shift & Scale) located in the middle main path.

[0095] The Conditional Attention Fusion (CAF) module in this embodiment of the invention enhances feature integration by combining a conditional weighting mechanism with multi-head self-attention. The structure of the Conditional Attention Fusion (CAF) module is as follows: Figure 7As shown, the Conditional Attention Fusion (CAF) module utilizes the weighting mechanism of conditional modulation and global attention to adaptively balance the importance of local and contextual features, thereby achieving more effective feature representation learning. The feature fusion process of the CAF module mainly includes two parts: adaptive feature weighting and multi-scale feature interaction. Three feature maps extracted from three different modules (the EA-VSS Block in the encoder part of one skip connection propagation path, the FDA in another skip connection propagation path, and the previous C-VSS Block) are used as the input feature maps for the CAF module. Including feature maps Feature map Feature map The Conditional Attention Fusion (CAF) module first uses bilinear interpolation to fuse the feature maps. Feature map Feature map The spatial dimensions are adjusted, and the specific formula is as follows:

[0096] ,

[0097] in, The feature map after size adjustment includes Interp represents bilinear interpolation. This represents the input feature map of the conditional attention fusion module. Represents the input feature map The size is ;

[0098] To achieve adaptive feature selection, the Conditional Attention Fusion (CAF) module includes a conditional weight generation module, which calculates the fusion weights based on conditional information extracted from the Conditional Visual State Space Block (C-VSS Block). The conditional weight generation module consists of sequentially connected 1×1 convolutions, linear transformation operations, sigmoid activation functions, shape transformations, and dimension expansion operations. The condition information is defined as... The conditional weight generation module calculates the fusion weight. The specific formula is expressed as follows:

[0099] ,

[0100] in, This represents the conditional information extracted from the conditional visual state space block. and It is a learnable linear transformation matrix obtained through linear transformation operations. This represents the Sigmoid activation function; View represents the shape transformation used to rearrange the dimensions of the tensor; Expand represents the dimension expansion operation used to expand the generated weights to the same spatial dimension as the input features.

[0101] Then, the three feature maps with adjusted sizes Separately with fusion weights Element-wise multiplication is performed to obtain the feature maps after three conditionally modulated weights. :

[0102] ,

[0103] Next, the three conditionally modulated and weighted feature maps are... Adding elements one by one yields the aggregate representation. :

[0104] ,

[0105] Conditional modulation weighting can adaptively emphasize information-rich spatial regions, but it does not demonstrate the ability to model long-range dependencies. Therefore, this invention introduces a multi-head attention (MHA) mechanism to capture cross-scale feature interactions. First, the feature maps adjusted for three sizes are processed... Each feature map is first transformed (View) to rearrange the tensor dimensions, and then fed into a multi-head attention (MHA) mechanism for feature interaction. The output feature map of the MHA is then transformed (View) again to rearrange the tensor dimensions, resulting in the attention-enhanced feature representation. The specific calculations are as follows:

[0106] ,

[0107] in, This represents the multi-head attention mechanism (MHA), which internally calculates multi-head scaling dot product attention and generates the final output after splicing and projection.

[0108] Finally, the aggregate representation is integrated by adding elements one by one. and feature representation The fused feature map is obtained. :

[0109] ,

[0110] The Conditional Attention Fusion (CAF) module employs a fusion strategy that balances local adaptability with global context modeling, effectively enhancing the representational ability of features in downstream tasks.

[0111] Step S3: Train the autism children's attention prediction model based on the Mamba-Unet structure constructed in Step S2 using data from the training set. Adjust the model parameters using data from the validation set and determine if overfitting occurs. After training, test the network performance of the model using data from the test set. In this embodiment of the invention, a hybrid loss function is used to guide model training and parameter tuning. This hybrid loss function consists of five different components, effectively overcoming the limitations of traditional saliency prediction methods, thereby achieving higher prediction accuracy and better matching the unique visual attention patterns of individuals with autism spectrum disorder. The calculation formula of the hybrid loss function is as follows:

[0112] ,

[0113] in, Represents the mixed loss function. A true saliency plot, s represents the true fixation map, and s is the predicted saliency map generated by the model. The weighting coefficients representing the KL divergence. Denotes the KL divergence function. The weighting coefficient represents the correlation coefficient. Represents the correlation coefficient function. Weighting coefficients representing similarity Represents the similarity function. The weighting coefficients represent the significance values ​​of the normalized scan path. This represents the saliency function of the normalized scan path. The weighting coefficients represent the mean squared error. This represents the mean square error function.

[0114] The five components of the hybrid loss function are as follows:

[0115] (1) KL divergence (KLD): KL divergence is used to quantify the difference between the predicted and the actual distribution, penalizing significant deviations from the true data distribution. The formula for calculating the KL divergence function is:

[0116] ;

[0117] in, A true saliency plot, This represents the predicted saliency map generated by the model, where n represents the index of the nth pixel, and N represents the total number of pixels in the image. Let represent the natural logarithm function, and ε be the regularization constant, denoted as 2.2 × 10⁻⁶. , This represents the significance value of the model's prediction at the nth pixel position. This represents the true saliency value at the nth pixel position.

[0118] (2) Linear Correlation Coefficient (CC): The linear correlation coefficient (CC) measures the linear relationship between the prediction and the true significance plot. A higher correlation indicates a closer match to reality, thus improving the reliability of the prediction. The formula for the correlation coefficient function is:

[0119] ;

[0120] in, For covariance, The standard deviation is denoted as .

[0121] (3) Similarity (SIM): Similarity (SIM) is used to evaluate the overlap between the predicted and true saliency maps, emphasizing the accurate identification of prominent regions. The formula for calculating the similarity function is:

[0122] ;

[0123] in, This means taking the minimum of the two values.

[0124] (4) Normalized Scan Path Saliency (NSS): Normalized scan path saliency (NSS) is used to evaluate the correlation between the normalized predicted saliency map and the true fixation map, reflecting the model's ability to simulate human visual attention patterns. The formula for calculating the normalized scan path saliency function is:

[0125] ;

[0126] in, This represents the mean of the prediction significance plot s. A binary label representing the true gaze map at the nth pixel position. A diagram representing the true gaze point.

[0127] (5) Mean Squared Error (MSE): MSE calculates the squared error per pixel between the predicted saliency map and the true saliency map, directly penalizing the deviation of the significance value. The mean squared error function is defined as:

[0128] .

[0129] Step S4: Using the autism children's attention prediction model based on the Mamba-Unet structure trained in Step S3, perform atypical visual attention prediction for autism spectrum disorder on the image to be tested, obtain the corresponding binary saliency map, and generate a prediction heatmap based on the map and the original image to visualize the visual attention area of ​​individuals with autism spectrum disorder, and obtain the visualized prediction results of the attention distribution points (i.e., gaze points) of individuals with autism spectrum disorder on the image to be tested (or the scene represented by the image to be tested).

[0130] The technical effects of the embodiments of the present invention are illustrated below with specific experimental data:

[0131] This embodiment of the invention utilizes the PyTorch framework for training, and was trained for 50 epochs on an NVIDIA RTX 4090 GPU (24 GB). Early stopping was applied in the 8th training epoch when the model training loss no longer decreased. The MLP module in the Conditional Visual State Space Block (C-VSS Block) consists of three linear layers with output dimensions (widths) of 128, 64, and 5, respectively. The GELU activation function is applied between layers to enhance non-linear modeling capabilities. The model operates using labels T, each with a dimension D=128. All data and labels were resized to 256×256, and training was performed on the Saliency4ASD dataset. For the atypical visual attention prediction task in autism spectrum disorder, the network was optimized using the Adam optimizer with an initial learning rate of 1× The step size is 5, and the learning rate adjustment factor (gamma) is 0.28. In determining the optimal mixed loss function, for the atypical visual attention prediction task in autism spectrum disorder, the weight coefficients are set as follows: To improve the model's robustness and reduce overfitting, data augmentation techniques were applied during training. The training dataset in Saliency4ASD underwent preprocessing, including random cropping and adjustments to brightness, contrast, saturation, and hue, before being fed into the model for training. These enhancements improved the model's ability to generalize across different image compositions and color distributions.

[0132] Five commonly used significance prediction metrics were used to evaluate the model's performance in the experiment: Kullback-Leibler Divergence (KLD), linear correlation coefficient (CC), similarity (SIM), standardized scan path significance (NSS), and area under the Judd curve (AUC-J). A detailed description of the Judd curve area under the curve metric is as follows:

[0133] The area under the Judd curve (AUC-J): AUC-J measures the consistency between the predicted saliency map and the true human fixation point (ground truth fixation map), and is a commonly used benchmark metric in saliency prediction evaluation. The AUC-J metric reflects the model's ability to distinguish fixations by constructing a receiver operating characteristic (ROC) curve and calculating the area under it. The formula for calculating the area under the Judd curve is:

[0134] AUC-J ;

[0135] in, TPR(t) represents the true positive rate at the threshold t. FPR(t) represents the false positive rate at threshold t, and TP, FP, TN, and FN are the number of true positive, false positive, true negative, and false negative pixels, respectively. The integral is calculated numerically by varying the changes in TPR(t) and FPR(t) at different thresholds t. The value of this index ranges between 0 and 1; a larger value indicates that the predicted saliency map is closer to the real human eye gaze pattern, suggesting that the model has a stronger discriminative ability.

[0136] To verify the effectiveness of the method, the prediction method of this invention was tested on the Saliency4ASD dataset. It was compared with nine existing neural network-based attention prediction methods for children with autism published in recent years, based on the five evaluation metrics mentioned above (KLD, CC, SIM, NSS, and AUC-J). Two of these existing prediction methods, ASD-SA and ASD-MF, are specifically designed for atypical visual attention prediction in autism spectrum disorder. The other seven existing prediction methods, including ML-Net, SalGAN, SAM-RES, SAM-VGG, DIN, SAL-DCNN, and EML-Net, are general-purpose typical visual attention prediction methods used for comparison, and were optimized for atypical visual attention prediction tasks in autism spectrum disorder. The saliency prediction images for all comparison methods were provided by the authors of each method or generated by running their publicly available code. Among the five evaluation indicators KLD, CC, SIM, NSS and AUC-J, the lower the KLD value, the better, and the higher the SIM, CC, NSS and AUC-J values, the better, as shown in Table 1. The method of the present invention is generally superior to all the comparative methods.

[0137] Table 1. Quantitative evaluation of our method against 9 other techniques on the Saliency4ASD dataset (bold data represents the best-performing experimental data for each method, and the last row shows the experimental evaluation data for our method):

[0138]

[0139] Table 1 shows that the experimental results of the method described in this paper are significantly better than the previous methods overall, demonstrating that our method has superior performance in predicting attention in children with autism. For example, compared with nine other existing prediction methods, the method described in this paper achieves the best results on all five evaluation metrics on the Saliency4ASD dataset. The visualization results of the method described in this paper and the nine existing prediction methods are shown in the figure below. Figure 8 As shown, Figure 8 The ground truth (GT) in the dataset is the true saliency map (i.e., the true labeled map in the Saliency4ASD dataset). Figure 8 The superiority of the invention method in terms of visualization is demonstrated.

[0140] like Figure 9 As shown, this embodiment generates a corresponding binary saliency map (Our_FixMap) using the proposed method, and further generates a predicted heatmap (Our_HeatMap) based on this map and the original image. Figure 9 In the diagram, Image represents the original natural image, ASD_HeatMap represents the actual gaze heatmap of individuals with autism spectrum disorder, and ASD_FixMap represents the actual binary saliency label map. By comparing the predicted results with the actual label map, it can be observed that this method has high accuracy and consistency in saliency region localization and can effectively simulate the visual attention patterns of individuals with ASD.

[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for predicting attention in autistic children based on the Mamba-Unet structure, characterized in that, Includes the following steps: S1: Construct a multimodal dataset for attention prediction in children with autism, and divide the multimodal dataset into a training set, a test set, and a validation set; S2: Construct an attention prediction model for autistic children based on the Mamba-Unet structure. The attention prediction model for autistic children based on the Mamba-Unet structure includes a preprocessing module, an encoder part, a bottleneck part, a decoder part, and an output part connected in sequence. The encoder part and the decoder part each include three stages connected in sequence. The outputs of the three stages of the encoder part are also propagated to the three stages of the decoder part through skip connections. The preprocessing module is used for dimensionality reduction of the input image; The encoder section is used for feature extraction from the feature map; each stage of the encoder section includes a sequentially connected efficient adaptive visual state space block and patch merging downsampling operation; The efficient adaptive visual state space block is used to efficiently capture the global spatial dependencies of feature maps and provides key context modeling capabilities; The patch merging downsampling operation is used to halve the spatial resolution and double the number of channels to generate a hierarchical feature map. The bottleneck section is used to perform deeper underground sampling of the feature map to capture deep features; The bottleneck part includes an efficient adaptive visual state space block, a frequency domain attention module, and a conditional visual state space block. The frequency domain attention module is used to achieve enhanced feature modeling with minimal computational overhead, while preserving key feature information. The conditional visual state space block is used to fuse the global context and optimize spatial dependencies; The decoder section is used to perform upsampling processing to obtain fused features; Each stage of the decoder includes a conditional attention fusion module, a patch extended upsampling operation, and a conditional visual state space block; The conditional attention fusion module is used to generate dynamic fusion weights to achieve weighted fusion of multi-scale features; the patch expansion upsampling operation is used to expand the channels and reconstruct the spatial dimension of the feature map to achieve progressive upsampling of spatial resolution and enhance the fusion capability of local and global features. The output section is used to output the prediction results; S3: Use the data in the training set to train the autism children's attention prediction model based on the Mamba-Unet structure, use the data in the validation set to adjust the model parameters, determine whether overfitting occurs, and use a hybrid loss function to guide the optimization of model parameters during model training and parameter tuning. After training, the network performance of the model is tested using data from the test set. S4: Using a trained autism children's attention prediction model based on the Mamba-Unet structure, atypical visual attention prediction for autism spectrum disorder is performed on the image to be tested, and the corresponding binary saliency map is obtained. Based on the binary saliency map and the original input image, a predicted heat map is generated to visualize the visual attention area of ​​individuals with autism spectrum disorder.

2. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 1, characterized in that, The output of the first stage of the encoder is propagated to the third stage of the decoder through skip connections. The output of the second stage of the encoder is propagated to the second stage of the decoder through skip connections. The output of the third stage of the encoder is propagated to the first stage of the decoder through skip connections. Each stage contains two skip connection methods. The first skip connection method is to propagate the feature map after efficient adaptive visual state space block processing in the encoder to the conditional attention fusion module of the decoder. The second skip connection method is to first propagate the feature map after patch merging and downsampling operation to the frequency domain attention module for further optimization, and then propagate the optimized feature map to the conditional attention fusion module of the decoder.

3. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 2, characterized in that, The efficient adaptive visual state space block includes an efficient adaptive module and two visual state space modules; the efficient adaptive module is used to perform deep feature extraction, giving priority to high-entropy pixels that contain more significant information to enhance feature representation. The visual state space module is used for feature extraction, applying bidirectional scanning along the horizontal and vertical axes to enhance the spatial dependencies of the captured feature maps by integrating local and global dependencies; The efficient adaptive visual state space block in the encoder contains two parallel paths. One path involves two visual state space modules directly extracting features from the input feature map and propagating them to the subsequent layers of the encoder, i.e., the patch merging and downsampling operation. The other path involves the efficient adaptive module extracting deep features from the input feature map and then propagating them to the conditional attention fusion module of the decoder through a skip connection. In the bottleneck section, the efficient adaptive visual state space block first processes the input feature map by the efficient adaptive module. The output of the efficient adaptive module enters the two visual state space modules for processing, and then the output feature map is propagated to the subsequent layer of the bottleneck section, namely the frequency domain attention module.

4. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 3, characterized in that, The efficient adaptive module includes a dual-branch feature extraction structure, which comprises a parallel efficient grouped feature branch and an adaptive branch. The efficient grouped feature branch includes grouped convolution, and dimension mapping is achieved through batch normalization layers, SiLU activation function, and rearrangement operation; the adaptive branch includes average pooling operation, 1×1 convolution, dimension mapping using rearrangement operation, and normalization using Softmax function. Then, the efficient adaptive module multiplies the output feature maps of the efficient grouping feature branch and the adaptive branch element by element, performs one-dimensional summation on the last layer of the obtained feature map tensor, and finally performs bilinear interpolation upsampling to obtain the output feature map of the efficient adaptive module.

5. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 4, characterized in that, The visual state space module includes a main path and a branch path. The main path includes a layer normalization layer, a linear transformation operation, a depthwise separable convolution, a two-dimensional state space modeling module, a layer normalization layer, and a linear transformation operation connected in sequence. The branch path has one linear transformation operation. The output of the first layer normalization layer on the main path is processed by the linear transformation operation on the branch path and then multiplied element-wise with the output of the second layer normalization layer on the main path. The result of the element-wise multiplication is then input into the second linear transformation operation on the main path for processing. At the same time, the output features of the second linear transformation operation on the main path are added to the initial input features of the visual state space module through residual connections to obtain the final output of the visual state space module.

6. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 5, characterized in that, The frequency domain attention module first performs a block operation along the channel dimension to process the input feature map. Divided into three subsets along the channel dimension , and Each subset is sequentially processed through a 1×1 convolution and a 3×3 depthwise convolution for feature extraction and spatial information modeling, respectively, to obtain feature representations. , and ; Then, the rearrangement operator is used. Representing features respectively and feature representation Perform a rearrangement operation to obtain the rearranged feature representation. and feature representation And respectively, the features after rearrangement are represented. and feature representation Fast Fourier Transform (FFT) is applied and projected into the frequency domain to extract frequency domain features and enhance the model's ability to model global structural information. Then, the attention map F is obtained by performing element-wise multiplication on the FFT result in the frequency domain. AB And attention map F AB The inverse operation involves performing a two-dimensional inverse Fourier transform to obtain the attention map A. Then, the attention map A is normalized through a layer normalization layer before being compared with the feature representation. Element-wise multiplication yields the attention-weighted feature map. Finally, an attention-weighted feature map is processed by a 1×1 convolution. After processing, it is then connected to the input feature map X via residual connections. F Element-wise addition is performed to refine the feature map, resulting in an output feature map. .

7. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 6, characterized in that, The conditional visual state space block includes a central main path, a parallel feedback path, and a right branch. The central main path consists of a layer normalization layer, an affine transformation operation, a linear transformation operation, a depthwise separable convolution, a SiLU activation function, a 2D state space modeling module, a scaling function, a layer normalization layer, an affine transformation operation, and a linear transformation operation, all connected in sequence. The parallel feedback path propagates category feedback information to the 2D state space modeling module and the two affine transformation operations in the central main path through an MLP module. The right branch consists of a linear transformation operation and a SiLU activation function, where the output of the first affine transformation operation in the central main path is multiplied element-wise by the output of the second affine transformation operation in the central main path after passing through the right branch, and then propagated to the second linear transformation operation in the central main path. Simultaneously, the output of the second linear transformation operation in the central main path is added element-wise to the input feature map of the conditional visual state space block through a residual connection to obtain the final output of the conditional visual state space block.

8. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 7, characterized in that, In the parallel feedback path of the conditional vision state space block, the category feedback information is extracted by the MLP module based on the data category information contained in the predefined label T; label T contains learnable tokens, and the dimension of label T is D; the tokens are fed into the MLP module for regressing the modulation parameter scaling factor. Translation factor The MLP module is represented as ): → ,in ) indicates an MLP module. For input features, Indicates model parameters, This indicates that the input features are a real matrix of size 4 rows and D columns. The output is a 4x5 real matrix Y, where each row of Y corresponds to an input token, and five key parameters are output: Select the row corresponding to input label T from matrix Y. , This represents the z-th row vector in matrix Y, which is the modulation parameter row corresponding to the input label T. Let it be a vector S, i.e., S = [ , , , , The vector S is the set of modulation parameters corresponding to the input marker T; each parameter in vector S is used to adjust the computational behavior at different locations in the model, wherein... The modulation operation is located in the second affine transformation operation in the middle main path. Used to control the intensity of features in the two-dimensional state-space modeling module. The modulation operation is the first affine transformation operation located in the middle main path; the specific formula for adjusting the computational behavior at different positions in the model according to the parameters in vector S is expressed as follows: ， in, This represents a feature map modulated by dynamic parameters. This represents the feature map before modulation; It is a scaling factor, including , , ; It is the translation factor, including , ; This indicates element-wise multiplication.

9. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 8, characterized in that, The conditional attention fusion module includes bilinear interpolation, an adaptive feature weighting branch, and a multi-scale feature interaction branch. The adaptive feature weighting branch includes a conditional weight generation module, which consists of a 1×1 convolution, a linear transformation operation, a sigmoid activation function, a shape transformation, and a dimension expansion operation connected in sequence. The multi-scale feature interaction branch includes shape transformation, a multi-head attention mechanism, and shape transformation. The conditional attention fusion module first uses bilinear interpolation to process the three received feature maps. , , The spatial dimensions are adjusted to obtain three feature maps with adjusted dimensions; Meanwhile, in the adaptive feature weighting branch, the conditional weight generation module calculates the fusion weights based on the conditional information extracted from the conditional visual state space block. Then, the three resized feature maps are respectively fused with the weights. Element-wise multiplication is performed to obtain three conditionally modulated weighted feature maps; then, the three conditionally modulated weighted feature maps are summed element-wise to obtain the aggregated representation. ; The multi-scale feature interaction branch first performs shape transformations on the three scaled feature maps, rearranging the tensor dimensions of the feature map representations, and then inputs them into the multi-head attention mechanism for feature interaction. Next, it performs shape transformations on the output feature map of the multi-head attention mechanism, rearranging the tensor dimensions of the feature map representations again to obtain the attention-enhanced feature representations. ; Finally, the aggregation representation and feature representation By adding elements one by one, we obtain the fused feature map. .

10. The method for predicting attention in autistic children based on the Mamba-Unet structure according to claim 1, characterized in that, The formula for calculating the hybrid loss function is as follows: ， in, Represents the mixed loss function. A true saliency plot, s represents the true fixation map, and s is the predicted saliency map generated by the model. The weighting coefficients representing the KL divergence. Denotes the KL divergence function. The weighting coefficient represents the correlation coefficient. Represents the correlation coefficient function. Weighting coefficients representing similarity Represents the similarity function. The weighting coefficients represent the significance values ​​of the normalized scan path. This represents the saliency function of the normalized scan path. The weighting coefficients represent the mean squared error. This represents the mean square error function.

Images