Image processing method and image processing device

Abstract

The invention provides an image processing method and an image processing device, and relates to the field of computers. According to the method, feature extraction is carried out on multi-modal image data of functional magnetic resonance image data and structural magnetic resonance image data of a brain, feature fusion is carried out, classification is carried out based on fused features, an image recognition result is obtained, feature extraction is carried out based on a channel attention and space attention decoupling method, and the recognition accuracy is improved. According to the method, the image features can be extracted more accurately, the accuracy of the image recognition result is further improved, and diagnosis of mental diseases such as bidirectional affective disorder and Alzheimer's disease can be better assisted.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and particularly to the fields of artificial intelligence, machine learning and healthcare technology, especially to image processing methods and image processing apparatus. Background Technology

[0002] Bipolar disorder is a serious mental illness characterized by significant mood swings, and its diagnosis relies primarily on clinical professionals assessing the patient's condition through face-to-face interviews and observation. Due to the complex and diverse symptoms of bipolar disorder, clinical diagnosis faces significant challenges.

[0003] In recent years, machine learning technology has shown great potential in the auxiliary diagnosis of bipolar disorder. Machine learning techniques typically involve first extracting features from magnetic resonance imaging (MRI) images of the brain, and then using a machine learning model to identify bipolar disorder based on these features.

[0004] Due to the diversity and complexity of bipolar disorder symptoms, the accuracy of image-based bipolar disorder identification based on machine learning techniques faces challenges. Summary of the Invention

[0005] This disclosure discloses an embodiment that performs image classification based on multimodal magnetic resonance image data and extracts features based on a method that decouples channel attention and spatial attention. This method can extract image features more accurately, thereby improving the accuracy of image recognition results and better assisting in the diagnosis of mental illnesses such as bipolar disorder and Alzheimer's disease.

[0006] This disclosure provides an image processing method in some embodiments, including:

[0007] The first feature is obtained by extracting channel features and two-dimensional spatial features from the functional magnetic resonance imaging data of the user's brain;

[0008] The structural magnetic resonance imaging data of the user's brain are processed independently by channel attention and spatial attention. Then, based on the results of channel attention and spatial attention, channel features and three-dimensional spatial features are extracted to obtain the second feature.

[0009] The first feature and the second feature are fused to obtain a fused feature;

[0010] Based on the fusion features, classification is performed to obtain image recognition results.

[0011] In some embodiments, fusing the first feature and the second feature to obtain a fused feature includes:

[0012] Expand the dimensions of the first feature to match the dimensions of the second feature;

[0013] The second feature and the expanded first feature are downsampled to achieve uniform size;

[0014] Connect the downsampled second feature with the first feature;

[0015] Self-attention processing is applied to the concatenated second and first features to obtain the fused features.

[0016] In some embodiments, obtaining the first feature includes: using a cascaded plurality of first feature extraction units to perform multiple channel feature and two-dimensional spatial feature extractions on the user's functional magnetic resonance imaging data of the brain to obtain the first feature; or,

[0017] The process of obtaining the second feature includes: using multiple cascaded second feature extraction units to perform multiple channel feature and three-dimensional spatial feature extractions on the structural magnetic resonance imaging data of the user's brain to obtain the second feature;

[0018] The process of obtaining the fused features includes fusing the first feature output by the last first feature extraction unit and the second feature output by the last second feature extraction unit to obtain the fused features.

[0019] In some embodiments, each first feature extraction unit performs the following processing:

[0020] Extract global features from the input of the first feature extraction unit at this level, where the input of the first feature extraction unit is functional magnetic resonance imaging data;

[0021] Based on global features, features are extracted at the channel dimension to obtain channel features;

[0022] Based on global features and channel features, features are extracted in the spatial dimension to obtain spatial features;

[0023] Based on global features, channel features, and spatial features, the output of the first feature extraction unit at this level is determined. The output of the last first feature extraction unit is the first feature, and the outputs of other first feature extraction units are used as the inputs of the next level first feature extraction unit.

[0024] In some embodiments, global features of the input to the first feature extraction unit at this level are extracted using a 1×1 convolution kernel;

[0025] By using a 3×1 convolution kernel, features are extracted along the channel dimension based on global features to obtain channel features; or

[0026] By using a 1×3 convolution kernel, features are extracted in the spatial dimension based on global and channel features to obtain spatial features.

[0027] In some embodiments, each second feature extraction unit performs the following processing:

[0028] Feature extraction is performed on the input of the second feature extraction unit at this level to obtain the features of the input, wherein the input of the first second feature extraction unit is structural magnetic resonance image data;

[0029] Based on the input and its characteristics, calculate the channel weight matrix and the spatial weight matrix respectively;

[0030] The input features are processed using the channel weight matrix and the spatial weight matrix to obtain the output of the second feature extraction unit at this level. The output of the last second feature extraction unit is the second feature, and the outputs of the other second feature extraction units are used as the input of the next level second feature extraction unit.

[0031] In some embodiments, the input features are processed using a channel weight matrix and a spatial weight matrix to obtain the output of the second feature extraction unit at this level, including:

[0032] The input features are weighted using a channel weight matrix to obtain a first weighted processing result.

[0033] The first weighted result is weighted using a spatial weight matrix to obtain the second weighted result.

[0034] Based on the second weighted processing result and the features of the input, the output of the second feature extraction unit at this level is determined.

[0035] In some embodiments, calculating the channel weight matrix based on the input and its characteristics includes:

[0036] The input is weighted using the first weight matrix of channel attention;

[0037] The second weight matrix of the channel attention is used to weight the weighted processing result and the features of the input to obtain the channel weight matrix.

[0038] In some embodiments, calculating the spatial weight matrix based on the input and its features includes:

[0039] The input is weighted using the third weight matrix of spatial attention;

[0040] The spatial weight matrix is ​​obtained by weighting the weighted processing result and the input features using the fourth weight matrix of spatial attention.

[0041] In some embodiments, classification based on the fused features yields image recognition results including:

[0042] The fusion features are sliced ​​to obtain a first fusion feature in the functional magnetic resonance image data processing branch and a second fusion feature in the structural magnetic resonance image data processing branch.

[0043] Based on the first feature and the first fused feature, a first feature to be processed is determined;

[0044] Based on the second feature and the second fused feature, a second feature to be processed is determined;

[0045] Align the first feature to be processed and the second feature to be processed, and connect them;

[0046] A classifier is used to process the first and second features to be processed, which are connected together, to obtain the image recognition result.

[0047] In some embodiments, the image processing method further includes: generating a heat map of the user's brain based on gradient information of the second feature to be processed, for characterizing the brain region of interest in the image recognition result.

[0048] This disclosure provides an image processing apparatus, including an image processing model, comprising:

[0049] The first feature extraction module is configured to extract channel features and two-dimensional spatial features from the user's functional magnetic resonance imaging data of the brain to obtain the first feature;

[0050] The second feature extraction module is configured to independently perform channel attention processing and spatial attention processing on the structural magnetic resonance imaging data of the user's brain, and then extract channel features and three-dimensional spatial features based on the channel attention processing results and spatial attention processing results to obtain the second feature;

[0051] The feature fusion module is configured to fuse the first feature and the second feature to obtain a fused feature;

[0052] The classification module is configured to perform classification based on the fused features to obtain image recognition results.

[0053] In some embodiments, the first feature extraction module includes a plurality of cascaded first feature extraction units, each first feature extraction unit including:

[0054] The global feature extraction subunit is configured to extract global features from the input of the first feature extraction unit at this level, wherein the input of the first feature extraction unit is functional magnetic resonance image data;

[0055] The channel feature extraction subunit is configured to extract features in the channel dimension based on global features to obtain channel features;

[0056] The spatial feature extraction subunit is configured to extract features in the spatial dimension based on global features and channel features to obtain spatial features.

[0057] The output subunit is configured to determine the output of the first feature extraction unit at this level based on global features, channel features, and spatial features. The output of the last first feature extraction unit is the first feature, and the outputs of other first feature extraction units are used as the inputs of the next level first feature extraction unit.

[0058] In some embodiments, the second feature extraction module includes a plurality of cascaded second feature extraction units, each second feature extraction unit including: a channel attention subunit and a spatial attention subunit that are independent of each other, and an output subunit;

[0059] The channel attention subunit is configured to extract features from the input of the second feature extraction unit at this level to obtain the features of the input, wherein the input of the first second feature extraction unit is structural magnetic resonance image data; and the channel weight matrix is ​​calculated based on the input and its features.

[0060] The spatial attention subunit is configured to extract features from the input of the second feature extraction unit at this level to obtain the features of the input, wherein the input of the first second feature extraction unit is structural magnetic resonance image data; and a spatial weight matrix is ​​calculated based on the input and its features.

[0061] The output subunit is configured to process the input features using the channel weight matrix and the spatial weight matrix to obtain the output of the second feature extraction unit at this level. The output of the last second feature extraction unit is the second feature, and the outputs of the other second feature extraction units are used as the inputs of the next level second feature extraction unit.

[0062] In some embodiments, the image processing apparatus further includes a visualization module configured to generate a heat map of the user's brain based on gradient information of the second feature to be processed, for characterizing the brain region of interest in the image recognition result.

[0063] In some embodiments, the first feature extraction module includes a plurality of cascaded first feature extraction units; the second feature extraction module includes a plurality of cascaded second feature extraction units; and the feature fusion module is cascaded at the output of the last first feature extraction unit and the output of the last second feature extraction unit to fuse the first feature output by the last first feature extraction unit and the second feature output by the last second feature extraction unit to obtain a fused feature.

[0064] Some embodiments of this disclosure provide an image processing apparatus, including: a memory; and a processor coupled to the memory, the processor being configured to perform an image processing method based on instructions stored in the memory.

[0065] Some embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of an image processing method.

[0066] Some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of an image processing method. Attached Figure Description

[0067] The accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. This disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings.

[0068] Obviously, the accompanying drawings described below are merely some embodiments of this disclosure. Those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0069] Figure 1 A schematic diagram illustrating an image processing model according to some embodiments of this disclosure is shown.

[0070] Figure 2 A schematic diagram of a first feature extraction unit according to some embodiments of the present disclosure is shown.

[0071] Figure 3 A schematic diagram of a second feature extraction unit according to some embodiments of the present disclosure is shown.

[0072] Figure 4 A schematic diagram illustrating a training method for an image processing model according to some embodiments of the present disclosure is shown.

[0073] Figure 5 A schematic diagram illustrating image processing methods according to some embodiments of the present disclosure is shown.

[0074] Figure 6 A schematic diagram of an image processing apparatus according to some embodiments of the present disclosure is shown.

[0075] Figure 7 The diagram illustrates the hyperparameter experimental results of parameter sensitivity analysis for some embodiments of this disclosure. Detailed Implementation

[0076] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0077] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0078] It should also be understood that in the embodiments disclosed herein, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0079] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0080] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0081] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0082] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0083] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0084] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0085] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0086] Furthermore, to avoid obscuring this disclosure with unnecessary detail, only processing steps and / or apparatus structures closely related to at least the solutions according to this disclosure are shown in the accompanying drawings, while other details less relevant to this disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it need not be discussed again in subsequent drawings. For brevity, the same technical implementation, once described in one place, will not be repeated elsewhere.

[0087] Figure 1 Schematic diagrams illustrating image processing models of some embodiments of this disclosure are shown. These image processing models are deployed, for example, in image processing devices such as computers. Figure 1 As shown, the image processing model of this embodiment includes, for example, the following modules.

[0088] The first feature extraction module (also known as the channel feature and two-dimensional spatial feature extraction module) 110 is configured to extract channel features and two-dimensional spatial features from the user's functional magnetic resonance imaging (fMRI) data of the brain to obtain the first feature. For fMRI data, especially resting-state functional magnetic resonance imaging data, its spatial dimension is generally two-dimensional, i.e., height × width (denoted as H × W). The channel feature and two-dimensional spatial feature extraction module is used to decouple channel information and spatial information.

[0089] The second feature extraction module (also known as the channel feature and three-dimensional spatial feature extraction module) 120 is configured to independently perform channel attention processing and spatial attention processing on the structural magnetic resonance imaging (sMRI) data of the user's brain, and then extract channel features and three-dimensional spatial features based on the channel attention processing results and spatial attention processing results to obtain the second feature.

[0090] The feature fusion module 130 is configured to fuse the first feature and the second feature to obtain a fused feature.

[0091] The classification module 140 is configured to perform classification based on the fusion features to obtain image recognition results, which can characterize the probability that the user has a preset type of mental illness.

[0092] By extracting and fusing features from multimodal image data of the brain using fMRI and sMRI, image classification is performed based on the fused features. Furthermore, feature extraction is conducted using a method that decouples channel attention from spatial attention, resulting in more accurate image feature extraction and improved accuracy of image recognition and disease probability prediction. This can better assist in the diagnosis of mental illnesses such as bipolar disorder and Alzheimer's disease. The fMRI data includes, for example, Rs-fMRI (resting-state fMRI) data, and the sMRI data includes, for example, T1w-MRI (T1-weighted MRI) data.

[0093] The complexity of clinical diagnosis of mental illness requires models to capture subtle differences in symptoms, which is a challenge for existing technologies. This disclosure employs a pyramid-structured feature extraction module (first feature extraction module 110, second feature extraction module 120) to extract more refined features, thereby capturing subtle differences in symptoms of mental illness.

[0094] The first feature extraction module (also known as the channel feature and two-dimensional spatial feature extraction module) 110 includes multiple cascaded first feature extraction units 110-i, i = 1, 2, ..., N, where N is the number of first feature extraction units. Generally, the more first feature extraction units (also known as channel feature and two-dimensional spatial feature extraction units), the more refined the extracted features, but the computational complexity also increases accordingly. If the computer performance is high, more first feature extraction units can be set; if computer performance is a priority, the number of first feature extraction units can be appropriately reduced. Based on experiments, the basic number of first feature extraction units that meet the feature refinement requirements can be determined. The figure exemplarily shows 5 first feature extraction units.

[0095] like Figure 2 As shown, each first feature extraction unit includes the following parts.

[0096] Global feature extraction subunit 111 is configured to extract global features from the input of the first feature extraction unit at this level, where the input of the first feature extraction unit is fMRI data. Global features from the input of the first feature extraction unit at this level are extracted through convolution using a 1×1 kernel. The formula is expressed as: in, This represents the input and output of the global feature extraction subunit 111, and Cov1 represents convolution processing based on a 1×1 convolution kernel.

[0097] Channel feature extraction subunit 112, connected to global feature extraction subunit 111, is configured to extract features in the channel dimension based on global features to obtain channel features. Channel features are obtained by convolution using a 3×1 convolution kernel, extracting features in the channel dimension based on global features. The formula is expressed as: in, represents the input and output of the channel feature extraction subunit 112, and Cov2 represents convolution processing based on a 3×1 convolution kernel.

[0098] Spatial feature extraction subunit 113, connected to global feature extraction subunit 111 and channel feature extraction subunit 112 respectively, is configured to extract features in the spatial dimension based on global and channel features to obtain spatial features. Convolution is performed using a 1×3 convolution kernel to extract features in the spatial dimension based on global and channel features, thus obtaining spatial features. The formula is expressed as: in, This represents the input and output of the channel feature extraction subunit 113, where Cov3 indicates convolution processing based on a 1×3 kernel. Subsequent pooling operations (such as average pooling or max pooling) can further reduce the spatial dimensionality of the features while preserving important channel information.

[0099] Output subunit 114, connected to global feature extraction subunit 111, channel feature extraction subunit 112, and spatial feature extraction subunit 113 respectively, is configured to determine the output of the current level's first feature extraction unit based on global features, channel features, and spatial features. For example, it adds the global features, channel features, and spatial features as the output of the current level's first feature extraction unit. The output of the last first feature extraction unit is the first feature, while the outputs of subsequent first feature extraction units serve as the input to the next level's first feature extraction unit.

[0100] The residual structure and related processing in the first feature extraction unit can improve the generalization ability of the model, so that the model's detection performance is not only good on specific patient datasets, but also on new or different patient groups, thus maintaining performance across different populations and clinical environments.

[0101] For sMRI data, the spatial dimension is generally three-dimensional, i.e., height × width × depth (denoted as H×W×D). The channel feature and 3D spatial feature extraction module is used to analyze the four-dimensional (4D) voxel information in sMRI data, where the first dimension represents the channel dimension and the other three dimensions represent spatial information. This module uses convolution (which can be combined with batch normalization layers and ReLU (Rectified Linear Unit) activation functions, such as Leaky ReLU) to extract features, and simultaneously calculates channel attention and spatial attention through a collaborative attention mechanism, thereby extracting features more precisely.

[0102] The second feature extraction module (also known as the channel feature and 3D spatial feature extraction module) 120 includes multiple cascaded second feature extraction units 120-i, i = 1, 2, ..., N, where N is the number of second feature extraction units. Generally, the more second feature extraction units (also known as channel feature and 3D spatial feature extraction units), the more refined the extracted features, but the computational complexity also increases accordingly. If the computer performance is high, more second feature extraction units can be set; if computer performance is a priority, the number of second feature extraction units can be appropriately reduced. Based on experiments, the basic number of second feature extraction units that meet the feature refinement requirements can be determined. The figure exemplarily shows 5 second feature extraction units. The number of second feature extraction units can be the same as the number of first feature extraction units to ensure that the feature refinement of the two network branches is basically equivalent, facilitating subsequent feature fusion.

[0103] Previous channel-spatial attention modules first calculated channel attention and then connected it to calculate spatial attention. However, the spatial weights obtained by this calculation method are affected by the channel weights to a certain extent. Therefore, this invention decouples channel attention and spatial attention and calculates channel weight information and spatial weight information at the same time, which can extract image features more accurately.

[0104] like Figure 3 As shown, each second feature extraction unit includes the following parts.

[0105] The channel attention subunit 121 is configured to extract features from the input of the second feature extraction unit at this level, obtaining the features of the input, wherein the input of the first second feature extraction unit is sMRI data; based on the input and its features, a channel weight matrix is ​​calculated. The input is weighted using the first weight matrix of the channel attention; the weighted result and the features of the input are weighted using the second weight matrix of the channel attention to obtain the channel weight matrix.

[0106] Spatial attention subunit 122 is configured to extract features from the input of the second feature extraction unit at this level, obtaining the features of the input, wherein the input of the first second feature extraction unit is sMRI data; based on the input and its features, a spatial weight matrix is ​​calculated. The input is weighted using the third weight matrix of spatial attention; the weighted result and the features of the input are weighted using the fourth weight matrix of spatial attention to obtain the spatial weight matrix.

[0107] The input F of the second feature extraction unit at this level s First, features are further extracted through a feature extraction layer, such as a convolutional layer, to obtain feature F′. s ; using F s and F′ s Simultaneously calculate the channel weight matrix h(F) s ,F′ s ) and spatial weight matrix g(F s ,F′ s ).

[0108]

[0109]

[0110] in Let represent the i-th weight matrix.

[0111] Output subunit 123, connected to the channel attention subunit and the spatial attention subunit respectively, is configured to process the input features using the channel weight matrix and the spatial weight matrix to obtain the output of the current level's second feature extraction unit. The input features are weighted using the channel weight matrix to obtain a first weighted processing result; the first weighted processing result is then weighted using the spatial weight matrix to obtain a second weighted processing result; based on the second weighted processing result and the input features, the output of the current level's second feature extraction unit is determined. The output of the last second feature extraction unit is used as the second feature, and the outputs of subsequent second feature extraction units are used as the input to the next level's second feature extraction unit.

[0112] That is, after obtaining the channel weights and spatial weights, the channel weights are first compared with F′. s Multiply, such that the product of F′ s The channel dimension contains weight information, and then the spatial weights are combined with the calculated F′. s Multiply to get Make The spatial dimension contains weight information, and finally, the residual mechanism is used to... With F′ sThe output features of the second feature extraction unit are obtained by summing the values. Since the second feature extraction unit calculates both channel weights and spatial weights simultaneously, it is referred to as the collaborative attention unit in this embodiment. The residual structure and related processing in the second feature extraction unit can improve the generalization ability of the model, enabling the model to perform well not only on specific patient datasets but also on new or different patient groups, thus maintaining performance across different populations and clinical environments.

[0113] The location of the feature fusion module 130 was determined through hyperparameter experiments using parameter sensitivity analysis. Figure 7 A schematic diagram of the hyperparameter experimental results for parameter sensitivity analysis is shown. Through experiments, the accuracy of the image processing model at different locations of the feature fusion module is obtained; this accuracy metric is, for example, Balanced Accuracy (BACC). Figure 7 As shown, when using a single feature fusion module, the later the feature fusion module is placed within the feature extraction module, the higher the performance (accuracy) of the image processing model. Image processing models using multiple feature fusion modules perform worse than those using a single feature fusion module. For example, setting two feature fusion modules at positions 1 and 2 results in worse performance (accuracy) compared to setting a single feature fusion module at position 3. This is because excessive or premature feature fusion can lead to confusion of individual modal information, resulting in insufficient feature extraction for each modality. Figure 7 It can be seen that the image processing model with a single feature fusion module at position 3 achieves optimal performance while maintaining appropriate storage space requirements. Therefore, through performance testing experiments, this embodiment sets a single feature fusion module at position 3. The feature fusion module is cascaded at the output of the last first feature extraction unit and the output of the last second feature extraction unit, fusing the first feature output by the last first feature extraction unit and the second feature output by the last second feature extraction unit to obtain the fused feature.

[0114] The feature fusion module 130 is configured to fuse the first feature and the second feature to obtain a fused feature. Specifically, for example, it expands the dimensions of the first feature to match the dimensions of the second feature using element copying; it downsamples the second feature and the expanded first feature to unify their sizes; it concatenates the downsampled second feature and the first feature; and it performs self-attention processing on the concatenated second feature and first feature to obtain the fused feature. For example, the concatenated second feature and first feature are input into a Transformer model, and the model outputs the fused feature.

[0115] The Transformer model consists of a self-attention layer and a feedforward network layer, with regularization layers between them to prevent overfitting. The self-attention layer first performs a linear transformation on the input vector to obtain three vectors: Q (query vector), K (key vector), and V (value vector). These are learnable weight matrices. The inner product of the matrix multiplications of Q and K is calculated to obtain the similarity score of each sequence element with other elements. To maintain numerical stability, the results are scaled by a scaling factor (d). k ) 1 / 2 , where d k The dimension of the key vector K is represented by the SoftMax function. Finally, the similarity score is normalized to obtain the attention weights. These attention weights are then weighted and summed with the vector V to obtain the final self-attention output (i.e., the self-attention matrix, denoted as Output). The self-attention layer is calculated as follows: Output = SoftMax(QK) T / (d k ) 1 / 2 V, where T denotes transpose. This self-attention matrix represents a weighted representation of the input features, capable of capturing the complex relationships between different features and fusing them into a unified representation, thereby enhancing the model's comprehensive understanding of data from different modalities. This fused feature is then fed into a classifier for processing, and the log-odds output by the classifier represents the probability that the user has a preset type of mental illness. The role of the feedforward network layer is to introduce nonlinear transformations to enhance the model's ability to model input features, which is achieved through multiple fully connected layers (linear layers) and nonlinear activation functions.

[0116] The classification module 140 is configured to perform classification based on the fusion features to obtain image recognition results, thereby determining the probability that the user suffers from a preset type of mental illness. Specifically, referring to the dimensions of the first feature and the second feature, the fusion features are sliced ​​to obtain a first fusion feature in the fMRI data processing branch and a second fusion feature in the sMRI data processing branch. The first fusion feature may have the same size as the first feature, and the second fusion feature may have the same size as the second feature. Based on the first feature and the first fusion feature, a first feature to be processed is determined, for example, by adding the first feature and the first fusion feature, and the result is used as the first feature to be processed. Based on the second feature and the second fusion feature, a second feature to be processed is determined, for example, by adding the second feature and the second fusion feature, and the result is used as the second feature to be processed. The first feature to be processed and the second feature to be processed are aligned and connected, for example, by reducing the dimension of the features in the two branches through an average pooling operation to align the two data features. A classifier (such as a logistic regression classifier) ​​is used to process the connected first feature to be processed and the second feature to be processed, and the image recognition result is output to determine the probability that the user suffers from a preset type of mental illness.

[0117] The image processing model also includes: a visualization module 150, the location of which is as follows: Figure 1 As shown, the system is configured to generate a heatmap of the user's brain based on gradient information of the second feature to be processed. This heatmap characterizes the brain regions of interest in image recognition results (such as pre-defined types of mental illness analysis). This enables the visualization of biomarkers, improving the interpretability and transparency of the model. In the medical field, model interpretability is crucial for gaining the trust of doctors and patients.

[0118] The visualization module 150, for example, employs Grad-CAM (Gradient-weighted Class Activation Mapping) visualization technology, which allows observers to see specific portions of the input images upon which the deep learning model relies for diagnostic decisions. Thus, Grad-CAM generates heatmaps of sMRI data, displaying the brain regions deemed important by the model in diagnosing predefined types of mental illnesses. These regions can be observed from sagittal, axial, and coronal planes. For instance, blurred areas in the heatmap indicate higher network attention, meaning the model considers areas more relevant to the diagnosis of the predefined type of mental illness, while clear areas indicate lower network attention, meaning the model considers areas less relevant. Based on the heatmap, observers can identify brain structures that the model particularly focuses on when analyzing MRI data, such as the cerebral cortex, parietal lobe, temporal lobe, and cerebellum. These regions are associated with abnormalities in mood swings, cognitive impairment, and sensory processing in patients with the predefined type of mental illness.

[0119] The image processing model achieves more refined feature extraction through a pyramid-structured feature extraction module and employs a multimodal fusion strategy using fMRI and sMRI data to overcome the complexity of clinical diagnosis, improve the accuracy of mental illness prediction, and accurately distinguish pre-existing mental illnesses from other (mental) disorders. The model's design fully utilizes residual techniques to enhance its generalization ability, enabling it to adapt to different patient groups and clinical environments. Supervised learning strategies and attention mechanisms improve performance under limited data conditions, reducing the amount of labeled data required. The model utilizes self-attention mechanisms to enhance feature integration between different modalities, improving storage efficiency and avoiding information confusion. The model is designed for high computational efficiency, achieving real-time or near-real-time rapid diagnostic capabilities to meet the growing demands of medical data processing, reducing computational resource requirements and enabling deployment in resource-constrained environments. Parameter sensitivity analysis determines the appropriate number and location of feature fusion modules, optimizing the feature fusion strategy to ensure the efficiency and effectiveness of the feature extraction process. A visualization module enhances the model's interpretability and transparency, allowing medical personnel to understand the model's decision-making process. Image processing models provide a more accurate, efficient, interpretable, and practical tool for the auxiliary diagnosis of mental illnesses, thereby improving the diagnostic process.

[0120] Based on the usage scenario, the image processing model is trained using the corresponding training data in that scenario, so that the image processing model can adapt to the specific usage scenario and make accurate predictions.

[0121] Figure 4 Schematic diagrams illustrating training methods for image processing models according to some embodiments of this disclosure are shown. Figure 4 As shown, the training method of the image processing model in this embodiment includes the following steps.

[0122] In step 410, training samples related to a preset type of mental illness are input into the image processing model for processing to obtain a probability prediction value of the image recognition result, that is, a probability prediction value of the user having a preset type of mental illness. The training samples include: functional magnetic resonance imaging data of the user's brain, structural magnetic resonance imaging data of the user's brain, and probability label values ​​of the image recognition result (such as the probability label value of the user having a preset type of mental illness).

[0123] In step 420, the parameters of the image processing model are updated based on the difference between the probability prediction value and the probability label value of the image recognition result (such as the probability prediction value of the user having a preset type of mental illness and the probability label value of the user having a preset type of mental illness), so that the difference between the probability prediction value and the probability label value becomes smaller and smaller.

[0124] For example, training samples related to bipolar disorder are input into the image processing model for processing to obtain a probability prediction value of the image recognition result of the user having bipolar disorder. Based on the difference information between the probability prediction value of the image recognition result of the user having bipolar disorder and the probability label value, the parameters of the image processing model are continuously updated so that the trained image processing model can accurately output image recognition results to predict the probability of the user having bipolar disorder, thereby assisting in the diagnosis of bipolar disorder.

[0125] For example, Alzheimer's-related training samples are input into the image processing model for processing to obtain a probability prediction value of the image recognition result of the user having Alzheimer's disease. Based on the difference between the probability prediction value of the image recognition result of the user having Alzheimer's disease and the probability label value, the parameters of the image processing model are continuously updated so that the trained image processing model can accurately output image recognition results to predict the probability of the user having Alzheimer's disease, thereby assisting in the diagnosis of Alzheimer's disease.

[0126] The raw training samples obtained from the data source can be preprocessed and then used to train the image processing model, thereby improving the quality of magnetic resonance image data and ensuring the accuracy and reliability of subsequent analysis.

[0127] Image preprocessing processes, such as center space cropping, histogram normalization, and intensity normalization, can be applied together to magnetic resonance image data to reduce variability and enhance image features.

[0128] Center-space cropping involves cropping out the central region of an image to reduce the impact of inconsistent head positioning. This method is commonly used for functional magnetic resonance imaging (fMRI) data because it reduces edge artifacts and improves data consistency across different subjects.

[0129] Histogram normalization can be used to adjust the intensity distribution of an image by transforming its histogram into a uniform or Gaussian distribution. This process helps enhance image contrast and keeps the statistical properties of the image consistent across different scans or datasets. Histogram normalization can be achieved using the Cumulative Distribution Function (CDF), whose formula can be expressed as:

[0130] I norm =CDF(I)

[0131] Where I is the intensity value of the original image, I norm It is the intensity value of the normalized image.

[0132] Intensity normalization scales the intensity of an image to a specific range, such as 0 to 1, or uses other normalization methods, such as Z-score normalization, to eliminate intensity differences between different scans. This method is helpful for subsequent image analysis because it ensures comparability between different images.

[0133] The trained image processing model can perform image processing based on the user's functional magnetic resonance imaging (fMRI) and structural magnetic resonance imaging (SMRI) data of the brain to obtain image recognition results and predict the probability that the user has a preset type of mental illness, thereby assisting in the diagnosis of the preset type of mental illness.

[0134] Figure 5 The diagram illustrates some embodiments of the image processing method disclosed herein. All steps of this image processing method are performed by an image processing device such as a computer.

[0135] like Figure 5 As shown, the image processing method of this embodiment includes the following steps 510-540, and step 550 is performed as needed.

[0136] In step 510, channel features and two-dimensional spatial features are extracted from the functional magnetic resonance imaging data of the user's brain to obtain the first feature.

[0137] Step 510 is implemented using the first feature extraction module of the image processing model. Multiple cascaded first feature extraction units are used to extract channel features and two-dimensional spatial features from the user's functional magnetic resonance imaging data of the brain multiple times to obtain the first features.

[0138] Each first feature extraction unit undergoes the following processing: Global features of the input to this first feature extraction unit are extracted using a 1×1 convolution kernel, where the input to the first first feature extraction unit is functional magnetic resonance imaging data; Channel features are extracted based on the global features using a 3×1 convolution kernel; Spatial features are extracted based on the global and channel features using a 1×3 convolution kernel; The output of this first feature extraction unit is determined based on the global, channel, and spatial features, where the output of the last first feature extraction unit is the first feature, and the outputs of subsequent first feature extraction units serve as the input to the next level's first feature extraction unit.

[0139] In step 520, the structural magnetic resonance imaging data of the user's brain are processed independently with channel attention and spatial attention. Then, based on the results of the channel attention and spatial attention processing, channel features and three-dimensional spatial features are extracted to obtain the second feature.

[0140] Step 520 is implemented using the second feature extraction module of the image processing model. Multiple cascaded second feature extraction units are used to extract channel features and three-dimensional spatial features from the structural magnetic resonance imaging data of the user's brain multiple times to obtain the second features.

[0141] Each second feature extraction unit performs the following processing: It extracts features from the input of its current second feature extraction unit to obtain the features of the input, wherein the input of the first second feature extraction unit is structural magnetic resonance image data; based on the input and its features, it calculates the channel weight matrix and the spatial weight matrix respectively; using the channel weight matrix and the spatial weight matrix, it processes the features of the input to obtain the output of its current second feature extraction unit, wherein the output of the last second feature extraction unit is the second feature, and the outputs of other than the last second feature extraction units are used as the input of the next level second feature extraction unit.

[0142] The process of processing the input features using the channel weight matrix and the spatial weight matrix to obtain the output of the second feature extraction unit at this level includes: weighting the input features using the channel weight matrix to obtain a first weighted processing result; weighting the first weighted processing result using the spatial weight matrix to obtain a second weighted processing result; and determining the output of the second feature extraction unit at this level based on the second weighted processing result and the input features.

[0143] The calculation of the channel weight matrix based on the input and its features includes: weighting the input using a first weight matrix of channel attention; and weighting the weighted result and the features of the input using a second weight matrix of channel attention to obtain the channel weight matrix.

[0144] The calculation of the spatial weight matrix based on the input and its features includes: weighting the input using the third weight matrix of spatial attention; and weighting the weighted result and the features of the input using the fourth weight matrix of spatial attention to obtain the spatial weight matrix.

[0145] In step 530, the first feature and the second feature are fused to obtain a fused feature.

[0146] Step 530 is implemented using the feature fusion module of the image processing model. In some embodiments, the first feature is dimensionally expanded to match the dimension of the second feature; the second feature and the expanded first feature are downsampled to unify their size; the downsampled second feature and the first feature are concatenated; and self-attention processing is performed on the concatenated second feature and the first feature to obtain the fused feature.

[0147] In step 540, classification is performed based on the fused features to obtain image recognition results, thereby determining the probability that the user suffers from a preset type of mental illness.

[0148] Step 540 is implemented using the classification module of the image processing model. In some embodiments, the fused features are sliced ​​to obtain a first fused feature in the functional magnetic resonance image data processing branch and a second fused feature in the structural magnetic resonance image data processing branch; a first feature to be processed is determined based on the first feature and the first fused feature; a second feature to be processed is determined based on the second feature and the second fused feature; the first feature to be processed and the second feature to be processed are aligned and connected; a classifier is used to process the connected first feature to be processed and the second feature to be processed to obtain an image recognition result, thereby obtaining the probability that the user suffers from a preset type of mental illness.

[0149] In step 550, a heat map of the user's brain is generated based on the gradient information of the second feature to be processed, which is used to characterize the brain region of interest in the image recognition results (such as the analysis of a preset type of mental illness).

[0150] Step 550 is implemented using the visualization module of the image processing model, for example, by using Grad-CAM to generate a heat map of the user's brain, and by displaying the heat map model in the brain regions considered important when diagnosing a preset type of mental illness.

[0151] The performance metrics and performance test results of the image processing model and its image processing methods are described below.

[0152] (1) BACC (Balanced Accuracy) is an evaluation metric used in machine learning classification tasks, especially in cases of class imbalance. BACC considers the true positive rate and true negative rate of a classification model, i.e., sensitivity and specificity. In medical diagnostics and other fields that require distinguishing between different classes, BACC is an important metric because it provides a balanced measure that is not biased towards either class.

[0153] (2) F1: The F1 score is an important evaluation metric in machine learning classification tasks, which comprehensively considers the model's precision and recall. The F1 score is particularly useful when dealing with imbalanced datasets or when classification accuracy has different sensitivities to false positives and false negatives.

[0154] (3) SEN: (Sensitivity), also known as True Positive Rate (TPR) or recall, is an important metric for evaluating the performance of classification models, especially crucial in medical diagnosis and machine learning. It measures the proportion of instances correctly identified as positive (i.e., having a disease or possessing a specific attribute) out of all actual positive instances.

[0155] (4) SPEC (Specificity): Also known as the true negative rate in machine learning and medical diagnosis, it is an important indicator for measuring the performance of classification models. Specificity reflects the proportion of instances that the model correctly identifies as negative (i.e., instances that do not have the disease or do not have the specific attribute) out of all actual negative instances.

[0156] (5) The image processing model and method disclosed herein demonstrate superior performance on two test datasets, including the OpenfMRI dataset and a self-collected dataset. Compared with the previous SoTA (best) model STF2Former, the scheme of this embodiment achieves a 10.8% improvement in BACC, a 12.8% improvement in F1, a 14.4% improvement in SEN, and a 7.1% improvement in SPEC on the self-collected dataset, and a 5.7% improvement in BACC, an 11.9% improvement in F1, a 1.1% improvement in SEN, and a 10.2% improvement in SPEC on the OpenfMRI dataset.

[0157] Figure 6 Schematic diagrams of image processing apparatuses according to some embodiments of the present disclosure are shown. Figure 6As shown, the image processing apparatus 600 of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute image processing methods in any of the embodiments based on instructions stored in the memory 610.

[0158] The image processing apparatus 600 may also include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650, as well as the memory 610 and the processor 620, may be connected, for example, via a bus 660.

[0159] The memory 610 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0160] The processor 620 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, or transistors, or other discrete hardware components.

[0161] The input / output interface 630 provides a connection interface for input / output devices such as monitors, mice, keyboards, and touchscreens. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 660 can use any bus architecture from a variety of bus structures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0162] Some embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of an image processing method.

[0163] This disclosure provides some embodiments of a computer program product, including a computer program, characterized in that, when executed by a processor, the computer program implements the steps of an image processing method.

[0164] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security. Data protection standards are strictly adhered to during model development and deployment.

[0165] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more (non-transitory) computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, cloud storage, etc.) containing computer program code. A computer program product should be understood as a software product that primarily implements its solution through a computer program.

[0166] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0167] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0169] The above description is only a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. An image processing method comprising: extracting channel features and two-dimensional spatial features from functional magnetic resonance image data of a brain of a user to obtain first features; independently performing channel attention processing and spatial attention processing on structural magnetic resonance image data of the brain of the user, and then extracting channel features and three-dimensional spatial features based on the channel attention processing result and the spatial attention processing result to obtain second features; fusing the first features and the second features to obtain fused features; and performing classification based on the fused features to obtain an image recognition result.

2. The image processing method of claim 1, wherein: the obtaining first features comprises: using a plurality of first feature extraction units connected in cascade to extract channel features and two-dimensional spatial features from functional magnetic resonance image data of a brain of a user multiple times to obtain first features; or the obtaining second features comprises: using a plurality of second feature extraction units connected in cascade to extract channel features and three-dimensional spatial features from structural magnetic resonance image data of the brain of the user multiple times to obtain second features; the obtaining fused features comprises: fusing the first features output by a last first feature extraction unit and the second features output by a last second feature extraction unit to obtain fused features.

3. The image processing method according to claim 1 or 2, wherein the obtaining fused features comprises: dimensionally expanding the first features to match the dimensions of the second features; down-sampling the second features and the expanded first features to unify the sizes; concatenating the down-sampled second features and the first features; and performing self-attention processing on the concatenated second features and first features to obtain the fused features.

4. The image processing method of claim 2, wherein, each first feature extraction unit performs the following processing: extracting global features of an input of the first feature extraction unit, wherein the input of a first first feature extraction unit is the functional magnetic resonance image data; extracting features in a channel dimension based on the global features to obtain channel features; extracting features in a spatial dimension based on the global features and the channel features to obtain spatial features; and determining an output of the first feature extraction unit according to the global features, the channel features, and the spatial features, wherein the output of a last first feature extraction unit is the first features, and the output of a non-last first feature extraction unit is used as an input of a next first feature extraction unit.

5. The image processing method of claim 4, wherein: the global features of the input of the first feature extraction unit are extracted by using a 1x1 convolution kernel; the channel features are extracted in the channel dimension based on the global features by using a 3x1 convolution kernel; or the spatial features are extracted in the spatial dimension based on the global features and the channel features by using a 1x3 convolution kernel. each second feature extraction unit performs the following processing:

6. The image processing method of claim 2, wherein, extracting features of an input of the second feature extraction unit to obtain the features of the input, wherein the input of a first second feature extraction unit is the structural magnetic resonance image data; calculating a channel weight matrix and a spatial weight matrix based on the input and the features of the input, respectively; and ​ The input feature is processed by using the channel weight matrix and the spatial weight matrix to obtain the output of the second feature extraction unit of the current stage, wherein the output of the last second feature extraction unit is the second feature, and the output of the non-last second feature extraction unit is used as the input of the second feature extraction unit of the next stage.

7. The image processing method of claim 6, wherein, The input feature is processed by using the channel weight matrix and the spatial weight matrix to obtain the output of the second feature extraction unit of the current stage, including: The input feature is weighted processed by using the channel weight matrix to obtain a first weighted processing result; The first weighted processing result is weighted processed by using the spatial weight matrix to obtain a second weighted processing result; The output of the second feature extraction unit of the current stage is determined according to the second weighted processing result and the input feature.

8. The image processing method of claim 6, wherein, According to the input and the feature thereof, The channel weight matrix is calculated, including: The input is weighted processed by using the first weight matrix of the channel attention; The input feature is weighted processed by using the second weight matrix of the channel attention to obtain the channel weight matrix.

9. The image processing method of claim 6, wherein, According to the input and the feature thereof, The spatial weight matrix is calculated, including: The input is weighted processed by using the third weight matrix of the spatial attention; The input feature is weighted processed by using the fourth weight matrix of the spatial attention to obtain the spatial weight matrix.

10. The image processing method of claim 1, wherein, The image recognition result is obtained by classifying based on the fusion feature, including: The fusion feature is sliced to obtain the first fusion feature of the fusion feature in the functional magnetic resonance image data processing branch and the second fusion feature in the structural magnetic resonance image data processing branch; The first to-be-processed feature is determined according to the first feature and the first fusion feature; The second to-be-processed feature is determined according to the second feature and the second fusion feature; The first to-be-processed feature and the second to-be-processed feature are aligned and connected; The first to-be-processed feature and the second to-be-processed feature connected are processed by using the classifier to obtain the image recognition result.

11. The image processing method of claim 10, further comprising: generating a heat map of the brain of the user according to gradient information of the second to-be-processed feature, to represent a region of the brain of interest of the image recognition result.

12. An image processing apparatus, comprising an image processing model, including: a first feature extraction module configured to extract channel features and two-dimensional spatial features from functional magnetic resonance image data of a brain of a user to obtain first features; a second feature extraction module configured to independently perform channel attention processing and spatial attention processing on structural magnetic resonance image data of the brain of the user, and then perform channel feature extraction and three-dimensional spatial feature extraction based on the channel attention processing result and the spatial attention processing result to obtain second features; a feature fusion module configured to fuse the first features and the second features to obtain fusion features; a classification module configured to classify based on the fusion features to obtain an image recognition result.

13. The image processing apparatus according to claim 12, wherein The first feature extraction module includes a plurality of first feature extraction units in cascade, each first feature extraction unit including: a global feature extraction subunit configured to extract global features of an input of the first feature extraction unit; a channel feature extraction subunit configured to extract features in a channel dimension based on the global features, to obtain channel features; a spatial feature extraction subunit configured to extract features in a spatial dimension based on the global features and the channel features, to obtain spatial features; an output subunit configured to determine an output of the first feature extraction unit according to the global features, the channel features and the spatial features, wherein the output of the last first feature extraction unit is the first feature, and the output of a non-last first feature extraction unit is an input of a next first feature extraction unit.

14. The image processing apparatus according to claim 12, wherein The second feature extraction module includes a plurality of second feature extraction units in cascade, each second feature extraction unit including: a channel attention subunit and a spatial attention subunit independent of each other, and an output subunit; the channel attention subunit is configured to extract features of an input of the second feature extraction unit, to obtain the features of the input, wherein the input of the first second feature extraction unit is the structural magnetic resonance image data; and a channel weight matrix is calculated according to the input and the features thereof; the spatial attention subunit is configured to extract features of an input of the second feature extraction unit, to obtain the features of the input, wherein the input of the first second feature extraction unit is the structural magnetic resonance image data; and a spatial weight matrix is calculated according to the input and the features thereof; the output subunit is configured to process the features of the input by using the channel weight matrix and the spatial weight matrix, to obtain an output of the second feature extraction unit, wherein the output of the last second feature extraction unit is the second feature, and the output of a non-last second feature extraction unit is an input of a next second feature extraction unit.

15. The image processing apparatus of claim 12, further comprising: a visualization module configured to generate a heat map of the brain of the user according to the gradient information of the second to-be-processed feature, to represent a region of the brain of interest of the image recognition result.

16. The image processing apparatus of claim 12, wherein: the first feature extraction module includes a plurality of first feature extraction units in cascade; the second feature extraction module includes a plurality of second feature extraction units in cascade; 17. An image processing apparatus comprising: a feature fusion module cascaded at an output end of the last first feature extraction unit and an output end of the last second feature extraction unit, to fuse the first feature output by the last first feature extraction unit and the second feature output by the last second feature extraction unit, to obtain a fused feature. a memory; and a processor coupled to the memory, the processor configured to execute the method of any one of claims 1-11 based on instructions stored in the memory.

18. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of any one of claims 1-11.

19. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any one of claims 1-11.

Images