Image segmentation and classification method and device

Abstract

The present application discloses an image segmentation and classification method and device, which is applied to the field of image processing technology. The method includes: obtaining a target image including a target area, segmenting and classifying the target image through a neural network model, and obtaining a contour map and a classification result of the target area; in the model, an encoder-decoder structure extracts features from the target image to obtain an initial feature map and multiple decoding feature maps, a first segmentation module obtains a segmentation feature map through the initial feature map, a first classification module obtains a classification feature map through the initial feature map and each decoding feature map, an information interaction module performs feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features, a second segmentation module obtains a contour map of the target area through the target segmentation features, and a second classification module obtains a classification result of the target area through the target classification features. The present application can simultaneously realize image classification and image segmentation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image segmentation and classification method and device. Background Art

[0002] Image classification and image segmentation are both common processing tasks in the field of Computer Vision (CV). Image segmentation is a technique that divides an image into several specific regions and proposes regions of interest, while image classification is a technique that determines the overall category or label of an image. Some related technologies usually regard image classification and image segmentation tasks as two independent tasks, such as using classification models to achieve image classification and using segmentation models to achieve image segmentation; while other related technologies try to explore the correlation between image classification and image segmentation tasks, but they often use one information to assist in processing the other information, such as using segmentation results to assist in image classification. It can be seen that there are few ways to simultaneously process image classification and image segmentation tasks in related technologies, and the efficiency of image classification and image segmentation needs to be improved. In addition, since segmentation models usually output high-dimensional image matrices, while classification models usually output low-dimensional vectors, it is difficult for a single segmentation network structure or classification network structure to simultaneously output results of different sizes, so the joint processing of image classification and image segmentation tasks has become a major challenge. Summary of the invention

[0003] The embodiments of the present application provide an image segmentation and classification method and device for simultaneously realizing image classification and image segmentation.

[0004] On the one hand, an embodiment of the present application provides an image segmentation and classification method, comprising the following steps:

[0005] Acquire a target image, wherein the target image includes a target area;

[0006] The target image is segmented and classified by a pre-trained neural network model to obtain a contour map and a classification result of the target area; wherein the neural network model includes:

[0007] An encoder-decoder structure is used to extract features from the target image to obtain an initial feature map and a plurality of decoded feature maps;

[0008] A first segmentation module, used for obtaining a segmentation feature map according to the initial feature map;

[0009] A first classification module, used for obtaining a classification feature map according to the initial feature map and each of the decoding feature maps;

[0010] An information interaction module, used for performing feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features;

[0011] A second segmentation module, used for obtaining a contour map of the target area according to the target segmentation features;

[0012] The second classification module is used to obtain a classification result of the target area according to the target classification feature.

[0013] On the other hand, an embodiment of the present application provides an image segmentation and classification device, comprising:

[0014] An acquisition module, used for acquiring a target image, wherein the target image includes a target area;

[0015] The image segmentation and classification module is used to perform segmentation and classification processing on the target image through a pre-trained neural network model to obtain a contour map and classification result of the target area; wherein the neural network model includes:

[0016] An encoder-decoder structure is used to extract features from the target image to obtain an initial feature map and a plurality of decoded feature maps;

[0017] A first segmentation module, used for obtaining a segmentation feature map according to the initial feature map;

[0018] A first classification module, used for obtaining a classification feature map according to the initial feature map and each of the decoding feature maps;

[0019] An information interaction module, used for performing feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features;

[0020] A second segmentation module, used for obtaining a contour map of the target area according to the target segmentation features;

[0021] The second classification module is used to obtain a classification result of the target area according to the target classification feature.

[0022] The beneficial effects of the present application are as follows: an image segmentation and classification method and device are provided, first, a target image is obtained, the target image includes a target area; then, the target image is segmented and classified by a pre-trained neural network model to obtain a contour map and a classification result of the target area, wherein the neural network model includes an encoder-decoder structure, a first segmentation module, a first classification module, an information interaction module, a second segmentation module and a second classification module; the encoder-decoder structure is used to extract features from the target image to obtain an initial feature map and multiple decoding feature maps; the first segmentation module is used to obtain a segmentation feature map based on the initial feature map; the first classification module is used to obtain a classification feature map based on the initial feature map and each decoding feature map; the information interaction module is used to perform feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features; the second segmentation module is used to obtain a contour map of the target area based on the target segmentation features; the second classification module is used to obtain a classification result of the target area based on the target classification features. In this way, the embodiment of the present application realizes the synchronous processing of image classification tasks and image segmentation tasks, which can effectively improve the accuracy and efficiency of image classification tasks and image segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the image segmentation and classification method provided by this application;

[0024] Figure 2 It is a structural diagram of the neural network model provided by this application;

[0025] Figure 3 It is a structural diagram of the coding perception module provided by this application;

[0026] Figure 4 is a structural diagram of the decoding perception module provided by this application;

[0027] Figure 5 It is a structural diagram of the information interaction module provided by this application;

[0028] Figure 6 It is a structural diagram of the fusion module provided in this application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] In view of the defects existing in the related technologies, the embodiments of the present application provide an image segmentation and classification method and device, aiming to realize image segmentation while realizing image classification, thereby improving the accuracy and efficiency of image classification tasks and image segmentation tasks.

[0031] The image segmentation and classification method and device provided in the embodiments of the present application mainly involve various application scenarios such as medical image processing, remote sensing image processing, etc. Those skilled in the art can understand that the image segmentation and classification method and device provided in the embodiments of the present application can be implemented in various application scenarios.

[0032] For example, the image segmentation and classification method and device provided in the embodiments of the present application can be applied to brain hematoma classification tasks and brain hematoma segmentation tasks. The target image is a computed tomography image of the brain area, and the target area in the target image is the brain area and the hematoma. Through the embodiments of the present application, the contour map of the brain area and the hematoma and the classification result of the target image can be obtained. The classification result of the target image can be the type of cerebral hemorrhage, and the type of cerebral hemorrhage includes at least one of basal ganglia hemorrhage, thalamic hemorrhage, cerebellar hemorrhage or brainstem hemorrhage.

[0033] For another example, the image segmentation and classification method and device provided in the embodiments of the present application can be applied to plant disease area classification tasks and plant disease area segmentation tasks. The target image is a plant tissue image, such as a leaf image, etc. The target area in the target image is the plant disease area. Through the embodiments of the present application, the contour map and classification results of the plant disease area can be obtained, and the classification result can be the type of insect disease suffered by the plant.

[0034] It is understandable that the above application scenarios are only exemplary and do not limit the practical application of the image segmentation and classification method and device provided in the embodiments of the present application. Those skilled in the art can understand that in different application scenarios, the image segmentation and classification method and device provided in the embodiments of the present application can be used to simultaneously perform image segmentation tasks and image classification tasks.

[0035] First, the implementation of the image segmentation and classification method provided by the present application will be described in detail below with reference to the accompanying drawings.

[0036] The image segmentation and classification method provided in the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0037] Reference Figure 1 and Figure 2 The image segmentation and classification method provided in this application may include the following steps S101-S102:

[0038] S101, acquiring a target image, where the target image includes a target area;

[0039] S102, segmenting and classifying the target image through a pre-trained neural network model to obtain a contour map and a classification result of the target area; wherein the neural network model includes an encoder-decoder structure, a first segmentation module, a first classification module, an information interaction module, a second segmentation module and a second classification module; the encoder-decoder structure is used to extract features of the target image to obtain an initial feature map and multiple decoding feature maps; the first segmentation module is used to obtain a segmentation feature map based on the initial feature map; the first classification module is used to obtain a classification feature map based on the initial feature map and each decoding feature map; the information interaction module is used to perform feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features; the second segmentation module is used to obtain a contour map of the target area based on the target segmentation features; the second classification module is used to obtain a classification result of the target area based on the target classification features.

[0040] In an embodiment of the present application, first, a target image is acquired, where the target image includes a target area, which refers to an area to be classified and segmented, that is, an area of ​​interest for an image classification task and an image segmentation task; then, the target image is input into a pre-trained neural network model, and the target image is segmented and classified by the neural network model, thereby obtaining a contour map and a classification result of the target area. The neural network model may include an encoder-decoder structure, a first segmentation module, a first classification module, an information interaction module, a second segmentation module and a second classification module. In the neural network model, first, the target image is feature extracted by the encoder-decoder structure to obtain feature information associated with the target area, namely, an initial feature map and multiple decoding feature maps; then, the initial feature map is preliminarily processed by the first segmentation module to obtain more detailed feature information associated with the image segmentation task, namely, a segmentation feature map, and the initial feature map and each decoding feature map are preliminarily processed by the first classification module to obtain richer feature information associated with the image classification task, namely, a classification feature map; then, the segmentation feature map and the classification feature map are feature interactively processed by the information interaction module to promote the complementarity and fusion between the feature information of the image segmentation task and the feature information of the image classification task, thereby obtaining the target segmentation feature and the target classification feature; finally, the target segmentation feature is processed by the second segmentation module to obtain the contour map of the target area, thereby realizing image segmentation, and the target classification feature is processed by the second classification module to obtain the classification result of the target area, such as thereby realizing image classification. In this way, the embodiment of the present application realizes the synchronous processing of the image classification task and the image segmentation task, which can effectively improve the accuracy and efficiency of the image classification task and the image segmentation task.

[0041] The above-mentioned target image and its target area can be set according to actual conditions, and the embodiments of the present application do not specifically limit this. For example, in the brain hematoma segmentation task and the brain hematoma classification task, the target image can be a computed tomography image of the brain area, and the target area can be the brain area and the hematoma, but is not limited thereto.

[0042] The classification result of the above-mentioned target area can be set according to the actual situation, and the embodiment of the present application does not specifically limit this. For example, in the brain hematoma classification task, the classification result of the target area can be the type of brain hematoma, and the type of brain hematoma can be flexibly set according to the actual situation. For example, according to the location of the brain hematoma in the brain area, the type of brain hematoma can be divided into four types: basal ganglia hematoma, thalamic hematoma, cerebellar hematoma and brainstem hematoma, but it is not limited to this.

[0043] In some embodiments, reference Figure 2When the encoder-decoder structure is used to extract features from a target image and obtain an initial feature map and multiple decoded feature maps, the specific operations performed may include:

[0044] Downsampling and encoding perception processing are performed on the target image to obtain a first encoding feature map;

[0045] Downsampling and encoding perception processing are performed on the first encoding feature map to obtain a second encoding feature map;

[0046] Downsampling and encoding perception processing are performed on the second encoding feature map to obtain a third encoding feature map;

[0047] Upsampling the third encoded feature map to obtain a first upsampled feature map, performing a jump connection between the first upsampled feature map and the second encoded feature map to obtain a first feature map to be perceived, and then performing a decoding perception process on the first feature map to be perceived to obtain a first decoded feature map;

[0048] Upsampling the first decoded feature map to obtain a second upsampled feature map, and performing a jump connection between the second upsampled feature map and the first encoded feature map to obtain a second feature map to be perceived, and then performing a decoding perception process on the second feature map to be perceived to obtain a second decoded feature map;

[0049] The second decoded feature map is upsampled to obtain an initial feature map.

[0050] In this embodiment, in the above-mentioned neural network model, the encoder-decoder structure may include a first encoder, a second encoder, a third encoder, a first decoder, a second decoder and a first upsampling layer connected in sequence, wherein the first encoder and the second decoder are jump-connected, the second encoder and the first decoder are jump-connected, and the jump connection adopts a series operation.

[0051] The input of the first encoder is the target image. In the first encoder, first, the target image is downsampled to obtain the downsampled target image, aiming to reduce the size of the target image; then, the downsampled target image is subjected to encoding perception processing to obtain the first encoding feature map as the output of the first encoder, aiming to preliminarily extract the feature information associated with the target area. The input of the second encoder is the output of the first encoder. In the second encoder, first, the first encoding feature map is downsampled to obtain the downsampled first encoding feature map, aiming to reduce the size of the first encoding feature map; then, the downsampled first encoding feature map is subjected to encoding perception processing to obtain the second encoding feature map as the output of the second encoder, aiming to further extract the feature information associated with the target area. The input of the third encoder is the output of the second encoder. In the third encoder, first, the second encoding feature map is downsampled to obtain the downsampled second encoding feature map, aiming to reduce the size of the second encoding feature map; then, the downsampled second encoding feature map is subjected to encoding perception processing to obtain the third encoding feature map as the output of the third encoder, aiming to further extract the feature information associated with the target area.

[0052] The input of the first decoder is the output of the third encoder. In the first decoder, first, the third encoded feature map is upsampled to obtain a first upsampled feature map, which is intended to restore the size of the third encoded feature map; then, the first upsampled feature map and the second encoded feature map are jump-connected to obtain a first feature map to be perceived, which is intended to pass the deep high-level detail information of the encoder to the shallow low-level detail information of the decoder, and combine the deep high-level detail information with the shallow low-level detail information, so as to capture more feature information associated with the target area; then, the first feature map to be perceived is decoded and perceived to obtain a first decoded feature map as the output of the first decoder, which is intended to preliminarily restore the feature information associated with the target area. The input of the second decoder is the output of the first decoder. In the second decoder, first, the first decoded feature map is upsampled to obtain the second upsampled feature map, which aims to restore the size of the first decoded feature map; then, the second upsampled feature map and the first encoded feature map are jump-connected to obtain the second feature map to be perceived, which aims to pass the deep high-level detail information of the encoder to the shallow low-level detail information of the decoder, and combine the deep high-level detail information with the shallow low-level detail information, so as to capture more feature information associated with the target area; then, the second feature map to be perceived is decoded and perceived to obtain the second decoded feature map as the output of the second decoder, which aims to further restore the feature information associated with the target area. The input of the first upsampling layer is the output of the second decoder. In the first upsampling layer, the second decoded feature map is upsampled to further restore the feature information associated with the target area, thereby obtaining the initial feature map. In addition, the first upsampling layer uses a transposed convolution with doubled feature width and height and halved output channel number.

[0053] Further, the specific implementation of the first encoder, the second encoder and the third encoder is as follows: Figure 3 , No. The encoder may include a first The encoding downsampling layer and Encoding perception module, , No. The encoding downsampling layer is used to The encoder input is downsampled. The encoding downsampling layer uses average pooling with half the feature width and height. The coding perception module may include the Local feature extraction layer, Encoding the global modeling layer and the Encoding channel interaction layer.

[0054] Specifically, no. The local feature extraction layer is used to extract the The output of the encoding downsampling layer is expanded by the number of channels to obtain Channel expansion feature map, and The channels of the channel expansion feature map are convolved to obtain the first The local encoding feature map can capture local detail features such as corner points of the target area in a targeted manner.

[0055] More specifically, in each encoder, in order to extract local detail features of the target area from multiple angles, The 1×1 convolution transforms the output of the encoding downsampling layer from Expand to , where the number of convolution output channels is equal to the number of convolution kernels to enrich the image features and obtain the channel expansion feature map , as shown in the following formula (1):

[0056] , (1);

[0057] In formula (1), Represents channel expansion feature map No. The width coordinates of the channels are And the height direction coordinate is The pixel value of represents the output of the encoding downsampling layer; Represents the convolution kernel for the output channels and The weights of the input channels; Represents the number of channels of the output of the encoding downsampling layer; Indicates the number of channels of the channel-expanded feature map.

[0058] After completing the channel number expansion, the channel expansion feature map Each channel of is convolved separately, which can identify local features within each channel. That is, using a size of The convolution kernel channel expansion feature map Each channel of is convolved separately to obtain a local encoding feature map ,by Represents the local encoding feature map No. The width coordinates of the channels are And the height direction coordinate is The pixel value is as shown in the following formula (2):

[0059] (2);

[0060] In formula (2), Indicates that for channels inside the convolution kernel The weight of the position.

[0061] Specifically, no. The encoding global modeling layer is used to The size of the output of the downsampling layer is encoded to obtain several sliding windows and move them to cover the first Encode the output of the downsampling layer and get Global encoding features, and then The global encoding feature is activated and regularized to obtain the activated The global encoding feature is then used to focus on the activated The global encoding features are processed to obtain the The global encoding feature map can accurately identify the overall image attribute characteristics such as color, texture, shape, etc. of the target area.

[0062] More specifically, in each encoder, first, multiple windows of the same size are divided according to the size of the output of the encoding downsampling layer, and the output of the encoding downsampling layer is fully covered by moving multiple windows at the same time to identify the global encoding feature. Then, the global encoding feature is activated using the ReLU activation function, and regularized to prevent overfitting, thereby obtaining the activated global encoding feature. , as shown in the following formula (3):

[0063] (3);

[0064] In formula (3), represents the ReLU activation function, is the input of the moving window function, i.e. the output of the encoding downsampling layer, represents a moving window function, represents the group normalization function, and are all preset weight matrices. and are all preset bias items. is the weight matrix The L2 regularization coefficient of is the weight matrix The L2 regularization coefficient of .

[0065] The global encoding feature after activation Afterwards, a multi-head attention mechanism is used to focus on the activated global encoding features After processing, each attention head in the multi-head attention mechanism will calculate its own attention weight, thereby obtaining its own attention output Finally, the outputs of all attention heads are connected in series and fused through linear transformation to obtain the feature map matrix based on multi-head attention, that is, the global encoding feature map , as shown in the following formula (4):

[0066] , (4);

[0067] In formula (4), , It is The attention output of the attention head, represents the Softmax activation function, is a preset scaling factor to prevent gradient vanishing, For the The activated global encoding features are Converted to the weight of the query vector, For the The activated global encoding features are The weights converted to key vectors, For the The activated global encoding features are The weights converted to value vectors, is the total number of attention heads in the multi-head attention mechanism, represents a learnable linear transformation matrix used to fuse the outputs of all attention heads, Represents a concatenation operation.

[0068] Specifically, no. The coding channel interaction layer is used to Local encoding feature map and The global encoding feature map is multiplied by the channel pixel value to obtain the Fusion encoding feature map, and The fused coding feature map is pooled to obtain the Pooling encoding feature map, and then The pooled encoding feature map undergoes at least two convolution operations and at least two activation processes to obtain the first Encode the channel weight vector, and then Encoding channel weight vector and The fused encoding feature map is multiplied by channels to obtain the The output of the encoder helps the neural network model focus on more important feature channels.

[0069] More specifically, in each encoder, the local feature map and the global encoding feature map The pixel values ​​of the corresponding channels of the two feature maps are multiplied to obtain the fused encoding feature map , and use global average pooling to fusion code feature map Pooling is performed to reduce its width and height to 1×1 to obtain a pooled coding feature map, thereby reducing the number of parameters. Considering that a single 1×1 convolution on the pooled coding feature map can only obtain a linear weighted combination of different channels, but cannot capture the nonlinear relationship between different channels well, after the pooling process is completed, a 1×1 convolution is first used to convolve the pooled coding feature map to compress the number of channels of the pooled coding feature map by half, which is conducive to reducing the number of parameters. Then, the Parametric ReLU activation function is used to activate the pooled coding feature map after the channel number is compressed, and then the 1×1 convolution is used again to convolve the activated pooled coding feature map to double the number of channels of the activated pooled coding feature map. Finally, the Sigmoid activation function is used to activate the pooled coding feature map after the channel number is expanded to obtain the coding channel weight vector , is the number of channels. After obtaining the encoding channel weight vector, the scalar contained in the encoding channel weight vector and the fused encoding feature map Multiply channel by channel to obtain the channel-weighted feature map, that is, the encoded feature map of the encoder .set up , then the first The elements can be represented as , which satisfies , represents the first A scalar for each channel, Represents the fused encoding feature map Middle The feature map matrix of channels.

[0070] Furthermore, the specific implementation of the first decoder and the second decoder is as follows:

[0071] Reference Figure 4 , No. The decoder may include a first Decoding upsampling layer, Decoding skip connection layer and Decoding perception module, , No. The decoding upsampling layer is used to The decoder input is upsampled to obtain the Upsampled feature map of decoder, The decoding upsampling layer uses a transposed convolution with doubled feature width and height and half the number of output channels. The decoding skip connection layer is used to The upsampled feature map of the decoder and the The decoder jumps to the output of the encoder, and gets the The decoder's feature map to be perceived, The decoding perception module includes the Local feature reconstruction layer, Decoding the global modeling layer and the Decoding channel interaction layer, where:

[0072] Specifically, no. The local feature reconstruction layer is used to The decoder's feature map to be perceived is compressed by the number of channels to obtain the first Channel compression feature map, and The channels of the channel compression feature map are convolved to obtain the The local decoding feature map can capture local detail features such as corner points of the target area in a targeted manner.

[0073] More specifically, in each decoder, in order to restore the feature information of the target area from multiple angles, a 1×1 convolution is used to reduce the number of channels of the feature map to be sensed from Compress to , to restore the features and obtain the channel compression feature map ,by Represents channel compression feature map No. The width coordinates of the channels are And the height direction coordinate is The pixel value is as shown in the following formula (5):

[0074] , (5);

[0075] In formula (5), represents the feature map to be perceived, is the convolution kernel for the output channels and The weights of the input channels, is the number of channels of the channel compression feature map, is the number of channels of the feature map to be perceived.

[0076] After completing the channel number compression, the channel compression feature map Each channel of is convolved separately, so that local features within each channel can be identified. That is, using a size of The convolution kernel channel compression feature map Each channel of is convolved separately to obtain a local decoding feature map ,by Represented in the local decoding feature map No. The width coordinates of the channels are And the height direction coordinate is The pixel value is as shown in the following formula (6):

[0077] (6);

[0078] In formula (6), Indicates that for The relative position of the channels in the convolution kernel The weight of the position.

[0079] Specifically, no. The decoding global modeling layer is used to The size of the feature map to be perceived by the decoder is obtained, and several sliding windows are moved to cover the first The decoder's feature map to be perceived is obtained Global decoding features, and then The global decoding feature is activated to obtain the activated The global decoding feature is then used to focus on the activated The global decoding features are processed to obtain the Global decoding feature map. It should be understood that the specific implementation of the decoding global modeling layer is the same as the specific implementation of the encoding global modeling layer, which will not be repeated here.

[0080] Specifically, no. The decoding channel interaction layer is used to Local decoding feature map and The global decoding feature map is multiplied by the channel pixel value to obtain the Fusion decoding feature map, and Fusion decoding feature map for pooling processing, get the Pooling decoded feature map, and then The pooled decoded feature map undergoes at least two convolution operations and at least two activation processes to obtain the first Decode the channel weight vector and then Decode channel weight vector and The fused decoded feature maps are multiplied by channels to obtain the The output of the decoder. It should be understood that the specific implementation of the decoding channel interaction layer is the same as the specific implementation of the encoding channel interaction layer mentioned above, which will not be described in detail.

[0081] In some embodiments, reference Figure 2 When the first segmentation module is used to obtain a segmentation feature map according to the initial feature map, the specific operations performed may include:

[0082] Perform multiple convolution operations and multiple activation processes on the initial feature map to obtain a first processed feature map;

[0083] Performing dimension transformation on the first processed feature map to obtain a two-dimensional segmentation feature map;

[0084] Regularization and group normalization are performed on the two-dimensional segmentation feature map to obtain the segmentation feature map.

[0085] In this implementation, in order to obtain more refined segmentation features, in the above neural network model, the first segmentation module may include a first convolution layer, a first activation layer, a second convolution layer, a second activation layer, a first flattening layer, a first regularization layer and a first group of normalization layers connected in sequence. The specific implementation of the first segmentation module is as follows:

[0086] First, the first convolutional layer is used to train the The initial feature map is convolved with a kernel size of 3×3, a padding of 1, and a stride of 1. is the number of channels, is the width, The height is then activated by applying the first activation layer of the Parametric ReLU activation function to the initial feature map after convolution to learn a more complex nonlinear representation, and the first convolution feature map is obtained. Then, the first convolution feature map is convolved once with a convolution kernel size of 3×3, a padding of 1, and a stride of 1 through the second convolution layer, and then the first convolution feature map after convolution is activated by the second activation layer of the Parametric ReLU activation function to obtain a size of The first processed feature map is obtained by using the flatten function. Then, in order to simplify the subsequent three-dimensional matrix multiplication to a two-dimensional matrix multiplication to reduce the number of parameters, the first processed feature map is reduced from a three-dimensional feature map to a two-dimensional feature map through the first flattening layer. More specifically, the width and height dimensions of the first processed feature map are flattened using the flatten function. The pixels are arranged in a row, so that each channel becomes a channel with a length of , thus obtaining a vector of size Two-dimensional segmentation feature map Finally, the first regularization layer is used to segment the two-dimensional feature map SpatialDropout regularization is performed to reduce the risk of overfitting of the neural network model, and then the regularized two-dimensional segmentation feature map is group normalized through the first group normalization layer to limit the value of each pixel in the feature map to the range of [0, 1], resulting in a size of Segmentation feature map , thereby improving the convergence speed of the neural network model. It can be seen that the first segmentation module is equivalent to an initial segmentation head. Placing the segmentation head at the back end of the encoder-decoder structure can realize that the image segmentation task shares the feature information of the encoder-decoder structure; in addition, through the various processing operations of the first segmentation module, richer and more detailed segmentation features can be obtained. In this way, the segmentation accuracy of the target area can be effectively improved.

[0087] In some embodiments, reference Figure 2 When the first classification module is used to obtain a classification feature map according to the initial feature map and each decoding feature map, the specific operations for performing may include:

[0088] Downsampling the initial feature map to obtain a first downsampled feature map, and performing a jump connection between the first downsampled feature map and the second decoded feature map to obtain a first feature map to be convolved, and then performing a convolution operation on the first feature map to be convolved to obtain a second processed feature map;

[0089] Downsampling the second processed feature map to obtain a second downsampled feature map, and performing a jump connection between the second downsampled feature map and the first decoded feature map to obtain a second feature map to be convolved, and then performing a convolution operation on the second feature map to be convolved to obtain a third processed feature map;

[0090] Downsampling the third processed feature map to obtain a third downsampled feature map;

[0091] Expanding the number of channels of the third down-sampled feature map to obtain an expanded third down-sampled feature map;

[0092] Performing dimension transformation on the expanded third down-sampled feature map to obtain a two-dimensional classification feature map;

[0093] The two-dimensional classification feature map is activated, regularized and group normalized to obtain the classification feature map.

[0094] In this embodiment, unlike the segmentation output, the classification output usually requires deep and low spatial resolution feature input, which means that the first classification module as the initial classification head needs to gradually reduce the spatial resolution and integrate deep multi-channel features. Therefore, in the above neural network model, the first classification module may include a second downsampling layer, a third convolutional layer, a third downsampling layer, a fourth convolutional layer, a fourth downsampling layer, a channel expansion layer, a second flattening layer, a third activation layer, a second regularization layer and a second group of normalization layers connected in sequence, the third convolutional layer and the second decoder are jump-connected, the fourth convolutional layer and the first decoder are jump-connected, and the jump connection adopts a series operation. The specific implementation of the first classification module is as follows:

[0095] First, the size is The initial feature map is max-pooled down-sampled to reduce the width of the initial feature map to , reducing the height of the initial feature map to , get the first down-sampled feature map, and add the first down-sampled feature map and the size The second decoded feature map of is skipped to pass the feature information of the second decoder to the first classification module, thereby obtaining a size of The first feature map to be convolved , and then the first feature map to be convolved is processed through the third convolution layer A convolution operation with a convolution kernel size of 3×3, a padding of 1, and a stride of 1 is performed to obtain the second processed feature map. Secondly, the second processed feature map is subjected to maximum pooling downsampling through the third downsampling layer to reduce the width of the second processed feature map to , reduce the height of the second processed feature map to , get the second down-sampled feature map, and add the second down-sampled feature map and the size The first decoded feature map of is skipped to pass the feature information of the first decoder to the first classification module, thereby obtaining a size of The second convolution feature map , and then the second feature map to be convolved is processed through the fourth convolution layer A convolution operation with a convolution kernel size of 3×3, a padding of 1, and a stride of 1 is performed to obtain the third processed feature map. Then, the third processed feature map is subjected to maximum pooling downsampling through the fourth downsampling layer to reduce the width of the third processed feature map to , reduce the height of the third processing feature map to , get the third down-sampled feature map, and use 1×1 convolution in the channel expansion layer to increase the number of channels of the third down-sampled feature map from Expand to , to further enrich the features, so as to obtain a size of The expanded third down-sampled feature map Afterwards, in the second flattening layer, the expanded third downsampled feature map The width and height of are flattened using the Flatten function. The pixels are arranged in a row, so that each channel becomes a channel with a length of , thus obtaining a vector of size Two-dimensional classification feature map of Finally, in the third activation layer, the Parametric ReLU activation function is used to transform the two-dimensional classification feature map Activate to learn more complex nonlinear representations, and then perform Spatial Dropout regularization on the activated two-dimensional classification feature map through the second regularization layer to reduce the risk of overfitting of the neural network model, and limit each pixel value of the regularized two-dimensional classification feature map to the range of [0, 1] through the second group of normalization layers, so as to obtain a size of Classification feature map of .

[0096] It can be seen that by taking the first classification module as the initial classification head and placing the initial classification head at the back end of the encoder-decoder structure, the image classification task can share the feature information of the encoder-decoder structure; in addition, the feature maps of the same spatial resolution in the decoder and the first classification module are fused in a jump connection manner and passed to the first classification module, so that the first classification module can obtain richer feature information of the target area; in addition, more detailed and richer classification features can be obtained through various processing operations of the first classification module. In this way, the classification accuracy of the target area can be effectively improved.

[0097] In some embodiments, when the information interaction module is used to perform feature interaction processing on the segmentation feature map and the classification feature map to obtain the target segmentation feature and the target classification feature, the specific operations for performing may include:

[0098] According to the segmentation feature map and the classification feature map, a first sub-classification feature, a first sub-segmentation feature and a first fusion feature are obtained;

[0099] According to the segmentation feature map and the first fusion feature, combined with the multi-head attention mechanism, the first attention feature map is obtained;

[0100] According to the classification feature map and the first fusion feature, combined with the multi-head attention mechanism, the second attention feature map is obtained;

[0101] According to the first sub-classification feature and the first sub-segmentation feature, a second sub-classification feature, a second sub-segmentation feature and a second fusion feature are obtained;

[0102] According to the segmentation feature map and the second fusion feature, the third attention feature map is obtained by combining the multi-head attention mechanism;

[0103] According to the classification feature map and the second fusion feature, combined with the multi-head attention mechanism, the fourth attention feature map is obtained;

[0104] According to the second sub-segmentation feature and the second sub-classification feature, a third sub-segmentation feature and a third fusion feature are obtained;

[0105] According to the segmentation feature map and the third fusion feature, combined with the multi-head attention mechanism, the fifth attention feature map is obtained;

[0106] According to the classification feature map and the third fusion feature, combined with the multi-head attention mechanism, the sixth attention feature map is obtained;

[0107] Obtaining target segmentation features according to the first attention feature map, the third attention feature map, the fifth attention feature map, and the segmentation feature map;

[0108] According to the second attention feature map, the fourth attention feature map, the sixth attention feature map and the classification feature map, the target classification feature is obtained.

[0109] In this embodiment, in order to realize the joint processing of image classification task and image segmentation task, a segmentation and classification multi-task information interaction module based on interactive multi-head attention is designed in the above-mentioned neural network model. In this information interaction module, firstly, parallel convolution operations are used on the segmentation feature map and the classification feature map respectively to further extract feature information, and then parallel pooling operations are used to reduce the amount of parameters to improve the running speed of the neural network model, and sub-segmentation features and sub-classification features are obtained. Then, the sub-segmentation features are deformed and transposed and fused with the sub-classification features to obtain fused features. Then, the segmentation feature map and the classification feature map are used as query vectors, respectively, and the fused features are used as key vectors and value vectors, and the attention mechanism is processed. The above operation is performed multiple times to obtain the target classification features associated with the image classification task and the target segmentation features associated with the image segmentation task, thereby realizing multi-task feature interaction.

[0110] Specifically, refer to Figure 5 The information interaction module may include a first fusion module, a second fusion module, a third fusion module, a first attention module, a second attention module, a third attention module, a fourth attention module, a fifth attention module, a sixth attention module, a first interaction module and a second interaction module.

[0111] Furthermore, the first fusion module, the second fusion module and the third fusion module are implemented as follows: Figure 6 , No. The fusion module includes Split branch, Classification branch and Fusion processing module, , No. The input of the split branch is Sub-segmentation features, The output of the split branch is Sub-segmentation features, where Time Sub-segmentation features refer to segmentation feature maps; The input of the classification branch is Sub-category characteristics, The output of the classification branch is Sub-classification features, where Time Sub-classification features refer to classification feature maps; The input of the fusion processing module is Sub-segmentation features and Sub-category characteristics, The output of the fusion processing module is Fusion features.

[0112] No. The split branch consists of four parallel connected Split convolutional layer, two concatenated Split pooling layer and two Split multiplier, four parallel connected The split convolutional layers are 1×1 convolutional layers with padding of 0 and stride of 1, 3×3 convolutional layers with padding of 1 and stride of 1, 5×5 convolutional layers with padding of 2 and stride of 1, and 7×7 convolutional layers with padding of 3 and stride of 1. The split pooling layers are the maximum pooling layer and the average pooling layer. In the split branch, first, through each The split convolutional layer The convolution operation of the sub-segmentation features is designed to introduce multi-scale convolution to extract richer underlying features; then, through the first The split multiplier divides the four The output of the split convolution layer is multiplied element by element, aiming to fully integrate the feature information of different dimensions and realize feature interaction; then, through each The split pooling layer is used for the first The output of the split multiplier is pooled in parallel to obtain the global maximum feature and average feature of each channel; finally, the second The split multiplier divides the two The output of the split pooling layer is element-wise multiplied to get the precise Sub-segmentation features.

[0113] No. The classification branch consists of four parallel Classification convolutional layer, two concatenated Classification pooling layer and two Classification multiplier, four parallel connected The classification convolutional layers are 1×1 convolutional layers with padding of 0 and stride of 1, 3×3 convolutional layers with padding of 1 and stride of 1, 5×5 convolutional layers with padding of 2 and stride of 1, and 7×7 convolutional layers with padding of 3 and stride of 1. The classification pooling layers are the maximum pooling layer and the average pooling layer. In the classification branch, first, through each The classification convolution layer The convolution operation of the sub-classification features is designed to introduce multi-scale convolution to extract richer underlying features; then, through the first The classification multiplier converts the four The output of the classification convolution layer is multiplied element by element, aiming to fully integrate the feature information of different dimensions and realize feature interaction; then, through each The classification pooling layer is the first The output of the classification multiplier is pooled in parallel to obtain the global maximum feature and average feature of each channel; finally, the second The classification multiplier converts the two The output of the classification pooling layer is multiplied element-wise to obtain the accurate Sub-category characteristics.

[0114] In the In the fusion processing module, first Sub-segmentation features and preset Multiply the fusion deformation matrix to get the Sub-segment deformation feature map, and Sub-segmentation deformation feature map is matrix transposed to obtain the transposed The sub-segmentation deformation feature map aims to make the Sub-segmentation features and The sub-classification features can satisfy the matrix multiplication condition during the interaction process, and then Sub-class features and the transposed The sub-segmentation deformation feature map is multiplied to obtain the Fusion features to achieve feature fusion and interaction.

[0115] Furthermore, the specific implementations of the first attention module, the second attention module, the third attention module, the fourth attention module, the fifth attention module, and the sixth attention module are as follows:

[0116] Reference Figure 5 The first attention module, the third attention module and the fifth attention module all correspond to the image segmentation task. The input of the first attention module includes the segmentation feature map output by the first segmentation module and the first fusion feature output by the first fusion module, and the output of the first attention module is the first attention feature map; the input of the third attention module includes the segmentation feature map output by the first segmentation module and the second fusion feature output by the second fusion module, and the output of the third attention module is the third attention feature map; the input of the fifth attention module includes the segmentation feature map output by the first segmentation module and the third fusion feature output by the third fusion module, and the output of the fifth attention module is the fifth attention feature map. The second attention module, the fourth attention module and the sixth attention module all correspond to the image classification task. The input of the second attention module includes the classification feature map output by the first classification module and the first fusion feature output by the first fusion module, and the output of the second attention module is the second attention feature map; the input of the fourth attention module includes the classification feature map output by the first classification module and the second fusion feature output by the second fusion module, and the output of the fourth attention module is the fourth attention feature map; the input of the sixth attention module includes the classification feature map output by the first classification module and the third fusion feature output by the third fusion module, and the output of the sixth attention module is the sixth attention feature map.

[0117] In the In the attention module, for , firstly, the segmentation feature map and the preset Multiply the attention deformation matrix to get Deform the feature map so that the segmentation feature map meets the conditions of multi-head attention calculation, and then The deformation feature map is used as the query vector to be input into the The fused features of the attention module are used as key vectors and value vectors for the Deformation feature map and input to the first The fusion features of the attention module are used for attention calculation to obtain the Attention feature map; for First, the classification feature map and the preset Multiply the attention deformation matrix to get Deform the feature map so that the classification feature map meets the conditions of multi-head attention calculation, and then The deformation feature map is used as the query vector to be input into the The fused features of the attention module are used as key vectors and value vectors for the Deformation feature map and input to the first The fusion features of the attention module are used for attention calculation to obtain the Attention feature map.

[0118] Based on the above, refer to Figure 5 The specific implementation of the information interaction module provided in the embodiment of the present application is as follows:

[0119] The segmentation feature map output by the first segmentation module Classification feature map output by the first classification module Input to the first fusion module, in the first fusion module, the segmentation feature map Four types of convolutions are used: 1×1 convolution with padding of 0 and stride of 1, 3×3 convolution with padding of 1 and stride of 1, 5×5 convolution with padding of 2 and stride of 1, and 7×7 convolution with padding of 3 and stride of 1. These four types of convolutions are processed in parallel and fused by element multiplication. Then, average pooling and maximum pooling are processed in parallel and fused by element multiplication to obtain a size of The first sub-segmentation feature At the same time, the classification feature map Four types of convolutions are used: 1×1 convolution with padding of 0 and stride of 1, 3×3 convolution with padding of 1 and stride of 1, 5×5 convolution with padding of 2 and stride of 1, and 7×7 convolution with padding of 3 and stride of 1. These four types of convolutions are processed in parallel and fused by element multiplication. Then, average pooling and maximum pooling are processed in parallel and fused by element multiplication to obtain a size of The first sub-category characteristics ; Then, the size is The first fusion deformation matrix and the first sub-segmentation feature Multiplying them, we get the size The first sub-segmentation deformation feature map , and transform the first sub-segmentation deformation feature map Transpose the matrix to obtain the first sub-segmentation deformation feature map after transposition, and finally transform the first sub-classification feature map Multiply it with the transposed first sub-segmentation deformation feature map to get a size of The first fusion feature .

[0120] In the first attention module, the segmentation feature map output by the first segmentation module is and size is Multiply the first attention deformation matrix by , and get a size of The first deformation feature map , followed by the first deformation feature map As the query vector, the first fusion feature As key vectors and value vectors, the calculation of the multi-head attention mechanism is carried out, aiming to make up for the feature information of the image segmentation task with the help of the fusion features that integrate the segmentation feature information and the classification feature information, and realize the interactive fusion of the image classification task and the image segmentation task, so as to obtain the attention output of each attention head , The size is Finally, the outputs of all attention heads are concatenated, and the feature size after concatenation becomes , and then perform linear transformation to obtain the size First attention feature map , where the linear transformation takes the dimension The linear transformation matrix of .

[0121] In the second attention module, the classification feature map output by the first classification module is and size is Multiply the second attention deformation matrix by the size of The second deformation feature map , followed by the second deformation feature map As the query vector, the first fusion feature As key vectors and value vectors, the calculation of the multi-head attention mechanism is carried out, aiming to make up for the feature information of the image classification task with the help of the fusion features that integrate the segmentation feature information and the classification feature information, and realize the interactive fusion of the image classification task and the image segmentation task, so as to obtain the attention output of each attention head , The size is Finally, the outputs of all attention heads are concatenated and linearly transformed to obtain a size of Second attention feature map .

[0122] The first sub-segmentation feature output by the first fusion module and the first sub-classification feature Input to the second fusion module, in the second fusion module, the first sub-segmentation feature Four types of convolutions are used: 1×1 convolution with padding of 0 and stride of 1, 3×3 convolution with padding of 1 and stride of 1, 5×5 convolution with padding of 2 and stride of 1, and 7×7 convolution with padding of 3 and stride of 1. These four types of convolutions are processed in parallel and fused by element multiplication. Then, average pooling and maximum pooling are processed in parallel and fused by element multiplication to obtain a size of The second sub-segmentation feature ; At the same time, for the first sub-classification feature Four types of convolutions are used: 1×1 convolution with padding of 0 and stride of 1, 3×3 convolution with padding of 1 and stride of 1, 5×5 convolution with padding of 2 and stride of 1, and 7×7 convolution with padding of 3 and stride of 1. These four types of convolutions are processed in parallel and fused by element multiplication. Then, average pooling and maximum pooling are processed in parallel and fused by element multiplication to obtain a size of The second sub-classification characteristics ; Then, the second sub-segmentation feature and size is Multiply the second blend deformation matrix by the size of The second sub-segmentation deformation feature map , and the second sub-segmentation deformation feature map Transpose the matrix to obtain the transposed second sub-segmentation deformation feature map, and finally transform the transposed second sub-segmentation deformation feature map and the second sub-classification feature map Multiplying them, we get the size The second fusion feature .

[0123] In the third attention module, the segmentation feature map output by the first segmentation module is and size is The third attention deformation matrix is ​​multiplied by , and the size is The third deformation feature map , and the third deformation feature map As the query vector, the second fusion feature As the key vector and value vector, the multi-head attention mechanism is calculated, aiming to further supplement the feature information of the image segmentation task with the fusion feature that integrates the segmentation feature information and the classification feature information, and realize the interactive fusion of the image classification task and the image segmentation task, so as to obtain the attention output of each attention head. , The size is Finally, the outputs of all attention heads are concatenated and linearly transformed to obtain a size of The third attention feature map .

[0124] In the fourth attention module, the classification feature map output by the first classification module is and size is The fourth attention deformation matrix is ​​multiplied by , and the size is The fourth deformation characteristic diagram , and the fourth deformation characteristic diagram As the query vector, the second fusion feature As the key vector and value vector, the multi-head attention mechanism is calculated, aiming to further supplement the feature information of the image classification task with the fusion feature that integrates the segmentation feature information and the classification feature information, and realize the interactive fusion of the image classification task and the image segmentation task, so as to obtain the attention output of each attention head. , The size is Finally, the outputs of all attention heads are concatenated and linearly transformed to obtain a size of The fourth attention feature map .

[0125] The second sub-segmentation features output by the second fusion module and the second sub-classification characteristics Input to the third fusion module, in which the second sub-segmentation feature Four types of convolutions are used: 1×1 convolution with padding of 0 and stride of 1, 3×3 convolution with padding of 1 and stride of 1, 5×5 convolution with padding of 2 and stride of 1, and 7×7 convolution with padding of 3 and stride of 1. These four types of convolutions are processed in parallel and fused by element multiplication. Then, average pooling and maximum pooling are processed in parallel and fused by element multiplication to obtain a size of The third sub-segmentation feature ; At the same time, for the second sub-classification feature Four types of convolutions are used: 1×1 convolution with padding of 0 and stride of 1, 3×3 convolution with padding of 1 and stride of 1, 5×5 convolution with padding of 2 and stride of 1, and 7×7 convolution with padding of 3 and stride of 1. These four types of convolutions are processed in parallel and fused by element multiplication. Then, average pooling and maximum pooling are processed in parallel and fused by element multiplication to obtain a size of The third sub-category characteristics ; Then, the third sub-segmentation feature and size is The third blend deformation matrix is ​​multiplied by the size of The third sub-segmentation deformation feature map , and the deformation feature map of the third sub-segmentation Transpose the matrix to obtain the third sub-segmentation deformation feature map after transposition, and finally transform the third sub-classification feature map Multiply it with the transposed third sub-segmentation deformation feature map to get a size of The third fusion feature .

[0126] In the fifth attention module, the segmentation feature map output by the first segmentation module is and size is Multiply the fifth attention deformation matrix by , and get a size of The fifth deformation feature map , and the fifth deformation feature map As the query vector, the third fusion feature As key vectors and value vectors, multi-head attention mechanism calculation is performed, aiming to further supplement the feature information of the image segmentation task with the fusion features that integrate segmentation feature information and classification feature information, and realize the interactive fusion of image classification task and image segmentation task, so as to obtain the attention output of each attention head , The size is Finally, the outputs of all attention heads are concatenated and linearly transformed to obtain a size of The fifth attention feature map .

[0127] In the sixth attention module, the classification feature map output by the first classification module is and size is Multiply the sixth attention deformation matrix by , and get a size of The sixth deformation characteristic diagram , and the sixth deformation feature map As the query vector, the third fusion feature As key vectors and value vectors, multi-head attention mechanism calculation is performed, aiming to further supplement the feature information of image classification tasks with the help of fusion features that integrate segmentation feature information and classification feature information, and realize the interactive fusion of image classification tasks and image segmentation tasks, so as to obtain the attention output of each attention head. , The size is Finally, the outputs of all attention heads are concatenated and linearly transformed to obtain a size of The fifth attention feature map .

[0128] In the first interaction module, first, the first attention feature map and size is Multiply the first interactive deformation matrix of The first interactive feature map , and the first interaction feature map And the third attention feature map Multiplying them, we get the size The second interactive feature map Then, the second interactive feature map and size is Multiply the second interactive deformation matrix by , and get a size of The third interactive feature map , and the third interactive feature map and the fifth attention feature map Multiplying them, we get the size The fourth interactive feature map After that, the fourth interactive feature map and size is Multiply the third interactive deformation matrix by the size of The fifth interactive feature map , and the fifth interactive feature map And the segmentation feature map output by the first segmentation module Multiply element by element, and the size is The initial target segmentation features Finally, the initial target segmentation features Use the reshape function to reshape the size to get Target segmentation features It can be seen that firstly fusing the attention feature maps of various scales by matrix multiplication can mine deeper feature information associated with the image segmentation task, and then multiplying and reshaping the segmentation feature map and the fused feature information by elements, thereby realizing the fusion and interaction of features at different levels, where features at different levels refer to deep features and deeper features, which can effectively improve the accuracy of the target segmentation features and ensure the feature extraction capability of the neural network model.

[0129] In the second interaction module, first, the second attention feature map and size is The fourth interactive deformation matrix is ​​multiplied by the size of The sixth interactive feature map , and the sixth interactive feature map and the fourth attention feature map Multiplying them, we get the size The seventh interactive feature map Then, the seventh interactive feature map and size is Multiply the fifth interactive deformation matrix by the size of The eighth interactive feature map , and the eighth interactive feature map and the sixth attention feature map Multiplying them, we get the size The ninth interactive feature map After that, the ninth interactive feature map and size is Multiply the sixth interactive deformation matrix by the size of The tenth interactive feature map , and the tenth interactive feature map And the classification feature map output by the first classification module Multiply element by element, and the size is The initial target classification features Finally, the initial target classification features Use the reshape function to reshape the size to get The target classification features It can be seen that firstly fusing the attention feature maps of various scales by matrix multiplication can mine deeper feature information associated with the image classification task, and then multiplying and reshaping the classification feature map and the fused feature information by elements, thereby realizing the fusion and interaction of features at different levels, where features at different levels refer to deep features and deeper features, which can effectively improve the accuracy of the target classification features and ensure the feature extraction capability of the neural network model.

[0130] It should be noted that all the above deformation matrices are pre-set matrices of specific size with all elements being 1. These matrices can be matrix-multiplied with the feature maps output by each layer of the model, thereby changing the size of the feature maps and achieving "deformation", so that the segmentation features and classification features can meet the matrix multiplication conditions during the interaction process, that is, the number of columns of the segmentation features is equal to the number of rows of the classification features, or the number of columns of the classification features is equal to the number of rows of the segmentation features. During training, the element values ​​of the deformation matrix can be updated through back propagation to achieve "learnability".

[0131] In some embodiments, reference Figure 2 When the second segmentation module is used to obtain a contour map of the target area according to the target segmentation features, the specific operations performed may include:

[0132] Performing pooling processing on the target segmentation features to obtain pooled target segmentation features;

[0133] Perform multiple convolution operations and multiple activation processes on the pooled target segmentation features to obtain the segmentation channel weight vector;

[0134] Multiply the segmentation channel weight vector and the target segmentation feature by channel to obtain the channel-weighted target segmentation feature;

[0135] The channel-weighted target segmentation features are subjected to multiple convolution operations and multiple activation processes to obtain a contour map of the target area.

[0136] In this embodiment, in the above-mentioned neural network model, the second segmentation module includes a first pooling layer, a fifth convolutional layer, a fourth activation layer, a sixth convolutional layer, a fifth activation layer, a first channel weighted layer, a seventh convolutional layer, a sixth activation layer, an eighth convolutional layer, a seventh activation layer, a ninth convolutional layer and an eighth activation layer connected in sequence.

[0137] Specifically, the target segmentation features are segmented through the first pooling layer Perform global average pooling to segment the target features The width and height of are reduced to 1×1 to reduce the number of parameters and obtain the pooled target segmentation features. Since only a single 1×1 convolution on the pooled target segmentation features can only obtain a linear weighted combination of different channels, it is not conducive to capturing the nonlinear relationship between different channels. Therefore, the fifth convolutional layer is first used to perform a single 1×1 convolution on the pooled target segmentation features to reduce the number of channels from Compress to , which is helpful to reduce the number of parameters. Then, the fourth activation layer of the ParametricReLU activation function is used to activate the target segmentation features after the channel number is compressed to learn more complex nonlinear representations. Then, a single 1×1 convolution is performed on the activated target segmentation features through the sixth convolutional layer to reduce the number of channels from Expand to , and activate the target segmentation features after the channel number expansion by applying the fifth activation layer of the Sigmoid function to obtain the segmentation channel weight vector , is the number of channels. In the first channel weighted layer, the scalar contained in the segmentation channel weight vector and the target segmentation feature Multiply each channel to obtain the channel-weighted target segmentation features .set up , then the channel weighted target segmentation feature The elements can be represented as , which satisfies , represents the first segmentation channel weight vector A scalar for each channel, Represents the target segmentation feature Middle The feature map matrix of channels.

[0138] After completing the channel weighting, the target segmentation features after channel weighting are convolved once with a convolution kernel size of 3×3, padding of 1, and stride of 1 through the seventh convolution layer, and then activated through the sixth activation layer with the Parametric ReLU activation function, and then the output of the sixth activation layer is convolved once with a convolution kernel size of 3×3, padding of 1, and stride of 1 through the eighth convolution layer, and activated through the seventh activation layer with the Parametric ReLU activation function, and finally, the output of the seventh activation layer is convolved once with a 1×1 convolution through the ninth convolution layer to change the number of channels of the output of the seventh activation layer to the first preset number, and activated through the eighth activation layer with the Softmax activation function to obtain the contour map of the target area. In this way, the accuracy of image segmentation can be improved.

[0139] Exemplarily, in the brain hematoma segmentation task, in the ninth convolutional layer, a 1×1 convolution is performed on the output of the seventh activation layer to change the number of channels of the output of the seventh activation layer to 6, representing the background, hematoma, cerebellum, brainstem, basal ganglia and thalamus, respectively, and the Softmax activation function is applied to obtain the contour map of the brain hematoma.

[0140] In some embodiments, reference Figure 2 When the second classification module is used to obtain the classification result of the target area according to the target classification feature, the specific operations to be performed may include:

[0141] Performing pooling processing on the target classification features to obtain pooled target classification features;

[0142] Perform multiple convolution operations and multiple activation processes on the pooled target classification features to obtain the classification channel weight vector;

[0143] Multiply the classification channel weight vector and the target classification feature by channel to obtain the channel-weighted target classification feature;

[0144] The target classification features after channel weighting are pooled to obtain target pooling features;

[0145] Compress the number of channels of the target pooling features to obtain the target classification vector;

[0146] The target classification vector is processed probabilistically to obtain the classification result of the target area.

[0147] In this embodiment, in the above-mentioned neural network model, the second classification module includes a second pooling layer, a tenth convolutional layer, a ninth activation layer, an eleventh convolutional layer, a tenth activation layer, a second channel weighted layer, a third pooling layer, a fully connected layer and an eleventh activation layer.

[0148] Specifically, the target classification features are pooled through the second pooling layer Perform global average pooling to classify target features The height and width of the pooled target classification feature are reduced to 1×1 to obtain the pooled target classification feature. Since a single 1×1 convolution on the pooled target classification feature can only obtain a linear weighted combination of different channels, which is not conducive to capturing the nonlinear relationship between different channels, the pooled target classification feature is firstly convolved by 1×1 through the tenth convolution layer to reduce the number of channels of the pooled target classification feature from Compress to , to reduce the number of parameters, and then activate the target classification features after channel compression through the ninth activation layer using the Parametric ReLU activation function, and then perform 1×1 convolution on the activated target classification features through the eleventh convolution layer to reduce the number of channels of the activated target classification features from Expand to , and activate the target classification features after the channel number expansion by applying the tenth activation layer of the Sigmoid activation function to obtain the classification channel weight vector In the second channel weighting layer, the scalar contained in the classification channel weight vector is combined with the target classification feature Multiply each channel to obtain the channel-weighted target classification features .set up , then the first channel weighted target classification feature The elements can be represented as , which satisfies , Represents the first in the classification channel weight vector A scalar for each channel, Represents the target classification feature Middle The feature map matrix of channels.

[0149] After completing the channel weighting, the target classification features after channel weighting are globally averaged pooled through the third pooling layer to reduce the width and height of the target classification features after channel weighting to 1, and obtain the target pooling features, and the number of channels of the target pooling features is compressed to the second preset number through the fully connected layer, thereby obtaining a target classification vector with a size of 1×the second preset number. Finally, in the eleventh activation layer, the target classification vector is probabilistically processed using the Sigmoid function to obtain a vector including multiple probability values, the number of probability values ​​is the second preset number, and a probability threshold is set. In this vector, if the probability value is higher than the preset threshold, the probability value is modified to 1, otherwise it is 0, thereby obtaining a classification result in vector form, and the multiple integers contained in the classification result represent the type of the area where the target area is located in turn, wherein the total number of integers contained in the classification result is the second preset number. In this way, the accuracy of image classification can be improved.

[0150] Exemplarily, in the task of classifying brain hematomas, when the second preset number is 4, a target classification vector of size 1×4 can be obtained, and then the Sigmoid function is used to perform probability processing on the target classification vector to obtain a vector including 4 probability values, and the vector is threshold processed to obtain a classification result in vector form. The four integers contained in the classification result represent whether the basal ganglia, thalamus, cerebellum, and brainstem are bleeding ("1" represents bleeding in the brain region, and "0" represents the opposite). For example, the classification result of (0, 1, 0, 0) represents thalamic hemorrhage only, and another example is that the classification result of (1, 1, 0, 0) represents basal ganglia hemorrhage and thalamic hemorrhage.

[0151] In some embodiments, the above method may further include the following steps:

[0152] During the training of the neural network model, an adaptive Dice loss function and an adaptive focus loss function of the neural network model are obtained;

[0153] The adaptive focus loss function and the adaptive Dice loss function are weighted to obtain the loss function of the neural network model.

[0154] In this embodiment, the data set used to train the neural network model may have a situation where the number of samples of certain bleeding types is much larger or smaller than the number of samples of other bleeding types, which causes the neural network model to be more inclined to identify the sample type as a high-frequency sample type, and is prone to identify low-frequency sample types as high-frequency sample types, thereby causing classification errors. In order to more effectively handle this category imbalance, for the image classification task, this embodiment introduces a weighted adaptive focal loss function that satisfies the following formula (7):

[0155] (7);

[0156] In formula (7), represents the weighted adaptive focal loss function, Represents the probability of the neural network model predicting a positive sample, For the The adjustment factor of each category is different from the traditional focal loss function. This adjustment factor is a dynamic variable. The rarer categories have smaller adjustment factors, thereby increasing In addition, Yes Category The class weight coefficient is used to ensure that the neural network model is not overly biased towards the majority class during training.

[0157] Specifically, no. The adjustment factors of each category satisfy , For the The number of samples in each category, is the number of samples of the category with the largest number of samples. The category weight coefficient satisfies , represents the total number of samples; is a hyperparameter that controls the weight gap between categories. The adjustment factor and category weight coefficient of each category are calculated and then applied to the calculation of the weighted adaptive focal loss function to more effectively handle the category imbalance problem of image classification tasks.

[0158] For the image segmentation task, this implementation introduces a weighted adaptive Dice loss function, which not only integrates the Dice loss functions of multiple categories, but also introduces an adaptive weight mechanism to balance the contributions of different categories to ensure that small targets can also receive sufficient attention. The weighted adaptive Dice loss function satisfies the following formula (8):

[0159] (8);

[0160] In formula (8), represents the weighted adaptive Dice loss function, represents the total number of categories, Represents the total number of pixels, Indicates Pixels The true labels of the categories, Indicates Pixels The predicted probability of each category, is a very small number, It is Adaptive weights for each class.

[0161] Specifically, no. The adaptive weights of the categories satisfy , For all tags The sum of the pixel areas of the categories, is the sum of the pixel areas of all labels.

[0162] Since the present embodiment can complete two types of tasks, image classification and image segmentation, by training one neural network model, the loss functions of the two tasks are combined into one loss function, as shown in the following formula (9):

[0163] (9);

[0164] In formula (9), is the loss function of the neural network model, is the weight parameter used to balance the two loss functions.

[0165] In some embodiments, before performing segmentation and classification processing on the target image using the pre-trained neural network model, the method may further include the following steps:

[0166] Preprocess the target image.

[0167] In this embodiment, in order to ensure the quality of the image input to the neural network model and the accuracy of the neural network model, the target image is preprocessed before being input to the neural network model.

[0168] For example, in the brain hematoma segmentation task and the brain hematoma classification task, the input image is a computer tomography image of the brain region. Since the skull in the computer tomography image of the brain region is usually high-brightness, and the brain hematoma is also high-brightness, the skull may interfere with the recognition and segmentation of the brain hematoma by the neural network model, so it is necessary to remove the skull in the computer tomography image of the brain region. Specifically, first obtain a computer tomography image of the brain region, then binarize the computer tomography image of the brain region to obtain a binary image, wherein the value of the skull region in the binary image is 1, and the value of the rest is 0, then multiply the binary image and its original computer tomography image of the brain region to obtain a skull image, and finally subtract the original computer tomography image of the brain region from the skull image to obtain a target image, thereby removing the skull in the computer tomography image of the brain region and reducing the interference of the skull on the recognition and segmentation of the brain hematoma.

[0169] In addition, an embodiment of the present application also provides an image segmentation and classification device, including: an acquisition module and an image segmentation and classification module, the acquisition module is used to acquire a target image, the target image includes a target area; the image segmentation and classification module is used to perform segmentation and classification processing on the target image through a pre-trained neural network model to obtain a contour map and a classification result of the target area; wherein the neural network model includes: an encoder-decoder structure, a first segmentation module, a first classification module, an information interaction module, a second segmentation module and a second classification module, the encoder-decoder structure is used to extract features from the target image to obtain an initial feature map and multiple decoding feature maps; the first segmentation module is used to obtain a segmentation feature map based on the initial feature map; the first classification module is used to obtain a classification feature map based on the initial feature map and each decoding feature map; the information interaction module is used to perform feature interaction between the segmentation feature map and the classification feature map to obtain a target segmentation feature and a target classification feature; the second segmentation module is used to obtain a contour map of the target area based on the target segmentation feature; the second classification module is used to obtain a classification result of the target area based on the target classification feature.

[0170] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0171] In summary, compared with the related art, the present application can realize image classification and image segmentation at the same time, and improve the efficiency of image classification and image segmentation. In addition, the present application can also bring the following technical effects:

[0172] On the one hand, the present application introduces encoding perception and decoding perception into the encoder-decoder structure, and uses encoding perception and decoding perception to specifically capture local detail features such as corner points of the target area and overall image attribute features such as color, texture, and shape of the target area, and then fuses the local detail features and the overall attribute features to obtain the output of the encoder and the output of the decoder. This can help the neural network model focus on more important feature channels, improve the feature extraction capability of the neural network model, and improve the accuracy of feature information associated with the target area.

[0173] On the other hand, the present application places both the first segmentation module and the first classification module at the back end of the encoder-decoder structure, so that the image segmentation task and the image classification task can share the feature information of the encoder and the feature information of the decoder, and only one neural network model needs to be trained to optimize the image segmentation task and the image classification task at the same time, which is more efficient and simple; in addition, the present application jumps the decoder in the encoder-decoder structure with the first classification module, that is, the feature maps of the same spatial resolution in each layer of the decoder and the first classification module are fused in a jump connection manner and passed to the first classification module, so that the first classification module can obtain more feature information. In this way, the image classification accuracy and image segmentation accuracy can be improved.

[0174] On the other hand, the present application designs an information interaction module based on interactive multi-head attention, which first uses parallel convolution and parallel pooling for segmentation features and classification features respectively, then deforms and transposes the segmentation features and multiplies and fuses them with the classification features, and then uses the segmentation features and classification features as query vectors respectively, and the segmentation classification fusion features as key and value vectors, so as to obtain information related to the segmentation and classification tasks in the segmentation classification fusion features, and realize segmentation classification interaction. In this way, the recognition ability of the neural network model for multi-task correlation can be improved, and the feature complementarity between image segmentation tasks and image classification tasks can be promoted, thereby further improving the image classification accuracy and image segmentation accuracy.

[0175] On the other hand, the present application designs a loss function suitable for joint training of image segmentation tasks and image classification tasks. Through the combined loss function of the weighted adaptive focus loss function and the weighted adaptive Dice loss function designed in the present application, the tendency of the neural network model to favor high-frequency category samples is reduced by increasing the attention to rare category samples, thereby making the segmentation and classification of the neural network model more accurate, thereby improving the image classification accuracy and image segmentation accuracy.

[0176] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0177] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. An image segmentation and classification method, characterized in that: The following steps are involved: Acquire a target image, wherein the target image includes a target area; The target image is segmented and classified by a pre-trained neural network model to obtain a contour map and a classification result of the target area; wherein the neural network model includes: An encoder-decoder structure is used to extract features from the target image to obtain an initial feature map and a plurality of decoded feature maps; A first segmentation module, used for obtaining a segmentation feature map according to the initial feature map; A first classification module, used for obtaining a classification feature map according to the initial feature map and each of the decoding feature maps; An information interaction module, used for performing feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features; A second segmentation module, used for obtaining a contour map of the target area according to the target segmentation features; A second classification module is used to obtain a classification result of the target area according to the target classification feature; The step of performing feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features includes: According to the segmentation feature map and the classification feature map, a first sub-classification feature, a first sub-segmentation feature and a first fusion feature are obtained; According to the segmentation feature map and the first fusion feature, a first attention feature map is obtained in combination with a multi-head attention mechanism; According to the classification feature map and the first fusion feature, a second attention feature map is obtained in combination with a multi-head attention mechanism; Obtaining a second sub-classification feature, a second sub-segmentation feature, and a second fusion feature according to the first sub-classification feature and the first sub-segmentation feature; According to the segmentation feature map and the second fusion feature, a third attention feature map is obtained in combination with a multi-head attention mechanism; According to the classification feature map and the second fusion feature, a fourth attention feature map is obtained in combination with a multi-head attention mechanism; According to the second sub-segmentation feature and the second sub-classification feature, a third sub-segmentation feature and a third fusion feature are obtained; According to the segmentation feature map and the third fusion feature, a fifth attention feature map is obtained in combination with a multi-head attention mechanism; According to the classification feature map and the third fusion feature, combined with a multi-head attention mechanism, a sixth attention feature map is obtained; Obtaining the target segmentation feature according to the first attention feature map, the third attention feature map, the fifth attention feature map and the segmentation feature map; The target classification feature is obtained according to the second attention feature map, the fourth attention feature map, the sixth attention feature map and the classification feature map.

2. The image segmentation and classification method according to claim 1, characterized in that: The step of extracting features from the target image to obtain an initial feature map and a plurality of decoded feature maps includes: Performing downsampling and encoding perception processing on the target image to obtain a first encoding feature map; Downsampling and encoding perception processing are performed on the first encoding feature map to obtain a second encoding feature map; Downsampling and encoding perception processing are performed on the second encoding feature map to obtain a third encoding feature map; Upsampling the third encoded feature map to obtain a first upsampled feature map, performing a jump connection between the first upsampled feature map and the second encoded feature map to obtain a first feature map to be perceived, and then performing decoding and perception processing on the first feature map to be perceived to obtain a first decoded feature map; Upsampling the first decoded feature map to obtain a second upsampled feature map, performing a jump connection between the second upsampled feature map and the first encoded feature map to obtain a second feature map to be perceived, and then performing decoding perception processing on the second feature map to be perceived to obtain a second decoded feature map; The second decoded feature map is upsampled to obtain the initial feature map.

3. The image segmentation and classification method according to claim 1, characterized in that: The step of obtaining a segmentation feature map according to the initial feature map comprises: Performing multiple convolution operations and multiple activation processes on the initial feature map to obtain a first processed feature map; Performing dimension transformation on the first processed feature map to obtain a two-dimensional segmentation feature map; Regularization and group normalization are performed on the two-dimensional segmentation feature map to obtain the segmentation feature map.

4. The image segmentation and classification method according to claim 1, characterized in that: The step of obtaining a classification feature map according to the initial feature map and each of the decoded feature maps comprises: Downsampling the initial feature map to obtain a first downsampled feature map, performing a jump connection between the first downsampled feature map and the second decoded feature map to obtain a first feature map to be convolved, and then performing a convolution operation on the first feature map to be convolved to obtain a second processed feature map; Downsampling the second processed feature map to obtain a second downsampled feature map, performing a jump connection between the second downsampled feature map and the first decoded feature map to obtain a second feature map to be convolved, and then performing a convolution operation on the second feature map to be convolved to obtain a third processed feature map; Downsampling the third processed feature map to obtain a third downsampled feature map; Expanding the number of channels of the third down-sampled feature map to obtain an expanded third down-sampled feature map; Performing dimension transformation on the expanded third down-sampling feature map to obtain a two-dimensional classification feature map; The two-dimensional classification feature map is activated, regularized, and group normalized to obtain the classification feature map.

5. The image segmentation and classification method according to claim 1, characterized in that: The step of obtaining a contour map of the target area according to the target segmentation features includes: Performing pooling processing on the target segmentation features to obtain the pooled target segmentation features; Performing multiple convolution operations and multiple activation processes on the pooled target segmentation features to obtain a segmentation channel weight vector; Perform channel multiplication on the segmentation channel weight vector and the target segmentation feature to obtain the channel-weighted target segmentation feature; The target segmentation features after channel weighting are subjected to multiple convolution operations and multiple activation processes to obtain a contour map of the target area.

6. The image segmentation and classification method according to claim 1, characterized in that: The obtaining the classification result of the target area according to the target classification feature includes: Performing pooling processing on the target classification features to obtain the pooled target classification features; Perform multiple convolution operations and multiple activation processes on the pooled target classification features to obtain a classification channel weight vector; Channel-multiplying the classification channel weight vector and the target classification feature to obtain the channel-weighted target classification feature; Performing pooling processing on the target classification features after channel weighting to obtain target pooling features; Compressing the number of channels of the target pooling feature to obtain a target classification vector; Probabilistic processing is performed on the target classification vector to obtain a classification result of the target area.

7. The image segmentation and classification method according to claim 1, characterized in that: The method further comprises the following steps: During the training of the neural network model, obtaining an adaptive Dice loss function and an adaptive focus loss function of the neural network model; The adaptive focus loss function and the adaptive Dice loss function are weighted to obtain the loss function of the neural network model.

8. The image segmentation and classification method according to claim 1, characterized in that: Before performing segmentation and classification processing on the target image by using the pre-trained neural network model, the method further comprises the following steps: The target image is preprocessed.

9. An image segmentation and classification device, characterized in that: include: An acquisition module, used for acquiring a target image, wherein the target image includes a target area; The image segmentation and classification module is used to perform segmentation and classification processing on the target image through a pre-trained neural network model to obtain a contour map and classification result of the target area; wherein the neural network model includes: An encoder-decoder structure is used to extract features from the target image to obtain an initial feature map and a plurality of decoded feature maps; A first segmentation module, used for obtaining a segmentation feature map according to the initial feature map; A first classification module, used for obtaining a classification feature map according to the initial feature map and each of the decoding feature maps; An information interaction module, used for performing feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features; A second segmentation module, used for obtaining a contour map of the target area according to the target segmentation features; A second classification module is used to obtain a classification result of the target area according to the target classification feature; The step of performing feature interaction between the segmentation feature map and the classification feature map to obtain target segmentation features and target classification features includes: According to the segmentation feature map and the classification feature map, a first sub-classification feature, a first sub-segmentation feature and a first fusion feature are obtained; According to the segmentation feature map and the first fusion feature, a first attention feature map is obtained in combination with a multi-head attention mechanism; According to the classification feature map and the first fusion feature, a second attention feature map is obtained in combination with a multi-head attention mechanism; Obtaining a second sub-classification feature, a second sub-segmentation feature, and a second fusion feature according to the first sub-classification feature and the first sub-segmentation feature; According to the segmentation feature map and the second fusion feature, a third attention feature map is obtained in combination with a multi-head attention mechanism; According to the classification feature map and the second fusion feature, a fourth attention feature map is obtained in combination with a multi-head attention mechanism; According to the second sub-segmentation feature and the second sub-classification feature, a third sub-segmentation feature and a third fusion feature are obtained; According to the segmentation feature map and the third fusion feature, a fifth attention feature map is obtained in combination with a multi-head attention mechanism; According to the classification feature map and the third fusion feature, combined with a multi-head attention mechanism, a sixth attention feature map is obtained; Obtaining the target segmentation feature according to the first attention feature map, the third attention feature map, the fifth attention feature map and the segmentation feature map; The target classification feature is obtained according to the second attention feature map, the fourth attention feature map, the sixth attention feature map and the classification feature map.

Images