Image processing method, training method, device, equipment and medium

Abstract

The present disclosure provides an image processing method, device, equipment, medium and program product, which relates to the field of artificial intelligence. The method includes: obtaining a first feature image based on M superpixels of the image to be segmented; processing the first feature image according to the attention mechanism to obtain first attention data, and performing Gaussian distribution processing on the first attention data to obtain second attention data; using the first attention data and the first feature image to obtain a second feature image; using the second attention data and the first feature image to obtain a third feature image; classifying each of the superpixels according to the second feature image and the third feature image; segmenting the image to be segmented according to the classification result of each superpixel to obtain the N targets to be identified. The present disclosure also provides a training method, device, equipment, medium and program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence or finance, and more specifically, to an image processing method, training method, device, equipment, medium and program product. Background Art

[0002] Image segmentation can be the process of dividing an image into several image regions. Depending on the image content or image type, the difficulty of segmenting different images varies. For example, medical images have a single background and low complexity, so they can achieve a high segmentation accuracy. Remote sensing images have a complex background, contain a rich variety of ground objects, and have a wide range of remote sensing spectra, so the segmentation accuracy may be low. Therefore, how to provide an improved image processing method to improve the segmentation effect of images containing rich information has become a problem that needs to be solved urgently. Summary of the invention

[0003] In view of the above problems, the present disclosure provides an image processing method, a model training method, an apparatus, a device, a medium and a program product for improving segmentation effect.

[0004] One aspect of an embodiment of the present disclosure provides an image processing method, including: obtaining a first feature image based on M superpixels of an image to be segmented, wherein the image to be segmented includes N targets to be identified, and N and M are integers greater than or equal to 1 respectively; processing the first feature image according to an attention mechanism to obtain first attention data, and performing Gaussian distribution processing on the first attention data to obtain second attention data, wherein the value of the second attention data obeys a Gaussian distribution; obtaining a second feature image using the first attention data and the first feature image; obtaining a third feature image using the second attention data and the first feature image; classifying each of the superpixels according to the second feature image and the third feature image; and segmenting the image to be segmented according to the classification result of each superpixel to obtain the N targets to be identified.

[0005] According to an embodiment of the present disclosure, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, the first attention data includes channel attention data and spatial attention data, and the processing of the first feature image according to the attention mechanism to obtain the first attention data includes: processing the first feature image according to the channel attention mechanism to obtain the channel attention data; processing the first feature image according to the spatial attention mechanism to obtain the spatial attention data.

[0006] According to an embodiment of the present disclosure, the method of obtaining the second feature image using the first attention data and the first feature image includes: obtaining the fourth feature image using the channel attention data and the first feature image; obtaining the fifth feature image using the spatial attention data and the first feature image; and obtaining the second feature image based on the fourth feature image and the fifth feature image.

[0007] According to an embodiment of the present disclosure, the second attention data includes modulated channel attention data and modulated spatial attention data, and the Gaussian distribution processing of the first attention data to obtain the second attention data includes: modulating the channel attention data based on a Gaussian function to obtain the modulated channel attention data, wherein the value of the modulated channel attention data obeys a Gaussian distribution; modulating the spatial attention data based on a Gaussian function to obtain the modulated spatial attention data, wherein the value of the modulated spatial attention data obeys a Gaussian distribution.

[0008] According to an embodiment of the present disclosure, the method of obtaining the third feature image using the second attention data and the first feature image includes: obtaining the sixth feature image using the modulated channel attention data and the first feature image; obtaining the seventh feature image using the modulated space attention data and the first feature image; and obtaining the third feature image based on the sixth feature image and the seventh feature image.

[0009] According to an embodiment of the present disclosure, classifying each of the superpixels based on the second feature image and the third feature image includes: obtaining an eighth feature image based on the second feature image and the third feature image; using the eighth feature image as an input of a convolutional neural network to obtain a first feature matrix; and using a graph convolutional neural network to process the first feature matrix to classify each of the superpixels.

[0010] According to an embodiment of the present disclosure, after obtaining the first feature matrix, it includes: using a graph convolutional attention mechanism to process the first feature matrix; wherein, using a graph convolutional neural network to process the first feature matrix and classifying each of the superpixels includes: using the processed first feature matrix as an input of the graph convolutional neural network to obtain a classification result for each of the superpixels.

[0011] According to an embodiment of the present disclosure, the method also includes constructing an undirected graph model based on M superpixels, wherein the M superpixels are M vertices of the undirected graph model, specifically including: setting the initial value of i to 1, and looping through the following operations, wherein i is an integer greater than or equal to 1, and i is less than or equal to M: determining at least one superpixel adjacent to the ith superpixel; calculating the Euclidean distance between the ith superpixel and the first adjacent superpixel in the RGB channel, wherein the first adjacent superpixel is any superpixel adjacent to the ith superpixel; when the Euclidean distance is less than or equal to a preset threshold, constructing an edge between the ith superpixel and the first adjacent superpixel; and adding 1 to the value of i.

[0012] Another aspect of the disclosed embodiment provides an image processing model training method, wherein the image processing model is used to implement the image processing method as described above, the image processing model includes an attention layer and a graph convolutional neural network layer, the method includes: obtaining a first training feature image based on m superpixels of a training sample image, wherein the training sample image includes n targets to be identified, and n and m are integers greater than or equal to 1; using the attention layer to process the first training feature image to obtain first training attention data, and performing Gaussian distribution processing on the first training attention data to obtain second training attention data, wherein the second training attention data The value obeys Gaussian distribution; the second training feature image is obtained by using the first training attention data and the first training feature image; the third training feature image is obtained by using the second training attention data and the first training feature image; each of the superpixels is classified by using the graph convolutional neural network layer according to the second training feature image and the third training feature image; the training sample image is segmented according to the classification result of each superpixel to obtain the predicted segmentation results of the n targets to be identified; the attention layer and the graph convolutional neural network layer are trained based on the pre-labeled category labels of the n targets to be identified and the contrast loss of the predicted segmentation results.

[0013] Another aspect of an embodiment of the present disclosure provides an image processing device, including: a first acquisition module, used to obtain a first feature image based on M superpixels of an image to be segmented, wherein the image to be segmented includes N targets to be identified, and N and M are integers greater than or equal to 1 respectively; a first processing module, used to process the first feature image according to an attention mechanism to obtain first attention data, and to perform Gaussian distribution processing on the first attention data to obtain second attention data, wherein the value of the second attention data obeys a Gaussian distribution; a first calculation module, used to obtain a second feature image using the first attention data and the first feature image; a second calculation module, used to obtain a third feature image using the second attention data and the first feature image; a first classification module, used to classify each of the superpixels according to the second feature image and the third feature image; a first segmentation module, used to segment the image to be segmented according to the classification result of each superpixel to obtain the N targets to be identified.

[0014] Another aspect of the disclosed embodiment provides an image processing model training device, wherein the image processing model includes an attention layer and a graph convolutional neural network layer, and the device includes: a second acquisition module, used to obtain a first training feature image based on m superpixels of a training sample image, wherein the training sample image includes n targets to be identified, and n and m are integers greater than or equal to 1; a second processing module, used to use the attention layer to process the first training feature image to obtain first training attention data, and to perform Gaussian distribution processing on the first training attention data to obtain second training attention data, wherein the value of the second training attention data obeys Gaussian distribution; a third calculation module, used to use the first training attention layer to obtain a first training feature image; The fourth calculation module is used to use the second training attention data and the first training feature image to obtain the third training feature image; the second classification module is used to classify each of the superpixels according to the second training feature image and the third training feature image using the graph convolutional neural network layer; the second segmentation module is used to segment the training sample image according to the classification result of each superpixel to obtain the predicted segmentation results of the n targets to be identified; the model training module is used to train the attention layer and the graph convolutional neural network layer based on the pre-labeled category labels of the n targets to be identified and the contrast loss of the predicted segmentation results.

[0015] Another aspect of an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method as described above.

[0016] Another aspect of the embodiments of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method as described above.

[0017] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0018] One or more of the above embodiments have the following beneficial effects:

[0019] 1) In order to prevent the first attention data from concentrating the attention weight on the important features in the first feature image, the probability distribution of the attention weight is re-determined by performing Gaussian distribution processing on the first attention data, so that the second attention data can be used to suppress the background features and important features. Therefore, the second attention data plays a role of feature compensation based on the single first attention data. The combination of the two can set a reasonable attention weight for the first feature image, improve the classification accuracy of each superpixel, and improve the segmentation effect of the segmented image.

[0020] 2) By constructing an image processing model including an attention layer and a graph convolutional neural network layer, the first attention data and the second attention data output by the attention layer are used to process the first training feature image respectively during training, and the classification result of each superpixel is obtained according to the graph convolutional neural network layer to perform model training. Among them, the second training attention data plays a role of feature compensation on the basis of the single first training attention data. The combination of the two can set a reasonable attention weight for the first training feature image, so that the trained image processing model has the function of setting a reasonable attention weight, thereby having better classification accuracy and improving the segmentation effect of the image to be segmented. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 A diagram schematically shows an application scenario of an image processing method or a model training method according to an embodiment of the present disclosure;

[0023] Figure 2 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown;

[0024] Figure 3 A flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown;

[0025] Figure 4 Schematically shows a flowchart of constructing an undirected graph model according to an embodiment of the present disclosure;

[0026] Figure 5 Schematically shows a flow chart of obtaining first attention data according to an embodiment of the present disclosure;

[0027] Figure 6a A schematic diagram of obtaining channel attention data according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 6b A schematic diagram of obtaining spatial attention data according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 6c A schematic diagram of obtaining a second characteristic image according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 7 Schematically shows a flow chart of obtaining a second feature image according to an embodiment of the present disclosure;

[0031] Figure 8 Schematically shows a flow chart of obtaining second attention data according to an embodiment of the present disclosure;

[0032] Figure 9a A schematic diagram of obtaining modulated channel attention data according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 9b A schematic diagram of obtaining modulated spatial attention data according to an embodiment of the present disclosure is schematically shown;

[0034] Fig.9c A schematic diagram of obtaining a third characteristic image according to an embodiment of the present disclosure is schematically shown;

[0035] Fig.10 Schematically shows a flow chart of obtaining a third characteristic image according to an embodiment of the present disclosure;

[0036] Fig.11 A flowchart for classifying each superpixel according to an embodiment of the present disclosure is schematically shown;

[0037] Fig.12 A flowchart for classifying each superpixel according to another embodiment of the present disclosure is schematically shown;

[0038] Fig.13 A flowchart of an image processing model training method according to an embodiment of the present disclosure is schematically shown;

[0039] Fig.14A flowchart of an image processing model training method according to another embodiment of the present disclosure is schematically shown;

[0040] Fig.15 The structure block diagram of the image processing device according to the embodiment of the present disclosure is schematically shown;

[0041] Fig.16 The structure block diagram of the image processing model training device according to the embodiment of the present disclosure is schematically shown;

[0042] Fig.17 A block diagram of an electronic device suitable for implementing an image processing method or a model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0043] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0044] In order to facilitate understanding of the technical solutions of the embodiments of the present disclosure, some technical terms involved are first introduced below.

[0045] Images to be segmented: They can be remote sensing images regularly taken and uploaded by professional satellite companies using satellites, which have the characteristics of large memory usage, high spectrum, high resolution, etc. They can also be other images taken by mobile phones, cameras and other devices.

[0046] Superpixel: A sub-region composed of a series of adjacent pixels in the image to be segmented with similar features such as color, brightness, texture, etc. Most of these sub-regions retain effective information for further image segmentation and generally do not destroy the boundary information of objects in the image.

[0047] Attention mechanism: A set of weight coefficients (i.e., attention data) are learned autonomously through the neural network, and the important information is strengthened while suppressing the unimportant information by dynamically weighting the feature map, so that the model can focus on the necessary part of the input information at each step.

[0048] Channel attention mechanism: Add an attention mechanism between different channels of the neural network, learn the channel feature weights based on global information, increase the weights of effective channel features, reduce the weights of invalid or ineffective channel features, and improve the model's sensitivity to channel features.

[0049] Spatial attention mechanism: Add an attention mechanism in the spatial domain of the neural network, dynamically allocate spatial attention weights according to the importance of different features in spatial positions, enhance useful features, and suppress minor features.

[0050] Modulation processing: mapping the original attention data to a new probability distribution. In some embodiments of the present disclosure, a Gaussian function may be used to map the original attention data to a Gaussian distribution.

[0051] Graph convolution attention mechanism: Similar to the attention mechanism, its function is to dynamically assign weight coefficients to different vertices in an undirected graph.

[0052] Undirected graph model: In the disclosed embodiment, each superpixel corresponds to a vertex, and the edges between the vertices have no direction.

[0053] It should be noted that the methods and devices determined by the image processing methods, model training methods, devices, equipment, media and program products disclosed in the present invention can be used in the relevant aspects of image processing in the field of artificial intelligence or finance, and can also be used in any field other than the field of artificial intelligence or finance. The application fields of the image processing methods, model training methods, devices, equipment, media and program products disclosed in the present invention are not limited.

[0054] Figure 1 An application scenario diagram of an image processing method or a model training method according to an embodiment of the present disclosure is schematically shown.

[0055] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0056] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0057] The terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. The terminal devices 101, 102, 103 may have cameras to capture images, or be connected to the Internet to obtain images.

[0058] The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101, 102, and 103 (for example only). The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device. The server 105 obtains images from the terminal devices 101, 102, and 103, and may also connect to the Internet to obtain images.

[0059] It should be noted that the image processing method or model training method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image processing device or model training device provided in the embodiment of the present disclosure can generally be set in the server 105.

[0060] The image processing method or model training method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image processing device or model training device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0062] The following will be based on Figure 1 The scene described by Figure 2~Figure 13 The image processing method and model training method of the embodiments of the present disclosure are described in detail.

[0063] Figure 2 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown.

[0064] like Figure 2 As shown, the image processing method of this embodiment includes operations S210 to S260.

[0065] In operation S210, a first feature image is obtained based on M superpixels of an image to be segmented, wherein the image to be segmented includes N objects to be identified, and N and M are integers greater than or equal to 1, respectively.

[0066] In some embodiments, a simple linear iterative clustering algorithm can be used to over-segment the image to be segmented to obtain M superpixels. Using the over-segmentation method to process the image can improve the segmentation efficiency and obtain superpixels of different sizes and shapes. These superpixels usually have regional consistency, fully retain neighborhood information, and can reduce computational complexity.

[0067] In operation S220, the first feature image is processed according to the attention mechanism to obtain first attention data, and the first attention data is subjected to Gaussian distribution processing to obtain second attention data, wherein the value of the second attention data obeys Gaussian distribution.

[0068] For example, the first attention data can perform weighted processing on the first feature image, thereby using different weight coefficients to enhance important information. After mapping the weight coefficients in the first attention data to the Gaussian distribution, the weight coefficients corresponding to the background features and important features are reduced, and the weight coefficients corresponding to the less important features are increased.

[0069] According to an embodiment of the present disclosure, performing Gaussian distribution processing on the first attention data includes: performing modulation processing on the first attention data based on a Gaussian function to obtain second attention data. Thus, the first attention data can be mapped into a Gaussian distribution using a Gaussian function to redistribute attention to the feature image.

[0070] For example, the attention mechanism originates from the study of human vision. In order to make rational use of limited visual information processing resources, it is necessary to select a specific part of the visual area and then pay attention to it. In computer vision, the attention mechanism has two main aspects: deciding which part of the input needs to be paid attention to and allocating limited information processing resources to the important part.

[0071] The attention mechanism determines the image information that needs to be paid attention to from the first feature image, and further allocates attention coefficients to achieve different degrees of attention. The role of the second attention data is to compensate for the coefficient allocation in the first attention data. For example, the first attention data may allocate greater attention to the important features determined by the attention mechanism, while ignoring the less important features, and easily fall into overfitting or local optimal states. Therefore, since the second attention data obeys a Gaussian distribution, the weight coefficients corresponding to the background and important features in the first attention data are suppressed, which can highlight the less important features to prevent them from being ignored.

[0072] For example, the important feature may be a feature with a larger weight coefficient value in the first attention data, and the less important feature may be a feature with a smaller weight coefficient value in the first attention data. From the perspective of visualization of the image to be segmented, the important feature may be a target area to be identified, or one of the color features, texture features, etc. of the image.

[0073] In operation S230, a second feature image is obtained using the first attention data and the first feature image.

[0074] In operation S240, a third feature image is obtained using the second attention data and the first feature image.

[0075] In operation S250, each of the superpixels is classified according to the second feature image and the third feature image.

[0076] In operation S260, the image to be segmented is segmented according to the classification result of each superpixel to obtain the N objects to be identified.

[0077] For example, the N targets to be identified in the image to be segmented include cars, buildings and roads, and if the classification result of each superpixel is any one of cars, buildings and roads, the superpixels with the same classification results can be classified as the same target to be identified, and then image segmentation is performed.

[0078] The inventors found in the process of implementing the inventive concept of the disclosed embodiment that in the related art, when segmenting an image, all channels and features may be sampled uniformly. For images such as remote sensing images that contain rich image information, this results in the same degree of attention to all image information, without distinction of primary and secondary, thus wasting computing resources and affecting the segmentation effect.

[0079] According to the embodiments of the present disclosure, in order to prevent the first attention data from concentrating the attention weight on the important features in the first feature image, the probability distribution of the attention weight is re-determined by performing Gaussian distribution processing on the first attention data, so that the second attention data can be used to achieve the effect of suppressing background features and important features. Therefore, the second attention data plays a role of feature compensation on the basis of the single first attention data. The combination of the two can set a reasonable attention weight for the first feature image, improve the classification accuracy of each superpixel, and improve the segmentation effect of the segmented image.

[0080] exist Figure 2 Based on the following Figure 3 The image processing flow of an embodiment of the present disclosure is described as a whole. Figure 3 Combining some of the steps in Figure 4~Figure 11Explain them separately.

[0081] Figure 3 The flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown.

[0082] like Figure 3 As shown, the image processing method of this embodiment includes operations S301 to S319.

[0083] In operation S301, an image to be segmented is acquired.

[0084] In operation S302, M superpixels are obtained based on the image to be segmented, for example, using a simple linear iterative clustering algorithm.

[0085] In operation S303, an undirected graph model is constructed.

[0086] Figure 4 A flowchart for constructing an undirected graph model according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 4 As shown, let the initial value of i be 1, i be an integer greater than or equal to 1, i be less than or equal to M, and execute operations S410 to S450 in a loop.

[0088] In operation S410, at least one superpixel adjacent to an i-th superpixel is determined.

[0089] Exemplarily, the superpixels adjacent to each superpixel may be determined by the position coordinates of the superpixel on the image to be segmented.

[0090] In operation S420, the Euclidean distance between the ith superpixel and the first adjacent superpixel in the RGB channel is calculated, wherein the first adjacent superpixel is any superpixel adjacent to the ith superpixel. The Euclidean distance between the ith superpixel and each adjacent superpixel in the RGB channel can be calculated, as shown in formula (1).

[0091] (1)

[0092] Among them, W ij is the Euclidean distance between the i-th superpixel and the j-th superpixel, r i , g i , b i are the average values ​​of the RGB channels of each pixel in the i-th superpixel, r j , g j , b j are the average values ​​of the RGB channels of each pixel in the j-th superpixel.

[0093] In operation S430, when the Euclidean distance is less than or equal to a preset threshold, an edge between the i-th superpixel and the first superpixel is constructed.

[0094] In operation S440, it is determined whether i is less than M. If so, operation S450 is performed. If not, the process ends.

[0095] In operation S450, the value of i is incremented by 1, and operation S410 is performed.

[0096] According to an embodiment of the present disclosure, a superpixel point is modeled as an undirected graph model G(V, E), where each superpixel corresponds to a vertex, wherein V and E represent the set of all vertices and edges, respectively.

[0097] The above-mentioned edge construction is mainly based on whether the Euclidean distance between the superpixel and the adjacent superpixel color is lower than a preset threshold. If it is lower than the preset threshold, it can be considered that the two superpixels are similar and meet the conditions for constructing an edge. This can make the undirected graph model more accurately reflect the relationship between each superpixel.

[0098] In operation S304, a first feature image is obtained.

[0099] Exemplarily, in order to extract the features of irregular superpixels, each superpixel is placed in a Patch block. The centroid of the superpixel is located at the center of the Patch block. Pixels in the Patch block that do not belong to the superpixel area are set to zero. The first feature image can be obtained directly based on each Patch block data, or it can be obtained by further processing based on each Patch block data.

[0100] Figure 5 The flowchart of obtaining the first attention data according to an embodiment of the present disclosure is schematically shown. The attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the first attention data includes channel attention data and spatial attention data.

[0101] like Figure 5 As shown, processing the first feature image according to the attention mechanism to obtain the first attention data in operation S220 may include operations S510 to S520.

[0102] In operation S510, the first feature image is processed according to the channel attention mechanism to obtain the channel attention data.

[0103] Exemplarily, the first feature image may be input into a channel attention layer (refer to operation S305 ) to obtain channel attention data (refer to operation S307 ).

[0104] Figure 6aA schematic diagram of obtaining channel attention data according to an embodiment of the present disclosure is schematically shown.

[0105] Reference Figure 6a , the channel attention layer can include the global average pooling layer (Avgpool), the maximum pooling layer (Maxpool), and the shared network (Shared MLP). When constructing the attention mechanism, the maximum pooling layer and the average pooling layer are introduced at the same time, which can introduce more feature information to enhance the diversity of features when obtaining the attention weight.

[0106] First, global average pooling and maximum pooling operations are used to aggregate the spatial information of the first feature image (F) to generate two different spatial context descriptors: and and They represent the global average pooling feature and the maximum pooling feature respectively. Then these two descriptors are input into a shared network respectively. The shared network is composed of a multi-layer perceptron (MLP). Finally, the two descriptors after the shared network are merged by summing each element to obtain the channel attention data (Channel Attention) Channel Attention Data The calculation formula is shown in formula (2):

[0107] (2)

[0108] in, Represents the activation function, such as the sigmoid function, the weight of the MLP and The two descriptors are shared.

[0109] In operation S520, the first feature image is processed according to the spatial attention mechanism to obtain the spatial attention data.

[0110] Exemplarily, the first feature image may be input into a spatial attention layer (refer to operation S306 ) to obtain spatial attention data (refer to operation S308 ).

[0111] Exemplarily, according to operation S510 to operation S520, the first attention data is obtained from the two dimensions of channel domain and spatial domain complementarily. The improved spatial and channel attention modules are connected in parallel to calibrate the input feature image from the two dimensions of space and channel. When the feature image obtains a higher weight in both space and channel correction, a higher weight can be obtained.

[0112] Figure 6b A schematic diagram of obtaining spatial attention data according to an embodiment of the present disclosure is schematically shown.

[0113] Reference Figure 6b , the spatial attention layer can include a global average pooling layer (Avgpool), a maximum pooling layer (Maxpool), and a convolution layer (convlayer). In order to introduce more information in the spatial attention layer to make up for the shortcomings of insufficient feature information, the maximum pooling and global average pooling are first used along the channel axis and linked to generate a valid feature map. After the two feature descriptors are connected, a convolution layer is applied to generate spatial attention data (Spatialattention): .

[0114] The above uses maximum pooling and global average pooling along the channel axis to generate two two-dimensional maps: and , which represent the average pooling feature and the maximum pooling feature of the channel dimension respectively. Then these two feature images are concatenated and convolved through a standard convolution layer to produce a two-dimensional spatial attention map The calculation of spatial attention data can be expressed as formula (3):

[0115] (3)

[0116] in It represents a convolution operation with a kernel size of 3x3 (just an example), which plays a role in dimensionality reduction. Represents an activation function, such as the sigmoid function.

[0117] Figure 6c A schematic diagram of obtaining a second characteristic image according to an embodiment of the present disclosure is schematically shown. Figure 7 The flowchart of obtaining the second feature image according to an embodiment of the present disclosure is schematically shown.

[0118] like Figure 7 As shown, obtaining the second characteristic image in this embodiment includes operations S710 to S730.

[0119] In operation S710, a fourth feature image is obtained using the channel attention data and the first feature image.

[0120] Exemplarily, the channel attention data is multiplied by the first feature image to obtain a fourth feature image. Figure 6c , the first feature image F is multiplied by the channel attention data to obtain the fourth feature image U c .

[0121] In operation S720, a fifth feature image is obtained using the spatial attention data and the first feature image.

[0122] Exemplarily, the spatial attention data is multiplied by the first feature image to obtain the fifth feature image. Figure 6c , the first feature image F is multiplied by the spatial attention data to obtain the fifth feature image U S .

[0123] In operation S730, the second feature image is obtained according to the fourth feature image and the fifth feature image.

[0124] Exemplarily, the fourth feature image and the fifth feature image are added to obtain the second feature image (refer to operation S313). Figure 6c , the second feature image U c With the third feature image U S Add together to get the second feature image U sc .

[0125] According to an embodiment of the present disclosure, the second feature image is obtained based on the first attention data and the first feature image, which represents that a higher degree of attention is allocated to the key area in the image to be segmented.

[0126] Figure 8 The flowchart of obtaining the second attention data according to an embodiment of the present disclosure is schematically shown.

[0127] like Figure 8 As shown, performing Gaussian distribution processing on the first attention data to obtain the second attention data in operation S220 includes operation S810 to operation S820. The second attention data includes modulated channel attention data and modulated spatial attention data (refer to operation S312).

[0128] In operation S810, the channel attention data is modulated based on a Gaussian function to obtain the modulated channel attention data (refer to operation S310), wherein the value of the modulated channel attention data obeys a Gaussian distribution.

[0129] Figure 9a A schematic diagram of obtaining modulated channel attention data according to an embodiment of the present disclosure is schematically shown.

[0130] Reference Figure 9a , after obtaining channel attention data After that, it is modulated based on the Gaussian function, and the channel attention data The calculation method is shown in formula (4):

[0131] (4)

[0132] in, represents the Gaussian function. Expressed in matrix form, in the process of calculating formula (4), The calculation methods of the mean μ and standard deviation σ are shown in formula (5) and formula (6):

[0133] (5)

[0134] (6)

[0135] in, There are R elements in total. for The rth element in .

[0136] In operation S820, the spatial attention data is modulated based on a Gaussian function to obtain the modulated spatial attention data (refer to operation S311), wherein the value of the modulated spatial attention data obeys a Gaussian distribution.

[0137] Figure 9b A schematic diagram of obtaining modulated spatial attention data according to an embodiment of the present disclosure is schematically shown.

[0138] Reference Figure 9b , after obtaining spatial attention data Afterwards, it is modulated based on the Gaussian function, and the spatial attention data The calculation method is shown in formula (7):

[0139] (7)

[0140] in, represents the Gaussian function. Expressed in matrix form, in the process of calculating formula (8), The calculation methods of the mean μ and standard deviation σ are shown in formula (8) and formula (9):

[0141] (8)

[0142] (9)

[0143] in, There are R elements in total. for The rth element in .

[0144] According to the embodiments of the present disclosure, the Gaussian function can remap the matrix, for example, the channel attention data and the spatial attention data can be further mapped to obey the Gaussian distribution, thereby redistributing the attention, suppressing the background features and important features, and highlighting the less important features. It can play a role in feature compensation for the single first attention data.

[0145] Fig.9c A schematic diagram of obtaining a third characteristic image according to an embodiment of the present disclosure is schematically shown. Fig.10 The flowchart of obtaining the third characteristic image according to the embodiment of the present disclosure is schematically shown.

[0146] like Fig.10 As shown, obtaining the third feature image using the second attention data and the first feature image in operation S240 includes operations S1010 to S1030.

[0147] In operation S1010, a sixth feature image is obtained using the modulated channel attention data and the first feature image.

[0148] Exemplarily, the modulated channel attention data is multiplied by the first feature image to obtain the sixth feature image. Fig.9c , the first feature image F is multiplied by the modulated channel attention data to obtain the sixth feature image U' c .

[0149] In operation S1020, a seventh feature image is obtained using the modulated spatial attention data and the first feature image.

[0150] Exemplarily, the modulated spatial attention data is multiplied by the first feature image to obtain the seventh feature image. Fig.9c , the first feature image F is multiplied by the modulated spatial attention data to obtain the seventh feature image U' S .

[0151] In operation S1030, the third characteristic image is obtained according to the sixth characteristic image and the seventh characteristic image (refer to operation S314).

[0152] Exemplarily, the sixth characteristic image and the seventh characteristic image are added to obtain the seventh characteristic image. Fig.9c , the sixth feature image U' c With the seventh characteristic image U' S Add together to get the third feature image U' sc .

[0153] According to the embodiments of the present disclosure, the spatial attention layer and the channel attention layer are connected in parallel and a modulation step is set, so that the less important areas that are easily ignored can be mined, and the accuracy of image segmentation can be improved. The third feature image can highlight other features that are easily ignored in the second feature image.

[0154] Fig.11 The flowchart of classifying each superpixel in operation S230 according to an embodiment of the present disclosure is schematically shown.

[0155] like Fig.11 As shown, in operation S250, classifying each of the superpixels according to the second feature image and the third feature image includes operations S1110 to S1130.

[0156] In operation S1110, an eighth feature image is obtained according to the second feature image and the third feature image.

[0157] Exemplarily, referring to operation S315, the second characteristic image and the third characteristic image may be added to obtain an eighth characteristic image.

[0158] In operation S1120, the eighth feature image is used as an input of a convolutional neural network to obtain a first feature matrix.

[0159] Exemplarily, referring to operation S316, a convolutional neural network structure is implemented, for example, using a convolutional neural network layer, which may include a convolutional layer, a fully connected layer, and a SoftMax layer.

[0160] In operation S1130, the first feature matrix is ​​processed using a graph convolutional neural network to classify each of the superpixels.

[0161] Exemplarily, referring to operation S318, the first feature matrix is ​​input into the graph convolutional neural network layer for processing using the graph convolutional neural network. The convolutional neural network is an extremely effective architecture in utilizing the translation invariance of the signal, so the output of the convolutional neural network can be used to obtain the features of the superpixel (i.e., the first feature matrix), and then the graph convolutional neural network is introduced to predict the classification of the superpixel. Fig.12 A flowchart for classifying each superpixel according to another embodiment of the present disclosure is schematically shown.

[0162] like Fig.12 As shown, this embodiment classifies each superpixel including operations S1210 to S1220.

[0163] In operation S1210, the first feature matrix is ​​processed using a graph convolutional attention mechanism. Referring to operation S317, the undirected graph model may be input into the graph convolutional attention layer. Referring to Figure 3 By processing the first feature matrix and the undirected graph model in this layer, the attention coefficient between each vertex can be obtained. It can be expressed as formula (10):

[0164] (10)

[0165] in, It is an attention-related training parameter, and a certain value can be obtained after the training is completed. and are two vertices that have a common edge. It refers to the importance of vertex j to vertex i in the segmentation task.

[0166] If vertex j is more important to vertex i, then the attention coefficient The larger the value of The SoftMax function is then used to normalize the attention coefficient of each node, as shown in Equation 11.

[0167] (11)

[0168] in, Refers to all adjacent vertices of vertex i in the graph, LeakyReLU( ) refers to the activation function, such as LeakyReLU, is the normalized attention coefficient.

[0169] After normalization, we have , making it easier to compare coefficients between different vertices. Assume Represents the attention coefficient vector of vertex i. If vertex j and vertex i are not adjacent, there is Then the coefficient vectors of all vertices form the attention coefficient matrix .

[0170] In operation S1220, the processed first feature matrix is ​​used as an input of the graph convolutional neural network to obtain a classification result of each of the superpixels.

[0171] For example, the graph convolutional neural network can be expressed as formula (12):

[0172] (12)

[0173] in is the input of the first convolution layer, is the trainable weight matrix, It is called the generalized graph Laplacian operator, which contains the structural information of the graph. In order to introduce the attention mechanism in the graph convolution, the following formula (13) is obtained:

[0174] (13)

[0175] Where * refers to the Hadamard product, which means the multiplication of elements in corresponding positions in two matrices. Updated generalized graph Laplace The elements of not only represent the structural information of the graph model (which points are connected to each other), but also the relationship between two adjacent vertices in the feature space.

[0176] According to an embodiment of the present disclosure, a two-layer graph convolution operation may be set, and the input graph structure signal is propagated layer by layer according to the rule of the following formula (14):

[0177] (14)

[0178] in For the Trainable weight matrix for layer graph convolution. Represents an activation function. Refers to the The input signal of the layer graph convolution.

[0179] According to an embodiment of the present disclosure, the output of the second graph convolution may be classified using a SoftMax function to output a classification result of each superpixel. In operation S319, the image is segmented according to the classification result of each superpixel.

[0180] According to an embodiment of the present disclosure, a graph convolutional attention layer is introduced to further assign different attention coefficients to the feature image extracted by the convolutional neural network and the undirected graph model, so that the graph convolutional neural network can focus on important features and improve the accuracy of the segmentation results.

[0181] It should be noted that the above-mentioned convolutional neural network layer, graph convolutional attention layer and graph convolutional neural network layer represent the names of the model structure, which are used to realize the functions of convolutional neural network, graph convolutional attention mechanism and graph convolutional neural network respectively.

[0182] According to an embodiment of the present disclosure, the above Figure 2 to Figure 12 The image processing method introduced can be implemented through the image processing model. Before using the image processing model for image segmentation, training can be performed to obtain a model with better segmentation effect. Fig.13 Further introduction.

[0183] Fig.13 A flowchart of an image processing model training method according to an embodiment of the present disclosure is schematically shown.

[0184] like Fig.13 As shown, the image processing model training method of this embodiment includes operations S1310 to S1370. The trained image processing model can be used to implement Figure 2 to Figure 12 image processing method.

[0185] In operation S1310, a first training feature image is obtained based on m superpixels of a training sample image, wherein the training sample image includes n targets to be identified, and n and m are integers greater than or equal to 1, respectively.

[0186] Exemplarily, a simple linear iterative clustering algorithm may be used to over-segment the image to be segmented to obtain m super-pixels.

[0187] In operation S1320, the first training feature image is processed using the attention layer to obtain first training attention data, and the first training attention data is subjected to Gaussian distribution processing to obtain second training attention data, wherein the value of the second training attention data obeys Gaussian distribution.

[0188] Exemplarily, the method of obtaining the first training attention data and the second training attention data is the same as the method of obtaining the first attention data and the second attention data, which is not described in detail here. The difference is that the first attention data and the second attention data are obtained using the trained attention layer.

[0189] In operation S1330, a second training feature image is obtained using the first training attention data and the first training feature image.

[0190] In operation S1340, a third training feature image is obtained using the second training attention data and the first training feature image.

[0191] In operation S1350, each of the superpixels is classified using the graph convolutional neural network layer according to the second training feature image and the third training feature image.

[0192] In operation S1360, the training sample image is segmented according to the classification result of each superpixel to obtain predicted segmentation results of the n objects to be identified.

[0193] In operation S1370, based on the pre-labeled category labels of the n targets to be identified and the contrast loss of the predicted segmentation results, the attention layer and the graph convolutional neural network layer are trained, that is, the relevant parameters of the attention layer and the graph convolutional neural network layer are updated.

[0194] According to an embodiment of the present disclosure, an image processing model including an attention layer and a graph convolutional neural network layer is constructed, and the first attention data and the second attention data output by the attention layer are used to process the first training feature image respectively during training, and the classification result of each superpixel is obtained according to the graph convolutional neural network layer to perform model training. Among them, the second training attention data plays a role of feature compensation on the basis of the single first training attention data, and the combination of the two can set a reasonable attention weight for the first training feature image, so that the trained image processing model has the function of setting a reasonable attention weight, thereby having better classification accuracy and improving the segmentation effect of the image to be segmented.

[0195] Fig.14 A flowchart of an image processing model training method according to another embodiment of the present disclosure is schematically shown.

[0196] like Fig.14 As shown, the model training method of this embodiment includes operations S1401 to S1412.

[0197] In operation S1401, S training samples are obtained, and reference may be made to operation S1310.

[0198] In operation S1402, M superpixels are obtained based on the image to be segmented, for example, using a simple linear iterative clustering algorithm.

[0199] In operation S1403, an undirected graph model is constructed.

[0200] In operation S1404, a majority voting strategy is used to obtain labels of superpixels.

[0201] For example, pixel-level labels are provided in the training set, but there are no superpixel labels. Although the simple linear iterative clustering algorithm takes into account various text features and boundary information during the over-segmentation process, it is still possible that different types of pixels are included in a superpixel. Therefore, a majority voting strategy can be used to obtain superpixel labels, such as formula (15).

[0202] (15)

[0203] Where D represents the number of pixels in the superpixel. It represents the label value of the d-th pixel, sign(*) is the sign function sign(true) = 1, sign(false) = 0, q refers to the type of label in the data set, and there are K types of labels in total.

[0204] In operation S1405, a first training feature image is obtained.

[0205] In operation S1406, the first training feature image is input into the attention layer. The process of processing the first training feature image in the attention layer may refer to operations S305 to S315.

[0206] According to an embodiment of the present disclosure, the image processing model further includes a convolutional neural network layer. In operation S1407, the processed first training feature image is used as an input of the convolutional neural network layer. Wherein, using the graph convolutional neural network layer to classify each superpixel includes: based on the output of the convolutional neural network layer, such as the input second feature matrix, using the graph convolutional neural network layer to perform a classification operation.

[0207] According to an embodiment of the present disclosure, the image processing model also includes a graph convolutional attention layer. In operation S1408, the output of the convolutional neural network layer (the second feature matrix) is used as the input of the graph convolutional attention layer, and the undirected graph model is used as the input of the graph convolutional attention layer.

[0208] In operation S1409, the output of the graph convolutional attention layer is used as the input of the graph convolutional neural network layer for classification. Specifically, the classification result of each superpixel can be obtained through the SoftMax function based on the output of the graph convolutional neural network layer.

[0209] In operation S1410, a loss function (such as a cross entropy function) is used to calculate a loss according to the classification result of each superpixel in the S training samples and the corresponding category label.

[0210] In operation S1411, the parameters of at least one of the attention layer, the convolutional neural network layer, the graph convolutional attention layer, and the graph convolutional neural network are updated using a stochastic gradient descent method.

[0211] In operation S1412, if the maximum number of iterations is reached or the loss function converges during the training process, the training is completed and the trained image processing model can be obtained. The trained image processing model can also be verified by a validation set. If the classification effect meets the requirements, it is used. If the classification effect does not meet the requirements, it is trained again.

[0212] Based on the above image processing method and model training method, the present disclosure also provides an image processing device and an image processing model training device. Fig.15 and Fig.16 Describe in detail.

[0213] Fig.15 The structure block diagram of the image processing device according to the embodiment of the present disclosure is schematically shown.

[0214] like Fig.15As shown, the image processing device 1500 of this embodiment may include a first obtaining module 1510 , a first processing module 1520 , a first calculating module 1530 , a second calculating module 1540 , a first classification module 1550 , and a first segmentation module 1560 .

[0215] The first obtaining module 1510 may perform operation S210 to obtain a first feature image based on M superpixels of the image to be segmented, wherein the image to be segmented includes N targets to be identified, and N and M are integers greater than or equal to 1, respectively.

[0216] The first processing module 1520 can execute operation S220, which is used to process the first feature image according to the attention mechanism to obtain first attention data, and perform Gaussian distribution processing on the first attention data to obtain second attention data, wherein the value of the second attention data obeys Gaussian distribution.

[0217] According to an embodiment of the present disclosure, the first processing module 1520 may further perform operations S510 to S520 and operations S810 to S820, which are not described in detail herein.

[0218] The first calculation module 1530 may perform operation S230 for obtaining a second feature image using the first attention data and the first feature image.

[0219] According to an embodiment of the present disclosure, the first calculation module 1530 may also perform operations S710 to S730, which will not be described in detail herein.

[0220] The second calculation module 1540 may perform operation S240 for obtaining a third feature image using the second attention data and the first feature image.

[0221] According to an embodiment of the present disclosure, the second calculation module 1540 may also perform operations S1010 to S1030, which will not be described in detail herein.

[0222] The first classification module 1550 may perform operation S250 for classifying each of the superpixels according to the second feature image and the third feature image.

[0223] According to an embodiment of the present disclosure, the first classification module 1550 may perform operations S1110 to S1130 and operations S1210 to S1220, which will not be described in detail herein.

[0224] The first segmentation module 1560 may perform operation S260 to segment the image to be segmented according to the classification result of each superpixel to obtain the N objects to be identified.

[0225] According to an embodiment of the present disclosure, the image processing device 1500 may include a superpixel segmentation module for performing operations S410 to S450, which will not be described in detail herein.

[0226] Fig.16 The structural block diagram of the image processing model training device according to an embodiment of the present disclosure is schematically shown.

[0227] like Fig.16 As shown, the image processing model training device 1600 of this embodiment is used to train the image processing model, and the trained image processing model can be used to implement the function of at least one module of the image processing device 1500. The image processing model training device 1600 may include a second obtaining module 1610, a second processing module 1620, a third calculation module 1630, a fourth calculation module 1640, a second classification module 1650, a second segmentation module 1660, and a model training module 1670.

[0228] The second obtaining module 1610 may perform operation S1310 to obtain a first training feature image based on m superpixels of a training sample image, wherein the training sample image includes n targets to be identified, and n and m are integers greater than or equal to 1, respectively.

[0229] The second processing module 1620 can execute operation S1320, which is used to use the attention layer to process the first training feature image to obtain first training attention data, and perform Gaussian distribution processing on the first training attention data to obtain second training attention data, wherein the value of the second training attention data obeys Gaussian distribution.

[0230] The third calculation module 1630 may perform operation S1330 for obtaining a second training feature image using the first training attention data and the first training feature image.

[0231] The fourth calculation module 1640 may perform operation S1340 for obtaining a third training feature image using the second training attention data and the first training feature image.

[0232] The second classification module 1650 may perform operation S1350 for classifying each of the superpixels using the graph convolutional neural network layer according to the second training feature image and the third training feature image.

[0233] The second segmentation module 1660 may perform operation S1360 to segment the training sample image according to the classification result of each superpixel to obtain predicted segmentation results of the n objects to be identified.

[0234] The model training module 1670 can perform operation S1370 to train the attention layer and the graph convolutional neural network layer based on the pre-labeled category labels of the n targets to be identified and the contrast loss of the predicted segmentation results.

[0235] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module / unit / sub-unit in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0236] According to an embodiment of the present disclosure, any multiple modules in the image processing device 1500 or the image processing model training device 1600 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0237] According to an embodiment of the present disclosure, at least one of the image processing device 1500 or the image processing model training device 1600 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or may be implemented in any one of the three implementation methods of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, at least one of the image processing device 1500 or the image processing model training device 1600 may be at least partially implemented as a computer program module, which may perform corresponding functions when the computer program module is executed.

[0238] Fig.17 A block diagram of an electronic device suitable for implementing an image processing method or a model training method according to an embodiment of the present disclosure is schematically shown.

[0239] like Fig.17As shown, the electronic device 1700 according to an embodiment of the present disclosure includes a processor 1701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1702 or a program loaded from a storage part 1708 into a random access memory (RAM) 1703. The processor 1701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1701 may also include an onboard memory for caching purposes. The processor 1701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0240] In RAM 1703, various programs and data required for the operation of electronic device 1700 are stored. Processor 1701, ROM 1702 and RAM 1703 are connected to each other via bus 1704. Processor 1701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1702 and / or RAM 1703. It should be noted that the program can also be stored in one or more memories other than ROM 1702 and RAM 1703. Processor 1701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0241] According to an embodiment of the present disclosure, the electronic device 1700 may further include an input / output (I / O) interface 1705, which is also connected to the bus 1704. The electronic device 1700 may further include one or more of the following components connected to the I / O interface 1705: an input portion 1706 including a keyboard, a mouse, etc. An output portion 1707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc. A storage portion 1708 including a hard disk, etc. And a communication portion 1709 including a network interface card such as a LAN card, a modem, etc. The communication portion 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as needed. A removable medium 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1710 as needed, so that the computer program read therefrom is installed into the storage portion 1708 as needed.

[0242] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0243] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1702 and / or RAM 1703 described above and / or one or more memories other than ROM 1702 and RAM 1703.

[0244] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0245] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0246] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1709, and / or installed from a removable medium 1711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0247] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1709, and / or installed from the removable medium 1711. When the computer program is executed by the processor 1701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0248] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0249] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image processing method, comprising: Obtaining a first feature image based on M superpixels of an image to be segmented, wherein the image to be segmented includes N objects to be identified, and N and M are integers greater than or equal to 1 respectively; Processing the first feature image according to the attention mechanism to obtain first attention data, and performing Gaussian distribution processing on the first attention data to obtain second attention data, wherein the value of the second attention data obeys Gaussian distribution; Obtaining a second feature image using the first attention data and the first feature image; Obtaining a third feature image using the second attention data and the first feature image; Classifying each of the superpixels using a graph convolutional neural network layer according to the second feature image and the third feature image; The image to be segmented is segmented according to the classification result of each superpixel to obtain the N objects to be identified.

2. The method according to claim 1, wherein: The attention mechanism includes a channel attention mechanism and a spatial attention mechanism, the first attention data includes channel attention data and spatial attention data, and the first attention data is obtained by processing the first feature image according to the attention mechanism, including: Processing the first feature image according to the channel attention mechanism to obtain the channel attention data; The first feature image is processed according to the spatial attention mechanism to obtain the spatial attention data.

3. The method according to claim 2, wherein: The obtaining a second feature image by using the first attention data and the first feature image comprises: Obtaining a fourth feature image using the channel attention data and the first feature image; Obtaining a fifth feature image using the spatial attention data and the first feature image; The second characteristic image is obtained according to the fourth characteristic image and the fifth characteristic image.

4. The method according to claim 2, wherein: The second attention data includes modulated channel attention data and modulated spatial attention data, and performing Gaussian distribution processing on the first attention data to obtain the second attention data includes: Performing modulation processing on the channel attention data based on a Gaussian function to obtain the modulated channel attention data, wherein the value of the modulated channel attention data obeys a Gaussian distribution; The spatial attention data is modulated based on a Gaussian function to obtain the modulated spatial attention data, wherein the value of the modulated spatial attention data obeys a Gaussian distribution.

5. The method according to claim 4, wherein: The obtaining a third feature image by using the second attention data and the first feature image comprises: Obtaining a sixth feature image using the modulation channel attention data and the first feature image; Obtaining a seventh feature image using the modulated spatial attention data and the first feature image; The third characteristic image is obtained according to the sixth characteristic image and the seventh characteristic image.

6. The method according to claim 1, wherein: The classifying each of the superpixels according to the second feature image and the third feature image comprises: Obtaining an eighth characteristic image according to the second characteristic image and the third characteristic image; Using the eighth feature image as an input of a convolutional neural network to obtain a first feature matrix; The first feature matrix is ​​processed using a graph convolutional neural network to classify each of the superpixels.

7. The method according to claim 6, wherein: After obtaining the first feature matrix, the method includes: Processing the first feature matrix using a graph convolution attention mechanism; The using a graph convolutional neural network to process the first feature matrix and classify each of the superpixels includes: The processed first feature matrix is ​​used as input of the graph convolutional neural network to obtain a classification result of each superpixel.

8. The method according to claim 1, wherein: The method further includes constructing an undirected graph model based on the M superpixels, wherein the M superpixels are M vertices of the undirected graph model, specifically including: Let the initial value of i be 1, and loop through the following operations, where i is an integer greater than or equal to 1 and i is less than or equal to M: Determine at least one superpixel adjacent to the i-th superpixel; Calculate the Euclidean distance between the i-th superpixel and the first adjacent superpixel in the RGB channel, wherein the first adjacent superpixel is any superpixel adjacent to the i-th superpixel; When the Euclidean distance is less than or equal to a preset threshold, constructing an edge between the i-th superpixel and the first adjacent superpixel; Let the value of i be increased by 1.

9. A method for training an image processing model, wherein: The image processing model is used to implement the method according to any one of claims 1 to 8, the image processing model comprises an attention layer and a graph convolutional neural network layer, and the method comprises: Obtaining a first training feature image based on m superpixels of a training sample image, wherein the training sample image includes n targets to be identified, and n and m are integers greater than or equal to 1 respectively; Processing the first training feature image using the attention layer to obtain first training attention data, and performing Gaussian distribution processing on the first training attention data to obtain second training attention data, wherein the value of the second training attention data obeys Gaussian distribution; Obtaining a second training feature image using the first training attention data and the first training feature image; Obtaining a third training feature image using the second training attention data and the first training feature image; Classifying each of the superpixels using the graph convolutional neural network layer according to the second training feature image and the third training feature image; Segmenting the training sample image according to the classification result of each superpixel to obtain predicted segmentation results of the n targets to be identified; The attention layer and the graph convolutional neural network layer are trained based on the pre-labeled category labels of the n targets to be identified and the contrast loss of the predicted segmentation results.

10. An image processing device, comprising: A first obtaining module, configured to obtain a first feature image based on M superpixels of an image to be segmented, wherein the image to be segmented includes N objects to be identified, and N and M are integers greater than or equal to 1 respectively; A first processing module, configured to process the first feature image according to an attention mechanism to obtain first attention data, and to perform Gaussian distribution processing on the first attention data to obtain second attention data, wherein the value of the second attention data obeys Gaussian distribution; A first calculation module, configured to obtain a second feature image using the first attention data and the first feature image; A second calculation module, configured to obtain a third feature image using the second attention data and the first feature image; A first classification module, configured to classify each of the superpixels using a graph convolutional neural network layer according to the second feature image and the third feature image; The first segmentation module is used to segment the image to be segmented according to the classification result of each superpixel to obtain the N objects to be identified.

11. A training device for an image processing model, wherein: The image processing model includes an attention layer and a graph convolutional neural network layer, and the device includes: A second acquisition module is used to obtain a first training feature image based on m superpixels of a training sample image, wherein the training sample image includes n targets to be identified, and n and m are integers greater than or equal to 1 respectively; A second processing module, configured to process the first training feature image using the attention layer to obtain first training attention data, and perform Gaussian distribution processing on the first training attention data to obtain second training attention data, wherein the value of the second training attention data obeys Gaussian distribution; A third calculation module, configured to obtain a second training feature image using the first training attention data and the first training feature image; a fourth calculation module, configured to obtain a third training feature image using the second training attention data and the first training feature image; A second classification module, configured to classify each of the superpixels using the graph convolutional neural network layer according to the second training feature image and the third training feature image; A second segmentation module is used to segment the training sample image according to the classification result of each superpixel to obtain the predicted segmentation results of the n objects to be identified; A model training module is used to train the attention layer and the graph convolutional neural network layer based on the pre-labeled category labels of the n targets to be identified and the contrast loss of the predicted segmentation results.

12. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 9.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images