Machine learning model training method, image processing method, and electronic device

Abstract

The present disclosure relates to a training method of a machine learning model, a processing method of an image and an electronic device, and relates to the technical field of computers. The training method comprises: determining an attention feature map of a sample image by using a global attention module of the machine learning model; performing semantic segmentation on the sample image by using a segmentation layer of the machine learning model according to the attention feature map, and determining a segmentation loss function according to a segmentation result; and training the machine learning model according to the segmentation loss function. The technical solution of the present disclosure can improve the semantic segmentation accuracy of the machine learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for a machine learning model, an image processing method, a training device for a machine learning model, an image processing device, an electronic device, and a non-volatile computer-readable storage medium. Background Art

[0002] Image semantic segmentation is a fundamental task in computer vision. Its goal is to assign a semantic label to each pixel in an image. Semantic segmentation methods are widely used in fields such as autonomous driving and video surveillance, helping machines identify and understand specific objects in a scene.

[0003] Semantic segmentation methods require that the images in the training set contain pixel-by-pixel annotation information, meaning that each pixel in the image has its own semantic label. Acquiring this kind of annotation information is very expensive and labor-intensive. Therefore, current work is developing semantic segmentation methods based on weakly annotated information.

[0004] In related technologies, weakly supervised segmentation methods based on image-level labels contain only image-level labels in the training set, providing only the category to which the image belongs. The core idea of ​​this type of weakly supervised segmentation method is to generate high-confidence pixel-level pseudo-labels based on image-level annotation information, which serve as the annotation information for training segmentation. Weakly supervised segmentation methods based on image-level labels can be divided into two categories: multi-stage and single-stage (end-to-end). Summary of the Invention

[0005] The inventors of the present disclosure have discovered that the above-mentioned related technologies have the following problem: the accuracy of semantic segmentation is low.

[0006] In view of this, the present disclosure proposes a training technology solution for a machine learning model, which can improve the semantic segmentation accuracy of the machine learning model.

[0007] According to some embodiments of the present disclosure, a method for training a machine learning model is provided, including: using a global attention module of the machine learning model to determine an attention feature map of a sample image; performing semantic segmentation on the sample image based on the attention feature map using a segmentation layer of the machine learning model, and determining a segmentation loss function based on the segmentation result; and training the machine learning model based on the segmentation loss function.

[0008] In some embodiments, the training method also includes: using a global attention module to determine the attention matrix of the sample image; based on the attention matrix, using the linear layer of the machine learning model to predict the semantic affinity relationship between pixels in the sample image, and determining the semantic affinity relationship loss function based on the prediction results; training the machine learning model includes: training the machine learning model based on the segmentation loss function and the semantic affinity relationship loss function.

[0009] In some embodiments, determining the semantic affinity loss function based on the prediction results includes: determining the difference parameters between different pixels based on the differences between different pixels in the sample image; generating a first category activation map based on the attention feature map using the classification layer of the machine learning model; processing the first category activation map using the difference parameters to generate a second category activation map; determining the semantic affinity label of each pixel based on the second category activation map; and determining the semantic affinity loss function based on the semantic affinity label and the prediction results.

[0010] In some embodiments, determining the difference parameters between different pixels includes: determining a first difference between a first pixel and a second pixel among the different pixels; determining a second difference between the first pixel and its multiple adjacent pixels, where the adjacent pixels are determined from pixels within a preset radius centered on the first pixel; and determining the difference parameter between the first pixel and the second pixel based on the sum of the first difference and the second difference.

[0011] In some embodiments, the differences between different pixels include color differences and position differences, and determining the difference parameters between different pixels includes: determining the color difference parameters between different pixels based on the color differences; determining the position difference parameters between different pixels based on the position differences; and determining the difference parameters based on the weighted sum of the color difference parameters and the position difference parameters.

[0012] In some embodiments, determining the semantic affinity relationship label of each pixel based on the second category activation map includes: dividing the second category activation map into multiple regions based on a preset threshold; setting different first pixel-level pseudo labels for pixels in different regions; and determining the semantic affinity relationship label based on the first pixel-level pseudo label.

[0013] In some embodiments, multiple areas include an ignorable area, and determining the semantic affinity label based on the first pixel-level pseudo-label includes: when the first pixel and the second pixel in different pixels have the same first pixel-level pseudo-label, and the first pixel and the second pixel are not located in the ignorable area, the semantic affinity label is the first label; when the first pixel and the second pixel have different first pixel-level pseudo-labels, and the first pixel and the second pixel are not located in the ignorable area, the semantic affinity label is the second label, and the value of the first label is greater than the value of the second label; when at least one of the first pixel and the second pixel is located in the ignorable area, the semantic affinity label is the third label, and the value of the third label is greater than the value of the first label.

[0014] In some embodiments, determining the semantic affinity loss function includes: determining a semantic affinity matrix based on an attention matrix and its transposed matrix, wherein each element in the semantic affinity matrix corresponds to a semantic affinity parameter of each pixel; calculating a first difference between a preset value and the semantic affinity parameter of each pixel with a first label, and calculating the mean of all first differences as a first mean; calculating the mean of the semantic affinity parameters of all pixels with a second label as a second mean; and determining the semantic affinity loss function based on a weighted sum of the first mean and the second mean.

[0015] In some embodiments, the threshold includes a first threshold and a second threshold, the first threshold is less than the second threshold, and according to the preset threshold, dividing the second category activation map into multiple areas includes: dividing the pixels corresponding to the values ​​less than or equal to the first threshold in the second category activation map into the background area; dividing the pixels corresponding to the values ​​greater than or equal to the second threshold in the second category activation map into the foreground area; dividing the pixels corresponding to the values ​​less than the second threshold and greater than the first threshold in the second category activation map into the ignorable area.

[0016] In some embodiments, the training method also includes: classifying the sample image according to the attention feature map using the classification layer of the machine learning model; determining the semantic affinity matrix according to the attention matrix and its transposed matrix, and each element in the semantic affinity matrix corresponds to the semantic affinity relationship parameter of each pixel; determining the third category activation map according to the semantic affinity matrix and the second category activation map; determining the second pixel-level pseudo-label of each pixel according to the third category activation map; determining the classification loss function according to the second pixel-level pseudo-label and the classification result; training the machine learning model includes: training the machine learning model according to the classification loss function, the segmentation loss function and the semantic affinity loss function.

[0017] In some embodiments, determining the third category activation map based on the semantic affinity matrix and the second category activation map includes: generating a diagonal matrix based on the sum of the elements in each row of the semantic affinity matrix; determining a semantic transition matrix based on the diagonal matrix and the semantic affinity matrix; and determining the third category activation map based on the semantic transition matrix and the second category activation map.

[0018] In some embodiments, the training method also includes: classifying the sample image according to the attention feature map using the classification layer of the machine learning model, and determining the classification loss function according to the classification results; training the machine learning model includes: training the machine learning model according to the classification loss function, the segmentation loss function and the semantic affinity loss function.

[0019] In some embodiments, using the global attention module of the machine learning model to determine the attention feature map of the sample image includes: dividing the sample image into multiple image blocks; using the global attention module to process the multiple image blocks to determine multiple feature vectors; and determining the attention feature map based on the multiple feature vectors.

[0020] According to other embodiments of the present disclosure, a method for processing an image is provided, including: using a global attention module of a machine learning model to determine an attention feature map of an image to be processed, the machine learning model being trained by the training method in any of the above embodiments; performing semantic segmentation on the image to be processed using a segmentation layer of the machine learning model based on the attention feature map to determine a segmentation result of the image to be processed.

[0021] According to some further embodiments of the present disclosure, a training device for a machine learning model is provided, including: a feature extraction unit for determining an attention feature map of a sample image using a global attention module of the machine learning model; a segmentation unit for performing semantic segmentation on the sample image based on the attention feature map and using a segmentation layer of the machine learning model, and determining a segmentation loss function based on the segmentation result; and a training unit for training the machine learning model based on the segmentation loss function.

[0022] In some embodiments, the feature extraction unit predicts the semantic affinity between pixels in the sample image based on the attention matrix and the linear layer of the machine learning model, and determines the semantic affinity loss function based on the prediction results; the training unit trains the machine learning model based on the segmentation loss function and the semantic affinity loss function.

[0023] In some embodiments, the training unit determines the difference parameters between different pixels based on the differences between different pixels in the sample image; generates a first category activation map based on the attention feature map using the classification layer of the machine learning model; processes the first category activation map using the difference parameters to generate a second category activation map; determines the semantic affinity relationship label of each pixel based on the second category activation map; and determines the semantic affinity relationship loss function based on the semantic affinity relationship label and the prediction result.

[0024] In some embodiments, the training unit determines a first difference between a first pixel and a second pixel among different pixels; determines a second difference between the first pixel and its multiple adjacent pixels, where the adjacent pixels are determined from pixels within a preset radius centered on the first pixel; and determines a difference parameter between the first pixel and the second pixel based on the sum of the first difference and the second difference.

[0025] In some embodiments, the differences between different pixels include color differences and position differences. The training unit determines the color difference parameters between different pixels based on the color differences; determines the position difference parameters between different pixels based on the position differences; and determines the difference parameters based on the weighted sum of the color difference parameters and the position difference parameters.

[0026] In some embodiments, the training unit divides the second category activation map into multiple regions according to a preset threshold; sets different first pixel-level pseudo labels for pixels in different regions; and determines semantic affinity relationship labels based on the first pixel-level pseudo labels.

[0027] In some embodiments, multiple areas include an ignorable area, and when the first pixel and the second pixel in different pixels of the training unit have the same first pixel-level pseudo-label, and the first pixel and the second pixel are not located in the ignorable area, the semantic affinity label is the first label; when the first pixel and the second pixel have different first pixel-level pseudo-labels, and the first pixel and the second pixel are not located in the ignorable area, the semantic affinity label is the second label, and the value of the first label is greater than the value of the second label; when at least one of the first pixel and the second pixel is located in the ignorable area, the semantic affinity label is the third label, and the value of the third label is greater than the value of the first label.

[0028] In some embodiments, the training unit determines a semantic affinity matrix based on the attention matrix and its transposed matrix, wherein each element in the semantic affinity matrix corresponds to a semantic affinity parameter of each pixel; calculates a first difference between a preset value and the semantic affinity parameter of each pixel with a first label, and calculates the mean of all first differences as a first mean; calculates the mean of the semantic affinity parameters of all pixels with a second label as a second mean; and determines a semantic affinity loss function based on the weighted sum of the first mean and the second mean.

[0029] In some embodiments, the threshold includes a first threshold and a second threshold, and the training unit divides the pixels corresponding to the values ​​less than or equal to the first threshold in the second category activation map into the background area; divides the pixels corresponding to the values ​​greater than or equal to the second threshold in the second category activation map into the foreground area; and divides the pixels corresponding to the values ​​less than the second threshold and greater than the first threshold in the second category activation map into the ignorable area.

[0030] In some embodiments, the training device also includes a classification unit for classifying the sample image according to the attention feature map using the classification layer of the machine learning model; the training unit determines the semantic affinity matrix based on the attention matrix and its transposed matrix, and each element in the semantic affinity matrix corresponds to the semantic affinity relationship parameter of each pixel; the training unit determines the third category activation map based on the semantic affinity matrix and the second category activation map; the training unit determines the second pixel-level pseudo-label of each pixel based on the third category activation map; the training unit determines the classification loss function based on the second pixel-level pseudo-label and the classification result; the training unit trains the machine learning model based on the classification loss function, the segmentation loss function and the semantic affinity loss function.

[0031] In some embodiments, the training unit generates a diagonal matrix based on the sum of the elements in each row of the semantic affinity matrix; determines a semantic transition matrix based on the diagonal matrix and the semantic affinity matrix; and determines a third category activation map based on the semantic transition matrix and the second category activation map.

[0032] In some embodiments, the classification unit classifies the sample image according to the attention feature map using the classification layer of the machine learning model, and the training unit determines the classification loss function according to the classification result; the training unit trains the machine learning model according to the classification loss function, the segmentation loss function and the semantic affinity loss function.

[0033] In some embodiments, the feature extraction unit divides the sample image into multiple image blocks; uses the global attention module to process the multiple image blocks to determine multiple feature vectors; and determines an attention feature map based on the multiple feature vectors.

[0034] According to some further embodiments of the present disclosure, an image processing device is provided, including: a feature extraction unit, used to use the global attention module of a machine learning model to determine the attention feature map of the image to be processed, and the machine learning model is trained by the training method in any of the above embodiments; a segmentation unit, used to use the segmentation layer of the machine learning model to perform semantic segmentation on the image to be processed according to the attention feature map, and determine the segmentation result of the image to be processed.

[0035] According to some further embodiments of the present disclosure, there is provided an electronic device comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the training method of the machine learning model or the image processing method in any one of the above-mentioned embodiments based on instructions stored in the memory device.

[0036] According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the machine learning model or the image processing method in any of the above embodiments is implemented.

[0037] In the above embodiment, the global object information in the image is well utilized through the global attention mechanism, which enables the machine learning model to discover complete semantic objects, thereby improving the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0039] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0040] Figure 1 A flowchart illustrating some embodiments of a method for training a machine learning model of the present disclosure;

[0041] Figures 2a-2b Schematic diagrams showing some embodiments of the training method of the machine learning model disclosed herein;

[0042] Figure 3a A flowchart illustrating some embodiments of the image processing method disclosed herein;

[0043] Figure 3b Schematic diagrams showing some embodiments of the image processing method disclosed herein;

[0044] Figure 4a A block diagram illustrating some embodiments of a training apparatus for a machine learning model of the present disclosure;

[0045] Figure 4b A block diagram showing some embodiments of an image processing apparatus according to the present disclosure;

[0046] Figure 5 A block diagram illustrating some embodiments of the electronic device of the present disclosure;

[0047] Figure 6 A block diagram showing other embodiments of the electronic device of the present disclosure. DETAILED DESCRIPTION

[0048] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0049] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0050] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0051] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0052] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0053] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0054] As mentioned above, both multi-stage and single-stage approaches, weakly supervised semantic segmentation methods based on image-level labels, primarily train classification networks using CNNs (Convolutional Neural Networks) to generate class activation maps as initial pixel-level pseudo-labels. Because these initial pseudo-labels are relatively coarse, they are typically post-processed using a fully connected Conditional Random Field (CRF) to obtain more refined pseudo-labels.

[0055] However, the above weakly supervised semantic segmentation methods based on image-level labels have the following problems.

[0056] The convolution operation in CNN has a limited receptive field and cannot make good use of the global information of the image, resulting in the generated category activation Figure 1 Generally, it can only contain local areas of objects. That is, the complete object cannot be well marked in the pseudo label, which affects the accuracy of semantic segmentation.

[0057] Due to downsampling and other operations within the network structure, the generated class activation maps have rough edges around objects. Semantic labels for adjacent, similar pixels often differ. CRF methods, which require considering the RGB (red, green, and blue) and position information of all pixels, are therefore slow and unsuitable for efficient end-to-end training.

[0058] To address the above technical issues, this paper proposes an end-to-end weakly supervised semantic segmentation network based on learning semantic affinities through visual attention. This paper introduces the visual Transformer architecture into an end-to-end weakly supervised semantic segmentation approach. The visual Transformer architecture effectively captures global image information (other attention-based networks are also applicable).

[0059] In some embodiments, a classification network is constructed based on a visual Transformer to extract image feature maps. Due to the global attention mechanism within the Transformer architecture, global object information within the image can be effectively utilized. As a result, the generated CAM (Class Activation Map) can detect complete semantic objects, effectively overcoming the limitations of CNNs.

[0060] In some embodiments, the potential of the Transformer structure is further explored to learn symmetrical pixel semantic affinity relationships from global attention, further improving the accuracy of the generated CAM.

[0061] In some embodiments, pixel-adaptive convolution is used to efficiently extract the underlying local neighborhood information of pixels, and the CAM is further processed so that the pseudo labels generated by the CAM are consistent with the underlying boundaries of the object.

[0062] By exploring the Transformer structure to generate initial labels for weakly supervised semantic segmentation networks, we solve the following technical problems: incomplete activation when the CNN structure generates CAM; and the technical problem of inaccurate semantics of pseudo labels generated based on CAM.

[0063] Through the pixel-adaptive processing module, we solved the following technical problems: the inconsistency between pseudo labels and underlying pixel boundaries; the high time complexity of methods such as CRF, which are not suitable for use in end-to-end training frameworks.

[0064] For example, the technical solutions of the present disclosure can be implemented through the following embodiments.

[0065] Figure 1 Flowchart illustrating some embodiments of the method for training a machine learning model of the present disclosure.

[0066] like Figure 1 As shown, in step 110, the global attention module of the machine learning model is used to determine the attention feature map of the sample image. For example, the self-attention module of the Transformer network or the attention mechanism module of other networks can be used to extract the attention feature map. For example, the sample image can be a face image, an image containing an object to be tracked, an image containing an object to be identified, etc. Semantic segmentation can be performed using the machine learning model to achieve face recognition, object tracking, object detection, etc.

[0067] In some embodiments, using the global attention module of the machine learning model to determine the attention feature map of the sample image includes: dividing the sample image into multiple image blocks; using the global attention module to process the multiple image blocks to determine multiple feature vectors; and determining the attention feature map based on the multiple feature vectors.

[0068] For example, using the Transformer as the backbone network for feature extraction, the Transformer backbone network first divides an input image into h×w image blocks, and then inputs each image block into the fully connected layer of the backbone network to obtain the same number of feature vectors.

[0069] In some embodiments, in each layer of the backbone network, a self-attention method is used to capture the global relationship between features to avoid incomplete activation of CAM caused by the local receptive field of the convolution operation.

[0070] For example, through the three different fully connected layers of the Transformer's self-attention module, the feature vector output by the fully connected layer is mapped into three sub-feature vectors: Q (query feature), K (key feature), and V (value feature).

[0071] For example, Transformer adopts a multi-head mechanism, each head has 3 different fully connected layers. The three sub-feature vectors of the i-th head are Q i , K i 、V i , calculate the attention matrix S of the i-th head by the following formula i :

[0072] X i =softmax(S i )V i

[0073] d k K i The feature dimension, X i is the output feature of this layer (i.e., the attention feature map of the i-th head). After repeating the above steps multiple times, the final output feature X i Used for subsequent modules. For example, the output features of all heads can be fused to obtain the final attention feature map X∈R h×w×c .

[0074] In step 120, semantic segmentation is performed on the sample image based on the attention feature map using the segmentation layer of the machine learning model, and a segmentation loss function is determined based on the segmentation result.

[0075] For example, for the segmentation loss First, a segmentation decoder (segmentation layer) is used to predict the feature map to obtain the pixel-level prediction of the image; then, the semantic affinity label is obtained based on the pixel-level prediction and the step of learning the semantic affinity from the attention matrix, and the cross entropy loss function is used to obtain

[0076] In some embodiments, a global attention module is used to determine the attention matrix of a sample image; based on the attention matrix, the linear layer of a machine learning model is used to predict the semantic affinity relationship between pixels in the sample image, and a semantic affinity loss function is determined based on the prediction results.

[0077] For example, according to the differences between different pixels in the sample image, the difference parameters between different pixels are determined.

[0078] For example, a first difference between a first pixel and a second pixel among different pixels is determined; a second difference between the first pixel and its multiple adjacent pixels is determined, where the adjacent pixels are determined from pixels within a preset radius centered on the first pixel; and a difference parameter between the first pixel and the second pixel is determined based on the sum of the first difference and the second difference.

[0079] For example, the difference parameter is positively correlated with the first difference and negatively correlated with the second difference.

[0080] For example, the differences between different pixels include color differences and position differences. Based on the color differences, the color difference parameters between different pixels are determined; based on the position differences, the position difference parameters between different pixels are determined; and based on the weighted sum of the color difference parameters and the position difference parameters, the difference parameters are determined.

[0081] For example, the input sample image is I∈R h×w×3 , for the pixels at positions (i, j) and (k, l), the color (RGB) difference parameters and position difference parameters between them are:

[0082]

[0083] are the color standard deviation and position standard deviation calculated based on the RGB information and position information of pixels (i, j) and (k, l), respectively; ω1 and ω6 are predetermined weight parameters between 0 and 1 (e.g., 0.3); |I ij -I kl | is the color difference, |P ij -P kl | is the position difference.

[0084] For example, according to the difference parameter and the position difference parameter, the difference parameter of pixels (i, j) and (k, l) is determined as:

[0085]

[0086] (x, y) is the adjacent pixel selected from the neighborhood of pixel (i, j). Figure 2b As shown in Figure 1, the neighborhood set of pixel (i, j) is the eight neighboring pixels of pixel (i, j). For example, neighboring pixels can be quickly obtained through dilated convolution with different dilation values. ω3 is a predetermined weight parameter between 0 and 1 (e.g., 0.01).

[0087] In some embodiments, based on the attention feature map, a first category activation map is generated using the classification layer of the machine learning model; the first category activation map is processed using the difference parameter to generate a second category activation map; based on the second category activation map, the semantic affinity label of each pixel is determined; based on the semantic affinity label and the prediction result, the semantic affinity loss function is determined.

[0088] In some embodiments, a pooling operation is performed on X to obtain a feature vector f∈R for classification c , c is the feature dimension.

[0089] For example, use GMP (global maximum pooling) for pooling operation; input the pooled f into the classification layer to predict the membership probability p∈R of the image-level label to each category k , k is the number of categories.

[0090] For example, it can be seen that the classification loss function L is calculated using the binary cross entropy loss function cls , to update the parameters of the backbone network and classification layer.

[0091] In some embodiments, the attention feature map X before pooling is input into the classification layer to obtain a first category activation map; and an initial pixel-level pseudo label is obtained based on the first category activation map.

[0092] For example, the first category activation map M∈R can be calculated based on the difference parameter h×w×k Perform multiple iterations, k is the number of categories. For the tth iteration, the second category activation map generated is:

[0093]

[0094] is the set of neighboring pixels of pixel (i, j).

[0095] In some embodiments, the second category activation map is divided into multiple regions according to a preset threshold; different first pixel-level pseudo labels are set for pixels in different regions; and semantic affinity relationship labels are determined based on the first pixel-level pseudo labels.

[0096] For example, the threshold includes a first threshold and a second threshold, the first threshold is less than the second threshold, and according to the preset threshold, the second category activation map is divided into multiple areas, including: dividing the pixels corresponding to the values ​​less than or equal to the first threshold in the second category activation map into the background area; dividing the pixels corresponding to the values ​​greater than or equal to the second threshold in the second category activation map into the foreground area; dividing the pixels corresponding to the values ​​less than the second threshold and greater than the first threshold in the second category activation map into the ignorable area.

[0097] In order to learn a high-confidence semantic affinity matrix A, it is necessary to obtain a high-confidence semantic affinity relationship label Y aff As supervision information of A.

[0098] For example, the affinity label Y can be derived based on the output of pixel adaptive processing aff . Set two background thresholds β between 0 and 1 l and β h , β l <β h (such as 0.35 and 0.55 respectively), the second category activation map is divided into the foreground area and background area with higher confidence, and the ignored area with lower confidence.

[0099] For example, for the second category activation map M∈R output after pixel adaptive processing h×w×k , calculate the first pixel-level pseudo label of each region according to the following formula

[0100]

[0101] 0 and 255 represent the first pixel-level pseudo labels of the background area and the ignored area, respectively, and k is M i,j,k The sequence number in the channel dimension (e.g., less than 100).

[0102] In some embodiments, the plurality of regions include an ignorable region, and according to the first pixel-level pseudo-label, when a first pixel and a second pixel in different pixels have the same first pixel-level pseudo-label and both the first pixel and the second pixel are not located in the ignorable region, the semantic affinity label is a first label; when the first pixel and the second pixel have different first pixel-level pseudo-labels and both the first pixel and the second pixel are not located in the ignorable region, the semantic affinity label is a second label, and the value of the first label is greater than the value of the second label; when at least one of the first pixel and the second pixel is located in the ignorable region, the semantic affinity label is a third label, and the value of the third label is greater than the value of the first label.

[0103] For example, after obtaining the first pixel-level pseudo-label, the affinity label Y aff ∈Rh×w×h×w The calculation method is: if the first pixel-level pseudo labels of pixels (i, j) and (k, l) are the same and neither is an ignored area, then If the first pixel-level pseudo labels of pixels (i, j) and (k, l) are different and neither is an ignored region, then If pixel (i, j) or (k, l) belongs to the ignored region, then

[0104] In some embodiments, a semantic affinity matrix is ​​determined based on an attention matrix and its transposed matrix, wherein each element in the semantic affinity matrix corresponds to a semantic affinity parameter of each pixel; a first difference between a preset value (such as a positive number such as 1) and the semantic affinity parameter of each pixel with a first label is calculated, and the mean of all first differences is calculated as a first mean; the mean of the semantic affinity parameters of all pixels with a second label is calculated as a second mean; and a semantic affinity loss function is determined based on the weighted sum of the first mean and the second mean.

[0105] For example, to get the pseudo label Y aff Afterwards, it can be used to supervise the predicted semantic affinity matrix A, and the semantic affinity loss function is:

[0106]

[0107] R + and R - Represents Y aff The area of ​​1 and 0, N + and N - Respectively represent R + and R - The number of pixels in , A(i, j) is the element in the i-th row and j-th column of A.

[0108] In some embodiments, the sample image is classified according to the attention feature map using the classification layer of the machine learning model; the semantic affinity matrix is ​​determined according to the attention matrix and its transposed matrix, and each element in the semantic affinity matrix corresponds to the semantic affinity relationship parameter of each pixel; the third category activation map is determined according to the semantic affinity matrix and the second category activation map; the second pixel-level pseudo-label of each pixel is determined according to the third category activation map; and the classification loss function is determined according to the second pixel-level pseudo-label and the classification result.

[0109] For example, the attention matrix is ​​input into the linear layer to obtain the predicted value of the semantic affinity relationship. Since the attention feature map is a directed graph, and the semantic affinity relationship is an undirected graph, the transpose of S and S is added to generate the semantic affinity matrix to achieve the conversion from directed graph to undirected graph:

[0110] A=MLP(S+S T )

[0111] MLP stands for linear layer function.

[0112] For example, a diagonal matrix is ​​generated based on the sum of the elements in each row of the semantic affinity matrix; a semantic transition matrix is ​​determined based on the diagonal matrix and the semantic affinity matrix; and a third category activation map is determined based on the semantic transition matrix and the second category activation map.

[0113] For example, L aff This can make the predicted semantic affinity matrix A more accurate. The learned high-confidence semantic affinity matrix can be further processed on the pseudo-label to obtain a more accurate label.

[0114] For example, for the learned semantic affinity matrix A, calculate the semantic transformation matrix T:

[0115] T=D -1 A α ,D(i,i)=∑ k A(i, j) α

[0116] α is a tuning parameter greater than 1 (e.g., it can be 2), which is used to remove smaller values ​​in A; D is a diagonal matrix, and D(i, i) is an element on the diagonal.

[0117] For example, the initial pseudo-label is processed according to the semantic transformation matrix T. For the second category activation map M∈R h×w×k , first vectorize it into vec(M)∈R (h×w)×k ; Determine the activation map of the third category as:

[0118] M aff =T×vec(M)

[0119] In step 130, a machine learning model is trained based on the segmentation loss function.

[0120] In some embodiments, a machine learning model is trained based on a segmentation loss function and a semantic affinity loss function.

[0121] In some embodiments, the machine learning model is trained based on a classification loss function, a segmentation loss function, and a semantic affinity loss function.

[0122] In some embodiments, the sample image is classified according to the attention feature map using the classification layer of the machine learning model, and the classification loss function is determined according to the classification result; the machine learning model is trained according to the classification loss function, the segmentation loss function and the semantic affinity loss function.

[0123] For example, the comprehensive loss function of the network is the weighted sum of the classification loss function, the segmentation loss function, and the affinity loss function:

[0124] L=L cls +λ1×L seg +λ2×L aff

[0125] λ1 and λ2 are preset weights between 0 and 1 (eg, 0.1).

[0126] Figures 2a-2b Schematic diagram showing some embodiments of the training method of the machine learning model of the present disclosure.

[0127] like Figure 2a As shown in the figure, in the attention feature map extraction step, the Transformer is used as the backbone network for feature extraction. In the Transformer backbone network, an input image is first divided into h×w image blocks; each image block is then input into the fully connected layer of the backbone network to obtain the same number of feature vectors.

[0128] In some embodiments, in each layer of the backbone network, a self-attention method is used to capture the global relationship between features to avoid incomplete activation of CAM caused by the local receptive field of the convolution operation.

[0129] For example, through the three different fully connected layers of the Transformer's self-attention module, the feature vector output by the fully connected layer is mapped into three sub-feature vectors: Q (query feature), K (key feature), and V (value feature).

[0130] For example, Transformer adopts a multi-head mechanism, each head has 3 different fully connected layers. The three sub-feature vectors of the i-th head are Q i , K i 、V i , calculate the attention matrix S of the i-th head by the following formula i :

[0131] X i =softmax(S i )V i

[0132] d k K i The feature dimension, X i is the output feature of this layer (i.e., the attention feature map of the i-th head). After repeating the above steps multiple times, the final output feature X i Used for subsequent modules. For example, the output features of all heads can be fused to obtain the final attention feature map X∈Rh×w×c .

[0133] In some embodiments, the classification loss calculation and initial pseudo-label generation steps are performed. Pooling operation is performed on X to obtain the feature vector f∈R for classification c , c is the feature dimension.

[0134] For example, use GMP (global maximum pooling) for pooling operation; input the pooled f into the classification layer to predict the membership probability A∈R of the image-level label to each category k , k is the number of categories.

[0135] For example, it can be seen that the classification loss function L is calculated using the binary cross entropy loss function cls , to update the parameters of the backbone network and classification layer.

[0136] In some embodiments, the attention feature map X before pooling is input into the classification layer to obtain a first category activation map; and an initial pixel-level pseudo label is obtained based on the first category activation map.

[0137] In some embodiments, a pixel-adaptive processing step is performed. The classification loss calculation is relatively coarse compared to the initial pseudo-labels generated in the initial pseudo-label generation step. Furthermore, the initial pseudo-labels are locally inconsistent, meaning that the semantic labels of adjacent pixels with similar RGB information are often inconsistent. The CRF method is slow in processing local consistency, so the present disclosure proposes a processing method based on local pixel RGB and position information.

[0138] For example, the input sample image is I∈R h×w×3 , for the pixels at positions (i, j) and (k, l), the color (RGB) difference parameters and position difference parameters between them are:

[0139]

[0140] are the color standard deviation and position standard deviation calculated based on the RGB information and position information of pixels (i, j) and (k, l), respectively; ω1 and ω2 are pre-determined weight parameters between 0 and 1 (e.g., 0.3); |I ij -I kl | is the color difference, |P ij -P kl | is the position difference.

[0141] For example, according to the difference parameter and the position difference parameter, the difference parameter of pixels (i, j) and (k, l) is determined as:

[0142]

[0143] (x, y) is the adjacent pixel selected from the neighborhood of pixel (i, j). Figure 2b As shown in Figure 1, the neighborhood set of pixel (i, j) is the eight neighboring pixels of pixel (i, j). For example, neighboring pixels can be quickly obtained through dilated convolution with different dilation values. ω3 is a predetermined weight parameter between 0 and 1 (e.g., 0.01).

[0144] For example, the first category activation map M∈R can be calculated based on the difference parameter h×w×k Perform multiple iterations, k is the number of categories. For the tth iteration, the second category activation map generated is:

[0145]

[0146] is the set of neighboring pixels of pixel (i, j).

[0147] In some embodiments, a step of learning semantic affinity relationships from an attention matrix is ​​performed. From the attention matrix S in the attention feature map extraction step, semantic affinity relationships between pixels are learned to improve the quality of pseudo labels.

[0148] For example, the attention matrix is ​​input into the linear layer to obtain the predicted value of the semantic affinity relationship. Since the attention feature map is a directed graph, and the semantic affinity relationship is an undirected graph, the transpose of S and S is added to generate the semantic affinity matrix to achieve the conversion from directed graph to undirected graph:

[0149] A=MLP(S+S T )

[0150] MLP stands for linear layer function.

[0151] In order to learn a high-confidence semantic affinity matrix A, it is necessary to obtain a high-confidence semantic affinity relationship label Y aff As supervision information of A.

[0152] For example, the affinity label Y can be derived based on the output of pixel adaptive processing aff . Set two background thresholds β between 0 and 1 l and β h , β l <β h (such as 0.35 and 0.55 respectively), the second category activation map is divided into the foreground area and background area with higher confidence, and the ignored area with lower confidence.

[0153] For example, for the second category activation map M∈R output after pixel adaptive processing h×w×k , calculate the first pixel-level pseudo label of each region according to the following formula

[0154]

[0155] 0 and 255 represent the first pixel-level pseudo labels of the background area and the ignored area, respectively, and k is M i,j,k The sequence number in the channel dimension (e.g., less than 100).

[0156] For example, after obtaining the first pixel-level pseudo-label, the affinity label Y aff ∈R h×w×h×w The calculation method is: if the first pixel-level pseudo labels of pixels (i, j) and (k, l) are the same and neither is an ignored area, then If the first pixel-level pseudo labels of pixels (i, j) and (k, l) are different and neither is an ignored region, then If pixel (i, j) or (k, l) belongs to the ignored region, then

[0157] For example, to get the pseudo label Y aff Afterwards, it can be used to supervise the predicted semantic affinity matrix A, and the semantic affinity loss function is:

[0158]

[0159] R + and R - Represents Y aff The area of ​​1 and 0, N + and N - Respectively represent R + and R - The number of pixels in , A(i, j) is the element in the i-th row and j-th column of A.

[0160] L aff This can make the predicted semantic affinity matrix A more accurate. The learned high-confidence semantic affinity matrix can be further processed on the pseudo-label to obtain a more accurate label.

[0161] For example, for the learned semantic affinity matrix A, calculate the semantic transformation matrix T:

[0162] T=D -1 A α ,D(i,i)=∑ k A(i, j) α

[0163] α is a tuning parameter greater than 1 (e.g., it can be 2), which is used to remove smaller values ​​in A; D is a diagonal matrix, and D(i, i) is an element on the diagonal.

[0164] For example, the initial pseudo-label is processed according to the semantic transformation matrix T. For the second category activation map M∈R h ×w×k , first vectorize it into vec(M)∈R (h×w)×k ; Determine the activation map of the third category as:

[0165] M aff =T×vec(M)

[0166] In the above-described embodiment, it is possible to suppress the erroneously activated background region and expand the partially activated object region.

[0167] In some embodiments, a network loss function is performed with the training step. The network loss function may include a classification loss function, a segmentation loss function, and an affinity loss function. and affinity loss This has been introduced in the previous steps.

[0168] For segmentation loss First, a segmentation decoder (segmentation layer) is used to predict the feature map to obtain the pixel-level prediction of the image; then, the semantic affinity label is obtained based on the pixel-level prediction and the step of learning the semantic affinity from the attention matrix, and the cross entropy loss function is used to obtain

[0169] For example, the comprehensive loss function of the network is the weighted sum of the classification loss function, the segmentation loss function, and the affinity loss function:

[0170] L=L cls +λ1×L seg +λ2×L aff

[0171] λ1 and λ2 are preset weights between 0 and 1 (eg, 0.1).

[0172] In the above embodiment, the classification loss function, the segmentation loss function and the affinity loss function can all be implemented efficiently, so that efficient end-to-end training can be supported during the training phase, avoiding the high complexity model of the multi-stage method.

[0173] Figure 3a A flowchart illustrating some embodiments of the image processing method disclosed herein.

[0174] like Figure 3a As shown, in step 310, the global attention module of the machine learning model is used to determine the attention feature map of the image to be processed, and the machine learning model is trained by the training method in any of the above embodiments.

[0175] In step 320, semantic segmentation is performed on the image to be processed based on the attention feature map using the segmentation layer of the machine learning model to determine the segmentation result of the image to be processed.

[0176] Figure 3b Schematic diagrams showing some embodiments of the image processing method disclosed herein.

[0177] like Figure 3b As shown in the figure, during the test and inference phase, the trained machine learning model is actually tested and deployed. For example, for an input image to be processed, the Transformer-based backbone network is first used to extract the attention feature map; then the semantic segmentation layer is used to perform predictions, obtaining the semantic segmentation result as the final output.

[0178] Figure 4a A block diagram illustrating some embodiments of a training apparatus for a machine learning model of the present disclosure.

[0179] like Figure 4a As shown, the training device 4a of the machine learning model includes: a feature extraction unit 41a, which is used to use the global attention module of the machine learning model to determine the attention feature map of the sample image; a segmentation unit 42a, which is used to perform semantic segmentation on the sample image according to the attention feature map and the segmentation layer of the machine learning model, and determine the segmentation loss function according to the segmentation result; a training unit 43a, which is used to train the machine learning model according to the segmentation loss function.

[0180] In some embodiments, the feature extraction unit 41a predicts the semantic affinity between pixels in the sample image based on the attention matrix and the linear layer of the machine learning model, and determines the semantic affinity loss function based on the prediction results; the training unit 43a trains the machine learning model based on the segmentation loss function and the semantic affinity loss function.

[0181] In some embodiments, the training unit 43a determines the difference parameters between different pixels based on the differences between different pixels in the sample image; generates a first category activation map based on the attention feature map using the classification layer of the machine learning model; processes the first category activation map using the difference parameters to generate a second category activation map; determines the semantic affinity label of each pixel based on the second category activation map; and determines the semantic affinity loss function based on the semantic affinity label and the prediction result.

[0182] In some embodiments, the training unit 43a determines a first difference between a first pixel and a second pixel among different pixels; determines a second difference between the first pixel and its multiple adjacent pixels, where the adjacent pixels are determined from pixels within a preset radius centered on the first pixel; and determines a difference parameter between the first pixel and the second pixel based on the sum of the first difference and the second difference.

[0183] In some embodiments, the differences between different pixels include color differences and position differences. The training unit 43a determines the color difference parameters between different pixels based on the color differences; determines the position difference parameters between different pixels based on the position differences; and determines the difference parameters based on the weighted sum of the color difference parameters and the position difference parameters.

[0184] In some embodiments, the training unit 43a divides the second category activation map into multiple regions according to a preset threshold; sets different first pixel-level pseudo labels for pixels in different regions; and determines semantic affinity relationship labels based on the first pixel-level pseudo labels.

[0185] In some embodiments, multiple areas include an ignorable area, and when the first pixel and the second pixel in different pixels of the training unit 43a have the same first pixel-level pseudo-label, and the first pixel and the second pixel are not located in the ignorable area, the semantic affinity label is the first label; when the first pixel and the second pixel have different first pixel-level pseudo-labels, and the first pixel and the second pixel are not located in the ignorable area, the semantic affinity label is the second label, and the value of the first label is greater than the value of the second label; when at least one of the first pixel and the second pixel is located in the ignorable area, the semantic affinity label is the third label, and the value of the third label is greater than the value of the first label.

[0186] In some embodiments, the training unit 43a determines a semantic affinity matrix based on the attention matrix and its transposed matrix, wherein each element in the semantic affinity matrix corresponds to a semantic affinity parameter of each pixel; calculates a first difference between a preset value and the semantic affinity parameter of each pixel with a first label, and calculates the mean of all first differences as a first mean; calculates the mean of the semantic affinity parameters of all pixels with a second label as a second mean; and determines a semantic affinity loss function based on the weighted sum of the first mean and the second mean.

[0187] In some embodiments, the threshold includes a first threshold and a second threshold, and the training unit 43a divides the pixels corresponding to the values ​​less than or equal to the first threshold in the second category activation map into the background area; divides the pixels corresponding to the values ​​greater than or equal to the second threshold in the second category activation map into the foreground area; and divides the pixels corresponding to the values ​​less than the second threshold and greater than the first threshold in the second category activation map into the ignorable area.

[0188] In some embodiments, the training device 4a also includes a classification unit 44a, which is used to classify the sample image according to the attention feature map and the classification layer of the machine learning model; the training unit 43a determines the semantic affinity matrix according to the attention matrix and its transposed matrix, and each element in the semantic affinity matrix corresponds to the semantic affinity relationship parameter of each pixel; the training unit 43a determines the third category activation map according to the semantic affinity matrix and the second category activation map; the training unit 43a determines the second pixel-level pseudo-label of each pixel according to the third category activation map; the training unit determines the classification loss function according to the second pixel-level pseudo-label and the classification result; the training unit 43a trains the machine learning model according to the classification loss function, the segmentation loss function and the semantic affinity loss function.

[0189] In some embodiments, the training unit 43a generates a diagonal matrix based on the sum of the elements in each row of the semantic affinity matrix; determines a semantic transition matrix based on the diagonal matrix and the semantic affinity matrix; and determines a third category activation map based on the semantic transition matrix and the second category activation map.

[0190] In some embodiments, the classification unit 44a classifies the sample image according to the attention feature map and the classification layer of the machine learning model, and the training unit 43a determines the classification loss function according to the classification result; the training unit 43a trains the machine learning model according to the classification loss function, the segmentation loss function and the semantic affinity loss function.

[0191] In some embodiments, the feature extraction unit 41a divides the sample image into multiple image blocks; uses the global attention module to process the multiple image blocks to determine multiple feature vectors; and determines an attention feature map based on the multiple feature vectors.

[0192] Figure 4b A block diagram illustrating some embodiments of an image processing apparatus according to the present disclosure.

[0193] like Figure 4b As shown, the image processing device 4b includes: a feature extraction unit 41b, which is used to use the global attention module of the machine learning model to determine the attention feature map of the image to be processed, and the machine learning model is trained by the training method in any of the above embodiments; a segmentation unit 42b, which is used to perform semantic segmentation on the image to be processed according to the attention feature map and use the segmentation layer of the machine learning model to determine the segmentation result of the image to be processed.

[0194] Figure 5 A block diagram illustrating some embodiments of the electronic device of the present disclosure is shown.

[0195] like Figure 5As shown, the electronic device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51, and the processor 52 is configured to execute the training method of the machine learning model or the image processing method in any embodiment of the present disclosure based on the instructions stored in the memory 51.

[0196] The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0197] Figure 6 A block diagram showing other embodiments of the electronic device of the present disclosure.

[0198] like Figure 6 As shown, the electronic device 6 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, and the processor 620 is configured to execute the training method of the machine learning model or the image processing method in any of the aforementioned embodiments based on the instructions stored in the memory 610.

[0199] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0200] The electronic device 6 may further include an input / output interface 630, a network interface 640, a storage interface 650, and the like. These interfaces 630, 640, 650, as well as the memory 610 and the processor 620, may be connected, for example, via a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0201] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] Thus far, the machine learning model training method, image processing method, machine learning model training device, image processing device, electronic device, and non-volatile computer-readable storage medium according to the present disclosure have been described in detail. To avoid obscuring the concepts of the present disclosure, some details well known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solutions disclosed herein.

[0203] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0204] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for training a machine learning model, comprising: Determine an attention matrix and an attention feature map for a sample image using a global attention module of the machine learning model; According to the attention feature map, using the segmentation layer of the machine learning model, performing semantic segmentation on the sample image, and determining a segmentation loss function according to the segmentation result; Predicting the semantic affinity between pixels in the sample image using the linear layer of the machine learning model according to the attention matrix, and determining a semantic affinity loss function based on the prediction result; Classify the sample image according to the attention feature map using the classification layer of the machine learning model, and determine a classification loss function according to the classification result; Training the machine learning model according to the classification loss function, the segmentation loss function and the semantic affinity loss function, The step of determining the semantic affinity loss function based on the prediction results includes: determining difference parameters between different pixels according to the differences between different pixels in the sample image; Generating a first category activation map using a classification layer of the machine learning model according to the attention feature map; Processing the first class activation map using the difference parameter to generate a second class activation map; Determining a semantic affinity label for each pixel based on the second category activation map; The semantic affinity loss function is determined according to the semantic affinity label and the prediction result.

2. The training method according to claim 1, wherein: Determining the difference parameters between different pixels includes: determining a first difference between a first pixel and a second pixel of the different pixels; determining a second difference between the first pixel and a plurality of adjacent pixels thereof, the adjacent pixels being determined from pixels within a preset radius centered on the first pixel; A difference parameter between the first pixel and the second pixel is determined according to the sum of the first difference and the second difference.

3. The training method according to claim 1, wherein: The differences between the different pixels include color differences and position differences. Determining the difference parameters between different pixels includes: determining a color difference parameter between the different pixels according to the color difference; determining a position difference parameter between the different pixels according to the position difference; The difference parameter is determined according to a weighted sum of the color difference parameter and the position difference parameter.

4. The training method according to claim 1, wherein: Determining the semantic affinity label of each pixel according to the second category activation map includes: Dividing the second category activation map into a plurality of regions according to a preset threshold; Set different first pixel-level pseudo labels for pixels in different regions; The semantic affinity relationship label is determined according to the first pixel-level pseudo label.

5. The training method according to claim 4, wherein: The plurality of regions include a negligible region, Determining the semantic affinity label according to the first pixel-level pseudo label includes: When a first pixel and a second pixel among the different pixels have the same first pixel-level pseudo label, and neither the first pixel nor the second pixel is located in the ignorable area, the semantic affinity label is the first label; When the first pixel and the second pixel have different first pixel-level pseudo labels, and neither the first pixel nor the second pixel is located in the ignorable area, the semantic affinity label is the second label, and the value of the first label is greater than the value of the second label; In a case where at least one of the first pixel and the second pixel is located in the ignorable area, the semantic affinity label is a third label, and a value of the third label is greater than a value of the first label.

6. The training method according to claim 5, wherein: Determining the semantic affinity loss function includes: Determining a semantic affinity matrix based on the attention matrix and its transposed matrix, wherein each element in the semantic affinity matrix corresponds to a semantic affinity relationship parameter of each pixel; Calculating a first difference between a preset value and a semantic affinity parameter of each pixel having the first label, and calculating an average of all the first differences as a first average; Calculating a mean of the semantic affinity parameters of all pixels having the second label as a second mean; The semantic affinity loss function is determined according to a weighted sum of the first mean and the second mean.

7. The training method according to claim 4, wherein: The threshold includes a first threshold and a second threshold, the first threshold is smaller than the second threshold, The dividing the second category activation map into a plurality of regions according to a preset threshold comprises: Classify pixels corresponding to values ​​less than or equal to the first threshold in the second category activation map into the background area; Classifying pixels corresponding to values ​​greater than or equal to the second threshold in the second category activation map into the foreground area; Pixels corresponding to values ​​smaller than the second threshold and larger than the first threshold in the second category activation map are divided into an ignorable area.

8. The training method according to claim 1, further comprising: Classify the sample image using the classification layer of the machine learning model according to the attention feature map; Determining a semantic affinity matrix based on the attention matrix and its transposed matrix, wherein each element in the semantic affinity matrix corresponds to a semantic affinity relationship parameter of each pixel; determining a third category activation map according to the semantic affinity matrix and the second category activation map; Determining a second pixel-level pseudo label for each pixel according to the third category activation map; The classification loss function is determined according to the second pixel-level pseudo-label and the classification result.

9. The training method according to claim 8, wherein: Determining a third category activation map according to the semantic affinity matrix and the second category activation map includes: Generating a diagonal matrix according to the sum of the elements of each row of the semantic affinity matrix; Determining a semantic conversion matrix according to the diagonal matrix and the semantic affinity matrix; The third category activation map is determined according to the semantic transition matrix and the second category activation map.

10. The training method according to any one of claims 1 to 9, wherein: Determining the attention matrix and attention feature map of the sample image using the global attention module of the machine learning model includes: Dividing the sample image into a plurality of image blocks; Processing the plurality of image blocks using the global attention module to determine a plurality of feature vectors; Determine the attention feature map based on the multiple feature vectors.

11. A method for processing an image, comprising: Determining an attention feature map of the image to be processed using a global attention module of the machine learning model, wherein the machine learning model is trained using the training method of any one of claims 1 to 10; According to the attention feature map, the segmentation layer of the machine learning model is used to perform semantic segmentation on the image to be processed to determine the segmentation result of the image to be processed.

12. A training device for a machine learning model, comprising: a feature extraction unit, configured to determine an attention matrix and an attention feature map of a sample image using a global attention module of the machine learning model; a segmentation unit, configured to perform semantic segmentation on the sample image according to the attention feature map and using the segmentation layer of the machine learning model, and determine a segmentation loss function according to the segmentation result; A semantic affinity unit, configured to predict the semantic affinity relationship between pixels in the sample image according to the attention matrix and using the linear layer of the machine learning model, and determine a semantic affinity loss function according to the prediction result; a classification unit, configured to classify the sample image according to the attention feature map and using the classification layer of the machine learning model, and determine a classification loss function according to the classification result; a training unit, configured to train the machine learning model according to the classification loss function, the segmentation loss function, and the semantic affinity loss function, The training unit determines a difference parameter between different pixels according to the difference between different pixels in the sample image; generates a first category activation map based on the attention feature map using the classification layer of the machine learning model; processes the first category activation map using the difference parameter to generate a second category activation map; and determines a semantic affinity label for each pixel based on the second category activation map; The semantic affinity loss function is determined according to the semantic affinity label and the prediction result.

13. An image processing device, comprising: a feature extraction unit, configured to determine an attention feature map of the image to be processed using a global attention module of the machine learning model, wherein the machine learning model is trained using the training method according to any one of claims 1 to 10; A segmentation unit is used to perform semantic segmentation on the image to be processed based on the attention feature map and utilize the segmentation layer of the machine learning model to determine the segmentation result of the image to be processed.

14. An electronic device comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the method for training a machine learning model according to any one of claims 1 to 10, or the method for processing an image according to claim 11, based on instructions stored in the memory.

15. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for training a machine learning model according to any one of claims 1 to 10, or the method for processing an image according to claim 11.

Images