Image semantic segmentation model training method, image semantic segmentation method and device

Abstract

This application relates to a method for training an image semantic segmentation model. Based on the ground truth labels of all pixels in the image to be segmented, edge detection is performed on the image to be segmented to obtain object edge images for each category. Image prediction results are obtained, and the gradient of each pixel in the prediction results is calculated to generate a pixel gradient map. The arctangent gradient map of each pixel is then obtained from the pixel gradient map. Based on the object edge images and the arctangent gradient maps, an auxiliary loss value is obtained for each pixel in the image to be segmented, predicted by the image semantic segmentation model to be trained. The overall loss value for each pixel in each category of the image to be segmented is calculated based on the auxiliary loss value and the loss value obtained from the main loss function. Based on the overall loss value, the parameters of the image semantic segmentation model are adjusted to obtain the trained image semantic segmentation model. This application can improve the training effect of the image semantic segmentation model and increase the accuracy of semantic segmentation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image semantic segmentation model training method, an image semantic segmentation method and apparatus. Background Technology

[0002] Image semantic segmentation is a crucial part of image processing and machine vision technology, focusing on image understanding, and is also an important branch of AI (Artificial Intelligence). Image semantic segmentation involves classifying each pixel in an image, determining its category (e.g., background, person, or vehicle), and thus dividing the image into regions. Currently, image semantic segmentation is widely used in scenarios such as autonomous driving.

[0003] Related techniques, when training image semantic segmentation models using deep learning, employ loss functions such as cross-entropy to calculate loss values. These loss values ​​are then used to adjust the parameters of the image semantic segmentation model, completing the training process. However, these techniques often use a global averaging approach when calculating the loss value using cross-entropy and similar loss functions. This means that even with a small number of poorly performing pixels (i.e., a few with large summation values), the final loss value will still be relatively small. In practical applications, it is observed that edge detection results exhibit a high number of false edge detections, meaning that pixels at edges are easily misclassified. Semantic segmentation of slender objects is also poor, leading to inaccurate loss values ​​obtained using cross-entropy and similar loss functions, resulting in poor training performance for the image semantic segmentation model. Consequently, the trained image semantic segmentation model exhibits inaccurate semantic segmentation of slender objects, resulting in low accuracy in semantic segmentation of such objects. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides an image semantic segmentation model training method, an image semantic segmentation method, and an apparatus, which can improve the training effect of the image semantic segmentation model and increase the semantic segmentation accuracy of the image semantic segmentation model.

[0005] The first aspect of this application provides a method for training an image semantic segmentation model, characterized in that it includes:

[0006] Based on the ground truth values ​​of the labels of all pixels in the image to be segmented, edge detection is performed on the image to be segmented to obtain the object edge image of each category in the image to be segmented;

[0007] Obtain the image prediction result, calculate the gradient of each pixel in the prediction result based on the image prediction result, generate a pixel gradient map, and obtain the gradient arctangent map of each pixel based on the pixel gradient map;

[0008] Based on the object edge image and the gradient arctangent map, the image semantic segmentation model to be trained is used to predict the auxiliary loss value corresponding to each pixel in the image to be segmented.

[0009] The overall loss value for each pixel of each category in the image to be segmented is calculated based on the auxiliary loss value and the loss value obtained from the main loss function;

[0010] Based on the overall loss value, adjust the parameters of the image semantic segmentation model to obtain the trained image semantic segmentation model.

[0011] A second aspect of this application provides an image semantic segmentation method, characterized in that it comprises: [the method is described in the original text].

[0012] Obtain the image to be segmented;

[0013] The image to be segmented is input into the image semantic segmentation model to obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained by the image semantic segmentation model training method described above.

[0014] A third aspect of this application provides an image semantic segmentation model training apparatus, characterized in that it comprises:

[0015] The first processing module is used to perform edge detection on the image to be segmented based on the true values ​​of the labels of all pixels in the image to be segmented, so as to obtain the object edge image of each category in the image to be segmented.

[0016] The second processing module is used to obtain the image prediction result, calculate the gradient of each pixel in the prediction result based on the image prediction result, generate a pixel gradient map, and obtain the gradient arctangent map of each pixel based on the pixel gradient map.

[0017] The third processing module is used to obtain the auxiliary loss value corresponding to each pixel in the image to be segmented based on the object edge image and the gradient arctangent map;

[0018] The fourth processing module is used to calculate the overall loss value of each pixel of each category in the image to be segmented based on the auxiliary loss value and the loss value obtained from the main loss function;

[0019] The fifth processing module is used to adjust the parameters of the image semantic segmentation model according to the overall loss value to obtain the trained image semantic segmentation model.

[0020] A fourth aspect of this application provides an image semantic segmentation apparatus, the apparatus comprising:

[0021] The image acquisition module is used to acquire the image to be segmented;

[0022] The image segmentation module is used to input the image to be segmented into the image semantic segmentation model and obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained by the image semantic segmentation model training device described above.

[0023] The fifth aspect of this application provides an electronic device, comprising:

[0024] Processor; and

[0025] The memory stores executable code, which, when executed by the processor, causes the processor to perform the image semantic segmentation model training method and the image semantic segmentation method described above.

[0026] The sixth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the image semantic segmentation model training method and the image semantic segmentation method described above.

[0027] The technical solution provided in this application may include the following beneficial effects:

[0028] The technical solution of this application involves edge detection of the image to be segmented based on the ground truth labels of all pixels in the image to be segmented, obtaining object edge images for each category of the image to be segmented; acquiring image prediction results, and calculating the gradient of each pixel in the prediction results to generate a pixel gradient map, and obtaining the arctangent gradient map of each pixel based on the pixel gradient map; obtaining the auxiliary loss value corresponding to each pixel in the image to be segmented based on the object edge images and the arctangent gradient map; calculating the overall loss value of each pixel of each category in the image to be segmented based on the auxiliary loss value and the loss value obtained by the main loss function; and adjusting the parameters of the image semantic segmentation model based on the overall loss value to obtain the trained image semantic segmentation model. This application determines the overall loss value by using an auxiliary loss function and the original main loss function, which can improve the training effect of the image semantic segmentation model, enabling the image semantic segmentation model to more accurately predict each pixel of each category in the image, thereby improving the semantic segmentation accuracy of the image semantic segmentation model.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0030] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0031] Figure 1 This is a schematic flowchart illustrating the image semantic segmentation model training method in an embodiment of this application;

[0032] Figure 2 This is another flowchart illustrating the image semantic segmentation model training method shown in the embodiments of this application;

[0033] Figure 3 This is a schematic diagram of the truth value of the label with category index K shown in the embodiments of this application;

[0034] Figure 4 This is a schematic diagram illustrating the true values ​​of the labels of neighboring pixels in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram illustrating the true values ​​of the image pixel value gradient in an embodiment of this application;

[0036] Figure 6 This is a schematic flowchart of an image semantic segmentation method shown in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of the structure of an image semantic segmentation model training device shown in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of the structure of an image semantic segmentation apparatus shown in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0040] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0041] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0043] In related technologies, the cross-entropy loss function can be expressed by the following formula:

[0044]

[0045] Where N is the total number of pixels in the sample image; C is the number of sample classes; and y is the truth value of the pixel belonging to that class, i.e., if the class id of a pixel is k, then

[0046]

[0047] p represents the probability that the pixel belongs to a certain category as predicted by the network. Its value ranges from [0,1], and the sum of the probabilities of all categories for the same pixel is 1.0, i.e.

[0048]

[0049] Because the cross-entropy loss function uses a global averaging approach, the final Lce value will still be relatively small when there are a few poorly performing pixels (i.e., a few pixels with large summation values). In practical use, it is found that edge detection results have a high number of false detections, meaning that it is easy to misclassify pixels at edge locations; the detection effect is particularly poor for long and thin objects, which are essentially edges everywhere.

[0050] This application improves upon the existing Cross Entropy (CE) loss function, thereby enhancing the training performance and semantic segmentation accuracy of image semantic segmentation models.

[0051] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0052] Figure 1 This is a schematic flowchart illustrating the image semantic segmentation model training method in an embodiment of this application.

[0053] See Figure 1 A method for training an image semantic segmentation model, comprising:

[0054] S110: Based on the ground truth values ​​of the labels of all pixels in the image to be segmented, perform edge detection on the image to be segmented to obtain the object edge images of each category in the image to be segmented.

[0055] In a specific embodiment, edge detection is performed on the image to be segmented based on the ground truth values ​​of the labels for all pixels of each category in the training set image, resulting in object edge images for each category of the image to be segmented. The object edge positions for each category of the image to be segmented can be detected based on the ground truth values ​​of the pixel labels, thus determining the object edge images for each category of the image to be segmented.

[0056] S120: Obtain the image prediction result, calculate the gradient of each pixel in the prediction result based on the image prediction result, generate a pixel gradient map, and obtain the gradient arctangent map of each pixel based on the pixel gradient map.

[0057] In a specific embodiment, based on the image prediction result, the gradient of each pixel in the prediction result is calculated to generate a pixel gradient map, and the arctangent map of the gradient of each pixel is obtained based on the pixel gradient map, including:

[0058] Based on the image prediction results, the gradient of each pixel in the prediction results is calculated using the Sobel operator to generate a pixel gradient map;

[0059] The arctangent map of the gradient of each pixel is obtained by a preset calculation formula based on the pixel gradient value. The preset calculation formula is ArctanMap(i,j,c)=arctan(α×Grad(i,j,c)), where α is a hyperparameter and Grad(i,j,c) is the pixel gradient map. G x (i,j,c) and G y (i,j,c) represent the gradient maps in the x and y directions, respectively.

[0060] S130: Based on the object edge image and the gradient arctangent map, obtain the image semantic segmentation model to be trained and predict the auxiliary loss value corresponding to each pixel in the image to be segmented.

[0061] S140: Calculate the overall loss value for each pixel of each category in the image to be segmented based on the auxiliary loss value and the loss value obtained from the main loss function.

[0062] S150: Adjust the parameters of the image semantic segmentation model based on the overall loss value to obtain a trained image semantic segmentation model.

[0063] In a specific embodiment, the auxiliary loss value corresponding to each pixel in the image to be segmented can be predicted by the image semantic segmentation model to be trained based on the object edge image and the gradient arctangent map; the overall loss value of each pixel of each category in the image to be segmented can be calculated based on the loss value obtained by the auxiliary loss value and the main loss function; then, the parameters of the image semantic segmentation model can be adjusted based on the overall loss value predicted by the image semantic segmentation model to obtain the trained image semantic segmentation model.

[0064] The image semantic segmentation model training method illustrated in this application involves: performing edge detection on the image to be segmented based on the ground truth labels of all pixels in the image to be segmented, obtaining object edge images for each category of the image to be segmented; acquiring image prediction results and calculating the gradient of each pixel in the prediction results to generate a pixel gradient map, and obtaining the arctangent gradient map of each pixel based on the pixel gradient map; obtaining the auxiliary loss value corresponding to each pixel in the image to be segmented predicted by the image semantic segmentation model to be trained based on the object edge images and the arctangent gradient maps; calculating the overall loss value of each pixel of each category in the image to be segmented based on the auxiliary loss value and the loss value obtained by the main loss function; and adjusting the parameters of the image semantic segmentation model based on the overall loss value to obtain the trained image semantic segmentation model. This application determines the overall loss value by using an auxiliary loss function and the original main loss function, which can improve the training effect of the image semantic segmentation model, enabling the image semantic segmentation model to more accurately predict each pixel of each category in the image, thereby improving the semantic segmentation accuracy of the image semantic segmentation model.

[0065] Figure 2 This is another schematic diagram of the image semantic segmentation model training method shown in the embodiments of this application. Figure 2 Compared to Figure 1 The scheme of this application is described in more detail.

[0066] See Figure 2 A method for training an image semantic segmentation model, comprising:

[0067] S210: Obtain the number of pixel categories in the image to be segmented based on the labeled image of the image to be segmented.

[0068] In a specific embodiment, a labeling tool is used to correctly label pixels in different regions or for different targets in the training set images, obtaining a labeled image (Ground Truth) with correctly labeled data for different pixels in the image. Based on the set values ​​of all pixels in the labeled image of the image to be segmented, duplicate set values ​​are removed to obtain the number of unique set values. The number of unique set values ​​is the number of pixel categories in the image. For example, if the number of unique set values ​​is 3 (i.e., there are 3 different set values), then the number of pixel categories in the image is 3.

[0069] In a specific embodiment, the pixel setting value can be a pixel value or a value set according to a set rule.

[0070] S220: Obtain the image histogram based on the image annotation and the number of pixel categories.

[0071] In a specific embodiment, encoding methods such as one-hot encoding can be used to encode each category of pixels in the labeled image according to the number of pixel categories in the image, thereby obtaining the image histogram.

[0072] S230: Based on the labeled data of the image to be segmented, obtain the true label values ​​of all pixels of each category in the image.

[0073] In a specific embodiment, for an image of size W x H, its label truth value can be represented as a three-dimensional matrix of size W x H x C, denoted as Label, where C is the number of categories. Elements in Label are either 1 or 0. For category k, Label[:,:,k] (i.e., the probability truth value of the k-th category for all pixels, with a shape of W x H) can be extracted. Assuming the image contains an object of category k, then '1' in a continuous area will form the pattern of that object. Figure 3 As shown in the example, the 1s in the matrix form a continuous area of ​​shaded region. In specific embodiments, the edge image in the figure can be found using various numerical methods.

[0074] S240: Based on the label ground value method of neighboring pixels or the image pixel value gradient method, perform edge detection on the label ground values ​​of all pixels in the image to be segmented to obtain the object edge image of each category in the image to be segmented.

[0075] In a specific embodiment, edge detection can be performed by determining whether there are unequal values ​​in the neighborhood. Specifically, a multidimensional matrix of the same size as the Label, denoted as EdgeMap, is generated, and the values ​​of its internal elements are filled with truth values ​​using formula (I):

[0076]

[0077] It should be noted that when a pixel has a value different from the ground truth value within its neighborhood of category c, that point is considered a boundary point, and its value is 1. When B=1, the criterion is an 8-neighborhood. The EdgeMap generated based on the above example is as follows: Figure 4 As shown, the first part represents the inner edge of the object, and the second part represents the outer edge. The range for edge detection can also be increased, i.e., B = 2, 3, ..., becoming a 15-neighborhood or a 24-neighborhood, thus increasing the width of the edge.

[0078] In specific embodiments, edge detection can also be performed based on the image pixel value gradient method. Specifically, the Sobel operator is a common operator for calculating the gradient of image pixel values. It contains operators in two directions, which can calculate the gradient in the horizontal (x-direction) and vertical (y-direction) directions respectively. For a Sobel operator of size (2k+1)x(2k+1), the specific values ​​of its internal elements are as follows:

[0079]

[0080] K y (i,j)=K x (j,i)

[0081] Co is a fixed coefficient, the smallest coefficient that makes the elements in Kx and Ky integers.

[0082] When k=1, according to the above formula, Co=2, and the 3x3 Soble operator is obtained as follows:

[0083]

[0084]

[0085] Where Kx is the convolution kernel used to calculate the gradient in the x-direction, and Ky is the convolution kernel used to calculate the gradient in the y-direction. By directly performing a convolution operation on the Label using these two kernels, we can obtain the gradient maps in the x and y directions, as shown in the formula:

[0086]

[0087]

[0088] Then the gradient map of the whole system is calculated.

[0089]

[0090] Since a Grad graph only has non-zero values ​​at its edges, it can be considered equivalent to an EdgeMap here. The results calculated from the example are as follows... Figure 5As shown. In other embodiments, Sobel operators of size 5x5 or 7x7 can also be used, in the same way, but the resulting edge width will be wider than that.

[0091] S250: Based on the set loss function, obtain the prediction loss value of each pixel of each category in the image predicted by the image semantic segmentation model.

[0092] In a specific embodiment, the image is input into the image semantic segmentation model to be trained. Based on the input image, the image semantic segmentation model predicts the n categories of pixels in the image through n channels, outputting the prediction results for each category of pixels. Based on the prediction results of each category of pixels in the image and the labeled image, a set main loss function, such as the cross-entropy loss function, is used to calculate the prediction loss value of each category of pixels in the image.

[0093] S260: Obtain the image prediction result, calculate the gradient of each pixel in the prediction result based on the image prediction result, generate a pixel gradient map, and obtain the gradient arctangent map of each pixel based on the pixel gradient map.

[0094] In a specific embodiment, the pixel gradient of the prediction result can be calculated using the Sobel operator. The gradients in the x and y directions are obtained through image convolution, as shown in the following equation:

[0095]

[0096]

[0097] Where Kx and Ky are the convolution kernels of the Sobel operator, Pred is the prediction result of the network, and Pred(I,j,c) represents the probability that the pixel coordinate (I,j) in the image is c.

[0098] The gradient map is further calculated from Gx and Gy:

[0099]

[0100] In other specific embodiments, Sobel operators of size 5x5 or 7x7 can also be used, and the usage is the same.

[0101] Then, obtain the ArctanMap based on the Grad graph:

[0102] ArctanMap(i,j,c)=arctan(α×Grad(i,j,c))

[0103] Here, α is a hyperparameter that can be adjusted based on the test results. In a specific test, α can be set to 2.

[0104] S270: Based on the object edge image and the gradient arctangent map, the image semantic segmentation model to be trained predicts the auxiliary loss value corresponding to each pixel in the image to be segmented;

[0105] In a specific embodiment, the auxiliary loss value can be calculated according to the following formula:

[0106]

[0107] Where N represents the element number in the ArctanMap and EdgeMap matrices, and k is calculated by the following formula to simplify the representation in the formula:

[0108]

[0109] S280: Calculate the overall loss value for each pixel of each category in the image to be segmented based on the auxiliary loss value and the loss value obtained from the main loss function.

[0110] It should be noted that the auxiliary loss function cannot be used alone; it must be used in conjunction with the main loss function to improve its effectiveness. In this embodiment, the cross-entropy loss function is used as an example:

[0111] L=βL CE +γL AG

[0112] Here, β and γ are hyperparameters that can be adjusted based on the training and testing results. In a specific implementation, the parameters that can be used are β = 1 and γ = 1. Similarly, γ can be gradually increased according to the training epochs, for example, γ = epoch * 0.005.

[0113] S290: Adjust the parameters of the image semantic segmentation model based on the overall loss value to obtain a trained image semantic segmentation model.

[0114] In a specific embodiment, the loss values ​​of each category of pixels can be backpropagated in the image semantic segmentation model to be trained, and the parameters of each channel of the image semantic segmentation model to be trained can be adjusted to determine the parameters of the image semantic segmentation model. Steps 210 to 270 can be iteratively adjusted to adjust the parameters of each channel of the image semantic segmentation model to be trained until the loss value of the prediction result of each channel of the image semantic segmentation model to be trained tends to stabilize or is less than a set loss threshold, at which point the training of the image semantic segmentation model is stopped, and the trained image semantic segmentation model is obtained.

[0115] In another specific embodiment, the loss value of each pixel in each category of the image can be predicted by the image semantic segmentation model to obtain the average loss value of the image to be segmented. Based on the average loss value, the parameters of the image semantic segmentation model are adjusted to obtain a trained image semantic segmentation model. The loss values ​​of each category of pixels in the image predicted by the image semantic segmentation model are summed to obtain the cumulative loss value of each category of pixels in the image predicted by the image semantic segmentation model. The cumulative loss value is divided by the number of pixel categories in the image to obtain the average loss value of the image semantic segmentation model. The average loss value is backpropagated in the image semantic segmentation model to be trained, and the parameters of each channel of the image semantic segmentation model to be trained are adjusted to determine the parameters of the image semantic segmentation model. Steps 210 to 270 can be iteratively adjusted to adjust the parameters of the image semantic segmentation model to be trained until the average loss value of the image semantic segmentation model to be trained tends to stabilize or is less than a set loss threshold, at which point the training of the image semantic segmentation model is stopped, and a trained image semantic segmentation model is obtained.

[0116] The image semantic segmentation model training method illustrated in this application involves: performing edge detection on the image to be segmented based on the ground truth labels of all pixels in the image to be segmented, obtaining object edge images for each category of the image to be segmented; acquiring image prediction results and calculating the gradient of each pixel in the prediction results to generate a pixel gradient map, and obtaining the arctangent gradient map of each pixel based on the pixel gradient map; obtaining the auxiliary loss value corresponding to each pixel in the image to be segmented predicted by the image semantic segmentation model to be trained based on the object edge images and the arctangent gradient maps; calculating the overall loss value of each pixel of each category in the image to be segmented based on the auxiliary loss value and the loss value obtained by the main loss function; and adjusting the parameters of the image semantic segmentation model based on the overall loss value to obtain the trained image semantic segmentation model. This application determines the overall loss value by using an auxiliary loss function and the original main loss function, which can improve the training effect of the image semantic segmentation model, enabling the image semantic segmentation model to more accurately predict each pixel of each category in the image, thereby improving the semantic segmentation accuracy of the image semantic segmentation model.

[0117] This application also provides an image semantic segmentation method.

[0118] Figure 6 This is a flowchart illustrating the image semantic segmentation method shown in the embodiments of this application.

[0119] See Figure 6 An image semantic segmentation method, comprising:

[0120] S610: Obtain the image to be segmented.

[0121] In one embodiment, the device can receive images to be segmented input by a user, or images to be segmented sent by other devices; it can also acquire images to be segmented through an image acquisition device.

[0122] S620: Input the image to be segmented into the image semantic segmentation model and obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model.

[0123] In a specific embodiment, the image to be segmented can be input into an image semantic segmentation model. The image semantic segmentation model predicts and outputs for each pixel of each category in the image to be segmented. Based on the predicted outputs of all pixels of each category in the image to be segmented, the segmentation result corresponding to the image to be segmented is output; thus, the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model is obtained. The image semantic segmentation model can be based on the above... Figure 1 , Figure 2 The image semantic segmentation model was trained using the given training method.

[0124] The image semantic segmentation method shown in this application focuses more on the position of object edges. It adjusts the weight of each pixel of the object edge in the image to be segmented, which can accurately predict each pixel of each category in the image and improve the accuracy of image semantic segmentation.

[0125] Corresponding to the aforementioned application function implementation method embodiments, this application also provides an image semantic segmentation model training device, an image semantic segmentation device, an electronic device, and corresponding embodiments.

[0126] Figure 7 This is a schematic diagram of the structure of an image semantic segmentation model training device shown in an embodiment of this application.

[0127] See Figure 7 An image semantic segmentation model training device includes a first processing module 701, a second processing module 702, a third processing module 703, a fourth processing module 704, and a fifth processing module 705.

[0128] The first processing module 701 is used to perform edge detection on the image to be segmented based on the label ground values ​​of all pixels in the image to be segmented, so as to obtain the object edge image of each category in the image to be segmented.

[0129] The second processing module 702 is used to acquire the image prediction result, calculate the gradient of each pixel in the prediction result based on the image prediction result, generate a pixel gradient map, and obtain the gradient arctangent map of each pixel based on the pixel gradient map.

[0130] The third processing module 703 is used to obtain the auxiliary loss value corresponding to each pixel in the image to be segmented based on the object edge image and the gradient arctangent map;

[0131] The fourth processing module 704 is used to calculate the overall loss value of each pixel of each category in the image to be segmented based on the auxiliary loss value and the loss value obtained by the main loss function;

[0132] The fifth processing module 705 is used to adjust the parameters of the image semantic segmentation model according to the overall loss value to obtain the trained image semantic segmentation model.

[0133] Figure 8 This is a schematic diagram of the structure of an image semantic segmentation apparatus shown in an embodiment of this application.

[0134] See Figure 8 An image semantic segmentation device 800 includes an image acquisition module 801 and an image segmentation module 802.

[0135] Image acquisition module 801 is used to acquire the image to be segmented.

[0136] The image segmentation module 802 is used to input the image to be segmented acquired by the image acquisition module 801 into the image semantic segmentation model, and obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained according to the image semantic segmentation model training device described above.

[0137] The technical solution shown in this application focuses more on the position of object edges. It adjusts the weight of each pixel of the object edge in the image to be segmented, which can accurately predict each pixel of each category in the image and improve the accuracy of image semantic segmentation.

[0138] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0139] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0140] See Figure 9 The electronic device 900 includes a memory 901 and a processor 902.

[0141] The processor 902 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0142] Memory 901 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 902 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 901 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 901 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, a high-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0143] The memory 901 stores executable code, which, when processed by the processor 902, can cause the processor 902 to execute part or all of the methods described above.

[0144] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0145] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0146] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training an image semantic segmentation model, characterized in that, The method comprises the following steps: performing edge detection on the image to be segmented according to the label true value of all pixels in the image to be segmented, to obtain an object edge image of each category of the image to be segmented; obtaining an image prediction result, calculating the gradient of each pixel in the prediction result according to the image prediction result, generating a pixel gradient map, and obtaining a gradient inverse tangent map of each pixel according to the pixel gradient map; obtaining an auxiliary loss value corresponding to each pixel in the image to be segmented by the image semantic segmentation model to be trained according to the object edge image and the gradient inverse tangent map; calculating the overall loss value of each pixel of each category in the image to be segmented according to the auxiliary loss value and the loss value obtained by the main loss function; adjusting the parameters of the image semantic segmentation model according to the overall loss value, to obtain a trained image semantic segmentation model; the auxiliary loss value corresponding to each pixel in the image to be segmented is obtained by the image semantic segmentation model to be trained according to the object edge image and the gradient inverse tangent map, which comprises: the auxiliary loss value is calculated by the following formula: ； Wherein, the is the auxiliary loss value, the is the gradient inverse tangent graph, the is the object edge image, the N is the and the number of elements in the matrix, the k is calculated by the following formula: ； the main loss function is a cross-entropy loss function.

2. The method of claim 1, wherein, the label true value of all pixels in the image to be segmented is obtained according to the annotation data of the image to be segmented; the label true value of all pixels in the image to be segmented is edge detected according to the label true value method of adjacent pixels or the image pixel value gradient method, to obtain an object edge image of each category of the image to be segmented. the gradient of each pixel in the prediction result is calculated according to the image prediction result, a pixel gradient map is generated, and a gradient inverse tangent map of each pixel is obtained according to the pixel gradient map, which comprises:

3. The method of claim 1, wherein, the gradient of each pixel in the prediction result is calculated according to the image prediction result, a pixel gradient map is generated, and a gradient inverse tangent map of each pixel is obtained according to the pixel gradient map, which comprises: the gradient of each pixel in the prediction result is calculated according to the image prediction result, a pixel gradient map is generated, and a gradient inverse tangent map of each pixel is obtained according to the pixel gradient map, which comprises: According to the pixel gradient value, the gradient arctangent graph of each pixel is obtained by a pre-designed calculation formula wherein, a is a hyperparameter, is a pixel gradient graph, , and are gradient graphs in x and y directions respectively.

4. The method of claim 1, wherein, the average loss value of the image to be segmented predicted by the image semantic segmentation model is obtained by predicting the loss value of each pixel of each category in the image by the image semantic segmentation model; the average loss value of the image to be segmented predicted by the image semantic segmentation model is obtained by predicting the loss value of each pixel of each category in the image by the image semantic segmentation model; The method comprises the following steps:

5. An image semantic segmentation method, characterized in that, obtaining an image to be segmented; inputting the image to be segmented into an image semantic segmentation model to obtain a segmentation result corresponding to the image to be segmented output by the image semantic segmentation model, wherein the image semantic segmentation model is trained by the image semantic segmentation model training method of any one of claims 1-4. The method comprises the following steps:

6. An image semantic segmentation model training apparatus, characterized in that, a first processing module is configured to perform edge detection on the image to be segmented according to the label true value of all pixels in the image to be segmented, to obtain an object edge image of each category of the image to be segmented; ​ The second processing module is configured to obtain an image prediction result, calculate a gradient of each pixel in the prediction result according to the image prediction result, generate a pixel gradient map, and obtain a gradient inverse tangent map of each pixel according to the pixel gradient map. The third processing module is configured to obtain an auxiliary loss value corresponding to each pixel in the image to be segmented by using the image semantic segmentation model to be trained according to the object edge image and the gradient inverse tangent map. The fourth processing module is configured to calculate an overall loss value of each pixel of each category in the image to be segmented according to the auxiliary loss value and a loss value obtained by using the main loss function. The fifth processing module is configured to adjust parameters of the image semantic segmentation model according to the overall loss value, and obtain a trained image semantic segmentation model. The third processing module is configured to: calculate the auxiliary loss value by using the following formula: ； Wherein the is the auxiliary loss value, the is the gradient inverse tangent diagram, the is the object edge image, the N is the and the number of elements in the matrix, the k is calculated by the following formula: ； The main loss function is a cross-entropy loss function.

7. The apparatus of claim 6, wherein, The first processing module is specifically configured to: obtain label true values of all pixels in the image to be segmented according to label data of the image to be segmented; perform edge detection on the label true values of all pixels in the image to be segmented according to a label true value method of a neighborhood pixel or an image pixel value gradient method, and obtain an object edge image of each category of the image to be segmented.

8. An image semantic segmentation apparatus, characterized by comprising: The image semantic segmentation model training device comprises: an image acquisition module configured to acquire an image to be segmented; an image segmentation module configured to input the image to be segmented into an image semantic segmentation model, and obtain a segmentation result corresponding to the image to be segmented and output by the image semantic segmentation model, wherein the image semantic segmentation model is trained by using the image semantic segmentation model training device according to any one of claims 6-7.

9. An electronic device, comprising: The image semantic segmentation model training device comprises: a processor; and a memory having executable code stored thereon, wherein when the executable code is executed by the processor, the processor is caused to execute the image semantic segmentation model training method according to any one of claims 1-4 and the image semantic segmentation method according to claim 5.

10. A computer-readable storage medium, characterized in that, The memory having executable code stored thereon, wherein when the executable code is executed by the processor of the electronic device, the processor is caused to execute the image semantic segmentation model training method according to any one of claims 1-4 and the image semantic segmentation method according to claim 5.

Images