Image semantic segmentation model training method, image semantic segmentation method and device

Abstract

The application relates to an image semantic segmentation model training method, an image semantic segmentation method and device. The method comprises the following steps: performing edge detection on a to-be-segmented image according to label true values of all pixels in the to-be-segmented image, to obtain object edge images of each category of the to-be-segmented image; obtaining weights corresponding to each pixel of each category in the to-be-segmented image predicted by a to-be-trained image semantic segmentation model according to the object edge images of each category of the to-be-segmented image; obtaining loss values of each pixel of each category in the to-be-segmented image predicted by the image semantic segmentation model according to the weights corresponding to each pixel of each category in the to-be-segmented image; and adjusting parameters of the image semantic segmentation model according to the loss values, to obtain a trained image semantic segmentation model. The scheme provided by the application can improve the training effect of the image semantic segmentation model and improve the semantic segmentation accuracy of the image semantic segmentation model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image semantic segmentation model training method, an image semantic segmentation method and apparatus. Background Technology

[0002] Image semantic segmentation is a crucial part of image processing and machine vision technology, focusing on image understanding, and is also an important branch of AI (Artificial Intelligence). Image semantic segmentation involves classifying each pixel in an image, determining its category (e.g., background, person, or vehicle), and thus dividing the image into regions. Currently, image semantic segmentation is widely used in scenarios such as autonomous driving.

[0003] Related techniques, when training image semantic segmentation models using deep learning, employ loss functions such as cross-entropy to calculate loss values. These loss values ​​are then used to adjust the parameters of the image semantic segmentation model, completing the training process. However, these techniques often use a global averaging approach when calculating the loss value using cross-entropy and similar loss functions. This means that even with a small number of poorly performing pixels (i.e., a few with large summation values), the final loss value will still be relatively small. In practical applications, it is observed that edge detection results exhibit a high number of false edge detections, meaning that pixels at edges are easily misclassified. Semantic segmentation of slender objects is also poor, leading to inaccurate loss values ​​obtained using cross-entropy and similar loss functions, resulting in poor training performance for the image semantic segmentation model. Consequently, the trained image semantic segmentation model exhibits inaccurate semantic segmentation of slender objects, resulting in low accuracy in semantic segmentation of such objects. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides an image semantic segmentation model training method, an image semantic segmentation method, and an apparatus, which can improve the training effect of the image semantic segmentation model and increase the semantic segmentation accuracy of the image semantic segmentation model.

[0005] The first aspect of this application provides a method for training an image semantic segmentation model, including:

[0006] Based on the ground truth values ​​of the labels of all pixels in the image to be segmented, edge detection is performed on the image to be segmented to obtain the object edge image of each category in the image to be segmented;

[0007] Based on the object edge images of each category in the image to be segmented, the weights corresponding to each pixel of each category in the image to be segmented are predicted by the image semantic segmentation model to be trained.

[0008] Based on the weights corresponding to each pixel of each category in the image to be segmented, the loss value of the image semantic segmentation model for predicting each pixel of each category in the image to be segmented is obtained.

[0009] Based on the loss value, the parameters of the image semantic segmentation model are adjusted to obtain a trained image semantic segmentation model.

[0010] A second aspect of this application provides an image semantic segmentation method, comprising:

[0011] Obtain the image to be segmented;

[0012] The image to be segmented is input into the image semantic segmentation model to obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained according to the image semantic segmentation model training method described above.

[0013] A third aspect of this application provides an image semantic segmentation model training apparatus, the apparatus comprising:

[0014] The first processing module is used to perform edge detection on the image to be segmented based on the true values ​​of the labels of all pixels in the image to be segmented, so as to obtain the object edge image of each category in the image to be segmented.

[0015] The second processing module is used to obtain the weights corresponding to each pixel of each category in the image to be segmented based on the object edge images of each category in the image to be segmented;

[0016] The third processing module is used to obtain the loss value of each pixel of each category in the image to be segmented by the image semantic segmentation model based on the weight corresponding to each pixel of each category in the image to be segmented.

[0017] The fourth processing module is used to adjust the parameters of the image semantic segmentation model according to the loss value to obtain the trained image semantic segmentation model.

[0018] A fourth aspect of this application provides an image semantic segmentation apparatus, the apparatus comprising:

[0019] The image acquisition module is used to acquire the image to be segmented;

[0020] The image segmentation module is used to input the image to be segmented into the image semantic segmentation model and obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained by the image semantic segmentation model training device described above.

[0021] The fifth aspect of this application provides an electronic device, comprising:

[0022] Processor; and

[0023] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0024] A sixth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0025] The technical solution provided in this application may include the following beneficial effects:

[0026] The technical solution of this application involves edge detection of the image to be segmented based on the ground truth values ​​of the labels of all pixels in the image to be segmented, obtaining object edge images for each category of the image to be segmented; obtaining the weights corresponding to each pixel of each category in the image to be segmented for prediction by the image semantic segmentation model to be trained based on the object edge images of each category in the image to be segmented; obtaining the loss value of the image semantic segmentation model for prediction of each pixel of each category in the image to be segmented based on the weights corresponding to each pixel of each category in the image to be segmented; and adjusting the parameters of the image semantic segmentation model based on the loss value to obtain the trained image semantic segmentation model. This application, by increasing the weights of pixels at edges and calculating the loss value for each pixel of each category in the image to be segmented, treats the object edge images of each category in the image to be segmented differently, enabling accurate prediction of each pixel of each category in the image to be segmented, accurately obtaining the loss value of each pixel of each category, improving the training effect of the image semantic segmentation model, and enabling the image semantic segmentation model to more accurately predict each pixel of each category in the image, thereby improving the semantic segmentation accuracy of the image semantic segmentation model.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0028] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0029] Figure 1 This is a schematic flowchart illustrating the image semantic segmentation model training method in an embodiment of this application;

[0030] Figure 2This is another flowchart illustrating the image semantic segmentation model training method shown in the embodiments of this application;

[0031] Figure 3 This is a schematic diagram of the truth value of the label with category index K shown in the embodiments of this application;

[0032] Figure 4 This is a schematic diagram illustrating the true values ​​of the labels of neighboring pixels in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram illustrating the true values ​​of the image pixel value gradient in an embodiment of this application;

[0034] Figure 6 This is a schematic flowchart of an image semantic segmentation method shown in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the structure of an image semantic segmentation model training device shown in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of the structure of an image semantic segmentation apparatus shown in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0038] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0039] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0040] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0041] In related technologies, the cross-entropy loss function can be expressed by the following formula:

[0042]

[0043] Where N is the total number of pixels in the sample image; C is the number of sample classes; and y is the truth value of the pixel belonging to that class, i.e., if the class id of a pixel is k, then

[0044]

[0045] p represents the probability that the pixel belongs to a certain category as predicted by the network. Its value ranges from [0,1], and the sum of the probabilities of all categories for the same pixel is 1.0, i.e.

[0046]

[0047] Because the cross-entropy loss function uses a global averaging approach, the final Lce value will still be relatively small when there are a few poorly performing pixels (i.e., a few pixels with large summation values). In practical use, it is found that edge detection results have a high number of false detections, meaning that it is easy to misclassify pixels at edge locations; the detection effect is particularly poor for long and thin objects, which are essentially edges everywhere.

[0048] This application improves upon the existing Cross Entropy (CE) loss function, thereby enhancing the training performance and semantic segmentation accuracy of image semantic segmentation models.

[0049] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0050] Figure 1 This is a schematic flowchart illustrating the image semantic segmentation model training method in an embodiment of this application.

[0051] See Figure 1 A method for training an image semantic segmentation model, comprising:

[0052] S110: Based on the ground truth values ​​of the labels of all pixels in the image to be segmented, perform edge detection on the image to be segmented to obtain the object edge images of each category in the image to be segmented.

[0053] In a specific embodiment, edge detection is performed on the image to be segmented based on the ground truth values ​​of the labels for all pixels of each category in the training set image, resulting in object edge images for each category of the image to be segmented. The object edge positions for each category of the image to be segmented can be detected based on the ground truth values ​​of the pixel labels, thus determining the object edge images for each category of the image to be segmented.

[0054] S120: Based on the object edge images of each category in the image to be segmented, obtain the weights corresponding to each pixel of each category in the image to be segmented by the image semantic segmentation model to be trained.

[0055] In a specific embodiment, based on the object edge images of each category in the image to be segmented, the weights corresponding to each pixel of each category in the image to be segmented predicted by the image semantic segmentation model to be trained are obtained, including: based on the object edge images of each category in the image to be segmented, the weights corresponding to each pixel of each category in the image to be segmented predicted by the image semantic segmentation model to be trained are obtained, and the weights corresponding to each pixel in the image to be segmented are equal to the product of the first hyperparameter plus the second hyperparameter and the true value of the label of each pixel in the corresponding image to be segmented.

[0056] S130: Based on the weights corresponding to each pixel of each category in the image to be segmented, obtain the loss value of the image semantic segmentation model for predicting each pixel of each category in the image to be segmented.

[0057] In a specific embodiment, the images of the training set are input into the image semantic segmentation model, and the image semantic segmentation model outputs the prediction results corresponding to each category of pixels in the image; based on the prediction results, the prediction loss value of each category of pixels in the image predicted by the image semantic segmentation model is obtained; based on the weight of each category of pixels in the image and the prediction loss value, the loss value of each category of pixels in the image to be segmented by the image semantic segmentation model is obtained.

[0058] S140: Adjust the parameters of the image semantic segmentation model based on the loss value to obtain the trained image semantic segmentation model.

[0059] In a specific embodiment, the loss value of each pixel of each category in the image can be predicted by the image semantic segmentation model to obtain the loss value of the image predicted by the image semantic segmentation model; the parameters of the image semantic segmentation model can be adjusted based on the loss value of the image predicted by the image semantic segmentation model to obtain the trained image semantic segmentation model.

[0060] The image semantic segmentation model training method illustrated in this application involves performing edge detection on the image to be segmented based on the ground truth values ​​of all pixels' labels, obtaining object edge images for each category of the image to be segmented; obtaining the weights corresponding to each pixel of each category in the image to be segmented as predicted by the image semantic segmentation model to be trained based on the object edge images of each category in the image to be segmented; obtaining the loss value of each pixel of each category in the image to be segmented as predicted by the image semantic segmentation model based on the weights corresponding to each pixel of each category in the image to be segmented; and adjusting the parameters of the image semantic segmentation model based on the loss values ​​to obtain the trained image semantic segmentation model. This application, by increasing the weights of pixels at edges and calculating the loss value for each pixel of each category in the image to be segmented, treats object edge images of different categories in the image differently, enabling accurate prediction of each pixel of each category in the image to be segmented, accurately obtaining the loss value of each pixel of each category, improving the training effect of the image semantic segmentation model, and enabling the image semantic segmentation model to more accurately predict each pixel of each category in the image, thereby improving the semantic segmentation accuracy of the image semantic segmentation model.

[0061] Figure 2 This is another schematic diagram of the image semantic segmentation model training method shown in the embodiments of this application. Figure 2 Compared to Figure 1 The scheme of this application is described in more detail.

[0062] See Figure 2 A method for training an image semantic segmentation model, comprising:

[0063] S210: Obtain the number of pixel categories in the image to be segmented based on the labeled image of the image to be segmented.

[0064] In a specific embodiment, a labeling tool is used to correctly label pixels in different regions or for different targets in the training set images, obtaining a labeled image (Ground Truth) with correctly labeled data for different pixels in the image. Based on the set values ​​of all pixels in the labeled image of the image to be segmented, duplicate set values ​​are removed from all pixels in the labeled image to obtain the number of unique set values. The number of unique set values ​​is the number of pixel categories in the image. For example, if the number of unique set values ​​is 3 (meaning there are 3 different set values), then the number of pixel categories in the image is 3.

[0065] In a specific embodiment, the pixel setting value can be a pixel value or a value set according to a set rule.

[0066] S220: Obtain the image histogram based on the image annotation and the number of pixel categories.

[0067] In a specific embodiment, encoding methods such as one-hot encoding can be used to encode each category of pixels in the labeled image according to the number of pixel categories in the image, thereby obtaining the image histogram.

[0068] In a specific embodiment, one-hot encoding is used to encode each category of pixels in the labeled image. One-hot encoding uses N channels to encode each category of pixels in the labeled image. The value of N in one-hot encoding is equal to the number of categories of pixels in the image. For example, if the number of categories of pixels in the image is 1, then the value of N in one-hot encoding is also 1.

[0069] S230: Based on the histogram of the image to be segmented, obtain the ground truth label values ​​of all pixels of each category in the image.

[0070] In a specific embodiment, for an image of size W x H, its label truth value can be represented as a three-dimensional matrix of size W x H x C, denoted as Label, where C is the number of categories. Elements in Label are either 1 or 0. For category k, Label[:,:,k] (i.e., the probability truth value of the k-th category for all pixels, with a shape of W x H) can be extracted. Assuming the image contains an object of category k, then '1' in a continuous area will form the pattern of that object. Figure 3 As shown in the example, the 1s in the matrix form a continuous area of ​​shaded region. In specific embodiments, the edge image in the figure can be found using various numerical methods.

[0071] S240: Based on the label ground value method of neighboring pixels or the image pixel value gradient method, perform edge detection on the label ground values ​​of all pixels in the image to be segmented to obtain the object edge image of each category in the image to be segmented.

[0072] In a specific embodiment, edge detection can be performed by determining whether there are unequal values ​​in the neighborhood. Specifically, a multidimensional matrix of the same size as the Label, denoted as EdgeMap, is generated, and the values ​​of its internal elements are filled with truth values ​​using formula (I):

[0073]

[0074] It should be noted that when a pixel has a value different from the ground truth value within its neighborhood of category c, that point is considered a boundary point, and its value is 1. When B=1, the criterion is an 8-neighborhood. The EdgeMap generated based on the above example is as follows: Figure 4As shown, the first part represents the inner edge of the object, and the second part represents the outer edge. The range for edge detection can also be increased, i.e., B = 2, 3, ..., becoming a 15-neighborhood or a 24-neighborhood, thus increasing the width of the edge.

[0075] In specific embodiments, edge detection can also be performed based on the image pixel value gradient method. Specifically, the Sobel operator is a common operator for calculating the gradient of image pixel values. It contains operators in two directions, which can calculate the gradient in the horizontal (x-direction) and vertical (y-direction) directions respectively. For a Sobel operator of size (2k+1)x(2k+1), the specific values ​​of its internal elements are as follows:

[0076]

[0077] K y (i,j)=K x (j,i)

[0078] Co is a fixed coefficient, the smallest coefficient that makes the elements in Kx and Ky integers.

[0079] When k=1, according to the above formula, Co=2, and the 3x3 Soble operator is obtained as follows:

[0080]

[0081]

[0082] Where Kx is the convolution kernel used to calculate the gradient in the x-direction, and Ky is the convolution kernel used to calculate the gradient in the y-direction. By directly performing a convolution operation on the Label using these two kernels, we can obtain the gradient maps in the x and y directions, as shown in the formula:

[0083]

[0084] Then the gradient map of the whole system is calculated.

[0085]

[0086] Since a Grad graph only has non-zero values ​​at its edges, it can be considered equivalent to an EdgeMap here. The results calculated from the example are as follows... Figure 5 As shown. In other embodiments, Sobel operators of size 5x5 or 7x7 can also be used, in the same way, but the resulting edge width will be wider than that.

[0087] S250: Based on the set loss function, obtain the prediction loss value of each pixel of each category in the image predicted by the image semantic segmentation model.

[0088] In a specific embodiment, the image is input into the image semantic segmentation model to be trained. Based on the input image, the image semantic segmentation model predicts the n categories of pixels in the image through n channels, outputting the prediction results for each category of pixels. Based on the prediction results of each category of pixels in the image and the labeled image, a set loss function, such as the cross-entropy loss function, is used to calculate the prediction loss value of each category of pixels in the image.

[0089] S260: Based on the weights and predicted loss values ​​of each pixel in each category of the image to be segmented, obtain the loss value predicted by the image semantic segmentation model for each pixel in each category of the image to be segmented.

[0090] In a specific embodiment, based on the object edge images of each category in the image to be segmented, the weights corresponding to each pixel of each category in the image to be segmented are predicted by the image semantic segmentation model to be trained. The weights corresponding to each pixel in the image to be segmented are equal to the product of the first hyperparameter plus the second hyperparameter and the true value of the label of each pixel in the corresponding image to be segmented.

[0091] In a specific embodiment, a weight map is generated based on the edge map of each category of objects in the image to be segmented, i.e., the EdgeMap, and then according to the following formula two:

[0092] Weight(i,j,c) = α + β × EdgeMap(i,j,c) (Formula 2)

[0093] Here, α and β are hyperparameters, with α > 0 and β > 0. Their optimal values ​​need to be obtained through experimental adjustments. For example, in a specific implementation, α = 1 and β = 1 can be set. Alternatively, they can be dynamically adjusted during training, such as gradually decreasing α and increasing β as the training epochs increase.

[0094] Based on the weights corresponding to each pixel of each category in the image to be segmented, the cross-entropy loss function is weighted to obtain the loss value of each pixel of each category in the image to be segmented predicted by the image semantic segmentation model. Among them, W n,c The above Weight represents the weight value corresponding to the pixel category.

[0095] S270: Based on the loss value of each pixel of each category in the image predicted by the image semantic segmentation model, adjust the parameters of the image semantic segmentation model to be trained to obtain the trained image semantic segmentation model.

[0096] In a specific embodiment, the loss values ​​of each category of pixels can be backpropagated in the image semantic segmentation model to be trained, and the parameters of each channel of the image semantic segmentation model to be trained can be adjusted to determine the parameters of the image semantic segmentation model. Steps 210 to 270 can be iteratively adjusted to adjust the parameters of each channel of the image semantic segmentation model to be trained until the loss value of the prediction result of each channel of the image semantic segmentation model to be trained tends to stabilize or is less than a set loss threshold, at which point the training of the image semantic segmentation model is stopped, and the trained image semantic segmentation model is obtained.

[0097] In another specific embodiment, the loss value of each pixel in each category of the image can be predicted by the image semantic segmentation model to obtain the average loss value of the image to be segmented. Based on the average loss value, the parameters of the image semantic segmentation model are adjusted to obtain a trained image semantic segmentation model. The loss values ​​of each category of pixels in the image predicted by the image semantic segmentation model are summed to obtain the cumulative loss value of each category of pixels in the image predicted by the image semantic segmentation model. The cumulative loss value is divided by the number of pixel categories in the image to obtain the average loss value of the image semantic segmentation model. The average loss value is backpropagated in the image semantic segmentation model to be trained, and the parameters of each channel of the image semantic segmentation model to be trained are adjusted to determine the parameters of the image semantic segmentation model. Steps 210 to 270 can be iteratively adjusted to adjust the parameters of the image semantic segmentation model to be trained until the average loss value of the image semantic segmentation model to be trained tends to stabilize or is less than a set loss threshold, at which point the training of the image semantic segmentation model is stopped, and a trained image semantic segmentation model is obtained.

[0098] The image semantic segmentation model training method illustrated in this application involves performing edge detection on the image to be segmented based on the ground truth values ​​of all pixels' labels, obtaining object edge images for each category of the image to be segmented; obtaining the weights corresponding to each pixel of each category in the image to be segmented based on the object edge images of each category in the image to be segmented; obtaining the loss value of the image semantic segmentation model for predicting each pixel of each category in the image to be segmented based on the weights corresponding to each pixel of each category in the image to be segmented; and adjusting the parameters of the image semantic segmentation model based on the loss values ​​to obtain the trained image semantic segmentation model. This application, by increasing the weights of pixels at edges and calculating the loss value for each pixel of each category in the image to be segmented, treats object edge images of different categories in the image differently, enabling accurate prediction of each pixel of each category in the image to be segmented, accurately obtaining the loss value for each pixel of each category, improving the training effect of the image semantic segmentation model, and enabling the image semantic segmentation model to more accurately predict each pixel of each category in the image, thereby improving the semantic segmentation accuracy of the image semantic segmentation model.

[0099] This application also provides an image semantic segmentation method.

[0100] Figure 6 This is a flowchart illustrating the image semantic segmentation method shown in the embodiments of this application.

[0101] See Figure 6 An image semantic segmentation method, comprising:

[0102] S610: Obtain the image to be segmented.

[0103] In one embodiment, the device can receive images to be segmented input by a user, or images to be segmented sent by other devices; it can also acquire images to be segmented through an image acquisition device.

[0104] S620: Input the image to be segmented into the image semantic segmentation model and obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model.

[0105] In a specific embodiment, the image to be segmented can be input into an image semantic segmentation model. The image semantic segmentation model predicts and outputs for each pixel of each category in the image to be segmented. Based on the predicted outputs of all pixels of each category in the image to be segmented, the segmentation result corresponding to the image to be segmented is output; thus, the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model is obtained. The image semantic segmentation model can be based on the above... Figure 1 , Figure 2 The image semantic segmentation model was trained using the given training method.

[0106] The image semantic segmentation method shown in this application focuses more on the position of object edges. It adjusts the weight of each pixel of the object edge in the image to be segmented, which can accurately predict each pixel of each category in the image and improve the accuracy of image semantic segmentation.

[0107] Corresponding to the aforementioned application function implementation method embodiments, this application also provides an image semantic segmentation model training device, an image semantic segmentation device, an electronic device, and corresponding embodiments.

[0108] Figure 7 This is a schematic diagram of the structure of an image semantic segmentation model training device shown in an embodiment of this application.

[0109] See Figure 7 An image semantic segmentation model training device includes a first processing module 701, a second processing module 702, a third processing module 703, and a fourth processing module 704.

[0110] The first processing module 701 is used to perform edge detection on the image to be segmented based on the label ground values ​​of all pixels in the image to be segmented, so as to obtain the object edge image of each category in the image to be segmented.

[0111] The second processing module 702 is used to obtain the weights of each pixel in each category of the image to be segmented based on the object edge images of each category in the image to be segmented;

[0112] The third processing module 703 is used to obtain the loss value of each pixel of each category in the image to be segmented by the image semantic segmentation model based on the weight corresponding to each pixel of each category in the image to be segmented.

[0113] The fourth processing module 704 is used to adjust the parameters of the image semantic segmentation model according to the loss value to obtain the trained image semantic segmentation model.

[0114] The technical solution illustrated in this application involves edge detection of the image to be segmented based on the ground truth values ​​of all pixels' labels, obtaining object edge images for each category of the image; obtaining the weights corresponding to each pixel of each category in the image to be segmented based on the object edge images of each category; obtaining the loss value of the image semantic segmentation model for predicting each pixel of each category in the image to be segmented based on the weights corresponding to each pixel of each category in the image to be segmented; and adjusting the parameters of the image semantic segmentation model based on the loss value to obtain a trained image semantic segmentation model. This application, by increasing the weights of pixels at edges and calculating the loss value for each pixel of each category in the image to be segmented, treats object edge images of different categories differently, enabling accurate prediction of each pixel of each category in the image to be segmented, accurately obtaining the loss value for each pixel of each category, improving the training effect of the image semantic segmentation model, and enabling the image semantic segmentation model to more accurately predict each pixel of each category in the image, thereby improving the semantic segmentation accuracy of the image semantic segmentation model.

[0115] Figure 8 This is a schematic diagram of the structure of an image semantic segmentation apparatus shown in an embodiment of this application.

[0116] See Figure 8 An image semantic segmentation device 800 includes an image acquisition module 801 and an image segmentation module 802.

[0117] Image acquisition module 801 is used to acquire the image to be segmented.

[0118] The image segmentation module 802 is used to input the image to be segmented acquired by the image acquisition module 801 into the image semantic segmentation model, and obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained according to the image semantic segmentation model training device described above.

[0119] The technical solution shown in this application focuses more on the position of object edges. It adjusts the weight of each pixel of the object edge in the image to be segmented, which can accurately predict each pixel of each category in the image and improve the accuracy of image semantic segmentation.

[0120] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0121] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0122] See Figure 9 The electronic device 900 includes a memory 901 and a processor 902.

[0123] The processor 902 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0124] Memory 901 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 902 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 901 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 901 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, a high-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0125] The memory 901 stores executable code, which, when processed by the processor 902, can cause the processor 902 to execute part or all of the methods described above.

[0126] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0127] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0128] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training an image semantic segmentation model, characterized in that, include: Based on the ground truth values ​​of the labels of all pixels in the image to be segmented, edge detection is performed on the image to be segmented to obtain the object edge image of each category in the image to be segmented; Based on the object edge images of each category in the image to be segmented, the image semantic segmentation model to be trained is used to predict the weight of each pixel in each category in the image to be segmented. The weight of each pixel in the image to be segmented is equal to the first hyperparameter plus the product of the second hyperparameter and the value of each pixel in the corresponding object edge image. Based on the weights corresponding to each pixel of each category in the image to be segmented, a weighted loss function is applied to obtain the loss value of the image semantic segmentation model for predicting each pixel of each category in the image to be segmented. The loss value is calculated using the following formula: Among them, W n,c The weights are as described above; Based on the loss value, the parameters of the image semantic segmentation model are adjusted to obtain a trained image semantic segmentation model.

2. The method according to claim 1, characterized in that, Before performing edge detection on the image to be segmented based on the ground truth values ​​of the labels of all pixels in the image to be segmented to obtain the object edge image of each category in the image to be segmented, the method further includes: Based on the histogram of the image to be segmented, obtain the true values ​​of the labels for all pixels in the image to be segmented; Edge detection is performed on the label ground values ​​of all pixels in the image to be segmented, based on either the label ground values ​​of neighboring pixels or the image pixel value gradient method, to obtain the object edge images of each category in the image to be segmented.

3. The method according to claim 1, characterized in that, The step of weighting a predetermined loss function according to the weights corresponding to each pixel of each category in the image to be segmented, to obtain the loss value of the image semantic segmentation model for predicting each pixel of each category in the image to be segmented, includes: Based on the set loss function, the predicted loss value of each pixel of each category in the image is obtained by the image semantic segmentation model; Based on the weights and predicted loss values ​​of each pixel in each category of the image to be segmented, the loss value of each pixel in each category of the image to be segmented is obtained by the image semantic segmentation model.

4. The method according to claim 3, characterized in that, The step of adjusting the parameters of the image semantic segmentation model based on the loss value to obtain the trained image semantic segmentation model includes: The loss value of each pixel of each category in the image is predicted by the image semantic segmentation model, and the average loss value of the image to be segmented is obtained by the image semantic segmentation model. Based on the average loss value, adjust the parameters of the image semantic segmentation model to obtain a trained image semantic segmentation model.

5. An image semantic segmentation method, characterized in that, include: Obtain the image to be segmented; The image to be segmented is input into the image semantic segmentation model to obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained by the image semantic segmentation model training method according to any one of claims 1-4.

6. An image semantic segmentation model training device, characterized in that, include: The first processing module is used to perform edge detection on the image to be segmented based on the true values ​​of the labels of all pixels in the image to be segmented, so as to obtain the object edge image of each category in the image to be segmented. The second processing module is used to obtain the weights corresponding to each pixel of each category in the image to be segmented based on the object edge images of each category in the image to be segmented. The weights corresponding to each pixel in the image to be segmented are equal to the first hyperparameter plus the product of the second hyperparameter and the value of each pixel in the corresponding object edge image. The third processing module is used to weight a predetermined loss function according to the weights corresponding to each pixel of each category in the image to be segmented, to obtain the loss value of the image semantic segmentation model for predicting each pixel of each category in the image to be segmented, wherein the loss value is calculated by the following formula: Among them, W n,c The weights are as described above; The fourth processing module is used to adjust the parameters of the image semantic segmentation model according to the loss value to obtain the trained image semantic segmentation model.

7. An image semantic segmentation device, characterized in that, include: The image acquisition module is used to acquire the image to be segmented; The image segmentation module is used to input the image to be segmented into the image semantic segmentation model and obtain the segmentation result corresponding to the image to be segmented output by the image semantic segmentation model. The image semantic segmentation model is trained by the image semantic segmentation model training device according to claim 6.

8. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-4.

9. A computer-readable storage medium, characterized in that, It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-4.

Images