Semantic segmentation model training method, semantic segmentation method and electronic device
By generating pixel weight matrix and calculating loss values to optimize semantic segmentation model parameters, the problem of inaccurate semantic segmentation boundaries in the prior art is solved, and more accurate segmentation results are achieved.
Patent Information
- Application Number
- CN202210297985.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-03-23
AI Technical Summary
Existing semantic segmentation methods have inaccuracies when processing pixel boundaries, especially when adjacent pixels are at the boundaries of segmented areas, resulting in inaccurate segmentation results.
By obtaining edge images and target segmentation results in the image training set, a pixel weight matrix is generated, the loss value between the predicted segmentation result and the target segmentation result is calculated, and the semantic segmentation model parameters are adjusted to optimize the accuracy of segmentation boundaries.
Improves the accuracy of the semantic segmentation model and ensures the accuracy of segmentation results, especially when dealing with boundary areas.
Smart Images

Figure CN114781489B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of semantic segmentation, and in particular to a semantic segmentation model training method, a semantic segmentation method and an electronic device. Background Art
[0002] As computer vision continues to develop, semantic segmentation technology has become more and more mature. Recent semantic segmentation architectures generally use convolutional neural networks (CNNs) to assign an initial category label to each pixel. Convolutional layers can effectively capture local features in an image and nest many such modules together in a hierarchical manner so that CNNs can try to extract larger structures. By capturing the complex features of an image through a series of convolutions, CNNs can encode the content of an image into a compact representation.
[0003] However, in the prior art, since the image information within the corresponding receptive fields of adjacent pixels is too similar, if the adjacent pixels all belong to the interior of the desired segmentation area, then this "similarity" is beneficial, but if the adjacent pixels happen to be on the boundary of the desired segmentation area, then this similarity will lead to inaccurate semantic segmentation results. Summary of the invention
[0004] In view of this, an embodiment of the present invention provides a semantic segmentation model training method, a semantic segmentation method and an electronic device, aiming to solve the problem of inaccurate boundary segmentation in existing semantic segmentation methods.
[0005] According to a first aspect, an embodiment of the present invention provides a semantic segmentation model training method, comprising:
[0006] Obtain edge images and target segmentation results corresponding to each training image in the image training set;
[0007] According to the pixel value of each pixel in the edge image, a pixel weight matrix corresponding to the edge image is generated;
[0008] Input each training image into the initial semantic segmentation model to obtain the predicted segmentation result;
[0009] Calculate the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix;
[0010] The parameters of the initial semantic segmentation model are adjusted based on the loss value to determine the target semantic segmentation model.
[0011] The semantic segmentation model training method provided by the embodiment of the present invention obtains the edge image and the target segmentation result corresponding to each training image in the image training set, and generates the pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image, thereby ensuring the accuracy of the pixel weight matrix corresponding to the generated edge image. Then, each training image is input into the initial semantic segmentation model to obtain the predicted segmentation result, and the loss value between the predicted segmentation result and the target segmentation result is calculated based on the pixel weight matrix. The accuracy of the loss value between the calculated predicted segmentation result and the target segmentation result is guaranteed, so that the pixels of the segmentation boundary can be determined more accurately, and the boundary optimization is achieved. Then, the parameters of the initial semantic segmentation model are adjusted based on the loss value to determine the target semantic segmentation model, thereby ensuring the accuracy of the determined target semantic segmentation model, and then ensuring the accuracy of the segmentation result obtained by using the target semantic segmentation model for semantic segmentation.
[0012] In combination with the first aspect, in a first implementation of the first aspect, calculating a loss value between a predicted segmentation result and a target segmentation result based on a pixel weight matrix includes:
[0013] Generate a sample weight sum based on the value of each element in the pixel weight matrix;
[0014] Based on the pixel weight matrix and the sum of sample weights, the cross entropy loss value between the predicted segmentation result and the target segmentation result is calculated.
[0015] The semantic segmentation model training method provided by the embodiment of the present invention generates a sample weight sum based on the value of each element in the pixel weight matrix, thereby ensuring the accuracy of the generated sample weight sum. Then, based on the pixel weight matrix and the sample weight sum, the cross entropy loss value between the predicted segmentation result and the target segmentation result is calculated. The accuracy of the calculated cross entropy loss value is ensured.
[0016] In combination with the first implementation of the first aspect, in the second implementation of the first aspect, generating a sample weight sum based on the value of each element in the pixel weight matrix includes:
[0017] Add the target parameter to the value of each element in the pixel weight matrix to obtain the target pixel weight matrix;
[0018] The values of each target element in the target pixel weight matrix are added together to generate the sum of the sample weights.
[0019] The semantic segmentation model training method provided by the embodiment of the present invention adds the target parameter to the value of each element in the pixel weight matrix to obtain the target pixel weight matrix, which takes into account the influence of the global loss on the predicted segmentation result, rather than just the influence of the segmentation boundary loss on the predicted segmentation result. Then, the values of each target element in the target pixel weight matrix are added to generate the sample weight sum, thereby ensuring the accuracy of the generated sample weight sum.
[0020] In combination with the second implementation of the first aspect, in the third implementation of the first aspect, based on the pixel weight matrix and the sum of sample weights, calculating the cross entropy loss value between the predicted segmentation result and the target segmentation result includes:
[0021] Calculate the initial loss value between the predicted result and the actual result for each pixel in the training image;
[0022] Multiply each initial loss value by the corresponding target element in the target pixel weight matrix to obtain the target loss value corresponding to each pixel;
[0023] After adding the target loss values corresponding to each pixel, divide it by the sum of the sample weights to obtain the cross entropy loss value between the predicted segmentation result and the target segmentation result.
[0024] The semantic segmentation model training method provided by the embodiment of the present invention calculates the initial loss value between the predicted result and the actual result of each pixel in the training image, thereby ensuring the accuracy of the initial loss value corresponding to each pixel calculated. Then, each initial loss value is multiplied by the corresponding target element in the target pixel weight matrix to obtain the target loss value corresponding to each pixel. As a result, the calculated target loss value takes into account both the influence of the global loss on the predicted segmentation result and the influence of the segmentation boundary loss on the predicted segmentation result, thereby ensuring the accuracy of the target loss value, so that the pixels at the segmentation edge can be better determined according to the target loss value. Then, after adding the target loss values corresponding to each pixel, the cross entropy loss value between the predicted segmentation result and the target segmentation result is obtained by dividing it by the sum of the sample weights. The accuracy of the calculated cross entropy loss value is guaranteed, so that the training result of the training image can be determined according to the cross entropy loss value, thereby ensuring the accuracy of the target semantic segmentation model trained according to the cross entropy loss value.
[0025] In combination with the first aspect, in a fourth implementation of the first aspect, a pixel weight matrix corresponding to the edge image is generated according to the pixel value of each pixel in the edge image, including:
[0026] Performing filtering on the edge image to generate a target edge image;
[0027] The pixel value of each pixel in the target edge image is normalized to generate a pixel weight matrix.
[0028] The semantic segmentation model training method provided by the embodiment of the present invention performs filtering processing on the edge image, thereby suppressing the noise that obeys the normal distribution, so that the generated target edge image has less noise, is more accurate and smooth. Then, the pixel value of each pixel in the target edge image is normalized to generate a pixel weight matrix, which ensures the accuracy of the generated pixel weight matrix and makes the generated pixel weight matrix related to each pixel value in the edge image, so that the segmentation boundary in the training image can be better determined according to the pixel weight matrix, thereby increasing the difference between the segmentation boundary and the inside of the segmentation area, and ensuring the accuracy of the loss value calculated based on the pixel weight matrix.
[0029] In combination with the first aspect, in a fifth implementation of the first aspect, obtaining an edge image corresponding to each training image in an image training set includes:
[0030] Obtain an image training set; each training image in the image training set is labeled;
[0031] According to the labels of each training image, the training images are converted into grayscale images;
[0032] The edges of the grayscale image are located to obtain edge images corresponding to each training image.
[0033] The semantic segmentation model training method provided by the embodiment of the present invention obtains an image training set, and converts the training image into a grayscale image according to the label of each training image, thereby ensuring the accuracy of the obtained grayscale image. Then, the edge of the grayscale image is located to obtain an edge image corresponding to each training image, thereby ensuring the accuracy of the obtained edge image.
[0034] According to the second aspect, an embodiment of the present invention further provides a semantic segmentation method, the method comprising:
[0035] Obtain the image to be segmented;
[0036] The image to be segmented is input into a target semantic segmentation model, and a segmentation result corresponding to the image to be segmented is output. The target semantic segmentation model is trained according to the semantic segmentation model training method in the first aspect or any one of the embodiments of the first aspect.
[0037] The semantic segmentation method provided by the embodiment of the present invention obtains an image to be segmented, inputs the image to be segmented into a target semantic segmentation model, and outputs a segmentation result corresponding to the image to be segmented, thereby ensuring the accuracy of the output segmentation result.
[0038] According to the third aspect, an embodiment of the present invention further provides a semantic segmentation model training device, comprising:
[0039] An acquisition module is used to acquire edge images and target segmentation results corresponding to each training image in the image training set;
[0040] A generation module, used for generating a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image;
[0041] An input module is used to input each training image into the initial semantic segmentation model to obtain a predicted segmentation result;
[0042] A calculation module, used for calculating the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix;
[0043] The adjustment module is used to adjust the parameters of the initial semantic segmentation model based on the loss value to determine the target semantic segmentation model.
[0044] The semantic segmentation model training device provided by the embodiment of the present invention obtains the edge image and the target segmentation result corresponding to each training image in the image training set, and generates the pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image, thereby ensuring the accuracy of the pixel weight matrix corresponding to the generated edge image. Then, each training image is input into the initial semantic segmentation model to obtain the predicted segmentation result, and the loss value between the predicted segmentation result and the target segmentation result is calculated based on the pixel weight matrix. The accuracy of the loss value between the calculated predicted segmentation result and the target segmentation result is ensured, so that the pixels of the segmentation boundary can be determined more accurately, and the boundary optimization is achieved. Then, the parameters of the initial semantic segmentation model are adjusted based on the loss value to determine the target semantic segmentation model, thereby ensuring the accuracy of the determined target semantic segmentation model, and then ensuring the accuracy of the segmentation result obtained by using the target semantic segmentation model for semantic segmentation.
[0045] According to a fourth aspect, an embodiment of the present invention further provides a semantic segmentation device, including:
[0046] An acquisition module, used for acquiring an image to be segmented;
[0047] A segmentation module is used to input the image to be segmented into a target semantic segmentation model and output a segmentation result corresponding to the image to be segmented. The target semantic segmentation model is trained according to the semantic segmentation model training method in the first aspect or any one of the embodiments of the first aspect.
[0048] The semantic segmentation device provided by the embodiment of the present invention obtains an image to be segmented, inputs the image to be segmented into a target semantic segmentation model, and outputs a segmentation result corresponding to the image to be segmented, thereby ensuring the accuracy of the output segmentation result.
[0049] According to the fifth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the computer instructions to execute the semantic segmentation model training method in the first aspect or any one of the embodiments of the first aspect and the semantic segmentation method in any one of the embodiments of the second aspect.
[0050] According to the sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the semantic segmentation model training method in the first aspect or any one of the embodiments of the first aspect and the semantic segmentation method in any one of the embodiments of the second aspect.
[0051] According to a seventh aspect, an embodiment of the present invention provides an engineering machinery equipment, the engineering machinery equipment comprising:
[0052] The device body and the electronic device described in the fifth aspect, the device body is connected to the electronic device, and the electronic device is used to execute the semantic segmentation model training method in the first aspect or any one of the embodiments of the first aspect and the semantic segmentation method in any one of the embodiments of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0054] Figure 1 is a schematic diagram of applying semantic segmentation provided by an embodiment of the present invention to autonomous driving;
[0055] Figure 2 is a flowchart of a semantic segmentation model training method provided by another embodiment of the present invention;
[0056] Figure 3 is a flowchart of a semantic segmentation model training method provided by another embodiment of the present invention;
[0057] Figure 4 is a flowchart of a semantic segmentation model training method provided by another embodiment of the present invention;
[0058] 5 is a schematic diagram of a training image, a grayscale image, and an edge image in a semantic segmentation model training method provided by another embodiment of the present invention;
[0059] Figure 6 is a schematic diagram of filtering an edge image in a semantic segmentation model training method provided by another embodiment of the present invention;
[0060] Figure 7 is a flow chart of a semantic segmentation method provided by another embodiment of the present invention;
[0061] Figure 8 It is a functional module diagram of a semantic segmentation model training device provided by an embodiment of the present invention;
[0062] Fig. 9 It is a functional module diagram of a semantic segmentation device provided by an embodiment of the present invention;
[0063] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0065] With the continuous development of computer vision, semantic segmentation technology has become more and more mature. Currently, semantic segmentation technology is applied in more and more technical fields.
[0066] For example, in the field of autonomous driving, autonomous driving vehicles need to obtain driving scenes in real time, perform semantic segmentation on the driving scenes, identify roads, vehicles, trees, and surrounding buildings in the driving scenes, and then perform autonomous driving based on the recognition results. For example, Figure 1 shown.
[0067] In computational photography, electronic devices can perform semantic segmentation on photos, and then perform semantic recognition based on the segmentation results. For example, semantic segmentation can be performed on trees and people in a photo, and then the trees and people in the photo can be classified as scenery and people according to their semantics. The scenery part is processed according to the scenery processing method, and the people part is processed according to the person processing method.
[0068] In terms of geographic information systems: electronic devices can perform semantic segmentation on input satellite remote sensing images, automatically identify roads, rivers, crops, buildings, etc., and label each pixel in the image.
[0069] In terms of medical image analysis and intelligent medical field, semantic segmentation is mainly used in tumor image segmentation and caries diagnosis.
[0070] However, in the prior art, since the image information within the corresponding receptive fields of adjacent pixels is too similar, if the adjacent pixels all belong to the interior of the desired segmentation area, then this "similarity" is beneficial, but if the adjacent pixels happen to be on the boundary of the desired segmentation area, then this similarity will lead to inaccurate semantic segmentation results.
[0071] Therefore, the embodiment of the present application provides a semantic segmentation model training method, which can ensure the accuracy of the target semantic segmentation model obtained through training, and further ensure the accuracy of the segmentation results obtained by performing semantic segmentation using the target semantic segmentation model.
[0072] It should be noted that the method for semantic segmentation model training provided in the embodiment of the present application, its execution subject can be a device for semantic segmentation model training, and the device for semantic segmentation model training can be implemented as part or all of a computer device through software, hardware, or a combination of software and hardware, wherein the computer device can be a server or a terminal, wherein the server in the embodiment of the present application can be a single server or a server cluster composed of multiple servers, and the terminal in the embodiment of the present application can be a smart phone, a personal computer, a tablet computer, a wearable device, an intelligent robot, and other intelligent hardware devices such as control components in intelligent vehicles or engineering machinery equipment. In the following method embodiments, the execution subject is an electronic device as an example for explanation.
[0073] In one embodiment of the present application, Figure 2 As shown, a semantic segmentation model training method is provided, which is described by taking the method applied to an electronic device as an example, and includes the following steps:
[0074] S11, obtaining edge images and target segmentation results corresponding to each training image in the image training set.
[0075] In an optional embodiment of the present application, the electronic device can receive edge images and target segmentation results corresponding to each training image in the image training set input by the user, and can also receive edge images and target segmentation results corresponding to each training image in the image training set sent by other devices, and can also identify and process each training image in the image training set to obtain edge images and target segmentation results corresponding to each training image.
[0076] The embodiment of the present application does not specifically limit the manner in which the electronic device obtains the edge image and the target segmentation result corresponding to each training image in the image training set.
[0077] S12. Generate a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image.
[0078] In an optional implementation of the present application, after acquiring the edge image corresponding to the training image, the electronic device may acquire the pixel value of each pixel in the edge image, and then extract the pixel value of each pixel in the edge image to generate a pixel weight matrix corresponding to the edge image.
[0079] This step will be described in detail below.
[0080] S13. Input each training image into the initial semantic segmentation model to obtain a predicted segmentation result.
[0081] Specifically, the electronic device inputs each training image into an initial semantic segmentation model, and the initial semantic segmentation model extracts features from the training image and outputs a predicted segmentation result based on the extracted features.
[0082] The initial semantic segmentation model may be a Fully Convolution Network (FCN), a Semantic Segmentation Network (SegNet), a Unity Networking (UNet), a DeepLab model, UNet++, PSPNet, etc. The embodiment of the present application does not specifically limit the initial semantic segmentation model.
[0083] S14. Calculate the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix.
[0084] In an optional embodiment, the electronic device may first calculate an initial loss value between the predicted segmentation result and the target segmentation result, and then multiply the initial loss value between the predicted segmentation result and the target segmentation result by a pixel weight matrix to obtain a loss value between the predicted segmentation result and the target segmentation result.
[0085] This step will be described in detail below.
[0086] S15. Adjust the parameters of the initial semantic segmentation model based on the loss value to determine the target semantic segmentation model.
[0087] Specifically, after calculating the loss value between the predicted segmentation result and the target segmentation result, the electronic device can adjust the parameters of the initial semantic segmentation model based on the loss value until the loss value between the predicted segmentation result and the target segmentation result tends to stabilize or is less than a preset loss threshold, and determine the target semantic segmentation model.
[0088] Furthermore, when training the above initial semantic segmentation model, the Adam optimizer can be selected to optimize the initial semantic segmentation model, or optimizers such as the stochastic gradient descent method SGD and AdaGrad can be selected, so that the initial semantic segmentation model can converge quickly and have good generalization ability.
[0089] When the Adam optimizer is used to optimize the initial semantic segmentation model, a learning rate can also be set for the optimizer. Here, the LR Range Test technique can be used to select the best learning rate and set it to the optimizer. The learning rate selection process of this test technique is as follows: first, the learning rate is set to a very small value, then the initial semantic segmentation model and the training image data are simply iterated several times, the learning rate is increased after each iteration, and the loss value (loss) between the predicted segmentation result and the target segmentation result in each training is recorded, and then the LR Range Test graph is drawn. Generally, the ideal LR Range Test graph contains three regions: the first region has a learning rate that is too small and the loss remains basically unchanged, the second region has a loss that decreases and converges quickly, and the last region has a learning rate that is too large so that the loss begins to diverge. Then the learning rate corresponding to the lowest point in the LR Range Test graph can be used as the best learning rate, and the best learning rate is used as the initial learning rate of the Adam optimizer and set to the optimizer.
[0090] The semantic segmentation model training method provided by the embodiment of the present invention obtains the edge image and the target segmentation result corresponding to each training image in the image training set, and generates the pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image, thereby ensuring the accuracy of the pixel weight matrix corresponding to the generated edge image. Then, each training image is input into the initial semantic segmentation model to obtain the predicted segmentation result, and the loss value between the predicted segmentation result and the target segmentation result is calculated based on the pixel weight matrix. The accuracy of the loss value between the calculated predicted segmentation result and the target segmentation result is guaranteed, so that the pixels of the segmentation boundary can be determined more accurately, and the boundary optimization is achieved. Then, the parameters of the initial semantic segmentation model are adjusted based on the loss value to determine the target semantic segmentation model, thereby ensuring the accuracy of the determined target semantic segmentation model, and then ensuring the accuracy of the segmentation result obtained by using the target semantic segmentation model for semantic segmentation.
[0091] In one embodiment of the present application, Figure 3 As shown, a semantic segmentation model training method is provided, which is described by taking the method applied to an electronic device as an example, and includes the following steps:
[0092] S21, obtaining edge images and target segmentation results corresponding to each training image in the image training set.
[0093] For more information about this step, see Figure 1 The introduction of S11 in the previous section will not be repeated here.
[0094] S22. Generate a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image.
[0095] For more information about this step, see Figure 1 The introduction of S12 in is not repeated here.
[0096] S23, input each training image into the initial semantic segmentation model to obtain the predicted segmentation result.
[0097] For more information about this step, see Figure 1 The introduction of S13 in will not be repeated here.
[0098] S24. Calculate the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix.
[0099] In an optional implementation of the present application, the above S24 “calculating the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix” may include the following steps:
[0100] S241. Generate a sample weight sum based on the value of each element in the pixel weight matrix.
[0101] In an optional implementation of the present application, the electronic device may add the values of each element in the pixel weight matrix to generate a sample weight sum.
[0102] In an optional implementation manner of the present application, the above S241 “generating a sample weight sum based on the value of each element in the pixel weight matrix” may include the following steps:
[0103] (1) Add the target parameter to the value of each element in the pixel weight matrix to obtain the target pixel weight matrix.
[0104] Specifically, the electronic device may add the target parameter to the value of each element in the pixel weight matrix to obtain a target pixel weight matrix.
[0105] Among them, the target parameter can be set according to actual needs. The target parameter can be 1, 2, or 0.5, etc. The embodiment of the present application does not specifically limit the target parameter.
[0106] (2) Add the values of each target element in the target pixel weight matrix to generate the sum of the sample weights.
[0107] Specifically, after calculating and obtaining the target pixel weight matrix, the electronic device adds the values of each target element in the target pixel weight matrix to generate a sample weight sum.
[0108] S242: Calculate the cross entropy loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix and the sum of the sample weights.
[0109] In an optional implementation of the present application, the above S242 “calculating the cross entropy loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix and the sum of sample weights” may include the following steps:
[0110] (1) Calculate the initial loss value between the predicted result and the true result of each pixel in the training image.
[0111] (2) Multiply each initial loss value by the corresponding target element in the target pixel weight matrix to obtain the target loss value corresponding to each pixel.
[0112] (3) After adding the target loss values corresponding to each pixel, divide it by the sum of the sample weights to obtain the cross entropy loss value between the predicted segmentation result and the target segmentation result.
[0113] Specifically, after the electronic device inputs the training image into the initial semantic segmentation model, it obtains the prediction result of each pixel, and then calculates the initial loss value between the prediction result and the true result of each pixel in the training image. The initial loss value corresponding to each pixel is multiplied by the corresponding target element in the target pixel weight matrix to obtain the target loss value corresponding to each pixel. After adding the target loss values corresponding to each pixel, it is divided by the sum of the sample weights to obtain the cross entropy loss value between the predicted segmentation result and the target segmentation result.
[0114] For example, in the prior art, taking binary classification semantic segmentation of an image sample as an example, the cross entropy loss function in the prior art is expressed as follows:
[0115]
[0116] Among them, a is the prediction result, which is generally the result of the Sigmoid function; y is the true value (0 or 1), and N is the total number of pixels in the sample. The cross entropy loss checks each pixel one by one and compares the prediction result (probability distribution vector) of each pixel category with the true label result. The loss of the entire image is the average loss of each pixel.
[0117] In the embodiment of the present application, the initial loss value between the predicted result and the true result of each pixel in the training image is first calculated. Then, the initial loss value corresponding to each pixel is multiplied by the corresponding target element in the target pixel weight matrix to obtain the target loss value corresponding to each pixel. After adding the target loss values corresponding to each pixel, it is divided by the sum of the sample weights to obtain the cross entropy loss value between the predicted segmentation result and the target segmentation result.
[0118] Exemplarily, in the embodiment of the present application, the improved cross entropy loss function is expressed as follows:
[0119]
[0120] Among them, a is the prediction result, generally the result of the Sigmoid function; y is the true value (0 or 1), M is the sum of sample weights, w is the pixel weight matrix, 1 is the target parameter, and 1+w is the target pixel weight matrix.
[0121] S25. Adjust the parameters of the initial semantic segmentation model based on the loss value to determine the target semantic segmentation model.
[0122] For more information about this step, see Figure 1 The introduction of S15 in the previous section will not be repeated here.
[0123] The semantic segmentation model training method provided by the embodiment of the present invention adds the target parameter to the value of each element in the pixel weight matrix to obtain the target pixel weight matrix, taking into account the influence of the global loss on the predicted segmentation result, rather than just the influence of the segmentation boundary loss on the predicted segmentation result. Then, the values of each target element in the target pixel weight matrix are added to generate the sample weight sum, thereby ensuring the accuracy of the generated sample weight sum. Then, the initial loss value between the predicted result and the true result of each pixel in the training image is calculated to ensure the accuracy of the initial loss value corresponding to each pixel calculated. Then, each initial loss value is multiplied by the corresponding target element in the target pixel weight matrix to obtain the target loss value corresponding to each pixel. Thus, the calculated target loss value takes into account both the influence of the global loss on the predicted segmentation result and the influence of the segmentation boundary loss on the predicted segmentation result, thereby ensuring the accuracy of the target loss value, so that the pixels at the segmentation edge can be better determined according to the target loss value. Then, after adding the target loss values corresponding to each pixel, it is divided by the sample weight sum to obtain the cross entropy loss value between the predicted segmentation result and the target segmentation result. The accuracy of the calculated cross entropy loss value is guaranteed, so that the training result of the training image can be determined according to the cross entropy loss value, thereby ensuring the accuracy of the target semantic segmentation model trained according to the cross entropy loss value.
[0124] In one embodiment of the present application, Figure 4 As shown, a semantic segmentation model training method is provided, which is described by taking the method applied to an electronic device as an example, and includes the following steps:
[0125] S31, obtaining edge images and target segmentation results corresponding to each training image in the image training set.
[0126] In an optional implementation manner of the present application, the “obtaining edge images corresponding to each training image in the image training set” in the above S11 may include the following steps:
[0127] S311, obtaining an image training set.
[0128] Each training image in the image training set has a label. The label of the training image can be the target segmentation result corresponding to the training image, and the label of the training image can be generated by pixel-by-pixel annotation of the electronic device using tools such as labelme.
[0129] In an optional implementation of the present application, the electronic device may receive an image training set input by a user, or may receive an image training set sent by other devices. The present application embodiment does not specifically limit the manner in which the electronic device obtains the image training set.
[0130] In an optional implementation of the present application, after acquiring the image training set, the electronic device may also pre-process the training images in the image training set, for example, performing image stretching, image cropping, format conversion, etc.
[0131] S312: Convert the training images into grayscale images according to the labels of the training images.
[0132] Specifically, after acquiring the training images, the electronic device may convert the training images into grayscale images according to the labels of the training images.
[0133] Exemplarily, the electronic device may determine the foreground and background of the training image according to the label of the training image, and then use 255 to represent the foreground in the training image and use 0 to represent the background in the training image, thereby converting the training image into a grayscale image.
[0134] S313: Locate the edge of the grayscale image to obtain edge images corresponding to each training image.
[0135] Specifically, the electronic device can use a traditional image edge operator to locate the edge of the grayscale image, and each training image corresponds to an edge image.
[0136] Among them, the edge operator can be a sobel operator, or a Roberts operator, a Prewitt operator, etc. The embodiment of the present application does not specifically limit the edge operator.
[0137] For example, the electronic device can use a traditional image edge operator to locate the edge of the grayscale image, and then use 255 to represent the edge pixels in the grayscale image and use 0 to represent the internal pixels in the grayscale image, thereby generating an edge image corresponding to the training image. As shown in FIG5 , FIG5(a) represents the training image, FIG5(b) represents the grayscale image, and FIG5(c) represents the edge image.
[0138] S32. Generate a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image.
[0139] In an optional implementation of the present application, the above S32 “generating a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image” may include the following steps:
[0140] S321, filter the edge image to generate a target edge image.
[0141] Specifically, the electronic device may perform Gaussian filtering on the edge image to generate a target edge image with smoothed edges.
[0142] Among them, Gaussian smoothing filters are very effective in suppressing noise that follows a normal distribution, which has been proven to conform to most physical laws. Edges may appear anywhere in the image, and the annotation process may be wrong by one or two pixels, so they obey the central limit theorem.
[0143] like Figure 6 As shown, based on the above content, it can be known that the edge pixels of the edge image are represented by 255, and the internal pixels are represented by 0. The electronic device can perform Gaussian filtering on the edge image, which will make the transition of the pixel value of the edge pixel from 255 to 0 smoother, so that the target edge image after filtering has a convex middle and concave edge effect.
[0144] S322, normalize the pixel value of each pixel in the target edge image to generate a pixel weight matrix.
[0145] Specifically, after acquiring the target edge image, the electronic device may normalize the pixel value of each pixel in the target edge image to generate a pixel weight matrix.
[0146] Optionally, as can be seen from the above content, after acquiring the target edge image, since the pixel values of each pixel in the target edge image are between 0-255, the electronic device can divide the pixel value of each pixel in the target edge image by 255, so that the pixel value of each pixel in the target edge image is normalized to between 0-1, thereby generating a pixel weight matrix.
[0147] S33, input each training image into the initial semantic segmentation model to obtain a predicted segmentation result.
[0148] S34. Calculate the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix.
[0149] S35. Adjust the parameters of the initial semantic segmentation model based on the loss value to determine the target semantic segmentation model.
[0150] The semantic segmentation model training method provided by the embodiment of the present invention obtains an image training set, and converts the training image into a grayscale image according to the label of each training image, thereby ensuring the accuracy of the obtained grayscale image. Then, the edge of the grayscale image is located to obtain an edge image corresponding to each training image, thereby ensuring the accuracy of the obtained edge image.
[0151] In addition, the embodiment of the present application performs filtering processing on the edge image, thereby suppressing the noise that obeys the normal distribution, so that the generated target edge image has less noise, is more accurate and smooth. Then, the pixel value of each pixel in the target edge image is normalized to generate a pixel weight matrix, which ensures the accuracy of the generated pixel weight matrix and makes the generated pixel weight matrix related to each pixel value in the edge image, so that the segmentation boundary in the training image can be better determined according to the pixel weight matrix, thereby increasing the difference between the segmentation boundary and the inside of the segmentation area, and ensuring the accuracy of the loss value calculated based on the pixel weight matrix.
[0152] It should be noted that the semantic segmentation method provided in the embodiment of the present application may be executed by a semantic segmentation device, which may be implemented as part or all of a computer device through software, hardware, or a combination of software and hardware, wherein the computer device may be a server or a terminal, wherein the server in the embodiment of the present application may be a single server or a server cluster composed of multiple servers, and the terminal in the embodiment of the present application may be a smart phone, a personal computer, a tablet computer, a wearable device, an intelligent robot, and other intelligent hardware devices such as control components in intelligent vehicles or engineering machinery equipment. In the following method embodiments, the execution subject is an electronic device as an example for explanation.
[0153] In one embodiment of the present application, Figure 7As shown, a semantic segmentation method is provided, which is described by taking the method applied to an electronic device as an example, and includes the following steps:
[0154] S41, obtaining an image to be segmented.
[0155] In an optional implementation of the present application, the electronic device may receive the image to be segmented input by the user, may receive the image to be segmented sent by other devices, and may also use its own image acquisition device to acquire the image to be segmented.
[0156] The embodiment of the present application does not specifically limit the manner in which the electronic device obtains the image to be segmented.
[0157] S42: Input the image to be segmented into the target semantic segmentation model, and output the segmentation result corresponding to the image to be segmented.
[0158] Among them, the target semantic segmentation model is trained according to the semantic segmentation model training method of any one of the above embodiments.
[0159] Specifically, the electronic device inputs the image to be segmented into the target semantic segmentation model, and the target semantic segmentation model extracts features from the image to be segmented, and outputs a segmentation result corresponding to the image to be segmented based on the extracted features.
[0160] The semantic segmentation method provided by the embodiment of the present invention obtains an image to be segmented, inputs the image to be segmented into a target semantic segmentation model, and outputs a segmentation result corresponding to the image to be segmented, thereby ensuring the accuracy of the output segmentation result.
[0161] It should be understood that although Figure 2-4 as well as Figure 7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-4 as well as Figure 7 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.
[0162] like Figure 8 As shown, this embodiment provides a semantic segmentation model training device, including:
[0163] An acquisition module 51 is used to acquire edge images and target segmentation results corresponding to each training image in an image training set;
[0164] A generating module 52, for generating a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image;
[0165] An input module 53, used to input each training image into the initial semantic segmentation model to obtain a predicted segmentation result;
[0166] A calculation module 54, used for calculating a loss value between a predicted segmentation result and a target segmentation result based on a pixel weight matrix;
[0167] The adjustment module 55 is used to adjust the parameters of the initial semantic segmentation model based on the loss value to determine the target semantic segmentation model.
[0168] In one embodiment of the present application, the above-mentioned calculation module 54 is specifically used to generate the sum of sample weights based on the values of each element in the pixel weight matrix; based on the pixel weight matrix and the sum of sample weights, calculate the cross entropy loss value between the predicted segmentation result and the target segmentation result.
[0169] In one embodiment of the present application, the above-mentioned calculation module 54 is specifically used to add the value of each element in the pixel weight matrix to the target parameter to obtain the target pixel weight matrix; and add the value of each target element in the target pixel weight matrix to generate the sum of sample weights.
[0170] In one embodiment of the present application, the above-mentioned calculation module 54 is specifically used to calculate the initial loss value between the predicted result and the actual result of each pixel in the training image; multiply each initial loss value by the corresponding target element in the target pixel weight matrix to obtain the target loss value corresponding to each pixel; add the target loss values corresponding to each pixel, and then divide them by the sum of the sample weights to obtain the cross entropy loss value between the predicted segmentation result and the target segmentation result.
[0171] In one embodiment of the present application, the above-mentioned generation module 52 is specifically used to filter the edge image to generate a target edge image; normalize the pixel value of each pixel in the target edge image to generate a pixel weight matrix.
[0172] In one embodiment of the present application, the above-mentioned acquisition module 51 is specifically used to acquire an image training set; each training image in the image training set is labeled; according to the label of each training image, the training image is converted into a grayscale image; the edge of the grayscale image is located to obtain an edge image corresponding to each training image.
[0173] like Fig. 9 As shown, this embodiment provides a semantic segmentation device, including:
[0174] An acquisition module 61 is used to acquire an image to be segmented;
[0175] The segmentation module 62 is used to input the image to be segmented into a target semantic segmentation model and output a segmentation result corresponding to the image to be segmented. The target semantic segmentation model is trained according to the semantic segmentation model training method in any of the above embodiments.
[0176] For the specific limitations and beneficial effects of the semantic segmentation model training device and the semantic segmentation device, please refer to the limitations of the semantic segmentation model training method and the semantic segmentation method above, which will not be repeated here. The various modules in the above-mentioned semantic segmentation model training device and the semantic segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0177] The embodiment of the present invention also provides an electronic device having the above Figure 8 The semantic segmentation model training device shown and Fig. 9 The semantic segmentation device shown.
[0178] like Fig.10 As shown, Fig.10 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the electronic device may include: at least one processor 71, such as a CPU (Central Processing Unit), at least one communication interface 73, a memory 74, and at least one communication bus 72. The communication bus 72 is used to realize the connection and communication between these components. The communication interface 73 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 73 may also include a standard wired interface and a wireless interface. The memory 74 may be a high-speed RAM memory (Random Access Memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 74 may optionally be at least one storage device located away from the aforementioned processor 71. The processor 71 may be combined with Figure 8 or Fig. 9 In the described device, the memory 74 stores an application program, and the processor 71 calls the program code stored in the memory 74 to execute any of the above method steps.
[0179] The communication bus 72 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 72 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0180] Among them, the memory 74 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 74 may also include a combination of the above types of memory.
[0181] The processor 71 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.
[0182] The processor 71 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0183] Optionally, the memory 74 is also used to store program instructions. The processor 71 can call the program instructions to implement the present application Figures 2 to 4 The semantic segmentation model training method shown in the embodiment and Figure 7 The semantic segmentation method shown.
[0184] The embodiment of the present invention also provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the semantic segmentation model training method and the semantic segmentation method in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.
[0185] This embodiment provides a kind of engineering machinery equipment, which can be an operating vehicle such as an intelligent vehicle, a heavy truck, a trailer, an excavator, an anchor digger, a bulldozer, a road roller, a concrete pump truck, or a mechanical operating equipment such as a tower crane, a construction elevator, and a material hoist. The engineering machinery equipment may include an equipment body and the above-mentioned electronic equipment. Among them, the electronic equipment is connected to the equipment body, and the electronic equipment can be connected to the equipment body according to the needs. The specific connection method and setting position are not limited here. The electronic equipment can be used to perform the above-mentioned Figure 2-Figure 4 Semantic segmentation model training method and Figure 7 The semantic segmentation method shown.
[0186] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A semantic segmentation model training method, characterized in that: include: Obtain edge images and target segmentation results corresponding to each training image in the image training set; Generating a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image; Inputting each of the training images into an initial semantic segmentation model to obtain a predicted segmentation result; Calculate the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix; Adjusting the parameters of the initial semantic segmentation model based on the loss value to determine a target semantic segmentation model; The step of calculating the loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix includes: Generate a sample weight sum based on the value of each element in the pixel weight matrix; Calculating a cross entropy loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix and the sum of the sample weights; The step of generating the sample weight sum based on the value of each element in the pixel weight matrix includes: Add the target parameter to the value of each element in the pixel weight matrix to obtain a target pixel weight matrix; Adding the values of each target element in the target pixel weight matrix to generate the sample weight sum; The step of calculating the cross entropy loss value between the predicted segmentation result and the target segmentation result based on the pixel weight matrix and the sum of the sample weights includes: Calculating an initial loss value between a predicted result and a true result for each pixel in the training image; Multiply each of the initial loss values by the corresponding target element in the target pixel weight matrix to obtain a target loss value corresponding to each pixel; After adding the target loss values corresponding to each pixel, the result is divided by the sum of the sample weights to obtain a cross entropy loss value between the predicted segmentation result and the target segmentation result.
2. The method according to claim 1, characterized in that Generating a pixel weight matrix corresponding to the edge image according to the pixel value of each pixel in the edge image includes: Performing filtering processing on the edge image to generate a target edge image; The pixel value of each pixel in the target edge image is normalized to generate the pixel weight matrix.
3. The method according to claim 1, characterized in that The step of obtaining edge images corresponding to each training image in the image training set includes: Acquire the image training set; each of the training images in the image training set is labeled; According to the labels of the training images, the training images are converted into grayscale images; The edges of the grayscale images are located to obtain edge images corresponding to the training images.
4. A semantic segmentation method, characterized in that: The method comprises: Obtain the image to be segmented; The image to be segmented is input into a target semantic segmentation model, and a segmentation result corresponding to the image to be segmented is output, wherein the target semantic segmentation model is trained according to the semantic segmentation model training method according to any one of claims 1-3.
5. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer instructions, and the processor executes the semantic segmentation model training method described in any one of claims 1 to 3 and the semantic segmentation method described in claim 4 by executing the computer instructions.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the semantic segmentation model training method described in any one of claims 1 to 3 and the semantic segmentation method described in claim 4.
7. An engineering machinery equipment, characterized in that: The engineering machinery equipment includes: A device body and the electronic device described in claim 5, wherein the device body is connected to the electronic device, and the electronic device is used to train a target semantic segmentation model and complete a semantic segmentation method based on the target semantic segmentation model.
