Training method of semantic segmentation model, semantic segmentation method and device

Abstract

The embodiment of the present application provides a training method for a semantic segmentation model, a semantic segmentation method and a device method and a device, which relate to the field of image processing technology, including: obtaining a target feature image by inputting a plurality of acquired sample images into a semantic segmentation model, calculating a loss value of the target feature image using a cross entropy loss function, and training the model according to the loss value, wherein the semantic segmentation model includes a plurality of attention models. Since the attention model can fully extract the feature information of the sample image, using the trained semantic segmentation model to segment the image can effectively improve the accuracy of image segmentation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for a semantic segmentation model, a semantic segmentation method and a device. Background Art

[0002] At present, more and more application scenarios require semantic segmentation of video images, such as autonomous driving, indoor navigation, virtual reality, image recognition, etc.

[0003] UNet, also known as U-Net convolutional neural network, is a commonly used semantic segmentation model. This model is based on a fully convolutional network and can perform convolution operations on images of any shape and size, achieving good results in complex image segmentation.

[0004] However, UNet directly uses convolutional networks to extract image features. Therefore, the accuracy of feature extraction will decrease, which will reduce the accuracy of image segmentation to a certain extent. Summary of the invention

[0005] The embodiments of the present application provide a training method for a semantic segmentation model, a semantic segmentation method and a device to improve the accuracy of image segmentation.

[0006] In a first aspect, an embodiment of the present application provides a method for training a semantic segmentation model, comprising:

[0007] Acquire multiple sample images; input the sample images into a semantic segmentation model to obtain a target feature image corresponding to the sample image; wherein the semantic segmentation model includes multiple attention models, each attention model includes left and right branches, the left branch is used to obtain feature parameters of the sample image layer dimension, and the right branch is used to obtain feature parameters of the sample image width and height dimensions; a cross entropy loss function is used to calculate a loss value for the target feature image; and the semantic segmentation model is trained according to the loss value.

[0008] Optionally, obtain multiple sample images, including:

[0009] Acquire multiple original images taken by different users in different shooting scenes; randomly rotate the original images to obtain multiple sample images, wherein the number of sample images is greater than the number of the original images.

[0010] Optionally, the semantic segmentation model includes: a feature extraction layer, a downsampling processing layer, and an upsampling processing layer; the downsampling processing layer includes a downsampling layer and a first attention model; the upsampling processing layer includes an upsampling layer and a second attention model, and the first attention model and the second attention model have the same structure;

[0011] Inputting the sample image into the semantic segmentation model to obtain a target feature image corresponding to the sample image includes:

[0012] The sample image is input into the feature extraction layer to obtain a first feature image; the first feature image is input into the first attention model to obtain a second feature image, and the second feature image is input into the downsampling layer to obtain a third feature image; the third feature image is input into the upsampling layer to obtain a fourth feature image, and the fourth feature image is input into the second attention model to obtain the target feature image.

[0013] Optionally, inputting the first feature image into the first attention model to obtain the second feature image includes:

[0014] The first feature image is input into the left branch of the first attention model to obtain the left feature image output by the left branch; the first feature image is input into the right branch of the first attention model to obtain the right feature image output by the right branch; the left feature image and the right feature image are dot-multiplied to obtain the second feature image.

[0015] Optionally, the left branch of the attention model includes: left 1 branch, left 2 branch, and left 3 branch; the left 1 branch includes a first processing layer and a second processing layer, and the left 2 branch includes a third processing layer;

[0016] Inputting the first feature image into the left branch of the first attention model to obtain the left feature image output by the left branch includes:

[0017] The first feature image is input into the first processing layer to obtain the first reduced dimension feature image; the first feature image is input into the third processing layer to obtain the second reduced dimension feature image; the first reduced dimension feature image and the second reduced dimension feature image are matrix multiplied and input into the second processing layer to obtain the fifth feature image; the fifth feature image is dot multiplied with the first feature image of the left 3 branches to obtain the left feature image.

[0018] Optionally, the right branch of the attention model includes: right branch 1, right branch 2, and right branch 3; the right branch 1 includes a fourth processing layer and a fifth processing layer, and the right branch 2 includes a sixth processing layer;

[0019] Inputting the first feature image into the right branch of the first attention model to obtain the right feature image output by the right branch includes:

[0020] The first feature image is input into the fourth processing layer to obtain the third reduced dimension feature image; the first feature image is input into the sixth processing layer to obtain the fourth reduced dimension feature image; the third reduced dimension feature image is matrix multiplied with the fourth reduced dimension feature image, and the result is input into the fifth processing layer to obtain the sixth feature image; the sixth feature image is dot multiplied with the first feature image of the right 3 branch to obtain the right feature image.

[0021] In a second aspect, an embodiment of the present application provides a segmentation method, including:

[0022] Obtain an image to be processed; input the image to be processed into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

[0023] In a third aspect, an embodiment of the present application provides a training device for a semantic segmentation model, comprising:

[0024] An acquisition module is used to acquire multiple sample images; a first training module is used to input the sample images into a semantic segmentation model to obtain feature images corresponding to the sample images; a calculation module is used to calculate the loss value of the feature image using a cross entropy loss function; and a second training module is used to train the semantic segmentation model according to the loss value.

[0025] In a fourth aspect, an embodiment of the present application provides a semantic segmentation device, including:

[0026] The acquisition module is used to acquire the image to be processed; the processing module is used to input the image to be processed into the semantic segmentation model to obtain the semantic segmentation result output by the semantic segmentation model.

[0027] In a fifth aspect, an embodiment of the present application provides a terminal device, including: a memory and a processor;

[0028] The memory is used to store computer instructions; the processor is used to execute the computer instructions stored in the memory to implement any method in the first aspect or the second aspect.

[0029] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the method of any one of the first aspect or the second aspect.

[0030] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method of any one of the first aspect or the second aspect when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a scenario provided for an embodiment of the present application;

[0032] Figure 2 A flowchart of a method for training a semantic segmentation model provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the structure of the semantic segmentation model provided in the embodiment of the present application;

[0034] Figure 4A schematic diagram of the structure of the attention model provided in the embodiment of the present application;

[0035] Figure 5 A flowchart of a semantic segmentation method provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of a training device for a semantic segmentation model provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of the structure of a semantic segmentation device provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0040] 1) Semantic segmentation refers to the classification of images at the pixel level, grouping pixels belonging to the same category into one category.

[0041] 2) U-Net convolutional neural network is a classic full convolutional network. There is no full connection operation in the network. It is mainly divided into two processes: upsampling and downsampling. The entire network is shaped like a "U", so it is called U-net. Except for the last output layer, all other convolutional layers in the network are 3*3 convolutions.

[0042] 3) The attention model can selectively extract a series of regions from the image, process only the extracted regions each time, and then combine the processed information to establish the corresponding scene information.

[0043] 4) Other terms

[0044] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects, and do not limit their order. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0045] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0046] The following is a detailed introduction to the training method of the semantic segmentation model provided in the embodiment of the present application in conjunction with the accompanying drawings. It should be noted that the "at..." in the embodiment of the present application can be the instant when a certain situation occurs, or it can be a period of time after a certain situation occurs, and the embodiment of the present application does not specifically limit this.

[0047] At present, more and more application scenarios require semantic segmentation of video images, such as autonomous driving, indoor navigation, virtual reality, image recognition, etc.

[0048] UNet, also known as U-Net convolutional neural network, is a commonly used semantic segmentation model. This model is based on a fully convolutional network and can perform convolution operations on images of any shape and size, achieving good results in complex image segmentation.

[0049] However, UNet directly uses convolutional networks to extract image features. Therefore, the accuracy of feature extraction will decrease, which will reduce the accuracy of image segmentation to a certain extent.

[0050] In view of this, the present application proposes a training method for a semantic segmentation model and a semantic segmentation method, which is based on the U-net model, introduces an attention model, and redesigns the model structure. It can more comprehensively extract and process the feature parameters of the image to be segmented, and obtain more accurate segmentation results.

[0051] The following is a detailed description of the technical solution of the present invention and how the technical solution of the present application solves the above-mentioned technical problems with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings. For ease of description, the embodiments of the present application are described using face recognition as an example, and this example does not constitute a limitation on the application scenarios of the present application.

[0052] Figure 1 This is a schematic diagram of a scenario of an embodiment of the present application, such as Figure 1 As shown, it includes a server and a terminal device, and the terminal device is connected to the server through a network for data transmission. The terminal device is provided with a client for collecting user face images, and the server is provided with a model for semantic segmentation of user face images. The terminal device sends the collected user face images to the server, and the server segments the face images and returns the segmentation results.

[0053] It is understandable that the server may also obtain multiple face images as sample images to train the semantic segmentation model to obtain the expected output result.

[0054] The above briefly describes the application scenarios provided by the embodiments of the present application. Figure 1 Taking the server in as an example, the training method of the semantic segmentation model provided in the embodiment of the present application is described in detail.

[0055] Figure 2 A flowchart of a method for training a semantic segmentation model provided in an embodiment of the present application includes the following steps:

[0056] S201. Acquire multiple sample images.

[0057] By interacting with the terminal device, the server can obtain facial images taken by multiple users in different shooting scenes, and use the facial images as sample images for training the semantic segmentation model.

[0058] Optionally, in order to further enrich sample images required for training, multiple face images acquired by different users in different shooting scenes may be randomly rotated to obtain richer sample images.

[0059] Exemplarily, the obtained user face image may be randomly rotated 90 degrees, flipped horizontally, and flipped upside down to obtain a plurality of sample images.

[0060] S202: Input the sample image into the semantic segmentation model to obtain a target feature image corresponding to the sample image.

[0061] The semantic segmentation model of the embodiment of the present application adopts the U-net model as the basic structure, and introduces the attention model into the U-net model. Correspondingly, the semantic segmentation model includes multiple attention models, each of which includes left and right branches. The left branch is used to obtain the feature parameters of the sample image layer dimension, and the right branch is used to obtain the feature parameters of the sample image width and height dimensions.

[0062] An image is usually composed of a three-dimensional array, which is height, width and number of layers. The number of layers is the red, green and blue values ​​of the image pixels. The three-dimensional array of an image can also be called the feature parameters of the image. Through the different branch processing of the attention model, the feature parameters in the sample image can be better extracted.

[0063] Optionally, the semantic segmentation model may further include a feature extraction layer for extracting feature parameters of the sample image to obtain a corresponding feature image.

[0064] S203, using a cross entropy loss function to calculate a cross entropy loss value for the target feature image.

[0065] Specifically, the cross entropy loss value can be obtained according to the true value of the sample image pixel and the pixel value of the sample image output by the model.

[0066] Exemplarily, the calculation method of the cross entropy loss value can refer to the following formula:

[0067]

[0068] Among them, ytrue is the true value of the sample image pixel, i represents the i-th sample image, n represents the number of sample images, and ypred is the pixel value of the sample image output by the model.

[0069] S204: Train the semantic segmentation model according to the loss value.

[0070] After obtaining the cross entropy loss value, determine whether the cross entropy loss value meets the preset value. If the cross entropy loss value does not meet the preset value, adjust the parameters of the semantic segmentation model based on the cross entropy loss value, and repeat the process shown in S201-S203 until the cross entropy loss value meets the preset value.

[0071] Exemplarily, the preset value of the cross entropy loss value may be 0.2. When the loss value of the target feature image calculated by the server is greater than 0.2, the parameters of each layer of the semantic segmentation model are adjusted, for example, the size of the convolution kernel may be adjusted. The process from S201 to S203 is repeated multiple times until the loss value of the target feature image is less than or equal to the preset value.

[0072] The training method of the semantic segmentation model provided in the embodiment of the present application obtains a target feature image by inputting a plurality of sample images into the semantic segmentation model, calculates the loss value of the target feature image using a cross entropy loss function, and trains the model according to the loss value, wherein the semantic segmentation model includes a plurality of attention models. Since the attention model can fully extract the feature information of the sample image, the image can be segmented using the trained semantic segmentation model, which can effectively improve the accuracy of image segmentation.

[0073] The following will Figure 2 On the basis of the illustrated embodiment, taking the input of a sample image into a semantic segmentation model as an example, the structure of the semantic segmentation model is described in detail.

[0074] like Figure 3 As shown, the semantic segmentation model includes a feature extraction layer, a downsampling processing layer, an upsampling processing layer and a convolution layer.

[0075] Among them, the downsampling processing layer includes a first downsampling processing layer, a second downsampling processing layer and a third downsampling processing, each downsampling processing layer includes an attention model and a downsampling layer, each downsampling layer includes a convolution layer (conv), an activation layer (relu) and a pooling layer (maxpool), the convolution kernel size of each convolution layer is 3*3, and the convolution kernel size of each pooling layer is 2*2.

[0076] The upsampling processing layer includes a first upsampling processing layer, a second upsampling processing layer and a third upsampling processing, each upsampling processing layer includes an attention model and an upsampling layer, each upsampling layer includes a convolution layer, an activation layer and an upsampling operation layer, the convolution kernel size of each convolution layer is 3*3, and the convolution kernel size of each upsampling operation layer is 2*2.

[0077] Correspondingly, inputting the sample image into the semantic segmentation model to obtain the target feature image corresponding to the sample image includes the following steps.

[0078] S301: Input a sample image into a feature extraction layer to obtain a first feature image.

[0079] The feature extraction layer is used to extract the feature parameters of the sample image, including feature parameters in three dimensions: number of layers, height, and width.

[0080] Exemplarily, the first feature image output by the feature extraction layer can be represented in the form of feature parameters, denoted as C*H*W, where C is the number of layers, H is the height, and W is the width.

[0081] S302: Input the first feature image into the first attention model to obtain the second feature image, and input the second feature image into the downsampling layer to obtain the third feature image.

[0082] The semantic segmentation model may include multiple downsampling processing layers. The embodiment of the present application takes three downsampling processing layers as an example for illustration. Each downsampling processing layer includes a first attention model and a downsampling layer. The detailed process of obtaining the second feature image from the first feature image and then obtaining the third feature image from the second feature image is shown as follows. In the process of obtaining the second feature image from the first feature image, multiple feature images will be output, which are distinguished by the Nth feature image and the N+1th feature image.

[0083] S3021. Input the first feature image into the first downsampling processing layer to obtain an Nth feature image.

[0084] Among them, the first feature image is first input into the first attention model in the first downsampling processing layer. The first attention model can weight the feature parameters of the image, but does not change the size of the image. The feature image output by the first attention model is still C*H*W. The feature image output by the first attention model is input into the downsampling layer to obtain the Nth feature image. The feature parameters of the Nth feature image are 2C*H / 2*W / 2.

[0085] S3022. Input the Nth feature image into the second downsampling processing layer to obtain the N+1th feature image.

[0086] The Nth feature image is input into the first attention model in the second downsampling processing layer to obtain the feature image corresponding to the output, and is input into the downsampling layer in the second downsampling processing layer to obtain the N+1th feature image. The feature parameters of the N+1th feature image can be expressed as 4C*H / 4*W / 4.

[0087] S3023. Input the N+1th feature image into the third downsampling processing layer to obtain a third feature image.

[0088] The N+1th feature image is input into the first attention model in the third downsampling processing layer to obtain the second feature image. The second feature image is input into the downsampling layer in the third downsampling processing layer to obtain the third feature image. The feature parameters of the third feature image can be expressed as 8C*H / 8*W / 8.

[0089] S303: Input the third feature image into the upsampling layer to obtain a fourth feature image, and input the fourth feature image into the second attention model to obtain a target feature image.

[0090] Each upsampling processing layer includes a second attention model and an upsampling layer. The process of obtaining the target image from the third feature image is shown as follows. In the process of obtaining the fourth feature image from the third feature image and obtaining the target feature image from the fourth feature image, multiple other feature images will also be output, and the naming method continues the method shown in S302.

[0091] S3031. Input the third feature image into the third upsampling processing layer to obtain the N+2th feature image.

[0092] The third feature image is input into the upsampling layer in the third upsampling processing layer to obtain the corresponding feature image, and the feature image is input into the second attention model in the third upsampling processing layer to obtain the N+2th feature image. The feature parameters of the N+2th feature image can be expressed as 8C*H / 4*W / 4.

[0093] S3032. Input the N+2th feature image into the second upsampling processing layer to obtain the N+3th feature image.

[0094] The N+2th feature image is input into the upsampling layer in the second upsampling processing layer to obtain the corresponding feature image, and the feature image is input into the second attention model in the second upsampling processing layer to obtain the N+3th feature image. The feature parameters of the N+3th feature image can be expressed as 4C*H / 2*W / 2.

[0095] S3033. Input the N+3th feature image into the first upsampling processing layer to obtain the N+4th feature image.

[0096] The N+3th feature image is input into the upsampling layer in the first upsampling processing layer to obtain the fourth feature image, and the fourth feature image is input into the second attention model in the first upsampling processing layer to obtain the N+4th feature image. The feature parameters of the N+4th feature image can be expressed as 2C*H*W.

[0097] S3034. Input the N+4th feature image into the convolution layer to obtain the target feature image.

[0098] The convolution layer is used to process the feature parameters of the N+4th feature image into feature parameters having the same scale as the first feature image.

[0099] Exemplarily, the feature parameters of the N+4th feature image are 2C*H*W, and the N+4th image is input into the convolution layer to obtain the target feature image, and the feature parameters of the target image are C*H*W.

[0100] The structure of the semantic segmentation model is described above. Figure 2 Based on the embodiments, the structure of the attention model in the semantic segmentation model is explained.

[0101] Figure 4 A schematic diagram of the structure of the attention model provided in the embodiment of the present application, such as Figure 4 As shown, the attention model consists of two branches, left and right.

[0102] The left branch includes: left branch 1, left branch 2, and left branch 3; left branch 1 includes the first processing layer and the second processing layer, and left branch 2 includes the third processing layer. The right branch includes: right branch 1, right branch 2, and right branch 3; right branch 1 includes the fourth processing layer and the fifth processing layer, and right branch 2 includes the sixth processing layer.

[0103] The first processing layer includes a convolution layer and a reconstruction layer (reshape); the second processing layer includes a convolution layer, a reconstruction layer and a classification layer (softmax); the third processing layer includes a convolution layer, a normalization layer (layernorm) and an activation layer (sigmod); the fourth processing layer includes a convolution layer and a reconstruction layer; a reconstruction layer and an activation layer; the sixth processing layer includes a convolution layer, a global pooling layer (globelpooling), a reconstruction layer and a classification layer. The convolution kernel size of each convolution layer is 1*1

[0104] Correspondingly, inputting the first feature image into the first attention model to obtain the second feature image includes:

[0105] S401. Input the first feature image into the left branch of the first attention model to obtain the left feature image output by the left branch.

[0106] In the specific implementation process, the first feature image is input into the first processing layer to obtain the first reduced dimension feature image; the first feature image is input into the third processing layer to obtain the second reduced dimension feature image; the first reduced dimension feature image and the second reduced dimension feature image are matrix multiplied and input into the second processing layer to obtain the fifth feature image; the fifth feature image is dot multiplied with the first feature image of the left 3 branches to obtain the left feature image.

[0107] Exemplarily, the characteristic parameter of the first feature map is C*H*W, which is input into the first processing layer. After being processed by the convolution layer, the characteristic parameter is C / 2*H*W, which is then input into the reconstruction layer for processing to obtain the first dimension reduction feature image, whose characteristic parameter is C / 2*HW. The reconstruction layer is used to reconstruct the shape of the feature image.

[0108] The first feature map is input into the third processing layer. After being processed by the convolution layer, the feature parameter is 1*H*W. It is then input into the reconstruction layer and the classification layer for processing to obtain the second dimensionality reduction feature image, whose feature parameter is HW*1*1.

[0109] The first reduced dimension feature image and the second reduced dimension feature image are matrix multiplied, and their characteristic parameters are C / 2*1*1, which are then input into the second processing layer, and processed by the convolution layer, the normalization layer, and the activation layer to obtain the fifth feature image, whose characteristic parameters are C*1*1. Among them, the normalization layer processing refers to the standardization of image features, pulling the data distribution to the unsaturated area of ​​the activation function, and has the characteristics of weight / data scaling invariance. It has the effect of alleviating gradient disappearance / explosion, accelerating training, and regularization.

[0110] The fifth feature image is dot-multiplied with the first feature image to obtain a left feature image, whose feature parameters are C*H*W.

[0111] S402. Input the first feature image into the right branch of the first attention model to obtain the right feature image output by the right branch.

[0112] In the specific implementation, the first feature image is input into the fourth processing layer to obtain the third reduced-dimensionality feature image; the first feature image is input into the sixth processing layer to obtain the fourth reduced-dimensionality feature image; the third reduced-dimensionality feature image is matrix multiplied with the second fourth reduced-dimensionality feature image and input into the fifth processing layer to obtain the sixth feature image; the sixth feature image is dot-multiplied with the first feature image of the right 3 branches to obtain the right feature image.

[0113] Exemplarily, the first feature map is input into the fourth processing layer, and after being processed by the convolution layer, the feature parameter is C / 2*H*W, and then input into the reconstruction layer for processing to obtain a third reduced-dimensional feature image, whose feature parameter is C / 2*HW.

[0114] The first feature map is input into the sixth processing layer. After being processed by the convolution layer, the feature parameter is C / 2*H*W. It is then input into the global pooling layer for processing with the feature parameter being C / 2*1*1. Finally, it is input into the reconstruction layer and the classification layer for processing to obtain the fourth dimensionality reduction feature image with the feature parameter being 1*C / 2.

[0115] The third dimensionality reduction feature image is matrix multiplied with the fourth dimensionality reduction feature image, and its feature parameter becomes 1*HW, which is then input into the fifth processing layer, and processed by the reconstruction layer and the activation layer to obtain the sixth feature image, whose feature parameter is 1*H*W.

[0116] The sixth feature image is dot-multiplied with the first feature image to obtain a right feature image, whose feature parameters are C*H*W.

[0117] S403: Perform a dot product of the left feature image and the right feature image to obtain a second feature image.

[0118] In the above method, the left branch of the attention model performs dimensionality reduction processing on the first feature image to obtain two feature images with different dimensionality reduction, and then performs matrix multiplication processing, and finally performs dot multiplication processing with the first feature image. It can not only better extract the layer dimension information in the first feature image, but also retain the information of the two dimensions of height and width. The right branch also uses a similar method to better extract the two dimensional information of height and width in the first feature image, and also retains the information of the layer dimension. Finally, the feature images output by the left and right branches are processed by dot multiplication to fully extract the information of the three dimensions in the first feature image. Thereby improving the output results of the model.

[0119] The present application also provides a semantic segmentation method, such as Figure 5 As shown, the following steps are included:

[0120] S501: Obtain an image to be processed.

[0121] S502: Input the image to be processed into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model.

[0122] Among them, the semantic segmentation model is Figure 2 The semantic segmentation model training method shown is obtained.

[0123] In the above method, an attention model is introduced into the semantic segmentation model. The attention model can take advantage of the weighted processing of image feature information, fully extract and process the feature information of the sample image, and improve the accuracy of image segmentation by the semantic segmentation model.

[0124] The present application also provides a training device for a semantic segmentation model. Figure 6As shown, it includes: an acquisition module 601, a first training module 602, a calculation module 603 and a second training module 604.

[0125] The acquisition module 601 is used to acquire a plurality of sample images.

[0126] The first training module 602 is used to input the sample image into the semantic segmentation model to obtain the feature image corresponding to the sample image.

[0127] The calculation module 603 is used to calculate the loss value of the feature image using the cross entropy loss function.

[0128] The second training module 604 is used to train the segmentation model according to the loss value.

[0129] The training device of the semantic segmentation model provided in this embodiment can execute Figure 2 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0130] Further, in Figure 6 Based on the illustrated embodiment, the semantic segmentation model training device provided by the present application further includes a preprocessing module 605.

[0131] The acquisition module 601 is further used to acquire multiple original images taken by different users in different shooting scenes.

[0132] The preprocessing module 605 is used to randomly rotate the original image to obtain multiple sample images.

[0133] The first training module 602 is also used to input the sample image into the feature extraction layer to obtain a first feature image; input the first feature image into the first attention model to obtain a second feature image, and input the second feature image into the downsampling layer to obtain a third feature image; input the third feature image into the upsampling layer to obtain a fourth feature image, and input the fourth feature image into the second attention model to obtain a target feature image.

[0134] The present application embodiment also provides a semantic segmentation device 70, such as Figure 7 As shown, it includes: an acquisition module 701 and a processing module 702.

[0135] An acquisition module, used for acquiring an image to be processed;

[0136] The processing module is used to input the image to be processed into the semantic segmentation model to obtain the semantic segmentation result output by the semantic segmentation model.

[0137] The semantic segmentation model device provided in this embodiment can be executed Figure 5The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0138] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the device 80 provided in this embodiment may include:

[0139] Processor 801.

[0140] The memory 802 is used to store executable instructions of the electronic device.

[0141] Among them, the processor is configured to execute the technical solution of the above-mentioned semantic segmentation model training method or semantic segmentation method embodiment by executing executable instructions. Its implementation principle and technical effect are similar and will not be repeated here.

[0142] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the technical solution of the above-mentioned semantic segmentation model training method or semantic segmentation method embodiment is implemented. The implementation principle and technical effect are similar and will not be repeated here.

[0143] In one possible implementation, a computer-readable medium may include a random access memory (RAM), a read-only memory (ROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage device, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of the medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVDs), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0144] A computer program product is also provided in an embodiment of the present application, including a computer program. When the computer program is executed by a processor, it implements the technical solution of the above-mentioned semantic segmentation model training method or semantic segmentation method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0145] In the specific implementation of the above-mentioned terminal device or server, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0146] Those skilled in the art will appreciate that all or part of the steps of any of the above method embodiments may be completed by hardware associated with program instructions. The aforementioned program may be stored in a computer-readable storage medium, and when the program is executed, all or part of the steps of the above method embodiments are executed.

[0147] If the technical solution of the present application is implemented in the form of software and sold or used as a product, it can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a computer program or several instructions. The computer software product enables a computer device (which can be a personal computer, a server, a network device, or a similar electronic device) to perform all or part of the steps of the method described in the embodiment of the present application.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a semantic segmentation model, characterized in that: include: Acquire multiple sample images; Input the sample image into a semantic segmentation model to obtain a target feature image corresponding to the sample image; wherein the semantic segmentation model includes a plurality of attention models, each of the attention models includes a left branch and a right branch, the left branch is used to obtain feature parameters of the sample image layer dimension, and the right branch is used to obtain feature parameters of the sample image width and height dimensions; Calculating a loss value of the target feature image using a cross entropy loss function; Training the semantic segmentation model according to the loss value; The semantic segmentation model further includes: a feature extraction layer, a downsampling processing layer and an upsampling processing layer; the downsampling processing layer includes a downsampling layer and a first attention model; the upsampling processing layer includes an upsampling layer and a second attention model, and the first attention model and the second attention model have the same structure; The step of inputting the sample image into a semantic segmentation model to obtain a target feature image corresponding to the sample image includes: Inputting the sample image into the feature extraction layer to obtain a first feature image; Inputting the first feature image into the first attention model to obtain a second feature image, and inputting the second feature image into the downsampling layer to obtain a third feature image; The third feature image is input into the upsampling layer to obtain a fourth feature image, and the fourth feature image is input into the second attention model to obtain the target feature image.

2. The method according to claim 1, characterized in that The acquiring of a plurality of sample images comprises: Acquire multiple original images taken by different users in different shooting scenes; The original image is randomly rotated to obtain a plurality of sample images, wherein the number of the sample images is greater than the number of the original images.

3. The method according to claim 1, characterized in that The step of inputting the first feature image into the first attention model to obtain a second feature image includes: Inputting the first feature image into the left branch of the first attention model to obtain a left feature image output by the left branch; Inputting the first feature image into the right branch of the first attention model to obtain a right feature image output by the right branch; Perform a dot product of the left feature image and the right feature image to obtain the second feature image.

4. The method according to claim 3, characterized in that The left branch of the attention model includes: left 1 branch, left 2 branch, and left 3 branch; the left 1 branch includes a first processing layer and a second processing layer, and the left 2 branch includes a third processing layer; The step of inputting the first feature image into the left branch of the first attention model to obtain the left feature image output by the left branch includes: Inputting the first feature image into the first processing layer to obtain a first dimension-reduced feature image; Inputting the first feature image into the third processing layer to obtain a second dimension-reduced feature image; Performing matrix multiplication on the first reduced-dimensionality feature image and the second reduced-dimensionality feature image, and inputting the matrix multiplication into the second processing layer to obtain a fifth feature image; The fifth characteristic image is dot-multiplied with the first characteristic image of the left 3 branches to obtain a left characteristic image.

5. The method according to claim 3, characterized in that: The right branch of the attention model includes: right 1 branch, right 2 branch, and right 3 branch; the right 1 branch includes a fourth processing layer and a fifth processing layer, and the right 2 branch includes a sixth processing layer; The step of inputting the first feature image into the right branch of the first attention model to obtain the right feature image output by the right branch includes: Inputting the first feature image into the fourth processing layer to obtain a third dimension-reduced feature image; Inputting the first feature image into the sixth processing layer to obtain a fourth dimension-reduced feature image; Performing matrix multiplication on the third reduced-dimensionality feature image and the fourth reduced-dimensionality feature image, and inputting the result into the fifth processing layer to obtain a sixth feature image; The sixth characteristic image is dot-multiplied with the first characteristic image of the right 3 branches to obtain a right characteristic image.

6. A semantic segmentation method, characterized in that: include: Get the image to be processed; The image to be processed is input into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model, wherein the semantic segmentation model is trained by the method described in any one of items 1 to 5 above.

7. A training device for a semantic segmentation model, characterized in that: include: An acquisition module, used for acquiring a plurality of sample images; A first training module is used to input the sample image into a semantic segmentation model to obtain a target feature image corresponding to the sample image; wherein the semantic segmentation model includes a plurality of attention models, each of the attention models includes a left branch and a right branch, the left branch is used to obtain feature parameters of the sample image layer dimension, and the right branch is used to obtain feature parameters of the sample image width and height dimensions; A calculation module, used for calculating a loss value of the feature image using a cross entropy loss function; A second training module, used for training the semantic segmentation model according to the loss value; The semantic segmentation model further includes: a feature extraction layer, a downsampling processing layer and an upsampling processing layer; the downsampling processing layer includes a downsampling layer and a first attention model; the upsampling processing layer includes an upsampling layer and a second attention model, and the first attention model and the second attention model have the same structure; The first training module is specifically used to input the sample image into the feature extraction layer to obtain a first feature image; input the first feature image into the first attention model to obtain a second feature image, and input the second feature image into the downsampling layer to obtain a third feature image; input the third feature image into the upsampling layer to obtain a fourth feature image, and input the fourth feature image into the second attention model to obtain the target feature image.

8. A semantic segmentation device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A processing module is used to input the image to be processed into a semantic segmentation model to obtain a semantic segmentation result output by the semantic segmentation model, wherein the semantic segmentation model is trained by the method described in any one of items 1 to 5 above.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 5, or to implement the method according to claim 6.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 5, or to implement the method according to claim 6.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented, or the method according to claim 6 is implemented.

Images