A semantic segmentation method, a method for training a semantic segmentation model, and an electronic device

Abstract

The application provides a semantic segmentation method, a method for training a semantic segmentation model and an electronic device, and relates to the technical field of image processing. The method can improve the feature extraction capability of the semantic segmentation model in the encoding process, thereby making the image segmentation result have more rich details and improving the accuracy of the image segmentation result. The method comprises: acquiring a first image; inputting the first image into a semantic segmentation model to obtain a semantic segmentation result; wherein the semantic segmentation model comprises first model parameters, the first model parameters are associated with a first loss function, the first loss function is determined according to a first feature map and a second feature map, the first feature map is a feature map generated in the encoding process of a basic semantic segmentation model used to generate the semantic segmentation model, the second feature map is a feature map generated in the decoding process of the basic semantic segmentation model, and the resolutions of the first feature map and the second feature map are the same.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a semantic segmentation method, a method for training a semantic segmentation model, and an electronic device. Background Technology

[0002] Image semantic segmentation is an important research area in computer vision. Its goal is to segment an image into regions with different semantic information and label each region with a corresponding semantic label.

[0003] Currently, using deep neural networks to process image semantic segmentation tasks is a common approach in the industry, but this approach suffers from poor semantic segmentation results. Summary of the Invention

[0004] This application provides a semantic segmentation method, a method for training a semantic segmentation model, and an electronic device, which can improve the feature extraction capability of the semantic segmentation model in the encoding process, thereby making the image segmentation result have richer details and reducing the risk of false positives in the image segmentation result.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, a semantic segmentation method is provided, comprising: acquiring a first image; inputting the first image into a semantic segmentation model to obtain a semantic segmentation result; wherein the semantic segmentation result is used to indicate the category information of each pixel in the first image. In this embodiment, the semantic segmentation model includes first model parameters, which are associated with a first loss function. The first loss function is determined based on a first feature map and a second feature map. The first feature map is a feature map generated during the encoding process of the basic semantic segmentation model used to generate the semantic segmentation model, and the second feature map is a feature map generated during the decoding process of the basic semantic segmentation model. The first feature map and the second feature map have the same resolution. Understandably, considering that the feature map in the decoding process often has more and richer features than the feature map in the encoding process, adjusting the first model parameters through the first loss function can distill the richer features from the decoding stage to the encoding stage, thereby improving the feature extraction capability of the semantic segmentation model in the encoding process. This results in the image segmentation result output by the semantic segmentation model having richer details and improving the accuracy of the image segmentation result.

[0007] In one embodiment provided in the first aspect, the semantic segmentation model is obtained by training a basic semantic segmentation model based on training samples, wherein the training process includes: iteratively training the basic semantic segmentation model based on training samples; if the basic semantic segmentation model does not converge, adjusting the model parameters of the basic semantic segmentation model based on a first loss function; if the basic semantic segmentation model converges, stopping the iterative training, and obtaining the semantic segmentation model.

[0008] In one implementation provided in the first aspect, the basic semantic segmentation model is used to perform N downsampling and N upsampling operations on the training samples. The first feature map is the feature map processed by the i-th downsampling operation of the basic semantic segmentation model, and the second feature map is the feature map obtained by the j-th upsampling operation of the basic semantic segmentation model, where i+j-1=N, i≤N. This ensures that the first and second feature maps have the same resolution.

[0009] In one embodiment provided in the first aspect, the semantic segmentation model includes an encoder-decoder network and a prediction head; inputting a first image into the semantic segmentation model to obtain a semantic segmentation result includes: performing N downsampling and N upsampling operations on the first image sequentially through the encoder-decoder network to obtain a first intermediate feature map; and processing the first intermediate feature map through the prediction head to obtain the semantic segmentation result.

[0010] In one embodiment provided in the first aspect, the encoder-decoder network includes a convolution module, N downsampling modules, a pooling module, and N upsampling modules. The encoder-decoder network sequentially performs N downsampling and N upsampling operations on the input image corresponding to the first image to obtain a first intermediate feature map, including: performing a convolution operation on the first image using the convolution module to obtain a second intermediate feature map; sequentially downsampling the second intermediate feature map using the N downsampling modules to obtain a third intermediate feature map; processing the third intermediate feature map using the pooling module to obtain a fourth intermediate feature map; and sequentially upsampling the fourth intermediate feature map using the N upsampling modules to obtain the first intermediate feature map.

[0011] In one embodiment provided in the first aspect, the N downsampling modules include a first downsampling module, a second downsampling module, and a third downsampling module. The first downsampling module includes a first residual network, the second downsampling module includes a first residual network and a second residual network, and the third downsampling module includes a first residual network, a second residual network, and a third residual network. The network structures of the first residual network, the second residual network, and the third residual network are different. Understandably, having different network structures among the multiple downsampling modules allows for the extraction of features from the image at different resolutions, which is beneficial for extracting more details.

[0012] In one embodiment provided in the first aspect, the pooling module includes a pyramid pooling module, which comprises six pooling layers. That is, the pyramid pooling module provided in this application expands from the traditional four channels to six channels, thereby preserving image details as much as possible while capturing a wider range of semantic features, which is beneficial for improving the accuracy of semantic segmentation results.

[0013] In one embodiment provided in the first aspect, each upsampling module includes an initialization module, which comprises two convolutional layers and a concatenation layer. The two convolutional layers are used to perform convolution processing on the received input and send the processing result to the concatenation layer. The concatenation layer is used to concatenate the processing result to obtain the output. This initialization module has a simple network structure, which can reduce the number of parameters in the semantic segmentation model.

[0014] In one embodiment provided in the first aspect, the feature map processed by the i-th downsampling of the encoder-decoder network is a first feature map generated when the basic semantic segmentation model converges, and the feature map output by the j-th upsampling of the encoder-decoder network is a second feature map generated when the basic semantic segmentation model converges, where i+j-1=N, i≤N.

[0015] In one implementation provided by the first aspect, the first model parameters include all parameters used by the encoder-decoder network before the i-th downsampling. That is, the first loss function is only used to adjust the parameters of the network layers necessary to generate the first feature map, thus avoiding the problem of an identity mapping between the first and second feature maps and achieving effective training.

[0016] In one embodiment provided in the first aspect, the first model parameters include at least the parameters of the convolution module.

[0017] In one implementation provided in the first aspect, N = 3 and i = 1.

[0018] In one embodiment provided in the first aspect, the first model parameters are also associated with a second loss function, which is determined based on the predicted semantic segmentation result of the training image and the label of the training image. The predicted semantic segmentation result is obtained by inputting the training image into the basic semantic segmentation model.

[0019] In one embodiment provided in the first aspect, the semantic segmentation model further includes second model parameters, which include all parameters of the semantic segmentation model and are determined according to a second loss function.

[0020] Secondly, embodiments of this application also provide a method for training a semantic segmentation model, the method comprising: iteratively training a basic semantic segmentation model based on training samples; if the basic semantic segmentation model does not converge, adjusting the model parameters of the basic semantic segmentation model based on a first loss function; wherein the first loss function is determined based on a first feature map and a second feature map, the first feature map being a feature map generated by the basic semantic segmentation model during the encoding process, the second feature map being a feature map generated by the basic semantic segmentation model during the decoding process, and the first feature map and the second feature map having the same resolution; if the basic semantic segmentation model converges, stopping the iterative training to obtain the semantic segmentation model.

[0021] In one embodiment provided in the second aspect, the semantic segmentation model includes an encoder-decoder network and a prediction head; inputting a training image into the semantic segmentation model to obtain a predicted semantic segmentation result includes: performing N downsampling and N upsampling operations on the training image sequentially through the encoder-decoder network to obtain a first intermediate feature map; and processing the first intermediate feature map through the prediction head to obtain the predicted semantic segmentation result.

[0022] In one embodiment provided in the second aspect, the encoder-decoder network includes a convolution module, N downsampling modules, a pooling module, and N upsampling modules. The encoder-decoder network sequentially performs N downsampling and N upsampling operations on the input image corresponding to the training image to obtain a first intermediate feature map, including: performing a convolution operation on the training image using the convolution module to obtain a second intermediate feature map; sequentially downsampling the second intermediate feature map using the N downsampling modules to obtain a third intermediate feature map; processing the third intermediate feature map using the pooling module to obtain a fourth intermediate feature map; and sequentially upsampling the fourth intermediate feature map using the N upsampling modules to obtain a first intermediate feature map.

[0023] In one embodiment provided in the second aspect, the N downsampling modules include a first downsampling module, a second downsampling module, and a third downsampling module. The first downsampling module includes a first residual network, the second downsampling module includes a first residual network and a second residual network, and the third downsampling module includes a first residual network, a second residual network, and a third residual network. The network structures of the first residual network, the second residual network, and the third residual network are different. Understandably, having different network structures among the multiple downsampling modules allows for the extraction of features from the image at different resolutions, which is beneficial for extracting more details.

[0024] In one embodiment provided in the second aspect, the pooling module includes a pyramid pooling module, which comprises six pooling layers. That is, the pyramid pooling module provided in this application embodiment expands from the traditional four channels to six channels, thereby preserving image details as much as possible while capturing a wider range of semantic features, which is beneficial for improving the accuracy of the predicted semantic segmentation results.

[0025] In one embodiment provided in the second aspect, each upsampling module includes an initial module, which comprises two convolutional layers and a concatenation layer. The two convolutional layers are used to perform convolution processing on the received input and send the processing result to the concatenation layer. The concatenation layer is used to concatenate the processing result to obtain the output. This initial module has a simple network structure, which can reduce the number of parameters in the semantic segmentation model.

[0026] Thirdly, embodiments of this application also provide an electronic device, which includes: a memory and one or more processors; the memory and processors are coupled; the memory is used to store computer program code, the computer program code including computer instructions, which, when executed by the electronic device, cause the electronic device to perform the methods described in the first aspect, the second aspect, and any one of the embodiments thereof.

[0027] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed in an electronic device, cause the electronic device to perform the methods described in the first aspect, the second aspect, and any one of their embodiments.

[0028] The technical effects of any of the design methods in the second to fourth aspects can be found in the technical effects of different design methods in the first aspect, and will not be repeated here. Attached Figure Description

[0029] Figure 1 A schematic diagram of a scenario provided for an embodiment of this application;

[0030] Figure 2 This application provides a schematic diagram of the structure of an AI system according to an embodiment of the present application.

[0031] Figure 3 Flowchart of the semantic segmentation method provided in the embodiments of this application Figure 1 ;

[0032] Figure 4 Flowchart of the semantic segmentation method provided in the embodiments of this application Figure 2 ;

[0033] Figure 5A schematic diagram illustrating the upsampling and downsampling processes performed on the encoder-decoder network provided in this application embodiment;

[0034] Figure 6 Flowchart of the semantic segmentation method provided in the embodiments of this application Figure 3 ;

[0035] Figure 7 This is a schematic diagram of the residual network structure provided in an embodiment of this application;

[0036] Figure 8 A further structural schematic diagram of the residual network provided in the embodiments of this application;

[0037] Figure 9 A network structure diagram of a pyramid pooling module provided in an embodiment of this application;

[0038] Figure 10 A network structure diagram of an initial module provided in an embodiment of this application;

[0039] Figure 11 A network structure diagram of a semantic segmentation model provided in an embodiment of this application;

[0040] Figure 12 A flowchart illustrating a training method for a semantic segmentation model provided in an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of a training basic semantic segmentation model provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0044] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first feature map" and "second feature map," etc., are used to distinguish different feature maps, not to describe a specific order of feature maps.

[0045] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0046] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0047] To ensure clarity and brevity in the description of the following embodiments, a brief introduction to the relevant technologies is given first.

[0048] 1) Image segmentation

[0049] Image segmentation is a crucial task in computer vision, aiming to classify each pixel in an image—a pixel-level classification task. Currently, there are three main types of image segmentation tasks: semantic segmentation, instance segmentation, and panoptic segmentation.

[0050] Semantic segmentation refers to identifying the category of each pixel in an image and assigning a corresponding category label to the pixel.

[0051] Instance segmentation refers to identifying the existence of each object or category instance in an image and assigning it a different mask or bounding box with a unique identifier.

[0052] Panoramic segmentation is a combination of semantic segmentation and instance segmentation, which can assign a semantic label (due to semantic segmentation) and a unique instance identifier to each pixel in an image.

[0053] Specifically, this application's solution mainly involves semantic segmentation, which will be explained below. Since an image is composed of many pixels, semantic segmentation can be understood as grouping or segmenting pixels according to their different semantic meanings expressed in the image. Through image semantic segmentation, the content in the image can be automatically segmented and identified. In semantic segmentation, the number of categories is first determined, and then an output channel is created for each category. A single channel represents the region where a specific category exists. The result of semantic segmentation can be represented as (C, H, W) or C×H×W.

[0054] For example, such as Figure 1 As shown, after semantic segmentation, the original image is divided into four semantic categories: people, sky, flowers and background, which can output four channels. The result of semantic segmentation can be represented as (4, H, W) or 4×H×W, where each channel classifies the pixel into a single category as 1 or 0.

[0055] 2) Deep Neural Networks

[0056] The image semantic segmentation task described above can be accomplished using deep neural networks. Deep neural networks can include convolutional neural networks and deconvolutional neural networks.

[0057] Convolutional neural networks (CNNs) are related to image feature extraction, transforming input images into multidimensional feature matrices. Deconvolutional neural networks (DNNs) function as region segmentation generators, utilizing the image features extracted by CNNs to perform semantic segmentation of objects.

[0058] The output of the entire deep neural network is a probability matrix graph, which is the same size as the input image. The value of each element at each position in the matrix graph represents the classification probability of the corresponding pixel in the image, that is, the probability that the object to which the pixel belongs is a certain category of object.

[0059] Understandably, a deep neural network can be viewed as a network structure that encodes and decodes. Through a convolutional network (also known as an encoding network), an image is convolved into a small matrix; through a deconvolutional network (also known as a decoding network), the small matrix is ​​restored to a large image.

[0060] 3) Feature map

[0061] Convolutional neural networks (CNNs) include, but are not limited to, one or more convolutional layers. Each convolutional layer can include multiple filters (or convolutional kernels), and each filter is essentially an array, where the numbers are called convolution weights or parameters. The function of the convolutional layers in a CNN is to perform convolution operations on the input. For example, the filters in a convolutional layer can slide across the samples with a set stride. At each sliding position, the filter array is multiplied by the sample data and then summed to obtain a value. All the values ​​obtained during the sliding process are combined into a new array, which is called a feature map. Each value in the feature map is a feature of that feature map.

[0062] The feature map can be represented as a C×H×W three-dimensional matrix, which may include C H×W two-dimensional matrices. Here, H represents the pixel height of the image to be processed, W represents the pixel width of the image to be processed, and C represents the number of channels of the image to be processed. For example, an RGB image has 3 channels.

[0063] In related technologies, using deep neural networks to handle image semantic segmentation tasks is a common approach in the industry. This approach involves training a convergent semantic segmentation model using supervised learning, and then using this model to process the input image to obtain the final semantic segmentation result. However, semantic segmentation models based on conventional supervised learning methods cannot accurately identify the texture details of the image, leading to poor semantic segmentation results and even semantic segmentation errors.

[0064] In view of this, embodiments of this application provide a semantic segmentation method that can obtain the semantic segmentation result of a first image using a semantic segmentation model. The first model parameters of this semantic segmentation model are associated with a first loss function, which is determined based on a first feature map generated during the encoding process and a second feature map generated during the decoding process. Thus, this semantic segmentation model can distill richer features from the decoding stage into the encoding stage, thereby improving the feature extraction capability of the semantic segmentation model during the encoding process. This results in image segmentation results with richer details and reduces the risk of false positives in image segmentation results.

[0065] The semantic segmentation method provided in this application can be applied to shooting scenarios, video call scenarios, medical scenarios, transportation scenarios, etc. For example, taking a shooting scenario as an example, the terminal can perform semantic segmentation on the captured preview image to obtain the semantic segmentation result; then, based on the semantic segmentation result, adjust the local brightness of a certain type of pixel in the preview image, for example, increase the brightness of pixels of category "sky" in the preview image and decrease the brightness of pixels of category "people" in the preview image; finally, display the preview image after brightness adjustment.

[0066] like Figure 2 As shown, Figure 2 A schematic diagram of the structure of the AI ​​system provided in this application. Figure 2 As shown, the AI ​​system includes a data center and multiple terminals (such as...). Figure 2 As shown in terminals 111 and 112, the data center can communicate with the terminals via a network, which can be the Internet or other networks. This network can include one or more network devices, such as routers or switches.

[0067] A data center includes one or more servers, such as Figure 2 The server 120 shown is, for example, an application server that supports application services, which can provide image services, video services, game services, and other AI processing services based on video or images. In an alternative scenario, server 120 refers to a server cluster with multiple servers deployed. This server cluster may have racks, and the racks can establish communication between the multiple servers via wired connections, such as Universal Serial Bus (USB) or Peripheral Component Interconnect Express (PCIe) high-speed buses.

[0068] Server 120 can also acquire data from terminals, process the data using AI, and then send the AI ​​processing results to the corresponding terminals. This AI processing can refer to using AI models to perform tasks such as object recognition, target detection, and semantic segmentation on the data, or it can refer to acquiring AI models that meet the requirements based on samples collected by the terminals.

[0069] in addition, Figure 2 The data center shown may also include other physical devices with AI processing capabilities, such as mobile phones, tablets, or other devices.

[0070] A terminal can also be called a terminal device, user equipment (UE), mobile station (MS), or mobile terminal (MT). A terminal can be a mobile phone (such as...). Figure 2 The terminal 111 shown has mobile payment function and facial recognition payment device, and camera device with data (such as image or video) acquisition and processing function (such as...). Figure 2The terminal shown (112) can also be a tablet computer, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, and so on. The embodiments of this application do not limit the specific technology or device form used in the terminal device.

[0071] It is worth noting that the terminal can access AI models stored on the server 120, and then use these AI models to perform various tasks on the information contained in the image. For example, the AI ​​model can be a semantic segmentation model that performs operations such as object detection and semantic segmentation on the data.

[0072] For example, the AI ​​model deployed in terminal 112 can be used to achieve functions such as object detection and semantic segmentation.

[0073] For example, the AI ​​model deployed in terminal 111 can realize functions such as automatically adjusting the local brightness of the image.

[0074] For example, the AI ​​model deployed in Terminal 111 enables functions such as facial recognition payment and object classification (such as product classification).

[0075] It should be noted that, Figure 2 These are merely illustrative images and should not be construed as limiting the scope of this application. The embodiments of this application do not limit the application scenarios for terminals and servers. For example, a frame of image captured by the terminal may contain multiple different categories of objects; the terminal can identify these objects based on the information in the image. Figure 2 The examples shown are Class 1 (such as people), Class 2 (flowers), Class 3 (such as the sky), etc.

[0076] To at least solve the above problems, the following will be discussed. Figure 2 Based on the AI ​​system shown, this embodiment provides a semantic segmentation method, such as... Figure 3 As shown, Figure 3 Flowchart of the semantic segmentation method provided in the embodiments of this application Figure 1 This semantic segmentation method can be applied to Figure 2 The AI ​​system shown can be executed by a terminal or a server. Here, we will take the execution of the semantic segmentation method provided in this embodiment by a terminal device as an example.

[0077] like Figure 3 As shown, the semantic segmentation method provided in this application includes steps S310 to S320.

[0078] S310, the terminal device acquires the first image.

[0079] The first image can be an image captured in real time by the terminal device, any frame of a video captured in real time by the terminal device, an image sent to the terminal device by another device, any frame of a video sent to the terminal device by another device, or an image pre-stored in the terminal device, etc., without specific restrictions.

[0080] S320, the terminal device inputs the first image into the semantic segmentation model to obtain the semantic segmentation result.

[0081] In this design, the semantic segmentation result is used to indicate the category information of each pixel in the first image. In one possible design, the category information can be represented by the predicted confidence score of a pixel belonging to a first category. In another possible design, the category information can also include the predicted confidence score of a pixel belonging to another category (e.g., a second category). In yet another possible design, the category information can directly indicate the category to which the pixel belongs, for example, as... Figure 1 Any one of the four categories shown.

[0082] In one possible design, the semantic segmentation result can be presented in the form of an image. The terminal device can label the category information of each pixel in a first image and use the labeled first image as the semantic segmentation result. For example, the first image can be... Figure 1 The original image shown can be used to perform semantic segmentation. Figure 1 The image shown is after semantic segmentation.

[0083] In this embodiment of the application, the terminal device can label the category information of each pixel into the image. The labeled image (i.e., the semantic segmentation result) can display the distribution of pixels of different categories in the image, thereby enabling users to quickly view the distribution of pixels of different categories from the semantic segmentation result displayed by the terminal device, thus improving the quality of experience (QoE).

[0084] In this embodiment, the semantic segmentation model can perform multiple downsampling and multiple upsampling processes. The process of multiple downsampling processes can also be called the encoding process, and the process of multiple upsampling processes can also be called the decoding process.

[0085] Downsampling is used to extract detailed features from an image to obtain feature maps. By sequentially performing multiple downsampling processes on the input image, multiple feature maps with different resolutions can be obtained to represent the details and semantic information of the first image at different resolutions. In this embodiment, downsampling can be implemented through multi-layer convolution and pooling operations.

[0086] Upsampling is used to improve image resolution. By performing multiple upsampling processes on the feature map output from downsampling, an image of the same size as the input image can be gradually obtained. In this embodiment, downsampling can be achieved through nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0087] The semantic segmentation model includes first model parameters, which are associated with a first loss function. The first loss function is determined based on a first feature map and a second feature map. The first feature map is generated during the encoding process of the basic semantic segmentation model based on the generative semantic segmentation model, and the second feature map is generated during the decoding process of the basic semantic segmentation model. The first and second feature maps have the same resolution. This semantic segmentation model can be understood as a convergent basic semantic segmentation model.

[0088] As can be seen, in the embodiments of this application, the first model parameters of the semantic segmentation model are associated with the first loss function. The first loss function is determined based on two feature maps (e.g., the first feature map and the second feature map) with the same resolution generated by the basic semantic segmentation model in the encoding and decoding processes. This allows richer features from the decoding process to be distilled into the encoding process, enabling the semantic segmentation model to have stronger feature extraction capabilities in the encoding process, extracting more details, thereby making the image segmentation results more accurate and reducing the probability of false positives in the image segmentation results.

[0089] In this embodiment, the first model parameters are also associated with a second loss function. In other words, the first model parameters are determined based on the first loss function and the second loss function. The second loss function is determined based on the predicted semantic classification result of the training image and the label of the training image. The predicted semantic classification result is the semantic segmentation result obtained by inputting the training image into the basic semantic segmentation model, and the label of the training image is used to indicate the actual semantic segmentation status of the training image.

[0090] In this embodiment of the application, the semantic segmentation model further includes a second model parameter, which is different from the first model parameter and is determined according to a second loss function.

[0091] In one possible design, the first loss function is the mean-square error (MSE) between the first and second feature maps, and the second loss function is the mean-square error between the test result and the standard result. In other possible designs, the first and second loss functions can also be other than those specified here.

[0092] Regarding the aforementioned process of obtaining semantic segmentation results using a semantic segmentation model in terminal devices, embodiments of this application provide a possible implementation method, such as... Figure 4As shown, Figure 4 Flowchart of the semantic segmentation method provided in the embodiments of this application Figure 2 The semantic segmentation model includes an encoder-decoder network and a prediction head.

[0093] like Figure 4 As shown, the semantic segmentation model can also preprocess the first image to obtain the input image corresponding to the first image.

[0094] In the embodiments of this application, preprocessing includes operations such as changing resolution and normalization.

[0095] The purpose of changing the resolution is to standardize the resolution of the input image, making it easier for the semantic segmentation model to process. For example, the semantic segmentation model can change the resolution of the first image to a preset resolution, such as 384×384.

[0096] The standardization operation adjusts the pixel values ​​in the first image to conform to a standard normal distribution or to have a specific mean and standard deviation. In one possible design, the mean and variance used by the semantic segmentation model for the standardization operation can be calculated based on large-scale image data from the ImageNet dataset.

[0097] In this embodiment, the semantic segmentation module can first adjust the resolution of the first image to a preset resolution, and then perform a standardization operation on the first image after adjusting the resolution to obtain the input image corresponding to the first image.

[0098] It is worth noting that the aforementioned preprocessing may also include other graphic processing operations, such as flipping, translating, and cropping, which are not limited here.

[0099] In one possible design, the semantic segmentation model may not preprocess the first image. After the terminal device obtains the input image corresponding to the first image, it can directly input the input image corresponding to the first image into the semantic segmentation model.

[0100] In the semantic segmentation method provided in this embodiment, the aforementioned S320 may include S410 to S420.

[0101] S410, the terminal device performs N downsampling and N upsampling operations on the input image corresponding to the first image through an encoder-decoder network to obtain the first intermediate feature map.

[0102] Among them, such as Figure 5 As shown, the process by which the encoder-decoder network performs N downsampling and N upsampling operations on the input image can include: performing a first downsampling operation on the input image to obtain an intermediate feature map m. d1The result obtained from the first downsampling (i.e., the intermediate feature map m) d1 A second downsampling process is performed to obtain the intermediate feature map m. d2 ...and so on, for the result obtained from the (N-1)th downsampling (i.e., the intermediate feature map m) d(N-1) Perform the Nth downsampling process to obtain the intermediate feature map m. dN .

[0103] Next, as before Figure 5 As shown, the result of the Nth downsampling (i.e., the intermediate feature map m) dN The first upsampling process is performed, and the result of the first upsampling is fused with the result of the (N-1)th downsampling (i.e., the intermediate feature map m). d(N-1) ), thus obtaining the intermediate feature map m uN ; for intermediate feature map m uN A second upsampling is performed, and the results of the second upsampling and the results of the (N-2)th downsampling (i.e., the intermediate feature map m) are fused together. d(N-2) ), thus obtaining the intermediate feature map m u(N -1); ...and so on, for the intermediate feature map m u3 Perform the (N-1)th upsampling and fuse the result of the (N-1)th upsampling with the structure of the first downsampling (i.e., the intermediate feature map m). d1 ), thus obtaining the intermediate feature map m u2 ; for intermediate feature map m u2 Perform the Nth upsampling and fuse the result of the Nth upsampling with the input image to obtain the intermediate feature map m. u1 (Also known as the first intermediate feature map).

[0104] Among them, the intermediate feature map m d1 To the intermediate feature map m dN The resolution decreases sequentially while the number of channels increases sequentially, indicating that the semantic segmentation model extracts more and more detailed features. Additionally, the intermediate feature map m... uN To the intermediate feature map m u1 The resolution increases sequentially, getting closer and closer to the resolution of the input image.

[0105] The basic semantic segmentation model and the semantic segmentation model have the same network structure; the difference is that the basic semantic segmentation model does not converge, while the semantic segmentation model does. Therefore, in Figure 4In the provided flowchart, the first feature map can specifically be the feature map processed by the i-th downsampling of the basic semantic segmentation model, and the second feature map can specifically be the feature map obtained by the j-th upsampling of the basic semantic segmentation model, where i+j-1=N, i≤N. Thus, the first feature map and the second feature map can be the feature maps generated by the basic semantic segmentation model during the encoding and decoding processes, respectively, and the resolution of the first feature map and the second feature map can be kept consistent.

[0106] Correspondingly, the feature map processed by the i-th downsampling of the semantic segmentation model is the first feature map generated when the basic semantic segmentation model converges, and the feature map output by the j-th upsampling of the semantic segmentation model is the second feature map generated when the basic semantic segmentation model converges.

[0107] In this case, the first model parameters may include the parameters used by the semantic segmentation model before the i-th downsampling, and the second model parameters may include all the parameters of the semantic segmentation model.

[0108] S420: The terminal device processes the first intermediate feature map through the prediction head to obtain the semantic segmentation result.

[0109] In this embodiment, the prediction head can convert the first intermediate feature map into a feature map of size C×H×W, where H×W is the resolution of the image, the same as the resolution of the input image, and C is the number of channels, the same as the number of types that the semantic segmentation model can segment. Then, the Softmax activation function is used to calculate a corresponding probability value for each category, and the Argmax function is used to obtain the type information corresponding to each pixel, thus completing the semantic segmentation task.

[0110] For example, if the size of the input image is 3×384×384, it means that the input image is an image with a resolution of 384×384 and 3 color channels, and the semantic segmentation model can identify 4 different categories; then the result of the prediction head can be 4×384×384.

[0111] Optionally, in Figure 4 Based on the S410 shown below, the following is combined with Figure 6 The process by which the terminal device obtains the first intermediate feature map is explained. Figure 6 Flowchart of the semantic segmentation method provided in the embodiments of this application Figure 3 The encoder-decoder network includes a convolutional module, N downsampling modules, a pooling module, and N upsampling modules.

[0112] Figure 6 It shows Figure 4 Possible implementations of S410 include S411 to S414.

[0113] S411, the terminal device performs a convolution operation on the input image corresponding to the first image through the convolution module to obtain the second intermediate feature map.

[0114] The function of the convolution module is to perform convolution operations on the input. For example, the filters in the convolutional layer can slide on the samples with a set stride. At each sliding position, the filter array is multiplied by the sample data and then added to obtain a value. All the values ​​obtained during the sliding process are combined into a new array, which is called a feature map.

[0115] Convolution operations can change the resolution and number of channels of an image. In this embodiment, the convolution module can use the conv2d function to perform a two-dimensional convolution operation on the input image to obtain a second intermediate feature. The number of channels of the second intermediate feature is greater than the number of channels of the input image, and the resolution of the second intermediate feature is less than the resolution of the input image.

[0116] S412, the terminal device performs downsampling processing on the second intermediate feature map in sequence through N downsampling modules to obtain the third intermediate feature map.

[0117] In this embodiment of the application, taking N downsampling modules including downsampling module 1, downsampling module 2, ..., downsampling module N as an example, the terminal device can use downsampling module 1 to perform the first downsampling process on the second intermediate feature map to obtain the intermediate feature map m. d1 ; Use downsampling module 2 to process intermediate feature map m d1 A second downsampling process is performed to obtain the intermediate feature map m. d2 ...and so on, using downsampling module N to sample intermediate feature maps m d(N-1) Perform the Nth downsampling process to obtain the intermediate feature map m. dN intermediate feature map m dN This is the third intermediate feature map.

[0118] The N downsampling modules have different network structures to extract features of the image at different resolutions.

[0119] In this embodiment, the N downsampling modules can be composed of various residual networks, and the network structures of these residual networks are different. In this embodiment, the various residual networks can be obtained by arranging and combining a first network layer, a second network layer, a third network layer, and a fourth network layer.

[0120] Taking multiple residual networks, including the first residual network, the second residual network, and the third residual network, as an example, Figure 7 The diagram shows the network structures of the first residual network, the second residual network, and the third residual network. Figure 7As shown, the first residual network includes two first network layers, two second network layers, and a third network layer; the second residual network includes two first network layers, a second network layer, and a third network layer; and the third residual network includes a first network layer, a second network layer, a third network layer, and a fourth network layer.

[0121] The first and fourth network layers can perform convolution, normalization, and linear rectification operations on the input. The parameters used for the convolution operations in the first and fourth network layers are different. The second network layer can be used to perform convolution and normalization operations on the input, and the third network layer can be used to perform linear rectification operations on the input.

[0122] In the first residual network, the input x is processed by two first network layers and a second network layer to obtain the intermediate quantity x1. The input x is processed by the second network layer to obtain the intermediate quantity x2. Then the intermediate quantities x1 and x2 are fused and the fused result is input into the third network layer to obtain the output y.

[0123] In the second residual network, the input x is processed through two first network layers and a second network layer to obtain the intermediate quantity x1. Then, the intermediate quantity x1 and the input x are fused, and the fused result is input into the third network layer to obtain the output y.

[0124] In the third residual network, the input x is processed through the first network layer, the fourth network layer, and the second network layer in sequence to obtain the intermediate quantity x3. Then, the intermediate quantity x3 and the input x are fused, and the fused result is input into the third network layer for processing to obtain the output y.

[0125] Figure 8 The diagram illustrates further network structures for the first, second, and third residual networks. Figure 8 As shown, the first network layer includes a first convolutional (conv) layer, a normalization layer, and a rectifying linear units (ReLU) layer; the second network layer includes a first convolutional layer and a normalization layer; the third network layer includes a rectifying linear unit (ReLU) layer; and the fourth network layer includes a second convolutional layer, a normalization layer, and a rectifying linear unit (ReLU) layer.

[0126] In one possible design, the first convolutional layer can be a Conv2d layer, the second convolutional layer can be a Conv2dwith Dilation layer, the normalization layer can be a BN (batch normalization) layer, and the linear rectification layer can be a LeakyReLU layer.

[0127] S413, the terminal device processes the third intermediate feature map through the pooling module to obtain the fourth intermediate feature map.

[0128] The pooling module can include a pyramid pooling module, which can use pooling of different sizes to perform multi-scale feature fusion.

[0129] For example, Figure 9 A network structure diagram of a pyramid pooling module is shown. Figure 9 As shown, the pyramid pooling module includes 6 channels and a splicing layer. Each channel includes a pooling layer, a first network layer, and an upsampling layer. The pooling layer can perform pooling operations, the first network layer can perform convolution operations, and the upsampling layer can perform upsampling operations.

[0130] Figure 9 The processing flow of the pyramid pooling module shown is as follows: First, the input (e.g., the third intermediate feature map) is pooled at various sizes to obtain feature maps of multiple sizes, such as 1×1, 2×2, 3×3, 4×4, 6×6, and 12×12. Then, convolution operations are performed on the feature maps of multiple sizes to reduce the number of channels. Next, the convolved feature maps are upsampled to obtain multiple feature maps of the same size. Finally, multiple feature maps of the same size are concatenated on the channels to obtain the output (e.g., the fourth intermediate feature map).

[0131] In one possible design, the pooling layer mentioned above can be an adaptive average pooling 2D layer, the first network layer can include a Conv2d layer, a BN layer and a LeakyReLU layer, and the concatenation layer can include a concatenation layer.

[0132] As can be seen, the pyramid pooling module provided in this application embodiment has been expanded from the traditional 4 channels to 6 channels, which can preserve image details as much as possible while capturing a larger range of semantic features, thus improving the accuracy of semantic segmentation results.

[0133] S414, the terminal device sequentially upsamples the fourth intermediate feature map through N upsampling modules to obtain the first intermediate feature map.

[0134] In this embodiment of the application, taking N upsampling modules including upsampling module 1, upsampling module 2, ... upsampling module N as an example, the terminal device can use upsampling module 1 to perform the first upsampling process on the fourth intermediate feature map, and then fuse the result of the first upsampling process with the intermediate feature map m. d(N-1) Obtain intermediate feature map m uN ; Use upsampling module 2 to process intermediate feature map m uN A second upsampling process is performed, and then the result of the second upsampling process is fused with the intermediate feature map m. d(N-2)Obtain intermediate feature map m uN-1 ...and so on, using the upsampling module N to sample the intermediate feature map m u2 Perform the Nth upsampling process, and then fuse the fourth intermediate feature map to obtain the intermediate feature map m. u1 intermediate feature map m u1 This is the first intermediate feature map.

[0135] In one possible design, the N upsampling modules may include an inception module. Figure 10 The network structure diagram of the Inception module is shown. Figure 10 As shown, the Inception module includes two second network layers and a concatenation layer. The input x is processed by the two second network layers and then concatenated by the concatenation layer to obtain the output y. The convolutional kernels of the two second network layers are 1×9 and 9×1, respectively. The second network layers may include Conv layers and BN layers, and the concatenation layer may include a concatenation layer.

[0136] The Inception module provided in this application has a simple network structure, which can reduce the number of parameters in the semantic segmentation model.

[0137] It is worth noting that the semantic segmentation model provided in this application embodiment may include not only the modules described above, but also other network layers or algorithm modules, and this application does not limit it in this regard.

[0138] The following uses N=3 as an example to illustrate in detail the network structure of a semantic segmentation model provided in the embodiments of this application. Figure 11 The diagram illustrates a network structure of a semantic segmentation model provided in an embodiment of this application. Figure 11 As shown, the semantic segmentation model includes a convolutional module, a residual network module, downsampling module 1, downsampling module 2, downsampling module 3, a pooling module, upsampling module 1, upsampling module 2, upsampling module 3, and a prediction head. The convolutional module includes three network layers consisting of Conv2d and BN2d layers. The residual network module includes the first and second residual networks mentioned above. Downsampling module 1 includes a MaxPool2d layer, a first residual network, and a second residual network. Downsampling modules 2 and 3 each include a MaxPool2d layer, a first residual network, a second residual network, and a third residual network. The pooling module includes a network layer consisting of a Conv2d layer, a BN2d layer, and a LeakyReLU layer, as well as a pyramid pooling module. Downsampling modules 1, 2, and 3 all include an inception module. The prediction head includes a network layer consisting of Conv2d and BN2d layers, a Conv2d layer, and an upsampling layer.

[0139] The process involves inputting a 3×384×384 input image into a convolution module to obtain a 64×96×96 feature map FM1. FM1 is then input into a residual network module to obtain a 128×96×96 feature map FM2. FM2 is input into downsampling module 1 to obtain a 256×48×48 feature map FM3. FM3 is input into downsampling module 2 to obtain a 512×24×24 feature map FM4. FM4 is input into downsampling module 3 to obtain a 1024×12×12 feature map FM5. FM5 is input into a pooling module to obtain a 128×12×12 feature map FM6. FM6 is then input into upsampling module 1, and finally fused with a feature map FM4 processed by Conv2d and BN2d to obtain a 1024×12×12 feature map. The 28×24×24 feature map FM7 is combined with the result of upsampling module 1 to obtain a feature map FM8 with a size of 128×24×24. Feature map FM8 is input into upsampling module 2, and then the feature map FM9 with a size of 128×48×48 obtained by Conv2d and BN2d processing of feature map FM3 is combined with the result of upsampling module 2 to obtain a feature map FM10 with a size of 128×48×48. Feature map FM10 is input into upsampling module 3, and then the feature map FM11 with a size of 128×96×96 obtained by Conv2d and BN2d processing of feature map FM2 is combined with the result of upsampling module 3 to obtain a feature map FM12 with a size of 128×96×96. Finally, feature map FM12 is input into the prediction head to obtain a semantic segmentation result with a size of 4×384×384.

[0140] As can be seen, the residual modules provided in this application can extract image features at different resolutions using three different network structures during the encoding process. This allows for more reasonable extraction of image details and semantic information at different stages of feature encoding, capturing a wider range of semantic information. Subsequently, during the decoding process, pyramid pooling and Inception modules are used to fuse image features at different scales, and the feature map is gradually restored to the original input resolution using an upsampling method based on bilinear interpolation. Notably, this invention extends the classic pyramid pooling module from 4 layers to 6 layers, thereby preserving as much image detail as possible while capturing a wider range of semantic features. At the same time, the concise design of the Inception module effectively reduces the number of model parameters. Finally, the model uses the Softmax activation function to calculate a corresponding probability value for each category, and uses the Argmax function to obtain the predicted category for each pixel, completing the semantic segmentation task.

[0141] This application also provides a training method for a semantic segmentation model, which distills richer image features from the decoding stage to the encoding stage, thereby improving the feature extraction capability of the encoder and thus enhancing the performance of the semantic segmentation model.

[0142] like Figure 12 The diagram shown illustrates the flowchart of the training method for the semantic segmentation model provided in this embodiment. This training method can be executed by the aforementioned server, the aforementioned terminal device, or other devices with computing capabilities. Here, we will illustrate the method using a terminal device to execute the training method for the semantic segmentation model provided in this embodiment. Figure 12 As shown, the training method for the semantic segmentation model provided in this application includes steps S710 to S740.

[0143] S710, the terminal device iteratively trains the basic semantic segmentation model based on training samples.

[0144] The training samples include labeled training images, where the labels indicate the true semantic segmentation result of the corresponding training image. During iterative training, the terminal device can input the labeled training images into the base semantic segmentation model to obtain the predicted semantic segmentation result of the training image.

[0145] It should be noted that the network structure of the basic semantic segmentation model is similar to that of the semantic segmentation model described above, and will not be repeated here.

[0146] S720: The terminal device determines whether the basic semantic segmentation model has converged.

[0147] If the basic semantic segmentation model does not converge, then execute S730; if the basic semantic segmentation model converges, then execute S740.

[0148] For example, the terminal device can determine whether the basic semantic segmentation model has converged based on the first loss function and the second loss function.

[0149] The first loss function can be determined based on the first feature map and the second feature map. The first feature map is a feature map generated by the basic semantic segmentation model during the encoding process, and the second feature map is a feature map generated by the basic semantic segmentation model during the decoding process. The first feature map and the second feature map have the same resolution.

[0150] The terminal device can perform convolution processing on the first feature map to map the first feature map to the dimension of the second feature map, and then calculate the mean square error between the second feature map and the first feature map after convolution processing. This mean square error is the first loss function mentioned above.

[0151] Understandably, the first loss function is used to reflect the difference between the first feature map and the second feature map. The smaller the first loss function, the closer the first feature map is to the second feature map.

[0152] The second loss function can be determined based on the labels of the training images and the predicted semantic segmentation results of the training images. The smaller the second loss function, the closer the predicted semantic segmentation results are to the labels; in other words, the better the performance of the basic semantic segmentation model.

[0153] S730: The terminal device updates the model parameters of the basic semantic segmentation model based on the first loss function and the second loss function.

[0154] The basic semantic segmentation model includes first model parameters and second model parameters. The first model parameters include the model parameters used in the process of generating the first feature map, and the second model parameters include all model parameters of the basic semantic segmentation model; that is, the second model parameters include the first model parameters. The terminal device can adjust the first model parameters based on the first loss function and the second loss function, and adjust the first model parameters based on the second loss function. This avoids the problem of an identity mapping between the first and second feature maps of the basic semantic segmentation model.

[0155] For example, such as Figure 13 As shown, the network structure of the basic semantic segmentation model is as follows: Figure 11 Taking the network structure shown as an example, the first feature map can be feature map FM2 output by the residual network module, and the second feature map can be feature map FM12. The terminal device can perform a Conv2d operation on feature map FM2, and then calculate the mean squared error between feature map FM12 and the Conv2d-operated feature map FM2. This mean squared error is used as the first loss function. This first loss function can be used to adjust the parameters of the convolutional module and the residual network module; that is, the first model parameters include the parameters of both the convolutional module and the residual network module.

[0156] The second loss function is determined based on the labels of the input image and the predicted semantic segmentation results output by the base semantic segmentation model. This second loss function can be used to update all model parameters of the base semantic segmentation model.

[0157] S740, the terminal device stops iterative training and obtains the semantic segmentation model.

[0158] As can be seen, in the process of training the basic semantic segmentation model, a first loss function and a second loss function can be obtained in one training cycle. Then, the first model parameters are adjusted according to the first loss function and the second loss function, and the second model parameters are adjusted according to the second loss function until the first loss function and the second loss function converge. The determined first model parameters and second model parameters are then used as the model parameters of the initial semantic segmentation model to obtain the semantic segmentation model.

[0159] Understandably, the training method for the semantic segmentation model provided in this application satisfies three conditions:

[0160] The first approach is to iteratively train the basic semantic segmentation model using labeled training images, which means optimizing the basic semantic segmentation model using supervised learning.

[0161] Second, the local network of the basic semantic segmentation model (i.e., the network between the first feature map and the second feature map) satisfies the local Markov property.

[0162] A stochastic process possesses the Markov property when, given its present state and all past states, the conditional probability distribution of its future states depends only on the current state; in other words, given the present state, it is conditionally independent of its past states (i.e., the historical path of the process). The local Markov property, on the other hand, states that given the neighboring variables of a variable, that variable is conditionally independent of the other variables. Since the first feature map is the unique input feature of the local network of the basic semantic segmentation model, and the second feature map is the unique output feature of the local network of the basic semantic segmentation model, the local network of the basic semantic segmentation model satisfies the local Markov property.

[0163] Third, the feature maps generated during the decoding process not only acquire detailed features extracted during the encoding process through the bridging network, but also capture rich semantic information from deeper networks. Therefore, the feature maps generated during the decoding process have richer features than those generated during the encoding process, and are better able to meet the needs of label prediction during supervised learning.

[0164] In other words, the training method of the semantic segmentation model provided in this application satisfies the condition that "under the supervised training paradigm, the information in the output features of the local Markov process is richer than that in the input features".

[0165] Therefore, in this embodiment, the first feature map generated during the encoding process is used as the student feature, and the second feature map generated during the decoding process is used as the teacher feature. This allows for the distillation of features from the decoding process back into the encoding process, achieving a spiral enhancement of feature representation during encoding and decoding, thereby improving the performance of the semantic segmentation model. Furthermore, experimental results show that, within a certain range, the more complex the semantic segmentation model, the more significant the performance improvement brought about by this training method.

[0166] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0167] This application also provides a computer storage medium that includes computer instructions. When the computer instructions are executed on the electronic device, the electronic device causes the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiment.

[0168] This application also provides a computer program product that, when run on a computer, causes the computer to perform various functions or steps performed by the mobile phone in the above method embodiments.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A semantic segmentation method, characterized in that, The method includes: Get the first image; The first image is input into the semantic segmentation model to obtain the semantic segmentation result; The semantic segmentation result is used to indicate the category information of each pixel in the first image. The semantic segmentation model includes first model parameters, which are associated with a first loss function and a second loss function. The first loss function is the mean squared error of the second feature map and the first feature map after convolution. The first feature map is a feature map generated by the basic semantic segmentation model used to generate the semantic segmentation model during the encoding process. The second feature map is a feature map generated by the basic semantic segmentation model during the decoding process. The first feature map and the second feature map have the same resolution. The second loss function is determined based on the predicted semantic segmentation result of the training image and the label of the training image. The predicted semantic segmentation result is obtained by inputting the training image into the basic semantic segmentation model.

2. The method according to claim 1, characterized in that, The semantic segmentation model is obtained by training the basic semantic segmentation model based on training samples, wherein the training process includes: The basic semantic segmentation model is iteratively trained based on the training samples. If the basic semantic segmentation model does not converge, the model parameters of the basic semantic segmentation model are adjusted based on the first loss function; If the basic semantic segmentation model converges, stop iterative training to obtain the semantic segmentation model.

3. The method according to claim 2, characterized in that, The basic semantic segmentation model is used to perform N downsampling and N upsampling on the training samples. The first feature map is the feature map processed by the i-th downsampling of the basic semantic segmentation model, and the second feature map is the feature map obtained by the j-th upsampling of the basic semantic segmentation model, i+j-1=N, i≤N.

4. The method according to any one of claims 1-3, characterized in that, The semantic segmentation model includes an encoder-decoder network and a prediction head; The step of inputting the first image into the semantic segmentation model to obtain the semantic segmentation result includes: The first image is downsampled N times and upsampled N times sequentially by the encoder-decoder network to obtain the first intermediate feature map; The semantic segmentation result is obtained by processing the first intermediate feature map through the prediction head.

5. The method according to claim 4, characterized in that, The encoder-decoder network includes a convolutional module, N downsampling modules, a pooling module, and N upsampling modules; The step of performing N downsampling and N upsampling operations on the input image corresponding to the first image through the encoder-decoder network to obtain the first intermediate feature map includes: The first image is convolved by the convolution module to obtain a second intermediate feature map. The second intermediate feature map is downsampled sequentially by the N downsampling modules to obtain the third intermediate feature map; The third intermediate feature map is processed by the pooling module to obtain the fourth intermediate feature map; The first intermediate feature map is obtained by sequentially upsampling the fourth intermediate feature map through the N upsampling modules.

6. The method according to claim 5, characterized in that, The N downsampling modules include a first downsampling module, a second downsampling module, and a third downsampling module. The first downsampling module includes a first residual network, the second downsampling module includes a first residual network and a second residual network, and the third downsampling module includes a first residual network, a second residual network, and a third residual network. The network structures of the first residual network, the second residual network, and the third residual network are different.

7. The method according to claim 5, characterized in that, The pooling module includes a pyramid pooling module, which comprises six pooling layers.

8. The method according to claim 5, characterized in that, Each of the upsampling modules includes an initialization module, which includes two convolutional layers and a concatenation layer. The two convolutional layers are used to perform convolution processing on the received input and send the processing result to the concatenation layer. The concatenation layer is used to concatenate the processing result to obtain the output.

9. The method according to any one of claims 5-8, characterized in that, The feature map processed by the i-th downsampling of the encoder-decoder network is the first feature map generated when the basic semantic segmentation model converges, and the feature map output by the j-th upsampling of the encoder-decoder network is the second feature map generated when the basic semantic segmentation model converges, where i+j-1=N, i≤N.

10. The method according to claim 9, characterized in that, The first model parameters include all parameters used by the encoder-decoder network before the i-th downsampling.

11. The method according to claim 10, characterized in that, The first model parameters include at least the parameters of the convolution module.

12. The method according to any one of claims 10-11, characterized in that, N=3, i=1.

13. The method according to any one of claims 1-3, 5-8 and 10-11, characterized in that, The semantic segmentation model further includes a second model parameter, which includes all parameters of the semantic segmentation model and is determined according to the second loss function.

14. A method for training a semantic segmentation model, characterized in that, The method includes: The basic semantic segmentation model is iteratively trained based on training samples; If the basic semantic segmentation model does not converge, the model parameters of the basic semantic segmentation model are adjusted based on a first loss function and a second loss function. The first loss function is the mean squared error between the second feature map and the first feature map after convolution. The first feature map is a feature map generated by the basic semantic segmentation model during the encoding process, and the second feature map is a feature map generated by the basic semantic segmentation model during the decoding process. The first and second feature maps have the same resolution. The second loss function is determined based on the predicted semantic segmentation result of the training image and the label of the training image. The predicted semantic segmentation result is obtained by inputting the training image into the basic semantic segmentation model. If the basic semantic segmentation model converges, stop iterative training to obtain the semantic segmentation model.

15. An electronic device, characterized in that, The electronic device includes: a memory and one or more processors; the memory and the processors are coupled; the memory is used to store computer program code, the computer program code including computer instructions, which, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1-14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed in an electronic device, cause the electronic device to perform the method as described in any one of claims 1-14.

Images