Semantic segmentation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a semantic segmentation method. The method comprises the following steps: acquiring an original image of a view angle of an unmanned aerial vehicle; inputting the original image into a pre-trained semantic segmentation model, and performing forward propagation to generate a semantic segmentation image; the semantic segmentation model comprises a pre-trained image coding head, a time step head, a diffusion segmentation coding head, an encoder, a decoder and an output layer; wherein the semantic segmentation model is obtained by training according to a plurality of training images; the training image is an unmanned aerial vehicle view angle image; the semantic segmentation image comprises category information of each pixel in the training image; the category information includes at least one of buildings, roads, trees, low vegetation, moving cars, stationary cars, humans and sundries.

Description

Technical Field

[0001] The present application belongs to the field of Internet technology, and in particular relates to a semantic segmentation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of drone technology, drones are increasingly being used in a variety of fields, including environmental monitoring, agricultural management, disaster response, and logistics delivery. To achieve autonomous navigation and operation in these tasks, drones require a deep understanding and perception of their surroundings. Traditional navigation methods rely on GPS and pre-set paths, but these methods are often inflexible and inaccurate in complex or dynamic environments. Therefore, drones require technology that can analyze and understand the environment in real time to ensure they can perform their missions safely and efficiently.

[0003] Semantic segmentation is a computer vision technique that labels each pixel in an image as a specific category, such as "person," "tree," or "road." This technology is crucial for drones because it provides detailed environmental information, helping them identify and distinguish between different objects and scenes. Semantic segmentation allows drones to better understand their surroundings, enabling them to make more intelligent decisions, such as obstacle avoidance, path planning, and object recognition.

[0004] Image segmentation from drone perspectives is a challenging task due to diverse viewpoints, extreme scale variations, and highly complex scenes. While traditional encoder-decoder architectures perform well in some scenarios, their limited receptive field and single-scale feature extraction methods make it difficult to address these challenges. Summary of the Invention

[0005] The embodiments of the present application provide a semantic segmentation method, apparatus, device, and storage medium that can output high-precision semantic segmentation images.

[0006] In a first aspect, an embodiment of the present application provides a semantic segmentation method, the method comprising:

[0007] Get the original image from the drone's perspective;

[0008] Input the original image into the pre-trained semantic segmentation model, perform forward propagation, and output the semantic segmentation image; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer;

[0009] The semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris.

[0010] In a second aspect, an embodiment of the present application provides a semantic segmentation device, the semantic segmentation device comprising:

[0011] The acquisition module is used to obtain the original image from the drone's perspective;

[0012] The output module is used to input the original image into the pre-trained semantic segmentation model, perform forward propagation, and output the semantic segmentation image; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer;

[0013] The semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris.

[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the method in the first aspect or any possible implementation of the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method in the first aspect or any possible implementation of the first aspect is implemented.

[0016] In an embodiment of the present application, an original image from the perspective of a drone is obtained; the original image is input into a pre-trained semantic segmentation model, forward propagation is performed, and a semantic segmentation image is output; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer; in the semantic segmentation model, the image encoding head is responsible for extracting high-level features of the original image, the time step head is responsible for processing time step information and previous segmentation results, the feature fusion module combines the time step feature vector and the first feature image, the diffusion segmentation encoding head performs denoising, the encoder gradually extracts features and reduces the resolution, the decoder gradually restores the resolution and generates a high-resolution feature image, and the output layer generates a final semantic segmentation image. Thus, the semantic segmentation model can generate high-quality semantic segmentation images.

[0017] Since the semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris; in the process of the semantic segmentation model, the gap between the predicted semantic segmentation image output by the semantic segmentation model and the preset semantic segmentation image can be continuously narrowed, that is, the semantic segmentation accuracy of the semantic segmentation model can be continuously improved. Therefore, the trained semantic segmentation model can quickly and accurately classify the pixel categories in the original image and output a high-precision semantic segmentation image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 is a schematic diagram of a diffusion model provided in an embodiment of the present application;

[0020] Figure 2 This is a flowchart of a semantic segmentation method provided in an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a noise recursive diffusion process provided by an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of an attention mechanism provided in an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of an attention mechanism provided in an embodiment of the present application;

[0024] Figure 6 This is a schematic diagram of a hierarchical multi-scale denoising process provided in an embodiment of the present application;

[0025] Figure 7 This is a schematic diagram of a model architecture provided in an embodiment of the present application;

[0026] Figure 8 This is a schematic diagram of a segmentation result provided in an embodiment of the present application;

[0027] Figure 9 Schematic diagram of a semantic segmentation device provided in an embodiment of the present application;

[0028] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0030] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0031] The following is a brief introduction to the technical terms involved in this application.

[0032] Fully autonomous drones: also known as unmanned aerial vehicles (UAVs), have many socially and economically important applications, such as infrastructure inspection, agricultural monitoring, search and rescue, disaster management, surveying, and delivery services.

[0033] Semantic segmentation: Samples in the dataset used for comparison with training samples.

[0034] Diffusion denoising diffusion probability model (DDPM): referred to as diffusion model, is a generative model.

[0035] Generative Adversarial Networks (GANs): A generative neural network based on game theory consisting of a generator and a discriminator.

[0036] Multi-class segmentation: Multi-class segmentation refers to the process of labeling each pixel in an image into more than two different categories.

[0037] Noise Schedule: A mechanism within the diffusion model that schedules the noise reduction amplitude over multiple iterations, with each noise reduction amplitude varying. This amplitude typically decreases.

[0038] Figure 1 This paper demonstrates the complete pipeline from image input to semantic segmentation. First, the input image is denoised using a diffusion model to generate a denoised feature map. The denoised feature map then passes through an encoder to gradually extract features and reduce its resolution. The decoder then gradually restores the resolution and generates a high-resolution feature map. Finally, the high-resolution feature map is fed into the output layer to generate the final semantic segmentation map.

[0039] Figure 2 This is a flowchart of a semantic segmentation method provided in an embodiment of the present application.

[0040] like Figure 2 As shown, the semantic segmentation method may include steps 210 to 240, and the method is applied to a semantic segmentation device, as shown below:

[0041] Step 210: obtaining an original image from the drone's perspective;

[0042] Step 220: Input the original image into a pre-trained semantic segmentation model, perform forward propagation, and output a semantic segmentation image; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer;

[0043] The semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris.

[0044] Raw images from a drone’s perspective: Images taken from a drone are usually used to obtain large-scale scene information.

[0045] Semantic segmentation model: A deep learning model that classifies each pixel in an image into a predefined category.

[0046] Image encoding head: The part of the semantic segmentation model responsible for extracting high-level features of the image.

[0047] Time step head: A part of the semantic segmentation model that processes time step information and helps the model understand the temporal sequence characteristics of the image.

[0048] Diffusion Segmentation Encoder Head: The part of the semantic segmentation model that is responsible for processing noise in the image and generating denoised feature maps.

[0049] Encoder: The part of the semantic segmentation model that progressively extracts features from the image and reduces the resolution through convolutional and pooling layers.

[0050] Decoder: Part of the semantic segmentation model that gradually restores the resolution of the image and generates high-resolution feature maps through upsampling and convolutional layers.

[0051] Output layer: The last part of the semantic segmentation model, responsible for generating the final semantic segmentation map.

[0052] In an embodiment of the present application, an original image from the perspective of a drone is obtained; the original image is input into a pre-trained semantic segmentation model, forward propagation is performed, and a semantic segmentation image is output; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer; in the semantic segmentation model, the image encoding head is responsible for extracting high-level features of the original image, the time step head is responsible for processing time step information and previous segmentation results, the feature fusion module combines the time step feature vector and the first feature image, the diffusion segmentation encoding head performs denoising, the encoder gradually extracts features and reduces the resolution, the decoder gradually restores the resolution and generates a high-resolution feature image, and the output layer generates a final semantic segmentation image. Thus, the semantic segmentation model can generate high-quality semantic segmentation images.

[0053] Since the semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris; in the process of the semantic segmentation model, the gap between the predicted semantic segmentation image output by the semantic segmentation model and the preset semantic segmentation image can be continuously narrowed, that is, the semantic segmentation accuracy of the semantic segmentation model can be continuously improved. Therefore, the trained semantic segmentation model can quickly and accurately classify the pixel categories in the original image and output a high-precision semantic segmentation image.

[0054] In a possible embodiment, step 220 includes:

[0055] The original image and the previous segmentation result are input into the image encoding head. The convolution layer of the image encoding head extracts the feature image of the original image, and the pooling layer of the image encoding head reduces the resolution of the feature image of the original image, and outputs the first feature image of the original image. The previous segmentation result is the previous semantic segmentation image generated by the semantic segmentation model when processing the previous input image. The previous segmentation image contains the category information of each pixel in the previous input image.

[0056] Input the time step information and the previous segmentation result into the time step header, and encode the time step information and the previous segmentation result into a time step feature vector through the time step header;

[0057] Obtaining a second feature image according to the time step feature vector and the first feature image;

[0058] Input the second feature image into the diffusion segmentation encoding head, perform denoising processing through the diffusion segmentation encoding head, and output a third feature image;

[0059] The third feature image is input to the encoder, the features of the third feature image are gradually extracted, and the resolution is reduced through the convolution layer and the pooling layer to output a multi-level feature image. An efficient attention mechanism is introduced between each encoder layer.

[0060] The multi-level feature image is input into the decoder, the resolution of the multi-level feature image is gradually restored, and the fourth feature image is generated through upsampling and convolution layers. An efficient attention mechanism is introduced between each decoder layer;

[0061] The fourth feature image is input to the output layer and the predicted semantic segmentation map is output.

[0062] The image encoding head is responsible for extracting high-level features from the original image. The convolutional layer takes as input the original image and the previous segmentation result. It performs a convolution operation on the original image using a convolution kernel (e.g., 3x3), extracting local features and outputting a feature image of the original image.

[0063] Pooling layer input: Feature image of the original image. Use pooling operation (such as 2x2 maximum pooling) to reduce the resolution of the feature image, the first feature image.

[0064] The time-step header processes the time-step information and the previous segmentation results. The time-step header inputs: time-step information and the previous segmentation results. It encodes the time-step information and the previous segmentation results into a time-step feature vector. The time-step header outputs: a time-step feature vector.

[0065] The feature fusion module is responsible for combining the time step feature vector and the first feature image to generate a second feature image, and combining the time step feature vector and the first feature image to generate a second feature image.

[0066] The diffusion segmentation encoder head is responsible for denoising. The diffusion segmentation encoder head inputs: the second feature image, performs denoising through the diffusion segmentation encoder head to reduce the noise in the image, and outputs: the third feature image.

[0067] The encoder is responsible for gradually extracting features and reducing the resolution. The encoder input is the third feature image. The encoder gradually extracts features from the third feature image and reduces the resolution through convolutional and pooling layers. An efficient attention mechanism is introduced between each encoder layer to help the model better focus on important features. The encoder output is a multi-layer feature image.

[0068] The decoder is responsible for gradually restoring the resolution and generating a high-resolution feature image. Decoder input: multi-layer feature images. The decoder gradually restores the resolution of these multi-layer feature images and generates a high-resolution feature image through upsampling and convolutional layers. An efficient attention mechanism is introduced between each decoder layer to help the model better integrate features. Decoder output: the fourth feature image.

[0069] The output layer is responsible for generating the final semantic segmentation map. The fourth feature image is input to the output layer to generate the predicted semantic segmentation map. Output of the output layer: predicted semantic segmentation map.

[0070] Given a drone-view RGB image x∈R W×H×3 , the goal is to predict a semantic segmentation map s∈R W ×H×d classes , where W and H are the width and height of the image respectively, d classes is the total number of categories.

[0071] The semantic segmentation model is a diffusion model, which consists of two processes:

[0072] The first process is the diffusion process or forward process q, which is a Markov chain that gradually adds Gaussian noise to the data. Given a data distribution X0~q(X0), define q as:

[0073]

[0074] Where: β t ∈(0,1) is the noise schedule, X1...,X t is the latent variable, and T is the number of time steps. t When T is large enough, it is approximately an isotropic Gaussian distribution (spherical Gaussian distribution).

[0075] q(X 1：T |X0) represents the joint distribution of latent variables given the initial state X0.

[0076] (X t |X t-1 ), indicating that at the given previous time step X t-1 In the case of the current time step X t The conditional distribution of .

[0077] X t , represents the latent variable at time step t.

[0078] The time step number T represents the total time steps of the diffusion process.

[0079] The second process is the denoising process or the inverse process p, which is also defined as the process from p0(X t )=N(X T ; 0, I) and gradually learn to remove Gaussian noise:

[0080]

[0081] At a given α t =1-β t and X can be sampled at any time step t t :

[0082]

[0083] α t is the noise variance β t The complement of , which represents the retention ratio of the signal at time step t.

[0084] In the semantic segmentation model, the image encoding head is responsible for extracting high-level features of the original image, the time step head is responsible for processing the time step information and the previous segmentation results, the feature fusion module combines the time step feature vector and the first feature image, the diffusion segmentation encoding head performs denoising, the encoder gradually extracts features and reduces the resolution, the decoder gradually restores the resolution and generates a high-resolution feature image, and the output layer generates the final semantic segmentation map. As a result, the semantic segmentation model can generate high-quality semantic segmentation maps.

[0085] Therefore, through multi-level feature extraction and restoration, the model can better capture the details and global information in the image, thereby improving the accuracy of segmentation; through recursive denoising and efficient attention mechanism, the model can better handle noise and complex scenes and improve the robustness of segmentation; through efficient attention mechanism and hierarchical multi-scale diffusion method, the model can reduce computational complexity and memory usage and improve processing efficiency; through the time step head and previous segmentation results, the model can maintain consistency between consecutive images; the model can adapt to images of different scales and resolutions, and generate high-quality semantic segmentation maps through gradual upsampling and denoising.

[0086] In a possible embodiment, before step 220, the method further includes:

[0087] Step 310: Acquire multiple sets of associated training images and preset semantic segmentation images;

[0088] Step 320: input the training image into the initialized neural network model and output the predicted semantic segmentation image;

[0089] Step 330: training a neural network model based on the preset semantic segmentation image and the predicted semantic segmentation image until a preset training condition is met, thereby obtaining a trained semantic segmentation model;

[0090] Among them, the neural network model includes: image encoding head, time step head, diffusion segmentation encoding head, encoder, decoder and output layer.

[0091] Training images: Image data used to train the neural network model, which are images from the drone's perspective.

[0092] Preset semantic segmentation image: A semantic segmentation image associated with the training image, containing the category information of each pixel.

[0093] Initialized neural network model: an untrained neural network model, consisting of an image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer.

[0094] Predicted semantic segmentation image: The semantic segmentation image generated by the neural network model during training.

[0095] Trained semantic segmentation model: A trained neural network model that can generate high-quality semantic segmentation images.

[0096] Step 310: Obtain multiple sets of associated training images and preset semantic segmentation images

[0097] Step 320: Input the training image into the initialized neural network model; the model generates a predicted semantic segmentation image through forward propagation; output: predicted semantic segmentation image.

[0098] Step 330: Calculate the loss value using the preset semantic segmentation image and the predicted semantic segmentation image. Update the model parameters using backpropagation and an optimization algorithm (e.g., AdamW). Repeat this process until the preset training conditions are met, such as reaching a preset training cycle or the loss value falling below a preset threshold, resulting in a trained semantic segmentation model.

[0099] Through training, the model can better capture details and global information in images, thereby improving segmentation accuracy. During training, the model can better handle noise and complex scenes, improving segmentation robustness. Through an efficient attention mechanism and a layered multi-scale diffusion method, the model can reduce computational complexity and memory usage, improving processing efficiency. By using a time step head and previous segmentation results, the model can maintain consistency between consecutive images, especially in video segmentation tasks. The model can adapt to images of different scales and resolutions, generating high-quality semantic segmentation maps through gradual upsampling and denoising.

[0100] In a possible embodiment, step 320 includes:

[0101] The training image and the previous segmentation result are input into the image encoding head, the feature image of the training image is extracted by the convolution layer of the image encoding head, and the resolution of the feature image of the training image is reduced by the pooling layer of the image encoding head, and the first training feature image of the training image is output; the previous segmentation result is the previous semantic segmentation image generated by the neural network model when processing the previous training image, and the previous segmentation image contains the category information of each pixel in the previous training image;

[0102] Input the time step information and the previous segmentation result into the time step header, and encode the time step information and the previous segmentation result into a time step feature vector through the time step header;

[0103] Obtaining a second training feature image according to the time step feature vector and the first training feature image;

[0104] Input the second training feature image into the diffusion segmentation encoding head, perform denoising processing through the diffusion segmentation encoding head, and output a third training feature image;

[0105] The third training feature image is input into the encoder, the features of the third training feature image are gradually extracted, and the resolution is reduced through the convolution layer and the pooling layer to output a multi-level training feature image. An efficient attention mechanism is introduced between each encoder layer.

[0106] Input the multi-level training feature image into the decoder, gradually restore the resolution of the multi-level training feature image, generate the fourth training feature image through upsampling and convolution layers, and introduce an efficient attention mechanism between each decoder layer;

[0107] The fourth training feature image is input to the output layer and the predicted semantic segmentation map is output.

[0108] The image encoding head is responsible for extracting high-level features of the training image. Its processing principle is as follows:

[0109] Convolutional layer input: training image and previous segmentation results. Use convolution kernel (e.g. 3x3) to convolve the training image to extract local features. Output: feature image.

[0110] Pooling layer input: Feature image. Processing: Reduce the resolution of the feature image using a pooling operation (e.g., 2x2 max pooling). Output: First training feature image.

[0111] The time-step header processes the time-step information and the previous segmentation results. Input: Time-step information and the previous segmentation results. The time-step header encodes the time-step information and the previous segmentation results into a time-step feature vector. Output: Time-step feature vector.

[0112] The feature fusion module is responsible for combining the time step feature vector and the first training feature image to generate the second training feature image and the second training feature image.

[0113] The diffusion segmentation encoder head is responsible for denoising. Input: The second training feature image. The diffusion segmentation encoder head performs denoising to reduce the noise in the image. Output: The third training feature image.

[0114] The encoder is responsible for progressively extracting features and reducing the resolution. Input: The third training feature image. Features are progressively extracted from the third training feature image and the resolution is reduced through convolutional and pooling layers. An efficient attention mechanism is introduced between each encoder layer to help the model better focus on important features. Output: A multi-layered training feature image.

[0115] To integrate the time step into the model, the time step head converts it into a sinusoidal position embedding. The image and segmentation heads have the same structure, each head includes two residual blocks, the residual block structure is shown in the following figure. Figure 4 The sum of the image and the output of the segmentation head is passed to the encoder-decoder. The architecture of the encoder-decoder is shown in the attached figure. Figure 5 As shown in , it incorporates time-step embeddings in each ResNet block. In addition, the model utilizes Efficient Attention, an attention mechanism with linear complexity.

[0116] Time step head: Converts time step information into sinusoidal position embeddings so that the model can understand time series characteristics. Sinusoidal position embedding: Encodes time step information into feature vectors using a sinusoidal function, making it easier for the model to process. The image and segmentation heads have the same structure, each consisting of two residual blocks.

[0117] Residual block: The residual block structure is as shown in the attached Figure 4 As shown in the figure, the original information is retained through skip connection, which improves the training efficiency and performance of the model.

[0118] The encoder-decoder architecture is shown in the attached Figure 5 As shown in , it incorporates time-step embeddings in each ResNet block.

[0119] Time-step embedding: Incorporating time-step embedding into each ResNet block helps the model better understand the temporal sequence characteristics of images.

[0120] Linear complexity: Efficient Attention is an attention mechanism with linear complexity, which improves the efficiency and performance of the model by reducing computational complexity and memory usage.

[0121] The decoder is responsible for gradually restoring the resolution and generating a high-resolution training feature image. Input: Multi-layer training feature images. The decoder gradually restores the resolution of these layers and generates a high-resolution training feature image through upsampling and convolutional layers. An efficient attention mechanism is introduced between each decoder layer to help the model better integrate features. Output: The fourth training feature image.

[0122] The output layer is responsible for generating the final semantic segmentation map. Input: The fourth training feature image. The fourth training feature image is input to the output layer to generate the predicted semantic segmentation map. Output: The predicted semantic segmentation map.

[0123] The embodiment of the present application modifies the diffusion process to make it more suitable for predicting semantic segmentation problems. Given an image x∈X, the corresponding one-hot encoded segmentation map s0∈(S|x) and the time step (noise level) t∈|0,T|, a forward noise process q is defined, which adds a variance of β t Gaussian noise of ∈(0, 1) can be defined as follows:

[0124] q(s t |s t-1 )=s t-1 +N(0,β t I)

[0125] Define the noise schedule β t for:

[0126] The total noise added can be expressed as ∈ t =s t -s0, using neural network ∈ θ To approximate the inverse process (denoising), ∈ θ prediction∈ t , which can be expressed as ∈ t ≈∈ θ (s t , x, t). Then for the segmentation map at any time, we can use s0≈st -∈ θ (s t , x, t) to represent its prediction process.

[0127] In addition, during the training process, the mean square error (MSE) of the prediction noise is used as the loss function, which can be expressed as:

[0128]

[0129] Second, recursive denoising can be described as:

[0130] When training a model on any step of the noise process, the model quickly overfits the training data and has no generalization ability. Therefore, the embodiments of this application consider that binary labels allow learning a simple denoising strategy, such as rounded logical values. This leads to two problems:

[0131] The model does not fully utilize the conditional image, but simply uses noise segmentation;

[0132] During testing, the model relies too much on the initial steps of the denoising process. To address this problem, embodiments of the present application propose training using recursive denoising. Training using recursive denoising involves recursively (as the name suggests) performing each time step t from T to 1, which allows a portion of the prediction error to propagate.

[0133] Figure 3 This paper outlines the recursive noise diffusion process, demonstrating the complete pipeline from pure noise to the final denoised predicted segmentation. The previously predicted segmentation is diffused using a noise function, denoised using a diffusion model, and finally compared with the ground truth. This method, through recursive denoising, gradually refines the segmentation results, improving both accuracy and robustness.

[0134] It is important to note that the labeled ground truth segmentation is not used as part of the model input. This process is agnostic to the choice of noise function, diffusion model, and loss function, and the entire process can be represented by the following pseudocode:

[0135]

[0136] The recursive denoising method involved in the embodiments of the present application can be viewed as a simulation test process that needs to be advanced through each time step. A key feature is its uncertainty in the selection of noise function, diffusion model, and loss function. This feature provides flexibility, allowing the most appropriate components to be selected for a specific task or dataset. This flexibility provides greater diversity and adaptability, enabling the embodiments of the present application to be widely applicable to various tasks and settings.

[0137] Recursive noise diffusion does not impose any restrictions on the noise function. However, the one-hot encoded binary label of the class is robust to low noise. The embodiments of this application propose a Softmax-Average noise function, which is both effective and controllable. The Softmax-Average noise function for each pixel can be defined as:

[0138]

[0139] where z is a softmax vector sampled from a normal distribution, each element is random noise, and:

[0140]

[0141] The Softmax-Average noise function is a function used to generate noise data, especially suitable for the noise addition process in the diffusion model. The noise function of each pixel can be defined as:

[0142] in: is the noise pixel value at time step t.

[0143] is the original pixel value or the previous pixel value.

[0144] z is a softmax vector sampled from a normal distribution, and each element is random noise.

[0145] λ noise,t is the noise intensity parameter, which varies with time step t.

[0146] T is the total number of time steps.

[0147] t is the current time step.

[0148] Noise intensity changes over time: λ noise,t As the time step t increases, it decreases, indicating that the noise intensity gradually weakens. At the initial time step t = 0, the noise intensity is the largest. At the final time step t = T, there is no noise.

[0149] The Softmax function is used to convert the input vector into a probability distribution, ensuring that each element is between 0 and 1 and the sum of all elements is 1. In the noise function, the Softmax function is used to convert the mixed original pixel value and noise value into a new pixel value distribution.

[0150] The Softmax-Average noise function generates a new pixel value distribution by mixing the original pixel values ​​with random noise and applying the Softmax function. The noise intensity parameter controls the intensity of the noise over time. This noise function is used in diffusion models to gradually add noise to generate new data samples.

[0151] In the semantic segmentation model, the image encoding head extracts high-level features from the training image, the time step head processes time step information and previous segmentation results, the feature fusion module combines the time step feature vector with the first training feature image, the diffusion segmentation encoding head performs denoising, the encoder gradually extracts features and reduces the resolution, the decoder gradually restores the resolution and generates a high-resolution training feature image, and the output layer generates the final semantic segmentation map. Through the collaborative work of these modules, the model can generate high-quality semantic segmentation maps.

[0152] In a possible embodiment, step 330 includes:

[0153] The loss value between the preset semantic segmentation image and the predicted semantic segmentation image is calculated through the cross entropy loss function;

[0154] The neural network model is trained according to the loss value until the preset training conditions are met to obtain a trained semantic segmentation model.

[0155] In deep learning, the cross-entropy loss function is a loss function that measures the difference between the predicted result and the true result. In semantic segmentation tasks, the cross-entropy loss function is used to calculate the loss value between the preset semantic segmentation image and the predicted semantic segmentation image, thereby guiding the model training process.

[0156] The preset semantic segmentation image and the predicted semantic segmentation image are converted into category labels and predicted probabilities; the cross entropy loss function is used to calculate the difference between the two to obtain the loss value, and the neural network model is trained based on the loss value.

[0157] Use the backpropagation algorithm to calculate the gradient of the loss with respect to the model parameters and use an optimization algorithm (such as AdamW) to update the model parameters. Repeat the above process until the preset training conditions are met, such as reaching the preset training cycle or the loss value is lower than the preset threshold, to obtain a trained semantic segmentation model.

[0158] The initialized neural network moxing can be UNetFormer. UNetFormer is a neural network model for image segmentation. Its loss function consists of two parts:

[0159] Main loss: used to measure the difference between the semantic segmentation map generated by the model and the real semantic segmentation map. Auxiliary loss: used to measure the difference between the features extracted from the global-local transformer block (GLTB) and the real features when the model processes the image.

[0160] In this example, the auxiliary loss of UNetFormer is retained. This loss measures the difference between the features extracted by the model from the global-local transformer block and the true features. The main loss of UNetFormer is simplified, retaining only the cross-entropy component. Cross-entropy is a commonly used loss function that measures the difference between the semantic segmentation maps generated by the model and the true semantic segmentation maps.

[0161] By retaining the auxiliary loss of UNetFormer and simplifying the main loss, the diversity of the embodiment method of this application is demonstrated. This method is not only applicable to UNetFormer, but also to other similar models. Simplifying the main loss can reduce computational complexity and improve model training efficiency. Despite simplifying the main loss, by retaining the auxiliary loss, the model can still maintain high segmentation performance.

[0162] The cross-entropy loss function is used to calculate the loss between the preset semantic segmentation image and the predicted semantic segmentation image. The neural network model is trained based on this loss value until the preset training conditions are met, resulting in a trained semantic segmentation model. The cross-entropy loss function effectively measures the difference between the predicted and true results, guiding the model's gradual optimization and ultimately generating a high-quality semantic segmentation map.

[0163] The neural network model is trained according to the loss value until the preset training conditions are met to obtain a trained semantic segmentation model, including:

[0164] Calculate the gradient of the loss value with respect to each model parameter using the chain rule to obtain the gradient of each model parameter;

[0165] The AdamW optimizer is used to update the parameters of the neural network model according to the calculated gradient of each model parameter; and within the preset training cycle, the learning rate of the neural network model is gradually adjusted through the cosine annealing learning rate adjustment strategy until the preset training conditions are met to obtain a trained semantic segmentation model.

[0166] A large number of time steps, in the range of hundreds or thousands, is used during training. By increasing the number of time steps, the model can better capture details and noise in the image, thereby improving segmentation performance.

[0167] Timesteps: During training, the number of timesteps increases from 100 to 1000.

[0168] Trend: As the number of time steps increases, the segmentation score of the model gradually improves.

[0169] In order to improve the robustness and generalization ability of the model, data augmentation technology is used during the training process. Specific data augmentation operations include:

[0170] Brightness adjustment: Randomly adjust the brightness of the image in the range of [-0.2, 0.2].

[0171] Contrast Adjustment: Randomly adjust the contrast of the image in the range [0.8, 1.2].

[0172] Saturation adjustment: Randomly adjust the saturation of the image in the range [0.8, 1.2].

[0173] Hue Adjustment: Randomly adjust the hue of the image in the range [-0.1, 0.1].

[0174] Random Horizontal Flip: Randomly flip the image horizontally with a probability of 50%.

[0175] During training, the following parameter settings were used:

[0176] Training epochs: 100 epochs.

[0177] Batch size: 8.

[0178] Optimizer: AdamW optimizer.

[0179] Learning rate: 6×10^(-4).

[0180] Cosine annealing weight decay coefficient: 0.01.

[0181] Hardware: Single NVIDIA Tesla V100 GPU.

[0182] This embodiment studies the impact of changes in the number of time steps during the training process.

[0183] As shown in Table 1, a clear trend emerges, indicating that more training time steps can produce higher segmentation scores.

[0184] Table 1

[0185]

[0186] Diffusion models tend to use a large number of time steps during training, ranging from hundreds to thousands. By increasing the number of time steps, the model can better capture details and noise in the image, thereby improving segmentation performance. Data augmentation techniques (such as brightness, contrast, saturation, and hue adjustments, as well as random horizontal flips) further improve the model's robustness and generalization ability. Using the AdamW optimizer and a cosine annealing learning rate adjustment strategy, the model is gradually optimized over 100 epochs, ultimately achieving a high segmentation score. Through these steps, the model is gradually optimized, ultimately producing high-quality semantic segmentation maps.

[0187] Therefore, when training the neural network model based on the loss value, the chain rule is used to calculate the gradient of the loss value with respect to each model parameter. The AdamW optimizer is used to update the neural network model parameters based on the calculated gradient of each model parameter. The learning rate is then gradually adjusted using a cosine annealing learning rate adjustment strategy until the preset training conditions are met, resulting in a trained semantic segmentation model. Through these steps, the model can be gradually optimized, ultimately generating a high-quality semantic segmentation map.

[0188] In a possible embodiment, the diffusion segmentation encoder head performs denoising processing at each scale, and the step of inputting the second training feature image into the diffusion segmentation encoder head, performing denoising processing by the diffusion segmentation encoder head, and outputting the third training feature image includes:

[0189] Define a scaling plan, which includes defining the minimum and maximum scales, and dividing the time steps for each scale;

[0190] Performing initial segmentation on the minimum-scale second training feature image through a diffusion segmentation encoding head, processing the noise in the image, and generating a denoised minimum-scale second training feature image;

[0191] Generate a preliminary semantic segmentation image based on the denoised minimum-scale second training feature image;

[0192] Through the diffusion segmentation encoding head, the noise in the current scale image is recursively denoised based on the preliminary semantic segmentation image to generate the current scale semantic segmentation image; the current scale is larger than the minimum scale and smaller than the maximum scale;

[0193] Upsampling the semantic segmentation image of the current scale using bilinear interpolation to generate an upsampled semantic segmentation image; the resolution of the upsampled semantic segmentation image is higher than the resolution of the semantic segmentation image of the current scale;

[0194] Denoised predictions are performed at each time step until the original scale of the second training feature image is reached, resulting in the third training feature image.

[0195] Scaling schedule: Defines the minimum and maximum scales, and the schedule for dividing the time steps for each scale.

[0196] Minimum scale: The starting scale for image processing, usually a lower resolution image.

[0197] Maximum scale: The final scale of image processing, usually a higher resolution image.

[0198] Time steps: The number of steps to perform denoising at each scale.

[0199] Bilinear interpolation: An image processing technique used to upsample a low-resolution image to a high-resolution image.

[0200] The diffusion segmentation encoder performs initial segmentation on the minimum-scale second training feature image, processes the noise in the image, and generates a denoised minimum-scale second training feature image. Output: The denoised minimum-scale second training feature image.

[0201] Generate a preliminary semantic segmentation image based on the denoised minimum-scale second training feature image. Output: Preliminary semantic segmentation image. Using the diffusion segmentation encoder, recursively denoise the noise in the current-scale image based on the preliminary semantic segmentation image to generate a semantic segmentation image at the current scale. Output: Semantic segmentation image at the current scale.

[0202] Upsampling Input: Semantic segmentation image of the current scale. Use bilinear interpolation to upsample the semantic segmentation image of the current scale to generate the upsampled semantic segmentation image. Output: Semantic segmentation image of the upsampled scale.

[0203] Denoising prediction input: upsampled semantic segmentation image. Denoising prediction is performed at each time step until the original scale of the second training feature image is reached, resulting in the third training feature image. Output: the third training feature image.

[0204] Performing multi-scale image segmentation can significantly improve the prediction results by exploiting the propagation properties of recursive denoising and using a hierarchical multi-scale denoising process, such as Figure 6 As shown, multi-scale processing: processing images at different scales to gradually capture different levels of details and global information.

[0205] Improved predictions: By processing at multiple scales, the model can better capture details and global information in the image, thereby improving segmentation accuracy.

[0206] Recursive denoising: When processing the current image, the model uses not only the information of the current image, but also the segmentation results of previously processed images.

[0207] Propagation characteristics: Through recursive denoising, the model can utilize previous segmentation results and gradually optimize the segmentation results, thereby improving the accuracy and robustness of segmentation.

[0208] Hierarchical multi-scale denoising: recursively denoising at different scales to gradually optimize the segmentation results.

[0209] Table 2

[0210]

[0211] Table 3

[0212]

[0213] The recursive denoising method provided by the embodiments of this application achieves good results in image segmentation tasks, outperforming other recently proposed methods such as BANet and UAVFormer. The following is a detailed explanation:

[0214] Recursive denoising outperforms other methods: The recursive denoising method outperforms BANet and UAVFormer in segmentation performance. See Table 2 for specific metrics. The human category is particularly challenging because it is underrepresented in the dataset and each object is small and isolated within the scene. Table 2 shows the performance of the recursive denoising method on multiple evaluation metrics (such as Intersection over Union (IoU) and Dice coefficient), which shows that it outperforms other methods.

[0215] The implementation data in Table 3 validates the effectiveness of the hierarchical multi-scale method and demonstrates its direct benefits when combined with recursive denoising. Table 3 shows the performance of the hierarchical multi-scale method on multiple evaluation metrics, indicating that this method can significantly improve segmentation performance.

[0216] The recursive denoising method provided in the embodiments of the present application has achieved good results in image segmentation tasks, outperforming other recently proposed methods such as BANet and UAVFormer. Despite the presence of challenging categories (such as Human), the recursive denoising method performs well. Since this is the first iteration, there is still room for improvement by optimizing hyperparameters. The effect of the hierarchical multi-scale method was verified by implementing data, demonstrating its direct benefits when used in combination with recursive denoising. Through these improvements, the model can further improve segmentation performance and generate higher quality semantic segmentation maps.

[0217] Since the recursive denoising method is independent of the choice of model, an efficient and effective model can be chosen for multi-class segmentation from an aerial perspective, such as UNetFormer. UNetFormer can be easily converted to a diffusion model by incorporating the previous segmentation as an additional input, which only requires minor modifications to the original architecture. Figure 7 Architectural changes are shown. Specifically, the previous segmentation input is scaled using bilinear interpolation and concatenated with channels that minimize the introduction of additional network complexity. This model is equivalent to a variant of UNetFormer with these modifications.

[0218] The recursive denoising method does not rely on a specific model architecture, so any efficient and effective model can be selected; it is suitable for multi-category segmentation tasks from an aerial perspective, such as buildings, roads, trees, etc.; UNetFormer is an efficient and effective model suitable for multi-category segmentation tasks from an aerial perspective.

[0219] By including the previous segmentation as an additional input, UNetFormer can be easily converted to a diffusion model, with Figure 7 The architecture changes are shown. Specifically, the input of the previous segmentation is scaled using bilinear interpolation and concatenated with the channels to minimize the introduction of additional network complexity. The input of the previous segmentation is scaled using bilinear interpolation to fit the input requirements of the model. The scaled input of the previous segmentation is concatenated with the channels of the model to minimize the introduction of additional network complexity.

[0220] Therefore, through multi-scale denoising, the semantic segmentation model can better capture details and global information in the image, thereby improving segmentation accuracy. Through recursive denoising, the model can better handle noise and complex scenes, improving segmentation robustness. Through layered multi-scale denoising methods, the model can reduce computational complexity and memory usage, improving processing efficiency. By combining time steps and previous segmentation results, the model can maintain consistency between consecutive images, especially in video segmentation tasks. The semantic segmentation model can adapt to images of different scales and resolutions, generating high-quality semantic segmentation maps through gradual upsampling and denoising.

[0221] In an embodiment of the present application, an original image from the perspective of a drone is obtained; the original image is input into a pre-trained semantic segmentation model, forward propagation is performed, and a semantic segmentation image is output; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer; in the semantic segmentation model, the image encoding head is responsible for extracting high-level features of the original image, the time step head is responsible for processing time step information and previous segmentation results, the feature fusion module combines the time step feature vector and the first feature image, the diffusion segmentation encoding head performs denoising, the encoder gradually extracts features and reduces the resolution, the decoder gradually restores the resolution and generates a high-resolution feature image, and the output layer generates a final semantic segmentation image. Thus, the semantic segmentation model can generate high-quality semantic segmentation images.

[0222] The multi-category segmentation method of the embodiment of the present application was tested on the UAvid verification dataset. Specifically, the trained semantic segmentation model was used to make predictions on the test set. Figure 8 The segmentation results are shown, showing that the embodiments of the present application demonstrate good segmentation capabilities in segmenting moving cars from stationary cars given a static input image. This demonstrates that the embodiments of the present application utilize contextual clues found in the surrounding scene to determine whether a car is parked or moving, and that in some cases, the embodiments of the present application can produce better results than labeled masks.

[0223] Figure 8The segmentation results are shown, and it can be seen that the embodiment of the present application shows good segmentation capabilities in segmenting moving cars and stationary cars. Given a static input image, the model is able to accurately segment moving cars and stationary cars. The embodiment of the present application uses contextual clues found in the surrounding scene to determine whether the car is parked or moving. The role of contextual clues: By analyzing the contextual information of the surrounding scene, the model can better understand the state of objects in the image, thereby improving the accuracy of segmentation.

[0224] Better in Some Cases: The embodiments of this application can produce better results than labeled masks in some cases. Labeled masks: Labeled masks are manually labeled segmentation results and may contain errors or inaccuracies. Model Advantages: By leveraging contextual clues and recursive denoising methods, the model can generate more accurate segmentation results.

[0225] Since the semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris; in the process of the semantic segmentation model, the gap between the predicted semantic segmentation image output by the semantic segmentation model and the preset semantic segmentation image can be continuously narrowed, that is, the semantic segmentation accuracy of the semantic segmentation model can be continuously improved. Therefore, the trained semantic segmentation model can quickly and accurately classify the pixel categories in the original image and output a high-precision semantic segmentation image.

[0226] Based on the above Figure 1 The semantic segmentation method shown in the embodiment of the present application also provides a semantic segmentation device, such as Figure 9 As shown, the semantic segmentation device 900 may include:

[0227] An acquisition module 910 is used to acquire an original image from the perspective of the drone;

[0228] Output module 920, used to input the original image into a pre-trained semantic segmentation model, perform forward propagation, and output a semantic segmentation image; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer;

[0229] The semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris.

[0230] In a possible embodiment, the output module 920 is specifically configured to:

[0231] The original image and the previous segmentation result are input into the image encoding head. The convolution layer of the image encoding head extracts the feature image of the original image, and the pooling layer of the image encoding head reduces the resolution of the feature image of the original image, and outputs the first feature image of the original image. The previous segmentation result is the previous semantic segmentation image generated by the semantic segmentation model when processing the previous input image. The previous segmentation image contains the category information of each pixel in the previous input image.

[0232] Input the time step information and the previous segmentation result into the time step header, and encode the time step information and the previous segmentation result into a time step feature vector through the time step header;

[0233] Obtaining a second feature image according to the time step feature vector and the first feature image;

[0234] Input the second feature image into the diffusion segmentation encoding head, perform denoising processing through the diffusion segmentation encoding head, and output a third feature image;

[0235] The third feature image is input to the encoder, the features of the third feature image are gradually extracted, and the resolution is reduced through the convolution layer and the pooling layer to output a multi-level feature image. An efficient attention mechanism is introduced between each encoder layer.

[0236] The multi-level feature image is input into the decoder, the resolution of the multi-level feature image is gradually restored, and the fourth feature image is generated through upsampling and convolution layers. An efficient attention mechanism is introduced between each decoder layer;

[0237] The fourth feature image is input to the output layer and the predicted semantic segmentation map is output.

[0238] In a possible embodiment, the acquisition module 910 is further configured to acquire multiple sets of associated training images and preset semantic segmentation images;

[0239] The output module 920 is further configured to input the training image into the initialized neural network model and output a predicted semantic segmentation image;

[0240] The semantic segmentation apparatus 900 may further include:

[0241] A training module is used to train a neural network model based on a preset semantic segmentation image and a predicted semantic segmentation image until a preset training condition is met to obtain a trained semantic segmentation model;

[0242] Among them, the neural network model includes: image encoding head, time step head, diffusion segmentation encoding head, encoder, decoder and output layer.

[0243] In a possible embodiment, the output module 920 is specifically configured to:

[0244] The training image and the previous segmentation result are input into the image encoding head, the feature image of the training image is extracted by the convolution layer of the image encoding head, and the resolution of the feature image of the training image is reduced by the pooling layer of the image encoding head, and the first training feature image of the training image is output; the previous segmentation result is the previous semantic segmentation image generated by the neural network model when processing the previous training image, and the previous segmentation image contains the category information of each pixel in the previous training image;

[0245] Input the time step information and the previous segmentation result into the time step header, and encode the time step information and the previous segmentation result into a time step feature vector through the time step header;

[0246] Obtaining a second training feature image according to the time step feature vector and the first training feature image;

[0247] Input the second training feature image into the diffusion segmentation encoding head, perform denoising processing through the diffusion segmentation encoding head, and output a third training feature image;

[0248] The third training feature image is input into the encoder, the features of the third training feature image are gradually extracted, and the resolution is reduced through the convolution layer and the pooling layer to output a multi-level training feature image. An efficient attention mechanism is introduced between each encoder layer.

[0249] Input the multi-level training feature image into the decoder, gradually restore the resolution of the multi-level training feature image, generate the fourth training feature image through upsampling and convolution layers, and introduce an efficient attention mechanism between each decoder layer;

[0250] The fourth training feature image is input to the output layer and the predicted semantic segmentation map is output.

[0251] In a possible embodiment, the training module is specifically configured to:

[0252] The loss value between the preset semantic segmentation image and the predicted semantic segmentation image is calculated through the cross entropy loss function;

[0253] The neural network model is trained according to the loss value until the preset training conditions are met to obtain a trained semantic segmentation model.

[0254] In a possible embodiment, the training module is specifically configured to:

[0255] Calculate the gradient of the loss value with respect to each model parameter using the chain rule to obtain the gradient of each model parameter;

[0256] The AdamW optimizer is used to update the parameters of the neural network model according to the calculated gradient of each model parameter; and within the preset training cycle, the learning rate of the neural network model is gradually adjusted through the cosine annealing learning rate adjustment strategy until the preset training conditions are met to obtain a trained semantic segmentation model.

[0257] In a possible embodiment, the output module 920 is specifically configured to:

[0258] Define a scaling plan, which includes defining the minimum and maximum scales, and dividing the time steps for each scale;

[0259] Performing initial segmentation on the minimum-scale second training feature image through a diffusion segmentation encoding head, processing the noise in the image, and generating a denoised minimum-scale second training feature image;

[0260] Generate a preliminary semantic segmentation image based on the denoised minimum-scale second training feature image;

[0261] Through the diffusion segmentation encoding head, the noise in the current scale image is recursively denoised based on the preliminary semantic segmentation image to generate the current scale semantic segmentation image; the current scale is larger than the minimum scale and smaller than the maximum scale;

[0262] The semantic segmentation image of the current scale is upsampled using bilinear interpolation to generate an upsampled semantic segmentation image; the resolution of the upsampled semantic segmentation image is higher than the resolution of the semantic segmentation image of the current scale.

[0263] In an embodiment of the present application, an original image from the perspective of a drone is obtained; the original image is input into a pre-trained semantic segmentation model, forward propagation is performed, and a semantic segmentation image is output; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer; in the semantic segmentation model, the image encoding head is responsible for extracting high-level features of the original image, the time step head is responsible for processing time step information and previous segmentation results, the feature fusion module combines the time step feature vector and the first feature image, the diffusion segmentation encoding head performs denoising, the encoder gradually extracts features and reduces the resolution, the decoder gradually restores the resolution and generates a high-resolution feature image, and the output layer generates a final semantic segmentation image. Thus, the semantic segmentation model can generate high-quality semantic segmentation images.

[0264] Since the semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris; in the process of the semantic segmentation model, the gap between the predicted semantic segmentation image output by the semantic segmentation model and the preset semantic segmentation image can be continuously narrowed, that is, the semantic segmentation accuracy of the semantic segmentation model can be continuously improved. Therefore, the trained semantic segmentation model can quickly and accurately classify the pixel categories in the original image and output a high-precision semantic segmentation image.

[0265] Figure 10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0266] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0267] Specifically, the processor 401 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0268] The memory 402 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 402 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 402 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 402 is a non-volatile solid-state memory. In a specific embodiment, the memory 402 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0269] The processor 401 implements any one of the semantic segmentation methods in the embodiments shown in the figure by reading and executing computer program instructions stored in the memory 402.

[0270] In one example, the electronic device may further include a communication interface 404 and a bus 410. Figure 4 As shown, the processor 401 , the memory 402 , and the communication interface 404 are connected via a bus 410 and communicate with each other.

[0271] The communication interface 404 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0272] Bus 410 comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 410 can comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0273] The electronic device can execute the semantic segmentation method in the embodiment of the present application, thereby realizing the combination Figure 2 Describe the semantic segmentation method.

[0274] In addition, in combination with the semantic segmentation method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the Figure 1 Semantic segmentation method.

[0275] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0276] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0277] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0278] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A semantic segmentation method, characterized in that: The method comprises: Get the original image from the drone's perspective; Input the original image into the pre-trained semantic segmentation model, perform forward propagation, and output the semantic segmentation image; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer; The semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris.

2. According to the method of claim 1, the step of inputting the original image into a pre-trained semantic segmentation model, performing forward propagation, and generating a semantic segmentation image comprises: The original image and the previous segmentation result are input into the image encoding head, the feature image of the original image is extracted through the convolution layer of the image encoding head, and the resolution of the feature image of the original image is reduced through the pooling layer of the image encoding head, and the first feature image of the original image is output; the previous segmentation result is a previous semantic segmentation image generated by the semantic segmentation model when processing the previous input image, and the previous segmentation image contains the category information of each pixel in the previous input image; Input the time step information and the previous segmentation result into the time step header, and encode the time step information and the previous segmentation result into a time step feature vector through the time step header; Obtaining a second feature image according to the time step feature vector and the first feature image; Input the second feature image to the diffusion segmentation encoding head, perform denoising processing through the diffusion segmentation encoding head, and output a third feature image; The third feature image is input into the encoder, the features of the third feature image are gradually extracted, and the resolution is reduced through the convolution layer and the pooling layer, and a multi-level feature image is output. An efficient attention mechanism is introduced between each encoder layer; Input the multi-level feature image into the decoder, gradually restore the resolution of the multi-level feature image, generate the fourth feature image through upsampling and convolution layers, and introduce an efficient attention mechanism between each decoder layer; The fourth feature image is input to the output layer and the predicted semantic segmentation map is output.

3. According to the method of claim 1, before inputting the original image into the pre-trained semantic segmentation model for forward propagation to generate the semantic segmentation image, the method further comprises: Acquire multiple sets of associated training images and preset semantic segmentation images; The preset semantic segmentation image includes category information of each pixel in the training image; the category information includes at least one of the following: building, road, tree, low vegetation, moving car, stationary car, human and debris; Input the training image into the initialized neural network model and output the predicted semantic segmentation image; According to the preset semantic segmentation image and the predicted semantic segmentation image, the neural network model is trained until the preset training conditions are met to obtain a trained semantic segmentation model; Among them, the neural network model includes: image encoding head, time step head, diffusion segmentation encoding head, encoder, decoder and output layer.

4. The method according to claim 3, wherein the step of inputting the training image into the initialized neural network model and outputting the predicted semantic segmentation image comprises: The training image and the previous segmentation result are input into the image encoding head, the feature image of the training image is extracted through the convolution layer of the image encoding head, and the resolution of the feature image of the training image is reduced through the pooling layer of the image encoding head, and the first training feature image of the training image is output; the previous segmentation result is a previous semantic segmentation image generated by the neural network model when processing the previous training image, and the previous segmentation image contains the category information of each pixel in the previous training image; Input the time step information and the previous segmentation result into the time step header, and encode the time step information and the previous segmentation result into a time step feature vector through the time step header; Obtaining a second training feature image according to the time step feature vector and the first training feature image; Input the second training feature image into the diffusion segmentation encoding head, perform denoising processing through the diffusion segmentation encoding head, and output a third training feature image; Input the third training feature image into the encoder, gradually extract the features of the third training feature image, reduce the resolution through the convolution layer and the pooling layer, output a multi-level training feature image, and introduce an efficient attention mechanism between each encoder layer; Input the multi-level training feature image into the decoder, gradually restore the resolution of the multi-level training feature image, generate the fourth training feature image through upsampling and convolution layers, and introduce an efficient attention mechanism between each decoder layer; The fourth training feature image is input into the output layer, and the predicted semantic segmentation map is output.

5. The method according to claim 3, wherein the initializing the neural network model is trained according to the preset semantic segmentation image and the predicted semantic segmentation image until the preset training conditions are met to obtain the trained semantic segmentation model, comprising: The loss value between the preset semantic segmentation image and the predicted semantic segmentation image is calculated through the cross entropy loss function; The neural network model is trained according to the loss value until the preset training conditions are met to obtain a trained semantic segmentation model.

6. The method according to claim 5, wherein the neural network model is trained according to the loss value until a preset training condition is met to obtain a trained semantic segmentation model, comprising: The gradient of the loss value with respect to each model parameter is calculated by the chain rule to obtain the gradient of each model parameter; The AdamW optimizer is used to update the parameters of the neural network model according to the calculated gradient of each model parameter; and within the preset training cycle, the learning rate of the neural network model is gradually adjusted through the cosine annealing learning rate adjustment strategy until the preset training conditions are met to obtain a trained semantic segmentation model.

7. The method according to claim 4, wherein the diffusion segmentation encoder head performs denoising processing at each scale, wherein the second training feature image is input to the diffusion segmentation encoder head, denoising processing is performed by the diffusion segmentation encoder head, and the third training feature image is output, comprising: Define a scaling plan, which includes: defining the minimum scale and the maximum scale, and dividing the time steps for each scale; Performing initial segmentation on the minimum-scale second training feature image through a diffusion segmentation encoding head, processing noise in the image, and generating a denoised minimum-scale second training feature image; Generate a preliminary semantic segmentation image based on the denoised minimum-scale second training feature image; Through the diffusion segmentation encoding head, the noise in the image of the current scale is recursively denoised according to the preliminary semantic segmentation image to generate the semantic segmentation image of the current scale; the current scale is greater than the minimum scale and less than the maximum scale; Upsampling the semantic segmentation image of the current scale using bilinear interpolation to generate an upsampled semantic segmentation image; the resolution of the upsampled semantic segmentation image is higher than the resolution of the semantic segmentation image of the current scale; Denoising prediction is performed at each time step until the original scale of the second training feature image is reached, and the third training feature image is obtained.

8. A semantic segmentation device, characterized in that: The semantic segmentation device comprises: The acquisition module is used to obtain the original image from the drone's perspective; The output module is used to input the original image into the pre-trained semantic segmentation model, perform forward propagation, and output the semantic segmentation image; the semantic segmentation model includes: a pre-trained image encoding head, a time step head, a diffusion segmentation encoding head, an encoder, a decoder, and an output layer; The semantic segmentation model is trained based on multiple training images; the training images are images from the perspective of a drone; the semantic segmentation images include category information for each pixel in the training images; the category information includes at least one of the following: buildings, roads, trees, low vegetation, moving cars, stationary cars, humans, and debris.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the semantic segmentation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the semantic segmentation method according to any one of claims 1 to 7 is implemented.

Images