Image semantic segmentation method, device, equipment, storage medium and program product

Abstract

The application provides a semantic segmentation method and device of an image, equipment, a storage medium and a program product. The method comprises: acquiring a to-be-segmented image comprising at least two objects and a depth image corresponding to the to-be-segmented image; encoding the depth image to obtain a depth encoding result; calling at least two segmentation encoding networks to perform iterative fusion encoding comprising spatial screening and channel recombination on the depth encoding result and the to-be-segmented image to obtain a target encoding result, wherein the spatial screening is used for feature screening of the to-be-segmented image in a spatial dimension, and the channel recombination is used for feature screening of the to-be-segmented image in a channel dimension; and performing semantic segmentation on the to-be-segmented image based on the target encoding result to obtain a semantic segmentation result corresponding to each object. Through the application, the complementarity and interdependence between the semantics of the image are fully exploited, and the accuracy of semantic segmentation is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a semantic segmentation method, apparatus, device, storage medium, and program product for images. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] In related technologies, semantic segmentation of images is usually performed by directly processing the image to be segmented to obtain the corresponding semantic segmentation result. However, since the image to be segmented is processed directly, it is impossible to fully explore the complementarity and interdependence between the semantics of the images. Summary of the Invention

[0004] This application provides a semantic segmentation method, apparatus, computer-readable storage medium, and computer program product for images, which can fully exploit the complementarity and interdependence between semantics in images and effectively improve the accuracy of semantic segmentation.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a semantic segmentation method for images, including:

[0007] Obtain an image to be segmented that includes at least two objects, and a depth image corresponding to the image to be segmented;

[0008] The depth image is encoded to obtain a depth encoding result;

[0009] At least two segmentation coding networks are invoked to perform iterative fusion coding, including spatial filtering and channel reconstruction, on the depth coding result and the image to be segmented to obtain the target coding result. The spatial filtering is used to perform feature filtering on the image to be segmented in the spatial dimension, and the channel reconstruction is used to perform feature filtering on the image to be segmented in the channel dimension.

[0010] Based on the target encoding result, semantic segmentation is performed on the image to be segmented to obtain the semantic segmentation result corresponding to each of the objects.

[0011] In some embodiments, the above-mentioned invocation of the channel attention layer to perform channel reconstruction on the spatial filtering result to obtain the channel reconstruction result includes: performing convolution processing on the spatial filtering result to obtain a second convolution processing result of the spatial filtering result; performing convolution processing on the first convolution processing result to obtain a third convolution processing result; performing normalization processing on the third convolution processing result to obtain a normalization processing result; performing dot product on the normalization processing result and the second convolution processing result to obtain a second dot product result; and determining the second dot product result as the channel reconstruction result.

[0012] This application provides an image semantic segmentation apparatus, comprising:

[0013] The acquisition module is used to acquire an image to be segmented, which includes at least two objects, and a depth image corresponding to the image to be segmented;

[0014] An encoding module is used to encode the depth image to obtain a depth encoding result;

[0015] The fusion coding module is used to call at least two segmentation coding networks to perform iterative fusion coding, including spatial filtering and channel reconstruction, on the deep coding result and the image to be segmented to obtain the target coding result. The spatial filtering is used to perform feature filtering on the image to be segmented in the spatial dimension, and the channel reconstruction is used to perform feature filtering on the image to be segmented in the channel dimension.

[0016] The semantic segmentation module is used to perform semantic segmentation on the image to be segmented based on the target encoding result, so as to obtain the semantic segmentation result corresponding to each of the objects.

[0017] In some embodiments, the above-described encoding module is further configured to perform downsampling processing on the depth image to obtain a downsampling processing result of the depth image; perform pooling processing on the downsampling processing result of the depth image to obtain a pooling processing result of the depth image; and invoke at least two depth encoding networks to perform iterative encoding processing on the pooling processing result of the depth image to obtain the depth encoding result.

[0018] In some embodiments, the above-described encoding module is further configured to invoke a first depth encoding network to encode the pooling result of the depth image to obtain a first depth encoding result; invoke an (i+1)th depth encoding network to encode the i-th depth encoding result to obtain an (i+1)-th depth encoding result; and determine the N-th depth encoding result as the depth encoding result; wherein 1≤i≤N-1, N represents the number of depth encoding networks, and the size of the (i+1)-th depth encoding network is smaller than the size of the i-th depth encoding network.

[0019] In some embodiments, the deep coding network includes at least two coding layers with identical structures, the at least two coding layers with identical structures including a first coding layer and a second coding layer; the coding module is further configured to call the first coding layer to encode the i-th deep coding result to obtain a first coding result; call the second coding layer to encode the first coding result to obtain a second coding result; and determine the second coding result as the (i+1)-th deep coding result.

[0020] In some embodiments, the above-described fusion coding module is further configured to perform downsampling processing on the image to be segmented to obtain a downsampling processing result of the image to be segmented; perform pooling processing on the downsampling processing result of the image to be segmented to obtain a pooling processing result of the image to be segmented; and invoke the at least two segmentation coding networks to perform iterative fusion coding including spatial filtering and channel reorganization on the pooling processing result of the image to be segmented and the depth coding result to obtain the target coding result.

[0021] In some embodiments, the depth coding result includes i depth coding results, 1≤i≤N-1, where N represents the number of depth coding networks encoding the depth image; the above-mentioned fusion coding module is further configured to call the first segmentation coding network to perform fusion coding including spatial filtering and channel reconstruction on the pooling processing result of the image to be segmented and the first depth coding result to obtain the first segmentation coding result; call the (i+1)th segmentation coding network to perform fusion coding including spatial filtering and channel reconstruction on the i-th segmentation coding result and the i-th depth coding result to obtain the (i+1)th segmentation coding result; and determine the N-th segmentation coding result as the target coding result; wherein the size of the (i+1)th segmentation coding network is smaller than the size of the i-th segmentation coding network.

[0022] In some embodiments, the segmentation coding network includes at least two residual layers and at least one attention residual layer; the fusion coding module is further configured to call the at least two residual layers to extract features from the i-th segmentation coding result to obtain the feature extraction result of the i-th segmentation coding result; and call the at least one attention residual layer to perform fusion coding, including spatial filtering and channel reorganization, on the feature extraction result and the i-th depth coding result to obtain the (i+1)-th segmentation coding result.

[0023] In some embodiments, when the number of attention residual layers is at least two, the above-mentioned fusion encoding module is further configured to call the first attention residual layer to perform fusion encoding including spatial filtering and channel reconstruction on the feature extraction result and the i-th depth encoding result to obtain the first fusion encoding result; call the j-th attention residual layer to perform fusion encoding including spatial filtering and channel reconstruction on the i-th depth encoding result and the (j-1)-th fusion encoding result to obtain the j-th fusion encoding result, where 2≤j≤M, M represents the number of attention residual layers; and determine the M-th fusion encoding result as the (i+1)-th segmentation encoding result.

[0024] In some embodiments, the attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer; the aforementioned fusion encoding module is further configured to invoke the spatial attention layer of the first attention residual layer to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain the spatial filtering result of the first attention residual layer; invoke the channel attention layer of the first attention residual layer to perform channel reconstruction on the spatial filtering result to obtain the channel reconstruction result of the first attention residual layer; and invoke the residual connection layer of the first attention residual layer to perform fusion processing on the channel reconstruction result and the feature extraction result to obtain the first fusion encoding result.

[0025] In some embodiments, the attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer; when the number of attention residual layers is one, the above-mentioned fusion encoding module is further configured to call the spatial attention layer to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain a spatial filtering result; call the channel attention layer to perform channel recombination on the spatial filtering result to obtain a channel recombination result; and call the residual connection layer to perform fusion processing on the channel recombination result and the feature extraction result to obtain the (i+1)-th segmentation encoding result.

[0026] In some embodiments, the above-mentioned fusion encoding module is further configured to perform convolution processing on the feature extraction result to obtain a first convolution processing result of the feature extraction result; perform dot product on the first convolution processing result and the i-th depth encoding result to obtain a first dot product result; and determine the first dot product result as the spatial filtering result.

[0027] In some embodiments, the above-mentioned fusion encoding module is further configured to perform convolution processing on the spatial filtering result to obtain a second convolution processing result of the spatial filtering result; perform convolution processing on the first convolution processing result to obtain a third convolution processing result; perform normalization processing on the third convolution processing result to obtain a normalization processing result; perform dot product on the normalization processing result and the second convolution processing result to obtain a second dot product result; and determine the second dot product result as the channel recombination result.

[0028] This application provides an electronic device, including:

[0029] Memory, used to store executable instructions;

[0030] The processor, when executing executable instructions stored in the memory, implements the semantic segmentation method for images provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the semantic segmentation method for images provided in this application.

[0032] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image semantic segmentation method described in this application.

[0033] The embodiments of this application have the following beneficial effects:

[0034] By encoding the depth image corresponding to the image to be segmented, a depth encoding result is obtained. The depth encoding result and the image to be segmented are then fused and encoded. Based on the obtained target encoding result, semantic segmentation is performed on the image to be segmented, yielding the corresponding semantic segmentation result. In this way, the depth image of the image to be segmented contains the distance information between each pixel in the image to be segmented and the camera. Iterative fusion encoding, including spatial filtering and channel reorganization, is performed on the depth encoding result and the image to be segmented. Feature selection in both spatial and channel dimensions can fully extract the semantic information of the image. Therefore, semantic segmentation based on the obtained target encoding result can fully exploit the complementarity and interdependence between the semantics of the image, effectively improving the accuracy of semantic segmentation. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the semantic segmentation system architecture for images provided in the embodiments of this application;

[0036] Figure 2 This is a schematic diagram of the structure of the semantic segmentation device for images provided in the embodiments of this application;

[0037] Figures 3A to 3D This is a flowchart illustrating the semantic segmentation method for images provided in an embodiment of this application;

[0038] Figures 4A to 4D This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in the embodiments of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0043] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0044] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0045] 1) Semantic segmentation: Semantic segmentation is a fundamental task in computer vision. In semantic segmentation, we need to divide the visual input into different semantically interpretable categories. Semantic interpretability means that the classification categories are meaningful in the real world.

[0046] 2) Depth Map: Each pixel value in the depth map of the original image represents the distance between the corresponding pixel in the original image and the camera.

[0047] 3) Convolutional Neural Networks (CNNs): These are a class of feedforward neural networks (FNNs) that incorporate convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning. CNNs possess representation learning capabilities, enabling them to perform shift-invariant classification of input images according to their hierarchical structure.

[0048] 4) Downsampling: For a sequence of sample values, samples are taken at intervals of several samples to obtain a new sequence, which is the downsampling of the original sequence. In fact, downsampling is extraction. The main purposes of reducing the size of an image (also called subsampled or downsampled) are twofold: to make the image fit the size of the display area; and to generate a thumbnail of the corresponding image.

[0049] 5) Pooling: Pooling is used to improve important feature information, compress features, reduce computation, and alleviate overfitting.

[0050] 6) Self-Attention: This is an attention mechanism that focuses on the correlation between different parts of the entire input.

[0051] 7) Feedforward Neural Network (FFN): This is an artificial neural network where neurons are arranged in layers. Each neuron is connected only to neurons in the previous layer. Each layer receives the output of the previous layer and outputs to the next layer; there is no feedback between layers. Feedforward neural networks include perceptron networks, backpropagation (BP) networks, and radial basis function (RBF) networks. The perceptron network is the simplest feedforward network, primarily used for pattern classification, but also in pattern-based learning control and multimodal control. Perceptron networks can be divided into single-layer and multi-layer perceptron networks. A backpropagation (BP) network is a feedforward network that uses the backpropagation learning algorithm for connection weight adjustment. Unlike the perceptron, the BP network uses a sigmoid function for neuron transformation, resulting in a continuous output between 0 and 1, allowing for arbitrary nonlinear mapping from input to output. An radial basis function (RBF) network is a feedforward network whose hidden layer neurons are composed of RBF neurons. An RBF neuron is a neuron whose transformation function is RBF (Radial Basis Function). A typical RBF network consists of three layers: an input layer, one or more RBF layers (hidden layers) composed of RBF neurons, and an output layer composed of linear neurons.

[0052] During the implementation of the embodiments of this application, the applicant discovered the following problems with the related technology:

[0053] In related technologies, semantic segmentation typically uses a single image to be segmented as input to a segmentation network. The network processes the image to obtain the corresponding semantic segmentation result. However, this approach often fails to fully exploit the complementary and interdependent information within the image to be segmented, resulting in poor accuracy in semantic segmentation.

[0054] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for semantic segmentation of images. These methods can fully exploit the complementarity and interdependence between semantic elements in images, effectively improving the accuracy of semantic segmentation. The following describes exemplary applications of the semantic segmentation device provided in this application. The device can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server.

[0055] See Figure 1 , Figure 1This is a schematic diagram of the architecture of the semantic segmentation system 100 for images provided in this application embodiment. To realize the application scenario of defect detection, the terminal (terminal 400 is shown as an example) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0056] Terminal 400 is used by users to access client 410, and is displayed on graphical interface 410-1 (graphical interface 410-1 is shown as an example). Terminal 400 and server 200 are interconnected via wired or wireless network.

[0057] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, in-vehicle terminal, etc., but is not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0058] In some embodiments, server 200 obtains the image to be segmented and the corresponding depth image from terminal 400, and sends the image to be segmented and the corresponding depth image to terminal 400. Based on the image to be segmented and the corresponding depth image, terminal 400 determines the semantic segmentation result of each object in the image to be segmented, and sends the semantic segmentation result to server 200.

[0059] In other embodiments, terminal 400 acquires the image to be segmented and the corresponding depth image, and sends the image to be segmented and the corresponding depth image to server 200. Server 200 determines the semantic segmentation result of each object in the image to be segmented based on the image to be segmented and the corresponding depth image, and sends the semantic segmentation result to terminal 400.

[0060] In other embodiments, the vehicle-mounted camera captures driving footage, the vehicle-mounted terminal receives the driving footage captured by the vehicle-mounted camera, and acquires the corresponding depth image of the driving footage. Based on the driving footage and the corresponding depth image, the semantic segmentation result of each object in the driving footage is determined, and based on the semantic segmentation result of each object in the driving footage, the driving status of the vehicle is determined.

[0061] In other embodiments, the embodiments of this application can be implemented with the aid of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0062] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0063] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a server 200 for the semantic segmentation method of images provided in this application embodiment. Figure 2 The server 200 shown includes at least one processor 210, memory 250, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0064] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0066] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0067] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0068] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0069] The network communication module 252 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0070] In some embodiments, the semantic segmentation apparatus for images provided in this application can be implemented in software. Figure 2 A semantic segmentation apparatus 255 for an image stored in memory 250 is shown. This apparatus can be software in the form of programs or plugins, and includes the following software modules: an acquisition module 2551, an encoding module 2552, a fusion encoding module 2553, and a semantic segmentation module 2554. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0071] In other embodiments, the image semantic segmentation apparatus provided in this application can be implemented in hardware. As an example, the image semantic segmentation apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image semantic segmentation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] The semantic segmentation method for images provided in this application will be described in conjunction with exemplary applications and implementations of the servers or terminals provided in the embodiments of this application.

[0073] In some embodiments, see Figure 4A , Figure 4A This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application embodiment. An image 42 to be segmented, including at least two objects, and a corresponding depth image 41 are acquired. The depth image 41 is encoded to obtain a depth encoding result. The depth encoding result and the image 42 to be segmented are fused and encoded to obtain a target encoding result. Based on the target encoding result, semantic segmentation is performed on the image 42 to be segmented to obtain semantic segmentation results 43 corresponding to each object.

[0074] See Figure 3A , Figure 3A This is a flowchart illustrating the semantic segmentation method for images provided in this application, which will be combined with... Figure 3A Steps 101 to 104 are described below. The execution subject of steps 101 to 104 can be a server or a terminal. The following description will take the execution subject as a server as an example.

[0075] In step 101, the server obtains an image to be segmented that includes at least two objects, and a depth image corresponding to the image to be segmented.

[0076] In some embodiments, the object can be a physical object in the image to be segmented, such as a tree, a cup, or a person. Each pixel value in the depth image corresponding to the image to be segmented represents the distance between the corresponding pixel in the original image and the camera.

[0077] As an example, see Figure 4B , Figure 4B This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in the embodiments of this application. Figure 4B The image to be segmented shown is an aerial image. Figure 4B The aerial image shown includes at least two objects, which can be objects in the image to be segmented, such as trees, houses, roads, etc.

[0078] In step 102, the depth image is encoded to obtain the depth encoding result.

[0079] In some embodiments, encoding, also known as image encoding, refers to the technique of representing an image or the information contained in an image with a smaller number of bits while meeting certain quality requirements (signal-to-noise ratio requirements or subjective evaluation scores).

[0080] In some embodiments, see Figure 3B , Figure 3B This is a schematic flowchart of the semantic segmentation method for images provided in the embodiments of this application. Figure 3B Step 102 shown can be achieved through steps 1021 to 1023.

[0081] In step 1021, the depth image is downsampled to obtain the downsampled result of the depth image.

[0082] In some embodiments, downsampling refers to sampling a sequence of samples at intervals of several samples, resulting in a new sequence that is a downsampled version of the original sequence. In essence, downsampling is extraction. The main purposes of reducing an image size (or downsampling) are twofold: to make the image fit the size of the display area; and to generate a thumbnail of the corresponding image.

[0083] In some embodiments, step 1021 above can also be implemented by calling a downsampling layer to perform downsampling processing on the depth image and obtaining the downsampling processing result of the depth image.

[0084] As an example, see Figure 4A , Figure 4A This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application embodiment. The downsampling layer 51 is invoked to downsample the depth image 41, resulting in the downsampled depth image.

[0085] In step 1022, the downsampling result of the depth image is pooled to obtain the pooling result of the depth image.

[0086] In some embodiments, pooling is used to improve important feature information, compress features, reduce computational load, and alleviate overfitting.

[0087] In some embodiments, step 1022 above can also be implemented by calling a pooling layer to perform pooling processing on the downsampling result of the depth image, thereby obtaining the pooling processing result of the depth image.

[0088] As an example, see Figure 4A The pooling layer 53 is called to perform pooling processing on the downsampling result of the depth image, and the pooling result of the depth image is obtained.

[0089] In step 1023, at least two depth coding networks are invoked to iteratively encode the pooling results of the depth image to obtain the depth coding results.

[0090] In some embodiments, a deep coding network is used to encode the pooling result of a depth image. The deep coding network includes at least two coding layers with identical structures, which are used to encode the input data.

[0091] As an example, depth coding networks 54, 55, 56, and 57 are invoked to iteratively encode the pooling results of the depth image to obtain the depth coding result.

[0092] In some embodiments, see Figure 3C , Figure 3C This is a schematic flowchart of the semantic segmentation method for images provided in the embodiments of this application. Figure 3C Step 1023 shown can be achieved through the following steps 10231 to 10233.

[0093] In step 10231, the first depth coding network is invoked to encode the pooling result of the depth image to obtain the first depth coding result.

[0094] As an example, see Figure 4A The first depth coding network 54 is called to encode the pooling result of the depth image to obtain the first depth coding result.

[0095] In step 10232, the (i+1)th depth coding network is invoked to encode the (i)th depth coding result, thereby obtaining the (i+1)th depth coding result.

[0096] In some embodiments, 1≤i≤N-1, where N represents the number of deep coding networks, the size of the (i+1)th deep coding network is smaller than the size of the ith deep coding network, and the structure of the (i+1)th deep coding network is the same as the structure of the ith deep coding network.

[0097] As an example, when i = 1, see [reference needed]. Figure 4A The second depth coding network 55 is called to encode the result of the first depth coding to obtain the result of the second depth coding.

[0098] As an example, when i = 2, see [reference needed]. Figure 4A The third depth coding network 56 is called to encode the second depth coding result to obtain the third depth coding result.

[0099] As an example, see when i = 3. Figure 4A The fourth depth coding network 57 is called to encode the third depth coding result to obtain the fourth depth coding result.

[0100] In some embodiments, the deep coding network includes at least two coding layers with the same structure, and the at least two coding layers with the same structure include a first coding layer and a second coding layer; the above step 10232 can be implemented in the following way: calling the first coding layer to encode the i-th depth coding result to obtain a first coding result; calling the second coding layer to encode the first coding result to obtain a second coding result; and determining the second coding result as the i+1-th depth coding result.

[0101] As an example, see Figure 4A The first encoding layer 541 is called to encode the encoding result at depth i to obtain the first encoding result; the second encoding layer 542 is called to encode the first encoding result to obtain the second encoding result; the second encoding result is determined as the encoding result at depth i+1.

[0102] As an example, see Figure 4A The first encoding layer 541 is called to encode the encoding result at depth i to obtain the first encoding result; the second encoding layer 542 is called to encode the first encoding result to obtain the second encoding result; the third encoding layer 543 is called to encode the second encoding result to obtain the third encoding result; and the third encoding result is determined as the encoding result at depth i+1.

[0103] In step 10233, the Nth depth encoding result is determined as the depth encoding result.

[0104] Where 1≤i≤N-1, N represents the number of deep coding networks, and the size of the (i+1)th deep coding network is smaller than the size of the ith deep coding network.

[0105] As an example, see Figure 4A The fourth deep coding result is determined as the deep coding result, and the number of deep coding networks 51 to 57 is 4.

[0106] Thus, by encoding the depth image of the image to be segmented through a deep coding network including at least two structurally identical coding layers, the multimodal features of the depth image can be explicitly aggregated at multiple different scales. This facilitates subsequent fusion coding based on the multimodal features of the depth image and the image to be segmented, allowing for full exploitation of the complementarity and interdependence between the semantics of the images and effectively improving the accuracy of semantic segmentation.

[0107] In step 103, at least two segmentation coding networks are invoked to perform iterative fusion coding, including spatial filtering and channel reorganization, on the depth coding result and the image to be segmented, to obtain the target coding result.

[0108] In some embodiments, spatial filtering is used to perform feature filtering on the image to be segmented in the spatial dimension, and channel reconstruction is used to perform feature filtering on the image to be segmented in the channel dimension. The spatial filtering is implemented through a spatial attention layer in the segmentation coding network, and the channel reconstruction is implemented through a channel attention layer in the segmentation coding network.

[0109] In some embodiments, fusion coding is used to encode at least two different inputs to obtain an encoded result. Fusion coding can be implemented using a segmented coding network, which includes at least two residual layers and at least one attention residual layer.

[0110] In some embodiments, see Figure 3B , Figure 3B This is a flowchart illustrating the semantic segmentation method for images provided in an embodiment of this application. Figure 3B Step 103 shown can be achieved through steps 1031 to 1033.

[0111] In step 1031, the image to be segmented is downsampled to obtain the downsampled result of the image to be segmented.

[0112] In some embodiments, downsampling refers to sampling a sequence of samples at intervals of several samples, resulting in a new sequence that is a downsampled version of the original sequence. In essence, downsampling is extraction. The main purposes of reducing an image size (or downsampling) are twofold: to make the image fit the size of the display area; and to generate a thumbnail of the corresponding image.

[0113] In some embodiments, step 1031 above can also be implemented by calling a downsampling layer to perform downsampling processing on the image to be segmented, thereby obtaining the downsampling processing result of the image to be segmented.

[0114] As an example, see Figure 4A , Figure 4A This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application embodiment. The downsampling layer 521 is invoked to perform downsampling processing on the image 42 to be segmented, obtaining the downsampling result of the image to be segmented.

[0115] In step 1032, the downsampling result of the image to be segmented is pooled to obtain the pooling result of the image to be segmented.

[0116] In some embodiments, pooling is used to improve important feature information, compress features, reduce computational load, and alleviate overfitting.

[0117] In some embodiments, step 1032 above can also be implemented by calling a pooling layer to perform pooling processing on the downsampling processing result of the image to be segmented, thereby obtaining the pooling processing result of the image to be segmented.

[0118] As an example, see Figure 4A The pooling layer 522 is called to perform pooling processing on the downsampling processing result of the image 42 to be segmented, so as to obtain the pooling processing result of the image 42 to be segmented.

[0119] In step 1033, at least two segmentation coding networks are invoked to perform iterative fusion coding, including spatial filtering and channel reorganization, on the pooling results and depth coding results of the image to be segmented, to obtain the target coding result.

[0120] In some embodiments, the segmentation coding network is used to perform fusion coding on at least two inputs to obtain the target coding result.

[0121] In some embodiments, the depth coding results include i depth coding results, 1 ≤ i ≤ N-1, where N represents the number of depth coding networks encoding the depth image; see also Figure 3D , Figure 3D This is a schematic flowchart of the semantic segmentation method for images provided in the embodiments of this application. Figure 3D Step 1033 shown can be achieved by following steps 10331 to 10333.

[0122] In step 10331, the first segmentation coding network is invoked to perform fusion coding, including spatial filtering and channel reconstruction, on the pooling processing result of the image to be segmented and the first depth coding result, to obtain the first segmentation coding result.

[0123] As an example, see Figure 4A The first segmentation coding network 58 is called to fuse the pooling result of the image to be segmented and the first depth coding result to obtain the first segmentation coding result.

[0124] In step 10332, the (i+1)th segmentation coding network is invoked to perform fusion coding, including spatial filtering and channel reorganization, on the (i+1)th segmentation coding result and the (i)th depth coding result to obtain the (i+1)th segmentation coding result.

[0125] As an example, see Figure 4A When i=1, the second segmentation coding network 59 is called to fuse the first segmentation coding result and the first depth coding result to obtain the second segmentation coding result.

[0126] As an example, see Figure 4AWhen i=2, the third segmentation coding network 60 is called to fuse the second segmentation coding result and the second depth coding result to obtain the third segmentation coding result.

[0127] As an example, see Figure 4A When i=3, the fourth segmentation coding network 61 is called to fuse the third segmentation coding result and the third depth coding result to obtain the fourth segmentation coding result.

[0128] In some embodiments, the segmentation coding network includes at least two residual layers and at least one attention residual layer; the above step 10332 can be implemented in the following way: calling at least two residual layers to extract features from the i-th segmentation coding result to obtain the feature extraction result of the i-th segmentation coding result; calling at least one attention residual layer to perform fusion coding including spatial filtering and channel reorganization on the feature extraction result and the i-th depth coding result to obtain the (i+1)-th segmentation coding result.

[0129] In some embodiments, a residual layer refers to a residual network, which is characterized by its ease of optimization and the ability to improve accuracy by increasing its depth considerably. The residual blocks within the residual network utilize skip connections to mitigate the vanishing gradient problem that arises when increasing depth in deep neural networks.

[0130] As an example, see Figure 4A At least two residual layers (residual layer 581 and residual layer 582) are invoked to extract features from the i-th segmentation encoding result, and the feature extraction result of the i-th segmentation encoding result is obtained; at least one attention residual layer (attention residual layer 583) is invoked to fuse the feature extraction result and the i-th depth encoding result, and the i+1-th segmentation encoding result is obtained.

[0131] Thus, feature extraction of the segmentation encoding result is performed through at least two residual layers. Since the residual layers can improve accuracy by increasing the depth considerably, the gradient vanishing problem caused by increasing the depth in deep neural networks is effectively alleviated.

[0132] In some embodiments, when the number of attention residual layers is at least two, the above-mentioned invocation of at least one attention residual layer to perform fusion encoding including spatial filtering and channel reconstruction on the feature extraction result and the i-th depth encoding result to obtain the (i+1)-th segmentation encoding result can be implemented as follows: invoking the first attention residual layer to perform fusion encoding including spatial filtering and channel reconstruction on the feature extraction result and the i-th depth encoding result to obtain the first fusion encoding result; invoking the j-th attention residual layer to perform fusion encoding including spatial filtering and channel reconstruction on the i-th depth encoding result and the (j-1)-th fusion encoding result to obtain the j-th fusion encoding result, where 2≤j≤M, M represents the number of attention residual layers; and determining the M-th fusion encoding result as the (i+1)-th segmentation encoding result.

[0133] As an example, see Figure 4A When j=2, the first attention residual layer 591 is called to perform fusion encoding on the feature extraction result and the i-th depth encoding result to obtain the first fusion encoding result; the second attention residual layer 592 is called to perform fusion encoding on the first depth encoding result and the first fusion encoding result to obtain the second fusion encoding result.

[0134] In some embodiments, the attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer. The above-mentioned invocation of the first attention residual layer to fuse the feature extraction result and the i-th depth encoding result to obtain the first fused encoding result can be achieved in the following way: Invoking the spatial attention layer of the first attention residual layer to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain the spatial filtering result of the first attention residual layer; invoking the channel attention layer of the first attention residual layer to perform channel reconstruction on the spatial filtering result to obtain the channel reconstruction result of the first attention residual layer; and invoking the residual connection layer of the first attention residual layer to fuse the channel reconstruction result and the feature extraction result to obtain the first fused encoding result.

[0135] In some embodiments, the spatial attention layer is used to perform spatial filtering on the feature extraction result and the i-th depth encoding result; the channel attention layer is used to perform channel reconstruction on the spatial filtering result; and the residual connection layer is used to perform fusion processing on the channel reconstruction result and the feature extraction result, wherein the fusion processing can be the summation of the channel reconstruction result and the feature extraction result.

[0136] As an example, see Figure 4C , Figure 4CThis is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application embodiment. The spatial attention layer of the first attention residual layer is invoked to perform spatial filtering on the feature extraction result and the i-th depth encoding result, obtaining the spatial filtering result of the first attention residual layer; the channel attention layer of the first attention residual layer is invoked to perform channel reconstruction on the spatial filtering result, obtaining the channel reconstruction result of the first attention residual layer; the residual connection layer of the first attention residual layer is invoked to fuse the channel reconstruction result and the feature extraction result, obtaining the first fused encoding result.

[0137] In other embodiments, the attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer. When there is only one attention residual layer, the above-mentioned invocation of at least one attention residual layer to fuse the feature extraction result and the i-th depth encoding result to obtain the (i+1)-th segmentation encoding result can be achieved as follows: the spatial attention layer is invoked to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain a spatial filtering result; the channel attention layer is invoked to perform channel recombination on the spatial filtering result to obtain a channel recombination result; and the residual connection layer is invoked to fuse the channel recombination result and the feature extraction result to obtain the (i+1)-th segmentation encoding result.

[0138] As an example, see Figure 4C Call the spatial attention layer to process the feature extraction result F. C and the encoding result F at depth i D Spatial filtering is performed to obtain spatial filtering result F. N ; Invoke the channel attention layer to filter the spatial result F N Channel reconfiguration is performed to obtain channel reconfiguration result F. Y ; Call the residual connection layer to reassemble the channel result F Y and feature extraction results F C The fusion process is performed to obtain the (i+1)th segmentation encoding result.

[0139] In some embodiments, the above-mentioned invocation of the spatial attention layer to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain the spatial filtering result can be achieved in the following way: performing convolution processing on the feature extraction result to obtain the first convolution processing result of the feature extraction result; performing dot product on the first convolution processing result and the i-th depth encoding result to obtain the first dot product result; and determining the first dot product result as the spatial filtering result.

[0140] In some embodiments, the dot product, also known as the scalar product in mathematics, is a binary operation that takes two vectors over a real number R and returns a real scalar. It is the standard inner product of Euclidean space.

[0141] In some embodiments, convolution processing in functional analysis, where convolution, spin, or convolution is a mathematical operation that generates a third function from two functions f and g, is essentially a special integral transformation that represents the integral of the product of the overlapping function values ​​of functions f and g after flipping and translation, with respect to the overlap length. If one of the functions involved in the convolution is considered as an indicator function of the interval, convolution can also be viewed as a generalization of "moving average".

[0142] As an example, see Figure 4C For the feature extraction result F C Perform convolution processing to obtain the first convolution processing result of the feature extraction result; combine the first convolution processing result with the encoding result F at depth i. D Perform a dot product to obtain the first dot product result; determine the first dot product result as the spatial filtering result F. N .

[0143] In some embodiments, the above-mentioned invocation of the channel attention layer to reconstruct the channel based on the spatial filtering result can be achieved in the following way: performing convolution processing on the spatial filtering result to obtain a second convolution processing result; performing convolution processing on the first convolution processing result to obtain a third convolution processing result; performing normalization processing on the third convolution processing result to obtain a normalized processing result; performing dot product on the normalized processing result and the second convolution processing result to obtain a second dot product result; and determining the second dot product result as the channel reconstruction result.

[0144] In some embodiments, normalization is a way to simplify calculations, that is, to transform a dimensional expression into a dimensionless expression, which becomes a scalar.

[0145] As an example, see Figure 4C For the spatial filtering results F N Perform convolution processing to obtain the second convolution result F of the spatial filtering result. Q The result of the first convolution is then convolved to obtain the result of the third convolution, F. P The result of the third convolution, F P Normalization is performed to obtain the normalized result F. X The result F after normalization X The result of the second convolution F Q Perform a dot product to obtain the second dot product result; determine the second dot product result as the channel recombination result F. Y .

[0146] In some embodiments, the sizes of the first convolution result and the third convolution result are different.

[0147] Thus, by designing an attention residual layer and using depth images as spatial weights, features of the image to be segmented are filtered in the spatial dimension. The channel attention layer of the attention residual layer adaptively selects features in the channel dimension by applying weights to different channels. Then, the residual connection layer fuses the selected features with the features of the input image to be segmented, effectively enhancing the accuracy of semantic segmentation. This can explicitly increase the differences between different categories of objects, fully utilize the complementary information in multimodal features for semantic segmentation, and effectively improve the accuracy of semantic segmentation.

[0148] In step 10333, the Nth segmentation encoding result is determined as the target encoding result.

[0149] The size of the (i+1)th segmentation coding network is smaller than that of the ith segmentation coding network, and the structure of the (i+1)th segmentation coding network is the same as that of the ith segmentation coding network.

[0150] In some embodiments, see Figure 4D , Figure 4D This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application embodiment. Network 50 includes a deep coding network and a segmentation coding network, both of which have 5 layers. The first layer of the deep coding network of network 50 outputs an image of size [size missing]. Where H represents the height of the image to be segmented, and W represents the width of the image to be segmented. The first layer of the depth coding network of Network 50 is used to downsample the depth image; the image size output by the second layer of the depth coding network of Network 50 is... The second layer of the deep coding network of Network 50 includes a pooling layer and at least two coding layers of size 64; the third layer of the deep coding network of Network 50 outputs an image of size [missing information]. The third layer of the deep coding network of Network 50 includes at least two coding layers of size 128; the fourth layer of the deep coding network of Network 50 outputs an image of size [missing information]. The fourth layer of the deep coding network of Network 50 includes at least two coding layers of size 256; the fifth layer of the deep coding network of Network 50 outputs an image of size [missing information]. The fourth layer of the deep coding network of Network 50 includes at least two coding layers of size 512.

[0151] In some embodiments, see Figure 4D The size of the output image of the first layer of the segmentation coding network of Network 50 is The first layer of the segmentation coding network of Network 50 is used for downsampling; the output image size of the second layer of the segmentation coding network of Network 50 is... The second layer of the segmentation coding network of Network 50 includes a pooling layer, two residual layers, and one attention residual layer; the output image size of the third layer of the segmentation coding network of Network 50 is... The third layer of the segmentation coding network of Network 50 consists of three residual layers and one attention residual layer; the size of the output image of the fourth layer of the segmentation coding network of Network 50 is... The fourth layer of the Network 50 segmentation coding network consists of five residual layers and one attention residual layer; the output image size of the fifth layer of the Network 50 segmentation coding network is... The fifth layer of the segmentation coding network of Network 50 consists of two residual layers and one attention residual layer.

[0152] In some embodiments, see Figure 4D Network 101 includes a deep coding network and a segmentation coding network. Both the deep coding network and the segmentation coding network have 5 layers. The first layer of the deep coding network of Network 101 outputs an image of size [size missing]. Where H represents the height of the image to be segmented, and W represents the width of the image to be segmented, the first layer of the depth coding network of network 101 is used to downsample the depth image; the image size output by the second layer of the depth coding network of network 101 is... The second layer of the deep coding network of Network 101 includes a pooling layer and at least two coding layers of size 64; the third layer of the deep coding network of Network 101 outputs an image of size [missing information]. The third layer of the deep coding network 101 includes at least two coding layers of size 128; the fourth layer of the deep coding network 101 outputs an image of size [missing information]. The fourth layer of the deep coding network 101 includes at least two coding layers of size 256; the fifth layer of the deep coding network 101 outputs an image of size [missing information]. The fourth layer of the deep coding network of Network 101 includes at least two coding layers of size 512.

[0153] In some embodiments, see Figure 4D The size of the output image of the first layer of the segmentation coding network of Network 101 is The first layer of the segmentation coding network of Network 101 is used for downsampling; the output image size of the second layer of the segmentation coding network of Network 101 is... The second layer of the segmentation coding network of Network 101 includes a pooling layer, two residual layers, and one attention residual layer; the output image size of the third layer of the segmentation coding network of Network 101 is... The third layer of the segmentation coding network 101 consists of three residual layers and one attention residual layer; the output image size of the fourth layer of the segmentation coding network 101 is... The fourth layer of the Network 101 segmentation coding network consists of 22 residual layers and one attention residual layer; the output image size of the fifth layer of the Network 101 segmentation coding network is... The fifth layer of the segmentation coding network of Network 101 consists of two residual layers and one attention residual layer.

[0154] In step 104, based on the target encoding result, semantic segmentation is performed on the image to be segmented to obtain the semantic segmentation result for each object.

[0155] In some embodiments, step 104 above can be implemented by decoding the target encoding result and determining the decoding result as the semantic segmentation result of each object in the image to be segmented.

[0156] In some embodiments, the decoding process can be implemented by invoking the context model to decode the target encoding result and obtain the decoding result.

[0157] In some embodiments, the context module includes a pyramid pooling module, a spatial pyramid pooling module, and a self-attention module. The pyramid pooling module (PPM) effectively increases the receptive field and improves the utilization efficiency of global information through multiple pooling operations. The spatial pyramid pooling module (ASPP) produces a fixed-size output regardless of the input image size. It uses multiple pooling windows and can use different scales of the same image as input to obtain pooled features of the same length. The self-attention module is used to find the degree of correlation between each vector and other vectors (including itself).

[0158] Thus, by encoding the depth image corresponding to the image to be segmented, a depth encoding result is obtained. The depth encoding result and the image to be segmented are then fused and encoded. Based on the obtained target encoding result, semantic segmentation is performed on the image to be segmented, yielding the corresponding semantic segmentation result. In this way, the depth image of the image to be segmented contains the distance information between each pixel in the image to be segmented and the camera. Iterative fusion encoding, including spatial filtering and channel reorganization, is performed on the depth encoding result and the image to be segmented. Feature selection in both the spatial and channel dimensions of the image to be segmented can fully extract the semantic information of the image. Therefore, semantic segmentation based on the obtained target encoding result can fully exploit the complementarity and interdependence between the semantics of the image, effectively improving the accuracy of semantic segmentation.

[0159] The following will describe an exemplary application of the embodiments of this application in a real semantic segmentation application scenario.

[0160] Image processing tasks in the field of computer vision can be broadly categorized into three types: image classification, object detection, and image segmentation. Image segmentation includes semantic segmentation and instance segmentation. The semantic segmentation method for images provided in this application belongs to the semantic segmentation task within image processing. Semantic segmentation is the most information-rich task among the aforementioned image processing tasks. It assigns high-level semantic labels to each pixel, essentially classifying each pixel in the image. High-level semantic labels refer to various object categories (e.g., people, animals, cars, etc.) and background categories (e.g., sky, grassland, etc.). Semantic segmentation tasks place high demands on both classification and localization accuracy: on the one hand, it requires accurate localization of object contours; on the other hand, it requires precise classification of the regions within the contours to effectively segment specific objects from the background. Therefore, maintaining a balance between localization accuracy and classification accuracy is an important issue in semantic segmentation. Generally speaking, to improve classification accuracy, it is necessary to increase the receptive field of deep networks so as to integrate more information. However, expanding the receptive field of deep networks will lead to a large loss of image details, which is not conducive to boundary localization. Therefore, one of the goals of improving semantic segmentation is to integrate more global information without losing local image details.

[0161] The semantic segmentation method for images provided in this application significantly improves the performance of semantic segmentation by using multimodal image data and depth image data, since there is complementary information between the image to be segmented and the corresponding depth image. The progressive attention fusion network provided in this application can explicitly aggregate multimodal image data and depth image data, enabling more accurate differentiation of multi-scale and easily confused objects in the image to be segmented, thus effectively improving the accuracy of semantic segmentation.

[0162] In some embodiments, see Figure 4A The progressive attention fusion network provided in this application includes an encoding network and a decoding network. The encoding network uses a progressive fusion encoder, which has two branches. It extracts features from both the image to be segmented and the depth image, and fuses the depth image into the branch corresponding to the image to be segmented at each scale to enhance the encoding network's ability to distinguish objects of different sizes. This application adds a dual attention residual layer to the branch corresponding to the image to be segmented to explicitly enhance the feature differences between different categories of targets and strengthen the ability to distinguish confused objects.

[0163] In some embodiments, see Figure 4B , Figure 4B This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application embodiment. When the image to be segmented is an aerial image, after semantic segmentation of the image to be segmented using the semantic segmentation method provided in this application embodiment, the segmentation result is as follows: Figure 4B As shown.

[0164] In some embodiments, the semantic segmentation method for images provided in this application uses a dual-stream progressive fusion encoder module, which decodes the image to obtain multi-scale image features. Then, a dual-attention residual network effectively aggregates the deep features into branches of the multi-scale image to be segmented. The decoding part includes a pyramid pooling module, a spatial pyramid pooling module, and a self-attention module to capture contextual information for semantic segmentation.

[0165] In some embodiments, see Figure 4D , Figure 4D This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application. To improve the model's ability to recognize multi-scale targets in the image to be segmented, this application designs a dual-stream progressive fusion encoder for extracting and fusing multi-modal features at multiple scales. Specifically, the image to be segmented and the depth image are feature-encoded using ResNet-50 / 101 and ResNet-34, respectively. Figure 4D As shown, considering that the dilated network can reduce the loss of spatial resolution while maintaining a large receptive field, dilated convolutions are used in the last two layers of the original dilated network and the downsampling operation is removed, so that the spatial size is 1 / 8 of the input image. At the same time, in order to fuse depth features into the image branches to be segmented at various scales, a dual attention residual module is proposed to replace the last residual block of each stage in the RGB branch.

[0166] In some embodiments, see Figure 4C , Figure 4C This is a schematic diagram illustrating the principle of the semantic segmentation method for images provided in this application embodiment. For example... Figure 4C The results of the dual attention residual module shown are used to fuse multimodal features at various scales. Given the features of the image to be segmented and the features of the depth image, the spatial attention layer and the channel attention layer are executed sequentially. The output of the channel attention layer is then fused with the input of the dual attention residual module to output the output of the dual attention residual module.

[0167] In some embodiments, see Figure 4C For the spatial attention layer, since depth images can reflect the semantic distribution of categories to some extent (e.g., buildings always have a large height, while permeable surfaces have almost zero height), depth features F are used. DAs spatial attention weights, they can adaptively preserve features of taller targets and filter features of shorter targets. Specifically, 1×1 and 3×3 convolutional processing can be used to obtain the dimensionality-reduced features F. M The expression for the output of the spatial attention layer can be:

[0168] F N =F M ⊙F D (1)

[0169] Among them, F N F represents the output of the spatial attention layer. M Characterizing the features after dimensionality reduction, F D Represents depth features, and ⊙ represents the dot product.

[0170] In some embodiments, see Figure 4C For the channel attention layer, channel rescaling is performed on the feature map F. N Select effective feature channels, specifically, use 1×1 convolution on feature map F. M and F N The convolutional feature map F is obtained. P and F Q This is used to ensure channel consistency between the two feature maps. Subsequently, global average pooling and fully connected layers adaptively acquire the weights for each channel, and then a normalization operation yields the channel weight vector. Finally, the expression for the output of the channel attention can be:

[0171]

[0172] Where i, j, and k represent the feature indices of the channel's length, height, and width dimensions, respectively.

[0173] In some embodiments, see Figure 4C The spatial attention layer uses deep features as spatial weights to filter RGB features in the spatial dimension, while the channel attention layer selects features in the channel dimension through adaptive weights for different channels. Finally, the residual connection layer fuses the selected features with the features of the input image to be segmented, enhancing the features of taller objects. Therefore, the dual attention residual module proposed in this application can explicitly increase the feature differences between categories and fully utilize the complementary information in multimodal features for semantic segmentation.

[0174] In some embodiments, see Figure 4A The following methods can be used to... Figure 4A The network shown is trained using a weighted cross-entropy loss function, taking into account the significant difference in the number of pixels for each category in the training set. Figure 4D The network shown can be expressed as follows: The expression for the weighted cross-entropy loss function is:

[0175]

[0176] Among them, y i p represents the actual data of the current pixel i. i Characterization Figure 4A The normalized prediction results of the network shown are w i The weight represents the weight of the i-th class of pixels, and N represents the total number of pixel classes in the image to be segmented.

[0177] The expression for the weight of the i-th class of pixels can be:

[0178]

[0179] In this embodiment of the application, by using two 1080Ti GPUs... Figure 4A The network shown is trained using a GPU with 11GB of memory per GPU. Stochastic gradient descent is used for optimization, with weight decay of 0.0001, momentum of 0.9, and an initial learning rate of 0.01. A specific learning rate policy is employed for learning rate updates, with the learning rate updated after each iteration. In addition, to improve batch size, this solution also performs batch normalization statistics on samples from all GPUs.

[0180] The semantic segmentation method for images provided in this application can accurately and efficiently achieve semantic segmentation of images in aerial photography scenarios using optical data and radar data as fusion input conditions. It can fully explore the complementarity and interdependence between semantics in images and effectively improve the accuracy of semantic segmentation.

[0181] It is understood that in the embodiments of this application, data such as images to be segmented are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0182] The following description continues to illustrate the exemplary structure of the image semantic segmentation device 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules in the semantic segmentation device 255 storing the image in the memory 240 may include: an acquisition module 2551, used to acquire an image to be segmented including at least two objects, and a depth image corresponding to the image to be segmented; an encoding module 2552, used to encode the depth image to obtain a depth encoding result; a fusion encoding module 2553, used to call at least two segmentation encoding networks to fuse the depth encoding result and the image to be segmented to obtain a target encoding result, wherein the spatial filtering is used to perform feature filtering on the image to be segmented in the spatial dimension, and the channel reassembly is used to perform feature filtering on the image to be segmented in the channel dimension; and a semantic segmentation module 2554, used to perform semantic segmentation on the image to be segmented based on the target encoding result to obtain semantic segmentation results corresponding to each object.

[0183] In some embodiments, the encoding module 2552 is further configured to perform downsampling processing on the depth image to obtain a downsampling processing result of the depth image; perform pooling processing on the downsampling processing result of the depth image to obtain a pooling processing result of the depth image; and invoke at least two depth encoding networks to perform iterative encoding processing on the pooling processing result of the depth image to obtain a depth encoding result.

[0184] In some embodiments, the encoding module 2552 is further configured to call the first depth encoding network to encode the pooling result of the depth image to obtain the first depth encoding result; call the (i+1)th depth encoding network to encode the i-th depth encoding result to obtain the (i+1)th depth encoding result; and determine the N-th depth encoding result as the depth encoding result; wherein 1≤i≤N-1, N represents the number of depth encoding networks, and the size of the (i+1)th depth encoding network is smaller than the size of the i-th depth encoding network.

[0185] In some embodiments, the deep coding network includes at least two coding layers with the same structure, and the at least two coding layers with the same structure include a first coding layer and a second coding layer; the coding module 2552 is further configured to call the first coding layer to encode the i-th depth coding result to obtain a first coding result; call the second coding layer to encode the first coding result to obtain a second coding result; and determine the second coding result as the i+1-th depth coding result.

[0186] In some embodiments, the fusion coding module 2553 is further configured to perform downsampling processing on the image to be segmented to obtain a downsampling processing result of the image to be segmented; perform pooling processing on the downsampling processing result of the image to be segmented to obtain a pooling processing result of the image to be segmented; and invoke at least two segmentation coding networks to perform iterative fusion coding including spatial filtering and channel reorganization on the pooling processing result and depth coding result of the image to be segmented to obtain a target coding result.

[0187] In some embodiments, the depth coding result includes i depth coding results, 1≤i≤N-1, where N represents the number of depth coding networks encoding the depth image; the aforementioned fusion coding module 2553 is further configured to call the first segmentation coding network to perform fusion coding including spatial filtering and channel reconstruction on the pooling processing result of the image to be segmented and the first depth coding result to obtain the first segmentation coding result; call the (i+1)th segmentation coding network to perform fusion coding including spatial filtering and channel reconstruction on the i-th segmentation coding result and the i-th depth coding result to obtain the (i+1)th segmentation coding result; and determine the N-th segmentation coding result as the target coding result; wherein, the size of the (i+1)th segmentation coding network is smaller than the size of the i-th segmentation coding network.

[0188] In some embodiments, the segmentation coding network includes at least two residual layers and at least one attention residual layer; the fusion coding module 2553 is further configured to call at least two residual layers to extract features from the i-th segmentation coding result to obtain the feature extraction result of the i-th segmentation coding result; and call at least one attention residual layer to perform fusion coding including spatial filtering and channel reorganization on the feature extraction result and the i-th depth coding result to obtain the (i+1)-th segmentation coding result.

[0189] In some embodiments, when the number of attention residual layers is at least two, the fusion encoding module 2553 is further configured to call the first attention residual layer to perform fusion encoding including spatial filtering and channel reorganization on the feature extraction result and the i-th depth encoding result to obtain the first fusion encoding result; call the j-th attention residual layer to perform fusion encoding including spatial filtering and channel reorganization on the i-th depth encoding result and the (j-1)-th fusion encoding result to obtain the j-th fusion encoding result, where 2≤j≤M, M represents the number of attention residual layers; and determine the M-th fusion encoding result as the (i+1)-th segmentation encoding result.

[0190] In some embodiments, the attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer; the aforementioned fusion coding module 2553 is further configured to call the spatial attention layer of the first attention residual layer to perform spatial filtering on the feature extraction result and the i-th depth coding result to obtain the spatial filtering result of the first attention residual layer; call the channel attention layer of the first attention residual layer to perform channel recombination on the spatial filtering result to obtain the channel recombination result of the first attention residual layer; and call the residual connection layer of the first attention residual layer to perform fusion processing on the channel recombination result and the feature extraction result to obtain the first fusion coding result.

[0191] In some embodiments, the attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer; when the number of attention residual layers is one, the above-mentioned fusion encoding module 2553 is further used to call the spatial attention layer to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain a spatial filtering result; call the channel attention layer to perform channel recombination on the spatial filtering result to obtain a channel recombination result; and call the residual connection layer to perform fusion processing on the channel recombination result and the feature extraction result to obtain the (i+1)-th segmentation encoding result.

[0192] In some embodiments, the above-mentioned fusion encoding module 2553 is further configured to perform convolution processing on the feature extraction result to obtain a first convolution processing result of the feature extraction result; perform dot product on the first convolution processing result and the encoding result at the i-th depth to obtain a first dot product result; and determine the first dot product result as the spatial filtering result.

[0193] In some embodiments, the above-mentioned fusion encoding module 2553 is further configured to perform convolution processing on the spatial filtering result to obtain a second convolution processing result of the spatial filtering result; perform convolution processing on the first convolution processing result to obtain a third convolution processing result; perform normalization processing on the third convolution processing result to obtain a normalization processing result; perform dot product on the normalization processing result and the second convolution processing result to obtain a second dot product result; and determine the second dot product result as the channel recombination result.

[0194] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image semantic segmentation method described in this application.

[0195] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the image semantic segmentation method provided in this application. For example, ... Figure 3A The semantic segmentation method for the image is shown.

[0196] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEP ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0197] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0198] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., files that store one or more modules, subroutines, or code sections).

[0199] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0200] In summary, the embodiments of this application have the following beneficial effects:

[0201] (1) By encoding the depth image corresponding to the image to be segmented, a depth encoding result is obtained; the depth encoding result and the image to be segmented are fused and encoded; based on the obtained target encoding result, semantic segmentation is performed on the image to be segmented to obtain the corresponding semantic segmentation result. In this way, the depth image of the image to be segmented contains the distance information between each pixel in the image to be segmented and the camera. By performing iterative fusion encoding, including spatial filtering and channel reorganization, on the depth encoding result and the image to be segmented, feature selection is performed on the image to be segmented in both spatial and channel dimensions, which can fully explore the semantic information of the image. Therefore, based on the obtained target encoding result, semantic segmentation is performed on the image to be segmented, which can fully explore the complementarity and interdependence between the semantics of the image and effectively improve the accuracy of semantic segmentation.

[0202] (2) By encoding the depth image of the image to be segmented through a deep coding network including at least two coding layers with the same structure, the multimodal features of the depth image can be explicitly aggregated at multiple different scales, which facilitates the subsequent fusion coding based on the multimodal features of the depth image and the image to be segmented. This allows for the full exploitation of the complementarity and interdependence between the semantics of the images, effectively improving the accuracy of semantic segmentation.

[0203] (3) Feature extraction is performed on the segmentation encoding result through at least two residual layers. Since the residual layer can improve the accuracy by increasing the depth, it effectively alleviates the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0204] (4) By designing an attention residual layer, depth images are used as spatial weights to filter features of the image to be segmented in the spatial dimension. The channel attention layer of the attention residual layer selects features in the channel dimension by adaptive weights for different channels. Then, the selected features are fused with the features of the input image to be segmented through the residual connection layer, which effectively enhances the accuracy of semantic segmentation. It can explicitly increase the differences between different categories of objects and make full use of the complementary information in multimodal features for semantic segmentation, thus effectively enhancing the accuracy of semantic segmentation.

[0205] (5) The progressive attention fusion network provided in this application includes an encoding network part and a decoding network part. The encoding network part adopts a progressive fusion encoder, which includes two branches. It extracts features from the image to be segmented and the depth image at the same time, and fuses the depth image into the branch corresponding to the image to be segmented at each scale to enhance the ability of the encoding network to distinguish objects of different sizes. In this application embodiment, by adding a dual attention residual layer in the branch corresponding to the image to be segmented, the feature differences between different categories of targets are explicitly enhanced, and the ability to distinguish confused objects is strengthened.

[0206] (6) The semantic segmentation method of the image provided in this application embodiment can accurately and efficiently realize the semantic segmentation of the image under the condition of using optical data and radar data as fusion input in the aerial image scene. It can fully explore the complementarity and interdependence between the semantics of the image and effectively improve the accuracy of semantic segmentation.

[0207] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A semantic segmentation method for images, characterized in that, The method includes: Obtain an image to be segmented that includes at least two objects, and a depth image corresponding to the image to be segmented; The depth image is encoded to obtain the depth encoding results output by N depth encoding networks; The first segmentation coding network among N segmentation coding networks is invoked to perform fusion coding, including spatial filtering and channel reconstruction, on the pooling processing result and the first depth coding result of the image to be segmented, to obtain the first segmentation coding result; The process involves calling at least one attention residual layer in the (i+1)th segmentation coding network among N segmentation coding networks to perform spatial filtering on the feature extraction result and the i-th depth coding result of the i-th segmentation coding result, obtaining a spatial filtering result, performing channel reconstruction on the spatial filtering result, obtaining a channel reconstruction result, and fusing the channel reconstruction result and the feature extraction result to obtain the (i+1)-th segmentation coding result; the i-th segmentation coding result is the output of the i-th segmentation coding network. The attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer; The Nth segmentation encoding result is determined as the target encoding result; The spatial filtering is used to perform feature filtering on the image to be segmented in the spatial dimension, and the channel recombination is used to perform feature filtering on the image to be segmented in the channel dimension. Based on the target encoding result, semantic segmentation is performed on the image to be segmented to obtain the semantic segmentation result corresponding to each of the objects.

2. The method according to claim 1, characterized in that, The process of encoding the depth image to obtain a depth encoding result includes: The depth image is downsampled to obtain the downsampled result of the depth image; The downsampling result of the depth image is subjected to pooling processing to obtain the pooling result of the depth image; At least two depth coding networks are invoked to iteratively encode the pooling result of the depth image to obtain the depth coding result.

3. The method according to claim 2, characterized in that, The process of calling at least two depth coding networks to iteratively encode the pooling result of the depth image to obtain the depth coding result includes: The first depth coding network is invoked to encode the pooling result of the depth image to obtain the first depth coding result; Call the (i+1)th depth coding network to encode the i-th depth coding result to obtain the (i+1)th depth coding result; The Nth depth encoding result is determined as the depth encoding result; in, N represents the number of deep coding networks, and the size of the (i+1)th deep coding network is smaller than the size of the ith deep coding network.

4. The method according to claim 3, characterized in that, The deep coding network includes at least two structurally identical coding layers, which include a first coding layer and a second coding layer; the step of calling the (i+1)th deep coding network to encode the i-th deep coding result to obtain the (i+1)-th deep coding result includes: The first encoding layer is invoked to encode the encoding result at the i-th depth, thereby obtaining the first encoding result; The second encoding layer is invoked to encode the first encoding result, thereby obtaining the second encoding result. The second encoding result is determined as the (i+1)th depth encoding result.

5. The method according to claim 1, characterized in that, The method further includes: The image to be segmented is downsampled to obtain the downsampled result of the image to be segmented. The downsampling result of the image to be segmented is subjected to pooling processing to obtain the pooling processing result of the image to be segmented.

6. The method according to claim 1, characterized in that, The size of the (i+1)th segmentation coding network is smaller than the size of the ith segmentation coding network.

7. The method according to claim 1, characterized in that, The segmentation coding network includes at least two residual layers and at least one attention residual layer; Before calling at least one attention residual layer in the (i+1)th segmentation coding network among the N segmentation coding networks to perform spatial filtering on the feature extraction result and the i-th depth coding result, the method further includes: The at least two residual layers are invoked to extract features from the i-th segmentation coding result, thereby obtaining the feature extraction result of the i-th segmentation coding result.

8. The method according to claim 1, characterized in that, When the number of attention residual layers is at least two, the process of calling at least one attention residual layer in the (i+1)th segmentation coding network of the N segmentation coding networks, performing spatial filtering on the feature extraction result and the i-th depth coding result of the i-th segmentation coding result to obtain a spatial filtering result, performing channel reconstruction on the spatial filtering result to obtain a channel reconstruction result, and fusing the channel reconstruction result and the feature extraction result to obtain the (i+1)th segmentation coding result, includes: The spatial attention layer of the first attention residual layer is invoked to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain the spatial filtering result of the first attention residual layer. The channel attention layer of the first attention residual layer is invoked to perform channel reconstruction on the spatial filtering result to obtain the channel reconstruction result of the first attention residual layer. The residual connection layer of the first attention residual layer is invoked to perform fusion processing on the channel reconstruction result and the feature extraction result to obtain the first fusion encoding result. The j-th attention residual layer is invoked to perform fusion encoding, including spatial filtering and channel reconstruction, on the i-th depth encoding result and the (j-1)-th fusion encoding result, to obtain the j-th fusion encoding result, wherein... M represents the number of attention residual layers; The Mth fusion encoding result is determined as the (i+1)th segmentation encoding result.

9. The method according to claim 1, characterized in that, When the number of attention residual layers is one, the process of calling at least one attention residual layer in the (i+1)th segmentation coding network of the N segmentation coding networks, performing spatial filtering on the feature extraction result and the i-th depth coding result of the i-th segmentation coding result to obtain a spatial filtering result, performing channel reconstruction on the spatial filtering result to obtain a channel reconstruction result, and fusing the channel reconstruction result and the feature extraction result to obtain the (i+1)th segmentation coding result, includes: The spatial attention layer is invoked to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain the spatial filtering result; The channel attention layer is invoked to perform channel reorganization on the spatial filtering result, resulting in a channel reorganization result. The residual connection layer is invoked to fuse the channel reconstruction result and the feature extraction result to obtain the (i+1)th segmentation encoding result.

10. The method according to claim 9, characterized in that, The step of calling the spatial attention layer to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain the spatial filtering result includes: The feature extraction results are subjected to convolution processing to obtain the first convolution processing result of the feature extraction results; The first convolutional processing result and the i-th depth encoding result are multiplied by a dot product to obtain the first dot product result; The first dot product result is determined as the spatial filtering result.

11. A semantic segmentation apparatus for images, characterized in that, The device includes: The acquisition module is used to acquire an image to be segmented, which includes at least two objects, and a depth image corresponding to the image to be segmented; The encoding module is used to encode the depth image to obtain the depth encoding results output by N depth encoding networks respectively; The fusion encoding module is used to call the first segmentation encoding network out of N segmentation encoding networks, and perform fusion encoding, including spatial filtering and channel reconstruction, on the pooling processing result and the first depth encoding result of the image to be segmented, to obtain the first segmentation encoding result; call at least one attention residual layer of the (i+1)th segmentation encoding network out of the N segmentation encoding networks, and perform spatial filtering on the feature extraction result and the i-th depth encoding result of the i-th segmentation encoding result, to obtain the spatial filtering result; perform channel reconstruction on the spatial filtering result, to obtain the channel reconstruction result; and fuse the channel reconstruction result and the feature extraction result to obtain the (i+1)th segmentation encoding result; the i-th segmentation encoding result is the output of the i-th segmentation encoding network. The attention residual layer includes a spatial attention layer, a channel attention layer, and a residual connection layer; the Nth segmentation encoding result is determined as the target encoding result; wherein, the spatial filtering is used to perform feature filtering on the image to be segmented in the spatial dimension, and the channel recombination is used to perform feature filtering on the image to be segmented in the channel dimension; The semantic segmentation module is used to perform semantic segmentation on the image to be segmented based on the target encoding result, so as to obtain the semantic segmentation result corresponding to each of the objects.

12. The apparatus according to claim 11, characterized in that, The encoding module is further configured to perform downsampling processing on the depth image to obtain a downsampling processing result of the depth image; and to perform pooling processing on the downsampling processing result of the depth image to obtain a pooling processing result of the depth image. At least two depth coding networks are invoked to iteratively encode the pooling result of the depth image to obtain the depth coding result.

13. The apparatus according to claim 12, characterized in that, The encoding module is also used to call the first depth encoding network to encode the pooling result of the depth image to obtain the first depth encoding result; Call the (i+1)th depth coding network to encode the i-th depth coding result to obtain the (i+1)th depth coding result; The Nth depth encoding result is determined as the depth encoding result; wherein... N represents the number of deep coding networks, and the size of the (i+1)th deep coding network is smaller than the size of the ith deep coding network.

14. The apparatus according to claim 12, characterized in that, The deep coding network includes at least two coding layers with identical structures, including a first coding layer and a second coding layer. The coding module is further configured to call the first coding layer to encode the i-th deep coding result to obtain a first coding result. The second encoding layer is invoked to encode the first encoding result, thereby obtaining the second encoding result. The second encoding result is determined as the (i+1)th depth encoding result.

15. The apparatus according to claim 11, characterized in that, The fusion encoding module is further configured to perform downsampling processing on the image to be segmented to obtain a downsampling processing result of the image to be segmented; and to perform pooling processing on the downsampling processing result of the image to be segmented to obtain a pooling processing result of the image to be segmented. The at least two segmentation coding networks are invoked to perform iterative fusion coding, including spatial filtering and channel recombination, on the pooling processing result and the depth coding result of the image to be segmented, to obtain the target coding result.

16. The apparatus according to claim 11, characterized in that, The size of the (i+1)th segmentation coding network is smaller than the size of the ith segmentation coding network.

17. The apparatus according to claim 11, characterized in that, The segmentation coding network includes at least two residual layers and at least one attention residual layer; the fusion coding module is further configured to call the at least two residual layers to extract features from the i-th segmentation coding result, thereby obtaining the feature extraction result of the i-th segmentation coding result; The at least one attention residual layer is invoked to perform fusion encoding, including spatial filtering and channel reorganization, on the feature extraction result and the i-th depth encoding result to obtain the (i+1)-th segmentation encoding result.

18. The apparatus according to claim 11, characterized in that, When the number of attention residual layers is at least two, the above-mentioned fusion coding module is further used to call the first attention residual layer to perform fusion coding including spatial filtering and channel reorganization on the feature extraction result and the i-th depth coding result to obtain the first fusion coding result; The j-th attention residual layer is invoked to perform fusion encoding, including spatial filtering and channel reorganization, on the i-th depth encoding result and the (j-1)-th fusion encoding result to obtain the j-th fusion encoding result, where, M represents the number of attention residual layers; The Mth fusion encoding result is determined as the (i+1)th segmentation encoding result.

19. The apparatus according to claim 11, characterized in that, The fusion encoding module is further configured to call the spatial attention layer of the first attention residual layer to perform spatial filtering on the feature extraction result and the i-th depth encoding result to obtain the spatial filtering result of the first attention residual layer; The channel attention layer of the first attention residual layer is invoked to perform channel reconstruction on the spatial filtering result, thereby obtaining the channel reconstruction result of the first attention residual layer; The residual connection layer of the first attention residual layer is invoked to fuse the channel reconstruction result and the feature extraction result to obtain the first fusion encoding result.

20. The apparatus according to claim 11, characterized in that, The fusion encoding module is further configured to perform convolution processing on the feature extraction result to obtain a first convolution processing result of the feature extraction result; and to perform a dot product between the first convolution processing result and the encoding result at the i-th depth to obtain a first dot product result; The first dot product result is determined as the spatial filtering result.

21. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions or computer programs stored in the memory, implements the semantic segmentation method for images according to any one of claims 1 to 10.

22. A computer-readable storage medium storing executable instructions or a computer program, characterized in that, When the executable instructions are executed by the processor, they implement the semantic segmentation method for images according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the semantic segmentation method for images according to any one of claims 1 to 10.

Images