Semantic segmentation method and device based on depth information position coding guidance
By embedding depth information into features and utilizing multi-scale attention mechanisms and feedforward neural networks, the high computational cost and reliance on prior knowledge of RGB-Depth fusion technology are addressed, resulting in more efficient image segmentation performance and model flexibility.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIMEI UNIV
- Filing Date
- 2024-07-05
- Publication Date
- 2026-06-02
AI Technical Summary
Existing Transformer-based RGB-Depth fusion techniques are computationally expensive and rely on prior knowledge, limiting the model's versatility and flexibility.
By employing a multi-scale attention mechanism and a feedforward neural network, deep information is embedded in the features to reduce computational costs and improve image segmentation performance.
It reduces computational costs, improves image segmentation performance, and enhances the model's versatility and flexibility.
Smart Images
Figure CN118691823B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image segmentation technology, and in particular to a semantic segmentation method guided by depth information location encoding, a computer-readable storage medium, a computer device, and a semantic segmentation apparatus guided by depth information location encoding. Background Technology
[0002] In recent years, the integration of depth information has proven to play a significant role in improving the accuracy of segmentation tasks, especially in scenes with blurred object boundaries or insufficient texture information; RGB images contribute rich color and texture information, while depth images provide important geometric and spatial information; combining data from these two modalities can theoretically enable semantic segmentation models to more accurately resolve scenes.
[0003] In the field of RGB-D data fusion research, the HHA encoding strategy proposed by Gupta et al. greatly enhanced the representational power of depth features by converting depth information into three dimensions: horizontal disparity, height difference, and normal angle. Subsequently, research focused on exploring how to more effectively fuse data from these two modalities. Some researchers explored multi-scale fusion strategies, aiming to incorporate depth information into different network layers to seek the optimal balance between local and global context. Other studies focused on optimizing network structures, such as introducing complex branches specifically for processing depth data, or exploring the intrinsic relationship between depth and color information through end-to-end training. Although convolutional neural networks have made significant progress in RGB-Depth fusion tasks, their local information processing characteristics limit their ability to achieve a global understanding of complex scenes. The introduction of Transformers, leveraging their global modeling capabilities through self-attention mechanisms, provides a new approach to overcome this limitation, driving a new trend in Transformer-based RGB-Depth fusion research.
[0004] RGB-Depth fusion techniques based on Transformers effectively integrate color and depth information through their self-attention mechanism, providing a more in-depth and nuanced information fusion strategy compared to traditional convolutional neural networks. These methods aim to uncover the complementary characteristics between different modalities and explore how self-attention mechanisms can enhance cognitive abilities regarding complex scenes. To more accurately capture the complementarity between color and depth information, some studies have proposed a two-branch Transformer architecture. In this architecture, RGB and depth information are processed and encoded through two independent Transformer paths, and then integrated through a specially designed fusion module. This approach has achieved significant success in promoting information fusion and transmission between modalities. However, despite the superior performance of the two-branch Transformer architecture in handling RGB-Depth fusion tasks, this type of model still faces some challenges. First, the computational cost of this architecture is relatively high, mainly because it requires processing information from two modalities separately and performing complex calculations during the fusion process. Second, the design of the fusion strategy relies heavily on prior knowledge and empirical judgment, which limits the model's generality and flexibility. Therefore, the huge computational burden and excessive reliance on depth information brought about by the dual Transformer structure are problems that urgently need to be solved by current technology. Summary of the Invention
[0005] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a semantic segmentation method guided by depth information location encoding. By embedding depth information into features, computational costs can be reduced, thereby improving image segmentation performance.
[0006] A second objective of this invention is to provide a computer-readable storage medium.
[0007] The third objective of this invention is to provide a computer device.
[0008] The fourth objective of this invention is to propose a semantic segmentation device guided by depth information location encoding.
[0009] To achieve the above objectives, the first embodiment of the present invention proposes a semantic segmentation method based on depth information location encoding guidance, comprising the following steps: acquiring an image to be processed; inputting the image to be processed into a pre-trained spatial depth perception auxiliary network to extract multi-scale features, and obtaining a corresponding depth feature map based on the multi-scale features; generating a multi-scale feature vector based on the multi-scale features and the corresponding depth feature map; processing the multi-scale feature vector using a multi-scale attention mechanism and a feedforward neural network to obtain the final features; fusing the final features, and performing semantic segmentation on the fused features to obtain a semantic segmentation map.
[0010] According to the semantic segmentation method based on depth information location encoding guided by embodiments of the present invention, firstly, an image to be processed is acquired; then, the image to be processed is input into a pre-trained spatial depth perception auxiliary network to extract multi-scale features, and corresponding depth feature maps are obtained based on the multi-scale features; next, multi-scale feature vectors are generated based on the multi-scale features and the corresponding depth feature maps; then, a multi-scale attention mechanism and a feedforward neural network are used to process the multi-scale feature vectors to obtain the final features; finally, the final features are fused, and semantic segmentation is performed on the fused features to obtain a semantic segmentation map; thus, by embedding depth information into the features, the computational cost can be reduced, thereby improving image segmentation performance.
[0011] Furthermore, the semantic segmentation method based on depth information location encoding guided by the above embodiments of the present invention may also have the following additional technical features:
[0012] Furthermore, the final layer feature output depth feature map is optimized according to the following loss function:
[0013]
[0014] Among them, Smooth L1 The calculation method for (ΔD(i,j)) is as follows:
[0015]
[0016] Where ΔD(i,j) represents the difference between the predicted depth and the true depth; h represents the height of the image to be processed; w represents the width of the image to be processed; i represents the row index; and j represents the column index.
[0017] Furthermore, the multi-scale feature vector is processed using a multi-scale attention mechanism and a feedforward neural network to obtain the final features, including: processing the multi-scale feature vector using a multi-scale attention mechanism to obtain attention features; and further processing the attention features using a feedforward neural network to obtain the final features.
[0018] Furthermore, the multi-scale feature vector is processed according to the following formula:
[0019]
[0020] Where Q, K, and V represent the query vector, key vector, and value vector, respectively. The dimension of the key vector is represented by Q, K, and V; Q, K, and V in the formula are all multi-scale feature vectors of the input.
[0021] Furthermore, the attention features are further processed according to the following formula:
[0022] FFN(x) = max(0, xW1+b1)W2+b2
[0023] Where W1, W2, b1, b2 represent network parameters.
[0024] Furthermore, the final features are fused, and the fused features are semantically segmented to obtain a semantic segmentation map. This includes: integrating the final features using a fully connected layer and upsampling them to obtain upsampled features; combining all upsampled features into a new feature vector to obtain fused features; and performing semantic segmentation on the fused features to obtain a semantic segmentation map.
[0025] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing a semantic segmentation program guided by depth information location encoding, which, when executed by a processor, implements the semantic segmentation method guided by depth information location encoding as described above.
[0026] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the semantic segmentation method guided by depth information location encoding as described above.
[0027] To achieve the above objectives, a fourth aspect of the present invention proposes a semantic segmentation device based on depth information location encoding guidance, comprising: an acquisition module for acquiring an image to be processed; a feature extraction module for inputting the image to be processed into a pre-trained spatial depth perception auxiliary network to extract multi-scale features and obtain a corresponding depth feature map based on the multi-scale features; a feature embedding module for generating a multi-scale feature vector based on the multi-scale features and the corresponding depth feature map; a feature processing module for processing the multi-scale feature vector using a multi-scale attention mechanism and a feedforward neural network to obtain final features; and a semantic segmentation module for fusing the final features and performing semantic segmentation on the fused features to obtain a semantic segmentation map.
[0028] According to an embodiment of the present invention, a semantic segmentation device based on depth information location encoding guidance acquires an image to be processed through an acquisition module; a feature extraction module inputs the image to be processed into a pre-trained spatial depth perception auxiliary network to extract multi-scale features and obtains corresponding depth feature maps based on the multi-scale features; a feature embedding module generates multi-scale feature vectors based on the multi-scale features and the corresponding depth feature maps; a feature processing module processes the multi-scale feature vectors using a multi-scale attention mechanism and a feedforward neural network to obtain final features; and a semantic segmentation module performs fusion processing on the final features and performs semantic segmentation on the processed fused features to obtain a semantic segmentation map. Thus, by embedding depth information into the features, computational costs can be reduced, thereby improving image segmentation performance. Attached Figure Description
[0029] Figure 1 This is a flowchart illustrating a semantic segmentation method based on depth information location encoding according to an embodiment of the present invention.
[0030] Figure 2 This is a schematic diagram of a network structure according to an embodiment of the present invention;
[0031] Figure 3 The present invention provides the semantic segmentation prediction results on the KITTI360 dataset.
[0032] Figure 4 The present invention provides the semantic segmentation prediction results on the NYU DepthV2 dataset.
[0033] Figure 5 The deep feature maps predicted in the KITTI360 and NYU DepthV2 datasets according to the present invention;
[0034] Figure 6 This is a block diagram of a semantic segmentation device based on depth information location encoding guided by an embodiment of the present invention. Detailed Implementation
[0035] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0036] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.
[0037] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.
[0038] Figure 1 This is a flowchart illustrating a semantic segmentation method based on depth information location encoding guided by an embodiment of the present invention, as shown below. Figure 1 As shown, this semantic segmentation method guided by depth information location encoding includes the following steps:
[0039] S101, Obtain the image to be processed.
[0040] It should be noted that the image to be processed is an RGB image.
[0041] S102, the image to be processed is input into a pre-trained spatial depth perception auxiliary network to extract multi-scale features and obtain the corresponding depth feature map based on the multi-scale features.
[0042] As a specific example, the predicted depth feature map is as follows: Figure 5 As shown, firstly, a spatial depth-aware auxiliary network is constructed and trained, and the network model parameters are saved for subsequent processing; then, the image to be processed (with dimensions of H×W×3) is processed, that is, multi-scale features are extracted through the spatial depth-aware auxiliary network, with a specific resolution equal to a fraction of the original image to be processed. Finally, the final layer features are used to output a depth feature map, which is optimized using SmoothL1 loss in conjunction with the real depth map; the specific loss function is defined as:
[0043]
[0044] Among them, Smooth L1The calculation method for (ΔD(i,j)) is as follows:
[0045]
[0046] Where ΔD(i,j) represents the difference between the predicted depth and the true depth; h represents the height of the image to be processed; w represents the width of the image to be processed; i represents the row index; and j represents the column index.
[0047] S103 generates multi-scale feature vectors based on multi-scale features and corresponding depth feature maps.
[0048] In other words, the output deep feature map is embedded into multi-scale features to generate multi-scale feature vectors. That is, the multi-scale features are added to the deep feature map to obtain multi-scale feature vectors, which facilitates the model's position encoding when calculating attention to avoid being deviated from the real scene.
[0049] S104 employs a multi-scale attention mechanism and a feedforward neural network to process multi-scale feature vectors to obtain the final features.
[0050] As a specific implementation, firstly, a multi-scale attention mechanism is applied to the multi-scale feature vectors, and its calculation formula is as follows:
[0051]
[0052] Where Q, K, and V are the query vector, key vector, and value vector, respectively. Q is the dimension of the key vector, where Q, K, and V are all input features.
[0053] Then, the attention features are further processed through a feedforward neural network, and the calculation formula is as follows:
[0054] FFN(x) = max(0, xW1+b1)W2+b2
[0055] Where W1, W2, b1, b2 are network parameters.
[0056] S105, perform fusion processing on the final features, and then perform semantic segmentation on the fused features to obtain a semantic segmentation map.
[0057] As a specific embodiment, the network structure diagram is as follows: Figure 2 As shown, firstly, the final features are integrated through a fully connected layer and upsampled to a size of [size missing]. Then, all feature levels are combined into a new feature vector with dimension . Finally, the new feature vectors are processed by the segmentation head to output the final semantic segmentation map.
[0058] It should be noted that the semantic segmentation method guided by depth information location encoding can generate high-quality semantic segmentation masks and provides a foundation for research on the robustness and security of 2D segmentation. The semantic segmentation prediction results for different datasets are shown in the following figures. Figure 3-4 As shown.
[0059] In summary, the semantic segmentation method based on depth information location encoding guided by embodiments of the present invention first acquires the image to be processed; then, the image to be processed is input into a pre-trained spatial depth perception auxiliary network to extract multi-scale features, and the corresponding depth feature map is obtained based on the multi-scale features; next, a multi-scale feature vector is generated based on the multi-scale features and the corresponding depth feature map; then, a multi-scale attention mechanism and a feedforward neural network are used to process the multi-scale feature vector to obtain the final features; finally, the final features are fused, and semantic segmentation is performed on the fused features to obtain a semantic segmentation map; thus, by embedding depth information into the features, the computational cost can be reduced, thereby improving image segmentation performance.
[0060] To implement the above embodiments, this invention proposes a computer-readable storage medium storing a semantic segmentation program guided by depth information location encoding. When the semantic segmentation program guided by depth information location encoding is executed by a processor, it implements the semantic segmentation method guided by depth information location encoding as described above.
[0061] According to embodiments of the present invention, a computer-readable storage medium stores a semantic segmentation program guided by depth information location encoding, so that when a processor executes the semantic segmentation program guided by depth information location encoding, it implements the semantic segmentation method guided by depth information location encoding as described above. Thus, by embedding depth information in features, computational costs can be reduced, thereby improving image segmentation performance.
[0062] To implement the above embodiments, this invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the semantic segmentation method guided by depth information location encoding as described above.
[0063] According to an embodiment of the present invention, a computer device stores a semantic segmentation program guided by depth information location encoding in a memory, so that when the processor executes the semantic segmentation program guided by depth information location encoding, it implements the semantic segmentation method guided by depth information location encoding as described above. Thus, by embedding depth information in the features, the computational cost can be reduced, thereby improving the image segmentation performance.
[0064] To implement the above embodiments, this invention also proposes a semantic segmentation device guided by depth information location encoding, such as... Figure 6 As shown, the semantic segmentation device based on depth information location encoding includes: an acquisition module 10, a feature extraction module 20, a feature embedding module 30, a feature processing module 40, and a semantic segmentation module 50.
[0065] The acquisition module 10 is used to acquire the image to be processed; the feature extraction module 20 is used to input the image to be processed into a pre-trained spatial depth perception auxiliary network to extract multi-scale features and obtain the corresponding depth feature map based on the multi-scale features; the feature embedding module 30 is used to generate multi-scale feature vectors based on the multi-scale features and the corresponding depth feature map; the feature processing module 40 is used to process the multi-scale feature vectors using a multi-scale attention mechanism and a feedforward neural network to obtain the final features; and the semantic segmentation module 50 is used to perform fusion processing on the final features and perform semantic segmentation on the processed fused features to obtain a semantic segmentation map.
[0066] As an example, the final layer feature output depth feature map is optimized according to the following loss function:
[0067]
[0068] Among them, Smooth L1 The calculation method for (ΔD(i,j)) is as follows:
[0069]
[0070] Where ΔD(i,j) represents the difference between the predicted depth and the true depth; h represents the height of the image to be processed; w represents the width of the image to be processed; i represents the row index; and j represents the column index.
[0071] As an example, a multi-scale attention mechanism and a feedforward neural network are used to process multi-scale feature vectors to obtain final features, including: processing multi-scale feature vectors using a multi-scale attention mechanism to obtain attention features; and further processing the attention features through a feedforward neural network to obtain final features.
[0072] As an example, the multi-scale feature vector is processed according to the following formula:
[0073]
[0074] Where Q, K, and V represent the query vector, key vector, and value vector, respectively. The dimension of the key vector is represented by Q, K, and V; Q, K, and V in the formula are all multi-scale feature vectors of the input.
[0075] As an example, the attention features are further processed according to the following formula:
[0076] FFN(x) = max(0, xW1+b1)W2+b2
[0077] Where W1, W2, b1, b2 represent network parameters.
[0078] As an example, the final features are fused and the fused features are semantically segmented to obtain a semantic segmentation map. This includes: integrating the final features using a fully connected layer and upsampling them to obtain upsampled features; combining all upsampled features into a new feature vector to obtain fused features; and performing semantic segmentation on the fused features to obtain a semantic segmentation map.
[0079] It should be noted that the above description and examples of the semantic segmentation method guided by depth information location encoding are also applicable to the semantic segmentation device guided by depth information location encoding in this embodiment, and will not be repeated here.
[0080] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0084] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0085] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0086] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
[0087] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0088] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0089] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.
[0090] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0091] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A semantic segmentation method based on depth information location coding guidance, characterized in that, Includes the following steps: Obtain the image to be processed; The image to be processed is input into a pre-trained spatial depth perception auxiliary network to extract multi-scale features, and the corresponding depth feature map is obtained based on the multi-scale features. Generate a multi-scale feature vector based on the multi-scale features and the corresponding depth feature map; The multi-scale feature vector is processed using a multi-scale attention mechanism and a feedforward neural network to obtain the final features; The final features are fused, and the fused features are semantically segmented to obtain a semantic segmentation map. The process involves inputting the image to be processed into a pre-trained spatial depth-aware auxiliary network to extract multi-scale features, and obtaining the corresponding depth feature map based on the multi-scale features, including: Construct and train a spatial depth perception auxiliary network, and save the network model parameters for subsequent processing; The image to be processed is processed, that is, multi-scale features are extracted through a spatial depth perception auxiliary network, and the specific resolution is 1 / 4 of the original image to be processed ; The final layer features output a depth feature map, which is optimized using the SmoothL1 loss function in conjunction with the true depth map; the specific loss function is defined as follows: wherein is calculated as follows: wherein, represents the difference between the predicted depth and the real depth; represents the height of the image to be processed, represents the width of the image to be processed, represents the row index, represents the column index. 2.The method of semantic segmentation guided by depth information position coding according to claim 1, wherein, The multi-scale feature vector is processed using a multi-scale attention mechanism and a feedforward neural network to obtain the final features, including: A multi-scale attention mechanism is used to process the multi-scale feature vector to obtain attention features; The attention features are further processed by a feedforward neural network to obtain the final features. 3.The method of semantic segmentation guided by depth information position coding according to claim 2, wherein, The multi-scale feature vector is processed according to the following formula: wherein, respectively represent a query vector, a key vector and a value vector, denotes the dimension of the key vector; in the formula, are all input multi-scale feature vectors.
4. The semantic segmentation method based on depth information location encoding as described in claim 3, characterized in that, The attention features are further processed according to the following formula: in, This represents network parameters.
5. The semantic segmentation method based on depth information location encoding as described in claim 1, characterized in that, The final features are fused, and the fused features are then semantically segmented to obtain a semantic segmentation map, including: The final features are integrated using a fully connected layer and then upsampled to obtain upsampled features. All upsampled features are combined into a new feature vector to obtain fused features; The fused features are semantically segmented to obtain a semantic segmentation map.
6. A computer-readable storage medium, characterized in that, It stores a semantic segmentation program guided by depth information location encoding, which, when executed by a processor, implements the semantic segmentation method guided by depth information location encoding as described in any one of claims 1-5.
7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the semantic segmentation method based on depth information location encoding as described in any one of claims 1-5.
8. A semantic segmentation device guided by depth information location encoding, characterized in that, include: The acquisition module is used to acquire the image to be processed; The feature extraction module is used to input the image to be processed into a pre-trained spatial depth perception auxiliary network to extract multi-scale features and obtain the corresponding depth feature map based on the multi-scale features. The feature embedding module is used to generate multi-scale feature vectors based on the multi-scale features and the corresponding depth feature maps; The feature processing module is used to process the multi-scale feature vector using a multi-scale attention mechanism and a feedforward neural network to obtain the final features; The semantic segmentation module is used to perform fusion processing on the final features and semantic segmentation on the fused features to obtain a semantic segmentation map. The feature extraction module is also used to construct and train a spatial depth-aware auxiliary network, and save the network model parameters for subsequent processing; it processes the image to be processed, that is, it extracts multi-scale features through the spatial depth-aware auxiliary network, with a specific resolution of [missing information - likely a resolution value] of the original image to be processed. The final layer features are used to output a depth feature map, which is optimized using SmoothL1 loss in conjunction with the true depth map; the specific loss function is defined as follows: in, The calculation method is as follows: in, This represents the difference between the predicted depth and the actual depth. Indicates the height of the image to be processed. Indicates the width of the image to be processed. Indicates row index, Indicates column index.