RGB-D segmentation method and system based on feature self-enhancement and training method
The RGB-D segmentation method enhances feature fusion using attention mechanisms and residual blocks to improve noise suppression and accuracy, addressing the challenges of fusing RGB and depth information in diverse environments.
Patent Information
- Application Number
- CN202411164238.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-13
AI Technical Summary
Existing RGB-D semantic segmentation methods struggle to effectively fuse RGB and depth information due to noise and omissions in depth data, leading to suboptimal performance in various environments.
A method and system that utilize feature self-enhancement through an encoder-decoder architecture, incorporating attention mechanisms and residual blocks to guide the fusion of RGB and depth features, suppressing noise and enhancing high-quality features.
Improves the robustness and accuracy of semantic segmentation by leveraging high-level RGB features to guide low-level fusion features, effectively handling noise and maintaining high performance across diverse conditions.
Smart Images
Figure CN119131386A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and machine learning technologies, specifically to the field of image semantic segmentation technologies, and more particularly to an RGB-D segmentation method, system, and training method based on feature self-enhancement. Background Art
[0002] Semantic segmentation is a key task in computer vision, which aims to assign a semantic label to each pixel in an image. With the development of deep learning technology, semantic segmentation has made significant progress in accuracy and efficiency.
[0003] However, compared to simple RGB images, RGB-D images (i.e., RGB images containing depth information) show greater potential in semantic segmentation because depth information can provide additional geometric and spatial clues to help the model better understand the shape and boundaries of scenes and objects.
[0004] Furthermore, RGB-D semantic segmentation holds broad application prospects in computer vision. Its challenges and potential in multimodal data processing are driving the continued exploration of more effective techniques and methods. Future research will continue to advance progress in data fusion, model design, and practical applications, driving the continued development of this field.
[0005] Effectively fusing RGB information and depth information is one of the core challenges of RGB-D semantic segmentation. The two types of information have different characteristics and noise patterns, and traditional fusion methods may not be able to fully utilize their complementary advantages. Specifically, depth data is often affected by sensor accuracy, viewing angle limitations, and environmental conditions, which may cause noise and omissions in depth images. How to deal with these problems during the fusion process is the key to improving segmentation performance. Developing more robust processing methods for noise and omissions in depth information so that the model can maintain high performance in various environments and conditions is the pursuit of this field, but current technical solutions have not yet achieved this goal. Summary of the Invention
[0006] In view of the shortcomings of the prior art, the object of the present invention is to provide an RGB-D segmentation method, system and training method based on feature self-enhancement.
[0007] To achieve the aforementioned object of the invention, the technical solutions adopted by the present invention include:
[0008] In a first aspect, the present invention provides a training method for an RGB-D image segmentation model based on feature self-enhancement, comprising:
[0009] Get the RGB image and the corresponding depth image, input the encoder to obtain the depth features and multiple levels of RGB features;
[0010] Fusing the depth feature and the primary RGB feature to obtain a preliminary fusion feature;
[0011] The preliminary fusion feature is used as the first input depth feature to obtain a preliminary self-enhancement feature under the guidance of the second-level RGB feature, and the higher-level RGB feature is used to guide the self-enhancement of the current-level fusion feature to obtain the final fusion feature;
[0012] The final fusion feature and the RGB features at multiple levels are input into a decoder to obtain a segmentation result, and parameter update is performed based on the guidance of the true value label.
[0013] In a second aspect, the present invention further provides an RGB-D image segmentation method based on feature self-enhancement, which performs semantic segmentation using the RGB-D image segmentation model obtained by the above training method, including:
[0014] Get the RGB image to be segmented and the corresponding depth image, input them into the encoder to obtain depth features and multiple levels of RGB features;
[0015] Fusing the depth feature and the primary RGB feature to obtain a preliminary fusion feature;
[0016] The preliminary fusion feature is used as the first input feature to perform self-enhancement under the guidance of the second-level RGB feature to obtain an output feature, and the output feature is further used as a new input feature to perform self-enhancement under the guidance of a higher-level RGB feature until the final fusion feature is obtained;
[0017] The final fusion feature and the RGB features at multiple levels are input into a decoder to obtain a segmentation result.
[0018] In a third aspect, the present invention further provides an RGB-D image segmentation system based on feature self-enhancement, comprising an RGB-D image segmentation model obtained by the above-mentioned training method, wherein the RGB-D image segmentation model comprises:
[0019] The encoder module is used to obtain depth features and multiple levels of RGB features using the RGB image to be segmented and the corresponding depth image;
[0020] A preliminary fusion module, used to fuse the depth features and primary RGB features to obtain preliminary fusion features;
[0021] A feature self-enhancement module is used to perform self-enhancement on the preliminary fused feature as the first input feature under the guidance of the second-level RGB feature to obtain an output feature, and further perform self-enhancement on the output feature as a new input feature under the guidance of a higher-level RGB feature until a final fused feature is obtained;
[0022] The decoder module is used to input the final fusion feature and the RGB features of multiple levels into a decoder to obtain a segmentation result.
[0023] In a fourth aspect, the present invention also provides a readable storage medium, which stores a computer program. When the computer program is run, it executes the steps of the above-mentioned training method or the above-mentioned RGB-D image segmentation method, or stores an RGB-D image segmentation model obtained by the above-mentioned training method.
[0024] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention include at least:
[0025] The model construction method provided by the present invention utilizes high-level RGB features to guide the self-enhancement of low-level fusion features, and can suppress the noise in the depth information through cyclic self-enhancement, thereby making full use of their complementary advantages and avoiding the interference of noise and omissions in the depth image on the training and image segmentation processes, thereby improving training efficiency, accuracy and robustness of segmentation edges, and enabling the model to maintain high performance in various environments and conditions.
[0026] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of this application and implement them according to the contents of the specification, the following is an explanation of the preferred embodiments of the present invention with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 1 is a flow chart of an RGB-D image segmentation method provided by a typical embodiment of the present invention;
[0028] Figure 2 It is a schematic diagram of the overall framework of the RGB-D image segmentation model and model training method provided by a typical implementation case of the present invention. DETAILED DESCRIPTION
[0029] In view of the shortcomings of the prior art, the inventors of this case, after long-term research and extensive practice, have proposed the technical solution of the present invention. The following will further explain this technical solution, its implementation process and principles.
[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.
[0031] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any actual relationship or order between these components or method steps.
[0032] See also Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a training method for an RGB-D image segmentation model based on feature self-enhancement, which includes the following steps:
[0033] Get the RGB image and the corresponding depth image, input the encoder to obtain the depth features and multiple levels of RGB features;
[0034] Fusing the depth feature and the primary RGB feature to obtain a preliminary fusion feature;
[0035] The preliminary fusion feature is used as the first input depth feature to obtain a preliminary self-enhancement feature under the guidance of the second-level RGB feature, and the higher-level RGB feature is used to guide the self-enhancement of the current-level fusion feature to obtain the final fusion feature. The specific process is, for example: the preliminary fusion feature is used as the first input feature to perform self-enhancement under the guidance of the second-level RGB feature to obtain the output feature, and the output feature is continued to be used as a new input feature to perform self-enhancement under the guidance of the higher-level RGB feature until the final fusion feature is obtained;
[0036] The final fusion feature and the RGB features at multiple levels are input into a decoder to obtain a segmentation result, and parameter update is performed based on the guidance of the true value label.
[0037] Regarding the specific fusion method, in some implementation schemes, the fusion of the depth features and the primary RGB features adopts an attention mechanism to dynamically adjust the importance of each of the depth features and primary RGB features for fusion.
[0038] A designed attention-based supplementary fusion module is used to fuse preliminary depth features with RGB features. This module employs an attention mechanism, enabling the network to automatically learn appropriate feature fusion methods to achieve high-quality fused features. Specifically, the attention mechanism allows the network to dynamically adjust the importance of each feature map, effectively enhancing meaningful feature information while suppressing irrelevant or redundant features based on the needs of the current task and scenario. This approach not only improves the adaptability of the semantic segmentation model in complex scenarios, but also enables it to handle noise and low-quality information that may be present in depth images, thereby further optimizing the quality and stability of the fused features.
[0039] Further in some embodiments, the fusion process under the attention mechanism is expressed as:
[0040]
[0041] in, represents the depth feature, which is the depth feature embedded in the initial layer; F1 represents the primary RGB feature, and in the present invention, Fi and They represent RGB feature embedding and fusion feature embedding at different levels respectively; || represents the connection operation, f 1×1 Represents feature aggregation. By default, f 1×1 is a convolutional layer with a 1×1 convolution kernel; f ACF represents the complementary fusion of attention; f ACM represents the attention complementary fusion operation.
[0042] In addition, in the self-enhancement process, the preferred embodiment of the present invention also introduces a residual block layer for feature optimization, that is, in some embodiments, the self-enhancement specifically includes:
[0043] Performing residual block enhancement on the input features to select and enhance high-quality features in the preliminary fusion features;
[0044] performing guided self-enhancement on the high-quality features;
[0045] The residual block enhancement is expressed as:
[0046]
[0047] Among them, f RBL represents the residual block layer; represents said high-quality features; Represents the input features.
[0048] The preferred embodiment described above introduces a residual block layer to select and enhance high-quality features from the initial fused features. Residual blocks typically include skip connections, which help avoid the vanishing gradient problem and accelerate training. Through these skip connections, the network can effectively transfer and combine low-level and high-level feature information, thereby enhancing the ability to extract valuable information from the initial fused features.
[0049] The overall self-enhancement process can be performed using a gated convolutional layer. In some embodiments, the self-enhancement process is expressed as:
[0050]
[0051] where α n Represents the attention map matrix, which is used to highlight relevant features; σ represents the sigmoid function activation function F nRepresents the non-primary RGB feature; when n is (2, 3, ..., N-1) Represents the output feature, when n is N represents the final fusion feature; ω represents the channel weight; ⊙ represents point-by-point multiplication.
[0052] By default, N=5, but it is not limited to this.
[0053] The preferred embodiment described above feeds the features output by the residual block layer and high-level RGB features into a gated convolutional layer. The gated convolutional layer leverages the high-level RGB features to guide feature self-enhancement, thereby suppressing noise in the fused features. Specifically, the gated convolutional layer dynamically adjusts the weights of the convolution kernels to emphasize useful features and suppress noise and irrelevant information. In this way, the network effectively filters out noise in deep features, improving the purity and resolution of the final fused features and ensuring accurate semantic segmentation even in complex environments.
[0054] Regarding the loss function used, in some embodiments, the loss function used includes semantic edge loss, and the semantic edge loss is calculated as follows:
[0055]
[0056] Among them, F E represents semantic edge prediction; f P represents the feature conversion function, which converts the concatenated features into semantic edge predictions F E , two 1×1 convolution operations are used; f IG represents a function that generates image gradients, such as the Marr-Hildreth edge detector, but not limited to this; X represents a pixel; L CE represents the standard cross entropy (CE) loss; E represents the true semantic edge, which is obtained by applying edge detection based on the Euclidean distance transform algorithm to the true segmentation Y of the pixel labels of each category; represents the semantic edge loss.
[0057] In some embodiments, the loss function is expressed as:
[0058]
[0059] in, Indicates total loss; represents semantic loss; λ1 represents the first weight, λ2 represents the second weight; F Y represents the segmentation result, and Y represents the true value label.
[0060] The loss function used in the embodiment of the present invention mainly consists of two parts: one is the semantic loss Emphasizes the semantic loss of the object, and the other part is the semantic edge loss Focus on the edges of objects. λ1 represents the loss assigned to edge-related The weight λ2 is assigned to the semantic segmentation loss The weight of .
[0061] As for the decoder, in some embodiments, the decoder adopts a standardized stream decoder, and the calculation formula of the standardized stream decoder is:
[0062] x=f U (F E ,......,F N-3 , F N-2 , F N-1 , F N )
[0063]
[0064] F Y =MLP(F L )
[0065] Among them, f U Represents an ALL-MLP combination function, which consists of a multilayer perceptron (MLP) and a connection operation; Represents a probability set, where each element represents the probability of x in different distributions; z l =φ l (z 1-1 ), is the output of the first layer of the normalized stream decoder, where z0 = x belongs to the space And z=φ(x) belongs to the space L represents the total number of layers; det represents the determinant, It is in l-1 Transformation φ l The Jacobian matrix of is a predefined prior distribution, F L represents the normalized flow approximation.
[0066] In some embodiments, the prior distribution log-likelihood is expressed as:
[0067]
[0068] In the above scheme, the normalizing flow is used to approximate the probability vector of pixel x Where C is equal to the feature dimension. Each element in represents the probability of x in a specific distribution. A series of operations that make up the normalized flow coupling layer are defined, denoted as Where L represents the total number of layers. We also define is the output of the lth layer, where z0=x belongs to the space and Belong to space Therefore, the distribution of x approximated by the normalized flow can be calculated as follows:
[0069]
[0070] Among them, det represents the determinant, It is in l-1 Transform The Jacobian matrix of . is a predefined prior distribution. In order to maintain The feature dimension of is the same as x. Assuming that z obeys multiple univariate Gaussian distributions, the log-likelihood can be expressed as follows:
[0071]
[0072] Finally, a multi-layer perceptron is used to transform the probability vector F into the final prediction result as follows:
[0073] F Y =MLP(F L )
[0075] Finally, in some embodiments, the iterations in the training method can be performed in batches, i.e., the training set is divided into multiple batches, with each batch corresponding to one iteration. The number of data in the batch can be greater than one, such as 2-256, or just one, in which case the training method is not performed in batches.
[0076] As some typical examples of the above technical solutions, the process of the above training method may specifically include the following steps:
[0077] 1) Obtain preliminary RGB features and depth features: Obtain RGB images and depth images as input, and input the RGB images and depth images into their respective encoders to obtain preliminary RGB features and depth features;
[0078] 2) Fusing the preliminary deep features with the RGB features to obtain high-quality fused features: The preliminary deep features and RGB features are fused using the designed attention complementary fusion module. This module uses an attention mechanism so that the network can automatically learn the appropriate feature fusion method to obtain high-quality fused features.
[0079] 3) Using the residual block layer to perform preliminary self-enhancement on the fused features: The residual block layer is introduced to select and enhance high-quality features in the preliminary fused features. The residual block usually includes skip connections, which helps to avoid the gradient vanishing problem and accelerate training.
[0080] 4) Use gated convolutional layers to further enhance the fused features using high-level RGB features: Use gated convolutional layers to process the extracted fused features. Gated convolutional layers can use high-level RGB features to suppress noise in the fused features, thereby improving segmentation accuracy and robustness.
[0081] 5) Introduce semantic edge loss to guide and optimize the final fusion features: This loss function is usually used to ensure that the edge of the segmentation result is clearer and more accurate;
[0082] 6) The final fused features and RGB features at different levels are injected into the normalized stream encoder. Gradient descent is then used to update the model parameters until convergence. Finally, the final fused features and RGB features at different levels are injected into the normalized stream encoder. The normalized stream encoder may use normalization flow technology to generate the final segmentation result, ensuring classification accuracy and spatial coherence for each pixel.
[0083] The basic technical concept of the above technical solution can be summarized as follows: first, a preliminary fusion feature is obtained, and then the fusion feature is self-enhanced through a series of methods. The semantic edge loss function and the semantic loss function are jointly trained to calculate the loss function and update the model parameters.
[0084] A second aspect of an embodiment of the present invention further provides an RGB-D image segmentation method based on feature self-enhancement, which performs semantic segmentation using an RGB-D image segmentation model obtained by the training method provided in any of the above embodiments, including the following steps:
[0085] Get the RGB image to be segmented and the corresponding depth image, input them into the encoder to obtain depth features and multiple levels of RGB features;
[0086] Fusing the depth feature and the primary RGB feature to obtain a preliminary fusion feature;
[0087] The preliminary fusion feature is used as the first input feature to perform self-enhancement under the guidance of the second-level RGB feature to obtain an output feature, and the output feature is further used as a new input feature to perform self-enhancement under the guidance of a higher-level RGB feature until the final fusion feature is obtained;
[0088] The final fusion feature and the RGB features at multiple levels are input into a decoder to obtain a segmentation result.
[0089] A third aspect of the embodiments of the present invention further provides an RGB-D image segmentation system based on feature self-enhancement, which includes an RGB-D image segmentation model obtained by the training method provided in any of the above embodiments, wherein the RGB-D image segmentation model includes the following modules:
[0090] The encoder module is used to obtain depth features and multiple levels of RGB features using the RGB image to be segmented and the corresponding depth image;
[0091] A preliminary fusion module, used to fuse the depth features and primary RGB features to obtain preliminary fusion features;
[0092] A feature self-enhancement module is used to perform self-enhancement on the preliminary fused feature as the first input feature under the guidance of the second-level RGB feature to obtain an output feature, and further perform self-enhancement on the output feature as a new input feature under the guidance of a higher-level RGB feature until a final fused feature is obtained;
[0093] The decoder module is used to input the final fusion feature and the RGB features of multiple levels into a decoder to obtain a segmentation result.
[0094] As a typical example of the above technical solution, a model training system based on a self-enhanced feature fusion network can be used in the training process to apply the above training method. The obtained model belongs to the above segmentation system, which includes:
[0095] The preliminary feature extraction module is used to obtain preliminary RGB features and depth features of the same shape. The training set used includes labeled data semantic labels and RGB-D data;
[0096] The attention complementary fusion module is used to fuse preliminary depth features and RGB features. Through the attention mechanism, it automatically learns the appropriate feature fusion method to obtain high-quality fusion features.
[0097] The residual block layer enhancement module is used to introduce the residual block layer to select and enhance the initially fused features. It uses the skip connection in the residual block to avoid the gradient vanishing problem and accelerate training, ensuring the effective extraction of high-quality features.
[0098] The gated convolutional layer denoising module is used to input the features output by the residual block layer and the high-level RGB features into the gated convolutional layer. The gated convolutional layer uses the high-level RGB features to guide the features to self-enhancement, effectively suppressing the noise in the fused features.
[0099] The semantic edge guidance module is used to introduce semantic edge loss to guide and optimize the final fusion features. This loss function is used to ensure that the edge of the segmentation result is clearer and more accurate.
[0100] Normalized stream decoding module, which is used to inject the final fusion features and different levels of RGB features into the normalized stream encoder, apply the normalized stream technology, and generate the final segmentation result to ensure the classification accuracy and spatial coherence of each pixel;
[0101] An iterative update module is used to iteratively update the parameters of the constructed model based on the semantic loss and the semantic edge loss, and jointly train the model.
[0102] An embodiment of the present invention also provides a readable storage medium, which stores a computer program. When the computer program is run, it executes the steps of the above-mentioned training method or the above-mentioned RGB-D image segmentation method, or stores an RGB-D image segmentation model obtained by the above-mentioned training method.
[0103] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.
[0104] Example 1
[0105] This embodiment illustrates a training method for an RGB-D image segmentation model (named SEFnet), and the execution steps include:
[0106] (1) Create preliminary depth features and RGB features using the Resnet101 encoder. Secondly, the training set consists of labeled RGB and depth data. The labeled data is divided into batches of a specific size and fed into the model for training.
[0107] (2) By using the designed attention complementary fusion module to fuse the preliminary depth features and RGB features, the corresponding formula can be as follows:
[0108]
[0109] in, is the deep feature embedded in the initial layer. i and They represent RGB feature embedding and fusion feature embedding at different levels. Use || to represent the connection operation, and use f 1×1 Indicates feature aggregation. By default, f 1×1 Is a convolution layer with a 1×1 convolution kernel. Use f ACF and f ACMto represent the attention complementary fusion and attention complementary fusion operations.
[0110] (3) After the above operations, the residual block layer is introduced to select and enhance the high-quality features in the preliminary fusion features. The formula is as follows:
[0111]
[0112] Among them, f RBL represents the residual block layer.
[0113] (4) After the above operations, the gated convolution layer is used to process the extracted fusion features. The corresponding formula is as follows:
[0114]
[0115] where α n Represents the attention map matrix, used to highlight relevant features. ω represents the channel weight, ⊙ represents the point-by-point multiplication, and σ represents the sigmoid activation function. By default, N = 5.
[0116] (5) After the above operations, semantic edge loss is introduced to guide and optimize the final fusion features. The corresponding formula is as follows:
[0117]
[0118] L E =L CE (F E , E)
[0119] Among them, f IG represents the function used to generate image gradients, using the Marr-Hildreth edge detector. P represents a feature conversion function that converts the concatenated features into semantic edge predictions F E , two 1×1 convolution operations are used. L CE represents the standard cross entropy (CE) loss. E represents the ground-truth semantic edge obtained by applying edge detection based on the Euclidean distance transform algorithm to the ground-truth segmentation Y of the pixel labels of each class.
[0120] (6) After the above operations, the final fusion features and different levels of RGB features are injected into the normalized stream encoder. The corresponding formula is as follows
[0121] x=f U (F E , F N-3 , F N-2 , F N-1 , F N )
[0122] Among them, f U Represents an ALL-MLP combination function, which consists of a multilayer perceptron (MLP) and a connection operation.
[0123] Next, we use the Normalizing Flow to approximate the probability vector of pixel x. Where C is equal to the feature dimension. Each element in represents the probability of x in a specific distribution. A series of operations that make up the normalized flow coupling layer are defined, denoted as Where L represents the total number of layers. We also define is the output of the lth layer, where z0=x belongs to the space and Belong to space Therefore, the distribution of x approximated by the normalized flow can be calculated as follows:
[0124]
[0125] Among them, det represents the determinant, It is in l-1 Transform The Jacobian matrix of . is a predefined prior distribution. In order to maintain The feature dimension of is the same as x. Assuming that z obeys multiple univariate Gaussian distributions, the log-likelihood can be expressed as follows:
[0126]
[0127] Finally, a multi-layer perceptron is used to transform the probability vector F into the final prediction result as follows:
[0128] F Y =MLP(F L )
[0129] The joint loss is used to update the parameters of the base model. The formula steps for joint training are as follows:
[0130]
[0131] The loss function mainly consists of two parts: one is the semantic loss Emphasizes the semantic loss of the object, and the other part is the semantic edge loss Focus on the edges of objects. λ1 represents the loss assigned to edge-related The weight λ2 is assigned to the semantic segmentation loss The weight of .
[0132] (7) Use gradient descent to update the two model parameters and repeat steps (1) to (6) until the model converges, thereby obtaining a self-enhanced network model that can be used for RGB-D semantic segmentation.
[0133] Specifically, step (1) is a common preliminary step to obtain the initial features. In each iteration, the network parameters are updated using gradient descent. In step (2), the preliminary depth features and RGB features are fused. In step (3), the fused features are preliminarily self-enhanced. In step (4), the fused features are further self-enhanced by using the RGB high-level features. In step (5), semantic edge loss is introduced to guide and optimize the final fused features. In step (6), the final fused features and RGB features of different levels are injected into the standardized stream encoder.
[0134] The above model training process is applied to the training experiment, and the advantages of the solution provided by this embodiment are demonstrated by comparison with the baseline solution:
[0135] Data augmentation methods, including translation, flipping, and scaling, were used in all experiments. All networks were trained using the stochastic gradient descent (SGD) optimizer. The initial learning rate was 0.00025, the momentum was 0.9, and a polynomial decay strategy of 1-(iter / max_iter) was used. 0.9 , used to adjust the learning rate.
[0136] Tables 1, 2, and 3 below show the comparison results of the RGB-D semantic segmentation method described in this example and existing RGB-D semantic segmentation methods on three different datasets, using different structural networks as the base model and different labeled data partitioning.
[0137] Table 1 Model prediction results of different training methods under NYUDv2 dataset
[0138]
[0139] Table 2 Model prediction results of different training methods under SUN-RGBD dataset
[0140]
[0141] Table 3 Model prediction results of different training methods under Cityscapes dataset
[0142]
[0143] NYUDv2 consists of 1449 indoor RGB-D image samples, each with dense pixel-level annotations for 40 categories. Following common segmentation practices, NYUDv2 is divided into 795 training and 654 test RGB-D images. SUN-RGBD contains 10,335 RGB-D images with a resolution of 530×730, where objects are divided into 37 categories. The Cityscapes dataset contains images from 27 cities in Germany and neighboring countries, with dense pixel-level annotations for 30 categories, 19 of which are used for training and evaluation. Using a standard segmentation procedure, Cityscapes is divided into 2975 training images, 500 validation images, and 1525 test images, each with a resolution of 1024×2048 pixels. As can be seen from Tables 1, 2, and 3, the method described in this embodiment effectively improves the performance of the RGB-D semantic segmentation model.
[0144] Example 2
[0145] This embodiment compares the training method provided by the present invention with the existing semi-supervised RGB-D semantic segmentation method based on multi-model mutual learning:
[0146] Table 4 below shows an ablation experiment on the NYUDv2 dataset. As can be seen from Table 4, the method described in this embodiment improves the performance of the RGB-D semantic segmentation model in various modules.
[0147] Table 4 Ablation experiments on the NYUDv2 dataset
[0148]
[0149] In addition, in the above embodiments, when the corresponding technical means in many preferred implementation schemes were removed, performance degradation was observed. Therefore, the above-mentioned attention mechanism, residual block, gated convolutional layer, semantic edge loss, and normalized stream encoder are also significantly helpful in improving training effects and model performance.
[0150] Based on the above embodiments and comparative examples, it can be clearly seen that the model construction method provided in the embodiments of the present invention utilizes high-level RGB features to guide the self-enhancement of low-level fusion features, and can suppress the noise in the depth information through cyclic self-enhancement, thereby making full use of their complementary advantages and avoiding the interference of noise and omissions in the depth image on the training and image segmentation processes, thereby improving the training efficiency, the accuracy and robustness of the segmentation edges, and enabling the model to maintain high performance in various environments and conditions.
[0151] This embodiment also provides an electronic device, comprising: one or more processors; and a memory, wherein the memory is used to store one or more executable instructions; the one or more processors are configured to execute the one or more executable instructions to perform the training steps described in this embodiment or run the RGB-D semantic segmentation model provided by the embodiment of the present invention.
[0152] The specific implementation means of the present invention can be a system, method and / or computer program product. The computer program product can include a computer-readable storage medium on which are loaded computer-readable program instructions for causing a processor to implement various aspects of the present invention. The computer-readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. The computer-readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof.
[0153] It should be understood that the above embodiments are merely illustrative of the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit and substance of the present invention are intended to be encompassed within the scope of protection of the present invention.
Claims
1. A training method for an RGB-D image segmentation model based on feature self-enhancement, characterized in that: include: Get the RGB image and the corresponding depth image, input them into the encoder to obtain the depth features and multiple levels of RGB features; Fusing the depth feature and the primary RGB feature to obtain a preliminary fusion feature; The preliminary fusion feature is used as the first input depth feature to obtain a preliminary self-enhancement feature under the guidance of the second-level RGB feature, and a higher-level RGB feature is used to guide the self-enhancement of the fusion feature of the current level to obtain the final fusion feature; The final fused features and the RGB features at multiple levels are input into a standardized stream decoder to obtain a segmentation result, and parameter update is performed based on the guidance of the true value label.
2. The training method according to claim 1, characterized in that: The fusion of the deep features and the primary RGB features adopts an attention mechanism to dynamically adjust the importance of each of the deep features and primary RGB features for fusion.
3. The training method according to claim 2, characterized in that: The fusion process under the attention mechanism is expressed as: in, represents the depth feature; F1 represents the primary RGB feature; |‖ represents the connection operation, f1×1 represents feature aggregation; f ACF represents complementary fusion of attention; f ACM Represents the attention complementary fusion operation.
4. The training method according to claim 1, characterized in that: The self-enhancement specifically includes: Performing residual block enhancement on the input features, selecting and enhancing high-quality features in the preliminary fusion features; Conducting guided self-enhancement on the high-quality features; The residual block enhancement is expressed as: Among them, f RBL represents the residual block layer; representing said high quality characteristics; represents the input features.
5. The training method according to claim 4, characterized in that: The self-enhancement process is expressed as: where α n represents the attention map matrix; σ represents the sigmoid function activation function F n Represents the non-primary RGB feature; when n is (2, 3, ..., N-1) Represents the output feature, when n is N represents the final fusion feature; ω represents the channel weight; ⊙ represents point-by-point multiplication.
6. The training method according to claim 1, characterized in that: The loss function used includes semantic edge loss, which is calculated as: Among them, F E represents semantic edge prediction; f P represents the feature conversion function; f IG represents the function that generates the image gradient; X represents the pixel; L CE represents the standard cross entropy loss; E represents the true semantic edge; represents the semantic edge loss.
7. The training method according to claim 6, characterized in that: The loss function is expressed as: in, represents the total loss; represents semantic loss; λ1 represents the first weight, λ2 represents the second weight; F Y represents the segmentation result, and Y represents the true value label.
8. The training method according to claim 7, characterized in that: The calculation formula of the standardized stream decoder is: x=f U (F E ,......,F N-3 ,F N-2 ,F N-1 ,F N ) F Y =MLP(F L ) Among them, f U represents an ALL-MLP combination function; p x (x) represents a probability set, each element of which represents the probability of x in different distributions; z l =φ l (z l-1 ), is the output of the first layer of the normalized stream decoder, where z0=x belongs to the space x, and z=φ(x) belongs to the space L represents the total number of layers; det represents the determinant, It is in l-1 Transform φ l The Jacobian matrix, p z (z) is a predefined prior distribution, F L represents the normalized flow approximation; Preferably, the prior distribution log-likelihood is expressed as:
9. A RGB-D image segmentation method based on feature self-enhancement, characterized in that: Performing semantic segmentation using the RGB-D image segmentation model obtained by the training method according to any one of claims 1 to 8, comprising: Get the RGB image to be segmented and the corresponding depth image, and input them into the encoder to obtain the depth features and multiple levels of RGB features; Fusing the depth feature and the primary RGB feature to obtain a preliminary fusion feature; The preliminary fusion feature is used as the first input feature to perform self-enhancement under the guidance of the second-level RGB feature to obtain an output feature, and the output feature is continued to be used as a new input feature to perform self-enhancement under the guidance of a higher-level RGB feature until a final fusion feature is obtained; The final fusion feature and the RGB features at multiple levels are input into a decoder to obtain a segmentation result.
10. A RGB-D image segmentation system based on feature self-enhancement, characterized in that: An RGB-D image segmentation model obtained by the training method according to any one of claims 1 to 8, wherein the RGB-D image segmentation model comprises: An encoder module, for obtaining depth features and multiple levels of RGB features using the RGB image to be segmented and the corresponding depth image; A preliminary fusion module, used to fuse the depth feature and the primary RGB feature to obtain a preliminary fusion feature; A feature self-enhancement module, used to use the preliminary fusion feature as the first input feature to perform self-enhancement under the guidance of the second-level RGB feature to obtain an output feature, and to continue to use the output feature as a new input feature to perform self-enhancement under the guidance of a higher-level RGB feature until a final fusion feature is obtained; The decoder module is used to input the final fusion feature and the RGB features of multiple levels into a decoder to obtain a segmentation result.