Semantic segmentation method for realizing RGB-D mirror image segmentation and frequency domain fusion based on comparative knowledge extraction

By comparing knowledge refining and frequency domain fusion technology and combining with distillation method, the accuracy of specular reflection processing in semantic segmentation is solved, the accuracy and robustness of RGB-D mirror segmentation are improved, and it is suitable for resource-constrained devices.

CN119068194APending Publication Date: 2024-12-03ENJOYOR COMPANY LIMITED
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411134430.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to effectively handle specular reflection in semantic segmentation, resulting in low segmentation accuracy, especially on resource-constrained mobile devices, and the complex relationships and global features between RGB and depth images are not fully utilized.

Method used

Using a method based on contrast knowledge refining, a convolutional neural network is constructed, combined with time-frequency fusion and multi-scale enhancement modules, image features are extracted, and model performance is improved through distillation technology to achieve RGB-D mirror segmentation and frequency domain fusion.

Benefits of technology

The accuracy and robustness of semantic segmentation are improved, and context feature matching is enhanced through precise weight allocation and feature alignment, and the performance of small models is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068194A_ABST
    Figure CN119068194A_ABST
Patent Text Reader

Abstract

The invention discloses a semantic segmentation method for realizing RGB-D mirror image segmentation and frequency domain fusion based on comparative knowledge extraction, and is applied to the technical field of image processing. The method comprises a training stage and a testing stage, in the training stage, a plurality of original mirror images are selected to form a training set; constructing a convolutional neural network; the method comprises the following steps: preprocessing an original RGB image and a depth image, taking an enhanced image obtained after preprocessing as input of a convolutional neural network, and training to obtain a corresponding predicted mirror image; calculating a loss function value between the multi-prediction mirror image and the corresponding real mirror segmentation image; obtaining a convolutional neural network classification training model until the loss function value is reduced to the minimum; in the test stage, a to-be-detected mirror image is input into the convolutional neural network classification training model, and the model network outputs a mirror prediction map corresponding to an original mirror image. According to the method, the global features and detail information of the image are better captured, so that the segmentation precision and robustness are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to a semantic segmentation method for realizing RGB-D mirror segmentation and frequency domain fusion based on contrast knowledge extraction. Background Art

[0002] In computer vision tasks, mirror reflections do pose challenges for object detection (SOD) and semantic segmentation. Mirror segmentation is a special and challenging task in the field of computer vision, which involves distinguishing reflective surfaces (such as mirrors or reflective objects) in an image from the scene behind it. The importance of this task lies in the fact that reflective surfaces may cause visual confusion, causing segmentation algorithms to mistakenly identify the reflected content as part of the actual scene. In order to improve the accuracy of segmentation, researchers began to combine traditional methods with deep learning to exploit the ability of deep learning models to automatically learn complex features. The application of deep learning in the field of semantic segmentation is mainly due to its ability to learn high-level representations of data directly from the pixel level in an end-to-end manner. This method does not require manual annotation and can automatically classify each pixel in the image. Fully convolutional neural network (CCN) is the foundation of this field. It automatically learns features through a multi-layer structure and can accept image inputs of any size. The first is the encoding-decoding architecture, in which the encoding layer gradually reduces the spatial dimension information through pooling and extracts abstract features; the decoding layer gradually restores object details and spatial dimension information. Generally speaking, there are so-called shortcut connections between the encoding layer and the decoding layer, which are used to help the decoding layer better restore object details. Different void rates allow the network to learn features at different levels, thereby improving robustness to object size, position, and orientation.

[0003] Although deep learning has made significant progress in semantic segmentation, it still faces some challenges in practical deployment, especially on resource-constrained mobile devices. In addition, some methods may not fully utilize the complex relationship between different image modes (such as RGB and depth images), or do not fully consider the impact of global features on feature extraction, which may lead to low accuracy of segmentation results. To address these problems, researchers are exploring more efficient network architectures and how to better fuse image information from different modalities. At the same time, they are also studying how to improve the network training process to better capture the global features and detail information of the image, thereby improving the accuracy and robustness of segmentation.

[0004] Therefore, it is an urgent problem for those skilled in the art to propose a semantic segmentation method based on contrast knowledge extraction to realize RGB-D image segmentation and frequency domain fusion to solve the difficulties existing in the prior art. Summary of the invention

[0005] In view of this, the present invention provides a semantic segmentation method based on contrast knowledge extraction to realize RGB-D mirror segmentation and frequency domain fusion, so as to better capture the global features and detail information of the image, thereby improving the accuracy and robustness of segmentation.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A semantic segmentation method based on contrast knowledge extraction to achieve RGB-D image segmentation and frequency domain fusion, including two processes: training phase and testing phase:

[0008] The specific steps of the training phase are as follows:

[0009] S1-1, select multiple original mirror images, including RGB images and depth images taken in different scenes, to form a training set;

[0010] S1-2, build a convolutional neural network;

[0011] S1-3. Preprocess the original RGB image and depth image, use the enhanced image obtained after preprocessing as the input of the convolutional neural network, train to obtain the corresponding predicted mirror image, the predicted mirror image includes the semantic segmentation predicted image and the boundary predicted image, and record the obtained predicted mirror scene image set as in, Predict a set of images for semantic segmentation, A set of images for boundary prediction;

[0012] S1-4, calculating the loss function value between the predicted mirror image and the corresponding real mirror segmentation image;

[0013] S1-5, repeat step S1-3 and step S1-4 for a total of V times until the convolutional neural network converges and the loss function value is minimized, thereby obtaining a convolutional neural network classification training model;

[0014] The testing phase includes the following steps:

[0015] S2-1, select the Pth group of mirror scene images to be detected in the test set and mark them as I p , where 1≤p≤P, P=392;

[0016] S2-2, input various channels of the mirror image of the P group to be detected into the trained convolutional neural network classification training model, and the model network outputs the mirror prediction image corresponding to the original mirror image. The obtained prediction image is recorded as

[0017] The above method, optionally, S1-1 further includes:

[0018] The qth original mirror image in the training set is denoted as {I q (i,j)}, and the training set is q (i,j)} corresponding to the real semantic segmentation image is recorded as Then, the one-hot encoding technique is used to process the real semantic segmentation image corresponding to each original mirror image in the training set into q one-hot encoded images. The set of processed q one-hot encoded images is recorded as Among them, the mirror image is an RGB color image, Q is a positive integer, Q=784, q is a positive integer, 1≤q≤Q, express The pixel value of the pixel with coordinate position (i, j) in the middle.

[0019] In the above method, optionally, the convolutional neural network in S1-2 includes a feature extraction part and a feature fusion part; the feature extraction part includes an RGB image input layer, a depth image input layer and eight transformers, and the feature fusion part includes three time-frequency fusion modules, one advanced feature fusion module and four multi-scale enhancement modules.

[0020] The above method, optionally, in a convolutional neural network:

[0021] The RGB image passes through the first transformer, the second transformer, the third transformer, and the fourth transformer in sequence, and the depth map passes through the fifth transformer, the sixth transformer, the seventh transformer, and the eighth transformer in sequence;

[0022] The outputs of the fourth transformer and the eighth transformer are input into the fourth high-level feature fusion module to extract global context information. After the shallow context information is fused from high to low through the fourth multi-scale enhancement module, the third time-frequency fusion module, the third multi-scale enhancement module, the second time-frequency fusion module, the second multi-scale enhancement module, the first time-frequency fusion module, and the first multi-scale enhancement module, an output of the convolutional neural network is finally obtained.

[0023] In the above method, optionally, the advanced feature fusion module includes five convolution blocks, two average pooling layers, and three activation functions;

[0024] Input 1 first enters the first convolution block for preliminary feature extraction, recorded as output 2, and multiplied with input 1 to filter out the messy information to obtain output 4; output 4 is superimposed with the information of input 1 to obtain high-dimensional RGB information output 6 that integrates the initial information;

[0025] Input 2 first enters the first convolution block for preliminary feature extraction, recorded as output 3, and multiplied with input 2 to filter out the messy information to obtain output 5; output 5 is superimposed with the information of input 2 to obtain high-dimensional RGB information output 7 that integrates the initial information;

[0026] The high-dimensional RGB information output six is ​​first processed through an average pooling process of one to obtain a compressed RGB information, and then processed through the fourth convolution block and the activation function one to obtain a concentrated RGB information output nine; the high-dimensional RGB information output seven is first processed through an average pooling process of two to obtain a compressed RGB information, and then processed through the fifth convolution block and the activation function three to obtain a concentrated RGB information output ten;

[0027] The fused high-dimensional information is concatenated through output six and output seven, and then passed through the third convolution block and activation function two to obtain output eight; after output eight is concatenated with output nine or output ten, the two concatenated results are added together to obtain the highest-level feature output eleven.

[0028] In the above method, optionally, the first transformer has the same structure as the fourth convolutional block, including a self-attention module, a feature pyramid network, and upsampling connected in sequence; wherein the self-attention includes eight self-attention head layers, and the feature pyramid network includes a convolutional layer, a normalization layer, and an activation layer connected in sequence;

[0029] The second transformer has the same structure as the fourth convolutional block, including a self-attention module, a feature pyramid network, and upsampling connected in sequence; wherein the self-attention includes four self-attention head layers, and the feature pyramid network includes a convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0030] The above method, optionally, S1-4 specifically includes:

[0031] Using multi-category task loss to calculate the true semantic segmentation image The collection of processed one-hot encoded images A collection of semantic segmentation prediction maps The sub-loss function value between Where i = 1; the cross entropy loss function is used to calculate the real semantic segmentation image The collection of processed one-hot encoded images The set of boundary prediction maps The loss function value between Where i=2, and the sum of the two sub-loss function values ​​is used as the loss function value.

[0032] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a semantic segmentation method based on comparative knowledge extraction to realize RGB-D image segmentation and frequency domain fusion, which has the following features:

[0033] Beneficial effects:

[0034] (1) The method of the present invention constructs a convolutional neural network, inputs the mirror scene image into the convolutional neural network for training, and obtains a convolutional neural network segmentation training model; then the mirror scene image to be semantically segmented is input into the convolutional neural network segmentation training model to obtain a corresponding predicted semantic segmentation image. The method of the present invention introduces time-frequency fusion technology when constructing a convolutional neural network, effectively mines the similarity between mirror features, and accurately assigns weights to fusion channels; then, a multi-scale feature enhancement module is used to match features at different scales to achieve contextual alignment of objects, and enhance contextual features by controlling the importance between different scales.

[0035] (2) The distillation method proposed in this paper designs feature alignment distillation; secondly, inspired by comparative learning, semantic complementary comparative learning distillation is designed; finally, in order to improve accuracy, the responses of the teacher network and the student network are screened through maximum similarity distillation to improve the matching degree of feature point distribution. The distillation method allows smaller models to have better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A flowchart of a semantic segmentation method based on comparative knowledge extraction to achieve RGB-D image segmentation and frequency domain fusion provided by the present invention;

[0038] Figure 2 A block diagram of the convolutional neural network classification training model of the present invention;

[0039] Figure 3 It is a structural diagram of the advanced feature fusion module of the present invention;

[0040] Figure 4 It is a structural diagram of the time-frequency fusion module of the present invention;

[0041] Figure 5 This is a structural diagram of the multi-scale enhancement module of the present invention;

[0042] Figure 6 This is the semantic distillation structure diagram of the present invention;

[0043] Figure 7a is the first original mirror image of the same scene;

[0044] Figure 7bIt is the depth map corresponding to the first picture;

[0045] Figure 7c In order to utilize the method of the present invention Figure 7a The original mirror image shown is predicted to obtain a predicted mirror segmentation image;

[0046] Figure 8a is the second original mirror image of the same scene;

[0047] Figure 8b It is the depth map corresponding to the second picture;

[0048] Figure 8c In order to utilize the method of the present invention Figure 8a The original mirror image shown is predicted to obtain a predicted mirror segmentation image;

[0049] Figure 9a is the third original mirror image of the same scene;

[0050] Figure 9b It is the depth map corresponding to the third picture;

[0051] Fig.9c In order to utilize the method of the present invention Figure 9a The original mirror image shown is predicted to obtain a predicted mirror segmentation image;

[0052] Fig.10a is the fourth original mirror image of the same scene;

[0053] Fig.10b It is the depth map corresponding to the third picture;

[0054] Fig.10c In order to utilize the method of the present invention Fig.10a The original mirror image shown is predicted to obtain a predicted mirror segmentation image. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Reference Figure 1 As shown, the present invention discloses a semantic segmentation method based on contrast knowledge extraction to realize RGB-D image segmentation and frequency domain fusion, including two processes: a training phase and a testing phase:

[0057] The specific steps of the training phase are as follows:

[0058] S1-1, select multiple original mirror images, including RGB images and depth images taken in different scenes, to form a training set;

[0059] S1-2, build a convolutional neural network;

[0060] S1-3. Preprocess the original RGB image and depth image, use the enhanced image obtained after preprocessing as the input of the convolutional neural network, train to obtain the corresponding predicted mirror image, the predicted mirror image includes the semantic segmentation predicted image and the boundary predicted image, and record the obtained predicted mirror scene image set as in, Predict a set of images for semantic segmentation, A set of images for boundary prediction;

[0061] S1-4, calculating the loss function value between the predicted mirror image and the corresponding real mirror segmentation image;

[0062] S1-5. Repeat steps S1-3 and S1-4 for V times until the convolutional neural network converges and the loss function value is minimized, and the convolutional neural network classification training model is obtained; a total of Q loss function values ​​are obtained during the training process, and we select the one with the smallest loss value. The weight and bias term corresponding to this point are considered to be optimal because they make the model perform best on the training data. These optimal parameters are then used to construct the final classification training model, which can make effective predictions on new data, where the optimal weight vector and the optimal bias term are denoted as W best and b best ; Among them, V>1, take V=300.

[0063] The testing phase includes the following steps:

[0064] S2-1, select the Pth group of mirror scene images to be detected in the test set and mark them as I p , where 1≤p≤P, P=392;

[0065] S2-2, input various channels of the mirror image of the P group to be detected into the trained convolutional neural network classification training model, and the model network outputs the mirror prediction image corresponding to the original mirror image. The obtained prediction image is recorded as

[0066] Furthermore, S1-1 also includes:

[0067] The qth original mirror image in the training set is denoted as {I q (i,j)}, and the training set is q(i,j)} corresponding to the real semantic segmentation image is recorded as Then, the one-hot encoding technique is used to process the real semantic segmentation image corresponding to each original mirror image in the training set into q one-hot encoded images. The set of processed q one-hot encoded images is recorded as Among them, the mirror image is an RGB color image, Q is a positive integer, Q=784, q is a positive integer, 1≤q≤Q, express The pixel value of the pixel with coordinate position (i, j) in the image. The original mirror image is mainly composed of RGB images and depth images taken in different scenes. RGB images are used to capture the basic color information of the scene. These images present the details of the image through three independent spectral bands: red (R), green (G), and blue (B). The depth image records the distance of each pixel in the scene relative to the camera or a reference plane.

[0068] Furthermore, the convolutional neural network in S1-2 includes a feature extraction part and a feature fusion part; the feature extraction part includes an RGB image input layer, a depth image input layer and eight transformers, and the feature fusion part includes three time-frequency fusion modules, one advanced feature fusion module and four multi-scale enhancement modules.

[0069] Going further, in the convolutional neural network:

[0070] like Figure 2 As shown, the RGB image passes through the first transformer, the second transformer, the third transformer, and the fourth transformer in sequence, and the depth map passes through the fifth transformer, the sixth transformer, the seventh transformer, and the eighth transformer in sequence;

[0071] The outputs of the fourth transformer and the eighth transformer are input into the fourth high-level feature fusion module to extract global context information. After the shallow context information is fused from high to low through the fourth multi-scale enhancement module, the third time-frequency fusion module, the third multi-scale enhancement module, the second time-frequency fusion module, the second multi-scale enhancement module, the first time-frequency fusion module, and the first multi-scale enhancement module, an output of the convolutional neural network is finally obtained.

[0072] Furthermore, the high-level feature fusion module includes five convolution blocks, two average pooling layers, and three activation functions;

[0073] like Figure 3 As shown, input one first enters the first convolution block for preliminary feature extraction, recorded as output two, and multiplied with input one to filter out the messy information to obtain output four; output four is superimposed with the information of input one to obtain high-dimensional RGB information output six that integrates the initial information;

[0074] Input 2 first enters the first convolution block for preliminary feature extraction, recorded as output 3, and multiplied with input 2 to filter out the messy information to obtain output 5; output 5 is superimposed with the information of input 2 to obtain high-dimensional RGB information output 7 that integrates the initial information;

[0075] The high-dimensional RGB information output six is ​​first processed through an average pooling process of one to obtain a compressed RGB information, and then processed through the fourth convolution block and the activation function one to obtain a concentrated RGB information output nine; the high-dimensional RGB information output seven is first processed through an average pooling process of two to obtain a compressed RGB information, and then processed through the fifth convolution block and the activation function three to obtain a concentrated RGB information output ten;

[0076] The fused high-dimensional information is concatenated through output six and output seven, and then passed through the third convolution block and activation function two to obtain output eight; after output eight is concatenated with output nine or output ten, the two concatenated results are added together to obtain the highest-level feature output eleven.

[0077] Furthermore, the input of the convolutional neural network is the width and height of each image's RGB image and heat map image, which are W=416 and H=416 (RGB channels are R channel components, G channel components and B channel components, and the depth map is a single channel, namely D channel). Both feature extraction is performed through the segformer backbone network. The structures of the two feature extraction flows are similar, specifically:

[0078] The first transformer and the fourth convolution block are composed of three parts: self-attention, feature pyramid network, and upsampling connected in sequence. The self-attention head layer maps the input data to the output data through a linear transformation (plus a learnable bias term) through three fully connected layers. Among them, self-attention is composed of eight self-attention head layers. The feature pyramid network is composed of convolutional layers, normalization layers, and activation layers connected in sequence. The convolution kernel (kernel_size) of the convolution layer is 7×7, the number of convolution kernels (filters) is 64, the stride is 7, and the edge padding is 3. After normalization, the feature map is output through the activation layer. Finally, the spatial resolution of the image is gradually restored through upsampling while maintaining the accuracy of segmentation. The feature map sets output by the first and sixth transformers are respectively denoted as The width of the feature map is Height is

[0079] The structures of the second transformer and the seventh transformer are composed of three parts: self-attention, feature pyramid network, and upsampling. The self-attention head layer maps the input data to the output data through a linear transformation (plus a learnable bias term) through three fully connected layers. Among them, self-attention is composed of four self-attention head layers. The feature pyramid network is composed of convolutional layers, normalization layers, and activation layers connected in sequence. The convolution kernel (kernel_size) of the convolution layer is 3×3, the number of convolution kernels (filters) is 128, the stride is 2, and the edge padding is 1. After normalization, the feature map is output through the activation layer. Finally, the spatial resolution of the image is gradually restored through upsampling while maintaining the accuracy of segmentation. The feature map sets output by the second and seventh transformers are respectively denoted as The width of the feature map is Height is

[0080] The third transformer has the same structure as the eighth transformer, and is composed of three parts: self-attention, feature pyramid network, and upsampling. The self-attention head layer maps the input data to the output data through a linear transformation (plus a learnable bias term) through three fully connected layers. Among them, self-attention is composed of two self-attention head layers. The feature pyramid network is composed of convolutional layers, normalization layers, and activation layers connected in sequence. The convolution kernel (kernel_size) of the convolution layer is 3×3, the number of convolution kernels (filters) is 160, the stride is 2, and the edge padding is 1. After normalization, the feature map is output through the activation layer. Finally, the spatial resolution of the image is gradually restored through upsampling while maintaining the accuracy of segmentation. The feature map sets output by the third transformer and the eighth transformer are respectively denoted as The width of the feature map is Height is

[0081] The fourth transformer has the same structure as the ninth transformer, and is composed of three parts: self-attention, feature pyramid network, and upsampling. The self-attention head layer maps the input data to the output data through a linear transformation (plus a learnable bias term) through three fully connected layers. Among them, self-attention consists of a self-attention head layer. The feature pyramid network is composed of convolutional layers, normalization layers, and activation layers connected in sequence. The convolution kernel (kernel_size) of the convolution layer is 3×3, the number of convolution kernels (filters) is 160, the stride is 2, and the edge padding is 1. After normalization, the feature map is output through the activation layer. Finally, the spatial resolution of the image is gradually restored through upsampling while maintaining the accuracy of segmentation. The feature map sets output by the fourth transformer and the ninth transformer are respectively denoted as The width of the feature map is Height is

[0082] For the time-frequency fusion module, the outputs of the first transformer, the second transformer, the third transformer and the outputs of the fifth transformer, the sixth transformer, and the seventh transformer are respectively input into the first time-frequency fusion module, the second time-frequency fusion module, and the third time-frequency fusion module. Figure 4 As shown, input one comes from the output of the transformer that extracts depth, input two comes from extracting high-level features of the previous layer (only in the third time-frequency fusion module) or the output of the frequency domain fusion module, and output three comes from the output of the transformer that extracts RGB. First, we divide the depth information input one and the RGB information input two into two paths for fusion. One branch extracts frequency domain weighted information through a multi-spectral attention layer. The other path multiplies the high-level features to obtain output twelve and output thirteen, and adds them to their respective frequency domain weighted information to obtain filtered depth information output fourteen and RGBD information output fifteen. After that, splicing is performed in the channel dimension to obtain the summarized fusion information output sixteen, and then the channel and size are adjusted through the convolution layer to obtain the final coarse-level context information output eleven.

[0083] For the multi-scale enhancement module, its input comes from the first time-frequency fusion module, the second time-frequency fusion module, the third time-frequency fusion module and the fourth high-level feature fusion module. Figure 5 As shown in FIG. 1 , the coarse context information obtained by the time-frequency fusion module is adjusted through the 20th convolution layer, and then multi-scale feature extraction is performed from four branches. The four branches use different dilated convolutions, and the dilation rates are 1, 2, 4, and 8, respectively. After that, the multi-scale information is filtered through the channel splicing operation and the 22nd convolution block to obtain the multi-scale information output 18. In order not to lose the original information, we superimpose the input on the multi-scale to obtain the final output 19.

[0084] Semantic distillation methods, such as Figure 6 As shown. In order to alleviate the performance degradation that may occur in lightweight models due to parameter reduction, we adopted knowledge distillation technology and comparative learning strategies to optimize the prediction accuracy of lightweight student networks and cultivate networks with better performance. The study first used feature alignment distillation technology. The inputs of the first feature alignment distillation, the second feature alignment distillation, the third feature alignment distillation, and the fourth feature alignment distillation came from the intermediate features of transformers one, two, three, and four that combined depth and RGBD information, respectively. Adjustments were made to the structural differences between the teacher network and the student network in order to alleviate the model instability problem caused by outliers. Given that traditional feature distillation technology mainly focuses on the direct transmission of features between the teacher and student networks, and in actual operations, the features of each level within the student network are often difficult to effectively make up for the semantic differences with the teacher network, this study introduced an innovative semantic complementary comparative learning distillation strategy. We borrowed the concept of contrastive learning and introduced the contrastive learning goal of the first foreground distillation and the first foreground distillation between the output of the first multi-scale enhancement module between the teacher network and the student network, achieving a teaching effect different from the traditional contrastive learning method, that is, by enhancing the similarity of the same foreground or background in the student network and the teacher network, while reducing the similarity between different foregrounds or backgrounds, to promote the effective transfer of knowledge. In addition, in order to further improve the accuracy of the segmentation task, this study used the maximum likelihood distillation technology for the prediction values ​​of the student and teacher networks, and screened the responses of the student and teacher networks in order to better adapt to the main influencing factors of the distribution of feature points.

[0085] Furthermore, S1-4 specifically includes:

[0086] Using multi-category task loss to calculate the true semantic segmentation image The collection of processed one-hot encoded images A collection of semantic segmentation prediction maps The sub-loss function value between Where i = 1; the cross entropy loss function is used to calculate the real semantic segmentation image The collection of processed one-hot encoded images The set of boundary prediction maps The loss function value between Where i=2, and the sum of the two sub-loss function values ​​is used as the loss function value.

[0087] In a specific embodiment, experiments were conducted to further verify the feasibility and effectiveness of the method of the present invention.

[0088] The architecture of the frequency domain fusion network for RGB-D mirror segmentation based on contrast knowledge extraction was built using the Python-based deep learning library Pytorch3.6. The test set of the RGB-D mirror segmentation dataset (Mirror Segmentation) was used to analyze the mirror segmentation effect using the method of the present invention. We used four commonly used objective parameters for evaluating semantic segmentation methods as evaluation indicators, including the intersection over union (IoU) of the segmented image and the labeled image, weighted f-measure, mean absolute error, and balanced error rate.

[0089] The method of the present invention is used to predict each mirror image in the test set of the mirror segmentation dataset (Mirror Segmentation) of the mirror scene image database, and a predicted mirror segmentation image corresponding to each mirror image is obtained. The evaluation indexes reflecting the semantic segmentation effect of the method of the present invention are listed in Table 1. From the evaluation result data of the method of the present invention on the test set listed in Table 1, it can be seen that the segmentation result of the mirror image obtained by the method of the present invention is good, indicating that it is feasible and effective to use the method of the present invention to obtain the predicted mirror segmentation image corresponding to the mirror image.

[0090] Table 1

[0091]

[0092] Figure 7a is the first original mirror image of the same scene, Figure 7b It is the depth map corresponding to the first picture; Figure 7c In order to utilize the method of the present invention Figure 7a The original mirror image shown is predicted to obtain a predicted mirror segmentation image; Figure 8a is the second original mirror image of the same scene, Figure 8b It is the depth map corresponding to the second picture; Figure 8c In order to utilize the method of the present invention Figure 8a The original mirror image shown is predicted to obtain a predicted mirror segmentation image; Figure 9a is the third original mirror image of the same scene. Figure 9b It is the depth map corresponding to the third picture; Fig.9c In order to utilize the method of the present invention Figure 9a The original mirror image shown is predicted to obtain a predicted mirror segmentation image; Fig.10a is the fourth original mirror image of the same scene. Fig.10b It is the depth map corresponding to the third picture; Fig.10c In order to utilize the method of the present invention Fig.10a The original mirror image shown is predicted to obtain the predicted mirror segmentation image. Figure 7a and Figure 7c ,contrast Figure 8a and Figure 8c ,contrast Figure 9a and Fig.9c ,contrast Fig.10a and Fig.10c It can be seen that the segmentation accuracy of the predicted mirror segmentation image obtained by the method of the present invention is high.

[0093] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0094] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic segmentation method based on contrast knowledge extraction to achieve RGB-D image segmentation and frequency domain fusion, It is characterized in that It includes two processes: training phase and testing phase: The specific steps of the training phase are as follows: S1-1, select multiple original mirror images, including RGB images and depth images taken in different scenes, to form a training set; S1-2, build a convolutional neural network; S1-3. Preprocess the original RGB image and depth image, use the enhanced image obtained after preprocessing as the input of the convolutional neural network, train to obtain the corresponding predicted mirror image, the predicted mirror image includes the semantic segmentation predicted image and the boundary predicted image, and record the obtained predicted mirror scene image set as in, Predict a set of images for semantic segmentation, A set of images for boundary prediction; S1-4, calculating the loss function value between the predicted mirror image and the corresponding real mirror segmentation image; S1-5, repeat step S1-3 and step S1-4 for a total of V times until the convolutional neural network converges and the loss function value is minimized, thereby obtaining a convolutional neural network classification training model; The testing phase includes the following steps: S2-1, select the Pth group of mirror scene images to be detected in the test set and mark them as I p , where 1≤p≤P, P=392; S2-2, input various channels of the mirror image of the P group to be detected into the trained convolutional neural network classification training model, and the model network outputs the mirror prediction image corresponding to the original mirror image. The obtained prediction image is recorded as 2. According to claim 1, a semantic segmentation method based on contrast knowledge extraction to achieve RGB-D image segmentation and frequency domain fusion, It is characterized in that S1-1 also includes: The qth original mirror image in the training set is denoted as {I q (i,j)}, and the training set is q (i,j)} corresponding to the real semantic segmentation image is recorded as Then, the one-hot encoding technique is used to process the real semantic segmentation image corresponding to each original mirror image in the training set into q one-hot encoded images. The set of processed q one-hot encoded images is recorded as Among them, the mirror image is an RGB color image, Q is a positive integer, Q=784, q is a positive integer, 1≤q≤Q, express The pixel value of the pixel with coordinate position (i, j) in the middle.

3. According to claim 1, a semantic segmentation method for realizing RGB-D image segmentation and frequency domain fusion based on contrast knowledge extraction, It is characterized in that The convolutional neural network in S1-2 includes a feature extraction part and a feature fusion part; the feature extraction part includes an RGB image input layer, a depth image input layer and eight transformers, and the feature fusion part includes three time-frequency fusion modules, one high-level feature fusion module and four multi-scale enhancement modules.

4. According to claim 3, a semantic segmentation method for realizing RGB-D image segmentation and frequency domain fusion based on contrast knowledge extraction, It is characterized in that In a convolutional neural network: The RGB image passes through the first transformer, the second transformer, the third transformer, and the fourth transformer in sequence, and the depth map passes through the fifth transformer, the sixth transformer, the seventh transformer, and the eighth transformer in sequence; The outputs of the fourth transformer and the eighth transformer are input into the fourth high-level feature fusion module to extract global context information. After the shallow context information is fused from high to low through the fourth multi-scale enhancement module, the third time-frequency fusion module, the third multi-scale enhancement module, the second time-frequency fusion module, the second multi-scale enhancement module, the first time-frequency fusion module, and the first multi-scale enhancement module, an output of the convolutional neural network is finally obtained.

5. According to claim 3, a semantic segmentation method based on contrast knowledge extraction to achieve RGB-D image segmentation and frequency domain fusion, It is characterized in that The high-level feature fusion module includes five convolution blocks, two average pooling layers, and three activation functions; Input 1 first enters the first convolution block for preliminary feature extraction, recorded as output 2, and multiplied with input 1 to filter out the messy information to obtain output 4; output 4 is superimposed with the information of input 1 to obtain high-dimensional RGB information output 6 that integrates the initial information; Input 2 first enters the first convolution block for preliminary feature extraction, recorded as output 3, and multiplied with input 2 to filter out the messy information to obtain output 5; output 5 is superimposed with the information of input 2 to obtain high-dimensional RGB information output 7 that integrates the initial information; The high-dimensional RGB information output six is ​​first processed through an average pooling process of one to obtain a compressed RGB information, and then processed through the fourth convolution block and the activation function one to obtain a concentrated RGB information output nine; the high-dimensional RGB information output seven is first processed through an average pooling process of two to obtain a compressed RGB information, and then processed through the fifth convolution block and the activation function three to obtain a concentrated RGB information output ten; The fused high-dimensional information is concatenated through output six and output seven, and then passed through the third convolution block and activation function two to obtain output eight; after output eight is concatenated with output nine or output ten, the two concatenated results are added together to obtain the highest-level feature output eleven.

6. According to claim 5, a semantic segmentation method for realizing RGB-D image segmentation and frequency domain fusion based on contrast knowledge extraction, It is characterized in that The first transformer has the same structure as the fourth convolutional block, including a self-attention module, a feature pyramid network, and upsampling connected in sequence; wherein the self-attention includes eight self-attention head layers, and the feature pyramid network includes a convolution layer, a normalization layer, and an activation layer connected in sequence; The second transformer has the same structure as the fourth convolutional block, including a self-attention module, a feature pyramid network, and upsampling connected in sequence; wherein the self-attention includes four self-attention head layers, and the feature pyramid network includes a convolutional layer, a normalization layer, and an activation layer connected in sequence.

7. According to claim 1, a semantic segmentation method for realizing RGB-D image segmentation and frequency domain fusion based on contrast knowledge extraction, It is characterized in that S1-4 specifically includes: Using multi-category task loss to calculate the true semantic segmentation image The collection of processed one-hot encoded images A collection of semantic segmentation prediction maps The sub-loss function value between Where i = 1; the cross entropy loss function is used to calculate the real semantic segmentation image The collection of processed one-hot encoded images The set of boundary prediction maps The loss function value between Where i=2, and the sum of the two sub-loss function values ​​is used as the loss function value.