Image segmentation method of mirror image semantic segmentation lightweight insight network based on double contrast knowledge distillation

By employing a lightweight insight network for mirror semantic segmentation through dual-contrast knowledge distillation, the problem of low segmentation accuracy in mirror scenarios is solved, achieving efficient mirror segmentation results and making it suitable for resource-constrained mobile environments.

CN118941788APending Publication Date: 2024-11-12ENJOYOR COMPANY LIMITED
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410949597.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing semantic segmentation methods have low segmentation accuracy in mirrored scenarios and are limited by mobile resources, making it impossible to fully utilize multimodal data and global context information.

Method used

We employ a lightweight insight network for mirror semantic segmentation based on dual contrastive knowledge distillation. Through cross-modal feature aggregation, channel enhancement, and geometric space modules, combined with adaptive adjustment and detail enhancement design, we achieve feature fusion and efficient segmentation.

Benefits of technology

It improves the accuracy and efficiency of mirror segmentation, adapts to different channel response intensities, coordinates information flow, comprehensively characterizes the differences between mirror reflection and non-mirror regions, and solves the obstacles of multi-level feature fusion and transmission in mirror segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941788A_ABST
    Figure CN118941788A_ABST
Patent Text Reader

Abstract

The invention discloses an image segmentation method of a mirror image semantic segmentation lightweight insight network based on double contrast knowledge distillation, and relates to the technical field of image processing. Comprising a training stage and a testing stage, a plurality of original mirror scene images and a depth map and a real semantic segmentation image corresponding to each original mirror scene image are selected, a training set is formed, a convolutional neural network is constructed, the training set is input to the neural network for training, and a training result is obtained; and taking the obtained weight vector and the bias item as an optimal weight vector and an optimal bias item of a convolutional neural network classification training model to obtain the convolutional neural network classification training model, inputting a to-be-detected mirror scene image in the test set, and outputting a mirror scene semantic segmentation prediction map corresponding to the original mirror scene image by the model network. According to the invention, the precision and generalization ability of mirror segmentation are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to an image segmentation method of a mirror semantic segmentation lightweight insight network based on double contrast knowledge distillation. Background Art

[0002] With the continuous development of technologies such as intelligent transportation systems, intelligent driving, augmented reality / virtual reality, semantic segmentation plays an increasingly important role in many fields such as scene understanding, object detection, and visual navigation. Semantic segmentation aims to classify each pixel in the input image and mark its semantic category. It is an important basic technology for achieving pixel-level refined understanding. Traditional semantic segmentation methods mainly rely on shallow features of images, such as texture, color, etc., as well as manually designed structural features. However, since only surface visual clues can be obtained, the segmentation results obtained are relatively rough and lack rich semantic information, which cannot meet the needs of practical applications.

[0003] In order to improve the segmentation accuracy and the richness of semantic information, researchers introduced deep learning technology into the semantic segmentation task. As an end-to-end data-driven method, deep learning models can automatically learn feature representations directly from a large amount of raw data, thus avoiding the lengthy process of manually designing classifiers. More importantly, deep neural networks have a hierarchical structure and can gradually learn high-level semantic concepts from underlying pixels, greatly improving the accuracy and efficiency of pixel-level classification tasks. Fully Convolutional Network (FCN) has the advantage of not relying on fully connected layers and can directly perform pixel-level predictions on input images of any size, becoming the mainstream framework in the field of semantic segmentation.

[0004] Based on FCN, innovative technologies such as the encoder-decoder architecture and dilated / atrous convolution have emerged, promoting the rapid development of scene understanding research. The encoder-decoder structure consists of two main modules: the encoder reduces the feature resolution layer by layer and extracts high-level semantic features through continuous convolution and downsampling operations; the decoder upsamples and restores the feature resolution layer by layer, and uses jump connections to fuse shallow and deep features to finally obtain fine-grained segmentation predictions. This bidirectional architecture can achieve a balance between feature extraction and spatial accuracy. The dilated convolution improves the detection capability of large-size targets by controllably expanding the receptive field of the convolution kernel and learning multi-scale features without losing resolution.

[0005] Despite this, existing semantic segmentation methods still face many challenges in actual deployment. First, these models usually have large parameters and high computational complexity, and their application in resource-constrained environments such as mobile terminals is often severely restricted. Secondly, when fusing multimodal data such as RGB images and depth images, most networks use a simple feature cascade or point-wise addition method, failing to fully utilize the complex correlation between the two modalities, resulting in redundant or missing information. Furthermore, existing methods lack efficient use of global contextual information, and features extracted only by relying on local receptive fields often have blind spots, making it difficult to restore the complete target morphology, affecting the segmentation quality.

[0006] Especially in mirror scenes, due to the existence of mirror reflection, the shape and texture of objects often undergo a certain degree of deformation and distortion, which brings great challenges to semantic segmentation. Most existing methods rely on fixed mirror elements as segmentation features, but this approach ignores the uneven quality of feature images and geometric distortion in complex real environments, resulting in low segmentation accuracy.

[0007] Therefore, it is an urgent problem for those skilled in the art to propose an image segmentation method based on a mirror semantic segmentation lightweight insight network with dual contrast knowledge distillation to solve the difficulties existing in the prior art. Summary of the invention

[0008] In view of this, the present invention provides an image segmentation method of a mirror semantic segmentation lightweight insight network based on double contrast knowledge distillation, which is used to solve the technical problems existing in the prior art.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] An image segmentation method based on dual contrast knowledge distillation and mirror semantic segmentation lightweight insight network includes two processes: training phase and testing phase;

[0011] The training phase consists of the following steps:

[0012] S1-1, select multiple original mirror scene images and corresponding depth maps, as well as real semantic segmentation images, and form a training set;

[0013] S1-2, build a convolutional neural network;

[0014] S1-3. After data enhancement, each original mirror image and the corresponding depth map in the training set are used as the original input image and input into the convolutional neural network for training to obtain the corresponding predicted mirror image. The predicted mirror image includes a semantic segmentation predicted image and a boundary predicted image. The obtained predicted mirror scene image set is recorded as in, Predict a set of images for semantic segmentation, A set of images for boundary prediction;

[0015] S1-4, calculating the loss function value between the semantic segmentation prediction image and the corresponding true semantic segmentation image;

[0016] S1-5, repeat S1-3 and S1-4 for V times in total until the convolutional neural network converges and the loss function value is minimized, and the convolutional neural network classification training model is obtained; the weight vector and bias term of the convolutional neural network currently obtained are used as the optimal weight vector and optimal bias term of the convolutional neural network classification training model, which are correspondingly recorded as W best and b best , where V>1;

[0017] The testing phase includes the following steps:

[0018] S2-1, select the Pth group of mirror scene images to be detected in the test set and mark them as I p , where 1≤p≤P;

[0019] S2-2, input various channel components of the Pth group of mirror scene images to be detected into the convolutional neural network classification training model, and the model network outputs the mirror scene semantic segmentation prediction image corresponding to the original mirror scene image. The obtained mirror scene semantic segmentation prediction image is recorded as

[0020] The above method, optionally, a convolutional neural network is used for feature extraction and feature fusion;

[0021] The feature extraction operations of the convolutional neural network include: RGB image input layer, depth image input layer, twenty-five convolution blocks, eight edge contour enhancement modules ARM, four cross-modal feature aggregation modules DFFM and two geometric space modules GRF.

[0022] In the above method, optionally, the convolutional neural network includes a teacher network and a student network; wherein the RGB image is input to the first convolution block, and the depth image is input to the fifth convolution block.

[0023] In the above method, optionally, the teacher network comprises: eleven convolution blocks, eight edge contour enhancement modules ARM, four cross-modal feature aggregation modules DFFM and two geometric space modules GRF;

[0024] The first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the first geometric module are connected in sequence; the fifth convolution block, the sixth convolution block, the seventh convolution block, the eighth convolution block, and the second geometric module are connected in sequence;

[0025] An output end of the first convolution block is connected to an input end of the first edge contour enhancement module, an output end of the first edge contour enhancement module is connected to a first input end of the first cross-modal feature aggregation module, an output end of the fifth convolution block is connected to an input end of the fifth edge contour enhancement module, and an output end of the fifth edge contour enhancement module is connected to a second input end of the first cross-modal feature aggregation module;

[0026] The output end of the second convolution block is connected to the first input end of the second edge contour enhancement module, the output end of the first cross-modal feature aggregation module is connected to the second input end of the second edge contour enhancement module, and the output end of the second edge contour enhancement module is connected to the first input end of the second cross-modal feature aggregation module; the output end of the sixth convolution block is connected to the first input end of the sixth edge contour enhancement module, the output end of the first cross-modal feature aggregation module is connected to the second input end of the sixth edge contour enhancement module, and the output end of the sixth edge contour enhancement module is connected to the second input end of the second cross-modal feature aggregation module;

[0027] The output end of the third convolution block is connected to the first input end of the third edge contour enhancement module, the output end of the second cross-modal feature aggregation module is connected to the second input end of the third edge contour enhancement module, and the output end of the third edge contour enhancement module is connected to the first input end of the third cross-modal feature aggregation module; the output end of the seventh convolution block is connected to the first input end of the seventh edge contour enhancement module, the output end of the second cross-modal feature aggregation module is connected to the second input end of the seventh edge contour enhancement module, and the output end of the seventh edge contour enhancement module is connected to the second input end of the third cross-modal feature aggregation module;

[0028] The output end of the fourth convolution block and the output end of the first geometric space module are both connected to the first input end of the fourth edge contour enhancement module, the output end of the third cross-modal feature aggregation module is connected to the second input end of the fourth edge contour enhancement module, the output end of the fourth edge contour enhancement module is connected to the first input end of the fourth cross-modal feature aggregation module, the output end of the eighth convolution block is connected to the output end of the second space geometry module and the first input end of the eighth edge contour enhancement module, the output end of the third modal feature aggregation module is connected to the second input end of the eighth edge contour enhancement module, and the output end of the eighth edge contour enhancement module is connected to the second input end of the fourth cross-modal feature aggregation module;

[0029] The output of the first cross-modal feature aggregation module and the output of the second cross-modal feature aggregation module are cascaded and input into the ninth convolution block, the output one of the ninth convolution block and the output of the third cross-modal feature aggregation module are cascaded and input into the tenth convolution block, the output two of the tenth convolution block and the output of the fourth cross-modal feature aggregation module are cascaded and input into the input of the eleventh convolution block, and the output of the eleventh convolution block is output three.

[0030] In the above method, optionally, the student network includes: eleven convolutional blocks identical to the teacher network and four cross-modal feature aggregation modules DFFM identical to the teacher network;

[0031] The first convolution block, the second convolution block, the third convolution block, and the fourth convolution block are connected in sequence; the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block are connected in sequence;

[0032] An output end of the first convolution block is connected to a first input end of the first cross-modal feature aggregation module, and an output end of the fifth convolution block is connected to a second input end of the first cross-modal feature aggregation module;

[0033] An output end of the second convolution block is connected to a first input end of the second cross-modal feature aggregation module, and an output end of the sixth convolution block is connected to a second input end of the second cross-modal feature aggregation module;

[0034] An output end of the third convolution block is connected to a first input end of the third cross-modal feature aggregation module, and an output end of the seventh convolution block is connected to a second input end of the third cross-modal feature aggregation module;

[0035] An output end of the fourth convolution block is connected to a first input end of the fourth cross-modal feature aggregation module, and an output end of the eighth convolution block is connected to a second input end of the fourth cross-modal feature aggregation module;

[0036] The output of the first cross-modal feature aggregation module and the output of the second cross-modal feature aggregation module are cascaded and input into the ninth convolution module. The output one of the ninth convolution module and the output of the third cross-modal feature aggregation module are both input into the tenth convolution module. The output two of the tenth convolution block and the output of the fourth cross-modal feature aggregation module are cascaded and input into the eleventh convolution module. The output of the eleventh convolution module is output three.

[0037] In the above method, optionally, the first convolution block and the fourth convolution block have the same structure, including two global awareness attention modules and two local enhancement attention modules;

[0038] The second convolutional block has the same structure as the fifth convolutional block, including two global awareness attention modules and two local enhancement attention modules;

[0039] The third convolutional block has the same structure as the sixth convolutional block, including eight global awareness attention modules and eight local enhancement attention modules;

[0040] The fourth convolutional block has the same structure as the eighth convolutional block, including two global awareness attention modules and two local enhancement attention modules.

[0041] In the above method, optionally, the specific content of the cross-modal feature aggregation module is:

[0042] Input one and two are respectively passed through the twelfth convolution block and the thirteenth convolution block to obtain output five and output six. The pixels of output five and output six are multiplied and input into the first channel enhancer. The output generated by the first channel enhancer is multiplied with output five and output six again by pixel, and the generated output is added with output five and output six pixels respectively. Then, the first feature aggregation and the second feature aggregation are successively passed through. The output of the first feature aggregation and the output of the second feature aggregation are cascaded to obtain output seven. The output of output seven after passing through the first dilated convolution, the second dilated convolution, and the third dilated convolution is cascaded, and the result of the residual generated by the fourteenth convolution of output seven is added pixel by pixel to obtain output eight.

[0043] In the above method, optionally, the specific content of the edge contour enhancement module is:

[0044] The input one is sequentially passed through the output of the fifteenth convolution block and the sixteenth convolution block, and the input one is sequentially passed through the output of the seventeenth convolution block and the eighteenth convolution block. The value obtained by cascading is multiplied with the auxiliary branch to obtain the output nine; the value obtained by pixel addition of the output nine and the output of the seventeenth convolution block, and the value obtained by pixel addition of the output nine and the output of the eighteenth convolution block are input into the linear fusion module as the variable of the linear fusion module, the first fusion factor and the second fusion factor are used as parameters, and finally pixel multiplication is performed with the input one to obtain the output ten;

[0045] The auxiliary branch is the result of input two being transposed after the first average pooling, then placed in the fully connected layer, and then pixel-wise multiplied with input two, and output nine is a weight.

[0046] In the above method, optionally, the specific content of the spatial geometry module is:

[0047] Input one passes through the output of the first convolution block, and input two passes through the output of the second convolution block, and then is input into the nineteenth convolution block. After pixel subtraction, the output of the nineteenth convolution block is pixel-added with the output of the first convolution block to obtain output one, and the other is pixel-added with the output of the first convolution block after the Fourier transform filter to obtain output two.

[0048] In the above method, optionally, the convolutional neural network further includes a cascade distillation module, and the specific content of the cascade distillation module is:

[0049] Input one is sequentially passed through the output of the first maximum pooling layer and the 21st convolution block, input two is sequentially passed through the output of the second maximum pooling layer and the 22nd convolution block, and input three is sequentially passed through the output of the third maximum pooling layer and the 23rd convolution block, and then cascaded and input into the 24th convolution block to obtain output one;

[0050] Input 1 passes through the output of the first maximum pooling layer and the 21st convolution block in sequence, input 2 passes through the output of the second maximum pooling layer and the 22nd convolution block in sequence, and output 1 is pixel-multiplied to obtain output 2;

[0051] Input 2 is sequentially passed through the output of the second maximum pooling layer and the 22nd convolution block, input 3 is sequentially passed through the output of the third maximum pooling layer and the 23rd convolution block, and output 1 is pixel-multiplied to obtain output 3;

[0052] After adding pixels of output 2 and output 3, the pixels are input into the 25th convolution block and pass through the activation function. The loss calculation with input 3 after the activation function is the process of the first cascade distillation.

[0053] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses an image segmentation method of a mirror semantic segmentation lightweight insight network based on double contrast knowledge distillation, and its beneficial effects are:

[0054] 1) This paper proposes an RGB-D mirror insight network, a cross-modal feature aggregation module, and a channel enhancement operation, adaptively adjusts the response strength of different channels, and coordinates the information flow at different granularities, which provides more valuable clues for mirror segmentation; and the geometric space module integrates the feature representation of the spatial domain and the frequency domain, which is expected to more comprehensively characterize the difference between the mirror reflection and the non-mirror area; in addition, the double gating mechanism and multi-path design are specially designed by applying detail enhancement, giving the network the ability to adaptively represent the details of the edge and angle of the mirror object, so as to improve the segmentation accuracy of the mirror area;

[0055] 2) The present invention introduces the "teaching assistant" mechanism, which realizes the adaptive fusion of features at different levels and the gradual knowledge transfer from high-level to low-level through cascade fusion and supervision, and effectively solves the obstacles of multi-level feature fusion and transfer;

[0056] 3) This paper is the first to use pixel-level contrastive learning in mirror segmentation. By adding learnable mixed noise to negative samples, various forms of interference in real scenes can be more comprehensively covered. In order to make up for the shortcomings of KL divergence and the uneven quality of negative samples, we use kernel loss to map the complex distributed positive and negative samples to a higher-dimensional feature space to make them linearly separable, thereby better measuring and comparing similarities. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0058] Figure 1 A flow chart of an image segmentation method based on a lightweight insight network for mirror semantic segmentation with dual contrast knowledge distillation provided by the present invention;

[0059] Figure 2 It is the overall implementation block diagram of the teacher network of the present invention;

[0060] Figure 3 It is the overall implementation block diagram of the student network of the present invention;

[0061] Figure 4 This is a block diagram of the cross-modal feature aggregation module of the present invention;

[0062] Figure 5 This is a block diagram of the edge contour enhancement module of the present invention;

[0063] Figure 6 This is a block diagram of the geometric space module implementation of the present invention;

[0064] Figure 7 This is a block diagram of the cascade distillation module of the present invention;

[0065] Figure 8 The overall distillation framework diagram of the present invention is

[0066] Figure 9a is the first original mirror scene image of the same scene;

[0067] Figure 9b In order to utilize the method of the present invention Figure 8 The original mirror scene image shown is predicted to obtain a predicted semantic segmentation image;

[0068] Fig.10a is the second original mirror scene image of the same scene;

[0069] Fig.10b In order to utilize the method of the present invention Fig.10a The original mirror scene image shown is predicted to obtain a predicted semantic segmentation image;

[0070] Fig.11a is the third original mirror scene image of the same scene;

[0071] Fig.11b In order to utilize the method of the present invention Fig.11a The original mirror scene image shown is predicted to obtain a predicted semantic segmentation image;

[0072] Fig.12a is the fourth original mirror scene image of the same scene;

[0073] Figure 12bIn order to utilize the method of the present invention Fig.12a The original mirror scene image shown is predicted to obtain a predicted semantic segmentation image. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1 As shown, the present invention discloses an image segmentation method of a mirror semantic segmentation lightweight insight network based on double contrast knowledge distillation, including two processes: a training phase and a testing phase; wherein,

[0076] The training phase consists of the following steps:

[0077] S1-1, select Q original mirror scene images and corresponding depth maps, as well as real semantic segmentation images, and form a training set, and record the qth original mirror scene image in the training set as {I q (i,j)}, and the training set is q (i,j)} corresponding to the real semantic segmentation image is recorded as Among them, the mirror scene image is an RGB color image, Q is a positive integer, such as Q = 3046, q is a positive integer, 1≤q≤Q, express The pixel value of the pixel point with coordinate position (i, j);

[0078] S1-2, build a convolutional neural network;

[0079] Furthermore, convolutional neural networks are used for feature extraction and feature fusion;

[0080] The feature extraction operations of the convolutional neural network include: RGB image input layer, depth image input layer, twenty-five convolution blocks, eight edge contour enhancement modules ARM, four cross-modal feature aggregation modules DFFM and two geometric space modules GRF.

[0081] Furthermore, the convolutional neural network includes a teacher network and a student network; wherein the RGB image is input to the first convolution block and the depth image is input to the fifth convolution block.

[0082] For further information, see Figure 2As shown, the teacher network includes: eleven convolutional blocks, eight edge contour enhancement modules ARM, four cross-modal feature aggregation modules DFFM and two geometric space modules GRF;

[0083] The first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the first geometric module are connected in sequence; the fifth convolution block, the sixth convolution block, the seventh convolution block, the eighth convolution block, and the second geometric module are connected in sequence;

[0084] An output end of the first convolution block is connected to an input end of the first edge contour enhancement module, an output end of the first edge contour enhancement module is connected to a first input end of the first cross-modal feature aggregation module, an output end of the fifth convolution block is connected to an input end of the fifth edge contour enhancement module, and an output end of the fifth edge contour enhancement module is connected to a second input end of the first cross-modal feature aggregation module;

[0085] The output end of the second convolution block is connected to the first input end of the second edge contour enhancement module, the output end of the first cross-modal feature aggregation module is connected to the second input end of the second edge contour enhancement module, and the output end of the second edge contour enhancement module is connected to the first input end of the second cross-modal feature aggregation module; the output end of the sixth convolution block is connected to the first input end of the sixth edge contour enhancement module, the output end of the first cross-modal feature aggregation module is connected to the second input end of the sixth edge contour enhancement module, and the output end of the sixth edge contour enhancement module is connected to the second input end of the second cross-modal feature aggregation module;

[0086] The output end of the third convolution block is connected to the first input end of the third edge contour enhancement module, the output end of the second cross-modal feature aggregation module is connected to the second input end of the third edge contour enhancement module, and the output end of the third edge contour enhancement module is connected to the first input end of the third cross-modal feature aggregation module; the output end of the seventh convolution block is connected to the first input end of the seventh edge contour enhancement module, the output end of the second cross-modal feature aggregation module is connected to the second input end of the seventh edge contour enhancement module, and the output end of the seventh edge contour enhancement module is connected to the second input end of the third cross-modal feature aggregation module;

[0087] The output end of the fourth convolution block and the output end of the first geometric space module are both connected to the first input end of the fourth edge contour enhancement module, the output end of the third cross-modal feature aggregation module is connected to the second input end of the fourth edge contour enhancement module, the output end of the fourth edge contour enhancement module is connected to the first input end of the fourth cross-modal feature aggregation module, the output end of the eighth convolution block is connected to the output end of the second space geometry module and the first input end of the eighth edge contour enhancement module, the output end of the third modal feature aggregation module is connected to the second input end of the eighth edge contour enhancement module, and the output end of the eighth edge contour enhancement module is connected to the second input end of the fourth cross-modal feature aggregation module;

[0088] The output of the first cross-modal feature aggregation module and the output of the second cross-modal feature aggregation module are cascaded and input into the ninth convolution block, the output one of the ninth convolution block and the output of the third cross-modal feature aggregation module are cascaded and input into the tenth convolution block, the output two of the tenth convolution block and the output of the fourth cross-modal feature aggregation module are cascaded and input into the input of the eleventh convolution block, and the output of the eleventh convolution block is output three.

[0089] Specifically, the RGB image passes through the first convolution block, the second convolution block, the third convolution block, and the fourth convolution block in sequence, and the depth map passes through the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block in sequence; the RGB image and the depth image corresponding to the vertical direction are first input into the first edge contour enhancement module and the fifth edge contour enhancement module; then the enhanced output is input into the first cross-modal feature aggregation module, and its output passes through the second edge contour enhancement module and the sixth edge contour enhancement module of the second layer, and the output is synthesized into the second cross-modal feature aggregation, and the second cross-modal feature aggregation is then respectively passed through the third edge contour enhancement module and the seventh edge contour enhancement module with the third convolution block and the seventh convolution block to obtain the third cross-modal feature aggregation output, and then, the fourth convolution block and the eighth convolution block fuse the spatial and frequency domain information under the guidance of the geometric space module, and after edge contour enhancement, they are placed in the fourth cross-modal feature aggregation module to obtain the fourth cross-modal feature aggregation output, and finally the four aggregation outputs are decoded.

[0090] For further information, see Figure 3 As shown, the student network includes: eleven convolutional blocks that are the same as the teacher network and four cross-modal feature aggregation modules DFFM that are the same as the teacher network;

[0091] The first convolution block, the second convolution block, the third convolution block, and the fourth convolution block are connected in sequence; the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block are connected in sequence;

[0092] An output end of the first convolution block is connected to a first input end of the first cross-modal feature aggregation module, and an output end of the fifth convolution block is connected to a second input end of the first cross-modal feature aggregation module;

[0093] An output end of the second convolution block is connected to a first input end of the second cross-modal feature aggregation module, and an output end of the sixth convolution block is connected to a second input end of the second cross-modal feature aggregation module;

[0094] An output end of the third convolution block is connected to a first input end of the third cross-modal feature aggregation module, and an output end of the seventh convolution block is connected to a second input end of the third cross-modal feature aggregation module;

[0095] An output end of the fourth convolution block is connected to a first input end of the fourth cross-modal feature aggregation module, and an output end of the eighth convolution block is connected to a second input end of the fourth cross-modal feature aggregation module;

[0096] The output of the first cross-modal feature aggregation module and the output of the second cross-modal feature aggregation module are cascaded and input into the ninth convolution module. The output one of the ninth convolution module and the output of the third cross-modal feature aggregation module are both input into the tenth convolution module. The output two of the tenth convolution block and the output of the fourth cross-modal feature aggregation module are cascaded and input into the eleventh convolution module. The output of the eleventh convolution module is output three.

[0097] Specifically, the student network is the same as the teacher network, the RGB image passes through the first convolution block, the second convolution block, the third convolution block, and the fourth convolution block in sequence, and the depth map passes through the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block in sequence; the RGB image and the depth image corresponding to the vertical direction are input into the cross-modal feature aggregation module to obtain four cross-modal feature aggregation outputs, which are then decoded, and the output of the first cross-modal feature aggregation module and the output of the second cross-modal feature aggregation module are cascaded and put into the ninth convolution block to obtain output one, and the output one is cascaded with the third cross-modal feature aggregation module, and output two is obtained after the tenth convolution, and the output two is cascaded with the fourth cross-modal feature aggregation module to obtain the final result output three.

[0098] First, the input of the convolutional neural network is the RGB image and depth image of each image. The width and height of the image are W = 416 and H = 416 (RGB channels are R channel components, G channel components, and B channel components, and the depth map is a single channel, namely the D channel). Both feature extraction is performed through the Deformer backbone network. The structures of the two feature extraction flows are similar, specifically:

[0099] The building blocks of this paper are mainly composed of the Global Aware Attention (GAA) module and the Local Enhanced Attention (LEA) module to build the interaction between RGB and depth patterns. GAA integrates depth information and aims to enhance the ability of object localization from a global perspective, while LEA adopts large kernel convolution to capture local clues from depth features, which can refine the details of RGB representation. Our GAA fuses depth and RGB features to establish relationships across the scene, enhance 3D awareness, and further help capture semantic objects. The queries in GAA are sampled to a fixed size, so the computational complexity can be reduced. The LEA module is used to capture more local details, which can be seen as a complement to the GAA module. A deep convolution with a large kernel is performed on the depth features, and the resulting features are used as attention weights.

[0100] Furthermore, the first convolutional block and the fourth convolutional block have the same structure, including two global awareness attention modules and two local enhancement attention modules; the initial number of channels is 4, the channels are adjusted to 96, the number of attention heads is 2, the hidden layer dimension inside the MLP layer is 8 times the input dimension, a 7*7 window size is used for local attention operation, and then a downsampling operation is performed, and finally the dimension is rearranged to restore it to its original shape. The feature map sets output by the first and fourth convolutional blocks are recorded as The width of the feature map is Height is

[0101] The second convolutional block has the same structure as the fifth convolutional block, including two global awareness attention modules and two local enhancement attention modules; the initial number of channels is 96 and adjusted to 192, the number of attention heads is 4, the hidden layer dimension inside the MLP layer is 8 times the input dimension, and a 7*7 window size is used for local attention operation, followed by downsampling operation, and finally the dimension is rearranged to restore it to its original shape. The feature map sets of the second convolutional block and the fifth convolutional block are respectively recorded as The width of the feature map is Height is

[0102] The structure of the third convolutional block is the same as that of the sixth convolutional block, including eight global awareness attention modules and eight local enhancement attention modules; the initial number of channels is 192 and adjusted to 288, the number of attention heads is 8, the hidden layer dimension inside the MLP layer is 8 times the input dimension, and a 7*7 window size is used for local attention operation, followed by downsampling operation, and finally the dimension is rearranged to restore it to its original shape. The feature map sets of the third convolutional block and the sixth convolutional block are respectively recorded as The width of the feature map is Height is

[0103] The structure of the fourth convolutional block is the same as that of the eighth convolutional block, including two global awareness attention modules and two local enhancement attention modules; the initial number of channels is 288 and adjusted to 526, the number of attention heads is 16, the hidden layer dimension inside the MLP layer is 4 times the input dimension, and a 7*7 window size is used for local attention operation, followed by downsampling operation, and finally the dimension is rearranged to restore it to its original shape. The feature map sets of the fourth convolutional block and the eighth convolutional block are respectively recorded as The width of the feature map is Height is

[0104] For further information, see Figure 4As shown in the figure, the specific content of the cross-modal feature aggregation module is:

[0105] Input one and two are respectively passed through the twelfth convolution block and the thirteenth convolution block to obtain output five and output six. The pixels of output five and output six are multiplied and input into the first channel enhancer. The output generated by the first channel enhancer is multiplied with output five and output six again by pixel, and the generated output is added with output five and output six pixels respectively. Then, the first feature aggregation and the second feature aggregation are successively passed through. The output of the first feature aggregation and the output of the second feature aggregation are cascaded to obtain output seven. The output of output seven after passing through the first dilated convolution, the second dilated convolution, and the third dilated convolution is cascaded, and the result of the residual generated by the fourteenth convolution of output seven is added pixel by pixel to obtain output eight.

[0106] Specifically, the void ratios are 1, 4, 8, or other appropriate void ratios.

[0107] For further information, see Figure 5 As shown in the figure, the specific contents of the edge contour enhancement module are:

[0108] The input one is sequentially passed through the output of the fifteenth convolution block and the sixteenth convolution block, and the input one is sequentially passed through the output of the seventeenth convolution block and the eighteenth convolution block. The value obtained by cascading is multiplied with the auxiliary branch to obtain the output nine; the value obtained by pixel addition of the output nine and the output of the seventeenth convolution block, and the value obtained by pixel addition of the output nine and the output of the eighteenth convolution block are input into the linear fusion module as the variable of the linear fusion module, the first fusion factor and the second fusion factor are used as parameters, and finally pixel multiplication is performed with the input one to obtain the output ten;

[0109] The auxiliary branch is the result of input two being transposed after the first average pooling, then placed in the fully connected layer, and then pixel-wise multiplied with input two, and output nine is a weight.

[0110] For further information, see Figure 6 As shown in Figure 1, the specific contents of the spatial geometry module are:

[0111] Input one passes through the output of the first convolution block, and input two passes through the output of the second convolution block, and then is input into the nineteenth convolution block. After pixel subtraction, the output of the nineteenth convolution block is pixel-added with the output of the first convolution block to obtain output one, and the other is pixel-added with the output of the first convolution block after the Fourier transform filter to obtain output two.

[0112] The role of the spatial geometry module is to strengthen the fourth convolution block and the eighth convolution block. Therefore, the outputs of the fourth convolution block and the eighth convolution block are directly input into the first geometric space module and the second geometric space module and then returned to the fourth convolution block and the eighth convolution block.

[0113] For further information, see Figure 7 As shown, the convolutional neural network also includes a cascade distillation module. The specific content of the cascade distillation module is:

[0114] Input one is sequentially passed through the output of the first maximum pooling layer and the 21st convolution block, input two is sequentially passed through the output of the second maximum pooling layer and the 22nd convolution block, and input three is sequentially passed through the output of the third maximum pooling layer and the 23rd convolution block, and then cascaded and input into the 24th convolution block to obtain output one;

[0115] Input 1 passes through the output of the first maximum pooling layer and the 21st convolution block in sequence, input 2 passes through the output of the second maximum pooling layer and the 22nd convolution block in sequence, and output 1 is pixel-multiplied to obtain output 2;

[0116] Input 2 is sequentially passed through the output of the second maximum pooling layer and the 22nd convolution block, input 3 is sequentially passed through the output of the third maximum pooling layer and the 23rd convolution block, and output 1 is pixel-multiplied to obtain output 3;

[0117] After adding pixels of output 2 and output 3, the pixels are input into the 25th convolution block and pass through the activation function. The loss calculation with input 3 after the activation function is the process of the first cascade distillation.

[0118] See also Figure 8 As shown, it is the overall distillation framework diagram disclosed in the present invention. For the decoder, the decoders of the teacher and the student are the same. Both of them cascade the outputs of cross-modal feature aggregation. The output of the first cross-modal feature aggregation is cascaded with the output of the second cross-modal feature aggregation. After passing through the ninth convolution block, output one is generated. Then, output one is cascaded with the output of the third cross-modal feature aggregation and placed in the tenth convolution block to generate output two. Finally, output two is cascaded with the output of the fourth cross-modal feature aggregation and placed in the eleventh convolution block to generate the final output three.

[0119] For knowledge distillation, this paper allows students to learn the teacher's knowledge step by step, starting with cascade distillation. Figure 2 The output of the first cross-modal feature aggregation module in is the intermediate feature one, and so on, there are a total of four intermediate feature outputs. Figure 7As shown in the figure, the intermediate feature one of the teacher network is used as the "teaching assistant", that is, input two, the intermediate feature two of the higher layer is used as input one of the teacher network, and the intermediate feature two of the student network symmetrical to the teacher network is used as input three. They pass through the maximum pooling layer and the convolution block in turn, and then the three outputs are cascaded and put into the twenty-fourth convolution block as output one, which is the fusion weight of this article. Output one is multiplied with the pixels of the twenty-first convolution block and the twenty-third convolution block to obtain output two and output three. The pixels of output three and output two are added together, and the loss is calculated with the original input three after the activation function after the activation function. This is the process of the first cascade distillation. By analogy, three cascade distillations are completed, and the loss used is the residual attention loss. The output after the activation function is normalized, and then the two normalized attention mechanisms are subtracted to get their difference. Finally, the mean square error of this difference is calculated to minimize the difference between the teacher and the student.

[0120] For kernel pixel contrast learning, we first need to define a Gaussian kernel class to calculate the Gaussian kernel function value between two matrices, then initialize various parameters including contrast temperature, contrast size, etc., and define the projection head project_head to project the student features to the dimension of the teacher features; then normalize the student and teacher features, reshape the effective students and teachers into two-dimensional tensors, and initialize a pixel queue shared by the student and teacher to store the pixel embeddings generated by the teacher during the distillation process. For each image in the mini-batch, only randomly extract the pixel embeddings from the same class and add them according to the teacher features of the higher layer. The features are continuously pushed into the queue for updating, providing continuous negative samples and increasing the diversity of negative samples. In this experiment, the intermediate features of the students are used as anchors, and the corresponding intermediate features of the teachers are used as positive samples. The negative samples are composed of the original negative samples of the continuously updated queue and the mixed noise of Bayesian noise and Cauchy noise. After that, we calculate the similarity t_logits between the teacher's pixel-level features and the negative sample features, and the similarity s_logits between the student's pixel-level features and the negative sample features respectively through matrix multiplication. After obtaining the two similarities, we finally use the kernel contrast loss class defined by the implementation and accept s_logits s and t_logits, for each batch, their kernel function matrices K_s and K_t are calculated. KL divergence is first used to directly minimize the difference between the probability distribution of the student network output and the probability distribution of the teacher network output to transfer the knowledge of the teacher network. However, KL divergence cannot ensure that the internal representation of the student network is consistent with the teacher network. It is not enough to just match the output distribution. Therefore, the kernel loss method is used to make up for this shortcoming. That is, s_logits and t_logits are first softmaxed to obtain the probability distribution, and then the representation of the probability distribution of the student and teacher networks in the Gaussian kernel space is calculated. The difference matrix of the probability distribution of the student network and the teacher network in the kernel space is represented by the square root sum of the difference matrix. The kernel function can map the probability distribution to a higher-dimensional kernel space. By measuring the distance of the probability distribution in the kernel space, more flexible and indirect knowledge transfer can be achieved. Secondly, the kernel loss constrains the consistency of the probability distribution in the kernel space, and indirectly requires the internal representation of the student network to be consistent with that of the teacher network. The entire loss combines the KL divergence loss and the kernel loss, allowing the student network to directly learn the output knowledge of the teacher network (KL divergence) and indirectly obtain the internal representation and potential structural information of the teacher network (kernel loss). The two complement each other and help to more comprehensively transfer the knowledge of the teacher network, making the learning effect better.

[0121] S1-3. After data enhancement, each original mirror image and the corresponding depth map in the training set are used as the original input image and input into the convolutional neural network for training to obtain the corresponding predicted mirror image. The predicted mirror image includes a semantic segmentation predicted image and a boundary predicted image. The obtained predicted mirror scene image set is recorded as in, Predict a set of images for semantic segmentation, It is a set of boundary prediction images; the enhancement content includes: brightness, contrast, saturation, color adjustment and random flipping, cropping, etc.

[0122] S1-4. Calculate the loss function value between the semantic segmentation prediction image and the corresponding true semantic segmentation image; use the overlap task loss to calculate the true semantic segmentation image A collection of independently coded images The set of images composed of semantic segmentation prediction images The sub-loss function value between Where i = 1;

[0123] S1-5, repeat S1-3 and S1-4 for V times, until the convolutional neural network converges and the loss function value is minimized, and a convolutional neural network classification training model is obtained; a total of Q loss function values ​​are obtained during the training process, and the loss function value with a small value is found out, and the weight vector and bias term of the convolutional neural network currently obtained are used as the optimal weight vector and optimal bias term of the convolutional neural network classification training model, which are correspondingly recorded as W best and b best , where V>1; in this embodiment, V=300.

[0124] The testing phase includes the following steps:

[0125] S2-1, select the Pth group of mirror scene images to be detected in the test set and mark them as I p , where 1≤p≤P, P=392;

[0126] S2-2, various channels (I p The R channel component, G channel component, B channel component and depth (Depth) component of the original mirror scene image are input into the convolutional neural network classification training model. The model network outputs the mirror scene semantic segmentation prediction image corresponding to the original mirror scene image. The obtained mirror scene semantic segmentation prediction image is recorded as

[0127] In order to further verify the feasibility and effectiveness of the method of the present invention, experiments were conducted.

[0128] The Python-based deep learning library Pytorch3.6 is used to build a lightweight insight network for mirror semantic segmentation based on double contrast knowledge distillation. The test set of the MIINet-T mirror scene database is used to analyze the mirror scene segmentation effect using the method of the present invention. We use three commonly used objective parameters for evaluating semantic segmentation methods as evaluation indicators, namely, the binary error rate (Ber), the ratio of the intersection over union (IoU) of the segmented image and the labeled image, and the mean absolute error (Mae) to evaluate the segmentation performance of the predicted semantic segmentation image, as shown in Table 1. It can be seen from the data listed in Table 1 that the segmentation result of the mirror scene image obtained by the method of the present invention is good, indicating that it is feasible and effective to use the method of the present invention to obtain the predicted semantic segmentation image corresponding to the mirror scene image.

[0129] Table 1 Evaluation results of the method of the present invention on the test set

[0130] IoU BER MAE Ours-T 85.22 5.55 0.026 Ours-S 79.75 8.075 0.039 Ours-S* 82.72 6.73 0.031

[0131] Figure 9a The first original mirror scene image of the same scene is given; Figure 9b The method of the present invention is given Figure 9a The original mirror scene image shown is predicted to obtain a predicted semantic segmentation image; Fig.10a The second original mirror scene image of the same scene is given; Fig.10b The method of the present invention is given Fig.10a The original mirror scene image shown is predicted to obtain a predicted semantic segmentation image; Fig.11a The third original mirror scene image of the same scene is given; Fig.11b The method of the present invention is given Fig.11a The original mirror scene image shown is predicted to obtain a predicted semantic segmentation image; Fig.12a The fourth original mirror scene image of the same scene is given; Figure 12b The method of the present invention is given Fig.12a The original mirror scene image shown in Figure 1 is predicted to obtain the predicted semantic segmentation image. Figure 9a and Figure 9b ,contrast Fig.10a and Fig.10b ,contrast Fig.10a and Fig.10b ,contrast Fig.12a and Figure 12b It can be seen that the segmentation accuracy of the predicted semantic segmentation image obtained by the method of the present invention is high.

[0132] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0133] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image segmentation method based on dual contrast knowledge distillation and mirror semantic segmentation lightweight insight network, characterized in that: It includes two processes: training phase and testing phase; The training phase consists of the following steps: S1-1, select multiple original mirror scene images and corresponding depth maps, as well as real semantic segmentation images, and form a training set; S1-2, build a convolutional neural network; S1-3. After data enhancement, each original mirror image and the corresponding depth map in the training set are used as the original input image and input into the convolutional neural network for training to obtain the corresponding predicted mirror image. The predicted mirror image includes a semantic segmentation predicted image and a boundary predicted image. The obtained predicted mirror scene image set is recorded as in, Predict a set of images for semantic segmentation, A set of images for boundary prediction; S1-4, calculating the loss function value between the semantic segmentation prediction image and the corresponding true semantic segmentation image; S1-5, repeat S1-3 and S1-4 for V times in total until the convolutional neural network converges and the loss function value is minimized, and the convolutional neural network classification training model is obtained; the weight vector and bias term of the convolutional neural network currently obtained are used as the optimal weight vector and optimal bias term of the convolutional neural network classification training model, which are correspondingly recorded as W best and b best , where V>1; The testing phase includes the following steps: S2-1, select the Pth group of mirror scene images to be detected in the test set and mark them as I p , where 1≤p≤P; S2-2, input various channel components of the Pth group of mirror scene images to be detected into the convolutional neural network classification training model, and the model network outputs the mirror scene semantic segmentation prediction image corresponding to the original mirror scene image. The obtained mirror scene semantic segmentation prediction image is recorded as 2. According to claim 1, the image segmentation method of the mirror semantic segmentation lightweight insight network based on dual contrast knowledge distillation is characterized in that: Convolutional neural networks for feature extraction and feature fusion; The feature extraction operations of the convolutional neural network include: RGB image input layer, depth image input layer, twenty-five convolution blocks, eight edge contour enhancement modules ARM, four cross-modal feature aggregation modules DFFM and two geometric space modules GRF.

3. The image segmentation method according to claim 2, characterized in that: The convolutional neural network includes a teacher network and a student network; wherein the RGB image is input to the first convolution block and the depth image is input to the fifth convolution block.

4. The image segmentation method according to claim 3, characterized in that: The teacher network consists of: eleven convolutional blocks, eight edge contour enhancement modules ARM, four cross-modal feature aggregation modules DFFM and two geometric space modules GRF; The first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the first geometric module are connected in sequence; the fifth convolution block, the sixth convolution block, the seventh convolution block, the eighth convolution block, and the second geometric module are connected in sequence; An output end of the first convolution block is connected to an input end of the first edge contour enhancement module, an output end of the first edge contour enhancement module is connected to a first input end of the first cross-modal feature aggregation module, an output end of the fifth convolution block is connected to an input end of the fifth edge contour enhancement module, and an output end of the fifth edge contour enhancement module is connected to a second input end of the first cross-modal feature aggregation module; The output end of the second convolution block is connected to the first input end of the second edge contour enhancement module, the output end of the first cross-modal feature aggregation module is connected to the second input end of the second edge contour enhancement module, and the output end of the second edge contour enhancement module is connected to the first input end of the second cross-modal feature aggregation module; the output end of the sixth convolution block is connected to the first input end of the sixth edge contour enhancement module, the output end of the first cross-modal feature aggregation module is connected to the second input end of the sixth edge contour enhancement module, and the output end of the sixth edge contour enhancement module is connected to the second input end of the second cross-modal feature aggregation module; The output end of the third convolution block is connected to the first input end of the third edge contour enhancement module, the output end of the second cross-modal feature aggregation module is connected to the second input end of the third edge contour enhancement module, and the output end of the third edge contour enhancement module is connected to the first input end of the third cross-modal feature aggregation module; the output end of the seventh convolution block is connected to the first input end of the seventh edge contour enhancement module, the output end of the second cross-modal feature aggregation module is connected to the second input end of the seventh edge contour enhancement module, and the output end of the seventh edge contour enhancement module is connected to the second input end of the third cross-modal feature aggregation module; The output end of the fourth convolution block and the output end of the first geometric space module are both connected to the first input end of the fourth edge contour enhancement module, the output end of the third cross-modal feature aggregation module is connected to the second input end of the fourth edge contour enhancement module, the output end of the fourth edge contour enhancement module is connected to the first input end of the fourth cross-modal feature aggregation module, the output end of the eighth convolution block is connected to the output end of the second space geometry module and the first input end of the eighth edge contour enhancement module, the output end of the third modal feature aggregation module is connected to the second input end of the eighth edge contour enhancement module, and the output end of the eighth edge contour enhancement module is connected to the second input end of the fourth cross-modal feature aggregation module; The output of the first cross-modal feature aggregation module and the output of the second cross-modal feature aggregation module are cascaded and input into the ninth convolution block, the output one of the ninth convolution block and the output of the third cross-modal feature aggregation module are cascaded and input into the tenth convolution block, the output two of the tenth convolution block and the output of the fourth cross-modal feature aggregation module are cascaded and input into the input of the eleventh convolution block, and the output of the eleventh convolution block is output three.

5. The image segmentation method of the mirror semantic segmentation lightweight insight network based on dual contrast knowledge distillation according to claim 4 is characterized in that: The student network consists of: eleven convolutional blocks and four cross-modal feature aggregation modules (DFFMs) that are the same as the teacher network; The first convolution block, the second convolution block, the third convolution block, and the fourth convolution block are connected in sequence; the fifth convolution block, the sixth convolution block, the seventh convolution block, and the eighth convolution block are connected in sequence; An output end of the first convolution block is connected to a first input end of the first cross-modal feature aggregation module, and an output end of the fifth convolution block is connected to a second input end of the first cross-modal feature aggregation module; An output end of the second convolution block is connected to a first input end of the second cross-modal feature aggregation module, and an output end of the sixth convolution block is connected to a second input end of the second cross-modal feature aggregation module; An output end of the third convolution block is connected to a first input end of the third cross-modal feature aggregation module, and an output end of the seventh convolution block is connected to a second input end of the third cross-modal feature aggregation module; An output end of the fourth convolution block is connected to a first input end of the fourth cross-modal feature aggregation module, and an output end of the eighth convolution block is connected to a second input end of the fourth cross-modal feature aggregation module; The output of the first cross-modal feature aggregation module and the output of the second cross-modal feature aggregation module are cascaded and input into the ninth convolution module. The output one of the ninth convolution module and the output of the third cross-modal feature aggregation module are both input into the tenth convolution module. The output two of the tenth convolution block and the output of the fourth cross-modal feature aggregation module are cascaded and input into the eleventh convolution module. The output of the eleventh convolution module is output three.

6. The image segmentation method of the mirror semantic segmentation lightweight insight network based on dual contrast knowledge distillation according to claim 5 is characterized in that: The first and fourth convolutional blocks have the same structure, including two global awareness attention modules and two local enhancement attention modules; The second convolutional block has the same structure as the fifth convolutional block, including two global awareness attention modules and two local enhancement attention modules; The third convolutional block has the same structure as the sixth convolutional block, including eight global awareness attention modules and eight local enhancement attention modules; The fourth convolutional block has the same structure as the eighth convolutional block, including two global awareness attention modules and two local enhancement attention modules.

7. The image segmentation method according to claim 3, characterized in that: The specific content of the cross-modal feature aggregation module is: Input one and two are respectively passed through the twelfth convolution block and the thirteenth convolution block to obtain output five and output six. The pixels of output five and output six are multiplied and input into the first channel enhancer. The output generated by the first channel enhancer is multiplied with output five and output six again by pixel, and the generated output is added with output five and output six pixels respectively. Then, the first feature aggregation and the second feature aggregation are successively passed through. The output of the first feature aggregation and the output of the second feature aggregation are cascaded to obtain output seven. The output of output seven after passing through the first dilated convolution, the second dilated convolution, and the third dilated convolution is cascaded, and the result of the residual generated by the fourteenth convolution of output seven is added pixel by pixel to obtain output eight.

8. The image segmentation method according to claim 3, characterized in that: Specific contents of the edge contour enhancement module: The input one is sequentially passed through the output of the fifteenth convolution block and the sixteenth convolution block, and the input one is sequentially passed through the output of the seventeenth convolution block and the eighteenth convolution block. The value obtained by cascading is multiplied with the auxiliary branch to obtain the output nine; the value obtained by pixel addition of the output nine and the output of the seventeenth convolution block, and the value obtained by pixel addition of the output nine and the output of the eighteenth convolution block are input into the linear fusion module as the variable of the linear fusion module, the first fusion factor and the second fusion factor are used as parameters, and finally pixel multiplication is performed with the input one to obtain the output ten; The auxiliary branch is the result of input two being transposed after the first average pooling, then placed in the fully connected layer, and then pixel-wise multiplied with input two, and output nine is a weight.

9. The image segmentation method according to claim 3, characterized in that: The specific contents of the spatial geometry module are: Input one passes through the output of the first convolution block, and input two passes through the output of the second convolution block, and then is input into the nineteenth convolution block. After pixel subtraction, the output of the nineteenth convolution block is pixel-added with the output of the first convolution block to obtain output one, and the other is pixel-added with the output of the first convolution block after the Fourier transform filter to obtain output two.

10. The image segmentation method of the mirror semantic segmentation lightweight insight network based on dual contrast knowledge distillation according to claim 3 is characterized in that: The convolutional neural network also includes a cascade distillation module. The specific content of the cascade distillation module is: Input one is sequentially passed through the output of the first maximum pooling layer and the 21st convolution block, input two is sequentially passed through the output of the second maximum pooling layer and the 22nd convolution block, and input three is sequentially passed through the output of the third maximum pooling layer and the 23rd convolution block, and then cascaded and input into the 24th convolution block to obtain output one; Input 1 passes through the output of the first maximum pooling layer and the 21st convolution block in sequence, input 2 passes through the output of the second maximum pooling layer and the 22nd convolution block in sequence, and output 1 is pixel-multiplied to obtain output 2; Input 2 is sequentially passed through the output of the second maximum pooling layer and the 22nd convolution block, input 3 is sequentially passed through the output of the third maximum pooling layer and the 23rd convolution block, and output 1 is pixel-multiplied to obtain output 3; After adding pixels of output 2 and output 3, the pixels are input into the 25th convolution block and pass through the activation function. The loss calculation with input 3 after the activation function is the process of the first cascade distillation.

Citation Information

Cited By

  • Open scene instance segmentation method based on decoupling distillation

    CN119888223A

  • An open scene instance segmentation method based on decoupled distillation

    CN119888223B

  • Lightweight multi-modal foundation cloud picture recognition and prediction method

    CN119992190A