Image segmentation method and device, equipment, storage medium and computer program product

Abstract

The invention provides an image segmentation method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: carrying out the processing of N layers of feature maps of image data, and obtaining N layers of image features; the N-layer feature map is output by an N-layer encoder of the visual self-supervision model; extracting N layers of depth features from depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features; performing feature fusion on the N layers of depth features and the corresponding N layers of image features, and performing layer-by-layer fusion from top to bottom to obtain global fusion features; and performing image segmentation based on the global fusion feature, and outputting an image segmentation result. The generalization ability of image segmentation can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image segmentation method and apparatus, equipment, storage medium, and computer program product. Background Art

[0002] Current humanoid robots widely use RGBD cameras, which can simultaneously acquire color images (RGB) and depth information (Depth), providing multimodal input for complex scene perception. Traditional RGB segmentation models rely on low-dimensional features such as texture and color. These models are prone to misjudgment in complex scenarios such as lighting changes, occlusions, and texture similarities, resulting in incomplete segmentation of target objects and insufficient generalization capabilities for image segmentation. Summary of the Invention

[0003] The present application provides an image segmentation method, apparatus, device, storage medium, and computer program product, which can improve the generalization capability of image segmentation.

[0004] The technical solution of this application is achieved as follows:

[0005] In a first aspect, the present application proposes an image segmentation method, the method comprising:

[0006] Processing N layers of feature maps of the image data respectively to obtain N layers of image features; the N layers of feature maps are output by the N layers of encoders of the visual self-supervisory model;

[0007] Extracting N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features;

[0008] Performing feature fusion on the N layers of depth features and the corresponding N layers of image features, and fusing them layer by layer from top to bottom to obtain a global fusion feature;

[0009] Perform image segmentation based on the global fusion features and output an image segmentation result.

[0010] In a second aspect, the present application proposes an image segmentation device, comprising:

[0011] An image feature processing module is used to process N layers of feature maps of the image data to obtain N layers of image features; the N layers of feature maps are output by the N layers of encoders of the visual self-supervisory model;

[0012] A depth feature extraction module is configured to extract N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are aligned with the feature sizes of the N layers of image features;

[0013] A feature fusion module is used to fuse the N layers of depth features with the corresponding N layers of image features, and fuse them layer by layer from top to bottom to obtain a global fusion feature;

[0014] The image segmentation module is used to perform image segmentation based on the global fusion feature and output the image segmentation result.

[0015] In a third aspect, the present application provides an electronic device, comprising:

[0016] a memory for storing computer-executable instructions or computer programs;

[0017] The processor is configured to implement the above-mentioned image segmentation method when executing the computer executable instructions or computer program stored in the memory.

[0018] In a fourth aspect, the present application proposes a computer-readable storage medium storing computer-executable instructions or a computer program, which implements the above-mentioned image segmentation method when executed by a processor.

[0019] In a fifth aspect, the present application proposes a computer program product, comprising computer-executable instructions or a computer program, which implements the above-mentioned image segmentation method when executed by a processor.

[0020] The present application provides an image segmentation method, apparatus, device, storage medium, and computer program product. The method comprises: processing N layers of feature maps of image data to obtain N layers of image features; the N layers of feature maps are output by an N-layer encoder of a visual self-supervised model; extracting N layers of depth features from depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are aligned with the feature sizes of the N layers of image features; performing feature fusion on the N layers of depth features and the corresponding N layers of image features, and fusing them layer by layer from top to bottom to obtain a global fused feature; performing image segmentation based on the global fused feature, and outputting an image segmentation result. By adopting the above implementation scheme, for the N-layer feature maps output by the N-layer encoder of the visual self-supervised model, feature reorganization can integrate multi-scale features to form a feature pyramid with hierarchical expression capabilities. Subsequent sampling can optimize the resolution of the features, thereby generating N-layer image features with a hierarchical structure; the feature fusion of the N-layer image features and the corresponding N-layer depth features can combine image features with different semantics and depth features that can filter the main area for image segmentation, more accurately mark targets and identify target edges, greatly improve the accuracy and integrity of image segmentation, and thus improve the generalization ability of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A flowchart of an image segmentation method provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of an exemplary image segmentation model provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the module composition of an exemplary feature fusion module provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the structure of an image segmentation device provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0029] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0031] The present application embodiment provides an image segmentation method, such as Figure 1 As shown, the method may include:

[0032] S101. Process N layers of feature maps of the image data respectively to obtain N layers of image features; the N layers of feature maps are output by the N layers of encoders of the visual self-supervision model.

[0033] In one embodiment, the visual self-supervision model can be a DINOV2 network, and accordingly, the encoder is a transformer encoder. Specifically, other networks can be selected as visual self-supervision models and feature extraction modules of other network applications can be selected as encoders according to actual conditions, and this embodiment of the application does not make specific limitations. The following description takes the visual self-supervision model as a DINOV2 network and the encoder as a transformer encoder as an example.

[0034] In one embodiment, the DINOV2 network is a network with a vison transformer (ViT) as its core, which achieves high-performance visual feature extraction through self-supervised learning. The DINOV2 network can be composed of 12 stacked transformer encoders. The image data is input into the first transformer encoder and processed by the 12 transformer encoders in sequence. Each transformer encoder can output a layer of features.

[0035] The embodiment of the present application is described using the DINOV2 network including 12 transformer encoders as an example. It should be noted that the number of transformer encoders included in the DINOV2 network can be 12, 24, 32 or other numbers, which can be adjusted according to actual conditions and is not specifically limited in the embodiment of the present application.

[0036] In one embodiment, N transformer encoders may be selected from the 12 transformer encoders to obtain N layers of feature maps outputted by the transformer encoders. The strategy for selecting N transformer encoders from the 12 transformer encoders may be adaptively selected based on actual conditions, and this application does not further limit the selection strategy.

[0037] It should be noted that N is a positive integer.

[0038] Exemplarily, N is taken as 4, and the features output by the 3rd transformer encoder, the 6th transformer encoder, the 9th transformer encoder, and the 12th transformer encoder in the 12 transformer encoders are respectively obtained as 4-layer features, where the 3rd transformer encoder outputs the first layer features, the 6th transformer encoder outputs the second layer features, the 9th transformer encoder outputs the third layer features, and the 12th transformer encoder outputs the fourth layer features.

[0039] It should be noted that the feature dimensions of N layers of feature maps can be the same. For example, taking the features output by the 3rd transformer encoder, the 6th transformer encoder, the 9th transformer encoder, and the 12th transformer encoder as an example, the feature dimensions of the corresponding 4 layers of output features can be 1×1024×768.

[0040] It should be noted that the feature dimensions of the N layers of feature maps are the same, but the semantic levels gradually deepen. For example, the feature maps of the initial layers (such as layers 1-3) mainly reflect low-level information such as local texture and edges, which are suitable for tasks such as super-resolution and denoising. The feature maps of the intermediate layers (such as layers 4-9) gradually aggregate the global context and begin to capture semantic information (such as object outlines and component relationships), which can detect small targets such as pedestrians and traffic signs. The feature maps of the deep layers (such as layers 10-12) output high-level semantic features, which contain global information such as scene categories and target locations, and can distinguish between roads, buildings, vegetation, etc.

[0041] In an embodiment of the present application, after obtaining the N layers of feature maps output by the N layers of encoder of the visual self-supervision model, the N layers of feature maps must be processed separately to obtain N layers of image features.

[0042] Specifically, feature reorganization and sampling are performed on N layers of feature maps in sequence to obtain N layers of image features.

[0043] In an embodiment of the present application, the first layer feature map to the Nth layer feature map are sequentially reorganized according to the number of N visual self-supervised model channels from shallow to deep to obtain N layers of intermediate features; thereafter, the N layers of intermediate features are adaptively sampled in order from high to low image resolution to obtain N layers of image features.

[0044] In an embodiment of the present application, according to the number of N visual self-supervised model channels from shallow to deep, the first-layer feature map to the N-th layer feature map are sequentially feature reorganized to obtain N layers of intermediate features: the N-layer feature maps are dimensionally expanded to obtain N layers of spatial features; according to the number of N visual self-supervised model channels from shallow to deep, the first-layer spatial features to the N-th layer spatial features are sequentially feature extracted and then channel compressed to obtain N layers of intermediate features.

[0045] It should be noted that dimensional expansion is the process of feature reshaping, which means reshaping the data dimensions without changing the data content. The purpose of reshaping is to facilitate subsequent convolution operations. Convolution operations are reflected in the subsequent feature extraction, channel compression adaptive sampling, and feature fusion processes.

[0046] In the embodiment of the present application, the N-layer feature map is a three-dimensional feature, and the three-dimensional feature can be expanded into a four-dimensional spatial feature through the feature reshaping process.

[0047] For example, the 1×1024×768 three-dimensional features are dimensionally expanded and split to obtain 1×768×32×32 four-dimensional spatial features.

[0048] It should be noted that feature extraction and channel compression together constitute the projection process, which can be implemented by two convolution operations. The first convolution operation is used to gradually extract features from the input channel number and expand them to the deeper channel number. Specifically, feature extraction is performed on N layers of spatial features in order from shallow to deep channel number. The second convolution operation is used to compress the number of channels while maintaining spatial resolution. Specifically, it restores the image features of N layers with the same number of channels. Both the first and second convolutions can be a 3×3 convolution.

[0049] In the embodiment of the present application, the number of N channels from shallow to deep can be determined based on the number of channels of the N-layer spatial features and N. The specific number can be selected according to actual conditions and is not specifically limited in the embodiment of the present application.

[0050] For example, for the 4-layer spatial feature 1×768×32×32, the number of channels of the visual self-supervised model is 768, then 768 is averaged 4-1 times to obtain the 4 visual self-supervised model channel numbers {96, 192, 384, 768} from shallow to deep.

[0051] In one embodiment, the first convolution operation extracts features of the four-layer spatial features 1×768×32×32 in sequence according to the number of channels of the visual self-supervised model {96, 192, 384, 768}, and obtains the four features of 1x96x32x32, 1x196x32x32, 1x384x32x32, and 1x768x32x32 respectively; then, the four features of 1x96x32x32, 1x196x32x32, 1x384x32x32, and 1x768x32x32 are channel compressed to restore the N-layer intermediate features of 1x128x32x32 with 128 channels.

[0052] It should be noted that after obtaining N layers of intermediate features, the N layers of intermediate features can be adaptively sampled to obtain N layers of image features. The N layers of intermediate features can be adaptively sampled in descending order of image resolution to obtain N layers of image features.

[0053] In the embodiment of the present application, the design of the adaptive sampling portion mimics the image resolution in the field of image segmentation, and sets different sampling rates for the N layers of intermediate features. The sampling rate may include a sampling multiple and a sampling type. The sampling multiple can be 8x, 4x, 2x, 1x, etc., and the specific selection can be based on actual conditions, and the embodiment of the present application does not make specific restrictions. The sampling type can be upsampling or downsampling, and the specific selection can be based on actual conditions, and the embodiment of the present application does not make specific restrictions.

[0054] It should be noted that to achieve higher image resolution, the corresponding sampling rate is higher. For shallow features, the image resolution is set higher, and for deep features, the image resolution is set lower. Since the N-layer intermediate features are in the order of dimension from shallow to deep, the corresponding image resolution is set in the order from high to low.

[0055] In one embodiment, deconvolution 4x upsampling, deconvolution 2x upsampling, retention, and 2x downsampling may be sequentially set for the 4-layer intermediate features.

[0056] Exemplarily, the first-layer intermediate features 1x128x32x32 are deconvolved and upsampled by 4 times to obtain the first-layer image features 1x128x128x128; the second-layer intermediate features 1x128x32x32 are deconvolved and upsampled by 2 times to obtain the second-layer image features 1x128x64x64, the third-layer intermediate features 1x128x32x32 are maintained to obtain the third-layer image features 1x128x32x32, and the fourth-layer intermediate features 1x128x32x32 are downsampled by 2 times to obtain the fourth-layer image features 1x128x16x16.

[0057] It can be understood that by processing the N layers of feature maps output by the N-layer transformer encoder of the DINOV2 network separately, the N layers of feature maps can be further processed without destroying the DINOV2 network structure, retaining the generalization ability of the DINOV2 network while forming a hierarchical feature representation, enhancing the model's ability to understand the global and local information of the image, and improving the feature quality of self-supervised learning.

[0058] S102 , extracting N layers of depth features from the depth map data corresponding to the image data; wherein feature sizes of the N layers of depth features are aligned with feature sizes of the N layers of image features.

[0059] In an embodiment of the present application, the depth map data is first downsampled N times to obtain N layers of depth map data; the sampling multiples of the N downsampling operations are determined according to the feature sizes of the N layers of image features; then, depth feature extraction is performed on the N layers of depth map data to obtain N layers of depth features.

[0060] It should be noted that for the depth map data, the sizes of the N layers of image features are first aligned through N downsampling operations to achieve subsequent feature fusion. After that, the depth features of the N layers of depth map data are extracted to obtain N layers of depth features.

[0061] In one embodiment, the depth map data is downsampled by a factor of 4 to match the size of the first layer image features; the depth map data is downsampled by a factor of 8 to match the size of the second layer image features; the depth map data is downsampled by a factor of 16 to match the size of the third layer image features; and the depth map data is downsampled by a factor of 32 to match the size of the fourth layer image features.

[0062] In an embodiment of the present application, a convolutional network can be used to extract depth features from N layers of depth map data.

[0063] Optionally, the convolutional network may be a convolutional network with five 3×3 convolutions and 128 channels.

[0064] S103 , respectively fuse the N layers of depth features with the corresponding N layers of image features, and fuse them layer by layer from top to bottom to obtain a global fusion feature.

[0065] In an embodiment of the present application, N layers of image features are concatenated with corresponding N layers of depth features to obtain N layers of fused features; and the N layers of fused features are sequentially upsampled and added from top to bottom to obtain global fused features. Optionally, the above method can be implemented by an N-layer feature map fusion unit. Specifically, the N layers of depth features and the N layers of image features are respectively input into the N-layer feature map fusion unit for feature fusion, and the N-layer feature map fusion units are sequentially connected from top to bottom to perform top-down layer-by-layer fusion.

[0066] In one embodiment, since the sizes of image features of two adjacent layers differ by 2 times, the upsampling operation may be specifically a 2-fold upsampling.

[0067] In an embodiment of the present application, each layer of feature fusion unit is composed of two convolution units, a first convolution unit and a second convolution unit, an adder, an upsampling unit, and a Project unit. Among them, the first convolution unit is used to perform convolution processing on the image features of the layer; for the first layer of feature fusion unit, its adder is used to add the depth features of the layer and the image features of the layer after convolution processing to obtain fusion features; for other layers of feature fusion units, its adder is used to add the depth features of the layer, the image features of the layer after convolution processing, and the features output from the previous layer to obtain fusion features; the fusion features are processed by the second convolution unit, the upsampling unit, and the Project unit in turn and output to the adder of the feature fusion unit of the next layer; until the Project unit of the feature fusion unit of the last layer outputs the global fusion features.

[0068] It should be noted that the above Project unit is composed of a common convolution, which can be a 3×3 convolution, for maintaining the dimension.

[0069] In one embodiment, the size of the global fusion feature may be 1×128×256×256.

[0070] S104: Perform image segmentation based on the global fusion features and output the image segmentation result.

[0071] In the embodiment of the present application, the global fusion features can be classified at the pixel level by the Segmentation Head module to output the image segmentation result, which can include the image segmentation mask and the category label.

[0072] It should be noted that the Seg Head is the core module in deep learning image segmentation models responsible for converting feature maps into pixel-level classification predictions. Its design directly affects segmentation accuracy and efficiency, and is commonly used in tasks such as object detection, instance segmentation, and medical image analysis. Located at the end of the model, the Seg Head receives globally fused features, restores spatial resolution through upsampling or a decoder, and ultimately generates a segmentation mask of the same size as the input image data. It also outputs pixel-level category predictions (such as a binary classification mask or a multi-class semantic label).

[0073] In one embodiment, the Seg Head is composed of a common convolution, which may be a 3×3 convolution, and its output channel is the category to be finally segmented.

[0074] In one embodiment, the Seg Head module is used to perform pixel-level classification on the image 1×128×256×256 to obtain an image segmentation result of 1×class_num×256×256, where class_num is the category of the image segmentation.

[0075] It can be understood that for the N-layer feature maps output by the N-layer encoder of the visual self-supervised model, feature reorganization can integrate multi-scale features to form a feature pyramid with hierarchical expression capabilities, and then sampling can optimize the resolution of the features, thereby generating N-layer image features with a hierarchical structure; the feature fusion of N-layer image features and corresponding N-layer depth features can combine image features with different semantics and depth features that can filter the main area for image segmentation, more accurately mark targets and identify target edges, greatly improve the accuracy and completeness of image segmentation, and thus improve the generalization ability of image segmentation.

[0076] It should be noted that the above method can be implemented by an image segmentation model, wherein the image segmentation model includes at least one of the following modules: an image feature processing module for processing N layers of feature maps separately, a depth feature extraction module for extracting N layers of depth features from depth map data, a feature fusion module for fusing the N layers of depth features with the corresponding N layers of image features and fusing them layer by layer from top to bottom, and an image segmentation module for performing image segmentation based on the global fusion feature. The N layers of feature maps and depth map data of the image data can be input into the image segmentation model to obtain an image segmentation result.

[0077] It should be noted that the image segmentation model may also include a DINOV2 network to process the image data to obtain N layers of feature maps.

[0078] The above-mentioned image segmentation model can be obtained through model training. During the initialization process of training, the convolution kernel parameters of the deep feature extraction module can be initialized to zero, and then the model training is performed to obtain the image segmentation model.

[0079] It can be understood that by setting the convolution kernel parameters of the deep feature extraction module to zero and then initializing the model for training, it is possible to ensure that the main branch fully utilizes the generalization function of DINOV2 in the initial stage of training, and then gradually perform transitional learning according to the deep features, gradually superimposing the guiding information of the deep features, and improving the accuracy and generalization of image segmentation. This can enable the image segmentation model to enhance the accuracy and generalization of image segmentation without destroying the generalization ability of the main branch.

[0080] It should be noted that the convolution kernel parameters can include weights (W) and bias terms (b). Among them, W is the numerical matrix of the convolution kernel, which is used to extract local features (such as edges, textures, etc.); b is a single number used to adjust the global offset of the convolution output.

[0081] It should be noted that the zero initialization for the convolution kernel parameters can be to initialize W to a zero matrix and initialize b to a zero vector.

[0082] It should be noted that the above convolution kernel parameters are the convolution kernel parameters of the convolution network used in the depth feature extraction module to perform depth feature extraction on N layers of depth map data.

[0083] During the model training phase, a joint loss function may be used, including at least one of cross entropy loss (L_CE), Dice loss (L_Dice), and depth consistency loss (L_Depth). Specific selections may be made based on actual conditions and are not specifically limited in this application embodiment.

[0084] It should be noted that during the model training phase, after obtaining the image segmentation training results through Seg Head, the image segmentation training results need to be upsampled to the training input size before performing the above loss calculation.

[0085] For example, if the training input size is 448, the result obtained by Seg Head needs to be upsampled to 448 to perform the above loss calculation.

[0086] Based on the above embodiment, an image segmentation model is exemplarily proposed, see Figure 2The image segmentation model consists of a DINOV2 network, an image feature processing module, a deep feature extraction module, a feature fusion module, and an image segmentation module. The four layers of features output by the 3rd transformer encoder, the 6th transformer encoder, the 9th transformer encoder, and the 12th transformer encoder in the DINOV2 network are obtained. The image feature processing module includes a 4-layer feature reorganization unit connected to the 4 transformer encoders, a deconvolution 4x upsampling unit connected to the 1st layer feature reorganization unit, a deconvolution 2x upsampling unit connected to the 2nd layer feature reorganization unit, and a 2x downsampling unit connected to the 4th layer feature reorganization unit. The deep feature extraction module includes 4 downsampling units, 8x downsampling units, 16x downsampling units, and 32x downsampling units. A convolutional network 1 connected to a 4x downsampling unit, a convolutional network 2 connected to an 8x downsampling unit, a convolutional network 3 connected to a 16x downsampling unit, and a convolutional network 4 connected to a 32x downsampling unit. The feature fusion module includes 4 layers of feature fusion units cascaded from top to bottom to achieve feature fusion layer by layer. The 4th layer feature fusion unit is also connected to the 2x downsampling unit and the convolutional network 4 respectively to achieve feature splicing of this layer. The 3rd layer feature fusion unit is also connected to the 3rd layer feature recombination unit and the convolutional network 3 respectively to achieve feature splicing of this layer. The 2nd layer feature fusion unit is also connected to the deconvolution 2x upsampling unit and the convolutional network 2 respectively to achieve feature splicing of this layer. The 1st layer feature fusion unit is also connected to the deconvolution 4x upsampling unit and the convolutional network 1 respectively to achieve feature splicing of this layer. The image segmentation module can be Seg Head.

[0087] See also Figure 2 ,like Figure 3 As shown in the figure, the feature fusion unit consists of convolution unit 1, convolution unit 2, an adder, a 2x upsampling unit, and a project unit. The image features of each layer are input into the adder through convolution unit 1, and the depth features of each layer are also input into the adder. The adder is sequentially connected to convolution unit 2, the 2x upsampling unit, and the project unit.

[0088] It should be noted that, for the 4th, 3rd, and 2nd layer feature fusion units, the output end of the Project unit is connected to the input end of the adder of the next layer feature fusion unit (not shown); for the 1st layer feature fusion unit, the output end of the Project unit is connected to the Seg Head (not shown).

[0089] The embodiment of the present application provides an image segmentation device. Figure 4 As shown, the image segmentation device 1 includes:

[0090] An image feature processing module 10 is configured to process N layers of feature maps of the image data to obtain N layers of image features; the N layers of feature maps are output by the N layers of encoders of the visual self-supervisory model;

[0091] A depth feature extraction module 11 is configured to extract N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are aligned with the feature sizes of the N layers of image features;

[0092] A feature fusion module 12 is configured to fuse the N layers of depth features with the corresponding N layers of image features, and fuse them layer by layer from top to bottom to obtain a global fusion feature;

[0093] The image segmentation module 13 is configured to perform image segmentation based on the global fusion features and output an image segmentation result.

[0094] Optionally, the image feature processing module 10 is also used to perform feature reorganization on the first layer feature map to the Nth layer feature map in sequence according to the number of N visual self-supervised model channels from shallow to deep, to obtain N layers of intermediate features; and adaptively sample the N layers of intermediate features in order from high to low image resolution to obtain N layers of image features.

[0095] Optionally, the depth feature extraction module 11 is also used to downsample the depth map data N times to obtain N layers of depth map data; the sampling multiples of the N downsampling are determined according to the feature sizes of the N layers of image features; and depth feature extraction is performed on the N layers of depth map data to obtain N layers of depth features.

[0096] Optionally, the image segmentation device may further include an image segmentation model, and the image segmentation device is further used to input the N-layer feature map of the image data and the depth map data into the image segmentation model to obtain the image segmentation result; the image segmentation model includes at least one of the following modules: an image feature processing module for processing the N-layer feature map respectively, a depth feature extraction module for extracting the N-layer depth features from the depth map data, a feature fusion module for fusing the N-layer depth features with the corresponding N-layer image features respectively and fusing them layer by layer from top to bottom, and an image segmentation module for performing image segmentation based on the global fusion feature.

[0097] Optionally, the device further comprises: a training module;

[0098] The training module is used to initialize the convolution kernel parameters of the depth feature extraction module to zero and then perform model training to obtain the image segmentation model.

[0099] Optionally, the image feature processing module 10 is also used to perform dimension expansion on the N layers of feature maps to obtain N layers of spatial features; and perform channel compression on the first layer of spatial features to the Nth layer of spatial features in sequence according to the number of channels of the N visual self-supervised models from shallow to deep to obtain the N layers of intermediate features.

[0100] The present invention provides an image segmentation device that processes N layers of feature maps of image data to obtain N layers of image features; the N layers of feature maps are output by an N-layer encoder of a visual self-supervised model; extracts N layers of depth features from depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are aligned with the feature sizes of the N layers of image features; performs feature fusion on the N layers of depth features and the corresponding N layers of image features, and fuses them layer by layer from top to bottom to obtain global fusion features; performs image segmentation based on the global fusion features and outputs an image segmentation result. It can be seen that the image segmentation device proposed in this embodiment, for the N layers of feature maps output by the N layers of encoder of the visual self-supervised model, performs feature reorganization on them to integrate multi-scale features to form a feature pyramid with hierarchical expression capability, and then performs sampling to optimize the resolution of the features, thereby generating N layers of image features with a hierarchical structure; the feature fusion of the N layers of image features and the corresponding N layers of depth features can combine image features with different semantics and depth features that can filter the main area for image segmentation, more accurately annotate objects and identify object edges, greatly improve the accuracy and integrity of image segmentation, and thus improve the generalization ability of image segmentation.

[0101] Figure 5 A schematic diagram of the structure of an electronic device 2 provided in an embodiment of the present application Figure 2 In practical applications, based on the same disclosure concept of the above embodiments, as Figure 5 As shown, the electronic device 2 of this embodiment includes: a processor 21 , a memory 22 and a communication bus 23 .

[0102] The processor 21 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the processor function may also be other, and this embodiment does not specifically limit this.

[0103] In the embodiment of the present application, the communication bus 23 is used to realize the connection and communication between the processor 21 and the memory 22; when the processor 21 executes the running program stored in the memory 22, the following image segmentation method is implemented:

[0104] N layers of feature maps of image data are processed separately to obtain N layers of image features; the N layers of feature maps are output by the N layers of encoder of the visual self-supervisory model; N layers of depth features are extracted from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are aligned with the feature sizes of the N layers of image features; feature fusion is performed on the N layers of depth features and the corresponding N layers of image features, and the fusion is performed layer by layer from top to bottom to obtain a global fusion feature; image segmentation is performed based on the global fusion feature, and an image segmentation result is output.

[0105] Furthermore, the above-mentioned processor 21 is also used to perform feature reorganization on the first layer feature map to the Nth layer feature map in sequence according to the number of N visual self-supervised model channels from shallow to deep, so as to obtain N layers of intermediate features; and adaptively sample the N layers of intermediate features in order from high to low image resolution to obtain N layers of image features.

[0106] Furthermore, the above-mentioned processor 21 is also used to downsample the depth map data N times to obtain N layers of depth map data; the sampling multiples of the N downsampling are determined according to the feature sizes of the N layers of image features; and the depth features of the N layers of depth map data are extracted to obtain N layers of depth features.

[0107] Furthermore, the above-mentioned processor 21 is also used to input the N-layer feature map of the image data and the depth map data into the image segmentation model to obtain the image segmentation result; the image segmentation model includes at least one of the following modules: an image feature processing module for processing the N-layer feature map respectively, a depth feature extraction module for extracting the N-layer depth features from the depth map data, a feature fusion module for performing feature fusion of the N-layer depth features with the corresponding N-layer image features and fusing them layer by layer from top to bottom, and an image segmentation module for performing image segmentation based on the global fusion feature.

[0108] Furthermore, the processor 21 is further configured to perform model training after initializing the convolution kernel parameters of the depth feature extraction module to zero, so as to obtain the image segmentation model.

[0109] Furthermore, the above-mentioned processor 21 is also used to expand the dimensions of the N layers of feature maps respectively to obtain N layers of spatial features; and perform channel compression on the first layer of spatial features to the Nth layer of spatial features in sequence according to the number of channels of the N visual self-supervised models from shallow to deep to obtain the N layers of intermediate features.

[0110] An embodiment of the present application provides a storage medium on which a computer program is stored. The computer-readable storage medium stores one or more programs. The one or more programs can be executed by one or more processors and applied to an electronic device. The computer program implements the image segmentation method as described above.

[0111] Based on the above embodiments, an embodiment of the present application provides a computer program product, including a computer program, which can be executed by one or more processors, and the computer program implements the image segmentation method as described above.

[0112] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); or various devices including one or any combination of the above memories.

[0113] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0114] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0115] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0116] To sum up, through the embodiments of the present application, for the N-layer feature maps output by the N-layer encoder of the visual self-supervised model, feature reorganization can integrate multi-scale features to form a feature pyramid with hierarchical expression capabilities, and then sampling can optimize the resolution of the features, thereby generating N-layer image features with a hierarchical structure; the feature fusion of the N-layer image features and the corresponding N-layer depth features can combine image features with different semantics and depth features that can filter the main area for image segmentation, more accurately mark targets and identify target edges, greatly improve the image segmentation accuracy and image segmentation integrity, and thus improve the generalization ability of image segmentation.

[0117] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling an image display device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present disclosure.

[0119] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image segmentation method, characterized in that: The method comprises: Processing N layers of feature maps of the image data respectively to obtain N layers of image features; the N layers of feature maps are output by the N layers of encoders of the visual self-supervisory model; Extracting N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features; Performing feature fusion on the N layers of depth features and the corresponding N layers of image features, and fusing them layer by layer from top to bottom to obtain a global fusion feature; Perform image segmentation based on the global fusion features and output an image segmentation result.

2. The method according to claim 1, characterized in that The processing of N layers of feature maps of the image data to obtain N layers of image features includes: According to the number of N visual self-supervised model channels from shallow to deep, the first layer feature map to the Nth layer feature map are sequentially reorganized to obtain N layers of intermediate features; The N layers of intermediate features are adaptively sampled in descending order of image resolution to obtain N layers of image features.

3. The method according to claim 1, characterized in that The extracting N layers of depth features from the depth map data corresponding to the image data includes: Downsampling the depth map data N times to obtain N layers of depth map data; the sampling multiples of the N downsampling times are determined according to the feature sizes of the N layers of image features; Depth features are extracted from the N layers of depth map data to obtain N layers of depth features.

4. The method according to claim 1, wherein The method further comprises: The N-layer feature maps of the image data and the depth map data are input into an image segmentation model to obtain the image segmentation result; the image segmentation model includes at least one of the following modules: an image feature processing module for processing the N-layer feature maps respectively, a depth feature extraction module for extracting the N-layer depth features from the depth map data, a feature fusion module for fusing the N-layer depth features with the corresponding N-layer image features and fusing them layer by layer from top to bottom, and an image segmentation module for performing image segmentation based on the global fusion features.

5. The method according to claim 4, characterized in that The method further comprises: The convolution kernel parameters of the depth feature extraction module are initialized to zero and then model training is performed to obtain the image segmentation model.

6. The method according to claim 2, characterized in that According to the number of N visual self-supervisory model channels from shallow to deep, the first layer feature map to the Nth layer feature map are sequentially reorganized to obtain N layers of intermediate features, including: Dimensionally expand the N layers of feature maps to obtain N layers of spatial features; According to the number of channels of N visual self-supervised models from shallow to deep, feature extraction is performed on the first layer spatial features to the Nth layer spatial features in sequence, and then channel compression is performed to obtain the N layers of intermediate features.

7. An image segmentation device, characterized in that: The device comprises: An image feature processing module is used to process N layers of feature maps of the image data to obtain N layers of image features; the N layers of feature maps are output by the N layers of encoders of the visual self-supervisory model; A depth feature extraction module is configured to extract N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are aligned with the feature sizes of the N layers of image features; A feature fusion module is used to fuse the N layers of depth features with the corresponding N layers of image features, and fuse them layer by layer from top to bottom to obtain a global fusion feature; The image segmentation module is used to perform image segmentation based on the global fusion feature and output the image segmentation result.

8. An electronic device, characterized in that: include: a memory for storing computer executable instructions or computer programs; The processor is configured to implement the image segmentation method according to any one of claims 1 to 6 when executing the computer executable instructions or computer program stored in the memory.

9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image segmentation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the image segmentation method according to any one of claims 1 to 6 is implemented.

Images