An image segmentation method and device, equipment, storage medium, computer program product

Abstract

The application provides an image segmentation method and device, equipment, a storage medium and a computer program product, comprising: processing N-layer feature maps of image data respectively to obtain N-layer image features; the N-layer feature maps are output by N-layer encoders of a visual self-supervised model; extracting N-layer depth features from depth map data corresponding to the image data; wherein the feature sizes of the N-layer depth features are respectively aligned with the feature sizes of the N-layer image features; respectively performing feature fusion on the N-layer depth features and the corresponding N-layer image features, and performing layer-by-layer fusion from top to bottom to obtain global fusion features; performing image segmentation based on the global fusion features, and outputting an image segmentation result. The generalization capability of image segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image segmentation method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Current humanoid robots widely use RGBD cameras, which can simultaneously acquire color images (RGB) and depth information, providing multimodal input for perception of complex scenes. Traditional RGB segmentation models rely on low-dimensional features such as texture and color, and are prone to misjudgment in complex scenes with changes in lighting, occlusion, and similar textures, resulting in incomplete segmentation of target objects and thus insufficient generalization ability of image segmentation. Summary of the Invention

[0003] This application provides an image segmentation method, apparatus, device, storage medium, and computer program product. These can improve the generalization ability of image segmentation.

[0004] The technical solution of this application is implemented as follows: In a first aspect, this application proposes an image segmentation method, the method comprising: The N-layer feature maps of the image data are processed to obtain N-layer image features; the N-layer feature maps are output by the N-layer encoder of the visual self-supervised model; wherein, the visual self-supervised model includes a number of transformer encoders, and N transformer encoders are selected from the number of transformer encoders to obtain their output N-layer feature maps. N layers of depth features are extracted from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features. The N-layer depth features and the corresponding N-layer image features are fused respectively, and the global fused features are obtained by fusing layer by layer from top to bottom; Image segmentation is performed based on the global fusion features, and the image segmentation result is output. The process of processing the N-layer feature maps of the image data to obtain N-layer image features includes: The N-layer feature maps are dimensionally expanded to obtain N-layer spatial features; Based on the N channels of the visual self-supervised model from shallow to deep, the spatial features from the first layer to the Nth layer are extracted sequentially, and then channel compression is performed to obtain the intermediate features of the Nth layer.

[0005] The intermediate features of the N layers are adaptively sampled in descending order of image resolution to obtain the N layers of image features.

[0006] Secondly, this application proposes an image segmentation apparatus, the apparatus comprising: The image feature processing module is used to process the N-layer feature maps of the image data to obtain N-layer image features; the N-layer feature maps are output by the N-layer encoder of the visual self-supervised model; wherein, the visual self-supervised model includes a number of transformer encoders, and N transformer encoders are selected from the number of transformer encoders to obtain their output N-layer feature maps. The depth feature extraction module is used to extract N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features; The feature fusion module is used to perform feature fusion on the N-layer depth features and the corresponding N-layer image features respectively, and to fuse them layer by layer from top to bottom to obtain global fused features; The image segmentation module is used to perform image segmentation based on the global fusion features and output the image segmentation result; The image feature processing module is further configured to: expand the dimensions of the N-layer feature maps to obtain N-layer spatial features; extract features from the first-layer spatial features to the N-layer spatial features according to the N channels of the visual self-supervised model from shallow to deep, and then compress the channels to obtain the N-layer intermediate features; and adaptively sample the N-layer intermediate features according to the image resolution from high to low to obtain N-layer image features.

[0007] Thirdly, this application proposes an electronic device, comprising: Memory is used to store executable instructions or computer programs. The processor is configured to implement the image segmentation method described above when executing computer-executable instructions or computer programs stored in the memory.

[0008] Fourthly, this application proposes a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the above-mentioned image segmentation method.

[0009] Fifthly, this application proposes a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the above-described image segmentation method.

[0010] This application provides an image segmentation method, apparatus, device, storage medium, and computer program product. The method includes: processing N-layer feature maps of image data to obtain N-layer image features; the N-layer feature maps are output by an N-layer encoder of a visual self-supervised model; wherein the visual self-supervised model includes a number of transformer encoders, and N transformer encoders are selected from the number of transformer encoders to obtain their output N-layer feature maps; extracting N-layer depth features from the depth map data corresponding to the image data; wherein the feature dimensions of the N-layer depth features are aligned with the feature dimensions of the N-layer image features; fusing the N-layer depth features with the corresponding N-layer image features, and fusing layer by layer from top to bottom to obtain global fused features; performing image segmentation based on the global fused features, and outputting the image segmentation result; wherein processing the N-layer feature maps of image data to obtain N-layer image features includes: expanding the dimensions of the N-layer feature maps to obtain N-layer spatial features; extracting features from the first layer to the Nth layer spatial features according to the N channels of the visual self-supervised model from shallow to deep, and then compressing the channels to obtain N-layer intermediate features. Adaptive sampling is performed on the intermediate features of the N layers according to the image resolution from high to low to obtain N layers of image features. Using the above implementation scheme, for the N-layer feature map output by the N-layer encoder of the visual self-supervised model, feature recombination can integrate multi-scale features to form a feature pyramid with hierarchical representation capabilities. Subsequent sampling can optimize the feature resolution, thereby generating N-layer image features with a hierarchical structure. Feature fusion of the N-layer image features and the corresponding N-layer depth features allows for image segmentation by combining image features with different semantics and depth features of selectable subject regions. This results in more accurate target annotation and target edge identification, greatly improving the accuracy and completeness of image segmentation, and thus enhancing the generalization ability of image segmentation. Attached Figure Description

[0011] Figure 1 A flowchart illustrating an image segmentation method provided in this application embodiment; Figure 2 A schematic diagram of an exemplary image segmentation model provided in this application embodiment; Figure 3 A schematic diagram illustrating the module composition of an exemplary feature fusion module provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0015] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0017] This application provides an image segmentation method, such as... Figure 1 As shown, the method may include: S101. Process the N-layer feature maps of the image data respectively to obtain N-layer image features; the N-layer feature maps are the output of the N-layer encoder of the visual self-supervised model.

[0018] In one embodiment, the visual self-supervised model can be a DINOV2 network, and the encoder is a transformer encoder. Specifically, other networks can be selected as the visual self-supervised model, and feature extraction modules of other network applications can be used as the encoder, depending on the actual situation. This application does not impose specific limitations on these embodiments. The following description uses a DINOV2 network as the visual self-supervised model and a transformer encoder as the example.

[0019] In one embodiment, the DINOV2 network is a network with a Vision Transformer (ViT) as its core that achieves high-performance visual feature extraction through self-supervised learning. The DINOV2 network can be composed of 12 stacked transformer encoders. Image data is input into the first transformer encoder and processed sequentially by the 12 transformer encoders. Each transformer encoder can output a layer of features.

[0020] This application describes an example of a DINOV2 network including 12 transformer encoders. It should be noted that the number of transformer encoders included in the DINOV2 network can be 12, 24, 32 or other numbers, and can be adjusted according to the actual situation. No specific limitation is made in this application embodiment.

[0021] In one embodiment, N transformer encoders can be selected from 12 transformer encoders to obtain N layers of feature maps output by them. The selection strategy for choosing N transformer encoders from the 12 transformer encoders can be adaptively selected based on actual circumstances, and this application does not further limit the selection strategy.

[0022] It should be noted that N is a positive integer.

[0023] For example, if N is set to 4, the features output by the 3rd, 6th, 9th, and 12th transformer encoders out of the 12 transformer encoders are obtained as 4 layers of features. The output of the 3rd transformer encoder is the first layer of features, the output of the 6th transformer encoder is the second layer of features, the output of the 9th transformer encoder is the third layer of features, and the output of the 12th transformer encoder is the fourth layer of features.

[0024] It should be noted that the feature dimensions of the N-layer feature maps can be the same. For example, taking the features output by the 3rd, 6th, 9th, and 12th transformer encoders as an example, the feature dimensions of the corresponding output 4-layer features can be 1×1024×768.

[0025] It should be noted that while the feature maps of layers N have the same feature dimension, their semantic levels gradually deepen. For example, the initial layer (e.g., layers 1-3) feature maps mainly reflect low-level information such as local texture and edges, suitable for tasks such as super-resolution and denoising; the intermediate layer (e.g., layers 4-9) feature maps gradually aggregate the global context and begin to capture semantic information (e.g., object outlines, component relationships), enabling the detection of small targets such as pedestrians and traffic signs; the deep layer (e.g., layers 10-12) feature maps output high-order semantic features, containing global information such as scene category and target location, and can distinguish information such as roads, buildings, and vegetation.

[0026] In this embodiment of the application, after obtaining the N-layer feature map output by the N-layer encoder of the visual self-supervised model, the N-layer feature map needs to be processed separately to obtain the N-layer image features.

[0027] Specifically, the N layers of feature maps are sequentially reconstructed and sampled to obtain N layers of image features.

[0028] In this embodiment, the first layer feature map to the Nth layer feature map are sequentially reorganized according to the N channels of the visual self-supervised model from shallow to deep to obtain N layers of intermediate features; then, the N layers of intermediate features are adaptively sampled in order of image resolution from high to low to obtain N layers of image features.

[0029] In this embodiment, features from the first layer to the Nth layer feature map are sequentially reorganized according to the N channels of the visual self-supervised model from shallow to deep to obtain N intermediate features; the N layers feature maps are dimensionally expanded to obtain N spatial features; and the N layers spatial features are sequentially extracted and then compressed according to the N channels of the visual self-supervised model from shallow to deep to obtain N intermediate features.

[0030] It should be noted that dimensional expansion is the same as feature reshaping, which involves readjusting the dimensions of the data without changing its content. The purpose of reshaping is to facilitate subsequent convolution operations. Convolution operations are involved in subsequent feature extraction, channel compression adaptive sampling, and feature fusion.

[0031] In this embodiment, the N-layer feature map is a three-dimensional feature, and the three-dimensional feature can be expanded into a four-dimensional spatial feature through the feature reshaping process.

[0032] For example, a 1×1024×768 three-dimensional feature is expanded and split to obtain a 1×768×32×32 four-dimensional spatial feature.

[0033] It should be noted that feature extraction and channel compression together constitute the projection process, which can be implemented by two convolution operations. The first convolution operation is used to progressively extract features from the input channels and expand to deeper channels; specifically, feature extraction is performed on N layers of spatial features in ascending order of channel number. The second convolution operation is used for channel compression while maintaining spatial resolution; specifically, it restores the image features to N layers with the same number of channels. Both the first and second convolutions can be a 3×3 convolution.

[0034] In this embodiment, the number of channels from shallow to deep can be determined based on the number of channels in the N-layer spatial features and N. The specific number can be selected according to the actual situation, and this embodiment does not impose any specific limitations.

[0035] For example, for a 4-layer spatial feature 1×768×32×32, the number of channels in its visual self-supervised model is 768. Then, the 768 is averaged 4-1 times to obtain the number of channels in the 4 visual self-supervised models from shallow to deep: {96, 192, 384, 768}.

[0036] In one embodiment, the first convolution operation extracts features from the four spatial features 1×768×32×32 sequentially according to the number of channels in the visual self-supervised model {96, 192, 384, 768}, resulting in four features: 1x96x32x32, 1x196x32x32, 1x384x32x32, and 1x768x32x32. Then, these four features are compressed to restore the 128-channel intermediate feature 1x128x32x32 from the N layers.

[0037] It should be noted that after obtaining N layers of intermediate features, adaptive sampling can be performed on these N layers of intermediate features to obtain N layers of image features. Specifically, the N layers of intermediate features can be adaptively sampled in descending order of image resolution to obtain N layers of image features.

[0038] In this embodiment, the adaptive sampling design mimics the image resolution in image segmentation, setting different sampling rates for the N intermediate features. The sampling rate can include a sampling multiplier and a sampling type. The sampling multiplier can be 8x, 4x, 2x, 1x, etc., and can be selected based on actual conditions; this embodiment does not impose specific limitations. The sampling type can be upsampling or downsampling, and can be selected based on actual conditions; this embodiment does not impose specific limitations.

[0039] It should be noted that a higher image resolution corresponds to a higher sampling rate. For shallow features, a higher image resolution is set, and conversely, for deep features, a lower image resolution is set. Since the intermediate features in the N layers are arranged in order of increasing dimensionality, the corresponding image resolutions are set in order of decreasing dimensionality.

[0040] In one embodiment, the four intermediate feature layers can be sequentially configured with deconvolution upsampling by 4x, deconvolution upsampling by 2x, hold, and downsampling by 2x.

[0041] For example, the first layer intermediate feature 1x128x32x32 is deconvolved and upsampled by 4 times to obtain the first layer image feature 1x128x128x128; the second layer intermediate feature 1x128x32x32 is deconvolved and upsampled by 2 times to obtain the second layer image feature 1x128x64x64; the third layer intermediate feature 1x128x32x32 is preserved to obtain the third layer image feature 1x128x32x32; and the fourth layer intermediate feature 1x128x32x32 is downsampled by 2 times to obtain the fourth layer image feature 1x128x16x16.

[0042] It is understandable that processing the N-layer feature maps output by the N-layer transformer encoder of the DINOV2 network separately can further process the features of the N-layer feature maps without destroying the DINOV2 network structure. This preserves the generalization ability of the DINOV2 network while forming a hierarchical feature representation, enhancing the model's ability to understand global and local information of the image and improving the feature quality of self-supervised learning.

[0043] S102. Extract N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features.

[0044] In this embodiment, the depth map data is first downsampled N times to obtain N layers of depth map data; the sampling factor of the N downsampling operations is determined according to the feature size of the N layers of image features; then, depth features are extracted from the N layers of depth map data to obtain N layers of depth features.

[0045] It should be noted that, for depth map data, the size of N layers of image features is first aligned through N downsampling operations to achieve subsequent feature fusion. Then, depth features are extracted from the N layers of depth map data to obtain N layers of depth features.

[0046] In one embodiment, the depth map data is downsampled by 4 times to match the size of the first layer of image features; the depth map data is downsampled by 8 times to match the size of the second layer of image features; the depth map data is downsampled by 16 times to match the size of the third layer of image features; and the depth map data is downsampled by 32 times to match the size of the fourth layer of image features.

[0047] In this embodiment of the application, a convolutional network can be used to extract depth features from N layers of depth map data.

[0048] Optionally, the above convolutional network can be a convolutional network with 5 3×3 convolutions and 128 channels.

[0049] S103. Perform feature fusion on the N-layer depth features and the corresponding N-layer image features respectively, and fuse them layer by layer from top to bottom to obtain the global fused features.

[0050] In this embodiment, N layers of image features and corresponding N layers of depth features are concatenated to obtain N layers of fused features; then, the N layers of fused features are sequentially upsampled and added from top to bottom to obtain global fused features. Optionally, the above method can be implemented through an N-layer feature map fusion unit. Specifically, the N layers of depth features and N layers of image features are respectively input into an N-layer feature map fusion unit for feature fusion, and the N-layer feature map fusion units are sequentially connected from top to bottom to perform top-down layer-by-layer fusion.

[0051] In one embodiment, since the size of the features in two adjacent layers of the image differs by a factor of 2, the upsampling operation can specifically be a 2x upsampling.

[0052] In this embodiment, each feature fusion unit consists of two convolutional units (a first convolutional unit and a second convolutional unit), an adder, an upsampling unit, and a project unit. The first convolutional unit performs convolution processing on the image features of that layer. For the first layer feature fusion unit, its adder adds the depth features of that layer to the convolutionally processed image features of that layer to obtain the fused features. For other layer feature fusion units, their adders add the depth features of that layer, the convolutionally processed image features of that layer, and the features output from the previous layer to obtain the fused features. The fused features are sequentially processed by the second convolutional unit, the upsampling unit, and the project unit before being output to the adder of the next layer feature fusion unit. This process continues until the project unit of the last layer feature fusion unit outputs the global fused features.

[0053] It should be noted that the Project unit mentioned above consists of a regular convolution, which can be a 3×3 convolution to preserve the dimensions.

[0054] In one embodiment, the size of the global fusion feature can be 1×128×256×256.

[0055] S104. Perform image segmentation based on global fusion features and output the image segmentation results.

[0056] In this embodiment, the global fused features can be classified at the pixel level using the Seg Head module to output image segmentation results. These results may include an image segmentation mask and category labels.

[0057] It should be noted that the Seg Head is the core module in a deep learning image segmentation model responsible for converting feature maps into pixel-level classification predictions. Its design directly affects segmentation accuracy and efficiency, and it is commonly used in tasks such as object detection, instance segmentation, and medical image analysis. The Seg Head is located at the end of the model, receives globally fused features, restores spatial resolution through upsampling or a decoder, and finally generates a segmentation mask of the same size as the input image data, and outputs pixel-level class predictions (such as binary classification masks or multi-class semantic labels).

[0058] In one embodiment, the Seg Head consists of a regular convolution, which can be a 3×3 convolution, and its output channel is the final category to be segmented.

[0059] In one embodiment, the Seg Head module is used to perform pixel-level classification on the 1×128×256×256 image, resulting in an image segmentation result of 1×class_num×256×256. Here, class_num represents the image segmentation category.

[0060] Understandably, for the N-layer feature map output by the N-layer encoder of the visual self-supervised model, feature recombination can integrate multi-scale features to form a feature pyramid with hierarchical expressive capabilities. Subsequent sampling can optimize the feature resolution, thereby generating N-layer image features with a hierarchical structure. Feature fusion of N-layer image features and corresponding N-layer depth features can combine image features with different semantics and depth features of selectable subject regions for image segmentation, more accurately labeling targets and identifying target edges, greatly improving the accuracy and completeness of image segmentation, and thus improving the generalization ability of image segmentation.

[0061] It should be noted that the above method can be implemented by an image segmentation model, which includes at least one of the following modules: an image feature processing module for processing N layers of feature maps respectively; a depth feature extraction module for extracting N layers of depth features from the depth map data; a feature fusion module for fusing the N layers of depth features with the corresponding N layers of image features layer by layer from top to bottom; and an image segmentation module for performing image segmentation based on global fused features. The N layers of feature maps and depth map data of the image data can be input into the image segmentation model to obtain the image segmentation result.

[0062] It should be noted that the image segmentation model can also include the DINOV2 network to process the image data to obtain N layers of feature maps.

[0063] The above image segmentation model can be obtained through model training. During the initialization process of training, the convolution kernel parameters of the deep feature extraction module can be initialized to zero before model training is performed to obtain the image segmentation model.

[0064] It is understandable that initializing the convolution kernel parameters of the deep feature extraction module to zero before training the model can ensure that the main branch fully utilizes the generalization function of DINOV2 in the initial stage of training. Then, transitional learning can be performed gradually according to the deep features, and the guiding information of the deep features can be gradually superimposed to improve the accuracy and generalization of image segmentation. This allows the image segmentation model to enhance the accuracy and generalization of image segmentation without destroying the generalization ability of the main branch.

[0065] It should be noted that the convolution kernel parameters can include weights (W) and bias terms (b). Here, W is the numerical matrix of the convolution kernel, used to extract local features (such as edges, textures, etc.); b is a single numerical value used to adjust the global offset of the convolution output.

[0066] It should be noted that zeroing the convolution kernel parameters can be done by initializing W as a zero matrix and b as a zero vector.

[0067] It should be noted that the above convolution kernel parameters are the convolution kernel parameters of the convolutional network used in the depth feature extraction module to extract depth features from N layers of depth map data.

[0068] During the model training phase, a joint loss function can be used, including at least one of cross-entropy loss (L_CE), Dice loss (L_Dice), and depth consistency loss (L_Depth). The specific loss function can be selected according to the actual situation, and this application does not impose any specific limitations.

[0069] It should be noted that during the model training phase, after obtaining the image segmentation training results through the Seg Head, the image segmentation training results need to be upsampled to the training input size before performing the aforementioned loss calculation.

[0070] For example, if the training input size is 448, the result obtained from the Seg Head needs to be upsampled to 448 to perform the above loss calculation.

[0071] Based on the above embodiments, an image segmentation model is proposed by way of example, see [link to relevant documentation]. Figure 2This image segmentation model consists of a DINOV2 network, an image feature processing module, a deep feature extraction module, a feature fusion module, and an image segmentation module. Specifically, it acquires the four layers of features output by the 3rd, 6th, 9th, and 12th transformer encoders in the DINOV2 network. The image feature processing module includes a four-layer feature reconstruction unit connected to the four transformer encoders, a 4x upsampling deconvolution unit connected to the 1st feature reconstruction unit, a 2x upsampling deconvolution unit connected to the 2nd feature reconstruction unit, and a 2x downsampling unit connected to the 4th feature reconstruction unit. The deep feature extraction module includes four downsampling units: a 4x downsampling unit, an 8x downsampling unit, a 16x downsampling unit, and a 32x downsampling unit. The convolutional network 1 is connected to a 4x downsampling unit, the convolutional network 2 is connected to an 8x downsampling unit, the convolutional network 3 is connected to a 16x downsampling unit, and the convolutional network 4 is connected to a 32x downsampling unit. The feature fusion module includes four cascaded feature fusion units from top to bottom to achieve layer-by-layer feature fusion. The fourth feature fusion unit is also connected to a 2x downsampling unit and the convolutional network 4 to achieve feature stitching at this layer. The third feature fusion unit is also connected to a third feature recombination unit and the convolutional network 3 to achieve feature stitching at this layer. The second feature fusion unit is also connected to a deconvolutional 2x upsampling unit and the convolutional network 2 to achieve feature stitching at this layer. The first feature fusion unit is also connected to a deconvolutional 4x upsampling unit and the convolutional network 1 to achieve feature stitching at this layer. The image segmentation module can be a Seg Head.

[0072] See Figure 2 ,like Figure 3 As shown, the feature fusion unit consists of convolutional unit 1, convolutional unit 2, an adder, a 2x upsampling unit, and a project unit. The image features of each layer are input into the adder via convolutional unit 1, and the depth features of each layer are also input into the adder. The adder is connected in sequence to convolutional unit 2, the 2x upsampling unit, and the project unit.

[0073] It should be noted that for the feature fusion units of layers 4, 3, and 2, the output of their Project units is connected to the input of the adder of the next layer feature fusion unit (not shown). For the feature fusion unit of layer 1, the output of its Project unit is connected to the Seg Head (not shown).

[0074] This application provides an image segmentation apparatus. For example... Figure 4 As shown, the image segmentation device 1 includes: The image feature processing module 10 is used to process the N-layer feature maps of the image data respectively to obtain N-layer image features; the N-layer feature maps are output by the N-layer encoder of the visual self-supervised model; The depth feature extraction module 11 is used to extract N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features; The feature fusion module 12 is used to perform feature fusion on the N-layer depth features and the corresponding N-layer image features respectively, and to fuse them layer by layer from top to bottom to obtain global fused features; Image segmentation module 13 is used to perform image segmentation based on the global fusion features and output the image segmentation result.

[0075] Optionally, the image feature processing module 10 is further configured to sequentially reorganize the first layer feature map to the Nth layer feature map according to the N channels of the visual self-supervised model from shallow to deep, to obtain N layers of intermediate features; and adaptively sample the N layers of intermediate features according to the image resolution from high to low, to obtain N layers of image features.

[0076] Optionally, the depth feature extraction module 11 is further configured to perform N downsampling operations on the depth map data to obtain N layers of depth map data; the sampling multiple of the N downsampling operations is determined according to the feature size of the N layers of image features; and to extract depth features from the N layers of depth map data to obtain N layers of depth features.

[0077] Optionally, the image segmentation device may further include an image segmentation model. The image segmentation device is further configured to input the N-layer feature map of the image data and the depth map data into the image segmentation model to obtain the image segmentation result. The image segmentation model includes at least one of the following modules: an image feature processing module for processing the N-layer feature map respectively; a depth feature extraction module for extracting the N-layer depth features from the depth map data; a feature fusion module for fusing the N-layer depth features with the corresponding N-layer image features and fusing them layer by layer from top to bottom; and an image segmentation module for performing image segmentation based on the global fusion features.

[0078] Optionally, the device further includes: a training module; The training module is used to initialize the convolution kernel parameters of the deep feature extraction module to zero and then train the model to obtain the image segmentation model.

[0079] Optionally, the image feature processing module 10 is further configured to perform dimensional expansion on the N-layer feature maps respectively to obtain N-layer spatial features; and to extract features from the first-layer spatial features to the N-layer spatial features in sequence according to the N channels of the visual self-supervised model from shallow to deep, and then perform channel compression to obtain the N-layer intermediate features.

[0080] This application provides an image segmentation apparatus that processes N-layer feature maps of image data to obtain N-layer image features. The N-layer feature maps are output by an N-layer encoder of a visual self-supervised model. N-layer depth features are extracted from the depth map data corresponding to the image data. The feature sizes of the N-layer depth features are aligned with the feature sizes of the N-layer image features. Feature fusion is performed between the N-layer depth features and the corresponding N-layer image features, and the fusion is performed layer by layer from top to bottom to obtain global fused features. Image segmentation is performed based on the global fused features, and the image segmentation result is output. Therefore, the image segmentation apparatus proposed in this embodiment, by recombining the features of the N-layer feature maps output by the N-layer encoder of the visual self-supervised model, can integrate multi-scale features to form a feature pyramid with hierarchical representation capabilities. Subsequent sampling can optimize the feature resolution, thereby generating N-layer image features with a hierarchical structure. Feature fusion between the N-layer image features and the corresponding N-layer depth features can combine image features with different semantics and depth features that can filter subject regions for image segmentation, more accurately labeling targets and identifying target edges, greatly improving the accuracy and completeness of image segmentation, and thus improving the generalization ability of image segmentation.

[0081] Figure 5 A schematic diagram of the composition structure of an electronic device 2 provided in this application embodiment. Figure 2 In practical applications, based on the same disclosed concept of the above embodiments, such as Figure 5 As shown, the electronic device 2 in this embodiment includes: a processor 21, a memory 22, and a communication bus 23.

[0082] The processor 21 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not impose specific limitations.

[0083] In this embodiment, the communication bus 23 is used to establish a connection between the processor 21 and the memory 22; when the processor 21 executes the running program stored in the memory 22, it implements the following image segmentation method: The N-layer feature maps of the image data are processed to obtain N-layer image features; the N-layer feature maps are output by the N-layer encoder of the visual self-supervised model; N-layer depth features are extracted from the depth map data corresponding to the image data; the feature sizes of the N-layer depth features are aligned with the feature sizes of the N-layer image features; the N-layer depth features and the corresponding N-layer image features are fused, and the fusion is performed layer by layer from top to bottom to obtain global fused features; image segmentation is performed based on the global fused features, and the image segmentation result is output.

[0084] Furthermore, the processor 21 is also used to sequentially reorganize the features from the first layer to the Nth layer feature map according to the N channels of the visual self-supervised model from shallow to deep, to obtain N layers of intermediate features; and to adaptively sample the N layers of intermediate features according to the image resolution from high to low, to obtain N layers of image features.

[0085] Furthermore, the processor 21 is also used to downsample the depth map data N times to obtain N layers of depth map data; the sampling multiple of the N downsampling is determined according to the feature size of the N layers of image features; and to extract depth features from the N layers of depth map data to obtain N layers of depth features.

[0086] Furthermore, the processor 21 is also used to input the N-layer feature map of the image data and the depth map data into an image segmentation model to obtain the image segmentation result; the image segmentation model includes at least one of the following modules: an image feature processing module for processing the N-layer feature map respectively, a depth feature extraction module for extracting the N-layer depth features from the depth map data, a feature fusion module for fusing the N-layer depth features with the corresponding N-layer image features and fusing them layer by layer from top to bottom, and an image segmentation module for performing image segmentation based on the global fusion features.

[0087] Furthermore, the processor 21 is also used to initialize the convolution kernel parameters of the depth feature extraction module to zero and then train the model to obtain the image segmentation model.

[0088] Furthermore, the processor 21 is also used to expand the dimensions of the N-layer feature maps respectively to obtain N-layer spatial features; and to extract features from the first-layer spatial features to the N-layer spatial features according to the N channels of the visual self-supervised model from shallow to deep, and then compress the channels to obtain the N-layer intermediate features.

[0089] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in an electronic device. The computer program implements the image segmentation method described above.

[0090] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the image segmentation method described above.

[0091] In some embodiments, the computer-readable storage medium may be a memory such as ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0092] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0093] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0094] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0095] In summary, through the embodiments of this application, the N-layer feature map output by the N-layer encoder of the visual self-supervised model can be reorganized to integrate multi-scale features into a feature pyramid with hierarchical representation capabilities. Subsequent sampling can optimize the feature resolution, thereby generating N-layer image features with a hierarchical structure. Feature fusion of the N-layer image features and the corresponding N-layer depth features can combine image features with different semantics and depth features of selectable subject regions for image segmentation, more accurately labeling targets and identifying target edges, greatly improving the accuracy and completeness of image segmentation, and thus improving the generalization ability of image segmentation.

[0096] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0098] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image segmentation method, characterized in that, The method includes: The N-layer feature maps of the image data are processed to obtain N-layer image features; the N-layer feature maps are output by the N-layer encoder of the visual self-supervised model; wherein, the visual self-supervised model includes a number of transformer encoders, and N transformer encoders are selected from the number of transformer encoders to obtain their output N-layer feature maps. N layers of depth features are extracted from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features. The N-layer depth features and the corresponding N-layer image features are fused respectively, and the global fused features are obtained by fusing layer by layer from top to bottom; Image segmentation is performed based on the global fusion features, and the image segmentation result is output. The process of processing the N-layer feature maps of the image data to obtain N-layer image features includes: The N-layer feature maps are dimensionally expanded to obtain N-layer spatial features; Based on the N channels of the visual self-supervised model from shallow to deep, the spatial features from the first layer to the Nth layer are extracted sequentially and then compressed to obtain the Nth layer of intermediate features. The intermediate features of the N layers are adaptively sampled in descending order of image resolution to obtain the N layers of image features.

2. The method according to claim 1, characterized in that, The step of extracting N layers of depth features from the depth map data corresponding to the image data includes: The depth map data is downsampled N times to obtain N layers of depth map data; the sampling factor of the N downsampling is determined according to the feature size of the N layers of image features; Depth features are extracted from the N-layer depth map data to obtain N-layer depth features.

3. The method according to claim 1, characterized in that, The method further includes: The image segmentation model is input into the N-layer feature map and the depth map data of the image data to obtain the image segmentation result. The image segmentation model includes at least one of the following modules: an image feature processing module for processing the N-layer feature map respectively; a depth feature extraction module for extracting the N-layer depth features from the depth map data; a feature fusion module for fusing the N-layer depth features with the corresponding N-layer image features and fusing them layer by layer from top to bottom; and an image segmentation module for performing image segmentation based on the global fusion features.

4. The method according to claim 3, characterized in that, The method further includes: After initializing the convolution kernel parameters of the depth feature extraction module to zero, the model is trained to obtain the image segmentation model.

5. An image segmentation apparatus, characterized in that, The device includes: The image feature processing module is used to process the N-layer feature maps of the image data to obtain N-layer image features; the N-layer feature maps are output by the N-layer encoder of the visual self-supervised model; wherein, the visual self-supervised model includes a number of transformer encoders, and N transformer encoders are selected from the number of transformer encoders to obtain their output N-layer feature maps. The depth feature extraction module is used to extract N layers of depth features from the depth map data corresponding to the image data; wherein the feature sizes of the N layers of depth features are respectively aligned with the feature sizes of the N layers of image features; The feature fusion module is used to perform feature fusion on the N-layer depth features and the corresponding N-layer image features respectively, and to fuse them layer by layer from top to bottom to obtain global fused features; The image segmentation module is used to perform image segmentation based on the global fusion features and output the image segmentation result; The image feature processing module is further configured to perform dimensional expansion on the N-layer feature maps to obtain N-layer spatial features; extract features from the first-layer spatial features to the N-layer spatial features according to the N channels of the visual self-supervised model from shallow to deep, and then compress the channels to obtain N-layer intermediate features; and adaptively sample the N-layer intermediate features according to the image resolution from high to low to obtain N-layer image features.

6. An electronic device, characterized in that, include: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the image segmentation method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the image segmentation method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the image segmentation method according to any one of claims 1 to 4 is implemented.

Images