Multi-source heterogeneous image segmentation method based on self-supervised learning

Abstract

The embodiments of the present application relate to the field of image processing technology, and in particular to a multi-source heterogeneous image segmentation method based on self-supervised learning, comprising: collecting multi-source heterogeneous sample images and superimposing them in the channel dimension to obtain multi-channel sample input data; constructing a multi-source heterogeneous image segmentation model composed of a spatial branch network and a channel branch network, the spatial branch network and the channel branch network respectively process the multi-channel sample input data to obtain spatial reconstruction features and channel reconstruction features to construct a mask loss function; taking visible light sample images and radar sample images as positive sample pairs, and performing projection transformation and prediction transformation respectively to obtain visible light contrast features and radar contrast features to construct a contrast loss function; constructing a total loss function based on the mask loss function and the contrast loss function, iteratively training the model until convergence, and obtaining a trained model to apply to segmentation tasks to achieve high-precision segmentation of multi-source heterogeneous images.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a multi-source heterogeneous image segmentation method based on self-supervised learning. Background Art

[0002] A radar image is an image formed when a radar transmitter emits radio waves toward a target and a radar receiver receives the scattered echoes. Radar images provide information such as the target's location, velocity, shape, and properties, and are widely used in military, meteorological, geological, and transportation fields. Radar images can be combined with visible light images, infrared images, depth images, and other sources to form multi-source heterogeneous images. These images complement each other in terms of channel information, spatial resolution, and observation time, providing a valuable data foundation for various visual recognition tasks.

[0003] With the widespread adoption and continuous development of radar imaging technology, the ability to acquire multi-source heterogeneous images has significantly increased. While the amount of image data continues to increase, and the underlying data required for image processing tasks is becoming increasingly abundant, the effective utilization of this data still faces significant challenges, particularly in the field of image segmentation. Image segmentation is the task of identifying and distinguishing different objects or regions within an image. Traditional supervised learning-based image segmentation methods typically require large amounts of annotated data to train the image segmentation model. Acquiring this annotated data is not only time-consuming and labor-intensive, but also costly, significantly limiting the scope and effectiveness of image segmentation applications.

[0004] To overcome this problem, a research team proposed applying SSL (Self-Supervised Learning) to the iterative training of image segmentation models. As an emerging learning paradigm, SSL does not require external annotations. Instead, it automatically generates learning signals by leveraging the intrinsic structure and features of unlabeled data, thereby reducing the need for manual annotation.

[0005] However, the inventors of the present application have found that the currently proposed SSL-based image segmentation methods perform poorly when processing multi-source heterogeneous images and cannot achieve the required segmentation accuracy. Summary of the Invention

[0006] In view of this, an embodiment of the present application proposes a multi-source heterogeneous image segmentation method based on self-supervised learning, which can extract useful features by utilizing the complementary characteristics of multi-source heterogeneous images through the mechanism of self-supervised learning without the need for manual labeling of data, thereby achieving high-precision segmentation of various targets in multi-source heterogeneous images.

[0007] In the first aspect, an embodiment of the present application proposes a multi-source heterogeneous image segmentation method based on self-supervised learning, comprising the following steps: collecting multi-source heterogeneous sample images, superimposing sample images from different sources in the channel dimension to obtain multi-channel sample input data; wherein the multi-source heterogeneous sample images include visible light sample images and radar sample images; constructing a multi-source heterogeneous image segmentation model, the multi-source heterogeneous image segmentation model consists of a spatial branch network, a channel branch network and a classifier, the spatial branch network is used to perform spatial masking processing, feature extraction and feature reconstruction on the multi-channel sample input data to obtain spatial reconstruction features, the channel branch network is used to perform channel masking processing, feature extraction and feature reconstruction on the multi-channel sample input data to obtain channel reconstruction features, and the classifier is used to classify the spatial reconstruction features and channel reconstruction features. The method comprises the following steps: first, constructing features for fusion, and performing segmentation based on the fused features; constructing a mask loss function based on multi-channel sample input data, spatial reconstruction features and channel reconstruction features; taking visible light sample images and radar sample images with the same content as positive sample pairs, performing projection transformation and prediction transformation on the visible light sample images and radar sample images respectively, and obtaining visible light contrast features and radar contrast features; constructing a contrast loss function based on the visible light contrast features and radar contrast features; constructing a total loss function based on the mask loss function and the contrast loss function, and iteratively training the multi-source heterogeneous image segmentation model based on the total loss function until convergence to obtain a trained model; inputting the multi-source heterogeneous image to be segmented into the trained model, and obtaining the segmentation result of the multi-source heterogeneous image to be segmented output by the trained model.

[0008] Optionally, the collecting of multi-source heterogeneous sample images and superimposing the sample images from different sources in the channel dimension to obtain multi-channel sample input data includes: superimposing the visible light sample image X1 and the radar sample image X2 with the same content in the channel dimension to obtain the multi-channel sample input data X; wherein the size of X1 is N×C1×P×P, the size of X2 is N×C2×P×P, N is the batch size, C1 and C2 are the number of channels of X1 and X2 respectively, P is the spatial dimension, and the size of X is N×(C1+C2)×P×P.

[0009] Optionally, the multi-source heterogeneous image segmentation model also includes a pre-processing unit, which is located before the spatial branch network and the channel branch network, and is connected to the spatial branch network and the channel branch network respectively; the pre-processing unit is used to divide the multi-channel sample input data X into image blocks of fixed size, including several spatial blocks x1 of size p×p×(C1+C2) and several channel blocks x2 of size p×p×(C1+C2); for the spatial dimension, each spatial block x1 is mapped to a high-dimensional space through convolution projection, and then a flattening operation is performed, and finally the flattened tensor is normalized to obtain a first tensor, and input into the spatial branch network; for the channel dimension, the spatial dimension of each channel block x2 is flattened, and then a fully connected layer is used to map the flattened tensor to a low-dimensional space, and finally the channel dimension is expanded to obtain a second tensor, and input into the channel branch network.

[0010] Optionally, the spatial branch network is composed of a spatial mask unit, a spatial encoder, and a spatial decoder; the spatial mask unit is used to randomly perform spatial masking on the first tensor according to a mask ratio of 0.3, that is, randomly masking some pixels to obtain the spatial dimension masked image x spa ; The spatial encoder consists of multiple Transformer blocks, each of which contains a self-attention mechanism and a feedforward neural network. The spatial encoder is used to process x spa The pixel tokens visible in the image are used to learn the relationship between pixels and perform feature extraction through multiple Transformer blocks to obtain the spatial features f spa ; The spatial decoder consists of multiple Transformer blocks, which are used to spa f spa Decode and obtain the spatial reconstruction feature R spa The channel branch network consists of a channel mask unit, a channel encoder, and a channel decoder. The channel mask unit is used to randomly perform channel masking on the second tensor according to a mask ratio of 0.3, that is, to randomly close some channels and obtain the channel dimension masked image x. chan ; The channel encoder consists of multiple Transformer blocks, each of which contains a self-attention mechanism and a feedforward neural network. The channel encoder is used to process x chan The pixel tokens visible in the image are used to learn the relationship between pixels and perform feature extraction through multiple Transformer blocks to obtain channel features f chan ; The channel decoder consists of multiple Transformer blocks. The channel decoder is used to chan f chan Decode and obtain the channel reconstruction feature R chan .

[0011] Optionally, the method of performing projection transformation and prediction transformation on the visible light sample image and the radar sample image respectively to obtain the visible light contrast feature and the radar contrast feature includes: using an asymmetric student-teacher structure network to perform projection transformation and prediction transformation on the visible light sample image and the radar sample image respectively, the asymmetric student-teacher structure network is composed of an upper branch network and a lower branch network; the upper branch network is composed of an upper branch feature encoder, an upper branch projection transformation unit and a prediction transformation unit, the upper branch projection transformation unit is composed of two layers of MLP, the prediction transformation unit is composed of one layer of MLP, each layer of MLP is composed of a first linear layer, a batch normalization layer, and a first linear layer. The upper branch feature encoder is used to extract features from the visible light sample image, the upper branch projection transformation unit is used to map the output of the upper branch feature encoder to a low-dimensional space, and the prediction transformation unit is used to predict the output of the upper branch projection transformation unit, thereby outputting visible light contrast features; the lower branch network is composed of a lower branch feature encoder and a lower branch projection transformation unit; the lower branch feature encoder is used to extract features from the radar sample image, and the lower branch projection transformation unit is used to map the output of the lower branch feature encoder to a low-dimensional space, thereby outputting radar contrast features.

[0012] Optionally, the mask loss function is constructed based on the multi-channel sample input data, the spatial reconstruction features and the channel reconstruction features, and is implemented by the following formula:

[0013] L mask =L MSE (X,R spa )+L MSE (X,R chan )

[0014] Among them, L MSE represents the mean square error loss function, L mask Represents the constructed mask loss function;

[0015] The contrast loss function is constructed based on the visible light contrast feature and the radar contrast feature, and is implemented by the following formula:

[0016] L contrast =L[t(V1),t′(V2)]+L[t′(V1),t(V2)]

[0017]

[0018] Among them, q θ (z θ ) represents the visible light contrast feature, z ζ Represents radar contrast characteristics, L contrast Represents the constructed contrast loss function;

[0019] The total loss function is constructed based on the mask loss function and the contrast loss function, which is implemented by the following formula:

[0020] L ALL =λ1L contrast +λ2L mask

[0021] Among them, λ1 and λ2 are preset weight coefficients, L ALL is the total loss function constructed.

[0022] Optionally, the parameters of the upper branch network are updated using a global stochastic gradient descent method, which is expressed as follows:

[0023]

[0024] Where optimizer(·) represents the preset optimizer, η is the preset learning rate, θ represents the parameters of the upper branch network before updating, and θ * Represents the parameters of the updated upper branch network;

[0025] The parameter update of the lower branch network adopts the momentum update method, which is expressed by the formula:

[0026] ζ * =τ·ζ+(1-τ)·ζ

[0027] Among them, τ is the preset momentum factor, ζ represents the parameters of the lower branch network before updating, and ζ * Represents the parameters of the updated lower branch network.

[0028] Through the above method, the present application constructs and trains a multi-source heterogeneous image segmentation model to perform multi-source heterogeneous image segmentation tasks. A spatial branch network and a channel branch network are set in the multi-source heterogeneous image segmentation model to process the spatial dimension and channel dimension of the multi-channel sample input data respectively, making full use of the complementary information between different source images and enhancing the feature recognition ability of the multi-source heterogeneous image segmentation model. A self-supervised learning method is adopted when training the multi-source heterogeneous image segmentation model, which can be effectively trained without manual labeling. In order to improve the training effect, an asymmetric student-teacher structure network is introduced for comparative learning, and the total loss function is constructed by combining the mask loss function and the contrast loss function, which improves the training speed and training efficiency of the multi-source heterogeneous image segmentation model, and finally achieves high-precision segmentation of various targets in multi-source heterogeneous images.

[0029] On the second aspect, the embodiment of the present application proposes a multi-source heterogeneous image segmentation system based on self-supervised learning, which specifically includes: a sample acquisition and preprocessing module, which is used to acquire multi-source heterogeneous sample images, and superimpose sample images from different sources in the channel dimension to obtain multi-channel sample input data, wherein the multi-source heterogeneous sample images include visible light sample images and radar sample images; a model construction module, which is used to construct a multi-source heterogeneous image segmentation model, and the multi-source heterogeneous image segmentation model consists of a spatial branch network, a channel branch network and a classifier, the spatial branch network is used to perform spatial masking, feature extraction and feature reconstruction on the multi-channel sample input data to obtain spatial reconstruction features, the channel branch network is used to perform channel masking, feature extraction and feature reconstruction on the multi-channel sample input data to obtain channel reconstruction features, the classifier is used to fuse the spatial reconstruction features and the channel reconstruction features, and based on the fusion The multi-source heterogeneous image segmentation model is iteratively trained based on the mask loss function and the contrast loss function. The multi-source heterogeneous image segmentation model is iteratively trained based on the mask loss function and the contrast loss function. The multi-source heterogeneous image segmentation model is iteratively trained based on the mask loss function and the contrast loss function. The multi-source heterogeneous image segmentation model is iteratively trained based on the total loss function until convergence. The multi-source heterogeneous image segmentation model is iteratively trained based on the total ...

[0030] In a third aspect, an embodiment of the present application proposes an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-source heterogeneous image segmentation method based on self-supervised learning as described in the first aspect above.

[0031] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the multi-source heterogeneous image segmentation method based on self-supervised learning as described in the first aspect above.

[0032] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the related technologies. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 This is a flowchart of a multi-source heterogeneous image segmentation method based on self-supervised learning provided in one embodiment of the present application;

[0035] Figure 2 is a structural diagram of a multi-source heterogeneous image segmentation model provided in one embodiment of the present application;

[0036] Figure 3 is a structural diagram of a multi-source heterogeneous image segmentation system based on self-supervised learning provided in another embodiment of the present application;

[0037] Figure 4 It is a structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the various embodiments of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is only for the convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0039] An embodiment of the present application proposes a multi-source heterogeneous image segmentation method based on self-supervised learning, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is explained using the server as an example. The implementation details of the multi-source heterogeneous image segmentation method based on self-supervised learning proposed in this embodiment are specifically described below. The following content is only the relevant implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0040] The specific process of the multi-source heterogeneous image segmentation method based on self-supervised learning proposed in this embodiment can be as follows: Figure 1 Shown, including:

[0041] Step 101 : Acquire multi-source heterogeneous sample images, and superimpose the sample images from different sources in the channel dimension to obtain multi-channel sample input data, wherein the multi-source heterogeneous sample images include visible light sample images and radar sample images.

[0042] In specific implementations, the server first needs to prepare training, test, and validation samples through data acquisition. The server collects heterogeneous sample images from multiple sources and overlays them along the channel dimension to generate multi-channel sample input data. Multi-source heterogeneous sample images include visible light and radar sample images. The server can divide the training, test, and validation sets into a 3:1:1 ratio.

[0043] In one example, when the server collects multi-source heterogeneous sample images, it needs to perform data cleaning and data preprocessing on the collected data, including image cropping and normalization, to ensure the consistency and comparability of the data.

[0044] In one example, the server needs to superimpose a visible light sample image X1 and a radar sample image X2 with the same content (that is, the objects photographed are the same object) in the channel dimension to obtain multi-channel sample input data X. Here, the size of X1 is N×C1×P×P, and the size of X2 is N×C2×P×P, where N is the batch size, C1 and C2 are the number of channels of X1 and X2 respectively, and P is the spatial dimension. The size of X is N×(C1+C2)×P×P.

[0045] Step 102, construct a multi-source heterogeneous image segmentation model, which consists of a spatial branch network, a channel branch network and a classifier. The spatial branch network is used to perform spatial masking, feature extraction and feature reconstruction on multi-channel sample input data to obtain spatial reconstruction features. The channel branch network is used to perform channel masking, feature extraction and feature reconstruction on multi-channel sample input data to obtain channel reconstruction features. The classifier is used to fuse the spatial reconstruction features and the channel reconstruction features, and perform segmentation based on the fused features.

[0046] In the specific implementation, while collecting multi-source heterogeneous sample images, the server needs to build a multi-source heterogeneous image segmentation model. The structure of the constructed multi-source heterogeneous image segmentation model can be as follows: Figure 2As shown, it is specifically composed of a spatial branch network, a channel branch network and a classifier. The spatial branch network is used to perform spatial mask processing, feature extraction and feature reconstruction on multi-channel sample input data to obtain spatial reconstruction features. The channel branch network is used to perform channel mask processing, feature extraction and feature reconstruction on multi-channel sample input data to obtain channel reconstruction features. The input of the classifier is the output of the spatial branch network and the output of the channel branch network. The classifier is used to fuse the spatial reconstruction features and the channel reconstruction features, and perform segmentation based on the fused features.

[0047] In one example, the multi-source heterogeneous image segmentation model further includes a pre-processing unit, which is located before the spatial branch network and the channel branch network and is connected to the spatial branch network and the channel branch network, respectively. The pre-processing unit is used to divide the multi-channel sample input data X into fixed-size image patches, which include a number of spatial patches x1 of size p×p×(C1+C2) and a number of channel patches x2 of size p×p×(C1+C2). The patching operation essentially converts X into a feature vector and serializes these feature vectors to form a serialized embedding.

[0048] For the spatial dimension, the pre-processing unit maps each spatial block x1 of size p×p×(C1+C2) to a high-dimensional space through convolution projection (Conv2d), then performs a flattening operation, and finally normalizes the flattened tensor to obtain the first tensor, which is input into the spatial branch network.

[0049] For the channel dimension, the pre-processing unit flattens the spatial dimensions (i.e., width and height) of each channel block x2 of size p×p×(C1+C2), and then uses a fully connected layer to map the flattened tensor to a low-dimensional space. Finally, the channel dimension is expanded to obtain the second tensor and input it into the channel branch network.

[0050] In an example, the size of x1 is 28×28×(C1+C2), and the pre-processing unit maps it to a high-dimensional space of 28×28×256 through convolution projection, and after performing a flattening operation, a tensor representation of 784×256 is obtained.

[0051] In an example, the size of x2 is 28×28×(C1+C2). The pre-processing unit flattens its spatial dimensions. At this time, the channel dimension must be retained. The shape of the flattened tensor is 784×(C1+C2). Then the 784 dimensions are mapped to a 256-dimensional space, and the channel dimension is expanded to 64, resulting in a 256×64 tensor representation.

[0052] In one example, the spatial branch network consists of a spatial mask unit, a spatial encoder, and a spatial decoder.

[0053] The spatial mask unit is used to randomly perform spatial masking on the first tensor according to a mask ratio of 0.3, that is, to randomly mask some pixels and obtain the spatial dimension masked image x spa .

[0054] The spatial encoder is composed of multiple Transformer blocks, each of which contains a self-attention mechanism and a feedforward neural network (FNN). The spatial encoder is used to process x spa The pixel tokens visible in the image are used to learn the relationship between pixels and perform feature extraction through multiple Transformer blocks to obtain the spatial features f spa .

[0055] The spatial decoder is also composed of multiple Transformer blocks. The spatial decoder is used to spa f spa Decode and obtain the spatial reconstruction feature R spa .

[0056] In one example, the channel branch network consists of a channel mask unit, a channel encoder, and a channel decoder.

[0057] The channel mask unit is used to randomly perform channel masking on the second tensor according to a mask ratio of 0.3, that is, to randomly close some channels and obtain the channel dimension masked image x chan .

[0058] The channel encoder is composed of multiple Transformer blocks, each of which contains a self-attention mechanism and a feedforward neural network. The channel encoder is used to process x chan The pixel tokens visible in the image are used to learn the relationship between pixels and perform feature extraction through multiple Transformer blocks to obtain channel features f chan .

[0059] The channel decoder is also composed of multiple Transformer blocks. The channel decoder is specifically used based on x chan f chan Decode and obtain the channel reconstruction feature R chan .

[0060] During the feature fusion stage, the classifier concatenates the features extracted by the two branches along the channel dimension to fuse multi-source features. To enhance the representation of global features, the classifier multiplies the fused features by 3, thereby emphasizing their importance in the classification task. The classifier is essentially a linear classification head, which maps the fused features to the corresponding class labels to achieve the final segmentation task.

[0061] Step 103: construct a mask loss function based on the multi-channel sample input data, the spatial reconstruction features and the channel reconstruction features.

[0062] In a specific implementation, after obtaining the spatial reconstruction features and the channel reconstruction features, the server can construct a mask loss function based on the multi-channel sample input data, the spatial reconstruction features and the channel reconstruction features.

[0063] In one example, the server constructs a mask loss function based on multi-channel sample input data, spatial reconstruction features, and channel reconstruction features, which can be implemented by the following formula:

[0064] L mask =L MSE (X,R spa )+L MSE (X,R chan )

[0065] Among them, L MSF represents the mean square error loss function, L mask Represents the constructed mask loss function.

[0066] In step 104 , the visible light sample image and the radar sample image with the same content are taken as a positive sample pair, and projection transformation and prediction transformation are performed on the visible light sample image and the radar sample image respectively to obtain visible light contrast features and radar contrast features.

[0067] In the specific implementation, in order to improve the training effect, the server enters the comparative learning to participate in the training. The server takes the visible light sample image and radar sample image with the same content as the positive sample pair, and performs projection transformation and prediction transformation on the visible light sample image and radar sample image respectively to obtain the visible light contrast feature and radar contrast feature.

[0068] In one example, the server uses an asymmetric student-teacher network to perform projective and predictive transformations on visible light sample images and radar sample images, respectively. In practice, the server needs to perform both projective and predictive transformations on visible light sample images, while only performing projective transformation on radar sample images. The asymmetric student-teacher network consists of an upper branch network and a lower branch network.

[0069] The upper branch network consists of an upper branch feature encoder, an upper branch projection transformation unit, and a prediction transformation unit. The upper branch projection transformation unit consists of two layers of MLPs, and the prediction transformation unit consists of one layer of MLPs. Each MLP layer consists of a first linear layer, a batch normalization layer, a ReLU activation layer, and a second linear layer. The output size of the first linear layer is 4096, and the output size of the second linear layer is 256.

[0070] The upper branch feature encoder is used to extract features from the visible light sample image, the upper branch projection transformation unit is used to map the output of the upper branch feature encoder to a low-dimensional space, and the prediction transformation unit is used to predict the output of the upper branch projection transformation unit to output the visible light contrast feature.

[0071] The lower branch network consists of a lower branch feature encoder and a lower branch projection transformation unit.

[0072] The lower branch feature encoder is used to extract features from the radar sample image, and the lower branch projection transformation unit is used to map the output of the lower branch feature encoder to a low-dimensional space, thereby outputting radar contrast features.

[0073] It is worth noting that although the upper branch network has the same projection structure, the parameter update method is different.

[0074] The parameter update of the upper branch network adopts the global stochastic gradient descent method, which is expressed by the formula:

[0075]

[0076] Where optimizer(·) represents the preset optimizer, η is the preset learning rate, θ represents the parameters of the upper branch network before updating, and θ * Represents the parameters of the updated upper branch network.

[0077] The parameter update of the lower branch network adopts the momentum update method, which is expressed by the formula:

[0078] ζ * =τ·ζ+(1-τ)·ζ

[0079] Among them, τ is the preset momentum factor, ζ represents the parameters of the lower branch network before updating, and ζ * Represents the parameters of the updated lower branch network.

[0080] Step 105: construct a contrast loss function based on the visible light contrast feature and the radar contrast feature.

[0081] In a specific implementation, after obtaining the visible light contrast features and the radar contrast features, the server can construct a contrast loss function based on the visible light contrast features and the radar contrast features.

[0082] In an example, the server constructs a contrast loss function based on visible light contrast features and radar contrast features, which can be implemented using the following formula:

[0083] L contrast =L[t(V1),t′(V2)]+L[t′(V1),t(V2)]

[0084]

[0085] Among them, q θ (z θ ) represents the visible light contrast feature, z ζ Represents radar contrast characteristics, L contrast Represents the constructed contrast loss function.

[0086] Step 106 : constructing a total loss function based on the mask loss function and the contrast loss function, and iteratively training the multi-source heterogeneous image segmentation model based on the total loss function until convergence, thereby obtaining a trained model.

[0087] In the specific implementation, after constructing the mask loss function and the contrast loss function, the server can construct the total loss function based on the mask loss function and the contrast loss function, and iteratively train the multi-source heterogeneous image segmentation model based on the total loss function until convergence to obtain a trained model.

[0088] In one example, the server constructs a total loss function based on the mask loss function and the contrast loss function, which can be implemented by the following formula:

[0089] L ALL =λ1L contrast +λ2L ,ask

[0090] Among them, λ1 and λ2 are preset weight coefficients, L ALL is the total loss function constructed.

[0091] In step 107 , the multi-source heterogeneous image to be segmented is input into the trained model to obtain a segmentation result of the multi-source heterogeneous image to be segmented output by the trained model.

[0092] In the specific implementation, after completing the training, testing and verification of the multi-source heterogeneous image segmentation model, the model can be deployed in the required scenario to prepare for the segmentation task. When performing the segmentation task, the multi-source heterogeneous image to be segmented is input into the trained model, and the trained model outputs the segmentation result of the multi-source heterogeneous image to be segmented.

[0093] In this embodiment, a multi-source heterogeneous image segmentation model is constructed and trained to perform multi-source heterogeneous image segmentation tasks. A spatial branch network and a channel branch network are set in the multi-source heterogeneous image segmentation model to process the multi-channel sample input data in the spatial dimension and the channel dimension respectively, making full use of the complementary information between different source images and enhancing the feature recognition ability of the multi-source heterogeneous image segmentation model. A self-supervised learning method is adopted when training the multi-source heterogeneous image segmentation model, which can be effectively trained without manual labeling. In order to improve the training effect, an asymmetric student-teacher structure network is introduced for comparative learning. The total loss function is constructed by combining the mask loss function and the contrast loss function, which improves the training speed of the multi-source heterogeneous image segmentation model and ultimately achieves high-precision segmentation of various targets in multi-source heterogeneous images.

[0094] The step division of the above various methods is only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0095] Another embodiment of the present application proposes a multi-source heterogeneous image segmentation system based on self-supervised learning. The following is a detailed description of the implementation details of the multi-source heterogeneous image segmentation system based on self-supervised learning proposed in this embodiment. The following content is only for the convenience of understanding the implementation details and is not required for the implementation of this embodiment. Figure 3 This is a structural diagram of a multi-source heterogeneous image segmentation system based on self-supervised learning proposed in this embodiment. The system includes: a sample acquisition and preprocessing module 201, a model construction module 202, a mask construction module 203, a contrast learning module 204, a contrast construction module 205, a model training module 206 and a model use module 207.

[0096] The sample acquisition and preprocessing module 201 is used to acquire multi-source heterogeneous sample images and superimpose sample images from different sources in the channel dimension to obtain multi-channel sample input data, wherein the multi-source heterogeneous sample images include visible light sample images and radar sample images.

[0097] The model construction module 202 is used to construct a multi-source heterogeneous image segmentation model. The multi-source heterogeneous image segmentation model consists of a spatial branch network, a channel branch network and a classifier. The spatial branch network is used to perform spatial masking, feature extraction and feature reconstruction on multi-channel sample input data to obtain spatial reconstruction features. The channel branch network is used to perform channel masking, feature extraction and feature reconstruction on multi-channel sample input data to obtain channel reconstruction features. The classifier is used to fuse the spatial reconstruction features and the channel reconstruction features and perform segmentation based on the fused features.

[0098] The mask construction module 203 is used to construct a mask loss function based on the multi-channel sample input data, the spatial reconstruction features and the channel reconstruction features.

[0099] The contrast learning module 204 is configured to take a visible light sample image and a radar sample image with the same content as a positive sample pair, perform projection transformation and prediction transformation on the visible light sample image and the radar sample image respectively, and obtain visible light contrast features and radar contrast features.

[0100] The comparison construction module 205 is used to construct a comparison loss function based on the visible light comparison feature and the radar comparison feature.

[0101] The model training module 206 is used to construct a total loss function based on the mask loss function and the contrast loss function, and iteratively train the multi-source heterogeneous image segmentation model based on the total loss function until convergence to obtain a trained model.

[0102] The model using module 207 is used to input the multi-source heterogeneous image to be segmented into the trained model to obtain the segmentation result of the multi-source heterogeneous image to be segmented output by the trained model.

[0103] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0104] It is not difficult to find that this embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and this embodiment can be implemented in conjunction with the above-mentioned method embodiment. The relevant technical details and technical effects mentioned in the above-mentioned embodiments are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above-mentioned embodiments.

[0105] Another embodiment of the present application provides an electronic device, the specific structure of which is as follows: Figure 4 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 to enable the at least one processor 301 to execute a multi-source heterogeneous image segmentation method based on self-supervised learning described in the above method embodiments.

[0106] The memory and processor are connected via a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and will not be described further in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by the processor is transmitted via an antenna on a wireless medium. Furthermore, the antenna also receives data and transmits it to the processor.

[0107] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0108] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a multi-source heterogeneous image segmentation method based on self-supervised learning as described in the above method embodiments.

[0109] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0110] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A multi-source heterogeneous image segmentation method based on self-supervised learning, characterized in that: include: Collect multi-source heterogeneous sample images and superimpose the sample images from different sources in the channel dimension to obtain multi-channel sample input data; wherein the multi-source heterogeneous sample images include visible light sample images and radar sample images; Construct a multi-source heterogeneous image segmentation model, which consists of a spatial branch network, a channel branch network, and a classifier. The spatial branch network is used to perform spatial masking, feature extraction, and feature reconstruction on multi-channel sample input data to obtain spatial reconstruction features. The channel branch network is used to perform channel masking, feature extraction, and feature reconstruction on multi-channel sample input data to obtain channel reconstruction features. The classifier is used to fuse the spatial reconstruction features and the channel reconstruction features and perform segmentation based on the fused features. Construct a mask loss function based on multi-channel sample input data, spatial reconstruction features and channel reconstruction features; The visible light sample image and the radar sample image with the same content are taken as positive sample pairs, and the visible light sample image and the radar sample image are respectively subjected to projection transformation and prediction transformation to obtain visible light contrast features and radar contrast features; Construct a contrast loss function based on visible light contrast features and radar contrast features; A total loss function is constructed based on the mask loss function and the contrast loss function, and the multi-source heterogeneous image segmentation model is iteratively trained based on the total loss function until convergence to obtain a trained model; Inputting the multi-source heterogeneous image to be segmented into the trained model, and obtaining the segmentation result of the multi-source heterogeneous image to be segmented output by the trained model; The projective transformation and the predictive transformation are respectively performed on the visible light sample image and the radar sample image to obtain the visible light contrast feature and the radar contrast feature, including: An asymmetric student-teacher structure network is used to perform projection transformation and prediction transformation on visible light sample images and radar sample images respectively. The asymmetric student-teacher structure network consists of an upper branch network and a lower branch network. The upper branch network consists of an upper branch feature encoder, an upper branch projection transformation unit, and a prediction transformation unit. The upper branch projection transformation unit consists of two layers of MLP, and the prediction transformation unit consists of one layer of MLP. Each layer of MLP consists of a first linear layer, a batch normalization layer, a ReLU activation layer, and a second linear layer. The upper branch feature encoder is used to extract features from the visible light sample image, the upper branch projection transformation unit is used to map the output of the upper branch feature encoder to a low-dimensional space, and the prediction transformation unit is used to predict the output of the upper branch projection transformation unit to output visible light contrast features; The lower branch network consists of a lower branch feature encoder and a lower branch projection transformation unit; The lower branch feature encoder is used to extract features from the radar sample image, and the lower branch projection transformation unit is used to map the output of the lower branch feature encoder to a low-dimensional space, thereby outputting radar contrast features.

2. The multi-source heterogeneous image segmentation method based on self-supervised learning according to claim 1, characterized in that: The method of collecting multi-source heterogeneous sample images and superimposing the sample images from different sources in the channel dimension to obtain multi-channel sample input data includes: The visible light sample image with the same content and radar sample images Superimpose in the channel dimension to obtain multi-channel sample input data ;in, The size is , The size is , is the batch size, and They are and The number of channels, is the spatial dimension, The size is .

3. The multi-source heterogeneous image segmentation method based on self-supervised learning according to claim 2, characterized in that: The multi-source heterogeneous image segmentation model further includes a pre-processing unit, which is located before the spatial branch network and the channel branch network and is connected to the spatial branch network and the channel branch network respectively; The pre-processing unit is used to input multi-channel samples into the data Divide into fixed-size image blocks, including several blocks of size Space block and several sizes of Channel block ; For the spatial dimension, each spatial block is projected into Map to a high-dimensional space, then perform a flattening operation, and finally normalize the flattened tensor to obtain the first tensor, which is input into the spatial branch network; For the channel dimension, each channel block The spatial dimension of the tensor is flattened, and then a fully connected layer is used to map the flattened tensor to a low-dimensional space. Finally, the channel dimension is expanded to obtain the second tensor, which is input into the channel branch network.

4. The multi-source heterogeneous image segmentation method based on self-supervised learning according to claim 3, characterized in that: The spatial branch network consists of a spatial mask unit, a spatial encoder, and a spatial decoder; The spatial mask unit is used to randomly perform spatial masking on the first tensor according to a mask ratio of 0.3, that is, to randomly mask some pixels to obtain the image after spatial dimension masking. ; The spatial encoder is composed of multiple Transformer blocks, each of which contains a self-attention mechanism and a feedforward neural network. The spatial encoder is used to process The pixel tokens visible in the image are used to learn the relationship between pixels and perform feature extraction through multiple Transformer blocks to obtain spatial features. ; The spatial decoder consists of multiple Transformer blocks. The spatial decoder is used to right Decode and obtain spatial reconstruction features ; The channel branch network consists of a channel mask unit, a channel encoder, and a channel decoder; The channel mask unit is used to randomly perform channel masking on the second tensor according to a mask ratio of 0.3, that is, to randomly close some channels and obtain the channel dimension masked image. ; The channel encoder is composed of multiple Transformer blocks, each of which contains a self-attention mechanism and a feedforward neural network. The channel encoder is used to process The pixel tokens visible in the image are used to learn the relationship between pixels and perform feature extraction through multiple Transformer blocks to obtain channel features. ; The channel decoder consists of multiple Transformer blocks. The channel decoder is used to right Decode and obtain channel reconstruction features .

5. The multi-source heterogeneous image segmentation method based on self-supervised learning according to claim 1, characterized in that: The mask loss function is constructed based on the multi-channel sample input data, spatial reconstruction features and channel reconstruction features, and is implemented by the following formula: ； in, represents the mean square error loss function, Represents the constructed mask loss function; The contrast loss function is constructed based on the visible light contrast feature and the radar contrast feature, and is implemented by the following formula: ； ； ； in, Represents the visible light contrast feature, represents the radar contrast signature, Represents the constructed contrast loss function; The total loss function is constructed based on the mask loss function and the contrast loss function, which is implemented by the following formula: ； in, and is the preset weight coefficient, is the total loss function constructed.

6. The multi-source heterogeneous image segmentation method based on self-supervised learning according to claim 5, characterized in that: The parameter update of the upper branch network adopts the global stochastic gradient descent method, which is expressed by the formula: ； in, Represents the preset optimizer, is the preset learning rate, represents the parameters of the upper branch network before updating, Represents the parameters of the updated upper branch network; The parameter update of the lower branch network adopts the momentum update method, which is expressed by the formula: ； in, is the preset momentum factor, represents the parameters of the lower branch network before updating, Represents the parameters of the updated lower branch network.

7. A multi-source heterogeneous image segmentation system based on self-supervised learning, characterized in that: include: The sample acquisition and preprocessing module is used to acquire multi-source heterogeneous sample images and superimpose the sample images from different sources in the channel dimension to obtain multi-channel sample input data. The multi-source heterogeneous sample images include visible light sample images and radar sample images. A model building module is used to build a multi-source heterogeneous image segmentation model. The multi-source heterogeneous image segmentation model consists of a spatial branch network, a channel branch network, and a classifier. The spatial branch network is used to perform spatial masking, feature extraction, and feature reconstruction on multi-channel sample input data to obtain spatial reconstruction features. The channel branch network is used to perform channel masking, feature extraction, and feature reconstruction on multi-channel sample input data to obtain channel reconstruction features. The classifier is used to fuse the spatial reconstruction features and the channel reconstruction features and perform segmentation based on the fused features. The mask construction module is used to construct a mask loss function based on multi-channel sample input data, spatial reconstruction features and channel reconstruction features; A contrastive learning module is used to take visible light sample images and radar sample images with the same content as positive sample pairs, perform projection transformation and prediction transformation on the visible light sample images and radar sample images respectively, and obtain visible light contrast features and radar contrast features; A contrast construction module is used to construct a contrast loss function based on visible light contrast features and radar contrast features; The model training module is used to construct a total loss function based on the mask loss function and the contrast loss function, and iteratively train the multi-source heterogeneous image segmentation model based on the total loss function until convergence to obtain a trained model; The model usage module is used to input the multi-source heterogeneous image to be segmented into the trained model, and obtain the segmentation result of the multi-source heterogeneous image to be segmented output by the trained model; The projective transformation and the predictive transformation are respectively performed on the visible light sample image and the radar sample image to obtain the visible light contrast feature and the radar contrast feature, including: An asymmetric student-teacher structure network is used to perform projection transformation and prediction transformation on visible light sample images and radar sample images respectively. The asymmetric student-teacher structure network consists of an upper branch network and a lower branch network. The upper branch network consists of an upper branch feature encoder, an upper branch projection transformation unit, and a prediction transformation unit. The upper branch projection transformation unit consists of two layers of MLP, and the prediction transformation unit consists of one layer of MLP. Each layer of MLP consists of a first linear layer, a batch normalization layer, a ReLU activation layer, and a second linear layer. The upper branch feature encoder is used to extract features from the visible light sample image, the upper branch projection transformation unit is used to map the output of the upper branch feature encoder to a low-dimensional space, and the prediction transformation unit is used to predict the output of the upper branch projection transformation unit to output visible light contrast features; The lower branch network consists of a lower branch feature encoder and a lower branch projection transformation unit; The lower branch feature encoder is used to extract features from the radar sample image, and the lower branch projection transformation unit is used to map the output of the lower branch feature encoder to a low-dimensional space, thereby outputting radar contrast features.

8. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-source heterogeneous image segmentation method based on self-supervised learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multi-source heterogeneous image segmentation method based on self-supervised learning according to any one of claims 1 to 6 can be implemented.

Images