Image semantic segmentation scene migration method, system and device based on three-branch collaborative training

Abstract

The present invention belongs to the field of image analysis and knowledge transfer, and specifically relates to an image semantic segmentation scene migration method, system and device based on three-branch collaborative training, which aims to solve the problem of insufficient feature alignment and style transfer accuracy in existing semantic segmentation technology, resulting in insufficient segmentation performance and insufficient robustness. The present invention includes: extracting features of source domain images and target domain images respectively through a feature extraction module to obtain source domain pixel-level high-dimensional feature maps, target domain pixel-level high-dimensional feature maps and mixed domain pixel-level high-dimensional feature maps, and performing semantic segmentation respectively; calculating the source domain loss, mixed domain loss, target domain loss and segmentation loss based on the semantic segmentation results and calculating the weighted loss, and then training and adjusting the parameters of the feature extraction module; obtaining the semantic category of each pixel in the measured image through the trained feature extraction module. The model training of the present invention has high stability and high flexibility, and effectively improves the subsequent network segmentation performance and robustness.

Description

Technical Field

[0001] The present invention belongs to the field of image analysis and knowledge transfer, and specifically relates to an image semantic segmentation scene transfer method, system and equipment based on three-branch collaborative training. Background Art

[0002] Semantic segmentation is a fundamental task in computer vision, aiming to assign a corresponding category label to each pixel in an image. In real-world applications, segmentation often needs to be performed on an unlabeled target domain, which raises the issue of scene transfer. The goal of scene transfer is to effectively transfer knowledge learned from a labeled source domain to an unlabeled target domain.

[0003] Unsupervised image scene transfer improves model performance in unlabeled target domains by utilizing well-labeled source domain data. This not only significantly reduces labeling costs, but also improves the adaptability and generalization ability of the model in different scenarios and environments.

[0004] Existing image scene transfer methods mainly rely on techniques such as feature alignment and style transfer to reduce the distribution gap between the source and target domains. However, these methods face many challenges in practical applications. Different semantically unified alignment methods are difficult to achieve optimal results, and the alignment time should be dynamically adjusted.

[0005] In applications such as autonomous driving and intelligent surveillance, the accuracy of semantic segmentation has a significant impact on the overall performance of the system. Therefore, studying how to achieve high-precision semantic segmentation in unlabeled target domains has important theoretical significance and practical application value. Summary of the Invention

[0006] In order to solve the above-mentioned problems in the prior art, namely, the problem that the existing semantic segmentation technology has insufficient feature alignment and style transfer accuracy, resulting in insufficient segmentation performance and insufficient robustness, the present invention provides an image semantic segmentation scene transfer method based on three-branch collaborative training, the method comprising:

[0007] Based on the source domain image and the target domain image, feature extraction and feature alignment fusion are performed respectively through the feature extraction module to obtain the fused source domain pixel-level high-dimensional feature map, target domain pixel-level high-dimensional feature map and mixed domain pixel-level high-dimensional feature map;

[0008] The feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding.

[0009] Performing semantic segmentation on the fused and decoded high-dimensional feature maps through a semantic segmentation module to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map;

[0010] According to the semantic segmentation results, the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss are calculated respectively, and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module;

[0011] The semantic category of each pixel in the measured image is obtained through the trained feature extraction module.

[0012] In some preferred embodiments, each level of the multi-level attention unit includes an attention module and a mixed domain-target domain alignment controller;

[0013] Each of the attention modules includes a source domain feature module, a target domain feature module and a mixed domain feature module;

[0014] All source domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the hybrid domain branch of the multi-level attention unit.

[0015] In some preferred embodiments, the mixed domain-target domain alignment controller performs weighted fusion of the target domain pixel-level high-dimensional feature map of the current level attention unit and the mixed domain pixel-level high-dimensional feature map according to the fusion weight to generate a new mixed domain pixel-level high-dimensional feature map to input into the next level attention unit.

[0016] In some preferred embodiments, the source domain loss, the target domain loss, and the mixed domain loss are obtained by performing semi-supervised learning on the source domain branch, the target domain branch, and the mixed domain branch, respectively, by generating pseudo labels.

[0017] In some preferred embodiments, a weighted loss dynamic adjustment step is further provided, including:

[0018] Convert the target domain pixel-level high-dimensional feature map into a vector form, and evaluate the segmentation effect by combining it with the corresponding pseudo-label;

[0019] Determining the weight of the target domain loss based on the product of the iteration round and the segmentation effect;

[0020] Based on the weight of the target domain loss, a weighted loss is constructed.

[0021] In some preferred embodiments, the sampling frequency of the source domain image is set based on the occurrence frequency of a specific pixel category.

[0022] In some preferred embodiments, the source domain loss is a cross entropy loss.

[0023] Another aspect of the present invention provides an image semantic segmentation scene migration system based on three-branch collaborative training, the system comprising:

[0024] The feature extraction module is configured to perform feature extraction and feature alignment fusion based on the source domain image and the target domain image, respectively, to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map;

[0025] A semantic segmentation module is configured to perform semantic segmentation on each of the fused and decoded high-dimensional feature maps to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map;

[0026] A model training module is configured to calculate the source domain loss, the mixed domain loss, the target domain loss, and the segmentation loss and calculate the weighted loss according to the semantic segmentation result, and then adjust the parameters of the feature extraction module to obtain a trained feature extraction module;

[0027] The semantic classification subsystem is configured to obtain the semantic category of each pixel in the measured image through the trained feature extraction module.

[0028] In some preferred embodiments, the feature extraction module includes a dimensionality-raising encoder for extracting features through dimensionality-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding.

[0029] A third aspect of the present invention provides an electronic device, comprising:

[0030] at least one processor; and

[0031] a memory communicatively connected to at least one of the processors; wherein,

[0032] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0033] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, which stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0034] Beneficial effects of the present invention:

[0035] (1) The present invention proposes an image semantic segmentation scene migration method based on three-branch collaborative training. Through three-branch collaborative training, the knowledge migration capability of the model is improved. With the help of the excellent feature selection and alignment fusion capabilities of the multi-level attention unit, efficient feature alignment and fusion of the source domain and the target domain are achieved, ultimately achieving improved segmentation performance in the target domain and improving the robustness of the model to different domains.

[0036] (2) The present invention is based on the image semantic segmentation scene migration method of three-branch collaborative training. The multi-level attention unit has higher effectiveness for feature alignment and fusion. In addition, different alignment methods are used for features of different semantic categories. By dynamically adjusting the alignment strength at different times, it is beneficial to improve the flexibility of network learning and the stability of network training. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 Schematic diagram of a module for the image semantic segmentation scene migration method based on three-branch collaborative training of the present invention;

[0039] Figure 2 This is a schematic diagram of the feature extraction module of the image semantic segmentation scene migration method based on three-branch collaborative training of the present invention. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0041] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] The present invention provides an image semantic segmentation scene migration method based on three-branch collaborative training. This method improves the knowledge transfer capability of the model through three-branch collaborative training; with the help of the excellent feature selection and alignment capabilities of the attention mechanism, efficient feature alignment of the source domain and the target domain is achieved, ultimately achieving improved segmentation performance in the target domain, while improving the robustness of the model for different domains.

[0043] The present invention provides an image semantic segmentation scene migration method based on three-branch collaborative training, the method comprising:

[0044] Based on the source domain image and the target domain image, feature extraction and feature alignment fusion are performed respectively through the feature extraction module to obtain the fused source domain pixel-level high-dimensional feature map, target domain pixel-level high-dimensional feature map and mixed domain pixel-level high-dimensional feature map;

[0045] The feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding.

[0046] Performing semantic segmentation on the fused and decoded high-dimensional feature maps through a semantic segmentation module to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map;

[0047] According to the semantic segmentation results, the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss are calculated respectively, and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module;

[0048] The semantic category of each pixel in the measured image is obtained through the trained feature extraction module.

[0049] In order to more clearly illustrate the image semantic segmentation scene migration method based on three-branch collaborative training of the present invention, the following Figure 1 and Figure 2 Each step in the embodiment of the present invention is described in detail.

[0050] The image semantic segmentation scene migration method based on three-branch collaborative training in the first embodiment of the present invention is described in detail below:

[0051] Step S100: Acquire a source domain image and a target domain image, and construct a mixed domain image.

[0052] In this embodiment, the sampling frequency of the source domain image is set based on the occurrence frequency of a specific pixel category.

[0053] Sampling frequency of source domain images P for:

[0054] ;

[0055] in, c Indicates a category, Indicates the current category, C Indicates all categories, e represents the base of natural logarithms, TIndicates the smoothness of the sampling frequency control. Indicates the frequency of a certain category of pixels in the data set, Indicates the frequency of pixels of the current category appearing in the dataset.

[0056] The frequency of pixels of a certain category appearing in the dataset for:

[0057] ;

[0058] in, Indicates the number of images in the source domain, H and W Represent the length and width of the image respectively, i Indicates the image sequence index. j Indicates the index of the pixel in the current image.

[0059] For each source domain image sample, according to the sampling frequency P Sampling a certain category c , then from the c Sample an image from the dataset subset of the category:

[0060] ;

[0061] in, express c A subset of the dataset for the category, Represents the sampled source domain image, and uniform represents uniform distribution.

[0062] In step S200, based on the source domain image, the target domain image and the mixed domain image, feature extraction and feature alignment fusion are performed respectively by the feature extraction module 100 to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map and a mixed domain pixel-level high-dimensional feature map.

[0063] The feature extraction module includes a dimensionality-enhancing encoder, a multi-level attention unit, and a decoder:

[0064] The dimensionality-raising encoder extracts features of the input image block by dimensionality-raising coding to obtain a source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map.

[0065] A multi-level attention unit, each level of which includes an attention module and a mixed domain-target domain alignment controller, and each of the attention modules includes a source domain feature module 101, a target domain feature module 102 and a mixed domain feature module 103.

[0066] All source domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the hybrid domain branch of the multi-level attention unit.

[0067] In one example of the present invention, the multi-level attention unit includes four levels, specifically attention module 1, alignment controller 1, attention module 2, alignment controller 2, attention module 3, alignment controller 3, attention module 4 and alignment controller 4 connected in sequence, and attention module 1, attention module 2, attention module 3, and attention module 4 are respectively connected to the decoder in sequence.

[0068] The source domain pixel-level high-dimensional feature map, the target domain pixel-level high-dimensional feature map and the mixed domain pixel-level high-dimensional feature map extracted by the dimensionality-raising encoder are input to the attention module 1 to fuse the input feature map and perform feature decoupling operation. The outputs are alignment controller 1 and attention module 2 respectively. The alignment controller 1 performs feature alignment and outputs to the attention module 2. The attention module 2 fuses the outputs of the attention module 1 and the alignment controller 1 and performs feature decoupling operation again. The outputs are alignment controller 2 and attention module 3 respectively. The alignment controller 2 performs feature alignment and outputs to the attention module 3. The attention module 3 fuses the outputs of the attention module 2 and the alignment controller 2 and performs feature decoupling operation again. The outputs are alignment controller 3 and attention module 4 respectively. The alignment controller 3 performs feature alignment and outputs to the attention module 4. The attention module 4 fuses the outputs of the attention module 3 and the alignment controller 3 and performs feature decoupling operation again. The outputs are alignment controller 4 and decoder respectively. The alignment controller 4 performs feature alignment and outputs to the decoder. The decoder fuses the outputs of the attention module 4 and the alignment controller 4 to perform decoding operation.

[0069] All source domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the hybrid domain branch of the multi-level attention unit.

[0070] Both the dimensionality-raising encoder and decoder are built using deep neural networks.

[0071] In this embodiment, the source domain loss is a cross entropy loss, which is as follows:

[0072] ;

[0073] in, Indicates the i The source domain loss of source domain images, Indicates that the parameter is , and only use the deep neural network of the self-attention mechanism in the attention module, C Represents the total number of semantic categories.

[0074] In this embodiment, the mixed domain-target domain alignment controller performs weighted fusion on the target domain pixel-level high-dimensional feature map of the current level attention unit and the mixed domain pixel-level high-dimensional feature map according to the fusion weight to generate a new mixed domain pixel-level high-dimensional feature map which is input into the next level attention unit.

[0075] The multi-level attention unit of the present invention can achieve efficient alignment of the source domain and the target domain in the feature space. Combined with the mixed domain branch weight size controller, it assists the model in adaptively aligning the optimal semantic content at the appropriate time during training, does not increase the burden of the model during testing, and can effectively improve the accuracy of semantic segmentation.

[0076] In addition to the multi-level attention unit described above, the present invention can also use a hybrid attention mechanism to enable the model to focus on the focused feature through channel attention, and to determine at which position of the focused feature in the input data is most important through spatial attention. This mechanism enables the model to adaptively focus on the information that is most helpful to the final task while suppressing irrelevant information.

[0077] The hybrid attention mechanism is:

[0078] Concatenate the keys and values ​​of images from different branches:

[0079] ;

[0080] ;

[0081] in, A key representing a mixed domain, represents the key of the source domain, represents the key of the target domain, Represents a mixed domain value, Represents the value of the source domain, represents the value of the target domain, represents the real number space, represents the dimension of the key vector, N Indicates the number of pictures.

[0082] Through the query vector of the target domain Keys with mixed domains Multiply to get the similarity matrix, divide the similarity matrix by the square root of the dimension to get the value of the mixed domain Perform weighting to obtain the calculation results of the attention mechanism:

[0083] ;

[0084] Among them, Softmax represents the discriminant function.

[0085] In some instances, a hybrid attention mechanism can choose CBAM (Convolutional Block Attention Module), which first emphasizes important channels in the input feature map through the channel attention module, and then emphasizes important spatial regions through the spatial attention module. This sequential processing ensures that the model can effectively focus on the most informative regions. In CBAM, the channel attention module aggregates the spatial information of the feature map through global average pooling and max pooling operations, and then generates the channel attention map through a shared multi-layer perceptron (MLP). The spatial attention module aggregates channel information by performing max pooling and average pooling operations on the feature map, and then uses standard convolutional layers to generate the spatial attention map.

[0086] The lightweight design of CBAM means that it imposes little computational and parameter overhead on the model, making it an efficient choice for improving model performance.

[0087] In an embodiment of the present invention, the hybrid domain-target domain alignment controller performs weighted fusion of the target domain pixel-level high-dimensional feature map of the current-level attention unit with the hybrid domain pixel-level high-dimensional feature map according to the fusion weight, specifically:

[0088] The source domain branch and the target domain branch are branches that use in-domain attention calculation. For the branches that use in-domain attention calculation, the feature representation is:

[0089] ;

[0090] in, represents the in-domain attention feature, represents a deep neural network, Image representing in-domain attention.

[0091] The mixed domain branch uses mixed domain attention, and the feature representation is:

[0092] ;

[0093] in, represents the mixed attention feature, Deep neural networks use mixed-domain attention computation for network forward propagation.

[0094] Calculate fusion weights :

[0095] ;

[0096] in, Represents a learnable weight generation module, which is used by the network to autonomously select the weight distribution of self-attention features and mixed attention features.

[0097] According to the fusion weight, perform element-by-element weighted summation to obtain the target domain pixel-level high-dimensional feature map :

[0098] .

[0099] Step S300: Perform semantic segmentation on the fused and decoded high-dimensional feature maps through a semantic segmentation module to obtain a semantic segmentation result, which includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map.

[0100] In step S400 , the source domain loss, mixed domain loss, target domain loss, and segmentation loss are calculated based on the semantic segmentation results, and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module.

[0101] In this embodiment, the target domain loss for:

[0102] .

[0103] In this embodiment, the target domain loss is calculated by performing semi-supervised learning on the target domain branch by generating pseudo labels.

[0104] Through the codec model The generated segmentation results and further pseudo labels are:

[0105] ;

[0106] in, Indicates the i The target domain image j Pixel c Pseudo labels for channels, Indicates the current channel, Indicates the i A picture of the target domain, Indicates Iverson brackets.

[0107] The confidence level is higher than the set confidence threshold The uncertainty of pseudo label calculation is:

[0108] ;

[0109] in, Indicates the i The uncertainty of the target domain image.

[0110] Calculate the target domain loss based on the uncertainty:

[0111] ;

[0112] Based on the target domain loss, the model is updated using backpropagation The weight of the t After step training The weights are updated:

[0113] ;

[0114] in, Indicates the t +1 step model parameters, Indicates the t The model parameters of the step, Represents the smoothing coefficient for model update.

[0115] In this embodiment, by The model generates high-quality pseudo labels for the target domain and uses a moving average to update parameters, which can smooth model updates, enhance stability, and minimize the impact of noise in self-training.

[0116] The present invention is also provided with a weighted loss dynamic adjustment step, including:

[0117] Convert the target domain pixel-level high-dimensional feature map into a vector form:

[0118] ;

[0119] in, Indicates the i Image No. j Pixels that match k The probability of the class, k represents the index of the category, Indicates the i Image No. j The vector form of the high-dimensional target domain feature map of pixels.

[0120] Evaluate the segmentation effect based on the target domain pixel-level high-dimensional feature map and pseudo-label in vector form:

[0121] ;

[0122] in, Represents the pseudo labels of the target domain images.

[0123] The weight of the target domain loss is determined based on the product of the iteration round and the segmentation effect :

[0124] ;

[0125] in, Indicates the current iteration round, represents a hyperparameter that controls the speed at which weights are increased.

[0126] Based on the weight of the target domain loss, a weighted loss is constructed:

[0127] .

[0128] Step S500: Obtain the semantic category of each pixel in the measured image through the trained feature extraction module.

[0129] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0130] The image semantic segmentation scene migration system based on three-branch collaborative training according to the second embodiment of the present invention includes:

[0131] The feature extraction module 100 is configured to perform feature extraction and feature alignment fusion based on the source domain image and the target domain image, respectively, to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map;

[0132] The semantic segmentation module 200 is configured to perform semantic segmentation on each of the fused and decoded high-dimensional feature maps to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map;

[0133] A model training module is configured to calculate the source domain loss, the mixed domain loss, the target domain loss, and the segmentation loss and calculate the weighted loss according to the semantic segmentation result, and then adjust the parameters of the feature extraction module to obtain a trained feature extraction module;

[0134] The semantic classification subsystem is configured to obtain the semantic category of each pixel in the measured image through the trained feature extraction module.

[0135] Among them, the feature extraction module includes a dimensionality-raising encoder for feature extraction through dimensionality-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding.

[0136] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0137] It should be noted that the image semantic segmentation scene migration system based on three-branch collaborative training provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules of the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations on the present invention.

[0138] An electronic device according to a third embodiment of the present invention includes:

[0139] at least one processor; and

[0140] a memory communicatively connected to at least one of the processors; wherein,

[0141] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0142] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are used to be executed by the computer to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0145] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0146] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0147] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A scene migration method for image semantic segmentation based on three-branch collaborative training, characterized in that: The method comprises: Based on the source domain image and the target domain image, feature extraction and feature alignment fusion are performed respectively through the feature extraction module to obtain the fused source domain pixel-level high-dimensional feature map, target domain pixel-level high-dimensional feature map and mixed domain pixel-level high-dimensional feature map; The feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment, and hierarchical feature fusion, and a decoder for feature decoding. The multi-level attention unit, each level of which includes an attention module and a hybrid domain-target domain alignment controller, each of which includes a source domain feature module, a target domain feature module and a hybrid domain feature module, all source domain feature modules and the corresponding hybrid domain-target domain alignment controller constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controller constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controller constitute the hybrid domain branch of the multi-level attention unit; The semantic segmentation module performs semantic segmentation on each high-dimensional feature map that is fused and decoded to obtain the semantic segmentation result; According to the semantic segmentation results, the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss are calculated respectively, and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module; The semantic category of each pixel in the measured image is obtained through the trained feature extraction module.

2. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 1 is characterized in that The hybrid domain-target domain alignment controller performs weighted fusion of the target domain pixel-level high-dimensional feature map of the current level attention unit and the hybrid domain pixel-level high-dimensional feature map according to the fusion weight, and generates a new hybrid domain pixel-level high-dimensional feature map which is input into the next level attention unit.

3. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 1 is characterized in that The source domain loss, the target domain loss, and the mixed domain loss are respectively obtained by performing semi-supervised learning on the source domain branch, the target domain branch, and the mixed domain branch by generating pseudo labels.

4. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 3 is characterized in that There is also a weighted loss dynamic adjustment step, including: Convert the target domain pixel-level high-dimensional feature map into a vector form, and evaluate the segmentation effect by combining it with the corresponding pseudo-label; Determining the weight of the target domain loss based on the product of the iteration round and the segmentation effect; Based on the weight of the target domain loss, a weighted loss is constructed.

5. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 1 is characterized in that The sampling frequency of the source domain image is set based on the occurrence frequency of a specific pixel category.

6. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 1 is characterized in that The source domain loss is cross entropy loss.

7. An image semantic segmentation scene migration system based on three-branch collaborative training, characterized by: The system comprises: The feature extraction module is configured to perform feature extraction and feature alignment fusion based on the source domain image and the target domain image, respectively, to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map; The feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding; The multi-level attention unit, each level of which includes an attention module and a hybrid domain-target domain alignment controller, each of which includes a source domain feature module, a target domain feature module and a hybrid domain feature module, all source domain feature modules and the corresponding hybrid domain-target domain alignment controller constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controller constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controller constitute the hybrid domain branch of the multi-level attention unit; A semantic segmentation module is configured to perform semantic segmentation on each of the fused and decoded high-dimensional feature maps to obtain a semantic segmentation result; A model training module is configured to calculate the source domain loss, the mixed domain loss, the target domain loss, and the segmentation loss and the weighted loss according to the semantic segmentation result, and then adjust the parameters of the feature extraction module to obtain a trained feature extraction module; The semantic classification subsystem is configured to obtain the semantic category of each pixel in the measured image through the trained feature extraction module.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the image semantic segmentation scene migration method based on three-branch collaborative training according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to be executed by the computer to implement the image semantic segmentation scene migration method based on three-branch collaborative training according to any one of claims 1 to 6.

Images