Image semantic segmentation scene migration method, system and equipment based on three-branch cooperative training

Abstract

The invention belongs to the field of image analysis and knowledge migration, particularly relates to an image semantic segmentation scene migration method, system and device based on three-branch cooperative training, and aims to solve the problems of insufficient segmentation performance and insufficient robustness due to insufficient feature alignment and style transfer precision in the existing semantic segmentation technology. The method comprises the steps that feature extraction is carried out on a source domain picture and a target domain picture through a feature extraction module to obtain a source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map and a mixed domain pixel-level high-dimensional feature map, and semantic segmentation is carried out on the source domain pixel-level high-dimensional feature map, the target domain pixel-level high-dimensional feature map and the mixed domain pixel-level high-dimensional feature map; respectively calculating source domain loss, mixed domain loss, target domain loss and segmentation loss based on a semantic segmentation result, and calculating weighted loss, so as to train and adjust parameters of a feature extraction module; and obtaining the semantic category of each pixel in the actually measured image through the trained feature extraction module. The model training stability is high, the flexibility is high, and the subsequent network segmentation performance and robustness are effectively improved.

Description

Background Art

[0002] Semantic segmentation is a basic task in computer vision, which aims to assign a corresponding category label to each pixel in an image. In real applications, segmentation is often required on an unlabeled target domain, which involves the problem of scene migration. The goal of scene migration is to effectively transfer the knowledge learned from the labeled source domain to the unlabeled target domain.

[0003] Unsupervised image scene transfer improves the model performance in the unlabeled target domain by utilizing well-labeled source domain data. This can not only significantly reduce the labeling cost, but also improve the adaptability and generalization ability of the model in different scenarios and environments.

[0004] Existing image scene migration methods mainly rely on techniques such as feature alignment and style transfer to reduce the distribution difference between the source domain and the target domain. However, these methods face many challenges in practical applications. Different semantically unified alignment methods are difficult to achieve the best results, and the alignment time should be adjusted dynamically.

[0005] In application scenarios such as unmanned driving and intelligent monitoring, the accuracy of semantic segmentation has an important impact on the overall performance of the system. Therefore, studying how to achieve high-precision semantic segmentation in unlabeled target domains has important theoretical significance and practical application value. Summary of the invention

[0006] In order to solve the above-mentioned problems in the prior art, that is, the existing semantic segmentation technology has insufficient feature alignment and style transfer accuracy, resulting in insufficient segmentation performance and insufficient robustness, the present invention provides an image semantic segmentation scene migration method based on three-branch collaborative training, the method comprising: Based on the source domain image and the target domain image, feature extraction and feature alignment fusion are performed respectively through the feature extraction module to obtain the fused source domain pixel-level high-dimensional feature map, the target domain pixel-level high-dimensional feature map and the mixed domain pixel-level high-dimensional feature map; The feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding; Performing semantic segmentation of the fused and decoded high-dimensional feature maps respectively through a semantic segmentation module to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map; According to the semantic segmentation results, the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss are respectively calculated, and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module; The semantic category of each pixel in the measured image is obtained through the trained feature extraction module.

[0007] In some preferred embodiments, each level of the multi-level attention unit includes an attention module and a hybrid domain-target domain alignment controller; Each of the attention modules includes a source domain feature module, a target domain feature module and a mixed domain feature module; All source domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the hybrid domain branch of the multi-level attention unit.

[0008] In some preferred embodiments, the mixed domain-target domain alignment controller weightedly fuses the target domain pixel-level high-dimensional feature map of the current level attention unit with the mixed domain pixel-level high-dimensional feature map according to the fusion weight, and generates a new mixed domain pixel-level high-dimensional feature map to input into the next level attention unit.

[0009] In some preferred embodiments, the source domain loss, the target domain loss and the mixed domain loss are respectively obtained by semi-supervised learning of the source domain branch, the target domain branch and the mixed domain branch by generating pseudo labels.

[0010] In some preferred embodiments, a weighted loss dynamic adjustment step is also provided, including: Convert the target domain pixel-level high-dimensional feature map into a vector form, and evaluate the segmentation effect by combining the corresponding pseudo-label; Determine the weight of the target domain loss based on the product of the iteration round and the segmentation effect; Based on the weight of the target domain loss, a weighted loss is constructed.

[0011] In some preferred embodiments, the sampling frequency of the source domain image is set based on the occurrence frequency of a specific pixel category.

[0012] In some preferred embodiments, the source domain loss is a cross entropy loss.

[0013] Another aspect of the present invention provides an image semantic segmentation scene migration system based on three-branch collaborative training, the system comprising: The feature extraction module is configured to perform feature extraction and feature alignment fusion based on the source domain image and the target domain image, respectively, to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map; A semantic segmentation module is configured to perform semantic segmentation of each fused and decoded high-dimensional feature map to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map; A model training module is configured to calculate the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss and the weighted loss according to the semantic segmentation result, and then adjust the parameters of the feature extraction module to obtain a trained feature extraction module; The semantic classification subsystem is configured to obtain the semantic category of each pixel in the measured image through the trained feature extraction module.

[0014] In some preferred embodiments, the feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding.

[0015] According to a third aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0016] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0017] Beneficial effects of the present invention: (1) The present invention proposes an image semantic segmentation scene transfer method based on three-branch collaborative training. Through three-branch collaborative training, the knowledge transfer ability of the model is improved. With the help of the excellent feature selection and alignment fusion capabilities of the multi-level attention unit, efficient feature alignment and fusion of the source domain and the target domain are achieved, and finally the segmentation performance in the target domain is improved, while the robustness of the model to different domains is improved.

[0018] (2) The present invention is based on the image semantic segmentation scene transfer method of three-branch collaborative training. The multi-level attention unit has higher effectiveness for feature alignment and fusion. In addition, different alignment methods are used for features of different semantic categories. By dynamically adjusting the alignment strength at different times, it is beneficial to improve the flexibility of network learning and the stability of network training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a module schematic diagram of the image semantic segmentation scene migration method based on three-branch collaborative training of the present invention; Figure 2 It is a schematic diagram of the feature extraction module of the image semantic segmentation scene migration method based on three-branch collaborative training of the present invention. DETAILED DESCRIPTION

[0020] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] The present invention provides an image semantic segmentation scene migration method based on three-branch collaborative training. The method improves the knowledge migration ability of the model through three-branch collaborative training. With the help of the excellent feature selection and alignment capabilities of the attention mechanism, efficient feature alignment of the source domain and the target domain is achieved, and finally the segmentation performance in the target domain is improved, while the robustness of the model for different domains is improved.

[0023] The present invention provides an image semantic segmentation scene migration method based on three-branch collaborative training, the method comprising: Based on the source domain image and the target domain image, feature extraction and feature alignment fusion are performed respectively through the feature extraction module to obtain the fused source domain pixel-level high-dimensional feature map, the target domain pixel-level high-dimensional feature map and the mixed domain pixel-level high-dimensional feature map; The feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding; Performing semantic segmentation of the fused and decoded high-dimensional feature maps respectively through a semantic segmentation module to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map; According to the semantic segmentation results, the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss are respectively calculated, and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module; The semantic category of each pixel in the measured image is obtained through the trained feature extraction module.

[0024] In order to more clearly illustrate the image semantic segmentation scene migration method based on three-branch collaborative training of the present invention, Figure 1 and Figure 2 Each step in the embodiment of the present invention is described in detail.

[0025] The image semantic segmentation scene migration method based on three-branch collaborative training in the first embodiment of the present invention, each step is described in detail as follows: Step S100, obtaining a source domain image and a target domain image, and constructing a mixed domain image.

[0026] In this embodiment, the sampling frequency of the source domain image is set based on the occurrence frequency of a specific pixel category.

[0027] The sampling frequency of the source domain image P for: ; in, c Indicates a category, Indicates the current category. C Indicates all categories, e represents the base of natural logarithms, T Indicates the smoothness of the sampling frequency control. Indicates the frequency of a certain category of pixels in the data set. Indicates the frequency of pixels of the current category appearing in the dataset.

[0028] The frequency of a certain category of pixels appearing in the dataset for: ; in, represents the number of images in the source domain, H and W Represent the length and width of the image respectively. i Indicates the image sequence index. j Indicates the index of the pixel in the current image.

[0029] For each source domain image sample, according to the sampling frequency P Sampling a category c , then from the c Sample an image from a subset of the dataset for the category: ; in, express c A subset of the dataset for the category, Represents the sampled source domain image, and uniform represents uniform distribution.

[0030] Step S200, based on the source domain image, the target domain image and the mixed domain image, feature extraction and feature alignment fusion are performed by the feature extraction module 100 respectively to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map and a mixed domain pixel-level high-dimensional feature map.

[0031] The feature extraction module includes a dimensional encoder, a multi-level attention unit, and a decoder: The up-dimensional encoder extracts features of the input image block by up-dimensional encoding to obtain a source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map.

[0032] A multi-level attention unit, each level of which includes an attention module and a mixed domain-target domain alignment controller, and each of the attention modules includes a source domain feature module 101, a target domain feature module 102 and a mixed domain feature module 103.

[0033] All source domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the hybrid domain branch of the multi-level attention unit.

[0034] In one example of the present invention, the multi-level attention unit includes four levels, specifically, attention module 1, alignment controller 1, attention module 2, alignment controller 2, attention module 3, alignment controller 3, attention module 4 and alignment controller 4 connected in sequence, and attention module 1, attention module 2, attention module 3, attention module 4 are respectively connected to the decoder in sequence.

[0035] The source domain pixel-level high-dimensional feature map, the target domain pixel-level high-dimensional feature map and the mixed domain pixel-level high-dimensional feature map extracted by the dimension-raising encoder are input to the attention module 1 to fuse the input feature map and then perform feature decoupling operation, and their outputs are alignment controller 1 and attention module 2 respectively. Alignment controller 1 performs feature alignment and outputs to attention module 2. Attention module 2 fuses the outputs of attention module 1 and alignment controller 1 and performs feature decoupling operation again. The outputs are alignment controller 2 and attention module 3 respectively. Alignment controller 2 performs feature alignment and outputs to attention module 3. Attention module 3 fuses the outputs of attention module 2 and alignment controller 2 and performs feature decoupling operation again. The outputs are alignment controller 3 and attention module 4 respectively. Alignment controller 3 performs feature alignment and outputs to attention module 4. Attention module 4 fuses the outputs of attention module 3 and alignment controller 3 and performs feature decoupling operation again. The outputs are alignment controller 4 and decoder respectively. Alignment controller 4 performs feature alignment and outputs to the decoder. The decoder fuses the outputs of attention module 4 and alignment controller 4 to perform decoding operation.

[0036] All source domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the hybrid domain branch of the multi-level attention unit.

[0037] Both the dimension-raising encoder and decoder are built using deep neural networks.

[0038] In this embodiment, the source domain loss is a cross entropy loss, which is as follows: ; in, Indicates i The source domain loss of source domain images, Indicates that the parameter is , and only use the self-attention mechanism in the attention module. C Represents the total number of semantic categories.

[0039] In this embodiment, the mixed domain-target domain alignment controller weightedly fuses the target domain pixel-level high-dimensional feature map of the current level attention unit with the mixed domain pixel-level high-dimensional feature map according to the fusion weight, and generates a new mixed domain pixel-level high-dimensional feature map to input into the next level attention unit.

[0040] The multi-level attention unit of the present invention can achieve efficient alignment of the source domain and the target domain in the feature space. Combined with the mixed domain branch weight size controller, it assists the model in adaptively aligning the optimal semantic content at the appropriate time during the training process, does not increase the burden of the model during testing, and can effectively improve the accuracy of semantic segmentation.

[0041] In addition to the multi-level attention unit described above in the present invention, the present invention can also use a hybrid attention mechanism, which uses channel attention to enable the model to focus on the focused feature, and uses spatial attention to determine at which position of the input data the focused feature is most important. This mechanism enables the model to adaptively focus on the information that is most helpful to the final task while suppressing irrelevant information.

[0042] The hybrid attention mechanism is: Concatenate the keys and values ​​of images from different branches: ; ; in, A key representing a mixed domain, represents the key of the source domain, represents the key of the target domain, Represents a mixed domain value, Represents the value of the source domain, represents the value of the target domain, represents the real number space, represents the dimension of the key vector, N Indicates the number of pictures.

[0043] Through the query vector of the target domain Keys with mixed domains Multiply to get the similarity matrix, divide the similarity matrix by the square root of the dimension to get the value of the mixed domain Perform weighted calculation to obtain the calculation result of the attention mechanism: ; Among them, Softmax represents the discriminant function.

[0044] In some instances, the hybrid attention mechanism can choose CBAM (Convolutional Block Attention Module), which first emphasizes important channels in the input feature map through the channel attention module, and then emphasizes important spatial regions through the spatial attention module. This sequential processing ensures that the model can effectively focus on the most informative areas. In CBAM, the channel attention module aggregates the spatial information of the feature map through global average pooling and maximum pooling operations, and then generates the channel attention map through a shared multi-layer perceptron (MLP). The spatial attention module aggregates the channel information by performing maximum pooling and average pooling operations on the feature map, and then uses a standard convolutional layer to generate the spatial attention map.

[0045] The lightweight design of CBAM means that it imposes little computational and parameter overhead on the model, making it an efficient choice for improving model performance.

[0046] In an embodiment of the present invention, the hybrid domain-target domain alignment controller performs weighted fusion of the target domain pixel-level high-dimensional feature map of the current level attention unit with the hybrid domain pixel-level high-dimensional feature map according to the fusion weight, specifically: The source domain branch and the target domain branch are branches that use in-domain attention calculation. For the branch that uses in-domain attention calculation, the feature representation is: ; in, represents the in-domain attention feature, represents a deep neural network, Image representing in-domain attention.

[0047] The mixed domain branch uses mixed domain attention, and the feature representation is: ; in, represents the mixed attention feature, Deep neural networks use mixed-domain attention computation for the forward propagation of the network.

[0048] Calculate fusion weights : ; in, Represents a learnable weight generation module for the network to autonomously select the weight allocation using self-attention features and mixed attention features.

[0049] According to the fusion weight, perform element-by-element weighted summation to obtain the pixel-level high-dimensional feature map of the target domain : .

[0050] Step S300, performing semantic segmentation on each of the fused and decoded high-dimensional feature maps through a semantic segmentation module to obtain a semantic segmentation result, wherein the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map.

[0051] Step S400, according to the semantic segmentation result, the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss are respectively calculated and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module.

[0052] In this embodiment, the target domain loss for: .

[0053] In this embodiment, the target domain loss is calculated by performing semi-supervised learning on the target domain branch by generating pseudo labels.

[0054] Through the codec model The generated segmentation results and further pseudo labels are: ; in, Indicates i The target domain image j Pixel c Pseudo labels for channels, Indicates the current channel. Indicates i A picture of the target domain, Indicates Iverson brackets.

[0055] The confidence level is higher than the set confidence threshold. The uncertainty of pseudo-label calculation is: ; in, Indicates i The uncertainty of the target domain image.

[0056] Calculate the target domain loss based on the uncertainty: ; Based on the target domain loss, the model is updated using backpropagation The weights are calculated based on the mean shift method. t After step training The weights are updated: ; in, Indicates t+1 step model parameters, Indicates t The model parameters of the step, Represents the smoothing coefficient for model updating.

[0057] In this embodiment, by The model generates high-quality pseudo labels for the target domain and uses a moving average to update parameters, which can smooth model updates, enhance stability, and minimize the impact of noise in self-training.

[0058] The present invention is also provided with a weighted loss dynamic adjustment step, including: Convert the target domain pixel-level high-dimensional feature map into a vector form: ; in, Indicates i Image No. j Pixels match the k The probability of the class, k represents the index of the category, Indicates i Image No. j The vector form of the high-dimensional target domain feature map of pixels.

[0059] Evaluate the segmentation effect based on the target domain pixel-level high-dimensional feature map and pseudo-label in vector form: ; in, Represents the pseudo labels of the target domain images.

[0060] The weight of the target domain loss is determined based on the product of the iteration round and the segmentation effect. : ; in, Indicates the current iteration round, represents a hyperparameter that controls how fast the weights are increased.

[0061] Based on the weight of the target domain loss, a weighted loss is constructed: .

[0062] Step S500, obtaining the semantic category of each pixel in the measured image through the trained feature extraction module.

[0063] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art can understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0064] The image semantic segmentation scene migration system based on three-branch collaborative training according to the second embodiment of the present invention comprises: The feature extraction module 100 is configured to perform feature extraction and feature alignment fusion based on the source domain image and the target domain image, respectively, to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map; The semantic segmentation module 200 is configured to perform semantic segmentation of each fused and decoded high-dimensional feature map to obtain a semantic segmentation result; the semantic segmentation result includes a source domain segmentation prediction map, a first target domain segmentation prediction map, and a second target domain segmentation prediction map; A model training module is configured to calculate the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss and the weighted loss according to the semantic segmentation result, and then adjust the parameters of the feature extraction module to obtain a trained feature extraction module; The semantic classification subsystem is configured to obtain the semantic category of each pixel in the measured image through the trained feature extraction module.

[0065] Among them, the feature extraction module includes a dimension-raising encoder for feature extraction through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding.

[0066] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0067] It should be noted that the image semantic segmentation scene migration system based on three-branch collaborative training provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations of the present invention.

[0068] An electronic device according to a third embodiment of the present invention includes: at least one processor; and a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0069] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned image semantic segmentation scene migration method based on three-branch collaborative training.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the storage device and processing device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0071] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the technical field. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0072] The terms "first", "second", etc. are used to distinguish similar objects rather than to describe or indicate a particular order or sequence.

[0073] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus / device.

[0074] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for image semantic segmentation scene migration based on three-branch collaborative training, characterized in that: The method comprises: Based on the source domain image and the target domain image, feature extraction and feature alignment fusion are performed respectively through the feature extraction module to obtain the fused source domain pixel-level high-dimensional feature map, the target domain pixel-level high-dimensional feature map and the mixed domain pixel-level high-dimensional feature map; The feature extraction module includes a dimension-raising encoder for extracting features by dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding; The semantic segmentation module performs semantic segmentation on each high-dimensional feature map that is fused and decoded, and obtains a semantic segmentation result; According to the semantic segmentation results, the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss are respectively calculated, and the weighted loss is calculated, and then the parameters of the feature extraction module are adjusted to obtain a trained feature extraction module; The semantic category of each pixel in the measured image is obtained through the trained feature extraction module.

2. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 1 is characterized in that: The multi-level attention unit, each level of which includes an attention module and a mixed domain-target domain alignment controller; Each of the attention modules includes a source domain feature module, a target domain feature module and a mixed domain feature module; All source domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the source domain branch of the multi-level attention unit, all target domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the target domain branch of the multi-level attention unit, and all hybrid domain feature modules and the corresponding hybrid domain-target domain alignment controllers constitute the hybrid domain branch of the multi-level attention unit.

3. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 2 is characterized in that: The hybrid domain-target domain alignment controller weightedly fuses the target domain pixel-level high-dimensional feature map of the current level attention unit with the hybrid domain pixel-level high-dimensional feature map according to the fusion weight, and generates a new hybrid domain pixel-level high-dimensional feature map to input into the next level attention unit.

4. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 2 is characterized in that: The source domain loss, the target domain loss and the mixed domain loss are respectively obtained by performing semi-supervised learning on the source domain branch, the target domain branch and the mixed domain branch by a method of generating pseudo labels.

5. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 4 is characterized in that: A weighted loss dynamic adjustment step is also provided, including: Convert the target domain pixel-level high-dimensional feature map into a vector form, and evaluate the segmentation effect by combining the corresponding pseudo-label; Determine the weight of the target domain loss based on the product of the iteration round and the segmentation effect; Based on the weight of the target domain loss, a weighted loss is constructed.

6. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 1 is characterized in that: The sampling frequency of the source domain image is set based on the occurrence frequency of a specific pixel category.

7. The image semantic segmentation scene migration method based on three-branch collaborative training according to claim 1 is characterized in that: The source domain loss is a cross entropy loss.

8. An image semantic segmentation scene migration system based on three-branch collaborative training, characterized in that: The system comprises: The feature extraction module is configured to perform feature extraction and feature alignment fusion based on the source domain image and the target domain image, respectively, to obtain a fused source domain pixel-level high-dimensional feature map, a target domain pixel-level high-dimensional feature map, and a mixed domain pixel-level high-dimensional feature map; The feature extraction module includes a dimension-raising encoder for extracting features through dimension-raising coding, a multi-level attention unit for performing hierarchical feature decoupling, hierarchical feature alignment and hierarchical feature fusion, and a decoder for feature decoding; A semantic segmentation module is configured to perform semantic segmentation of each fused and decoded high-dimensional feature map to obtain a semantic segmentation result; A model training module is configured to calculate the source domain loss, the mixed domain loss, the target domain loss and the segmentation loss and the weighted loss according to the semantic segmentation result, and then adjust the parameters of the feature extraction module to obtain a trained feature extraction module; The semantic classification subsystem is configured to obtain the semantic category of each pixel in the measured image through the trained feature extraction module.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the image semantic segmentation scene migration method based on three-branch collaborative training as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the image semantic segmentation scene migration method based on three-branch collaborative training as described in any one of claims 1-7.

Images