Passive domain adaptive semantic segmentation method and device enhanced by diffusion guidance pseudo label

Abstract

The application provides a diffusion guidance pseudo-label enhanced passive domain adaptive semantic segmentation method and device, and relates to the technical field of semantic segmentation. The method comprises the following steps: obtaining an original target domain image and enhancing the original target domain image; predicting the original image and the enhanced image by using a source domain model to obtain initial pseudo-labels; performing confidence sorting and filtering on the initial pseudo-labels respectively to obtain a pseudo-label set, constructing seed pseudo-labels, and adding Gaussian noise to obtain pseudo-labels with different noise levels; extracting context features from the original target domain image by using an encoder, training a diffusion model, and generating pseudo-labels; pairing the pseudo-labels with the original image, performing supervised training on the source domain model according to cross-entropy loss, obtaining a trained model, and completing semantic segmentation of target domain data. The application introduces a diffusion model, starts from a small amount of easily obtained high-quality pseudo-labels, propagates the pseudo-labels to a complete pseudo-label set under the premise of ensuring the quality, and thus realizes pseudo-label optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic segmentation, in particular to a diffusion guided pseudo label enhanced unsupervised domain adaptive semantic segmentation method and device. BACKGROUND

[0002] Unsupervised domain adaptation is a method that does not require source domain data, but only uses target domain unlabeled data and a pre-trained source domain model to adapt the model to a new environment. This method can ensure the privacy and security of the source domain data, and solve the problem of sharing original data due to data protection or security requirements in practical work. It is particularly suitable for scenarios that require data security, such as remote sensing images, medical images, etc., so that the model can still achieve good performance in a new scenario without accessing sensitive data.

[0003] Self-training is a commonly used model optimization method, which refers to using the model's prediction results on unlabeled data as pseudo labels, and then using these pseudo labels to iteratively train the model itself. By continuously updating the pseudo labels and model parameters, the model can gradually improve its performance in new data or new environments. The self-training method can effectively utilize a large number of unlabeled samples, thereby improving the model's generalization ability and application effect, and is particularly suitable for scenarios where labeled data is scarce. The core lies in how to improve the accuracy and reliability of the pseudo labels to ensure the final adaptation effect of the model.

[0004] For the source-free domain adaptation method of semantic segmentation, the existing technology is usually based on self-training, which continuously generates and optimizes pseudo-labels to improve the performance of the model in the target domain. Early methods mainly use confidence filtering, which only uses the high-confidence results predicted by the model on the unlabeled data in the target domain as pseudo-labels for training to reduce the impact of pseudo-label noise. With the in-depth study, the source-free domain adaptation method has proposed various innovative strategies to improve the quality of pseudo-labels. For example, the SFDASEG (Source-Free Domain Adaptive Segmentation) method generates multiple source domain data through data augmentation and combines conditional prior constraints to improve the quality of pseudo-labels generated by the model; the DT-ST (Dynamic Teacher-Update and Self-Training) method introduces a dynamic teacher model update mechanism and combines consistency-based resampling to effectively enhance the stability of the self-training process and the reliability of the pseudo-labels; the CROTS (Cross-Domain Teacher-Student Learning) method uses spatial mixing and rare class mining to alleviate the problems of class imbalance and pseudo-label noise; the SND (Stable Neighbor Denoising) method filters pseudo-labels based on prediction stability and achieves denoising through stable neighbor retrieval and class compensation, further improving the quality of pseudo-labels. The above methods have achieved certain application results in the source-free domain adaptation semantic segmentation task, providing an effective technical path for improving the model's generalization ability and actual deployment performance under the condition of lacking labeled and inaccessible source domain data. However, due to the poor performance of the source domain model in the target domain, the initial pseudo-labels generated by the model are generally of low quality. In order to obtain as much supervision signal as possible, existing methods can only filter part of the pseudo-labels with low quality and high noise, resulting in a large amount of noise in the overall pseudo-labels, which limits the effect of pseudo-label optimization and self-training.

[0005] Therefore, existing methods extract part of the reliable supervision information from the initial pseudo-labels through confidence filtering, but due to the insufficient discrimination ability of the model in detail or ambiguous areas, the model may predict high confidence but be wrong in these areas. This part of the noise reduces the overall quality of the pseudo-labels and cannot be effectively filtered out.

[0006] At the same time, the initial pseudo-labels generated by the source domain model in the target domain usually contain a large amount of noise, and existing methods cannot filter these low-quality pseudo-labels on a large scale to ensure the richness of the supervision information. This part of the pseudo-labels with high confidence but insufficient actual accuracy is mixed with a large amount of noise, which is difficult to filter or correct, severely limiting the effect of subsequent pseudo-label optimization. SUMMARY

[0007] To solve the technical problem of how to effectively improve the accuracy and robustness of target domain semantic segmentation without source domain data, an embodiment of the present application provides a diffusion guide pseudo label enhanced source-free domain adaptive semantic segmentation method and device. The technical solution is as follows:

[0008] On the one hand, a diffusion guide pseudo label enhanced source-free domain adaptive semantic segmentation method is provided, which is realized by a source-free domain adaptive semantic segmentation device, and the method comprises:

[0009] S1, an original target domain image is obtained, and the original target domain image is enhanced through a super-resolution model to obtain an enhanced target domain image.

[0010] S2, the original target domain image and the enhanced target domain image are respectively predicted through a source domain model to obtain an initial pseudo label of the original target domain image and an initial pseudo label of the enhanced target domain image.

[0011] S3, the initial pseudo label of the original target domain image and the initial pseudo label of the enhanced target domain image are respectively subjected to confidence sorting and confidence filtering to obtain a pseudo label set of the original target domain image and a pseudo label set of the enhanced target domain image; and a seed pseudo label is constructed according to the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image.

[0012] S4, Gaussian noise is added to the seed pseudo label to obtain pseudo labels with different noise levels; and a context feature is extracted from the original target domain image by using an encoder.

[0013] S5, a diffusion model is trained according to the pseudo labels with different noise levels, the context feature and the seed pseudo label to obtain a trained diffusion model.

[0014] S6, random Gaussian noise and the context feature are input into the trained diffusion model to generate a pseudo label.

[0015] S7, the pseudo label is paired with the original target domain image, a source domain model is supervisedly trained according to a pairing result and a cross-entropy loss to obtain a trained model, and semantic segmentation of target domain data is completed according to the trained model.

[0016] Optionally, the confidence sorting and the confidence filtering of the initial pseudo label of the original target domain image and the initial pseudo label of the enhanced target domain image in S3 to obtain the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image comprises:

[0017] The confidence of each initial pseudo label in the initial pseudo label of the original target domain image and the initial pseudo label of the enhanced target domain image is sorted in a percentage distribution sorting manner, and a preset percentage of pixels are selected as candidate pseudo labels for confidence filtering according to the sorting result to construct the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image.

[0018] Optionally, the constructing of the seed pseudo label according to the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image in S3 comprises:

[0019] The pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image are subjected to pixel-by-pixel consistency verification to obtain the seed pseudo label.

[0020] Optionally, the adding of the Gaussian noise to the seed pseudo label to obtain the pseudo label with different noise levels in S4 comprises:

[0021] A time step is randomly sampled, and Gaussian noise is added to the seed pseudo label according to the noise intensity corresponding to the time step to obtain the pseudo label with different noise levels.

[0022] On the other hand, a diffusion-guided pseudo label augmented unsupervised domain adaptation semantic segmentation device is provided, which is applied to a diffusion-guided pseudo label augmented unsupervised domain adaptation semantic segmentation method, and the device comprises:

[0023] An image enhancement module is configured to obtain an original target domain image, enhance the original target domain image through a super-resolution model, and obtain an enhanced target domain image.

[0024] An initial pseudo label construction module is configured to predict the original target domain image and the enhanced target domain image through a source domain model to obtain an initial pseudo label of the original target domain image and an initial pseudo label of the enhanced target domain image.

[0025] A seed pseudo label construction module is configured to perform confidence sorting and confidence filtering on the initial pseudo label of the original target domain image and the initial pseudo label of the enhanced target domain image to obtain a pseudo label set of the original target domain image and a pseudo label set of the enhanced target domain image, and construct a seed pseudo label according to the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image.

[0026] A feature extraction module is configured to add Gaussian noise to the seed pseudo label to obtain a pseudo label with different noise levels, and extract context features from the original target domain image by using an encoder.

[0027] The training module is configured to train the diffusion model according to the pseudo labels with different noise levels, the context features and the seed pseudo labels, and obtain a trained diffusion model.

[0028] The generation module is configured to input the random Gaussian noise and the context features into the trained diffusion model, and generate the pseudo labels.

[0029] The output module is configured to pair the pseudo labels with the original target domain images, perform supervised training on the source domain model according to a pairing result and a cross-entropy loss, obtain a trained model, and complete semantic segmentation of the target domain data according to the trained model.

[0030] Optionally, the seed pseudo label construction module is further configured to:

[0031] The confidence of each initial pseudo label in the initial pseudo labels of the original target domain image and the initial pseudo labels of the enhanced target domain image is sorted in a percentage distribution sorting manner, and a preset percentage of pixels is selected as candidate pseudo labels according to a sorting result to realize confidence filtering, and to construct a pseudo label set of the original target domain image and a pseudo label set of the enhanced target domain image.

[0032] Optionally, the seed pseudo label construction module is further configured to:

[0033] The pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image are subjected to pixel-by-pixel consistency verification to obtain the seed pseudo label.

[0034] Optionally, the feature extraction module is further configured to:

[0035] A time step is randomly sampled, and Gaussian noise is added to the seed pseudo label according to a noise intensity corresponding to the time step to obtain the pseudo label with different noise levels.

[0036] In another aspect, a kind of passive field adaptive semantic segmentation equipment is provided, and the passive field adaptive semantic segmentation equipment includes: processor;Memory, computer readable instructions are stored on the memory, the computer readable instructions are executed by the processor, and any one method in the diffusion guided pseudo label enhanced passive domain adaptive semantic segmentation method described above is realized.

[0037] In another aspect, a kind of computer readable storage medium is provided, and at least one instruction is stored in the storage medium, the at least one instruction is loaded and executed by processor to realize any one method in the diffusion guided pseudo label enhanced passive domain adaptive semantic segmentation method described above.

[0038] The technical scheme provided by the embodiment of the application has at least the following beneficial effects:

[0039] In the present application, the accuracy and robustness of target domain semantic segmentation can be effectively improved without source domain data. By fully mining and utilizing the structure and context information in the target domain, combined with the diffusion model, high-quality pseudo-label automatic generation and efficient propagation are realized, providing efficient and reliable technical support for the field adaptive segmentation task under the limited conditions such as unavailability of source data. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 is a diffusion guided pseudo-label enhanced source-free domain adaptive semantic segmentation method flowchart provided by the embodiment of the present application;

[0042] Figure 2 is a structural schematic diagram of a diffusion guided pseudo-label enhanced source-free domain adaptive semantic segmentation method provided by the embodiment of the present application;

[0043] Figure 3 is a diffusion guided pseudo-label enhanced source-free domain adaptive semantic segmentation device block diagram provided by the embodiment of the present application;

[0044] Figure 4 is a structural schematic diagram of a source-free domain adaptive semantic segmentation device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the present application will be described below in combination with the drawings.

[0046] In the embodiments of the present application, the words such as "example", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be either one of the two.

[0047] In the embodiments of the present application, "image" and "picture" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "relevant" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0048] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0049] In order to make the technical problems, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the drawings.

[0050] The embodiments of the present application provide a diffusion guided pseudo label enhanced passive domain adaptive semantic segmentation method, which can be implemented by a passive domain adaptive semantic segmentation device, which can be a terminal or a server. As shown in the flow chart of the diffusion guided pseudo label enhanced passive domain adaptive semantic segmentation method, the processing flow of the method can include the following steps: Figure 1 、 Figure 2 As shown in the flow chart of the diffusion guided pseudo label enhanced passive domain adaptive semantic segmentation method, the processing flow of the method can include the following steps:

[0051] S1, obtaining an original target domain image, enhancing the original target domain image through a super-resolution model to obtain an enhanced target domain image.

[0052] In a feasible implementation, an existing super-resolution model is used to improve the definition and detail performance of the target domain image to obtain the enhanced target domain image.

[0053] S2, predicting the original target domain image and the enhanced target domain image through a source domain model respectively to obtain initial pseudo labels of the original target domain image and initial pseudo labels of the enhanced target domain image.

[0054] In a feasible implementation, a segmentation model trained on the source domain (source domain model) is used to predict the original target domain image and the enhanced image respectively to obtain two groups of initial pseudo labels (including class and confidence).

[0055] S3, performing confidence sorting and confidence filtering on the initial pseudo labels of the original target domain image and the initial pseudo labels of the enhanced target domain image respectively to obtain a pseudo label set of the original target domain image and a pseudo label set of the enhanced target domain image; and constructing seed pseudo labels according to the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image.

[0056] Optionally, the step S3 can include:

[0057] S31, respectively, for each initial pseudo label of each class in the initial pseudo label of the original target domain image and the initial pseudo label of the enhanced target domain image, confidence sorting is performed in a percentage distribution sorting manner, and a preset percentage of pixels are selected as candidate pseudo labels to realize confidence filtering, and a pseudo label set of the original target domain image and a pseudo label set of the enhanced target domain image are constructed.

[0058] S32, the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image are subjected to pixel-by-pixel consistency verification, and seed pseudo labels are obtained.

[0059] In a feasible implementation, for each category, confidence filtering is performed in a percentage distribution sorting manner, for each category, the confidence of all pixels belonging to the category is sorted, and the top percentage of pixels are taken as candidate pseudo labels. This process is separately performed on the original image pseudo label and the enhanced image pseudo label to obtain two sets of high-confidence pseudo label sets.

[0060] Further, the two filtered pseudo label sets are subjected to pixel-by-pixel consistency verification, that is, only those pixels that exist in both sets of pseudo labels and have consistent category predictions are retained as final seed pseudo labels.

[0061] S4, Gaussian noise is added to the seed pseudo label to obtain pseudo labels with different noise levels; and context features are extracted from the original target domain image by using an encoder.

[0062] Optionally, the step S4 of adding Gaussian noise to the seed pseudo label to obtain pseudo labels with different noise levels includes:

[0063] Gaussian noise is added to the seed pseudo label according to the noise intensity corresponding to the time step to obtain pseudo labels with different noise levels.

[0064] In a feasible implementation, for each training sample of the target domain, a time step t is randomly sampled under a diffusion modeling framework, and the step corresponds to a specific noise intensity. Gaussian noise is added to the original seed pseudo label according to the noise level to obtain a noisy pseudo label.

[0065] Meanwhile, context features are extracted from the original target domain image by using an encoder, and the features are provided as conditional input to the diffusion model to guide the model to learn how to recover related pseudo labels according to the context content of the image.

[0066] S5, training the diffusion model with the pseudo labels with different noise levels, the context features and the seed pseudo labels to obtain a trained diffusion model.

[0067] In an implementable embodiment, during the training process, the diffusion model takes the pseudo labels with different noise levels and the context features of the target domain images as joint inputs, and learns how to recover accurate segmentation predictions from different noise levels under the supervision of the seed pseudo labels.

[0068] S6, inputting the random Gaussian noise and the context features into the trained diffusion model to generate pseudo labels.

[0069] In an implementable embodiment, after the training is completed, the diffusion model will enter the inference stage. At this time, the prior label input is no longer relied on, but the random Gaussian noise is taken as the initial state of the pseudo label, and through the joint action with the context features of the target domain images, the segmentation regions conforming to the real semantics are extracted and gradually formed from the pure noise through multi-step iteration denoising, and finally the complete pseudo label segmentation map is generated.

[0070] S7, pairing the pseudo labels with the original target domain images, and performing supervised training on the source domain model according to the pairing results and the cross-entropy loss to obtain a trained model, and completing semantic segmentation of the target domain data according to the trained model.

[0071] In an implementable embodiment, model optimization: after obtaining the high-quality pseudo labels generated by the diffusion model, these labels are regarded as pseudo labels in the target domain and used in the subsequent model optimization stage. In this stage, the pseudo labels are paired with the target domain unlabeled images, and the segmentation model is supervised trained end-to-end through the cross-entropy loss. The model performs segmentation prediction on the target images according to the current parameters in each iteration, and compares the prediction results with the pseudo labels at the pixel level, calculates the cross-entropy loss, and then guides the model to continuously adjust the parameters and improve its adaptability to the distribution characteristics of the target domain.

[0072] For example, remote sensing image data, medical image data and the like to be subjected to semantic segmentation are obtained, the obtained data is enhanced according to the trained model, initial pseudo labels are generated and seed pseudo labels are constructed, and then the diffusion model is used to generate complete high-quality pseudo labels for supervising the training of the model in the target domain, thereby improving the semantic segmentation effect of the model in the target domain.

[0073] From the experimental results, compared with the previous optimal results, the semantic segmentation index miou of the application on the migration task GTA5→Cityscapes is increased by 2.5%, the semantic segmentation index miou on the migration task SYNTHIA→Cityscapes is increased by 2.3%, the semantic segmentation index miou on the migration task Vaihingen→Potsdam is increased by 2.45%, and the semantic segmentation index miou on the migration task Rural→Urban is increased by 2.18%.

[0074] In the embodiment of the application, the accuracy and robustness of target domain semantic segmentation can be effectively improved without source domain data. By fully mining and utilizing the structure and context information in the target domain, combined with the diffusion model, high-quality pseudo-label automatic generation and efficient propagation are realized, providing efficient and reliable technical support for domain adaptive segmentation tasks under limited conditions such as unavailability of source data.

[0075] Figure 3 It is a kind of diffusion guided pseudo label enhanced source free domain adaptive semantic segmentation device diagram according to an exemplary embodiment, the device is used for diffusion guided pseudo label enhanced source free domain adaptive semantic segmentation method. Refer to Figure 3 The device includes an image enhancement module 310, an initial pseudo label construction module 320, a seed pseudo label construction module 330, a feature extraction module 340, a training module 350, a generation module 360 and an output module 370. Wherein:

[0076] The image enhancement module 310 is used for obtaining the original target domain image, enhancing the original target domain image through the super-resolution model, and obtaining the enhanced target domain image.

[0077] The initial pseudo label construction module 320 is used for predicting the original target domain image and the enhanced target domain image through the source domain model respectively, obtaining the initial pseudo label of the original target domain image and the initial pseudo label of the enhanced target domain image.

[0078] The seed pseudo label construction module 330 is used for respectively performing confidence sorting and confidence filtering on the initial pseudo label of the original target domain image and the initial pseudo label of the enhanced target domain image, obtaining the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image, and constructing the seed pseudo label according to the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image.

[0079] The feature extraction module 340 is used for adding Gaussian noise to the seed pseudo label to obtain pseudo labels with different noise levels, and extracting context features from the original target domain image using an encoder.

[0080] The training module 350 is configured to train the diffusion model according to the pseudo label with different noise levels, the context feature and the seed pseudo label, and obtain a trained diffusion model.

[0081] The generating module 360 is configured to input the random Gaussian noise and the context feature into the trained diffusion model, and generate a pseudo label.

[0082] The output module 370 is configured to pair the pseudo label with an original target domain image, perform supervised training on a source domain model according to a pairing result and a cross-entropy loss, obtain a trained model, and complete semantic segmentation of target domain data according to the trained model.

[0083] In the embodiment of the present application, the precision and robustness of the target domain semantic segmentation can be effectively improved without the source domain data. By fully mining and utilizing the structure and context information in the target domain, and combining the diffusion model, the automatic generation and efficient propagation of high-quality pseudo labels are realized, which provides efficient and reliable technical support for the field adaptive segmentation task under the limited conditions such as unavailability of source data.

[0084] Figure 4 is a structural schematic diagram of a passive field adaptive semantic segmentation device provided by the embodiment of the present application, as shown in Figure 4 The passive field adaptive semantic segmentation device can include the diffusion guided pseudo label enhanced passive domain adaptive semantic segmentation apparatus shown in Figure 3 Optionally, the passive field adaptive semantic segmentation device 410 can include the first processor 2001.

[0085] Optionally, the passive field adaptive semantic segmentation device 410 can further include the memory 2002 and the transceiver 2003.

[0086] The first processor 2001, the memory 2002 and the transceiver 2003 can be connected through a communication bus.

[0087] The following will be described in detail Figure 4 The various constituent components of the passive field adaptive semantic segmentation device 410 will be described in detail:

[0088] The first processor 2001 is a control center of the passive domain adaptive semantic segmentation device 410, and can be one processor or a collective name of multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), and can also be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0089] Optionally, the first processor 2001 can execute various functions of the passive domain adaptive semantic segmentation device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0090] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, such as the CPU0 and the CPU1 shown in FIG. 2. Figure 4

[0091] In a specific implementation, as an embodiment, the passive domain adaptive semantic segmentation device 410 can also include multiple processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 2. Each of the processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (for example, computer program instructions). Figure 4

[0092] The memory 2002 is configured to store software programs for implementing the schemes of the present application, and the first processor 2001 is configured to control execution. For specific implementation, refer to the above method embodiments, which will not be repeated here.

[0093] ​​Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled to the first processor 2001 through an interface circuit (not shown in the figure) of the passive field adaptive semantic segmentation device 410. The embodiments of the present application do not make specific limitations hereon. Figure 4

[0094] The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0095] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown separately in the figure). The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function. Figure 4

[0096] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently and be coupled to the first processor 2001 through an interface circuit (not shown in the figure) of the passive field adaptive semantic segmentation device 410. The embodiments of the present application do not make specific limitations hereon. Figure 4

[0097] It should be noted that the structure of the passive field adaptive semantic segmentation device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements. ​

[0098] In addition, the technical effects of the passive field adaptive semantic segmentation device 410 can refer to the technical effects of the diffusion guided pseudo label enhanced passive domain adaptive semantic segmentation method described in the above method embodiments, which will not be repeated here.

[0099] ​​​​It is to be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0100] It is also to be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0101] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0102] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0103] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0104] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0105] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0107] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0109] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0110] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0111] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A diffusion guided pseudo-label augmented passive domain adaptation semantic segmentation method, characterized in that, The method comprises: S1, obtaining an original target domain image, enhancing the original target domain image through a super-resolution model to obtain an enhanced target domain image; S2, predicting the original target domain image and the enhanced target domain image through a source domain model respectively to obtain initial pseudo labels of the original target domain image and initial pseudo labels of the enhanced target domain image; S3, performing confidence sorting and confidence filtering on the initial pseudo labels of the original target domain image and the initial pseudo labels of the enhanced target domain image respectively to obtain a pseudo label set of the original target domain image and a pseudo label set of the enhanced target domain image; and constructing seed pseudo labels according to the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image; S4, adding Gaussian noise to the seed pseudo labels to obtain pseudo labels with different noise levels; and extracting context features from the original target domain image by using an encoder; S5, training a diffusion model according to the pseudo labels with different noise levels, the context features and the seed pseudo labels to obtain a trained diffusion model; S6, inputting random Gaussian noise and the context features into the trained diffusion model to generate pseudo labels; S7, pairing the pseudo labels with the original target domain image, performing supervised training on the source domain model according to a pairing result and a cross-entropy loss to obtain a trained model, and completing semantic segmentation of target domain data according to the trained model.

2. The diffusion guidance pseudo-label augmented passive domain adaptation semantic segmentation method according to claim 1, characterized in that, In the S3, the confidence sorting and the confidence filtering on the initial pseudo labels of the original target domain image and the initial pseudo labels of the enhanced target domain image respectively to obtain the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image comprise: Each of the initial pseudo labels of the original target domain image and the initial pseudo labels of the enhanced target domain image is subjected to confidence sorting in a percentage distribution sorting manner, and a preset percentage of pixels is selected as candidate pseudo labels to realize confidence filtering according to a sorting result, so as to construct the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image.

3. The diffusion guidance pseudo-label augmented passive domain adaptation semantic segmentation method according to claim 1, characterized in that, In the S3, the seed pseudo labels are constructed according to the pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image, which comprises: The pseudo label set of the original target domain image and the pseudo label set of the enhanced target domain image are subjected to pixel-by-pixel consistency verification to obtain the seed pseudo labels.

4. The diffusion guidance pseudo-label augmented passive domain adaptation semantic segmentation method according to claim 1, characterized in that, In the S4, the seed pseudo labels are added with Gaussian noise to obtain the pseudo labels with different noise levels, which comprises: A time step is randomly sampled, and Gaussian noise is added to the seed pseudo labels according to a noise intensity corresponding to the time step to obtain the pseudo labels with different noise levels.

5. A device for diffusion guided pseudo label augmented passive domain adaptive semantic segmentation, the device being configured to implement the diffusion guided pseudo label augmented passive domain adaptive semantic segmentation method according to any one of claims 1 to 4, wherein, The device comprises: An image enhancement module configured to obtain an original target domain image, enhance the original target domain image through a super-resolution model, and obtain an enhanced target domain image; An initial pseudo label construction module configured to predict the original target domain image and the enhanced target domain image through a source domain model respectively to obtain initial pseudo labels of the original target domain image and initial pseudo labels of the enhanced target domain image; An initial pseudo label construction module configured to predict the original target domain image and the enhanced target domain image through a source domain model respectively to obtain initial pseudo labels of the original target domain image and initial pseudo labels of the enhanced target domain image; The seed pseudo-label construction module is configured to perform confidence sorting and confidence filtering on the initial pseudo-labels of the original target domain images and the initial pseudo-labels of the enhanced target domain images respectively, to obtain a pseudo-label set of the original target domain images and a pseudo-label set of the enhanced target domain images; and construct seed pseudo-labels according to the pseudo-label set of the original target domain images and the pseudo-label set of the enhanced target domain images. The feature extraction module is configured to add Gaussian noise to the seed pseudo-labels to obtain pseudo-labels with different noise levels; and extract context features from the original target domain images by using an encoder. The training module is configured to train the diffusion model according to the pseudo-labels with different noise levels, the context features and the seed pseudo-labels, to obtain a trained diffusion model. The generation module is configured to input random Gaussian noise and the context features into the trained diffusion model to generate pseudo-labels. The output module is configured to pair the pseudo-labels with the original target domain images, and perform supervised training on a source domain model according to a pairing result and a cross-entropy loss, to obtain a trained model, and complete semantic segmentation of target domain data according to the trained model.

6. The diffusion guidance pseudo-label augmented passive domain adaptation semantic segmentation apparatus of claim 5, wherein, The confidence sorting and confidence filtering on the initial pseudo-labels of the original target domain images and the initial pseudo-labels of the enhanced target domain images respectively, to obtain the pseudo-label set of the original target domain images and the pseudo-label set of the enhanced target domain images, includes: The confidence sorting and confidence filtering on each initial pseudo-label in the initial pseudo-labels of the original target domain images and the initial pseudo-labels of the enhanced target domain images respectively are performed in a percentage distribution sorting manner, a preset percentage of pixels are selected as candidate pseudo-labels according to a sorting result to realize confidence filtering, and the pseudo-label set of the original target domain images and the pseudo-label set of the enhanced target domain images are constructed.

7. The diffusion guidance pseudo-label augmented passive domain adaptation semantic segmentation apparatus of claim 5, wherein, The seed pseudo-labels are constructed according to the pseudo-label set of the original target domain images and the pseudo-label set of the enhanced target domain images, including: Pixel-by-pixel consistency checking is performed on the pseudo-label set of the original target domain images and the pseudo-label set of the enhanced target domain images to obtain the seed pseudo-labels.

8. The diffusion guidance pseudo-label augmented passive domain adaptation semantic segmentation apparatus of claim 5, wherein, The Gaussian noise is added to the seed pseudo-labels to obtain the pseudo-labels with different noise levels, including: A time step is randomly sampled, and Gaussian noise is added to the seed pseudo-labels according to a noise intensity corresponding to the time step to obtain the pseudo-labels with different noise levels.

9. A device for passive domain-adaptive semantic segmentation, characterized in that The source domain adaptive semantic segmentation device includes: A processor; A memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of any one of claims 1 to 4.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code that can be invoked and executed by the processor to implement the method of any one of claims 1 to 4.

Images