Domain adaptation video semantic segmentation learning method based on optical flow cross-domain robustness

Abstract

The embodiment of the application provides a domain adaptation video semantic segmentation learning method based on optical flow cross-domain robustness, and belongs to the technical field of video semantic segmentation. The method comprises the following steps: presetting a source domain labeled training data set, a target domain training set and a target domain test set; constructing a semantic segmentation model for performing semantic segmentation on the source domain video, wherein the semantic segmentation model comprises a SegNet network, a FlowNet network and an SFM network; the SegNet network is used for extracting semantic segmentation information of the input source domain video; the FlowNet network is used for extracting an optical flow label of the input source domain video; the SFM network is used for extracting final semantic segmentation information from the semantic segmentation information under the supervision of the optical flow label; the FlowNet network is trained by using the training data set, so that the FlowNet network can extract the optical flow label from the input source domain video; the SFM network is trained by using the trained FlowNet network and the training data set; and the SegNet network is trained by using the target training data set, the training set, the trained FlowNet network and the SFM network.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation technology for videos, and more specifically to a domain-adaptive video semantic segmentation learning method based on the cross-domain robustness of optical flow. Background Technology

[0002] In recent years, deep learning has achieved remarkable success in the field of computer vision. However, its excellent performance largely depends on a large number of high-quality labeled datasets. For pixel-level prediction tasks such as semantic segmentation, the high time and human costs make manually labeled datasets impractical. Traditional deep learning methods also fail to generalize well to new datasets due to the domain bias problem. To address this, unsupervised domain adaptation leverages knowledge learned in a source domain with a large number of labeled samples to help the model learn in a target domain that is related to the source domain but lacks labels. By reducing domain bias, it can save on the labeling costs in the target domain.

[0003] Common domain adaptation methods involve constructing effective supervision signals on the target domain to adapt the model to the target domain. These methods are either adversarial or based on pseudo-labels. In the patent "A Joint Adversarial Learning Method and System for Domain-Adaptive Semantic Segmentation," multi-level joint adversarial learning ensures cross-domain consistency of learned features across different levels, achieving inter-domain distribution alignment; a weight transfer module is introduced to alleviate overfitting caused by the semantic segmentation model shifting towards the source domain. In the patent "An Unsupervised Domain-Adaptive Semantic Segmentation Method Based on Least Squares Loss," least squares loss is applied to the target domain to increase training on difficult samples, improving the discriminability of target domain features; supervision signals are constructed using pseudo-labels in the target domain to improve semantic segmentation quality in the target domain. In the patent "An Unsupervised Domain-Adaptive Semantic Segmentation Method Based on Region Feature Alignment," pseudo-labels of cross-domain mixed samples are used to apply consistency loss to the student-teacher model, reducing domain differences while improving segmentation quality in the target domain; a region contrast loss is applied to the target domain to further improve the discriminability of target domain features. In the patent "An Unsupervised Domain-Adaptive Semantic Segmentation Method Based on Category Similarity and Dissimilarity Guidance," inter-domain distribution alignment is achieved through image-level and feature-level adversarial losses. A loss function employing a parallel mechanism of intra-class aggregation and inter-class dispersion is used to bring features of the same class closer together and disperse features of different classes, improving the discriminability of target domain features. In the patent "A Semi-Supervised Domain-Adaptive Semantic Segmentation Method and System Based on Feature Prototypes," the feature extraction process is constrained through feature prototype contrastive learning, enhancing the network's ability to extract semantic features and achieving better segmentation results on the target domain data distribution. Most of the above methods use a segmentation model trained in the source domain to construct a supervisory signal in the target domain. However, models biased towards the source domain cannot extract high-quality target domain features, leading to inaccurate supervisory signals. Summary of the Invention

[0004] The purpose of this invention is to provide a domain-adaptive video semantic segmentation learning method based on optical flow cross-domain robustness, which has a stronger ability to extract semantic features.

[0005] To achieve the above objectives, embodiments of the present invention provide a domain-adaptive video semantic segmentation learning method based on optical flow cross-domain robustness, comprising:

[0006] The pre-defined training dataset with source domain annotations, the training set for the target domain, and the test set for the target domain are used.

[0007] A semantic segmentation model for semantic segmentation of source domain video is constructed, wherein the semantic segmentation model includes a SegNet network, a FlowNet network, and an SFM network, wherein the SegNet network is used to extract semantic segmentation information of the input source domain video, the FlowNet network is used to extract optical flow labels of the input source domain video, and the SFM network is used to extract the final semantic segmentation information from the semantic segmentation information under the supervision of the optical flow labels.

[0008] The FlowNet network is trained using the training dataset so that it can extract optical flow labels from the input source domain video.

[0009] The SFM network was trained using the pre-trained FlowNet network and the training dataset.

[0010] The SegNet network is trained using the target training dataset, the training set, the trained FlowNet network, and the SFM network to obtain the trained semantic segmentation model.

[0011] Optionally, the loss function used to train the SFM network is Equation (1).

[0012]

[0013] in, Let F be the loss function, S be the source domain, u and v be the horizontal and vertical components of the optical flow, respectively, and N be the total number of pixels.

[0014] Optionally, the consistency constraint used to train the SFM network is Equation (2).

[0015]

[0016] in, This is the optical flow consistency loss function.

[0017] Optionally, the loss function used to train the SegNet network includes formula (3).

[0018]

[0019] in, Let the loss function be... The loss is cross-entropy, where p(y=i|x) represents the number of classes of pixel i, C is the total number of pixels, and y i denoted by , where x is the set of pixels.

[0020] Optionally, the loss function used to train the SegNet network includes formula (4).

[0021]

[0022] in, Let λ be the loss function. flow This is the weight value.

[0023] Optionally, the pre-labeled training dataset of the source domain, the training set of the target domain, and the test set of the target domain include:

[0024] The image is scaled to 512*1024.

[0025] Normalize the scaled image.

[0026] Optionally, the semantic segmentation learning method further includes:

[0027] The SegNet network is tested using the test set and the trained FlowNet network and SFM network to obtain the trained semantic segmentation model.

[0028] On the other hand, the present invention also provides a domain-adaptive video semantic segmentation learning device based on optical flow cross-domain robustness, the semantic segmentation learning device including a processor for executing the semantic segmentation learning method as described above.

[0029] The domain-adaptive video semantic segmentation learning method based on cross-domain robustness of optical flow provided by this invention constructs a semantic segmentation model consisting of SFM, FlowNet, and SegNet networks. Then, it trains the FlowNet, SFM, and SegNet networks sequentially using a source domain-annotated training dataset, a target domain training set, and a test set, respectively. This approach achieves two goals: firstly, it enables cross-domain robust optical flow to construct a supervisory signal for semantic segmentation in the target domain, resulting in a higher-quality supervisory signal; secondly, the addition of the SFM network (segmentation-optical flow conversion module) establishes a connection between segmentation and optical flow, allowing constraints imposed on optical flow to fully apply to semantic segmentation, thereby improving the quality of unsupervised domain-adaptive semantic segmentation.

[0030] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0031] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0032] Figure 1 This is a flowchart of a domain-adaptive video semantic segmentation learning method based on optical flow cross-domain robustness according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the network structure of a semantic segmentation model according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the Accel network according to one embodiment of the present invention. Detailed Implementation

[0035] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0036] like Figure 1 The diagram shows a flowchart of a domain-adaptive video semantic segmentation learning method based on optical flow cross-domain robustness according to an embodiment of the present invention. Figure 1 In this context, semantic segmentation learning methods can include:

[0037] In step S10, a pre-labeled training dataset for the source domain, a training set for the target domain, and a test set for the target domain are pre-defined.

[0038] In step S11, a semantic segmentation model for semantic segmentation of the source domain video is constructed;

[0039] In step S12, the FlowNet network is trained using the training dataset so that the FlowNet network can extract optical flow labels from the input source domain video.

[0040] In step S13, the SFM network is trained using the pre-trained FlowNet network and the training dataset;

[0041] In step S14, the SegNet network is trained using the target training dataset, the training set, the trained FlowNet network, and the SFM network to obtain the trained semantic segmentation model.

[0042] In such Figure 1 In the semantic segmentation learning method shown, step S10 is used to pre-set the training dataset, training set, and test set. To facilitate image standardization, the image can be scaled to 512*1024 before training, and the scaled image can be normalized.

[0043] Step S11 is used to construct the network structure of the semantic segmentation model, and the network structure can be as follows: Figure 2 As shown. In this Figure 2 In this model, the semantic segmentation model can include SegNet, FlowNet and SFM networks. SegNet is used to extract semantic segmentation information from the input source domain video, FlowNet is used to extract optical flow labels from the input source domain video, and SFM is used to extract the final semantic segmentation information from the semantic segmentation information under the supervision of optical flow labels.

[0044] Step S12 can be used to train the FlowNet network using a training dataset, enabling the FlowNet network to extract optical flow labels from the input source domain video. In one embodiment of the invention, considering that a single FlowNet network cannot be directly trained, the FlowNet network can be first placed into an Accel network, and then removed after training. The structure of the Accel network can be as follows: Figure 3 As shown.

[0045] Step S13 can be used to train the SFM network using the trained FlowNet network and the training dataset. In one example of the present invention, the loss function used to train the SFM network can be Equations (1) and (2).

[0046]

[0047] in, Let F be the loss function, S be the source domain, u and v be the horizontal and vertical components of the optical flow, respectively, and N be the total number of pixels.

[0048]

[0049] in, This is the optical flow consistency loss function.

[0050] Step S14 can be used to train the SegNet network using the target training dataset, training set, trained FlowNet network, and SFM network to obtain a trained semantic segmentation model. In one example of the present invention, the loss function used to train the SegNet network includes formulas (3) and (4).

[0051]

[0052] in, For loss function, The loss is cross-entropy, where p(y=i|x) represents the number of classes of pixel i, C is the total number of pixels, and y i denoted by , where x is the set of pixels.

[0053]

[0054] in, Let λ be the loss function. flow This is the weight value.

[0055] In addition, to ensure the accuracy after training, the SegNet network can be tested using a test set, a pre-trained FlowNet network, and an SFM network to obtain a trained semantic segmentation model.

[0056] On the other hand, the present invention also provides a domain-adaptive video semantic segmentation learning device based on optical flow cross-domain robustness, the semantic segmentation learning device including a processor for executing the semantic segmentation learning method as described above.

[0057] The domain-adaptive video semantic segmentation learning method based on cross-domain robustness of optical flow provided by this invention constructs a semantic segmentation model consisting of SFM, FlowNet, and SegNet networks. Then, it trains the FlowNet, SFM, and SegNet networks sequentially using a source domain-annotated training dataset, a target domain training set, and a test set, respectively. This approach achieves two goals: firstly, it enables cross-domain robust optical flow to construct a supervisory signal for semantic segmentation in the target domain, resulting in a higher-quality supervisory signal; secondly, the addition of the SFM network (segmentation-optical flow conversion module) establishes a connection between segmentation and optical flow, allowing constraints imposed on optical flow to fully apply to semantic segmentation, thereby improving the quality of unsupervised domain-adaptive semantic segmentation.

[0058] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0066] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A domain-adaptive video semantic segmentation learning method based on optical flow cross-domain robustness, characterized in that, The semantic segmentation learning method includes: The pre-defined training dataset with source domain annotations, the training set for the target domain, and the test set for the target domain are used. A semantic segmentation model for semantic segmentation of source domain video is constructed, wherein the semantic segmentation model includes a SegNet network, a FlowNet network, and an SFM network, wherein the SegNet network is used to extract semantic segmentation information of the input source domain video, the FlowNet network is used to extract optical flow labels of the input source domain video, and the SFM network is used to extract the final semantic segmentation information from the semantic segmentation information under the supervision of the optical flow labels. The FlowNet network is trained using the training dataset so that it can extract optical flow labels from the input source domain video. The SFM network was trained using the pre-trained FlowNet network and the training dataset. The SegNet network is trained using the training dataset of the target domain, the training set, the trained FlowNet network, and the SFM network to obtain the trained semantic segmentation model. The loss function used to train the SFM network is Equation (1). ，（1） in, Let the loss function be... For full-map optical flow, For the source domain, and These represent the horizontal and vertical components of the optical flow, respectively. Total number of pixels; The consistency constraint used to train the SFM network is given by formula (2). ，（2） in, Let the optical flow consistency loss function be used. The loss function used to train the SegNet network includes formula (3). ，（3） in, Let the loss function be... For cross-entropy loss, For the pixel point Categories Total number of pixels For pixel annotations, A collection of pixels; The loss function used to train the SegNet network includes formula (4). ，（4） in, Let the loss function be... This is the weight value.

2. The semantic segmentation learning method according to claim 1, characterized in that, The pre-labeled training dataset for the source domain, the training set for the target domain, and the test set for the target domain include: Resize the image to 512*1024 pixels; Normalize the scaled image.

3. The semantic segmentation learning method according to claim 1, characterized in that, The semantic segmentation learning method also includes: The SegNet network is tested using the test set and the trained FlowNet network and SFM network to obtain the trained semantic segmentation model.

4. A domain-adaptive video semantic segmentation learning device based on optical flow cross-domain robustness, characterized in that, The semantic segmentation learning apparatus includes a processor for executing the semantic segmentation learning method as described in any one of claims 1 to 3.

Images