Training method and device for road segmentation model based on adversarial network
By constructing an adversarial network-based road segmentation model and utilizing pixel feature comparison and temporal contrast loss, the problems of large training dataset requirements and insufficient generalization ability in existing technologies are solved, achieving high-accuracy road segmentation in different scenarios.
Patent Information
- Application Number
- CN202411300812.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-18
AI Technical Summary
Existing highway pavement segmentation and recognition algorithms require large training datasets, especially for intensive prediction tasks such as semantic segmentation. Furthermore, adversarial network models cannot generalize in practical applications, leading to the need for re-optimization or hardware updates.
A road segmentation model based on adversarial networks is constructed. By combining a feature extraction network and an adversarial network, images of the same type are selected for training using pixel feature comparison. By combining temporal contrast loss and image patch loss, training samples are adaptively acquired to improve segmentation accuracy.
It achieves high accuracy and generalization ability in road segmentation without requiring a large number of training samples, reduces the need for hardware updates, and improves the segmentation accuracy of the model in different scenarios.
Smart Images

Figure CN118823356B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a training method and device for a road segmentation model based on an adversarial network. Background Art
[0002] The highway pavement segmentation and recognition algorithm is a key technology that is of great significance for improving road maintenance efficiency, ensuring traffic safety, reducing management costs, and promoting the development of intelligent transportation systems. The highway pavement segmentation and recognition algorithm can automatically detect road cracks, potholes and other damages, and provide timely repair information for road maintenance teams, thereby avoiding potential traffic accidents and extending the service life of the road ahead. In addition, the use of the highway pavement segmentation and recognition algorithm also reduces the need for manual inspections, significantly improves detection efficiency and reduces labor costs. In the construction of intelligent transportation systems, the pavement segmentation and recognition algorithm provides the basic data required for traffic flow monitoring and vehicle behavior analysis, and at the same time provides support for the environmental understanding ability of autonomous driving vehicles, ensuring the safety of autonomous driving. The high environmental adaptability of the algorithm enables it to work stably under different lighting and weather conditions, providing reliable data support for road asset management systems and helping managers to formulate scientific maintenance plans and budgets. With the advancement of technology, the pavement segmentation and recognition algorithm is not only applicable to highways, but can also be extended to urban and rural roads, with a wide range of cross-domain application potential. It also helps traffic management departments better comply with and implement standards and regulations on road maintenance and safety. In short, this algorithm is an indispensable technical means to achieve intelligent management of road infrastructure, enhance traffic safety, and promote the development of intelligent transportation systems.
[0003] However, conventional highway pavement segmentation and recognition algorithms usually require a large amount of training data sets, especially for intensive prediction tasks such as semantic segmentation, which requires pixel-level annotations, which is expensive and time-consuming, and is particularly demanding in video scenes. Therefore, existing technologies generally choose synthetic data and corresponding annotations, but there are domain differences between the source domain synthetic data and the target domain real data, such as unstable factors such as lighting, weather, and camera parameters. The adversarial network model can transfer the knowledge-labeled source domain data to the unlabeled target domain environment to reduce the domain difference between the source domain and the target domain. However, the tasks of the adversarial network are mostly specific and can only be trained for image samples or video samples alone. In practical applications, they cannot be generalized to each other and need to be re-optimized or updated in hardware.
[0004] In summary, there is an urgent need for a method that can perform accurate road segmentation without requiring a large number of training samples. Summary of the invention
[0005] The embodiments of the present application provide a training method and device for a road segmentation model based on an adversarial network, which selects images of the same type as the current scene based on the comparison of pixel features, and adaptively obtains corresponding subsequent training samples based on the adversarial network, thereby achieving the effect of improving segmentation accuracy without changing the network architecture.
[0006] In a first aspect, an embodiment of the present application provides a method for training a road segmentation model based on an adversarial network, the method comprising:
[0007] Obtain road videos marked with road information as source domain data, and obtain unmarked road videos as target domain data;
[0008] Constructing a road segmentation architecture including a feature extraction network and an adversarial network, wherein the feature extraction network is used to extract features from source domain data and target domain data to obtain source domain pixel features and target domain pixel features, the adversarial network includes a student network and a teacher network, the teacher network is trained using source domain pixel features as training samples and is used for road segmentation, the target domain pixel features are input into the teacher network to generate pseudo labels, and the pseudo labels are the results of the teacher network performing road segmentation on the target domain data;
[0009] The student network uses the target domain pixel features as training samples, and performs adversarial learning with the teacher network based on pseudo labels to obtain a classification head. In the adversarial learning, if the current training sample input to the student network is a static scene, an image with high similarity to the current training sample is selected from the target domain pixel features as a subsequent training sample. If the current training sample input to the student network is a dynamic scene, a similar image with high temporal continuity with the current training sample is selected from the target domain pixel features as a subsequent training sample. The subsequent training sample is the next training sample to be used in this round of iterative training. The static scene is a scene without vehicles passing, and the dynamic scene is a scene with vehicles passing. The feature extraction network and the classification head are combined to obtain a trained road segmentation model.
[0010] In a second aspect, an embodiment of the present application provides a training device for a road segmentation model based on an adversarial network, comprising:
[0011] An acquisition module, used to acquire road videos marked with road information as source domain data, and acquire unmarked road videos as target domain data;
[0012] A construction module is used to construct a road segmentation architecture including a feature extraction network and an adversarial network, wherein the feature extraction network is used to extract features from source domain data and target domain data to obtain source domain pixel features and target domain pixel features, the adversarial network includes a student network and a teacher network, the teacher network is trained with source domain pixel features as training samples and is used for road segmentation, the target domain pixel features are input into the teacher network to generate pseudo labels, and the pseudo labels are the results of the teacher network performing road segmentation on the target domain data;
[0013] Adversarial learning module, the student network uses the target domain pixel features as training samples, and performs adversarial learning with the teacher network based on pseudo labels to obtain a classification head. In adversarial learning, if the current training sample input to the student network is a static scene, an image with high similarity to the current training sample is selected from the target domain pixel features as a subsequent training sample. If the current training sample input to the student network is a dynamic scene, a similar image with high temporal continuity with the current training sample is selected from the target domain pixel features as a subsequent training sample. The subsequent training sample is the next training sample to be used in this round of iterative training. The static scene is a scene without vehicles passing, and the dynamic scene is a scene with vehicles passing. The feature extraction network and the classification head are combined to obtain a trained road segmentation model.
[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a training method for a road segmentation model based on an adversarial network.
[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a training method for a road segmentation model based on an adversarial network.
[0016] The main contributions and innovations of the present invention are as follows:
[0017] The embodiment of the present application trains the student network by constructing an adversarial network architecture including a student network and a teacher network, so that the student network can perform accurate road surface detection without the need for a large number of training sample annotations; the present solution ensures the temporal continuity in the video sequence by constructing a temporal contrast loss during adversarial learning, and by comparing the pixel-level features between adjacent frames or key frames and reference frames in the video sequence, the temporal contrast loss helps to learn consistent representations across frames, thereby better capturing the motion information and dynamic changes of objects in the video; the present solution enhances the consistency of features in local areas by constructing image block losses, and by performing contrast learning in local areas, the block-level contrast loss helps the model learn richer contextual information; the present solution selects images of the same type as the current scene for training according to the differences in the current training samples in the scene and based on the comparison of pixel features, thereby ensuring the accuracy of the model in road surface segmentation in different scenes.
[0018] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1 is a flow chart of a training method for a road segmentation model based on an adversarial network according to an embodiment of the present application;
[0021] Figure 2 It is a structural schematic diagram of a training method of a road segmentation model based on an adversarial network according to an embodiment of the present application;
[0022] Figure 3 It is a structural block diagram of a training device for a road segmentation model based on an adversarial network according to an embodiment of the present application;
[0023] Figure 4 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0025] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0026] Embodiment 1
[0027] The embodiment of the present application provides a training method for a road segmentation model based on an adversarial network, which selects images of the same type as the current scene based on the comparison of pixel features, and adaptively obtains corresponding subsequent training samples based on the adversarial network, thereby achieving the effect of improving the segmentation accuracy without changing the network architecture. Specifically, refer to Figure 1 as well as Figure 2 , the method comprising:
[0028] Obtain road videos marked with road information as source domain data, and obtain unmarked road videos as target domain data;
[0029] Constructing a road segmentation architecture including a feature extraction network and an adversarial network, wherein the feature extraction network is used to extract features from source domain data and target domain data to obtain source domain pixel features and target domain pixel features, the adversarial network includes a student network and a teacher network, the teacher network is trained using source domain pixel features as training samples and is used for road segmentation, the target domain pixel features are input into the teacher network to generate pseudo labels, and the pseudo labels are the results of the teacher network performing road segmentation on the target domain data;
[0030] The student network uses the target domain pixel features as training samples, and performs adversarial learning with the teacher network based on pseudo labels to obtain a classification head. In the adversarial learning, if the current training sample input to the student network is a static scene, an image with high similarity to the current training sample is selected from the target domain pixel features as a subsequent training sample. If the current training sample input to the student network is a dynamic scene, a similar image with high temporal continuity with the current training sample is selected from the target domain pixel features as a subsequent training sample. The subsequent training sample is the next training sample to be used in this round of iterative training. The static scene is a scene without vehicles passing, and the dynamic scene is a scene with vehicles passing. The feature extraction network and the classification head are combined to obtain a trained road segmentation model.
[0031] In some specific embodiments, the videos in the source domain data and the target domain data are collected from cameras on highways or other roads. In order to obtain better detection effects, the collection of source domain data and target domain data covers images under various lighting conditions as much as possible, such as evening, early morning, rainy and snowy weather, etc.
[0032] In this solution, the feature extraction network uses Transformer as a feature extractor to extract features from each image data in the source domain data to obtain source domain image features, and then uses a fully connected layer to project each source domain image feature into a one-dimensional space to obtain a feature representation of each pixel as a source domain pixel feature. The feature extraction network uses Transformer as a feature extractor to extract features from each image data in the target domain data to obtain target domain image features, and then uses a fully connected layer to project each target domain image feature into a one-dimensional space to obtain a feature representation of each pixel as a target domain pixel feature. The formula for feature extraction by the feature extraction network is as follows:
[0033]
[0034] in, is the feature representation of each pixel, is the image feature, FC represents the fully connected layer, and Transformer is the feature extractor.
[0035] Specifically, a feature extraction loss is constructed for the feature extraction network, and the feature extraction loss includes a source domain loss and a target domain loss. The formula of the source domain loss is expressed as follows:
[0036]
[0037] in, represents the source domain loss, and U is the source domain pixel feature The number of images in is corresponding to The feature representation of the pixel point is is a fully connected layer used to predict segmentation categories, Source Domain Image features after passing through the feature extractor Transformer.
[0038] The formula of the target domain loss is as follows:
[0039]
[0040] in, represents the target domain loss, V is the target domain The number of pictures in is corresponding to The feature representation of the pixel point is is the classification function of the fully connected layer used to predict the segmentation category, Target Domain Image features after passing through the feature extractor Transformer.
[0041] Furthermore, the feature extraction loss also includes pixel contrast loss, and the formula of the pixel contrast loss is expressed as follows:
[0042]
[0043] in, , , Indicates the feature representation of different pixels, is the pixel comparison loss, and sim is the similarity comparison.
[0044] Specifically, the formula for similarity comparison is as follows:
[0045]
[0046] Among them, sim is the similarity function, and They represent the feature representation of two pixels in the same image respectively, cos is the cosine similarity, and τ is the temperature parameter, which is used to control the smoothness of the distribution.
[0047] Specifically, this scheme ensures the accuracy of feature extraction by the feature extraction network by constructing feature extraction loss, and can also cluster pixel features of the same category together as much as possible and keep pixel features of different categories away from each other, further ensuring that the subsequent learning network or teacher network performs accurate road segmentation.
[0048] In this scheme, the target domain data is input into the teacher network, and the teacher network performs road segmentation on each image in the target domain data to obtain the segmentation result as a pseudo label, which is expressed as follows:
[0049]
[0050] in, Target Domain 's pseudo-labels.
[0051] In this solution, the structures of the student network and the teacher network can be the same or different. In adversarial learning, the student network updates parameters in a self-supervised manner.
[0052] In this scheme, adversarial learning between the student network and the teacher network can enable the student network to produce better road segmentation effects. That is to say, the segmentation results generated by the student network in the target domain are handed over to the teacher network for judgment based on the pseudo-labels, and the parameters of the student network are guided based on the judgment results. In this adversarial process, the segmentation results generated by the student network will become better and better, and the judgment of the teacher network will become more and more strict. The student network and the teacher network will make progress together to complete the construction of the road segmentation model.
[0053] In adversarial learning, the current training sample of the student network is taken as the current frame, and any image is selected from the historical data as the comparison frame. The temporal contrast loss function is constructed based on the cosine similarity between the current frame and each comparison frame. The formula is as follows:
[0054]
[0055] in, is the time contrast loss, is a key frame, is the historical frame in the historical data, is the similarity of the positive sample pair, where the positive sample is an image pair whose cosine similarity with the key frame is greater than a set threshold, is the cosine similarity between the key frame and each historical frame in the historical data.
[0056] Specifically, the student network uses the temporal contrast loss function to ensure the temporal continuity between consecutive frames in the video sequence. That is, by comparing the pixel-level features of the key frames in the time series with each frame in the historical data, it helps the student network to learn consistent representations across frames, thereby better capturing the motion information and dynamic changes of objects in the video. In addition, the temporal contrast loss reduces the impact of the student network on the segmentation of the road surface caused by fast-passing vehicles in the video by encouraging the model to learn more robust feature representations in the temporal dimension.
[0057] Specifically, the pixel features of the key frame and each historical frame are first obtained, and then the cosine similarity is compared based on the pixel features of the key frame and each historical frame. The formula for obtaining the pixel features is expressed as follows:
[0058]
[0059]
[0060] in, is the pixel feature of the key frame, is the pixel feature of the historical frame.
[0061] In adversarial learning, the first image block and the second image block are cropped in the current training sample of the student network, and the overlapping part of the first image block and the second image block is the same road surface. The image block loss function is constructed based on the cosine similarity of the first image block and the second image block. The formula is as follows:
[0062]
[0063] in, is the image block loss, Represents the current training sample The first image block The pixel m and the second image block The similarity of pixel n, is the sum of similarities between all pixels in the first image block and the second image block.
[0064] Specifically, by constructing an image block loss function, the consistency of features in local regions can be enhanced, which means that in adjacent regions, similar or identical content should have similar feature representations. This scheme helps the model learn richer contextual information by performing comparative learning in local regions.
[0065] In some specific embodiments, the learning network is trained in batch training, and a batch will have multiple training samples for iterative training. In the field of road segmentation, the presence or absence of vehicles in the image will have a relatively large impact on the detection results. Therefore, this solution adopts an intelligent sampling strategy in the process of adversarial learning to ensure the consistency of each batch of training samples.
[0066] Specifically, if the current training sample is a static scene, the pixel features of the current training sample are obtained, and a training sample with a high similarity to the pixel features of the current training sample is selected in the target domain as a subsequent training sample.
[0067] Specifically, if the current training sample is a dynamic scene, the pixel features of the current training sample are obtained, and training samples with high similarity to the pixel features of the current training sample in adjacent time are selected in the target domain as subsequent training samples.
[0068] Therefore, the overall loss function of the road segmentation architecture in this scheme is expressed as follows:
[0069]
[0070] in, is the pixel contrast loss, Represents the teacher network based on the source domain is the loss function for training samples, Represents the teacher network with the target domain is the loss function for training samples, is the time contrast loss, is the image patch loss, α, β, γ are hyperparameters used to balance the contribution of each loss.
[0071] The classification head generated by the student network through adversarial learning has better classification ability than the teacher network because it has undergone adversarial learning. It can better classify the features extracted by the feature extraction network to obtain road segmentation results.
[0072] Embodiment 2
[0073] Based on the same idea, refer to Figure 3 , the present application also proposes a training device for a road segmentation model based on an adversarial network, comprising:
[0074] An acquisition module is used to acquire road videos marked with road information as source domain data and acquire unmarked road videos as target domain data;
[0075] A construction module is used to construct a road segmentation architecture including a feature extraction network and an adversarial network, wherein the feature extraction network is used to extract features from source domain data and target domain data to obtain source domain pixel features and target domain pixel features, the adversarial network includes a student network and a teacher network, the teacher network is trained with source domain pixel features as training samples and is used for road segmentation, the target domain pixel features are input into the teacher network to generate pseudo labels, and the pseudo labels are the results of the teacher network performing road segmentation on the target domain data;
[0076] Adversarial learning module, the student network uses the target domain pixel features as training samples, and performs adversarial learning with the teacher network based on pseudo labels to obtain a classification head. In adversarial learning, if the current training sample input to the student network is a static scene, an image with high similarity to the current training sample is selected from the target domain pixel features as a subsequent training sample. If the current training sample input to the student network is a dynamic scene, a similar image with high temporal continuity with the current training sample is selected from the target domain pixel features as a subsequent training sample. The subsequent training sample is the next training sample to be used in this round of iterative training. The static scene is a scene without vehicles passing, and the dynamic scene is a scene with vehicles passing. The feature extraction network and the classification head are combined to obtain a trained road segmentation model.
[0077] Embodiment 3
[0078] This embodiment also provides an electronic device, referring to Figure 4 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.
[0079] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0080] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.
[0081] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .
[0082] The processor 402 implements any one of the training methods for the road segmentation model based on the adversarial network in the above embodiments by reading and executing the computer program instructions stored in the memory 404 .
[0083] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .
[0084] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.
[0085] The input / output device 408 is used to input or output information. In this embodiment, the input information may be source domain data and target domain data, etc., and the output information may be road segmentation results, etc.
[0086] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:
[0087] Obtain road videos marked with road information as source domain data, and obtain unmarked road videos as target domain data;
[0088] Constructing a road segmentation architecture including a feature extraction network and an adversarial network, wherein the feature extraction network is used to extract features from source domain data and target domain data to obtain source domain pixel features and target domain pixel features, the adversarial network includes a student network and a teacher network, the teacher network is trained using source domain pixel features as training samples and is used for road segmentation, the target domain pixel features are input into the teacher network to generate pseudo labels, and the pseudo labels are the results of the teacher network performing road segmentation on the target domain data;
[0089] The student network uses the target domain pixel features as training samples, and performs adversarial learning with the teacher network based on pseudo labels to obtain a classification head. In the adversarial learning, if the current training sample input to the student network is a static scene, an image with high similarity to the current training sample is selected from the target domain pixel features as a subsequent training sample. If the current training sample input to the student network is a dynamic scene, a similar image with high temporal continuity with the current training sample is selected from the target domain pixel features as a subsequent training sample. The subsequent training sample is the next training sample to be used in this round of iterative training. The static scene is a scene without vehicles passing, and the dynamic scene is a scene with vehicles passing. The feature extraction network and the classification head are combined to obtain a trained road segmentation model.
[0090] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0091] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.
[0092] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 4 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.
[0093] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0094] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A training method for road segmentation model based on adversarial network, It is characterized in that The following steps are involved: Obtain road videos marked with road information as source domain data, and obtain unmarked road videos as target domain data; Constructing a road segmentation architecture including a feature extraction network and an adversarial network, wherein the feature extraction network is used to extract features from source domain data and target domain data to obtain source domain pixel features and target domain pixel features, the adversarial network includes a student network and a teacher network, the teacher network is trained using source domain pixel features as training samples and is used for road segmentation, the target domain pixel features are input into the teacher network to generate pseudo labels, and the pseudo labels are the results of the teacher network performing road segmentation on the target domain data; The student network uses the target domain pixel features as training samples, and performs adversarial learning with the teacher network based on pseudo labels to obtain a classification head. In adversarial learning, if the current training sample input to the student network is a static scene, an image with high similarity to the current training sample is selected from the target domain pixel features as a subsequent training sample. If the current training sample input to the student network is a dynamic scene, a similar image with high temporal continuity with the current training sample is selected from the target domain pixel features as a subsequent training sample. The subsequent training sample is the next training sample to be used in this round of iterative training. The static scene is a scene without vehicle traffic, and the dynamic scene is a scene with vehicle traffic. The feature extraction network and the classification head are combined to obtain a trained road segmentation model. In adversarial learning, the current training sample of the student network is used as the current frame, and any image is selected from the historical data as a comparison frame. The temporal contrast loss function is constructed based on the cosine similarity between the current frame and each comparison frame. The formula is expressed as follows: in, is the time contrast loss, is a key frame, is the historical frame in the historical data, is the similarity of the positive sample pair, where the positive sample is an image pair whose cosine similarity with the key frame is greater than a set threshold, is the cosine similarity between the key frame and each historical frame in the historical data.
2. According to the training method of a road segmentation model based on an adversarial network as described in claim 1, It is characterized in that A feature extraction loss is constructed for the feature extraction network. The feature extraction loss includes a source domain loss and a target domain loss. The formula of the source domain loss is expressed as follows: in, represents the source domain loss, and U is the source domain pixel feature The number of images in is corresponding to The feature representation of the pixel point is is a fully connected layer used to predict segmentation categories, Source Domain Image features after passing through the feature extractor Transformer; The formula of the target domain loss is as follows: in, represents the target domain loss, V is the target domain The number of pictures in is corresponding to The feature representation of the pixel point is is the classification function of the fully connected layer used to predict the segmentation category, Target Domain Image features after passing through the feature extractor Transformer.
3. According to the training method of the road segmentation model based on the adversarial network of claim 2, It is characterized in that The feature extraction loss also includes pixel contrast loss, and the formula of the pixel contrast loss is expressed as follows: in, , , Indicates the feature representation of different pixels, is the pixel comparison loss, and sim is the similarity comparison.
4. The training method of a road segmentation model based on an adversarial network according to claim 1, It is characterized in that In adversarial learning, the first image block and the second image block are cropped in the current training sample of the student network, and the overlapping part of the first image block and the second image block is the same road surface. The image block loss function is constructed based on the cosine similarity of the first image block and the second image block. The formula is as follows: in, is the image block loss, Represents the current training sample The first image block The pixel m and the second image block The similarity of pixel n, is the sum of similarities between all pixels in the first image block and the second image block.
5. According to the training method of the road segmentation model based on the adversarial network of claim 1, It is characterized in that If the current training sample is a static scene, the pixel features of the current training sample are obtained, and a training sample with a high similarity to the pixel features of the current training sample is selected in the target domain as a subsequent training sample.
6. The training method of a road segmentation model based on an adversarial network according to claim 1, It is characterized in that If the current training sample is a dynamic scene, the pixel features of the current training sample are obtained, and training samples with high similarity to the pixel features of the current training sample in adjacent time are selected in the target domain as subsequent training samples.
7. A training device for a road segmentation model based on an adversarial network, It is characterized in that include: An acquisition module is used to acquire road videos marked with road information as source domain data and acquire unmarked road videos as target domain data; A construction module is used to construct a road segmentation architecture including a feature extraction network and an adversarial network, wherein the feature extraction network is used to extract features from source domain data and target domain data to obtain source domain pixel features and target domain pixel features, the adversarial network includes a student network and a teacher network, the teacher network is trained with source domain pixel features as training samples and is used for road segmentation, the target domain pixel features are input into the teacher network to generate pseudo labels, and the pseudo labels are the results of the teacher network performing road segmentation on the target domain data; Adversarial learning module, the student network uses the target domain pixel features as training samples, and performs adversarial learning with the teacher network based on pseudo labels to obtain a classification head. In adversarial learning, if the current training sample input to the student network is a static scene, an image with high similarity to the current training sample is selected from the target domain pixel features as a subsequent training sample. If the current training sample input to the student network is a dynamic scene, a similar image with high temporal continuity with the current training sample is selected from the target domain pixel features as a subsequent training sample. The subsequent training sample is the next training sample to be used in this round of iterative training. The static scene is a scene without vehicle traffic, and the dynamic scene is a scene with vehicle traffic. The feature extraction network and the classification head are combined to obtain a trained road segmentation model. In adversarial learning, the current training sample of the student network is taken as the current frame, and any image is selected from the historical data as a comparison frame. The temporal contrast loss function is constructed based on the cosine similarity between the current frame and each comparison frame. The formula is as follows: in, is the time contrast loss, is a key frame, is the historical frame in the historical data, is the similarity of the positive sample pair, where the positive sample is an image pair whose cosine similarity with the key frame is greater than a set threshold, is the cosine similarity between the key frame and each historical frame in the historical data.
8. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to run the computer program to execute the training method of a road segmentation model based on an adversarial network as described in any one of claims 1-6.
9. A readable storage medium, It is characterized in that The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes a training method for a road segmentation model based on an adversarial network according to any one of claims 1-6.
