Construction method and device of training data for model training and electronic equipment

Abstract

The invention provides a construction method and device of training data for model training and electronic equipment. According to the specific implementation scheme, three-dimensional scene data of a target road are obtained through three-dimensional reconstruction according to a real image of the target road corresponding to a required road scene, and the required road scene is a scene to which a visual perception model to be trained is applied; according to the three-dimensional scene data and the required road scene, performing simulation to obtain a digital twinborn scene of the target road; according to the needed road scene, a simulation image of a needed view angle is collected from the digital twin scene, and the simulation image comprises first image semantic information; the simulation image is redrawn, a live-action image of the real painting style is obtained, the live-action image is constructed into training data of the visual perception model to be trained, and the live-action image comprises second image semantic information corresponding to the first image semantic information. According to the scheme of the invention, live-action virtual training data can be automatically constructed based on a real image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device and electronic device for constructing training data for model training. Background Art

[0002] The demand for applying neural network models in the automotive industry is constantly increasing. However, simply using the collected real road data as training data to build a data set cannot meet the accuracy requirements of model training. Therefore, it is necessary to construct virtual training data to expand the sample diversity of the data set to improve the accuracy of model training. Summary of the invention

[0003] The present disclosure provides a method, device and electronic device for constructing training data for model training to solve or alleviate one or more technical problems in the prior art.

[0004] In one aspect, the present disclosure provides a method for constructing training data for model training, comprising:

[0005] According to a real image of a target road corresponding to a required road scene, three-dimensional scene data of the target road is obtained through three-dimensional reconstruction, wherein the required road scene is the scene to which the visual perception model to be trained is applied;

[0006] According to the three-dimensional scene data and the required road scene, the digital twin scene of the target road is simulated;

[0007] According to the required road scene, a simulation image of a required perspective is collected from the digital twin scene, wherein the simulation image includes first image semantic information; and

[0008] The simulated image is redrawn to obtain a realistic image of a real painting style, so as to construct the realistic image as training data of a visual perception model to be trained, wherein the realistic image includes second image semantic information corresponding to the first image semantic information.

[0009] On the other hand, the present disclosure provides a device for constructing training data for model training, comprising:

[0010] A three-dimensional reconstruction module, used to obtain three-dimensional scene data of the target road through three-dimensional reconstruction according to a real image of the target road corresponding to the required road scene, wherein the required road scene is the scene to which the visual perception model to be trained is applied;

[0011] A simulation module is used to simulate and obtain a digital twin scene of a target road based on the three-dimensional scene data and the required road scene;

[0012] A collection module, used for collecting a simulation image of a required perspective from the digital twin scene according to a required road scene, wherein the simulation image includes first image semantic information; and

[0013] A redrawing module is used to redraw the simulated image to obtain a realistic image of a real style, so as to construct the realistic image as training data for the visual perception model to be trained, wherein the realistic image includes second image semantic information corresponding to the first image semantic information.

[0014] In another aspect, the present disclosure provides an electronic device, comprising:

[0015] At least one processor. And

[0016] A memory in communication with the at least one processor.

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any embodiment of the present disclosure.

[0018] On the other hand, the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any embodiment of the present disclosure.

[0019] On the other hand, the present disclosure provides a computer program product, including a computer program, which implements the method according to any embodiment of the present disclosure when executed by a processor.

[0020] According to the solution disclosed in the present invention, realistic virtual training data annotated with image semantic information can be automatically constructed based on real images to reduce the cost of constructing training data and improve the accuracy of model training.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0023] Figure 1 It is a flowchart of a method for constructing training data for model training according to an embodiment of the present disclosure.

[0024] Figure 2 It is a schematic diagram of a realistic image obtained by the method for constructing training data for model training according to an embodiment of the present disclosure.

[0025] Figure 3 is a schematic diagram of a device for constructing training data for model training according to an embodiment of the present disclosure.

[0026] Figure 4 It is a block diagram of an electronic device used to implement the method for constructing training data for model training in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0028] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0029] like Figure 1 As shown, the embodiment of the present disclosure provides a method for constructing training data for model training, including:

[0030] Step S101: obtaining three-dimensional scene data of the target road through three-dimensional reconstruction according to a real image of the target road corresponding to the required road scene, wherein the required road scene is the scene to which the visual perception model to be trained is applied.

[0031] Step S102: According to the three-dimensional scene data and the required road scene, a digital twin scene of the target road is simulated.

[0032] Step S103: According to the required road scene, collect a simulation image of the required viewing angle from the digital twin scene, wherein the simulation image includes the first image semantic information. And

[0033] Step S104: redraw the simulated image to obtain a realistic image of a real style, so as to construct the realistic image as training data for the visual perception model to be trained, wherein the realistic image includes second image semantic information corresponding to the first image semantic information.

[0034] According to the embodiments of the present disclosure, it should be noted that:

[0035] The visual perception model trained by the training data constructed in the embodiment of the present disclosure can be applied to the visual sensors of the road-side equipment and / or the visual sensors of the vehicle-side in the vehicle-road collaboration, so that the visual sensors of the vehicle-side or road-side have the ability of visual perception and monitoring. Among them, the visual sensors include but are not limited to cameras, laser radars, millimeter wave radars, etc.

[0036] The required road scene can be understood as the scene where the visual sensor needs to use the visual perception model for visual perception. The required road scene can include various scenes in vehicle driving, such as pedestrians or non-motor vehicles running red lights at intersections, traffic accidents on highways, pedestrians breaking into highways, and rear-end collisions with vehicles ahead on the road. Any scene that needs to be perceived in advance and informed to the vehicle to make driving decisions can be understood as the required road scene.

[0037] The target road can be understood as different types of roads such as intersections, highways, three-way intersections, urban ground roads, etc. in the examples of the required road scenes mentioned above, and no specific limitation is made here.

[0038] The real image may be a target road collected in advance. For example, when the category of the target road is an intersection, the real image may include an image of a certain intersection, or multiple images from different perspectives.

[0039] The three-dimensional scene data of the target road is obtained by three-dimensional reconstruction, which can be understood as obtaining a three-dimensional model of the target road by using the three-dimensional reconstruction technology through the two-dimensional real image corresponding to the target road. The three-dimensional reconstruction technology can adopt any method in the prior art and is not specifically limited here. For example, through the 3D Gaussian Splatting algorithm, the Neural Radiance Fields (NeRF) algorithm, etc.

[0040] The three-dimensional scene data is generated based on the static objects (buildings, roads, basic traffic facilities, etc.) and / or dynamic objects (pedestrians, motor vehicles, non-motor vehicles, pets, etc.) originally contained in the real image of the target road. The digital twin scene adds dynamic objects such as pedestrians, motor vehicles, and non-motor vehicles to the three-dimensional scene data to simulate the required traffic scene. For example, the real image of the target road is that the vehicles at the intersection are driving according to traffic rules, and in the corresponding simulated digital twin scene, additional vehicles that do not comply with traffic rules and run red lights can be added at the intersection, or additional vehicles that collide at the intersection can be added, or additional pedestrians or non-motor vehicles that do not comply with traffic rules and run red lights can be added at the intersection.

[0041] The simulation technology for simulating three-dimensional scene data into a digital twin scene can be implemented by any simulation engine or autonomous driving simulator in the prior art, and is not specifically limited here. For example, Unity Engine, Unreal Engine 4, etc. are used.

[0042] Which dynamic objects need to be added to the digital twin scene can be determined according to the required road scene requirements. For example, if the required road scene is a scene where pedestrians or non-motor vehicles run a red light at an intersection, then pedestrians or non-motor vehicles running a red light are added as dynamic objects.

[0043] The simulated image collected from the digital twin scene can be an image from a certain perspective, or multiple frames of images from a certain perspective, or the same frame or multiple frames of images from different perspectives. The simulated image contains the first image semantic information of static objects and dynamic objects. The first image semantic information can be any information that can be used to characterize a specific static object and dynamic object, for example, the bounding box information of the static object and the dynamic object, object category information, grid information, texture information, vehicle type information, color information, position information, speed information, etc.

[0044] The redrawing technology of the simulated image into a realistic image of a real scene can adopt any generation model in the existing image technology, which is not specifically limited here. For example, it can adopt a diffusion model, a GAN (Generative Adversarial Networks) model, etc.

[0045] like Figure 2 As shown, the realistic image can be understood as an image that is redrawn based on the simulated image and is close to the real scene in terms of its realistic style.

[0046] The second image semantic information may be the same as the first image semantic information, or may be image semantic information adjusted based on the first image semantic information based on the scene-realized image.

[0047] Training data can be understood as the training samples in the dataset used to train the visual perception model.

[0048] When the simulation image and the real-scene image are continuous multiple-frame images, it can be understood that the multiple-frame simulation image is the first video, and the multiple-frame real-scene image is the second video.

[0049] The demand for training data for model training in the autonomous driving and automotive industries is constantly rising. However, due to the relatively small amount of training data for real scenes, especially the training data for special traffic scenes is difficult to collect or rarely occurs, it is not enough to meet the needs of model training. In addition, in order to facilitate model training, the training data needs to be labeled. However, the labeling cost and labeling cycle required to construct the training data are usually long, which will cause trouble for the version replacement of the model. In order to solve the above problems, it is necessary to add virtual training data to the data set for model training to make up for the above problems. However, this will cause another problem. The virtual training data generated by simulation often has a sense of unreality, and the virtual training data often has a large gap in style from the real image, resulting in a loss of accuracy in model training. According to the technology of the embodiment of the present disclosure, by redrawing the simulated image of the target road in a real style (realistic rendering), the redrawn realistic image can have the characteristic style of the real picture and make the style closer to the real image, so that the information represented by the virtual training data is close to the real training data, greatly improving the effect and accuracy of model training. According to the technology of the embodiments of the present disclosure, since the simulated image obtained in the digital twin scene contains image semantic information, the real-life image redrawn based on the simulated image also automatically has image semantic information, thereby realizing that the real-life virtual training data annotated with image semantic information can be automatically constructed based on the real image, reducing the annotation cost of the training data without manual annotation. According to the technology of the embodiments of the present disclosure, corresponding real-life images can be quickly generated for real images of different required road scenes to form general-purpose training data and fine tune data sets, which not only reduces the construction cost of the data set (including multiple training data) for model training, but also improves the efficiency of model training and application deployment.

[0050] In one embodiment, the method for constructing training data for model training provided in the embodiment of the present disclosure includes steps S101 to S104, wherein step S101: obtaining three-dimensional scene data of the target road by three-dimensional reconstruction according to a real image of the target road corresponding to the required road scene, includes:

[0051] According to the real image of the target road corresponding to the required road scene, the static object information in the real image is determined.

[0052] According to the static object information, three-dimensional scene data of the target road is obtained through three-dimensional reconstruction, wherein the three-dimensional scene data includes static object data in the target road.

[0053] According to the embodiments of the present disclosure, it should be noted that:

[0054] Static object information includes information on buildings, roads, and basic transportation facilities on the target road.

[0055] According to the technology of the embodiments of the present disclosure, background three-dimensional scene data that does not contain dynamic objects in the target road can be reconstructed in three dimensions, so as to add the dynamic objects required for the required road scene when simulating the digital twin scene.

[0056] In one implementation, the method for constructing training data for model training provided in the embodiment of the present disclosure includes steps S101 to S104, wherein step S102: simulating a digital twin scene of a target road according to three-dimensional scene data and a required road scene includes:

[0057] Determine the dynamic object data to be deployed based on the required road scene.

[0058] Based on the three-dimensional scene data and dynamic object data, a digital twin scene of the target road is simulated, wherein the digital twin scene includes static objects and dynamic objects.

[0059] According to the embodiments of the present disclosure, it should be noted that:

[0060] Which dynamic objects need to be added to the digital twin scene can be determined according to the required road scene requirements. For example, if the required road scene is a scene where pedestrians or non-motor vehicles run a red light at an intersection, then pedestrians or non-motor vehicles running a red light are added as dynamic objects.

[0061] Dynamic object data may include bounding box information, object category information, mesh information, texture information, vehicle model information, color information, position information, speed information, etc. of the dynamic object.

[0062] The simulation technology for simulating three-dimensional scene data into a digital twin scene can be implemented by any simulation engine or autonomous driving simulator in the prior art, and is not specifically limited here. For example, Unity Engine, Unreal Engine 4, etc. are used.

[0063] According to the technology of the embodiments of the present disclosure, the required dynamic objects can be deployed accordingly in the three-dimensional scene data based on the needs of model training, thereby forming a digital twin scene of the road scene required for model training, and then realizing the simulation image required for redrawing in a real style based on the collection of the digital twin scene.

[0064] In one implementation, the method for constructing training data for model training provided in the embodiment of the present disclosure includes steps S101 to S104, wherein step S103: collecting a simulation image of a required perspective from a digital twin scene according to a required road scene, including:

[0065] Step S1031: According to the required road scene and data set requirement information, collect at least one frame of simulation image from at least one perspective from the digital twin scene.

[0066] According to the embodiments of the present disclosure, it should be noted that:

[0067] The data set requirement information may include the amount of training data required, the perspective information of the training data, etc.

[0068] According to the technology of the embodiments of the present disclosure, by collecting different numbers of simulation images from different perspectives, a fine tune data set for the required road scene to which the model is applied can be generated.

[0069] In one implementation, the method for constructing training data for model training provided in the embodiment of the present disclosure includes steps S101 to S104, and step S1031, and further includes:

[0070] Step S1032: performing data annotation on the simulation image to obtain first image semantic information of static objects and dynamic objects in the simulation image.

[0071] According to the technology of the embodiments of the present disclosure, automatic data annotation of simulation images collected in the digital twin scene can be achieved through the simulation engine.

[0072] In one implementation, the method for constructing training data for model training provided in the embodiment of the present disclosure includes steps S101 to S104, wherein step S104: redrawing the simulation image to obtain a realistic image of a real painting style includes:

[0073] Step S1041: performing a framing operation on a static object or a dynamic object in the simulation image, and / or performing a segmentation operation on the static object and the dynamic object in the simulation image to obtain a preprocessed image.

[0074] Step S1042: redraw the preprocessed image to obtain a realistic image.

[0075] Step S1043: Generate corresponding second image semantic information according to the first image semantic information and the scene-realized image and associate it with the scene-realized image.

[0076] According to the embodiments of the present disclosure, it should be noted that:

[0077] The segment anything segmentation model with prompt can be used to segment static and dynamic objects in the simulation image.

[0078] In order to maintain the invariance of the posture and frame contour of dynamic objects and / or static objects, the object edges and postures can be framed by adding a ControlNet model.

[0079] According to the technology of the embodiments of the present disclosure, it is possible to achieve detailed control over the process of redrawing the style of the simulation image, thereby ensuring the accuracy of the annotation of the generated realistic image (second image semantic information), making it highly close to the realistic image, reducing the gap between the virtual style and the real style, greatly improving the quality of the training data set, and improving the model training effect and accuracy.

[0080] In one implementation, the method for constructing training data for model training provided in the embodiment of the present disclosure includes steps S101 to S104, and steps S1041 to S1043, wherein step S1042: redrawing the preprocessed image to obtain a realistic image of a real painting style includes:

[0081] Customized conditions are determined according to the required road scene, wherein the customized conditions include at least one of lighting conditions, weather conditions, and architectural style conditions.

[0082] According to customized conditions, the pre-processed image is redrawn to obtain a realistic image with a realistic style.

[0083] According to the embodiments of the present disclosure, it should be noted that:

[0084] Lighting conditions may include day, night, dusk, early morning, etc.

[0085] Weather conditions may include sunny, rainy, foggy, etc.

[0086] Architectural styles can include cities, rural streets, Chinese architecture, European architecture, etc.

[0087] According to the technology of the embodiments of the present disclosure, diversified training data of the same realistic image under different lighting, weather or architectural style conditions can be redrawn, so that the constructed training data can further improve the performance and accuracy of the trained model.

[0088] like Figure 3 As shown, the embodiment of the present disclosure provides a device for constructing training data for model training, including:

[0089] The three-dimensional reconstruction module 310 is used to obtain three-dimensional scene data of the target road through three-dimensional reconstruction based on the real image of the target road corresponding to the required road scene, wherein the required road scene is the scene to which the visual perception model to be trained is applied.

[0090] The simulation module 320 is used to simulate and obtain a digital twin scene of the target road based on the three-dimensional scene data and the required road scene.

[0091] The acquisition module 330 is used to acquire a simulation image of a required perspective from the digital twin scene according to the required road scene, wherein the simulation image includes the first image semantic information. And

[0092] The redrawing module 340 is used to redraw the simulated image to obtain a realistic image in a realistic style, so as to construct the realistic image as training data for the visual perception model to be trained, wherein the realistic image includes second image semantic information corresponding to the first image semantic information.

[0093] In one embodiment, the 3D reconstruction module 310 is used to:

[0094] According to the real image of the target road corresponding to the required road scene, the static object information in the real image is determined.

[0095] According to the static object information, three-dimensional scene data of the target road is obtained through three-dimensional reconstruction, wherein the three-dimensional scene data includes static object data in the target road.

[0096] In one embodiment, the simulation module 320 is used to:

[0097] Determine the dynamic object data to be deployed based on the required road scene.

[0098] Based on the three-dimensional scene data and dynamic object data, a digital twin scene of the target road is simulated, wherein the digital twin scene includes static objects and dynamic objects.

[0099] In one embodiment, the acquisition module 330 includes:

[0100] The acquisition submodule is used to acquire at least one frame of simulation image from at least one perspective from the digital twin scene according to the required road scene and data set requirement information.

[0101] In one embodiment, the acquisition module 330 further includes:

[0102] The annotation submodule is used to perform data annotation on the simulation image to obtain first image semantic information of static objects and dynamic objects in the simulation image.

[0103] In one embodiment, the redraw module 340 includes:

[0104] The operation submodule is used to perform a framing operation on a static object or a dynamic object in a simulation image, and / or to perform a segmentation operation on a static object and a dynamic object in a simulation image to obtain a preprocessed image.

[0105] The redrawing submodule is used to redraw the preprocessed image to obtain a realistic image with a realistic style.

[0106] The association submodule is used to generate corresponding second image semantic information according to the first image semantic information and the scene-realized image and associate the second image semantic information with the scene-realized image.

[0107] In one embodiment, the redraw submodule is used to:

[0108] Customized conditions are determined according to the required road scene, wherein the customized conditions include at least one of lighting conditions, weather conditions, and architectural style conditions.

[0109] According to customized conditions, the pre-processed image is redrawn to obtain a realistic image with a realistic style.

[0110] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0111] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0112] Figure 4 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 4 As shown, the electronic device includes: a memory 410 and a processor 420, and the memory 410 stores a computer program that can be run on the processor 420. The number of memories 410 and processors 420 can be one or more. The memory 410 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 430, which is used to communicate with external devices and perform data exchange transmission.

[0113] If the memory 410, the processor 420 and the communication interface 430 are implemented independently, the memory 410, the processor 420 and the communication interface 430 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0114] Optionally, in a specific implementation, if the memory 410, the processor 420 and the communication interface 430 are integrated on a chip, the memory 410, the processor 420 and the communication interface 430 can communicate with each other through an internal interface.

[0115] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0116] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0117] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by the computer, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0118] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0119] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0120] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0121] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0122] The above description is only an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for constructing training data for model training, comprising: According to a real image of a target road corresponding to a required road scene, three-dimensional scene data of the target road is obtained by three-dimensional reconstruction, wherein the required road scene is a scene to which the visual perception model to be trained is applied; According to the three-dimensional scene data and the required road scene, a digital twin scene of the target road is obtained by simulation; According to the required road scene, collecting a simulation image of a required perspective from the digital twin scene, wherein the simulation image includes first image semantic information; and The simulated image is redrawn to obtain a realistic image of a real painting style, so as to construct the realistic image as training data of the visual perception model to be trained, wherein the realistic image includes second image semantic information corresponding to the first image semantic information.

2. The method according to claim 1, wherein: According to the real image of the target road corresponding to the required road scene, three-dimensional scene data of the target road is obtained through three-dimensional reconstruction, including: Determining static object information in a real image of a target road corresponding to a desired road scene; According to the static object information, three-dimensional scene data of the target road is obtained through three-dimensional reconstruction, wherein the three-dimensional scene data includes static object data in the target road.

3. The method according to claim 1, wherein: According to the three-dimensional scene data and the required road scene, a digital twin scene of the target road is simulated, including: Determining dynamic object data to be deployed according to the required road scene; A digital twin scene of the target road is simulated based on the three-dimensional scene data and the dynamic object data, wherein the digital twin scene includes static objects and dynamic objects.

4. The method according to claim 1, wherein: According to the required road scene, a simulation image of a required viewing angle is collected from the digital twin scene, including: According to the required road scene and data set requirement information, at least one frame of simulation image from at least one perspective is collected from the digital twin scene.

5. The method according to claim 4, further comprising: Data annotation is performed on the simulation image to obtain first image semantic information of static objects and dynamic objects in the simulation image.

6. The method according to any one of claims 1 to 5, wherein: The simulation image is redrawn to obtain a realistic image of a real scene, including: Performing a framing operation on a static object or a dynamic object in the simulation image, and / or performing a segmentation operation on the static object and the dynamic object in the simulation image to obtain a preprocessed image; Redrawing the preprocessed image to obtain a realistic image; According to the first image semantic information and the scene-realized image, corresponding second image semantic information is generated and associated with the scene-realized image.

7. The method according to claim 6, wherein: Redrawing the preprocessed image to obtain a realistic image of a real scene, including: Determining customized conditions according to the required road scene, wherein the customized conditions include at least one of lighting conditions, weather conditions, and architectural style conditions; According to the customized conditions, the pre-processed image is redrawn to obtain a realistic image with a realistic style.

8. A device for constructing training data for model training, comprising: A three-dimensional reconstruction module, used to obtain three-dimensional scene data of the target road through three-dimensional reconstruction according to a real image of the target road corresponding to the required road scene, wherein the required road scene is the scene to which the visual perception model to be trained is applied; A simulation module, used for simulating and obtaining a digital twin scene of the target road according to the three-dimensional scene data and the required road scene; A collection module, configured to collect a simulation image of a required viewing angle from the digital twin scene according to the required road scene, wherein the simulation image includes first image semantic information; and A redrawing module is used to redraw the simulated image to obtain a realistic image in a realistic style, so as to construct the realistic image as training data for the visual perception model to be trained, wherein the realistic image includes second image semantic information corresponding to the first image semantic information.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images