Image generation method and device, model training method and device, and electronic device

Abstract

The present disclosure provides an image generation method and device, a model training method and device, and an electronic device, which can be applied to the technical field of intelligent driving. The image generation method comprises: acquiring a road intersection environment test depth map associated with a road intersection where an autonomous vehicle is located; generating a traffic signal lamp test sketch according to traffic signal lamp position information of the road intersection and a preset traffic signal lamp state; and performing image fusion on the road intersection environment test depth map and the traffic signal lamp test sketch to generate a traffic signal lamp test image, so as to perform driving simulation according to the traffic signal lamp test image, wherein the traffic signal lamp test image can represent the road intersection environment and the preset traffic signal lamp state.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving technology, and more specifically to an image generation method and apparatus, a model training method and apparatus, equipment, media, and program products. Background Technology

[0002] In autonomous driving systems, traffic light modules need to analyze traffic light images to determine the state of the traffic lights. Therefore, during autonomous driving simulation, test images of traffic lights in specified states need to be generated and transmitted to the traffic light module for testing.

[0003] In the process of realizing this disclosure, it was found that the traffic light test images obtained by existing image acquisition methods have the following shortcomings: for example, it is difficult to reproduce or derive traffic light images in dangerous scenarios, or the image distortion is high, or the image information is incomplete. Summary of the Invention

[0004] In view of the above problems, this disclosure provides an image generation method and apparatus, a model training method and apparatus, a device, a medium, and a program product.

[0005] One aspect of this disclosure provides an image generation method, comprising:

[0006] Obtain a test depth map of the intersection environment associated with the intersection where the autonomous vehicle is located;

[0007] Based on the location information of the traffic lights at the intersection and the preset traffic light status, a traffic light test sketch is generated. The traffic light area in the intersection environment test depth map corresponds to the location of the traffic light area in the traffic light test sketch. The traffic light area in the traffic light test sketch is displayed as the preset traffic light status.

[0008] The intersection environment test depth map and the traffic light test sketch are fused to generate a traffic light test image, so as to carry out driving simulation based on the traffic light test image. The traffic light test image can represent the intersection environment and the preset traffic light state.

[0009] According to embodiments of this disclosure, the process of generating a traffic light test image by image fusion of an intersection environment test depth map and a traffic light test sketch includes:

[0010] The intersection environment test depth map and the traffic light test sketch are stitched together in the channel dimension to generate a test stitched image;

[0011] The test stitched image is input into the image generation model so that the image generation model can output a traffic light test image.

[0012] According to embodiments of this disclosure, obtaining the intersection environment test depth map includes:

[0013] Acquire point cloud test data of the intersection environment;

[0014] An intersection environment test depth map is generated based on the intersection environment point cloud test data.

[0015] According to embodiments of this disclosure, the intersection environmental point cloud test data includes three-dimensional coordinate values ​​of the environmental point cloud in a radar coordinate system, and generating an intersection environmental test depth map based on the intersection environmental point cloud test data includes:

[0016] Based on the three-dimensional coordinates of the environmental point cloud in the radar coordinate system, determine the two-dimensional coordinates of the environmental point cloud in the pixel coordinate system, as well as the depth of the environmental point cloud.

[0017] An intersection environment test depth map is generated based on the two-dimensional coordinates and depth values ​​of the environmental point cloud.

[0018] According to embodiments of this disclosure, determining the two-dimensional coordinates of the environmental point cloud in the pixel coordinate system and the depth value of the environmental point cloud based on the three-dimensional coordinates of the environmental point cloud in the radar coordinate system includes:

[0019] Convert the 3D coordinates of the environmental point cloud in the radar coordinate system to the 3D coordinates of the environmental point cloud in the camera coordinate system.

[0020] The three-dimensional coordinates of the environment point cloud in the camera coordinate system are projected onto the camera plane to obtain the two-dimensional coordinates of the environment point cloud in the pixel coordinate system.

[0021] The depth value of the environmental point cloud is determined based on the 3D coordinates of the environmental point cloud in the camera coordinate system.

[0022] According to embodiments of this disclosure, generating an intersection environment test depth map based on two-dimensional coordinate values ​​and depth values ​​of the environmental point cloud includes:

[0023] Based on the two-dimensional coordinate values ​​of the environmental point cloud, the environmental point cloud is marked as pixels in the first blank image;

[0024] Based on the environmental point cloud depth value, pixels at different coordinate positions in the first blank image are marked with different colors to generate an intersection environmental test depth map.

[0025] According to embodiments of this disclosure, the traffic light location information includes the three-dimensional coordinate values ​​of the traffic lights in a predetermined map coordinate system. The step of generating a traffic light test sketch based on the traffic light location information at the intersection and a preset traffic light state includes:

[0026] Convert the three-dimensional coordinates of traffic lights in the predetermined map coordinate system into two-dimensional coordinates of traffic lights in the pixel coordinate system;

[0027] Draw the outline of a traffic light at the target location in the second blank image, where the coordinates of the target location are the same as the two-dimensional coordinates of the traffic light in the pixel coordinate system.

[0028] Based on the preset traffic light status, the outline of the traffic light is filled with a color associated with the preset traffic light status to generate a traffic light test sketch.

[0029] Another aspect of this disclosure provides a model training method, including:

[0030] Obtain a depth map of the intersection environment samples associated with the intersection where the autonomous vehicle is located;

[0031] Based on the location information of the traffic lights at the intersection and the training status of the traffic lights, a traffic light sample sketch is generated. The location of the traffic light area in the intersection environment sample depth map corresponds to the location of the traffic light area in the traffic light sample sketch. The traffic light area in the traffic light sample sketch is displayed as the training status of the traffic lights.

[0032] The intersection environment sample depth map and the traffic light sample sketch are stitched together in the channel dimension to generate a sample stitched image;

[0033] Input the stitched image of the sample into the image generation model to be trained, so that the traffic light training image can be output by the image generation model to be trained;

[0034] Calculate the pixel difference between the training image of the traffic light and the image captured at the intersection;

[0035] The training images of traffic lights and the images captured at the intersection are input into the discrimination model. The model outputs the probability values ​​that the sample images of traffic lights and the images captured at the intersection are the same. When the pixel difference and the probability value meet the preset termination conditions, the trained image generation model is obtained.

[0036] Another aspect of this disclosure provides an image generation apparatus, including a first acquisition module, a first generation module, and a fusion module.

[0037] The first acquisition module is used to acquire a test depth map of the intersection environment associated with the intersection where the autonomous vehicle is located.

[0038] The first generation module is used to generate a traffic light test sketch based on the traffic light location information and preset traffic light status at the intersection. The traffic light area in the intersection environment test depth map corresponds to the location of the traffic light area in the traffic light test sketch, and the traffic light area in the traffic light test sketch is displayed as the preset traffic light status.

[0039] The fusion module is used to fuse the intersection environment test depth map and the traffic light test sketch to generate a traffic light test image, so as to perform driving simulation based on the traffic light test image. The traffic light test image can represent the intersection environment and the preset traffic light state.

[0040] According to embodiments of this disclosure, the fusion module includes a splicing submodule and an input / output submodule.

[0041] The stitching submodule is used to stitch the intersection environment test depth map and the traffic light test sketch in the channel dimension to generate a test stitched image.

[0042] The input / output submodule is used to input the test stitched image into the image generation model, so that the traffic light test image can be output through the image generation model.

[0043] According to embodiments of this disclosure, the first acquisition module includes a first acquisition submodule and a first generation submodule.

[0044] The first acquisition submodule is used to acquire point cloud test data of the intersection environment.

[0045] The first generation submodule is used to generate a test depth map of the intersection environment based on the intersection environment point cloud test data.

[0046] According to an embodiment of this disclosure, the intersection environmental point cloud test data includes three-dimensional coordinate values ​​of the environmental point cloud in a radar coordinate system, and the first generation submodule includes a determination unit and a generation unit.

[0047] The determining unit is used to determine the two-dimensional coordinates of the environmental point cloud in the pixel coordinate system and the depth value of the environmental point cloud based on the three-dimensional coordinates of the environmental point cloud in the radar coordinate system.

[0048] The generation unit is used to generate an intersection environment test depth map based on the two-dimensional coordinate values ​​and depth values ​​of the environmental point cloud.

[0049] According to embodiments of this disclosure, the determining unit includes a first conversion subunit, a second conversion subunit, and a determining subunit.

[0050] The first transformation subunit is used to convert the three-dimensional coordinate values ​​of the environmental point cloud in the radar coordinate system into the three-dimensional coordinate values ​​of the environmental point cloud in the camera coordinate system.

[0051] The second transformation subunit is used to project the three-dimensional coordinates of the environment point cloud in the camera coordinate system onto the camera plane to obtain the two-dimensional coordinates of the environment point cloud in the pixel coordinate system.

[0052] The sub-unit is determined based on the 3D coordinates of the environmental point cloud in the camera coordinate system to determine the depth value of the environmental point cloud.

[0053] According to embodiments of this disclosure, the generation unit includes a first annotation subunit and a second annotation subunit.

[0054] The first annotation subunit is used to annotate the environmental point cloud in the form of pixels in the first blank image based on the two-dimensional coordinate values ​​of the environmental point cloud.

[0055] The second annotation subunit is used to mark pixels at different coordinate positions in the first blank image with different colors according to the environmental point cloud depth value, so as to generate an intersection environmental test depth map.

[0056] According to an embodiment of this disclosure, the traffic light location information includes the three-dimensional coordinate values ​​of the traffic light in a predetermined map coordinate system, and the first generation module includes a conversion unit, a drawing unit, and a filling unit.

[0057] The conversion unit is used to convert the three-dimensional coordinate values ​​of traffic lights in a predetermined map coordinate system into two-dimensional coordinate values ​​of traffic lights in a pixel coordinate system.

[0058] A drawing unit is used to draw the outline of a traffic light at the target location in the second blank image, wherein the coordinate values ​​of the target location are the same as the two-dimensional coordinate values ​​of the traffic light in the pixel coordinate system.

[0059] The fill unit is used to fill the outline of the traffic light with a color associated with the preset traffic light state to generate a traffic light test sketch.

[0060] Another aspect of this disclosure provides a model training apparatus, including a second acquisition module, a second generation module, a splicing module, a first input-output module, a calculation module, and a second input-output module.

[0061] The second acquisition module is used to acquire a depth map of the intersection environment sample associated with the intersection where the autonomous vehicle is located.

[0062] The second generation module is used to generate a traffic light sample sketch based on the traffic light location information and the training traffic light status at the intersection. The traffic light area in the intersection environment sample depth map corresponds to the location of the traffic light area in the traffic light sample sketch, and the traffic light area in the traffic light sample sketch is displayed as the training traffic light status.

[0063] The stitching module is used to stitch together the depth map of the intersection environment sample and the sketch of the traffic light sample in the channel dimension to generate a sample stitched image.

[0064] The first input / output module is used to input the sample stitched image into the image generation model to be trained, so that the traffic light training image can be output by the image generation model to be trained.

[0065] The calculation module is used to calculate the pixel difference between the traffic light training image and the intersection captured image;

[0066] The second input / output module is used to input the traffic light training image and the intersection captured image into the discrimination model, so that the discrimination model outputs the probability value that the traffic light sample image and the intersection captured image are the same. When the pixel difference and the probability value meet the preset termination condition, the trained image generation model is obtained.

[0067] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the image generation method described above.

[0068] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the image generation method described above.

[0069] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the above-described image generation method. Attached Figure Description

[0070] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0071] Figure 1 The illustrations depict application scenarios of image generation methods, apparatus, devices, media, and program products according to embodiments of the present disclosure.

[0072] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically;

[0073] Figure 3 A schematic diagram illustrating the principle of an image generation method according to an embodiment of the present disclosure is shown.

[0074] Figure 4 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.

[0075] Figure 5 A schematic diagram illustrating the principle of a model training method according to an embodiment of the present disclosure is shown.

[0076] Figure 6 A schematic block diagram of an image generation apparatus according to an embodiment of the present disclosure is shown.

[0077] Figure 7 A schematic block diagram of a model training apparatus according to an embodiment of the present disclosure is shown.

[0078] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing an image generation method according to an embodiment of the present disclosure. Detailed Implementation

[0079] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0080] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0081] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0082] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0083] In autonomous driving systems, traffic light modules need to analyze traffic light images to determine the state of the traffic lights. Therefore, during autonomous driving simulation, test images of traffic lights in specified states need to be generated and transmitted to the traffic light module for testing.

[0084] Related technologies include generating traffic light test images through intermediate representation methods, data acquisition methods, and rendering methods.

[0085] The intermediate representation method only directly transmits the traffic light status to the traffic light module and cannot transmit other information, such as environmental information. Therefore, the intermediate representation method can only test some functions of the traffic light module, such as determining the viewpoint of the traffic light. It cannot verify the effectiveness of the traffic light module in inferring the traffic light status.

[0086] Data acquisition methods utilize real-world data to reproduce or derive traffic light images for hazardous scenarios. However, since hazardous scenarios are rarely encountered in natural driving, data acquisition methods cannot capture traffic light images for all hazardous scenarios. Therefore, traffic light module simulations based on data acquisition struggle to reach the safety boundaries of autonomous driving algorithms. Furthermore, errors occur during data reproduction or derivation, causing the traffic light state and position in the generated images to mismatch with the time and actual location of the autonomous vehicle at the intersection.

[0087] The rendering method uses a game engine to generate traffic light images. However, these rendered images often exhibit obvious artifacts from compositing and lack realism. They cannot provide sufficient accuracy and detail for the traffic light module. Furthermore, the rendering method is inefficient, requiring significant manual work and consuming substantial computational resources.

[0088] It is evident that the traffic light test images obtained by the image acquisition methods in the relevant technologies have some shortcomings, such as difficulty in reproducing or deriving traffic light images in dangerous scenarios, high image distortion, or incomplete image information.

[0089] In view of the above, embodiments of this disclosure provide an image generation method, including:

[0090] Obtain a test depth map of the intersection environment associated with the intersection where the autonomous vehicle is located;

[0091] Based on the location information of the traffic lights at the intersection and the preset traffic light status, a traffic light test sketch is generated. The traffic light area in the intersection environment test depth map corresponds to the location of the traffic light area in the traffic light test sketch. The traffic light area in the traffic light test sketch is displayed as the preset traffic light status.

[0092] The intersection environment test depth map and the traffic light test sketch are fused to generate a traffic light test image, so as to carry out driving simulation based on the traffic light test image. The traffic light test image can represent the intersection environment and the preset traffic light state.

[0093] According to embodiments of this disclosure, and because the intersection environment depth map can express more realistic information, the image generated by depth map fusion is distortion-free and can indicate real environmental information, providing sufficient accuracy and detail for the traffic light module, thus solving the technical problem of image distortion generated by rendering methods in the prior art. Furthermore, the traffic light test image generated by fusing the intersection environment test depth map and the traffic light test sketch is similar to a camera-captured image, including not only the traffic light area but also a background area that can display two-dimensional and three-dimensional information of the real intersection environment. The traffic light area can display preset traffic light states, providing more complete information and allowing testing of all functions of the traffic light module, such as verifying the effectiveness of the traffic light module in inferring traffic light states, thus solving the technical problem of incomplete image information generated by intermediate representation methods in the prior art. In addition, because the traffic light state in the traffic light test sketch can be any preset state, including extreme states that violate traffic regulations, it can simulate traffic light images under unconventional and dangerous scenarios, solving the technical problem that existing image acquisition methods cannot reach the safety boundaries of autonomous driving algorithms, and better meeting the needs of vehicle autonomous driving simulation testing.

[0094] Figure 1 The illustrations depict application scenarios of image generation methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0095] like Figure 1 As shown, application scenario 100 according to this embodiment may include vehicle 100 and server 102. Vehicle 100 and server 102 can communicate via a network, which may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0096] The vehicle 100 is equipped with an autonomous driving module to guide the vehicle to achieve autonomous driving. The autonomous driving module includes a traffic light unit, which is used by the vehicle 100 to perform image processing and analysis on received traffic light photos when it approaches a traffic light intersection to determine the status of the traffic lights.

[0097] Before the autonomous driving module of a vehicle is put into production, its performance needs to be tested. Therefore, autonomous driving simulation is required. During the autonomous driving simulation, the server 102 needs to generate a traffic light test image in a specified state and pass it to the traffic light unit in the autonomous driving module of the vehicle 100 to test the traffic light unit.

[0098] Server 102 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0099] Server 102 can be used to execute the image generation method described in the embodiments of this disclosure. For example, it can first obtain a test depth map of the intersection environment and a test sketch of the traffic light, perform image fusion of the test depth map of the intersection environment and the test sketch of the traffic light to generate a test image of the traffic light, and send the test image of the traffic light to the autonomous driving module in the vehicle 100 through the network. The traffic light unit in the autonomous driving module performs a test on the traffic light unit according to the test image of the traffic light.

[0100] It should be noted that the image generation method provided in this embodiment can generally be executed by server 102. Correspondingly, the image generation apparatus provided in this embodiment can generally be located in server 102. The image generation method provided in this embodiment can also be executed by a server or server cluster that is different from server 102 but capable of communicating with server 102. Correspondingly, the image generation apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 102 but capable of communicating with server 102.

[0101] The following will be based on Figure 1 The described scene, through Figures 2-8 The image generation of the disclosed embodiments will be described in detail.

[0102] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically.

[0103] like Figure 2As shown, the image generation method of this embodiment includes operations S201 to S202.

[0104] In operation S201, obtain the intersection environment test depth map associated with the intersection where the autonomous vehicle is located;

[0105] In operation S202, a traffic light test sketch is generated based on the traffic light location information and preset traffic light status at the intersection. Both the intersection environment test depth map and the traffic light test sketch include the traffic light area. The traffic light area in the intersection environment test depth map corresponds to the traffic light area in the traffic light test sketch. The traffic light area in the traffic light test sketch is displayed with the preset traffic light status. In the intersection environment test depth map, pixels representing different depth values ​​are marked and displayed in different colors.

[0106] In operation S203, the intersection environment test depth map and the traffic light test sketch are image fused to generate a traffic light test image, so as to perform driving simulation based on the traffic light test image. The traffic light test image can represent the intersection environment and the preset traffic light state.

[0107] According to embodiments of this disclosure, an autonomous driving module is installed in the autonomous vehicle to guide the vehicle to achieve autonomous driving. The autonomous driving module includes a traffic light unit, which performs image processing and analysis on received traffic light photographs received at the intersection when the vehicle approaches a traffic light intersection to determine the status of the traffic lights.

[0108] Before the autonomous driving module of a vehicle is put into production, its performance needs to be tested. Therefore, autonomous driving simulation is required. During the autonomous driving simulation, a test image of a traffic light in a specified state is generated in advance and transmitted to the traffic light unit in the vehicle's autonomous driving module to test the traffic light unit.

[0109] The image generation method described in the embodiments of this disclosure can generate traffic light test images in any specified state, including traffic light states under special circumstances, and even traffic light test images under extreme states such as traffic light violations, such as when traffic lights are broken or when different directions at the same intersection are both showing green lights.

[0110] According to embodiments of this disclosure, intersection environment test depth maps and traffic light test sketches can be pre-generated and stored in a predetermined storage area of ​​a computer.

[0111] The intersection environment test depth map associated with the intersection where the autonomous vehicle is located can be obtained by processing real intersection environment point cloud data acquired by the vehicle's radar. The traffic light test sketch can be generated based on the set traffic light states.

[0112] According to embodiments of this disclosure, in the intersection environment test depth map, pixels representing different depth values ​​are marked with different colors. The depth value of each pixel in the image represents the distance between the environmental entity and the camera. Therefore, the intersection environment test depth map can represent not only the two-dimensional information of the intersection environmental entity but also its three-dimensional information. However, the traffic lights displayed in the intersection environment test depth map can only represent their geometric information and cannot display their color; therefore, they cannot reflect the status of the traffic lights.

[0113] Both the intersection environment test depth map and the traffic light test sketch include the traffic light area. The intersection environment test depth map includes a background area that displays the actual intersection environment, in addition to the traffic light area. The traffic light test sketch only shows the traffic light area, and the position of the traffic lights in the traffic light area can be indicated by red, yellow, or green to show the traffic light status. The traffic light area in the intersection environment test depth map may not display the traffic light status, while the traffic light area in the traffic light test sketch displays any preset traffic light status. Furthermore, the positions of the traffic light area in the intersection environment test depth map correspond to those in the traffic light test sketch to facilitate accurate image fusion later.

[0114] According to embodiments of this disclosure, and because the intersection environment depth map can express more realistic information, the image generated by depth map fusion is distortion-free and can indicate real environmental information, providing sufficient accuracy and detail for the traffic light module, thus solving the technical problem of image distortion generated by rendering methods in the prior art. Furthermore, the traffic light test image generated by fusing the intersection environment test depth map and the traffic light test sketch is similar to a camera-captured image, including not only the traffic light area but also a background area that can display two-dimensional and three-dimensional information of the real intersection environment. The traffic light area can display preset traffic light states, providing more complete information and allowing testing of all functions of the traffic light module, such as verifying the effectiveness of the traffic light module in inferring traffic light states, thus solving the technical problem of incomplete image information generated by intermediate representation methods in the prior art. In addition, because the traffic light state in the traffic light test sketch can be any preset state, including extreme states that violate traffic regulations, it can simulate traffic light images under unconventional and dangerous scenarios, solving the technical problem that existing image acquisition methods cannot reach the safety boundaries of autonomous driving algorithms, and better meeting the needs of vehicle autonomous driving simulation testing.

[0115] Figure 3 A schematic diagram illustrating the principle of an image generation method according to an embodiment of the present disclosure is shown.

[0116] like Figure 3 As shown, further, the image fusion of the intersection environment test depth map and the traffic light test sketch to generate the traffic light test image can be achieved using a trained image generation model. Specific methods include:

[0117] First, the intersection environment test depth map and the traffic light test sketch are stitched together in the channel dimension to generate a test stitched image.

[0118] Next, the stitched image will be input into the image generation model to output a test image of the traffic light. The image generation model can, for example, use a pre-trained Unet network.

[0119] For example, based on the location of the autonomous vehicle, surrounding point cloud information is acquired and converted into a test depth map of the intersection environment; pre-set traffic light state information in the simulation scenario is acquired and converted into a traffic light test sketch. The intersection environment test depth map and the traffic light test sketch are then stitched together, and a trained Unet network is used to generate a traffic light test image for the specified state. The generated image is then passed to the traffic light module for simulation testing.

[0120] According to embodiments of this disclosure, the intersection environment test depth map can be obtained by processing real intersection environment point cloud data acquired by vehicle radar. Specifically, obtaining the intersection environment test depth map includes: acquiring intersection environment point cloud test data; and generating the intersection environment test depth map based on the intersection environment point cloud test data.

[0121] According to embodiments of this disclosure, obtaining intersection environmental point cloud test data can be achieved by receiving point cloud data sent by vehicle radar. During the data acquisition process, the vehicle travels for a predetermined period of time, such as half an hour. During this period, multiple radars in the vehicle can acquire environmental point cloud data within a 360° field of view around the vehicle. From these point cloud data, the point cloud data corresponding to the field of view of the intersection to be tested is selected as the intersection environmental point cloud test data.

[0122] Furthermore, the final generated intersection environment test depth map needs to be from the camera's perspective. Therefore, during the generation of the depth map, coordinate transformation is required to convert the radar point cloud data from the radar coordinate system to the camera coordinate system.

[0123] Specifically, the intersection environmental point cloud test data includes the three-dimensional coordinate values ​​of the environmental point cloud in the radar coordinate system. The intersection environmental test depth map generated based on the intersection environmental point cloud test data includes:

[0124] First, based on the 3D coordinates of the environmental point cloud in the radar coordinate system, the 2D coordinates of the environmental point cloud in the pixel coordinate system and the depth value of the environmental point cloud are determined. Specifically, this includes: converting the 3D coordinates of the environmental point cloud in the radar coordinate system to the 3D coordinates of the environmental point cloud in the camera coordinate system; projecting the 3D coordinates of the environmental point cloud in the camera coordinate system onto the camera plane based on the camera intrinsic parameters to obtain the 2D coordinates of the environmental point cloud in the pixel coordinate system; and determining the depth value of the environmental point cloud based on the 3D coordinates of the environmental point cloud in the camera coordinate system.

[0125] Next, a test depth map of the intersection environment is generated based on the two-dimensional coordinates and depth values ​​of the environmental point cloud. Specifically, this involves: labeling the environmental point cloud as pixels in a first blank image based on its two-dimensional coordinates; and marking pixels at different coordinate positions in the first blank image with different colors based on their depth values ​​to generate the intersection environment test depth map. Thus, by combining the two-dimensional pixel coordinates and depth values ​​of the point cloud, a depth map corresponding to the point cloud can be obtained. Each pixel in the generated intersection environment test depth map not only represents two-dimensional information (i.e., its position in the image) but also the three-dimensional information of the image entities (the depth value represents the distance of the entity's surface from the camera). Because the intersection environment depth map can express more realistic information, the image generated through depth map fusion is distortion-free, indicating real environmental information and providing sufficient accuracy and detail for traffic light modules.

[0126] According to embodiments of this disclosure, a traffic light test sketch can be generated based on the location information of traffic lights at an intersection and the set traffic light states. Specifically, generating a traffic light test sketch includes:

[0127] First, the location information of the traffic lights at the intersection is obtained. This information includes the three-dimensional coordinates of the traffic lights in a predetermined map coordinate system, indicating the location of the traffic lights at the current test intersection on the map. The specific coordinates of the traffic lights can be obtained based on the traffic light ID and the map. Pre-set traffic light states are also included, for example, the states of the traffic lights in each direction at the current test intersection could be red, green, or yellow.

[0128] Next, a traffic light test sketch is generated based on the traffic light location information and the preset states of the traffic lights. This may include the following operations:

[0129] Operation 1: Convert the three-dimensional coordinates of the traffic lights in the predetermined map coordinate system to the two-dimensional coordinates of the traffic lights in the pixel coordinate system; that is, first convert the coordinates of the traffic lights from the map coordinate system to the camera coordinate system, and convert the three-dimensional coordinates to two-dimensional coordinates.

[0130] Step 2: Draw the outline of the traffic light at the target location in the second blank image, where the coordinates of the target location are the same as the two-dimensional coordinates of the traffic light in the pixel coordinate system; for example, the corresponding rectangular traffic light outline can be drawn in the blank image based on the coordinates of the traffic light obtained in the previous step.

[0131] Step 3: Based on the preset states of the traffic lights, fill the outline of the traffic lights with colors associated with those preset states to generate a traffic light test sketch. That is, fill the drawn rectangle with the corresponding color, such as red, green, yellow, or black, to represent red, green, yellow, and unknown states, respectively, thus obtaining the traffic light sketch.

[0132] It should be noted that the image resolution of the generated traffic light test sketch must be consistent with the resolution of the camera. Furthermore, the image resolution of both the traffic light test sketch and the intersection environment test depth map must be the same, consistent with the camera resolution. Therefore, when the two images are the same size, the position of the traffic light area in the traffic light test sketch obtained by the above method corresponds to the position of the traffic light area in the intersection environment test depth map (i.e., points that are in the same position in the three-dimensional space will be in the same position in the two images). This ensures accurate image fusion of the traffic light test sketch and the intersection environment test depth map in the subsequent process.

[0133] Another aspect of this disclosure provides a model training method. Figure 4 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically. Figure 5 A schematic diagram illustrating the principle of a model training method according to an embodiment of the present disclosure is shown.

[0134] According to embodiments of this disclosure, such as Figure 5 As shown above, Figure 3 The methods used in the illustrated embodiments for image fusion using an image generation model to generate traffic light test images differ slightly. Figure 5 The model training process shown improves the training performance of the image generation model Unet by introducing the discriminative model PatchGAN.

[0135] During training, the image generation model Unet and the discriminative model PatchGAN are trained alternately.

[0136] The following section first introduces the training method for the Unet image generation model during the alternating training process. For example... Figure 4 , Figure 5 As shown, the model training method in this embodiment includes operations S401 to S406.

[0137] In operation S401, obtain the intersection environment sample depth map associated with the intersection where the autonomous vehicle is located;

[0138] In operation S402, a traffic light sample sketch is generated based on the traffic light location information and the training traffic light status at the intersection. The traffic light area in the intersection environment sample depth map corresponds to the location of the traffic light area in the traffic light sample sketch, and the traffic light area in the traffic light sample sketch is displayed as the training traffic light status.

[0139] In operation S403, the intersection environment sample depth map and traffic light sample sketch are stitched together in the channel dimension to generate a sample stitched image.

[0140] In operation S404, the stitched image of the samples is input into the image generation model to be trained, so that the model can output the traffic light training image; for example... Figure 5 As shown, the sample depth map corresponding to the intersection environment point cloud and the sample sketch of the traffic light can be stitched together in the channel dimension and used as input to the Unet network. The Unet network is then used as an image generation model to generate training images containing the traffic light states. During the image generation process of the Unet network, the upsampled feature map is fused with the feature map from the feature extraction process to obtain contextual and high-resolution information.

[0141] In operation S405, the pixel difference between the traffic light training image and the intersection captured image is calculated;

[0142] In operation S406, the training images of traffic lights and the images captured at the intersection are input into the discriminative model PatchGAN for classification (during this process, the model parameters of the discriminative model remain unchanged). The discriminative model outputs a probability value that the traffic light sample image and the intersection captured image are the same, thus obtaining the image authenticity judgment result. When the pixel difference and probability value meet the preset termination condition, the trained image generation model is obtained. For example, BCELoss and L1Loss can be used as loss functions to calculate the corresponding loss values ​​for pixel difference and probability value, thus completing the training of the image generation model Unet.

[0143] According to embodiments of this disclosure, the operation process for training the discriminative model PatchGAN during alternating training is largely the same as the operation process for training the image generation model Unet, and will not be repeated here. The difference is that during the training phase of the discriminative model, the network parameters of the image generation model remain unchanged, and the loss function is BCELoss, which is used to calculate the accuracy of classifying the traffic light training images and the intersection images. When the accuracy of image classification reaches a preset threshold, the trained discriminative model is obtained.

[0144] According to embodiments of this disclosure, by introducing the discriminative model PatchGAN, the generated image and the real image are input into PatchGAN for classification to obtain an image authenticity judgment, which can improve the authenticity of the generated image and improve the ability to generate details of high-resolution images. It can be seen that by introducing and using PatchGAN, the training effect of the image generation model Unet can be improved.

[0145] Based on the above image generation method, this disclosure also provides an image generation apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0146] Figure 6 A schematic diagram illustrating the structure of an image generation apparatus according to an embodiment of the present disclosure is shown, such as... Figure 6 As shown, the image generation device 600 of this embodiment includes a first acquisition module 601, a first generation module 602, and a fusion module 603.

[0147] The system includes a first acquisition module 601, used to acquire a test depth map of the intersection environment associated with the intersection where the autonomous vehicle is located; a first generation module 602, used to generate a traffic light test sketch based on the traffic light location information and preset traffic light states at the intersection, wherein the traffic light area in the intersection environment test depth map corresponds to the location of the traffic light area in the traffic light test sketch, and the traffic light area in the traffic light test sketch displays the preset traffic light states; and a fusion module 602, used to perform image fusion of the intersection environment test depth map and the traffic light test sketch to generate a traffic light test image, so as to perform driving simulation based on the traffic light test image, wherein the traffic light test image can represent the intersection environment and the preset traffic light states.

[0148] According to embodiments of this disclosure, a test depth map of the intersection environment and a test sketch of the traffic lights are obtained through a first acquisition module 601 and a first generation module 602. Since the intersection environment depth map can express more realistic information, the fusion module 602 fuses the intersection environment test depth map and the traffic lights test sketch to generate a traffic lights test image. The fused image is distortion-free and can indicate real environmental information, providing sufficient accuracy and detail for the traffic lights module, thus solving the technical problem of image distortion generated by rendering methods in the prior art. The traffic lights test image generated after image fusion is similar to a camera-captured image, including not only the traffic lights area but also a background area that can display two-dimensional and three-dimensional information of the real intersection environment. Furthermore, the traffic lights area can display preset traffic lights states, providing more complete information and allowing testing of all functions of the traffic lights module, such as verifying the effectiveness of the traffic lights module in inferring traffic lights states. This solves the technical problem of incomplete image information generated by intermediate representation methods in the prior art. Furthermore, since the traffic light state in the traffic light test sketch can be any preset state, including extreme states that violate traffic regulations, it can simulate traffic light images under unconventional and dangerous scenarios, solving the technical problem that existing image acquisition methods cannot reach the safety boundaries of autonomous driving algorithms, and better meeting the needs of vehicle autonomous driving simulation testing.

[0149] According to embodiments of this disclosure, the fusion module includes a splicing submodule and an input / output submodule.

[0150] The stitching submodule is used to stitch the intersection environment test depth map and the traffic light test sketch in the channel dimension to generate a test stitched image.

[0151] The input / output submodule is used to input the test stitched image into the image generation model, so that the traffic light test image can be output through the image generation model.

[0152] According to embodiments of this disclosure, the first acquisition module includes a first acquisition submodule and a first generation submodule.

[0153] The first acquisition submodule is used to acquire point cloud test data of the intersection environment.

[0154] The first generation submodule is used to generate a test depth map of the intersection environment based on the intersection environment point cloud test data.

[0155] According to an embodiment of this disclosure, the intersection environmental point cloud test data includes three-dimensional coordinate values ​​of the environmental point cloud in a radar coordinate system, and the first generation submodule includes a determination unit and a generation unit.

[0156] The determining unit is used to determine the two-dimensional coordinates of the environmental point cloud in the pixel coordinate system and the depth value of the environmental point cloud based on the three-dimensional coordinates of the environmental point cloud in the radar coordinate system.

[0157] The generation unit is used to generate an intersection environment test depth map based on the two-dimensional coordinate values ​​and depth values ​​of the environmental point cloud.

[0158] According to embodiments of this disclosure, the determining unit includes a first conversion subunit, a second conversion subunit, and a determining subunit.

[0159] The first transformation subunit is used to convert the three-dimensional coordinate values ​​of the environmental point cloud in the radar coordinate system into the three-dimensional coordinate values ​​of the environmental point cloud in the camera coordinate system.

[0160] The second transformation subunit is used to project the three-dimensional coordinates of the environment point cloud in the camera coordinate system onto the camera plane to obtain the two-dimensional coordinates of the environment point cloud in the pixel coordinate system.

[0161] The sub-unit is determined based on the 3D coordinates of the environmental point cloud in the camera coordinate system to determine the depth value of the environmental point cloud.

[0162] According to embodiments of this disclosure, the generation unit includes a first annotation subunit and a second annotation subunit.

[0163] The first annotation subunit is used to annotate the environmental point cloud in the form of pixels in the first blank image based on the two-dimensional coordinate values ​​of the environmental point cloud.

[0164] The second annotation subunit is used to mark pixels at different coordinate positions in the first blank image with different colors according to the environmental point cloud depth value, so as to generate an intersection environmental test depth map.

[0165] According to an embodiment of this disclosure, the traffic light location test data includes the three-dimensional coordinate values ​​of the traffic lights in a predetermined map coordinate system. The first generation module includes a transformation unit, a drawing unit, and a filling unit.

[0166] The conversion unit is used to convert the three-dimensional coordinate values ​​of traffic lights in a predetermined map coordinate system into two-dimensional coordinate values ​​of traffic lights in a pixel coordinate system.

[0167] A drawing unit is used to draw the outline of a traffic light at the target location in the second blank image, wherein the coordinate values ​​of the target location are the same as the two-dimensional coordinate values ​​of the traffic light in the pixel coordinate system.

[0168] The fill unit is used to fill the outline of the traffic light with a color associated with the preset traffic light state to generate a traffic light test sketch.

[0169] Based on the above model training method, this disclosure also provides a model training apparatus. Figure 7 A schematic block diagram of a model training apparatus according to an embodiment of the present disclosure is shown.

[0170] like Figure 7 As shown, the model training device 700 of this embodiment includes a second acquisition module 701, a second generation module 702, a splicing module 703, a first input / output module 704, a calculation module 705, and a second input / output module 706.

[0171] The second acquisition module 701 is used to acquire a depth map of the intersection environment sample associated with the intersection where the autonomous vehicle is located.

[0172] The second generation module 702 is used to generate a traffic light sample sketch based on the traffic light location information and the training traffic light status at the intersection. The traffic light area in the intersection environment sample depth map corresponds to the location of the traffic light area in the traffic light sample sketch, and the traffic light area in the traffic light sample sketch is displayed as the training traffic light status.

[0173] The stitching module 703 is used to stitch the intersection environment sample depth map and the traffic light sample sketch in the channel dimension to generate a sample stitched image.

[0174] The first input / output module 704 is used to input the sample stitched image into the image generation model to be trained, so that the traffic light training image can be output by the image generation model to be trained.

[0175] The calculation module 705 is used to calculate the pixel difference between the traffic light training image and the intersection captured image;

[0176] The second input / output module 706 is used to input the traffic light training image and the intersection captured image into the discrimination model, so that the discrimination model outputs the probability value that the traffic light sample image and the intersection captured image are the same. When the pixel difference and the probability value meet the preset termination condition, the trained image generation model is obtained.

[0177] According to embodiments of this disclosure, traffic light training images and intersection captured images are input into a discrimination model via a second input / output module 706. The generated images and real images are then input into the discrimination model for classification to obtain an image authenticity judgment. This can improve the authenticity of the generated images, enhance the ability to generate details from high-resolution images, and improve the training effect of the image generation model.

[0178] According to embodiments of this disclosure, any and multiple modules among the first acquisition module 601, first generation module 602, fusion module 603, second acquisition module 701, second generation module 702, splicing module 703, first input / output module 704, calculation module 705, and second input / output module 706 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first acquisition module 601, first generation module 602, fusion module 603, second acquisition module 701, second generation module 702, splicing module 703, first input / output module 704, calculation module 705, and second input / output module 706 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 601, first generation module 602, fusion module 603, second acquisition module 701, second generation module 702, splicing module 703, first input / output module 704, calculation module 705, and second input / output module 706 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0179] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing an image generation method according to an embodiment of the present disclosure.

[0180] like Figure 8As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0181] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0182] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0183] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0184] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0185] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the image generation method provided in the embodiments of this disclosure.

[0186] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0187] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0188] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0189] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0190] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0191] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0192] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image generation method, comprising: obtaining an intersection environment test depth map associated with an intersection where an autonomous vehicle is located; generating a traffic light test sketch according to position information of a traffic light of the intersection and a preset traffic light state, wherein a traffic light region in the traffic light test sketch corresponds to a position of a traffic light region in the intersection environment test depth map, and the traffic light region in the traffic light test sketch is displayed as the preset traffic light state; performing image fusion on the intersection environment test depth map and the traffic light test sketch to generate a traffic light test image, so that driving simulation is performed according to the traffic light test image, wherein the traffic light test image can represent an intersection environment and the preset traffic light state, wherein according to the preset traffic light state, a traffic light contour is filled with a color associated with the preset traffic light state to generate the traffic light test sketch.

2. The method of claim 1, wherein, The image fusion on the intersection environment test depth map and the traffic light test sketch to generate a traffic light test image comprises: stitching the intersection environment test depth map and the traffic light test sketch in a channel dimension to generate a test stitched image; inputting the test stitched image into an image generation model, so that the traffic light test image is output by the image generation model.

3. The method of claim 1, wherein, The obtaining of the intersection environment test depth map comprises: obtaining intersection environment point cloud test data; generating the intersection environment test depth map according to the intersection environment point cloud test data.

4. The method of claim 3, wherein, The intersection environment point cloud test data comprises environment point cloud three-dimensional coordinate values in a radar coordinate system, and the generating of the intersection environment test depth map according to the intersection environment point cloud test data comprises: determining environment point cloud two-dimensional coordinate values and environment point cloud depth values in a pixel coordinate system according to the environment point cloud three-dimensional coordinate values in the radar coordinate system; generating the intersection environment test depth map according to the environment point cloud two-dimensional coordinate values and the environment point cloud depth values.

5. The method of claim 4, wherein, The determining of the environment point cloud two-dimensional coordinate values and the environment point cloud depth values in the pixel coordinate system according to the environment point cloud three-dimensional coordinate values in the radar coordinate system comprises: converting the environment point cloud three-dimensional coordinate values in the radar coordinate system into environment point cloud three-dimensional coordinate values in a camera coordinate system; projecting the environment point cloud three-dimensional coordinate values in the camera coordinate system onto a camera plane to obtain the environment point cloud two-dimensional coordinate values in the pixel coordinate system; determining the environment point cloud depth values according to the environment point cloud three-dimensional coordinate values in the camera coordinate system.

6. The method of claim 4, wherein, The generating of the intersection environment test depth map according to the environment point cloud two-dimensional coordinate values and the environment point cloud depth values comprises: annotating environment point cloud in the form of pixel points in a first blank picture according to the environment point cloud two-dimensional coordinate values; displaying pixel points at different coordinate positions in the first blank picture as different colors according to the environment point cloud depth values, to generate the intersection environment test depth map.

7. The method of claim 1, wherein, The traffic signal lamp position information includes a traffic signal lamp three-dimensional coordinate value in a predetermined map coordinate system, and the traffic signal lamp test sketch is generated according to the traffic signal lamp position information of the intersection and a preset traffic signal lamp state, including: Converting the traffic signal lamp three-dimensional coordinate value in the predetermined map coordinate system into a traffic signal lamp two-dimensional coordinate value in a pixel coordinate system; Drawing a traffic signal lamp contour at a target position in the second blank picture, wherein the coordinate value of the target position is the same as the traffic signal lamp two-dimensional coordinate value in the pixel coordinate system; According to the preset traffic signal lamp state, the traffic signal lamp contour is filled with a color associated with the preset traffic signal lamp state to generate the traffic signal lamp test sketch.

8. A model training method, comprising: obtaining an intersection environment sample depth map associated with an intersection where an autonomous vehicle is located; generating a traffic signal lamp sample sketch containing a training traffic signal lamp state according to traffic signal lamp position information of the intersection and the training traffic signal lamp state, wherein a traffic signal lamp region in the traffic signal lamp sample sketch corresponds to a position of a traffic signal lamp region in the intersection environment sample depth map, and the traffic signal lamp region in the traffic signal lamp sample sketch is displayed as the training traffic signal lamp state, wherein a corresponding color represents a corresponding training traffic signal lamp state; splicing the intersection environment sample depth map and the traffic signal lamp sample sketch in a channel dimension to generate a sample spliced image; inputting the sample spliced image into a to-be-trained image generation model, so as to output a traffic signal lamp training image containing the training traffic signal lamp state through the to-be-trained image generation model; calculating a pixel difference value of the traffic signal lamp training image and a road intersection captured image; inputting the traffic signal lamp training image and the road intersection captured image into a discriminant model to output a probability value that the traffic signal lamp sample image is the same as the road intersection captured image through the discriminant model, and obtaining a trained image generation model when the pixel difference value and the probability value satisfy a preset termination condition.

9. An image generation device, comprising: a first obtaining module configured to obtain an intersection environment test depth map associated with an intersection where an autonomous vehicle is located; a first generating module configured to generate a traffic signal lamp test sketch according to traffic signal lamp position information of the intersection and a preset traffic signal lamp state, wherein a traffic signal lamp region in the traffic signal lamp test sketch corresponds to a position of a traffic signal lamp region in the intersection environment test depth map, and the traffic signal lamp region in the traffic signal lamp test sketch is displayed as the preset traffic signal lamp state; a fusion module configured to perform image fusion on the intersection environment test depth map and the traffic signal lamp test sketch to generate a traffic signal lamp test image, so as to perform driving simulation according to the traffic signal lamp test image, wherein the traffic signal lamp test image can represent an intersection environment and the preset traffic signal lamp state. The first generation module comprises a filling unit configured to fill a traffic signal lamp contour with a color associated with the preset traffic signal lamp state to generate the traffic signal lamp test sketch according to the preset traffic signal lamp state.

10. A model training apparatus, comprising: a second acquisition module configured to acquire an intersection environment sample depth map associated with an intersection where an autonomous vehicle is located; a second generation module configured to generate a traffic signal lamp sample sketch containing a training traffic signal lamp state according to traffic signal lamp position information of the intersection and the training traffic signal lamp state, wherein a traffic signal lamp region in the traffic signal lamp sample sketch corresponds to a position of a traffic signal lamp region in the intersection environment sample depth map, and the traffic signal lamp region in the traffic signal lamp sample sketch is displayed as the training traffic signal lamp state, wherein a corresponding training traffic signal lamp state is represented by a corresponding color; a splicing module configured to splice the intersection environment sample depth map and the traffic signal lamp sample sketch in a channel dimension to generate a sample spliced image; a first input / output module configured to input the sample spliced image into a to-be-trained image generation model so as to output a traffic signal lamp training image containing the training traffic signal lamp state by the to-be-trained image generation model; a calculation module configured to calculate a pixel difference value of the traffic signal lamp training image and an intersection photographed image; a second input / output module configured to input the traffic signal lamp training image and the intersection photographed image into a discrimination model so as to output a probability value that the traffic signal lamp sample image is the same as the intersection photographed image by the discrimination model, and obtain a trained image generation model when the pixel difference value and the probability value satisfy a preset termination condition.

11. An electronic device, comprising: one or more processors; a storage device configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-8.

12. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-8.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images