Virtual camera image generation method and device, equipment and medium

Abstract

The invention relates to the technical field of image processing, in particular to a virtual camera image generation method and device, equipment and a medium. According to the invention, a target point cloud detected by a laser radar is projected to an original coordinate system of an image acquisition device to obtain a sparse depth map; meanwhile, a depth estimation map is obtained based on estimation of the depth of the original image. And fusing the depth estimation map and the sparse depth map to fill the point cloud of the point cloud sparse region on the sparse depth map so as to obtain a dense depth map. The method comprises the following steps: firstly, fusing an original image, then directly projecting the original image to a virtual coordinate system by using a dense depth map obtained by fusion, completing conversion from the original coordinate system to the virtual coordinate system so as to obtain an initial virtual image with relatively high geometric fidelity, and finally, complementing holes in the initial virtual image by using a diffusion model so as to obtain a virtual image without holes. Therefore, the quality of the virtual image is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method, apparatus, device, and medium for generating virtual camera images. Background Technology

[0002] Vehicles, robots, drones, and other intelligent mobile devices are all equipped with perception models. These models analyze raw images captured by cameras on the intelligent mobile device to obtain environmental information about its surroundings, enabling the control system to move the device accordingly. However, for unproduced intelligent mobile devices, it is difficult to collect enough images to train the perception model. Therefore, a large number of raw images are needed to train the model. Furthermore, the virtual coordinate system on unproduced intelligent mobile devices differs from that on mass-produced intelligent mobile devices. Therefore, the raw images need to be converted to a virtual coordinate system to obtain a virtual image. Existing technologies directly convert raw images into virtual images based on the intrinsic and extrinsic parameters of two cameras. However, because the raw images may contain occlusions and depth discontinuities, these areas appear as holes in the virtual image, resulting in voids in the generated virtual image and thus reducing its quality.

[0003] In summary, the virtual images generated by existing technologies are of low quality.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method, apparatus, device, and medium for generating virtual camera images, which solves the problem of low-quality virtual images generated by existing technologies.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for generating virtual camera images, comprising:

[0008] The original image of the target scene captured by the image acquisition device is acquired, the target point cloud of the target scene detected by the lidar is acquired, and the target point cloud is projected onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

[0009] Estimate the depth of the original image to obtain a depth estimation map, and fuse the depth estimation map and the sparse depth map to obtain a dense depth map of the target scene;

[0010] Based on the dense depth map, the original image is projected onto the virtual coordinate system to obtain an initial virtual image;

[0011] A diffusion model is applied to the initial virtual image to fill in the holes in the initial virtual image, thereby obtaining the target virtual image of the target scene in the virtual coordinate system.

[0012] In one implementation, projecting the target point cloud onto the original coordinate system of the image acquisition device to obtain a sparse depth map includes:

[0013] Obtain the intrinsic and extrinsic parameters of the image acquisition device, and obtain the projection function of the radar coordinate system of the lidar relative to the original coordinate system;

[0014] Based on the intrinsic and extrinsic parameters and the projection function, the target point cloud is projected onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

[0015] In one implementation, the depth estimation map and the sparse depth map are fused to obtain a dense depth map of the target scene, including:

[0016] Identify low-density regions on the sparse depth map where the point cloud density is below a threshold.

[0017] The point cloud depth of the low-density region is interpolated to obtain the interpolation result.

[0018] Based on the interpolation results of the low-density region on the sparse depth map and the depth estimation of the low-density region on the depth estimation map, the fusion depth of the low-density region is obtained;

[0019] The point cloud depth of the low-density region on the sparse depth map is replaced by the fusion depth to obtain the dense depth map of the target scene.

[0020] In one implementation, based on the dense depth map, the original image is projected onto a virtual coordinate system to obtain an initial virtual image, including:

[0021] Based on the dense depth map, the pixels on the original image are projected onto the real-world coordinate system to obtain the real coordinates of the pixels in the real coordinate system;

[0022] The real coordinates of the pixel are projected onto the virtual coordinate system to obtain the virtual coordinates of the pixel in the virtual coordinate system.

[0023] The pixel values ​​of the pixels in the original image are assigned to the virtual coordinates in the virtual coordinate system to obtain the initial virtual image.

[0024] In one implementation, projecting the real coordinates of a pixel onto the virtual coordinate system to obtain the virtual coordinates of the pixel in the virtual coordinate system includes:

[0025] The intrinsic and extrinsic parameters of the virtual image acquisition device in which the virtual coordinate system is located are obtained, as well as the projection function of the virtual coordinate system relative to the real world coordinate system. Based on the projection function and the intrinsic and extrinsic parameters of the virtual image acquisition device, the real coordinates of the pixel are projected onto the virtual coordinate system to obtain the virtual coordinates of the pixel in the virtual coordinate system.

[0026] In one implementation, the diffusion model is a stable diffusion model, which is a trained model, and the training method includes:

[0027] Obtain a sample image, generate holes in the sample image, and obtain a hole image;

[0028] The hole image is input into the stable diffusion model to complete the image on the hole image through the stable diffusion model, thereby obtaining a hole-completed image;

[0029] A loss function is constructed based on the hole-filled image and the sample image, and the parameters of the stable diffusion model are adjusted according to the loss function to complete the training of the stable diffusion model.

[0030] In one implementation, the image acquisition device and the lidar are located on the same operational vehicle, and the target virtual image is used to train a perception model on an unoperated vehicle.

[0031] Secondly, embodiments of the present invention also provide a virtual camera image generation apparatus, wherein the apparatus comprises the following components:

[0032] The sparse depth map generation module is used to acquire the original image of the target scene captured by the image acquisition device, acquire the target point cloud of the target scene detected by the lidar, and project the target point cloud onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

[0033] A fusion module is used to estimate the depth of the original image to obtain a depth estimation map, and fuse the depth estimation map and the sparse depth map to obtain a dense depth map of the target scene;

[0034] A virtual image generation module is used to project the original image onto a virtual coordinate system based on the dense depth map to obtain an initial virtual image;

[0035] The hole completion module is used to apply a diffusion model to the initial virtual image to complete the holes in the initial virtual image, thereby obtaining the target virtual image of the target scene in the virtual coordinate system.

[0036] Thirdly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a virtual camera image generation program stored in the memory and executable on the processor, wherein when the processor executes the virtual camera image generation program, it implements the steps of the virtual camera image generation method described above.

[0037] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a virtual camera image generation program, wherein when the virtual camera image generation program is executed by a processor, it implements the steps of the virtual camera image generation method described above.

[0038] Beneficial Effects: This invention projects the target point cloud detected by lidar onto the original coordinate system of the image acquisition device to obtain a sparse depth map. Simultaneously, based on the estimated depth of the original image, a depth estimation map is obtained. The depth estimation map and the sparse depth map are fused to fill in the sparse regions of the point cloud on the sparse depth map, resulting in a dense depth map. Then, the original image is directly projected onto a virtual coordinate system using the fused dense depth map, completing the transformation from the original coordinate system to the virtual coordinate system to obtain an initial virtual image with high geometric fidelity. Finally, a diffusion model is used to fill in the holes in the initial virtual image to obtain a hole-free virtual image, thereby improving the quality of the virtual image. Attached Figure Description

[0039] Figure 1 This is an overall flowchart of the present invention;

[0040] Figure 2 This is a flowchart illustrating the generation of high-quality virtual camera images and the training of a diffusion model in an embodiment of the present invention.

[0041] Figure 3 This is a flowchart illustrating the training process of the diffusion model in this embodiment of the invention.

[0042] Figure 4 A structural diagram of the virtual camera image generation device provided by the present invention;

[0043] Figure 5 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0045] Research has revealed that intelligent mobile devices such as vehicles, robots, and drones are equipped with perception models. These models analyze raw images captured by cameras on the intelligent mobile device to obtain environmental information about its surroundings, enabling the control system to move the device accordingly. However, for unproduced intelligent mobile devices, it is difficult to collect enough images to train the perception model. Therefore, a large number of raw images are needed to train the perception model. Furthermore, the virtual coordinate system on unproduced intelligent mobile devices differs from that on mass-produced intelligent mobile devices. Therefore, the raw images need to be converted to a virtual coordinate system to obtain virtual images. Existing technologies directly convert raw images into virtual images based on the intrinsic and extrinsic parameters of two cameras. However, due to potential occlusions and depth discontinuities in the raw images, these areas appear as holes in the virtual image, resulting in voids in the generated virtual images and thus reducing their quality.

[0046] To address the aforementioned technical problems, this invention provides a method, apparatus, device, and medium for generating virtual camera images, which solves the problem of low-quality virtual images generated by existing technologies.

[0047] The virtual camera image generation method of this embodiment can be applied to a terminal device, which can be a terminal product with image processing capabilities, such as a computer. In this embodiment, as... Figure 1 As shown, the virtual camera image generation method specifically includes the following steps:

[0048] S100: Acquire the original image of the target scene captured by the image acquisition device, acquire the target point cloud of the target scene detected by the lidar, and project the target point cloud onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

[0049] S200, estimate the depth of the original image to obtain a depth estimation map, and fuse the depth estimation map and the sparse depth map to obtain a dense depth map of the target scene;

[0050] S300, Based on the dense depth map, the original image is projected onto the virtual coordinate system to obtain an initial virtual image;

[0051] S400, apply a diffusion model to the initial virtual image to fill in the holes in the initial virtual image, and obtain the target virtual image of the target scene in the virtual coordinate system.

[0052] Combination Figure 2 Detailed explanation of steps S100, S200, S300, and S400. Figure 2 The original camera image in step S100 is the original image. Figure 2 The lidar point cloud mentioned is the target point cloud in step S100. Figure 2 In step S100, the sparse depth map is obtained by projecting the target point cloud onto the original coordinate system of the image acquisition device from the original camera image and the lidar point cloud. Figure 2 The depth estimation map is obtained from the monocular depth estimation and the original camera image, i.e., the depth of the original image is estimated in step S200 to obtain the depth estimation map; Figure 2 The process of applying multi-source depth adaptive fusion to the depth estimation map and the sparse depth map in step S200, i.e., fusing the depth estimation map and the sparse depth map, is... Figure 2 The result of the multi-source depth adaptive fusion is the dense depth map in step S200. Figure 2 In step S300, the result obtained from the multi-source depth adaptive fusion is used for virtual camera viewpoint transformation, i.e., based on the dense depth map, the original image is projected onto the virtual coordinate system. Figure 2 The virtual camera perspective transformation in step S300 yields the initial virtual image. Figure 2 The hole-filling process, which involves applying a diffusion model to the initial virtual image in step S400, is performed on the result obtained from the virtual camera perspective transformation to fill in the holes in the initial virtual image. Figure 2 The high-quality virtual camera image in step S400 is the target virtual image.

[0053] The above diffusion model is the diffusion model after training, such as... Figure 2 As shown, when training the diffusion model, the original camera images are used as training samples to train the diffusion model.

[0054] The virtual camera image generation method based on steps S100, S200, S300, and S400 can be used to generate virtual images needed to train a perception model on a vehicle not yet in production. Since the vehicle is not driving in a real-world scenario, it's impossible to use its camera to capture scene images and train the perception model. Therefore, it's necessary to project the original image captured by the camera on a vehicle already in production (i.e., the real camera) onto the coordinate system of the camera on the vehicle not yet in production (i.e., the virtual camera) to obtain a virtual image. Finally, this virtual image is used to train the perception model on the vehicle not yet in production. The specific application process is as follows:

[0055] The process involves acquiring raw images from cameras on production vehicles of different models, as well as point clouds detected by LiDAR on those vehicles. A depth map (i.e., a depth estimation map) is estimated for the raw images captured by the cameras on each vehicle model. The point clouds detected by the LiDAR on each vehicle model are then projected onto the camera's original coordinate system to obtain a sparse depth map. The depth estimation map and the sparse depth map for each vehicle model are then fused to obtain a dense depth map. The raw images are then projected onto a virtual coordinate system based on the dense depth map to obtain an initial virtual image. Finally, holes are filled into the initial virtual image to obtain the target virtual image. A perception model for non-production vehicles is trained using the target virtual images corresponding to each vehicle model.

[0056] Step S100, which projects the target point cloud onto the original coordinate system of the image acquisition device to obtain a sparse depth map, includes the following specific steps: obtaining the intrinsic and extrinsic parameters of the image acquisition device, and obtaining the projection function of the lidar's radar coordinate system relative to the original coordinate system; based on the intrinsic and extrinsic parameters and the projection function, projecting the target point cloud onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

[0057] In this process, the target point cloud is projected onto the original coordinate system of the image acquisition device to obtain a sparse depth map. That is, each point in the target point cloud (represented by lidar, which has three-dimensional coordinates) is projected onto the original coordinate system to obtain the projection point of the point in the original coordinate system (represented by a pixel (u,v)). All the projection points and their depth values ​​constitute the sparse depth map.

[0058] Use D lidar (u,v) represents the depth value of the pixel (u,v) in the original coordinate system:

[0059]

[0060] In the formula, P lidarπ represents the depth value corresponding to a point in the radar coordinate system. π represents the projection function of the radar coordinate system relative to the original coordinate system. The original coordinate system is a two-dimensional coordinate system, while the radar coordinate system is a three-dimensional coordinate system. The role of π is to project the three-dimensional coordinates P... lidar Converted to two-dimensional D lidar (u,v). R c The rotation matrix t represents the image acquisition device (i.e., the camera on a vehicle already in production). c The translation and rotation matrices represent the image acquisition device's translation and rotation matrices, which are extrinsic parameters of the device. Since extrinsic parameters are coordinate system dependent, transforming the point cloud in the radar coordinate system to the original coordinate system of the image acquisition device requires considering the transformation of extrinsic parameters. c This represents the internal parameters of the image acquisition device.

[0061] Because some objects in the target scene have very low reflectivity to the light waves emitted by the lidar, meaning some objects cannot reflect the light waves at all, their depth information cannot be matched on the sparse depth map. If the object is far from the lidar, its corresponding point cloud will be too sparse to be used to project the original image onto the virtual coordinate system. Therefore, it is necessary to estimate the depth of the original image and fuse the estimated depth with the sparse depth map to obtain the dense depth map in step S200. Step S200, fusing the estimated depth map and the sparse depth map to obtain the dense depth map of the target scene, includes the following specific steps S201, S202, S203, and S204:

[0062] S201, identify low-density regions on the sparse depth map where the point cloud density is below a threshold.

[0063] S202, interpolate the point cloud depth of the low-density region to obtain the interpolation result.

[0064] The sparse depth map is divided into several regions, and the point cloud density of each region is calculated. If the point cloud density of a region is below a threshold, the region is considered a low-density region. The point cloud depth in the low-density regions is interpolated to obtain the interpolated result for the low-density regions, and D is used as the interpolation point depth. mono (u,v) represents the interpolated depth value of pixel (u,v) in the low-density region.

[0065] S203, based on the interpolation result of the low-density region on the sparse depth map and the depth estimate of the low-density region on the depth estimation map, the fusion depth of the low-density region is obtained.

[0066] S204, replace the point cloud depth of the low-density region on the sparse depth map with the fusion depth to obtain the dense depth map of the target scene.

[0067] Steps S203 and S204 are based on the following formula to obtain the fusion depth value of the point (u,v) on the original coordinate system. The fusion depth values ​​of all points constitute the dense depth map.

[0068]

[0069] When (u,v) is located in a non-low-density region, use the depth value D of (u,v) on the sparse depth map. lidar (u,v) represents the depth value of (u,v) on the dense depth map; when (u,v) is located in a low-density region, D interp (u,v) represents the depth value of (u,v) on the depth estimation map, and D interp (u,v) and D mono By weighting (u,v), we obtain the depth-weighted value of (u,v), which is then used as the depth value of (u,v) on the dense depth map. α in the formula represents the weighting coefficient.

[0070] The above approach directly uses the depth information provided by Lidar in areas with dense Lidar point coverage, and in areas with sparse Lidar points, it fuses the Lidar neighbor point interpolation results and monocular depth prediction results according to a confidence-weighted strategy, which can prevent the blurring of object edges.

[0071] Step S300 includes the following specific steps S301 and S302:

[0072] S301, Based on the dense depth map, project the pixels on the original image onto the real-world coordinate system to obtain the real coordinates of the pixels in the real coordinate system.

[0073]

[0074] P w This represents the coordinates of pixel (u,v) in the real-world coordinate system. Formula (3) means that the coordinates of pixel (u,v) in the dense depth map are... lidar The expression (u,v) projects the pixel (u,v) onto the real coordinate system, which means finding the point corresponding to pixel (u,v) in the real coordinate system. By using the same method to project all pixels in the original image onto the real coordinate system, we obtain the true coordinates of all pixels in the real coordinate system.

[0075] S302, obtain the intrinsic and extrinsic parameters of the virtual image acquisition device where the virtual coordinate system is located, and obtain the projection function of the virtual coordinate system relative to the real world coordinate system. Based on the projection function and the intrinsic and extrinsic parameters of the virtual image acquisition device, project the real coordinates of the pixel point onto the virtual coordinate system to obtain the virtual coordinates of the pixel point on the virtual coordinate system.

[0076]

[0077] P w p represents the real-world coordinates of a pixel (u,v) in the original image. v R represents the virtual coordinates of pixel (u,v) in the virtual coordinate system. v The rotation matrix t represents the virtual image acquisition device (i.e., the virtual camera on an unproduced vehicle). v The translation matrix represents the virtual image acquisition device. Both the translation and rotation matrices are extrinsic parameters of the virtual image acquisition device. K v This represents the intrinsic parameters of the virtual image acquisition device. π′ represents the projection function of the real coordinate system relative to the virtual coordinate system, and P... w The coordinates are in a three-dimensional coordinate system, while the virtual coordinate system is a two-dimensional coordinate system. The function of π′ is to convert the two-dimensional P... w Convert to three-dimensional p v .

[0078] S302, assign the pixel value of the pixel point on the original image to the virtual coordinates on the virtual coordinate system to obtain the initial virtual image.

[0079] Assign the pixel values ​​of (u,v) on the original image to p in the virtual coordinate system. v That is, to set the coordinates p in the virtual coordinate system v The pixel values ​​of the points are set to the pixel values ​​of (u,v) on the original image. Using the same method, the pixel values ​​of all pixels on the virtual coordinate system are set to obtain the initial virtual image.

[0080] This embodiment directly utilizes the real and dense depth information provided by multi-source depth fusion technology to complete the three-dimensional spatial perspective transformation, and finally obtains a high-quality geometrically accurate initial virtual image.

[0081] The diffusion model in step S400 is a stable diffusion model, also known as a stable diffusion model. The training method for the stable diffusion model includes the following specific steps: S401, S402, and S403:

[0082] S401, acquire a sample image, generate holes in the sample image, and obtain a hole image.

[0083] like Figure 3 As shown, holes are randomly generated on the sample image (i.e., Figure 3 Random simulated holes in the model are used to obtain hole images, which are then used as ground values ​​for training a stable diffusion model (the ground value is...). Figure 3 (The holed ground truth image in the sample image), where the hole is the pixel value on the sample image set to 0 or 255.

[0084] S402, the hole image is input into the stable diffusion model to complete the image on the hole image through the stable diffusion model, so as to obtain the hole-completed image.

[0085] The hole image is input into the stable diffusion model, which fills in the hole in the image, that is, restores the holes in the hole image to the original pixel values ​​of the sample image as much as possible.

[0086] S403, construct a loss function based on the hole-filled image and the sample image, and adjust the parameters of the stable diffusion model according to the loss function to complete the training of the stable diffusion model.

[0087] That is, a loss function is constructed based on the difference between the hole-filled image and the sample image, and then fine-tuned based on this loss function (fine-tuning is... Figure 3 The parameters of the Finetune diffusion model are fine-tuned, and the diffusion model is further trained based on ControlNet. ControlNet is a neural network architecture for controlling diffusion models, designed to improve the controllability and accuracy of the hole-filled images generated by the diffusion model by adding additional control conditions, including edge and depth parameters.

[0088] This embodiment also provides a virtual camera image generation device, such as... Figure 4 As shown, the device comprises the following components:

[0089] The sparse depth map generation module 01 is used to acquire the original image of the target scene captured by the image acquisition device, acquire the target point cloud of the target scene detected by the lidar, and project the target point cloud onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

[0090] The fusion module 02 is used to estimate the depth of the original image, obtain a depth estimation map, and fuse the depth estimation map and the sparse depth map to obtain a dense depth map of the target scene;

[0091] The virtual image generation module 03 is used to project the original image onto a virtual coordinate system based on the dense depth map to obtain an initial virtual image;

[0092] Hole completion module 04 is used to apply a diffusion model to the initial virtual image to complete the holes on the initial virtual image, thereby obtaining the target virtual image of the target scene in the virtual coordinate system.

[0093] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 5 As shown, the terminal device includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a virtual camera image generation method. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0094] Those skilled in the art will understand that Figure 5 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0095] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a virtual camera image generation program stored in the memory and executable on the processor. When the processor executes the virtual camera image generation program, it implements the following operation instructions:

[0096] The original image of the target scene captured by the image acquisition device is acquired, the target point cloud of the target scene detected by the lidar is acquired, and the target point cloud is projected onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

[0097] Estimate the depth of the original image to obtain a depth estimation map, and fuse the depth estimation map and the sparse depth map to obtain a dense depth map of the target scene;

[0098] Based on the dense depth map, the original image is projected onto the virtual coordinate system to obtain an initial virtual image;

[0099] A diffusion model is applied to the initial virtual image to fill in the holes in the initial virtual image, thereby obtaining the target virtual image of the target scene in the virtual coordinate system.

[0100] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of virtual camera image generation, the method comprising: include: The original image of the target scene captured by the image acquisition device is acquired, the target point cloud of the target scene detected by the lidar is acquired, and the target point cloud is projected onto the original coordinate system of the image acquisition device to obtain a sparse depth map. Estimate the depth of the original image to obtain a depth estimation map, and fuse the depth estimation map and the sparse depth map to obtain a dense depth map of the target scene; Based on the dense depth map, the original image is projected onto the virtual coordinate system to obtain an initial virtual image; A diffusion model is applied to the initial virtual image to fill in the holes in the initial virtual image, thereby obtaining the target virtual image of the target scene in the virtual coordinate system.

2. The virtual camera image generation method of claim 1, wherein, Projecting the target point cloud onto the original coordinate system of the image acquisition device yields a sparse depth map, including: Obtain the intrinsic and extrinsic parameters of the image acquisition device, and obtain the projection function of the radar coordinate system of the lidar relative to the original coordinate system; Based on the intrinsic and extrinsic parameters and the projection function, the target point cloud is projected onto the original coordinate system of the image acquisition device to obtain a sparse depth map.

3. The virtual camera image generation method of claim 1, wherein, By fusing the depth estimation map and the sparse depth map, a dense depth map of the target scene is obtained, including: Identify low-density regions on the sparse depth map where the point cloud density is below a threshold. The point cloud depth of the low-density region is interpolated to obtain the interpolation result. Based on the interpolation results of the low-density region on the sparse depth map and the depth estimation of the low-density region on the depth estimation map, the fusion depth of the low-density region is obtained; The point cloud depth of the low-density region on the sparse depth map is replaced by the fusion depth to obtain the dense depth map of the target scene.

4. The virtual camera image generation method of claim 1, wherein, Based on the dense depth map, the original image is projected onto a virtual coordinate system to obtain an initial virtual image, including: Based on the dense depth map, the pixels on the original image are projected onto the real-world coordinate system to obtain the real coordinates of the pixels in the real coordinate system; The real coordinates of the pixel are projected onto the virtual coordinate system to obtain the virtual coordinates of the pixel in the virtual coordinate system. The pixel values ​​of the pixels in the original image are assigned to the virtual coordinates in the virtual coordinate system to obtain the initial virtual image.

5. The virtual camera image generation method of claim 4, wherein, Projecting the real coordinates of a pixel onto the virtual coordinate system to obtain the virtual coordinates of the pixel in the virtual coordinate system includes: The intrinsic and extrinsic parameters of the virtual image acquisition device in which the virtual coordinate system is located are obtained, as well as the projection function of the virtual coordinate system relative to the real world coordinate system. Based on the projection function and the intrinsic and extrinsic parameters of the virtual image acquisition device, the real coordinates of the pixel are projected onto the virtual coordinate system to obtain the virtual coordinates of the pixel in the virtual coordinate system.

6. The virtual camera image generation method of any one of claims 1-5, wherein, The diffusion model is a stable diffusion model, which is a trained model. The training method includes: Obtain a sample image, generate holes in the sample image, and obtain a hole image; The hole image is input into the stable diffusion model to complete the image on the hole image through the stable diffusion model, thereby obtaining a hole-completed image; A loss function is constructed based on the hole-filled image and the sample image, and the parameters of the stable diffusion model are adjusted according to the loss function to complete the training of the stable diffusion model.

7. The virtual camera image generation method of any one of claims 1-5, wherein, The image acquisition device and the lidar are located on the same vehicle already in production, and the target virtual image is used to train the perception model on vehicles not yet in production.

8. A virtual camera image generation device, characterized in that, The device comprises the following components: The sparse depth map generation module is used to acquire the original image of the target scene captured by the image acquisition device, acquire the target point cloud of the target scene detected by the lidar, and project the target point cloud onto the original coordinate system of the image acquisition device to obtain a sparse depth map. A fusion module is used to estimate the depth of the original image to obtain a depth estimation map, and fuse the depth estimation map and the sparse depth map to obtain a dense depth map of the target scene; A virtual image generation module is used to project the original image onto a virtual coordinate system based on the dense depth map to obtain an initial virtual image; The hole completion module is used to apply a diffusion model to the initial virtual image to complete the holes in the initial virtual image, thereby obtaining the target virtual image of the target scene in the virtual coordinate system.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a virtual camera image generation program stored in the memory and executable on the processor. When the processor executes the virtual camera image generation program, it implements the steps of the virtual camera image generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a virtual camera image generation program, which, when executed by a processor, implements the steps of the virtual camera image generation method as described in any one of claims 1-7.

Images