Occupancy training data generation apparatus and occupancy training data generation method

Abstract

An occupancy training data generation device and an occupancy training data generation method are disclosed. The occupancy training data generation device comprises one or more processors configured to: obtain camera parameters corresponding to a virtual camera, a visual image, and a depth ground truth image; determine a field of view space corresponding to the virtual camera and point cloud data corresponding to a simulation scene in the field of view space based on the camera parameters and the depth ground truth image; determine a voxel state corresponding to a first voxel in the field of view space based on the point cloud data; determine a voxel state corresponding to a second voxel based on light rays passing through voxels corresponding to the first voxel and the second voxel; and generate occupancy training data corresponding to the field of view space based on the visual image, the voxel state corresponding to the first voxel, and the voxel state corresponding to the second voxel. The disclosed method can generate occupancy training data that meets the training requirements of an occupancy network.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to an apparatus for generating training data and a method for generating training data. Background Technology

[0002] In fields such as intelligent driving and intelligent robots, occupancy networks (OCC) have become one of the key technologies for intelligent agents to perceive the three-dimensional environment because they can explicitly model the voxel state of each location in three-dimensional space in the form of voxels.

[0003] The training performance of Occupied Class Control (OCC) models heavily relies on large-scale, high-quality occupancy training data. Ideal occupancy training data needs to accurately label the three occupancy states of each voxel in 3D space: "Occupied" (actually occupied by an object), "Free" (without any object), and "Unknown" (unobservable due to object occlusion). Therefore, a method is urgently needed to generate occupancy training data that meets the training requirements of occupancy networks. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides an occupation training data generation apparatus and an occupation training data generation method to generate occupation training data that meets the training requirements of an occupation network.

[0005] A first aspect of this disclosure provides a training data generation apparatus, including one or more processors, the one or more processors being configured to: Obtain the camera parameters, visual image, and depth ground truth image corresponding to the virtual camera; Based on the camera parameters and the ground truth depth image, determine the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space; Based on the point cloud data, the voxel state corresponding to the first voxel in the field of view is determined; wherein, the first voxel is a voxel that contains at least a preset number of point cloud points in the point cloud data. Based on the ray path voxels corresponding to the first voxel and the second voxel, the voxel state corresponding to the second voxel is determined; wherein, the second voxel is other voxels in the field of view space besides the first voxel, and the ray path voxels corresponding to the second voxel include the voxels through which the ray path from the virtual camera to the second voxel passes; Based on the visual image, the voxel state corresponding to the first voxel, and the voxel state corresponding to the second voxel, occupancy training data corresponding to the field of view space is generated.

[0006] A second aspect of this disclosure provides a method for generating training data, comprising: Obtain the camera parameters, visual image, and depth ground truth image corresponding to the virtual camera; Based on the camera parameters and the ground truth depth image, determine the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space; Based on the point cloud data, the voxel state corresponding to the first voxel in the field of view is determined; wherein, the first voxel is a voxel that contains at least a preset number of point cloud points in the point cloud data. Based on the ray path voxels corresponding to the first voxel and the second voxel, the voxel state corresponding to the second voxel is determined; wherein, the second voxel is other voxels in the field of view space besides the first voxel, and the ray path voxels corresponding to the second voxel include the voxels through which the ray path from the virtual camera to the second voxel passes; Based on the visual image, the voxel state corresponding to the first voxel, and the voxel state corresponding to the second voxel, occupancy training data corresponding to the field of view space is generated.

[0007] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program that is executed by a processor to perform the occupation training data generation method provided in the second aspect of the present disclosure.

[0008] A fourth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the occupation training data generation method provided in the second aspect of the disclosure.

[0009] A fifth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs the occupation training data generation method provided in a second aspect of this disclosure.

[0010] This disclosure provides an occupancy training data generation apparatus and a occupancy training data generation method. The occupancy training data generation apparatus includes one or more processors configured to: acquire camera parameters, a visual image, and a depth ground truth image corresponding to a virtual camera; determine, based on the camera parameters and the depth ground truth image, the field of view space corresponding to the virtual camera and point cloud data corresponding to the simulated scene within the field of view space; determine, based on the point cloud data, the voxel state corresponding to a first voxel in the field of view space; wherein the first voxel is a voxel containing at least a preset number of point cloud points from the point cloud data; determine the voxel state corresponding to a second voxel based on the ray path voxels corresponding to the first voxel and the second voxel; wherein the second voxel is any voxel in the field of view other than the first voxel, and the ray path voxels corresponding to the second voxel include the voxels traversed by the ray path from the virtual camera to the second voxel; and generate occupancy training data corresponding to the field of view space based on the visual image, the voxel state corresponding to the first voxel, and the voxel state corresponding to the second voxel. The processor determines whether the second voxel is occluded by the first voxel by judging whether the ray path of the second voxel in the field of view contains the first voxel, and thus determines whether the voxel state corresponding to the second voxel is free or unknown. In this way, the processor generates occupancy training data that meets the training requirements of the occupancy network.

[0011] Furthermore, in this embodiment, the processor can not only label the voxel states of the first voxels (i.e., voxels occupied by entities) containing at least a preset number of point cloud points in the virtual camera's field of view as occupied states, but also, based on the visual occlusion relationships between voxels, label the voxel states of all second voxels (i.e., other voxels in the field of view besides the first voxels) in the virtual camera's field of view as free states or unknown states, thereby generating occupied training data containing three voxel states: occupied, free, and unknown. That is, the generation process of this occupied training data considers not only whether voxels are occupied by entities, but also the visual occlusion relationships between voxels, improving the accuracy of the occupied training data. Correspondingly, training the occupied network based on this occupied training data can also improve the training effect of the occupied network. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the structure of a training data generation apparatus provided in an exemplary embodiment of this disclosure.

[0013] Figure 2 This is a schematic diagram of the voxel state of a second voxel provided in an exemplary embodiment of this disclosure.

[0014] Figure 3 This is a schematic diagram of the voxel state of a second voxel provided in another exemplary embodiment of this disclosure.

[0015] Figure 4This is a flowchart illustrating an exemplary embodiment of the present disclosure of a method for generating training data.

[0016] Figure 5 This is a flowchart illustrating a method for generating training data in accordance with another exemplary embodiment of this disclosure.

[0017] Figure 6 This is a flowchart illustrating a method for generating training data in accordance with another exemplary embodiment of this disclosure.

[0018] Figure 7 This is a flowchart illustrating a method for generating training data in accordance with another exemplary embodiment of this disclosure.

[0019] Figure 8 This is a flowchart illustrating a method for generating training data in accordance with another exemplary embodiment of this disclosure.

[0020] Figure 9 This is a flowchart illustrating a method for generating training data in accordance with another exemplary embodiment of this disclosure.

[0021] Figure 10 This is a flowchart illustrating a method for generating training data in accordance with another exemplary embodiment of this disclosure.

[0022] Figure 11 This is a schematic diagram of the structure of a training data generation apparatus provided in an exemplary embodiment of this disclosure.

[0023] Figure 12 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0024] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0025] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0026] Application Overview The intelligent driving described in this disclosure can encompass multiple fields such as autonomous driving and assisted driving, and robotic systems can also utilize the intelligent driving described in this disclosure. Autonomous driving technology aims to achieve fully autonomous driving of vehicles in various complex road conditions without human intervention; it is also known as driverless driving and represents an advanced form of intelligent driving. Assisted driving, through a series of sensors and algorithms, provides drivers with real-time road condition information, warnings, and some driving operation support, such as automatic parking and adaptive cruise control, aiming to improve driving safety and convenience. Robotic systems further extend intelligent driving technology to service robots, industrial robots, and other fields, enabling robots to autonomously navigate, avoid obstacles, and complete specific tasks in complex environments, such as logistics delivery and warehouse management, demonstrating the broad application potential of intelligent driving technology in different scenarios.

[0027] In fields such as autonomous driving and intelligent robotics, Occupation Networks (OCCs) have become a key technology for intelligent agents to perceive their 3D environment because they can explicitly model the voxel state at each location in 3D space in a voxelized form. The training effect of OCC models highly depends on large-scale, high-quality occupation training data. Ideal occupation training data needs to accurately label the three occupation states of each voxel in 3D space: the "occupied state" where it is actually occupied by an object, the "free state" where there are no objects, and the "unknown state" where it cannot be observed due to object occlusion.

[0028] In this embodiment of the disclosure, the occupancy training data generation device includes one or more processors, which are configured to: acquire camera parameters, visual images, and depth ground truth images corresponding to a virtual camera; determine the field of view space corresponding to the virtual camera and point cloud data corresponding to the simulated scene within the field of view space based on the camera parameters and the depth ground truth images; determine the voxel state corresponding to a first voxel in the field of view space based on the point cloud data; wherein the first voxel is a voxel containing at least a preset number of point cloud points in the point cloud data; determine the voxel state corresponding to a second voxel based on the ray path voxels corresponding to the first voxel and the second voxel; wherein the second voxel is any voxel in the field of view other than the first voxel, and the ray path voxels corresponding to the second voxel include the voxels traversed by the ray path from the virtual camera to the second voxel; and generate occupancy training data corresponding to the field of view space based on the visual images, the voxel states corresponding to the first voxel, and the voxel states corresponding to the second voxel. The processor determines whether the second voxel is occluded by the first voxel by judging whether the ray path of the second voxel in the field of view contains the first voxel, and thus determines whether the voxel state corresponding to the second voxel is free or unknown. In this way, the processor generates occupancy training data that meets the training requirements of the occupancy network.

[0029] Furthermore, in this embodiment, the processor can not only label the voxel states of the first voxels (i.e., voxels occupied by entities) containing at least a preset number of point cloud points in the virtual camera's field of view as occupied states, but also, based on the visual occlusion relationships between voxels, label the voxel states of all second voxels (i.e., other voxels in the field of view besides the first voxels) in the virtual camera's field of view as free states or unknown states, thereby generating occupied training data containing three voxel states: occupied, free, and unknown. That is, the generation process of this occupied training data considers not only whether voxels are occupied by entities, but also the visual occlusion relationships between voxels, improving the accuracy of the occupied training data. Correspondingly, training the occupied network based on this occupied training data can also improve the training effect of the occupied network.

[0030] Exemplary System Figure 1 This is a schematic diagram of the structure of a training data generation apparatus provided in an exemplary embodiment of this disclosure. Figure 1 As shown, the occupancy training data generation device 100 includes one or more processors 110, which are configured to: acquire camera parameters, visual images, and depth ground truth images corresponding to a virtual camera; determine the field of view space corresponding to the virtual camera and point cloud data corresponding to the simulated scene within the field of view space based on the camera parameters and depth ground truth images; determine the voxel state corresponding to a first voxel in the field of view space based on the point cloud data; wherein the first voxel is a voxel containing at least a preset number of point cloud points in the point cloud data; determine the voxel state corresponding to a second voxel based on the ray path voxels corresponding to the first voxel and the second voxel; wherein the second voxel is any voxel in the field of view other than the first voxel, and the ray path voxels corresponding to the second voxel include the voxels traversed by the ray path from the virtual camera to the second voxel; and generate occupancy training data corresponding to the field of view space based on the visual images, the voxel states corresponding to the first voxel, and the voxel states corresponding to the second voxel.

[0031] For example, to ensure that the virtual camera maintains consistency with the real camera, the processor 110 can use the camera parameters corresponding to the real camera as the camera parameters corresponding to the virtual camera. The virtual camera is a virtual imaging device constructed in a simulation space. The virtual camera acquires visual images and depth ground truth images by simulating the imaging principle of a real camera. Camera parameters may include camera intrinsic parameters, camera extrinsic parameters, minimum observation depth, and maximum observation depth, etc., which are not limited in this embodiment.

[0032] The camera intrinsic parameters describe the optical characteristics of the virtual camera, and may include focal length and optical center coordinates. The camera extrinsic parameters describe the pose of the virtual camera in the world coordinate system, and may include baseline length. In this embodiment, the virtual camera may be a virtual monocular camera or a virtual binocular camera, and the baseline length of the virtual monocular camera is 0. The minimum observation depth is the closest distance at which the virtual camera can effectively image; objects smaller than this distance cannot be effectively observed by the virtual camera. The maximum observation depth is the farthest distance at which the virtual camera can effectively image; objects larger than this distance cannot be effectively observed by the virtual camera. For example, if a virtual camera has a minimum observation depth of 0.5 meters and a maximum observation depth of 50 meters, it means that only objects within a distance of 0.5 meters to 50 meters from the virtual camera can be observed by the virtual camera. Accordingly, only objects within this distance range can be imaged by the virtual camera.

[0033] The processor 110 can control a virtual camera to move along a preset trajectory in the simulation space. When the virtual camera moves to a point on the preset trajectory, the processor 110 can determine the transformation matrix from the world coordinate system to the image coordinate system based on the virtual camera's position at that point and the camera parameters. Then, for objects in the simulation space, the processor 110 can project the spatial coordinates of spatial points on the object onto the imaging plane of the virtual camera through the transformation matrix to obtain the pixel coordinates of the corresponding pixels on the object. Since the imaging plane of the virtual camera is a plane of fixed size, the pixel coordinates of some pixels on the object will fall on the imaging plane, while the pixel coordinates of others will not. Therefore, the processor 110 can generate a visual image and a depth ground truth image corresponding to the virtual camera based on the pixels that fall on the imaging plane. That is, the visual image and the depth ground truth image only contain a portion of the objects in the simulation scene, which can be considered as objects observable by the virtual camera. The visual image is an image reflecting the surface color, texture, and lighting characteristics of the objects observable by the virtual camera. A depth ground truth image is an image that reflects the depth value of a spatial point on an object observable by a virtual camera along the optical axis in the camera coordinate system. The pixel value of each pixel in this depth ground truth image represents the depth value, which is the Z-axis distance of the corresponding point cloud point in the camera coordinate system. For a monocular virtual camera, the visual image corresponding to the virtual camera includes both the visual image corresponding to the monocular virtual camera and the depth ground truth image corresponding to the virtual camera. For a binocular virtual camera, the visual image corresponding to the virtual camera includes both the visual image corresponding to the left and right virtual cameras. The depth ground truth image corresponding to the virtual camera includes both the depth ground truth image corresponding to the left and right virtual cameras.

[0034] After obtaining the camera parameters and ground truth depth image of the virtual camera, the processor 110 can determine the field of view space corresponding to the virtual camera based on these parameters and the ground truth depth image. The field of view space corresponding to the virtual camera is the three-dimensional spatial range that the virtual camera can effectively observe; it is a truncated cone enclosed by six planes, meaning the field of view space is a portion of the simulation space. Subsequently, the processor 110 can also determine the point cloud data corresponding to the simulation scene within the field of view space based on the camera parameters and the ground truth depth image. The processing procedure for the processor 110 to determine the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space based on the camera parameters and the ground truth depth image will be described in detail later and will not be repeated here.

[0035] The processor 110 can divide the simulation space into a regular three-dimensional voxel grid according to a preset voxel resolution. A voxel is the basic unit constituting the voxel grid; a voxel grid includes at least one voxel, and the initial voxel state corresponding to each voxel in the voxel grid is an undetermined state. After obtaining point cloud data, the processor 110 can traverse each point in the point cloud data and determine the voxel it belongs to based on the three-dimensional spatial coordinates of that point. Then, for each voxel in the field of view, the processor 110 can count the number of point points inside it and compare this number with a preset threshold. This preset threshold can be set to a positive integer greater than or equal to 1. If the number of point points corresponding to the voxel is greater than or equal to the preset threshold, the processor 110 can identify the voxel as the first voxel and determine its corresponding voxel state as an occupied state. The occupied state indicates that the space where the first voxel is located is occupied by an entity. If the number of point cloud points corresponding to the voxel is less than a preset threshold, the processor 110 retains the voxel state corresponding to the voxel as a pending state.

[0036] For each voxel other than the first voxel in the field of view (i.e., the second voxel), the processor 110 can acquire the ray path voxel corresponding to that second voxel. The ray path voxel corresponding to the second voxel includes the voxels traversed by the ray path from the virtual camera to the second voxel. Then, the processor 110 can further determine whether the second voxel is occluded by the first voxel in the ray path voxel corresponding to the second voxel (i.e., whether the second voxel can be directly observed by the virtual camera) in the ray propagation direction from the virtual camera to the second voxel by determining whether the second voxel is in a free state or an unknown state. In this context, the free state indicates that the second voxel can be directly observed by the virtual camera because it is not occluded by the first voxel. Based on the point cloud data determined from the ground truth depth image of the virtual camera, it can be determined that the second voxel does not contain any point cloud points, or the number of point cloud points it contains is less than a preset threshold. Therefore, the free state of the second voxel indicates that the space in which the voxel is located is not occupied by any entity. The unknown state indicates that the second voxel cannot be directly observed by the virtual camera because it is occluded by the first voxel. Based on the point cloud data determined from the ground truth depth image of the virtual camera, although it can be determined that the second voxel does not contain any point cloud points, or the number of point cloud points it contains is less than a preset threshold, it is possible that the second voxel cannot be observed by the virtual camera because it is occluded. That is, the unknown state of the second voxel indicates that the space in which the voxel is located may be unoccupied by any entity, or may be occupied by an entity.

[0037] After obtaining the voxel states corresponding to the first voxel and the second voxel, the processor 110 can merge the occupied state corresponding to the first voxel with the free / unknown state corresponding to the second voxel to generate a three-dimensional occupancy ground truth label matrix. The dimension of the three-dimensional occupancy ground truth label matrix is ​​consistent with the dimension of the voxel grid, and each element in the matrix corresponds one-to-one with a voxel in the grid, with the element's value representing the voxel state of its corresponding voxel. Then, the processor 110 can merge the visual image and the occupancy ground truth label matrix to obtain occupancy training data corresponding to the field of view. This training data is used for supervised training of the occupancy network (OCC). The visual image serves as the input data for Occupancy network training, and the occupancy ground truth label matrix serves as the supervision label for training. Based on the input visual image, the occupancy network outputs an occupancy prediction label matrix corresponding to the field of view. Then, by minimizing the error between the occupancy ground truth label matrix and the occupancy prediction label matrix, the model parameters of the occupancy network are iteratively optimized to obtain the trained occupancy network.

[0038] In this embodiment, the processor 110 determines that the voxel state corresponding to the first voxel in the field of view is occupied by checking whether it contains point cloud points from the point cloud data. Then, the processor 110 determines whether the second voxel is occluded by the first voxel by judging whether the ray path voxel corresponding to the second voxel in the field of view contains the first voxel, thereby determining whether the voxel state corresponding to the second voxel is free or unknown. In this way, the processor can ultimately generate occupancy training data that meets the training requirements of the occupancy network.

[0039] Furthermore, in this embodiment, the processor can not only label the voxel states of the first voxels (i.e., voxels occupied by entities) containing at least a preset number of point cloud points in the virtual camera's field of view as occupied states, but also, based on the visual occlusion relationships between voxels, label the voxel states of all second voxels (i.e., other voxels in the field of view besides the first voxels) in the virtual camera's field of view as free states or unknown states, thereby generating occupied training data containing three voxel states: occupied, free, and unknown. That is, the generation process of this occupied training data considers not only whether voxels are occupied by entities, but also the visual occlusion relationships between voxels, improving the accuracy of the occupied training data. Correspondingly, training the occupied network based on this occupied training data can also improve the training effect of the occupied network.

[0040] In one implementation, when the processor 110 executes the step of determining the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space based on camera parameters and a ground truth depth image, it is specifically configured to: construct the field of view space corresponding to the virtual camera based on the image size corresponding to the camera parameters and the ground truth depth image; and generate the point cloud data corresponding to the simulation scene within the field of view space based on the camera parameters, the first image coordinates of each pixel in the ground truth depth image, and the depth value.

[0041] For example, after obtaining the camera parameters and the ground truth depth image, the processor 110 can construct the field of view space corresponding to the virtual camera based on the image size corresponding to the camera parameters and the ground truth depth image. The field of view space corresponding to the virtual camera is used to determine the voxels observable by the virtual camera. After obtaining the camera parameters and the ground truth depth image, the processor 110 generates point cloud data corresponding to the simulated scene within the field of view space based on the camera parameters, the first image coordinates of each pixel in the ground truth depth image, and the depth value. The point cloud data is used to determine the voxel state of the voxels occupied by entities within the field of view space.

[0042] In this embodiment, the processor 110 determines the field of view of the virtual camera and only determines the state of voxels within that field of view. This effectively filters out invalid voxels outside the field of view and avoids determining the state of invalid voxels, thereby significantly reducing computational load. Simultaneously, by generating point cloud data corresponding to the simulated scene within the field of view, the processor 110 can accurately reconstruct the three-dimensional geometric position of objects in the simulated space, providing a reliable geometric basis for subsequently determining the voxels occupying their positions using point cloud data.

[0043] In one implementation, the camera parameters include camera intrinsic parameters, camera extrinsic parameters, minimum observation depth, and maximum observation depth. When the processor 110 executes the step of constructing the field of view space corresponding to the virtual camera based on the camera parameters and the image size corresponding to the ground truth depth image, it is specifically configured to: determine the minimum observation depth plane and the maximum observation depth plane of the virtual camera in the camera coordinate system based on the minimum observation depth and the maximum observation depth, respectively; determine the second image coordinates of each corner point in the ground truth depth image based on the image size corresponding to the ground truth depth image; back-project each corner point to the minimum observation depth plane and the maximum observation depth plane based on the camera intrinsic parameters, camera extrinsic parameters, and the second image coordinates corresponding to each corner point, respectively, to obtain the first spatial coordinates of each corner point on the minimum observation depth plane and the second spatial coordinates on the maximum observation depth plane; and construct the field of view space corresponding to the virtual camera based on the first spatial coordinates and the second spatial coordinates corresponding to each corner point.

[0044] For example, after obtaining the camera parameters of the virtual camera, the processor 110 can use the minimum observation depth as the Z-coordinate and the plane formed by all spatial points whose Z-coordinate is the minimum observation depth as the minimum observation depth plane. Similarly, the processor 110 can use the maximum observation depth as the Z-coordinate and the plane formed by all spatial points whose Z-coordinate is the maximum observation depth as the maximum observation depth plane.

[0045] After the processor 110 obtains the depth ground truth image corresponding to the virtual camera, it can determine the second image coordinates of each corner point in the depth ground truth image based on the image size of the depth ground truth image. The image size of the depth ground truth image includes its width (W) and height (H), which are typically expressed in pixels. Each corner point in the depth ground truth image includes the top-left, top-right, bottom-left, and bottom-right corner points. Taking the top-left corner point as the origin of the pixel coordinates, the image coordinates corresponding to the top-left corner point are (0, 0), the top-right corner point is (W, 0), the bottom-left corner point is (0, H), and the bottom-right corner point is (W, H). For example, when the depth ground truth image has a width of 640 pixels and a height of 480 pixels, the second image coordinates corresponding to its four corner points are the top left corner (0, 0), the top right corner (640, 0), the bottom left corner (0, 480), and the bottom right corner (640, 480).

[0046] After obtaining the second image coordinates of each corner point in the ground truth depth image, the processor 110 can further back-project each corner point onto the minimum observation depth plane and the maximum observation depth plane based on the camera intrinsic parameters, camera extrinsic parameters, and the corresponding second image coordinates of each corner point, thereby obtaining the first spatial coordinates of each corner point on the minimum observation depth plane and the second spatial coordinates on the maximum observation depth plane. The first spatial coordinates of each corner point are its three-dimensional spatial coordinates in the camera coordinate system after back-projection onto the minimum observation depth plane, with its Z-coordinate representing the minimum observation depth. Similarly, the second spatial coordinates of each corner point are its three-dimensional spatial coordinates in the camera coordinate system after back-projection onto the maximum observation depth plane, with its Z-coordinate representing the maximum observation depth.

[0047] The field of view is the three-dimensional spatial range that the virtual camera can effectively observe, and it is a frustum enclosed by six planes. After obtaining the first and second spatial coordinates corresponding to each corner point, the processor 110 can sequentially connect the first spatial coordinates corresponding to the four corner points on the minimum observation depth plane to obtain the near clipping plane of the frustum. Then, the processor 110 can sequentially connect the second spatial coordinates corresponding to the four corner points on the maximum observation depth plane to obtain the far clipping plane of the frustum. Afterwards, the processor 110 can connect the first and second spatial coordinates corresponding to the same corner point to obtain the four lateral faces of the frustum. Thus, the frustum enclosed by the near clipping plane, the far clipping plane, and the four lateral faces constitutes the field of view of the virtual camera.

[0048] It should be noted that for a monocular virtual camera, the baseline length in the camera extrinsic parameters of the monocular virtual camera is 0. The processor 110 only needs to construct the field of view space corresponding to the monocular virtual camera. However, for a binocular virtual camera, the field of view space corresponding to the left virtual camera and the field of view space corresponding to the right virtual camera can be constructed separately, using any monocular virtual camera included in the binocular virtual camera as a reference. For example, if the left virtual camera is used as a reference, the baseline length in the camera extrinsic parameters of the left virtual camera is 0, and the baseline length in the camera extrinsic parameters of the right virtual camera is the optical center distance between the left and right virtual cameras. The processor 110 needs to construct the field of view space corresponding to the left virtual camera and the field of view space corresponding to the right virtual camera. Other multi-view cameras, wide-angle fisheye cameras, and binocular virtual cameras are similar, and will not be described in detail in this embodiment.

[0049] In this embodiment, the processor 110 determines the field of view of the virtual camera based on the same camera parameters as the real camera, ensuring that the constructed field of view of the virtual camera accurately reflects the actual observation range of the real camera in the physical world. Simultaneously, the field of view of the virtual camera constructed by the processor 110 can strictly distinguish between the observable space (i.e., the virtual camera's field of view) and the unobservable space (i.e., other spaces within the simulation space besides the field of view), providing an accurate spatial reference range for subsequent voxel occlusion discrimination based on ray tracing. This effectively reduces the computational load of voxel occlusion discrimination and increases the generation rate of training data.

[0050] In one implementation, when the processor 110 executes the step of generating point cloud data corresponding to the simulated scene in the field of view space based on camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth depth image, it is specifically configured to: back-project each pixel to the camera coordinate system based on the camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth depth image, to obtain the third spatial coordinates of each pixel in the camera coordinate system; and generate point cloud data corresponding to the simulated scene in the field of view space based on the third spatial coordinates of each pixel.

[0051] For example, after obtaining the camera parameters and the ground truth depth image, the processor 110 can obtain the first image coordinates (u, v) and the corresponding depth value d for each pixel in the ground truth depth image. Then, based on the first image coordinates, depth value, and camera parameters, the processor 110 can back-project the pixel to the camera coordinate system to obtain the third spatial coordinates (X, v) of the point cloud point corresponding to the pixel in the camera coordinate system. c Y c Z cThe camera coordinate system of the virtual camera is a right-handed coordinate system with the optical center of the virtual camera as the origin, the X-axis to the right, the Y-axis downward, and the Z-axis forward along the optical axis. For a binocular virtual camera, the camera coordinate system can be either the left or right camera coordinate system; this embodiment does not limit this. This embodiment uses the example of the left camera coordinate system as the camera coordinate system for the binocular virtual camera; other cases are similar and will not be described in detail. Afterward, the processor 110 repeats the above steps, back-projecting all pixels to the camera coordinate system to obtain the third spatial coordinates of the point cloud points corresponding to all pixels in the camera coordinate system. Based on the third spatial coordinates of the point cloud points corresponding to all pixels, it can generate point cloud data corresponding to the simulated scene in the field of view. In one embodiment, firstly, the processor 110 can directly determine the depth value of the pixel as the Z-axis of the point cloud point corresponding to the pixel in the third spatial coordinates of the camera coordinate system. c Then, the processor 110 can base its analysis on the pixel's first image coordinates (u, v) and depth value d, as well as the focal length (f) in the camera parameters. x f y ) and optical center coordinates (c x c y ), calculate the X coordinates of the point cloud corresponding to the pixel in the third space coordinates of the camera coordinate system. c and Y c Among them, X c =(uc) x )×d / f x Y c =(vc y )×d / f y .

[0052] It should be noted that for binocular virtual cameras, either the left or right virtual camera can be used as a reference. If the left virtual camera is used as the reference, for the left virtual camera, the processor 110 can back-project each left-eye pixel to the left-eye camera coordinate system based on the camera parameters of the left virtual camera, the first image coordinates and depth values ​​of each left-eye pixel in the ground truth depth image, to obtain the third spatial coordinates of each left-eye pixel in the left-eye camera coordinate system. Then, for the right virtual camera, the processor 110 can back-project each right-eye pixel to the right-eye camera coordinate system based on the camera parameters of the right virtual camera, the first image coordinates and depth values ​​of each right-eye pixel in the ground truth depth image, to obtain the third spatial coordinates of each right-eye pixel in the right-eye camera coordinate system. Afterwards, for the right virtual camera, the processor 110 can perform coordinate system transformation processing on the third spatial coordinates of each right-eye pixel in the right-eye camera coordinate system based on the baseline length between the left and right virtual cameras, to obtain the third spatial coordinates of each right-eye pixel in the left-eye camera coordinate system. Finally, the processor 110 can generate point cloud data corresponding to the simulated scene in the field of view space based on the third spatial coordinates of each left-eye pixel and each right-eye pixel.

[0053] In this embodiment, the processor 110 generates point cloud data corresponding to the simulated scene within the field of view space based on camera parameters, the first image coordinates of each pixel in the ground truth depth image, and the depth value. In this way, the processor 110 can subsequently accurately determine the occupied voxel state of each voxel based on the distribution of the point cloud data in the voxel grid.

[0054] In one embodiment, when the processor 110 executes the step of determining the voxel state corresponding to the second voxel based on the ray path voxel corresponding to the first voxel and the second voxel, it is specifically configured as follows: if the virtual camera is a monocular virtual camera, and the ray path voxel corresponding to the second voxel is the first ray path voxel in the monocular virtual camera's view, in response to the presence of the first voxel in the first ray path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state; in response to the absence of the first voxel in the first ray path voxel, the voxel state corresponding to the second voxel is determined to be a free state; if the virtual camera is a binocular virtual camera, and the ray path voxel corresponding to the second voxel includes the second ray path voxel in the left virtual camera's view and the third ray path voxel in the right virtual camera's view, in response to the presence of the first voxel in both the second and third ray path voxels, the voxel state corresponding to the second voxel is determined to be an unknown state; in response to the absence of the first voxel in the second ray path voxel and / or the third ray path voxel, the voxel state corresponding to the second voxel is determined to be a free state.

[0055] For example, after the processor 110 determines that the voxel state corresponding to the first voxel in the field of view is occupied, it can further determine whether the second voxel in the field of view is occluded by the first voxel, thereby determining whether the second voxel can be directly observed by the virtual camera, and further determining whether the voxel state corresponding to the second voxel is free or unknown. Here, a free state indicates that the second voxel can be directly observed by the virtual camera because it is not occluded by the first voxel; an unknown state indicates that the second voxel cannot be directly observed by the virtual camera because it is occluded by the first voxel.

[0056] When the virtual camera is a monocular virtual camera, the ray-path voxel corresponding to the second voxel only contains the first ray-path voxel from the monocular virtual camera's perspective. For each second voxel, the processor 110 can determine whether the first voxel exists in the first ray-path voxel corresponding to that second voxel. If the first voxel exists in the first ray-path voxel, it means that the second voxel is occluded by the first voxel and cannot be directly observed by the virtual camera. Accordingly, in response to the presence of the first voxel in the first ray-path voxel, the processor 110 determines that the voxel state corresponding to the second voxel is an unknown state. Similarly, if the first voxel does not exist in the first ray-path voxel, it means that the second voxel is not occluded by the first voxel and can be directly observed by the virtual camera. Accordingly, in response to the absence of the first voxel in the first ray-path voxel, the processor 110 determines that the voxel state corresponding to the second voxel is a free state.

[0057] When the virtual camera is a binocular virtual camera, because the optical center coordinates of the left virtual camera and the right virtual camera are different, the ray path voxel corresponding to the second voxel may be different from the left and right virtual camera perspectives. That is, the ray path voxel corresponding to the second voxel includes the second ray path voxel from the left virtual camera perspective and the third ray path voxel from the right virtual camera perspective. For each second voxel, the processor 110 can determine whether the first voxel is present in both the second and third ray path voxels corresponding to that second voxel from the left and right virtual camera perspectives. If the first voxel is present in both the second and third ray path voxels, the processor 110 determines that the voxel state corresponding to the second voxel is unknown. For example, such as... Figure 2As shown, in the left-eye virtual camera view, the second ray path voxels corresponding to the second voxel V7 include [V2, V4, V6]. Among them, voxel V4 is the first voxel. In the right-eye virtual camera view, the third ray path voxels corresponding to the second voxel V7 include [V1, V3, V5]. Among them, voxel V5 is the first voxel. Accordingly, the voxel state corresponding to the second voxel V7 is unknown. If, in either the left-eye or right-eye virtual camera view, the first voxel is absent from the second ray path voxels and / or the third ray path voxels corresponding to the second voxel, then the processor 110 determines that the voxel state corresponding to the second voxel is free. Where the first voxel is absent in the voxel through which the second ray passes and / or the voxel through which the third ray passes includes: (1) the first voxel is absent in the voxel through which the second ray passes, but the first voxel is present in the voxel through which the third ray passes; (2) the first voxel is present in the voxel through which the second ray passes, but the first voxel is absent in the voxel through which the third ray passes; (3) the first voxel is absent in both the voxel through which the second ray passes and the first voxel is absent in both the voxel through which the third ray passes. For example, such as Figure 3 As shown, from the perspective of the left virtual camera, the second ray path voxels corresponding to the second voxel V7 include [V2, V4, V6]. Among them, voxel V4 is the first voxel. From the perspective of the right virtual camera, the third ray path voxels corresponding to the second voxel V7 include [V1, V3, V5]. Correspondingly, the voxel state corresponding to the second voxel V7 is a free state.

[0058] In this embodiment, the processor 110 determines whether the second voxel is occluded by the first voxel by judging whether the voxel through which the ray corresponding to the second voxel passes in the field of view contains the first voxel, and thus determines whether the voxel state corresponding to the second voxel is a free state or an unknown state. In this way, the processor 110 can ultimately generate occupancy training data containing three voxel states: occupied, free, and unknown.

[0059] In one implementation, the processor 110 is further configured to: construct a ray path from the virtual camera to a voxel based on the optical center coordinates corresponding to the virtual camera and the voxel center coordinates of each voxel in the field of view. The voxels traversed by the ray path are then identified as the voxels through which the ray passes.

[0060] For example, after obtaining the field of view space corresponding to the virtual camera, the processor 110 can obtain the voxel center coordinates for each voxel in the field of view space. The voxel center coordinates are the three-dimensional spatial coordinates of the voxel's center point in the camera coordinate system. Then, the processor 110 can further construct a ray path from the virtual camera to the voxel, starting from the optical center coordinates corresponding to the virtual camera and ending at the voxel's voxel center coordinates. Afterwards, the processor 110 can arrange all voxels traversed by the ray path in order from the optical center to the voxel, obtaining the ray path voxels corresponding to the voxel. The processor 110 can store the voxel's index as the key and the corresponding ray path voxel as the value in a structured ray path voxel lookup table. In this way, the processor 110 can directly obtain the ray path voxel corresponding to the voxel through a table lookup, without needing to perform repeated calculations. The processor 110 determines the voxel through which the light path corresponds to the voxel. This is a general processing procedure. Those skilled in the art can specifically use voxel traversal algorithms such as the Bresenham algorithm and the Amanatides-Woo algorithm to determine the voxel through which the light path corresponds to the voxel. This embodiment of the present disclosure does not limit this process.

[0061] It should be noted that the above processing procedure is described using a monocular virtual camera as an example. For a binocular virtual camera, a ray path from the left virtual camera to a voxel is constructed based on the optical center coordinates of the left virtual camera and the voxel center coordinates of each voxel in the field of view. The voxels along this ray path are then designated as the second ray path voxels. Similarly, a ray path from the right virtual camera to a voxel is constructed based on the optical center coordinates of the right virtual camera and the voxel center coordinates of each voxel in the field of view. The voxels along this ray path are then designated as the third ray path voxels. Because the optical center coordinates of the left and right virtual cameras are different, the ray path voxels corresponding to the same voxel may differ between the left and right virtual camera perspectives. For example, in the left virtual camera perspective, the ray path voxels corresponding to voxel V7 include [V2, V4, V6]. However, in the right virtual camera perspective, the ray path voxels corresponding to voxel V7 include [V1, V3, V5].

[0062] In this embodiment of the disclosure, the processor 110 constructs a ray path from the virtual camera to the voxel to determine the voxel through which the ray passes. Subsequently, the processor 110 can determine, based on the voxel through which the ray passes, whether the voxel through which the ray passes for the second voxel in the field of view contains the first voxel, thereby determining whether the second voxel is occluded by the first voxel, and thus determining whether the voxel state corresponding to the second voxel is a free state or an unknown state.

[0063] In one embodiment, the processor 110 is further configured to: determine that the voxel state corresponding to a third voxel in the simulation space is an unknown state; wherein the third voxel is a voxel located outside the field of view. Based on the visual image, the voxel state corresponding to the first voxel, the voxel state corresponding to the second voxel, and the voxel state corresponding to the third voxel, occupancy training data corresponding to the simulation space is generated.

[0064] For example, voxels located outside the virtual camera's field of view (i.e., the third voxel) cannot be directly observed by the virtual camera. Therefore, regardless of whether the space where the third voxel is located is occupied by an entity, the processor 110 can directly determine that the voxel state corresponding to the third voxel is an unknown state. After obtaining the voxel states corresponding to the first voxel, the second voxel, and the third voxel, the processor 110 can merge the occupied state corresponding to the first voxel, the free / unknown state corresponding to the second voxel, and the unknown state corresponding to the third voxel to generate a three-dimensional occupancy ground truth label matrix with dimensions completely consistent with the voxel grid corresponding to the simulation space. Then, the processor 110 can merge the visual image and the occupancy ground truth label matrix to obtain the occupancy training data corresponding to the simulation space.

[0065] In this embodiment of the present disclosure, the processor 110 marks the voxel states of all third voxels outside the field of view as unknown states, thereby ensuring that the occupation of training data conforms to the objective physical laws of camera observation.

[0066] Exemplary methods Figure 4 This is a flowchart illustrating an exemplary embodiment of the present disclosure of a method for generating training data. Figure 4 As shown, it includes the following steps: Step 401: Obtain the camera parameters, visual image, and depth ground truth image corresponding to the virtual camera.

[0067] Step 402: Based on camera parameters and ground truth depth images, determine the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space.

[0068] Step 403: Based on the point cloud data, determine the voxel state corresponding to the first voxel in the field of view space. The first voxel is a voxel containing at least a preset number of point cloud points from the point cloud data.

[0069] Step 404: Determine the voxel state corresponding to the second voxel based on the ray path voxels corresponding to the first voxel and the second voxel. Here, the second voxel refers to any voxel in the field of view other than the first voxel, and the ray path voxels corresponding to the second voxel include the voxels traversed by the ray path from the virtual camera to the second voxel.

[0070] Step 405: Based on the visual image, the voxel state corresponding to the first voxel, and the voxel state corresponding to the second voxel, generate occupancy training data corresponding to the field of view space.

[0071] In one implementation, such as Figure 5 As shown above, in the above Figure 4 Based on the illustrated embodiment, step 402 may include the following steps: Step 501: Based on the camera parameters and the image size corresponding to the ground truth depth image, construct the field of view space corresponding to the virtual camera.

[0072] Step 502: Based on camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth depth image, generate point cloud data corresponding to the simulated scene in the field of view space.

[0073] In one implementation, the camera parameters include camera intrinsic parameters, camera extrinsic parameters, minimum observation depth, and maximum observation depth. For example... Figure 6 As shown above, in the above Figure 5 Based on the illustrated embodiment, step 501 may include the following steps: Step 601: Based on the minimum observation depth and the maximum observation depth, determine the minimum observation depth plane and the maximum observation depth plane of the virtual camera in the camera coordinate system, respectively.

[0074] Step 602: Based on the image size corresponding to the ground truth depth image, determine the second image coordinates of each corner point in the ground truth depth image.

[0075] Step 603: Based on the camera intrinsic parameters, camera extrinsic parameters, and the second image coordinates corresponding to each corner point, back-project each corner point to the minimum observation depth plane and the maximum observation depth plane respectively, and obtain the first spatial coordinates of each corner point on the minimum observation depth plane and the second spatial coordinates on the maximum observation depth plane.

[0076] Step 604: Construct the field of view space corresponding to the virtual camera based on the first spatial coordinates and the second spatial coordinates corresponding to each corner point.

[0077] In one implementation, such as Figure 7 As shown above, in the above Figure 5 Based on the illustrated embodiment, step 502 may include the following steps: Step 701: Based on the camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth depth image, back-project each pixel to the camera coordinate system to obtain the third spatial coordinates of each pixel in the camera coordinate system.

[0078] Step 702: Based on the third spatial coordinates corresponding to each pixel, generate point cloud data corresponding to the simulation scene in the field of view space.

[0079] In one implementation, such as Figure 8 As shown above, in the above Figure 4 Based on the illustrated embodiment, step 404 may include the following steps: Step 801a: If the virtual camera is a monocular virtual camera, and the ray-path voxel corresponding to the second voxel is the first ray-path voxel in the monocular virtual camera's viewpoint, in response to the presence of the first voxel in the first ray-path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state. In response to the absence of the first voxel in the first ray-path voxel, the voxel state corresponding to the second voxel is determined to be a free state.

[0080] Step 801b: If the virtual camera is a binocular virtual camera, the ray path voxel corresponding to the second voxel includes the second ray path voxel from the left virtual camera's perspective and the third ray path voxel from the right virtual camera's perspective. In response to the presence of a first voxel in both the second and third ray path voxels, the voxel state corresponding to the second voxel is determined to be an unknown state. In response to the absence of a first voxel in the second and / or third ray path voxels, the voxel state corresponding to the second voxel is determined to be a free state.

[0081] In one implementation, such as Figure 9 As shown, this method for generating training data may also include the following steps: Step 901: Based on the optical center coordinates of the virtual camera and the voxel center coordinates of each voxel in the field of view, construct the ray path from the virtual camera to the voxel.

[0082] Step 902: Determine the voxels through which the light path passes as the voxels through which the light path passes.

[0083] In one implementation, such as Figure 10 As shown, this method for generating training data may also include the following steps: Step 1001: Determine that the voxel state corresponding to the third voxel in the simulation space is an unknown state. Here, the third voxel is a voxel located outside the field of view.

[0084] Step 1002: Based on the visual image, the voxel state corresponding to the first voxel, the voxel state corresponding to the second voxel, and the voxel state corresponding to the third voxel, generate occupancy training data corresponding to the simulation space.

[0085] Regarding the occupation training data generation method in the above embodiments, the specific execution methods of each step and the corresponding beneficial effects have been described in detail in the corresponding embodiment section of the aforementioned occupation training data generation device section. Please refer to the corresponding execution methods and beneficial technical effects of the aforementioned exemplary system section, which will not be repeated here.

[0086] Exemplary device Figure 11 This is a schematic diagram of the structure of a training data generation apparatus provided in an exemplary embodiment of this disclosure. Figure 11 As shown, the training data generation device includes a data acquisition module 1110, a data determination module 1120, a first voxel state determination module 1130, a second voxel state determination module 1140, and a first training data determination module 1150.

[0087] Data acquisition module 1110 is used to acquire camera parameters, visual images and depth ground truth images corresponding to the virtual camera; The data determination module 1120 is used to determine the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space based on the camera parameters and the depth ground truth image. The first voxel state determination module 1130 is used to determine the voxel state corresponding to the first voxel in the field of view based on the point cloud data; wherein, the first voxel is a voxel that contains at least a preset number of point cloud points in the point cloud data. The second voxel state determination module 1140 is used to determine the voxel state corresponding to the second voxel based on the light path voxels corresponding to the first voxel and the second voxel; wherein, the second voxel is other voxels in the field of view space besides the first voxel, and the light path voxels corresponding to the second voxel include the voxels through which the light path from the virtual camera to the second voxel passes. The first training data determination module 1150 is used to generate occupancy training data corresponding to the field of view space based on the visual image, the voxel state corresponding to the first voxel and the voxel state corresponding to the second voxel.

[0088] In one embodiment, the data determination module 1120 includes: The field of view space construction unit is used to construct the field of view space corresponding to the virtual camera based on the camera parameters and the image size corresponding to the ground truth depth image; The point cloud data generation unit is used to generate point cloud data corresponding to the simulated scene in the field of view space based on the camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth depth image.

[0089] In one embodiment, the camera parameters include camera intrinsic parameters, camera extrinsic parameters, minimum observation depth, and maximum observation depth; the field-of-view spatial construction unit is specifically used for: Based on the minimum observation depth and the maximum observation depth, the minimum observation depth plane and the maximum observation depth plane of the virtual camera in the camera coordinate system are determined respectively; Based on the image size corresponding to the ground truth depth image, determine the second image coordinates of each corner point in the ground truth depth image; Based on the camera intrinsic parameters, the camera extrinsic parameters, and the second image coordinates corresponding to each corner point, each corner point is back-projected onto the minimum observation depth plane and the maximum observation depth plane respectively to obtain the first spatial coordinates of each corner point on the minimum observation depth plane and the second spatial coordinates on the maximum observation depth plane. Based on the first spatial coordinates and the second spatial coordinates corresponding to each corner point, the field of view space corresponding to the virtual camera is constructed.

[0090] In one embodiment, the point cloud data generation unit is specifically used for: Based on the camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth image, each pixel is back-projected onto the camera coordinate system to obtain the third spatial coordinates of each pixel in the camera coordinate system. Based on the third spatial coordinates corresponding to each pixel, point cloud data corresponding to the simulated scene within the field of view is generated.

[0091] In one embodiment, the second voxel state determination module 1140 includes: If the virtual camera is a monocular virtual camera, the ray path voxel corresponding to the second voxel is the first ray path voxel under the monocular virtual camera's viewpoint. In response to the presence of the first voxel in the first ray path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state. The second voxel state determination unit determines the voxel state corresponding to the second voxel to be a free state in response to the absence of the first voxel in the voxel through which the first light passes. The third voxel state determination unit, if the virtual camera is a binocular virtual camera, the light path voxel corresponding to the second voxel includes the second light path voxel under the view of the left virtual camera and the third light path voxel under the view of the right virtual camera. In response to the presence of the first voxel in both the second light path voxel and the third light path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state. The fourth voxel state determination unit determines that the voxel state corresponding to the second voxel is a free state in response to the absence of the first voxel in the voxel through which the second ray passes and / or the voxel through which the third ray passes.

[0092] In one embodiment, the device further includes: The ray path determination module is used to construct a ray path from the virtual camera to the voxel based on the optical center coordinates corresponding to the virtual camera and the voxel center coordinates of each voxel in the field of view. The light path voxel determination module is used to determine the voxels through which the light path passes as the light path voxels corresponding to the voxels.

[0093] In one embodiment, the device further includes: The third voxel state determination module is used to determine that the voxel state corresponding to the third voxel in the simulation space is unknown; wherein, the third voxel is a voxel located outside the field of view. The second training data determination module is used to generate occupancy training data corresponding to the simulation space based on the visual image, the voxel state corresponding to the first voxel, the voxel state corresponding to the second voxel, and the voxel state corresponding to the third voxel.

[0094] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0095] Exemplary electronic devices Figure 12 A structural diagram of an electronic device provided in an embodiment of this disclosure includes at least one processor 11 and a memory 12.

[0096] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0097] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the training data generation methods and / or other desired functions of the various embodiments of this disclosure described above.

[0098] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0099] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0100] The output device 14 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0101] Of course, for the sake of simplicity, Figure 12 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0102] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the occupation training data generation method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0103] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0104] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the occupied training data generation methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0105] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0107] Various modifications and variations can be made to this disclosure without departing from its spirit and scope. Therefore, this disclosure is also intended to include such modifications and variations if they fall within the scope of the claims of this disclosure and their equivalents.

Claims

1. An apparatus for generating training data, comprising one or more processors, said one or more processors being configured to: Obtain the camera parameters, visual image, and depth ground truth image corresponding to the virtual camera; Based on the camera parameters and the ground truth depth image, determine the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space; Based on the point cloud data, the voxel state corresponding to the first voxel in the field of view is determined; wherein, the first voxel is a voxel that contains at least a preset number of point cloud points in the point cloud data. Based on the ray path voxels corresponding to the first voxel and the second voxel, the voxel state corresponding to the second voxel is determined; wherein, the second voxel is other voxels in the field of view space besides the first voxel, and the ray path voxels corresponding to the second voxel include the voxels through which the ray path from the virtual camera to the second voxel passes; Based on the visual image, the voxel state corresponding to the first voxel, and the voxel state corresponding to the second voxel, occupancy training data corresponding to the field of view space is generated.

2. The apparatus according to claim 1, wherein, The step of determining the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulated scene within the field of view space based on the camera parameters and the ground truth depth image includes: Based on the camera parameters and the image size corresponding to the ground truth depth image, the field of view space corresponding to the virtual camera is constructed; Based on the camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth depth image, point cloud data corresponding to the simulated scene in the field of view space is generated.

3. The apparatus according to claim 2, wherein, The camera parameters include camera intrinsic parameters, camera extrinsic parameters, minimum observation depth, and maximum observation depth; the construction of the field of view space corresponding to the virtual camera based on the camera parameters and the image size corresponding to the ground truth depth image includes: Based on the minimum observation depth and the maximum observation depth, the minimum observation depth plane and the maximum observation depth plane of the virtual camera in the camera coordinate system are determined respectively; Based on the image size corresponding to the ground truth depth image, determine the second image coordinates of each corner point in the ground truth depth image; Based on the camera intrinsic parameters, the camera extrinsic parameters, and the second image coordinates corresponding to each corner point, each corner point is back-projected onto the minimum observation depth plane and the maximum observation depth plane respectively to obtain the first spatial coordinates of each corner point on the minimum observation depth plane and the second spatial coordinates on the maximum observation depth plane. Based on the first spatial coordinates and the second spatial coordinates corresponding to each corner point, the field of view space corresponding to the virtual camera is constructed.

4. The apparatus according to claim 2, wherein, The step of generating point cloud data corresponding to the simulated scene within the field of view space based on the camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth depth image, includes: Based on the camera parameters, the first image coordinates and depth values ​​of each pixel in the ground truth image, each pixel is back-projected onto the camera coordinate system to obtain the third spatial coordinates of each pixel in the camera coordinate system. Based on the third spatial coordinates corresponding to each pixel, point cloud data corresponding to the simulated scene within the field of view is generated.

5. The apparatus according to claim 1, wherein, The step of determining the voxel state corresponding to the second voxel based on the light path voxel corresponding to the first voxel and the second voxel includes: If the virtual camera is a monocular virtual camera, the light path voxel corresponding to the second voxel is the first light path voxel under the monocular virtual camera's viewpoint. In response to the presence of the first voxel in the first light path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state. In response to the absence of the first voxel in the voxel through which the first light ray passes, the voxel state corresponding to the second voxel is determined to be a free state. If the virtual camera is a binocular virtual camera, the light path voxel corresponding to the second voxel includes the second light path voxel under the view of the left virtual camera and the third light path voxel under the view of the right virtual camera. In response to the presence of the first voxel in both the second light path voxel and the third light path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state. In response to the absence of the first voxel in the voxel through which the second ray passes and / or the voxel through which the third ray passes, the voxel state corresponding to the second voxel is determined to be a free state.

6. The apparatus of claim 1, wherein the one or more processors are further configured to: Based on the optical center coordinates of the virtual camera and the voxel center coordinates of each voxel in the field of view, a ray path from the virtual camera to the voxel is constructed. The voxels along the light path are identified as the voxels through which the light path passes.

7. The apparatus of claim 1, wherein the one or more processors are further configured to: The voxel state corresponding to the third voxel in the simulation space is determined to be an unknown state; whereby... The third voxel is a voxel located outside the field of view. Based on the visual image, the voxel state corresponding to the first voxel, the voxel state corresponding to the second voxel, and the voxel state corresponding to the third voxel, occupancy training data corresponding to the simulation space is generated.

8. A method for generating training data, comprising: Obtain the camera parameters, visual image, and depth ground truth image corresponding to the virtual camera; Based on the camera parameters and the ground truth depth image, determine the field of view space corresponding to the virtual camera and the point cloud data corresponding to the simulation scene within the field of view space; Based on the point cloud data, the voxel state corresponding to the first voxel in the field of view is determined; wherein, the first voxel is a voxel that contains at least a preset number of point cloud points in the point cloud data. Based on the ray path voxels corresponding to the first voxel and the second voxel, the voxel state corresponding to the second voxel is determined; wherein, the second voxel is other voxels in the field of view space besides the first voxel, and the ray path voxels corresponding to the second voxel include the voxels through which the ray path from the virtual camera to the second voxel passes; Based on the visual image, the voxel state corresponding to the first voxel, and the voxel state corresponding to the second voxel, occupancy training data corresponding to the field of view space is generated.

9. The method according to claim 8, wherein, The step of determining the voxel state corresponding to the second voxel based on the light path voxel corresponding to the first voxel and the second voxel includes: If the virtual camera is a monocular virtual camera, the light path voxel corresponding to the second voxel is the first light path voxel under the monocular virtual camera's viewpoint. In response to the presence of the first voxel in the first light path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state. In response to the absence of the first voxel in the voxel through which the first light ray passes, the voxel state corresponding to the second voxel is determined to be a free state. If the virtual camera is a binocular virtual camera, the light path voxel corresponding to the second voxel includes the second light path voxel under the view of the left virtual camera and the third light path voxel under the view of the right virtual camera. In response to the presence of the first voxel in both the second light path voxel and the third light path voxel, the voxel state corresponding to the second voxel is determined to be an unknown state. In response to the absence of the first voxel in the voxel through which the second ray passes and / or the voxel through which the third ray passes, the voxel state corresponding to the second voxel is determined to be a free state.

10. An electronic device, the electronic device comprising: The apparatus as described in any one of claims 1 to 7; or, The electronic device includes a processor and a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the occupation training data generation method according to any one of claims 8 to 9.

11. A computer-readable storage medium storing a computer program that is executed by a processor to perform the occupational training data generation method according to any one of claims 8 to 9.

Images