Depth map generation method and device, equipment and storage medium

Abstract

The invention provides a depth map generation method and device, equipment and a storage medium, and relates to the technical field of data processing and image processing, in particular to the technical field of artificial intelligence, big data, automatic driving, autonomous parking and the like. According to the specific implementation scheme, the method comprises the steps of obtaining a target dynamic occupation grid corresponding to a target frame, and obtaining a target static occupation grid corresponding to the target frame; the target dynamic occupation grid is constructed based on a dynamic target body in the original point cloud data of the target frame; the target static occupation grid is constructed based on a static target body in the original point cloud data of the target frame; the target frame is acquired based on radar equipment of a target vehicle; based on a target static occupancy grid and the target dynamic occupancy grid, obtaining a total occupancy grid corresponding to the target frame in a vehicle coordinate system of the target vehicle; and obtaining a target depth map corresponding to the target frame by using the total occupied grid.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of data processing and image processing, and in particular to the technical fields of artificial intelligence, big data, autonomous driving, and autonomous parking. Background Art

[0002] In autonomous driving scenarios, producing dense depth images is a key technology, which is essential for vehicles to understand the surrounding environment, perform path planning and avoid obstacles. However, in practical applications, especially in long-distance scenarios and scenarios involving dynamic objects, the problem of inaccurate depth accuracy is often encountered, which greatly challenges the safety and reliability of autonomous driving systems or assisted driving systems. Summary of the invention

[0003] The present disclosure provides a depth map generation method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a depth map generation method is provided, comprising:

[0005] Obtain a target dynamic occupied grid corresponding to a target frame, and obtain a target static occupied grid corresponding to the target frame; wherein the target dynamic occupied grid is constructed based on a dynamic target body in the original point cloud data of the target frame; the target static occupied grid is constructed based on a static target body in the original point cloud data of the target frame; the target frame is acquired based on radar equipment of the target vehicle;

[0006] Based on the target statically occupied grid and the target dynamically occupied grid, obtaining a total occupied grid corresponding to the target frame in the vehicle coordinate system of the target vehicle;

[0007] The target depth map corresponding to the target frame is obtained by using the total occupied grid.

[0008] According to another aspect of the present disclosure, there is provided a depth map generating apparatus, comprising:

[0009] A grid determination unit is used to obtain a target dynamic occupied grid corresponding to a target frame, and obtain a target static occupied grid corresponding to the target frame; wherein the target dynamic occupied grid is constructed based on a dynamic target body in the original point cloud data of the target frame; the target static occupied grid is constructed based on a static target body in the original point cloud data of the target frame; the target frame is acquired based on radar equipment of a target vehicle;

[0010] A grid processing unit, configured to obtain a total occupied grid corresponding to the target frame in a vehicle coordinate system of the target vehicle based on the target static occupied grid and the target dynamic occupied grid;

[0011] An image generating unit is used to obtain a target depth map corresponding to the target frame by using the total occupied grid.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] In this way, the disclosed solution can use the voxel occupancy grid (that is, the total occupied grid mentioned above) as reference information for depth estimation, effectively solving the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance). In this way, it provides high-precision densified depth data for subsequent scenarios, such as autonomous driving, assisted driving, or subsequent model training, thereby providing strong support for enriching and improving user experience.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 This is a schematic flow chart of a depth map generation method according to an embodiment of the present application. Figure 1 ;

[0022] Figure 2 This is a schematic flow chart of a depth map generation method according to an embodiment of the present application. Figure 2 ;

[0023] Figure 3is a schematic diagram of a scene with pre-annotated multi-frame point cloud data according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a scene of dynamic occupation grid completion and reconstruction according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of a scene of static occupancy grid reconstruction according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of a scene for determining visibility of dynamic and static occupied grids according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of a scene for depth estimation according to an embodiment of the present application;

[0028] Figure 8 is a schematic structural diagram of a depth map generating device according to an embodiment of the present application;

[0029] Fig. 9 It is a block diagram of an electronic device used to implement the depth map generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, it can mean including any one or more elements selected from the set consisting of A, B and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0032] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0033] The disclosed solution provides a solution to solve the problem of inaccurate depth accuracy of distant scenes and / or dynamic objects when producing dense depth images in autonomous driving scenarios or assisted driving scenarios. The solution can be applied to the production line of dense depth image data.

[0034] Figure 1 This is a schematic flow chart of a depth map generation method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0035] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0036] Step S101: Obtain the target dynamic occupation grid corresponding to the target frame.

[0037] Here, the target dynamic occupation grid is constructed based on the dynamic target body in the original point cloud data of the target frame.

[0038] Here, in one example, each obstacle detected in the foreground can be regarded as a dynamic target body (or can be called a foreground target body) in the disclosed solution. It can be understood that the obstacles detected in the foreground can be considered as potential dynamic objects, so the assumption that each obstacle detected in the foreground is a dynamic target body in this example is reasonable. Furthermore, it should be noted that the obstacles identified in the actual foreground detection process may be dynamic targets (foreground parts) or static parts (background parts), and the disclosed solution does not limit this; in other words, the "dynamic" in the dynamic target body described in the disclosed solution can be understood as a potential dynamic object, and is not used to limit the target body to be in motion.

[0039] Further, the target frame is acquired based on the radar device of the target vehicle. For example, in one example, the radar device of the target vehicle acquires N frames, that is, N frames of original point cloud data. At this time, the target frame can be specifically any feature in the N frames; here, N is a natural number greater than or equal to 1. Further, N is a natural number greater than or equal to 2.

[0040] Step S102: Obtain the target static occupation grid corresponding to the target frame.

[0041] Here, the target static occupancy grid is constructed based on the static target body in the original point cloud data of the target frame.

[0042] Here, in one example, other targets other than the dynamic targets described above can be identified as static targets. Further, in one example, the dynamic targets can be mainly the ground, curbs, fences, green plants or walls, etc., and the disclosed solution does not limit this.

[0043] Step S103: Based on the target statically occupied grid and the target dynamically occupied grid, obtain the total occupied grid corresponding to the target frame in the vehicle coordinate system of the target vehicle.

[0044] For example, in one example, the target static occupancy grid and the target dynamic occupancy grid can be superimposed together to obtain a global occupancy grid (i.e., the total occupancy grid described above). At this time, the total occupancy grid includes not only the dynamic target body but also the static target body, thus providing strong support for obtaining a high-precision dense depth map in the subsequent step.

[0045] Furthermore, it should be noted that the vehicle coordinate system in this example can be used to describe a reference coordinate system for the movement and position of the vehicle, which can usually be established based on the structure and motion characteristics of the vehicle itself. For example, the geometric center of the target vehicle can be used as the origin, the forward direction of the target vehicle can be used as the X-axis, the Y-axis is perpendicular to the X-axis, for example, it can point to one side of the target vehicle (for example, when the target vehicle is in the forward direction, the Y-axis points to the left side of the vehicle), and the Z-axis is perpendicular to the horizontal plane of the target vehicle and points to the sky. It should be noted that in actual applications, the establishment of the vehicle coordinate system can also follow the right-hand rule, that is, if the thumb of the right hand points to the positive direction of the X-axis and the index finger points to the positive direction of the Y-axis, then the direction pointed by the middle finger is the positive direction of the Z-axis.

[0046] It can be understood that the above is only an exemplary description, and in actual application, the vehicle coordinate system can be established based on actual needs.

[0047] Step S104: using the total occupied grid, obtaining a target depth map corresponding to the target frame.

[0048] For example, in one example, at least the total occupied grid may be input into a depth estimation model to obtain a target depth map corresponding to the target frame.

[0049] In a specific example, the depth estimation model is a trained model that can perform dense depth map estimation. Further, it should be noted that the target depth map obtained by the disclosed solution is a dense depth map.

[0050] Here, dense depth map is an important concept in the field of computer vision and 3D reconstruction. It refers to an image that contains depth information for each pixel in the scene. It can provide a more detailed description of the environment than a sparse depth map and can better understand and reconstruct the 3D world.

[0051] It should be noted that the occupancy grid described in the present disclosure, such as the dynamic occupancy grid or the static occupancy grid described above, is used to represent data in three-dimensional space, such as specifically indicating whether each point in three-dimensional space is occupied by an object. Furthermore, in autonomous driving or assisted driving technology, the occupancy grid can provide a fine-grained representation of the environment and has strong robustness to occlusion. For example, in one example, the occupancy grid can be used to accurately perceive and represent dynamic outdoor traffic scenes.

[0052] In this way, the disclosed solution can use the voxel occupancy grid (that is, the total occupied grid mentioned above) as reference information for depth estimation, effectively solving the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance). In this way, it provides high-precision densified depth data for subsequent scenarios, such as autonomous driving, assisted driving, or subsequent model training, thereby providing strong support for enriching and improving user experience.

[0053] Figure 2 This is a schematic flow chart of a depth map generation method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0054] Further, the method includes at least part of the following contents. Figure 2 As shown, including:

[0055] Step S201: Obtain the target dynamic occupation grid corresponding to the target frame.

[0056] Here, the target dynamic occupation grid is constructed based on the dynamic target body in the original point cloud data of the target frame.

[0057] Furthermore, the target frame is acquired based on radar equipment of the target vehicle;

[0058] It should be noted that, for the relevant contents about the target frame, dynamic target body, etc., please refer to the above description, which will not be repeated here.

[0059] Step S202: Obtain the target static occupation grid corresponding to the target frame.

[0060] Here, the target static occupancy grid is constructed based on the static target body in the original point cloud data of the target frame.

[0061] Step S203: Based on the target statically occupied grid and the target dynamically occupied grid, obtain the total occupied grid corresponding to the target frame in the vehicle coordinate system of the target vehicle.

[0062] Step S204: performing visibility detection on the voxels in the total occupied grid to obtain target detection results corresponding to the voxels.

[0063] Here, voxel is short for volume pixel, which is the smallest unit in three-dimensional space. Similar to pixels in two-dimensional space, voxels can be used to represent data in three-dimensional space. For example, in three-dimensional modeling, voxels are the basic units that constitute volume data, similar to the role of pixels in two-dimensional images.

[0064] Furthermore, the target detection result is used to indicate whether the voxel is visible from the vehicle perspective of the target vehicle; that is, the target detection result corresponding to each voxel indicates whether the voxel is visible from the vehicle perspective.

[0065] For example, in one example, the visibility detection may be performed in the following manner; specifically, the above-mentioned visibility detection of the voxels in the total occupied grid to obtain the target detection result corresponding to the voxels (for example, step S204) may specifically include:

[0066] Step S204 - 1 : Projecting feature points of voxels in the total occupied grid onto the image captured by the image acquisition component of the target vehicle to obtain projection points corresponding to the feature points.

[0067] That is to say, in this example, whether the voxel is visible from the perspective of the vehicle can be determined by projecting the feature points of the voxel onto the image captured by the image acquisition component.

[0068] Here, the feature point of a voxel may specifically refer to a point such as a center point or a vertex of the voxel that can reflect the feature of the voxel. For example, in one example, the feature point of a voxel may specifically include a center point and a plurality of vertices of the voxel.

[0069] Step S204-2: Detect whether the projection point corresponding to the feature point is occupied on the image captured by the image acquisition component, so as to obtain the target detection result based on the result of whether it is occupied.

[0070] That is to say, whether the voxel is visible from the perspective of the vehicle may be determined based on the occupancy status of the projection point corresponding to the feature point of the voxel and whether the voxel is occupied.

[0071] Here, it should be noted that, in practical applications, voxels can also be classified, and then the target detection result of the voxel can be jointly determined according to the type of voxel and whether the projection point corresponding to the feature point of the voxel is occupied on the image captured by the image acquisition component.

[0072] In this way, the disclosed solution provides a specific method for visibility judgment, which is simple, efficient, and highly interpretable, and provides strong support for effectively solving the problem of providing wrong reference information to model estimation when multiple frames are superimposed. At the same time, it effectively avoids the negative impact of invisible voxels on depth estimation.

[0073] Step S205: Based on the target detection result, remove the grids of voxels that are invisible from the vehicle's perspective from the total occupied grid, and update the total occupied grid.

[0074] Step S206: using the total occupied grid (eg, the updated total occupied grid) to obtain a target depth map corresponding to the target frame.

[0075] In a specific example, a depth estimation model may be used to obtain a target depth map. Specifically, the above-described use of the total occupied grid (for example, the updated total occupied grid) to obtain a target depth map corresponding to the target frame may specifically include:

[0076] At least the updated total occupied grid is input into a depth estimation model to obtain a target depth map corresponding to the target frame.

[0077] For instance, in one example, the image captured by the image acquisition device of the target vehicle (the image corresponding to the target frame), the external parameters of the image acquisition device, and the total occupied grid can be input into the depth estimation model to obtain a dense depth map of the target frame, that is, a target depth map.

[0078] In this way, the depth estimation model can be used to quickly obtain the target depth map. At the same time, it effectively solves the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance). In this way, it provides high-precision densified depth data for subsequent scenarios, such as autonomous driving, assisted driving, or subsequent model training, thereby providing strong support for enriching and improving user experience.

[0079] Furthermore, in one example, voxel information can also be added to the total occupied grid, so as to provide data support for further improving the accuracy of depth estimation, and then provide high-precision densified depth data for subsequent scenes, such as autonomous driving, assisted driving or subsequent model training, and then provide strong support for enriching and improving user experience. For example, the above-mentioned use of the total occupied grid to obtain the target depth map corresponding to the target frame can specifically include: adding the voxel information of each voxel in the total occupied grid (for example, the new total occupied grid described above) to the total occupied grid to obtain the total occupied grid with voxel information added; further, using the total occupied grid with voxel information added, the target depth map corresponding to the target frame is obtained. For example, in one example, at least the total occupied grid with voxel information added can be input into the depth estimation model to obtain the target depth map corresponding to the target frame. In this way, the target depth map can be quickly obtained using the depth estimation model.

[0080] For instance, in one example, the voxel information of each voxel in the updated total occupied grid is added to the updated total occupied grid to obtain the total occupied grid with added voxel information, and the total occupied grid with added voxel information, the image captured by the image acquisition device of the target vehicle (the image corresponding to the target frame), the external parameters of the image acquisition device, etc. are input into the depth estimation model to obtain the target depth map corresponding to the target frame.

[0081] In this way, the disclosed solution solves the problem of occluded voxel conflicts by judging visible voxels. In this way, it can effectively solve the problem of providing wrong depth reference information to the model estimation when multiple frames are superimposed, resulting in low accuracy of the estimated depth image. In other words, the disclosed solution provides the model with high accuracy of reference information for depth estimation, which lays the foundation for improving the accuracy of the obtained target depth map. At the same time, it also effectively avoids the negative impact of invisible voxels on depth estimation.

[0082] The following is a detailed description of the specific scheme for obtaining the target dynamically occupied grid:

[0083] In a specific example of the disclosed solution, the target dynamic occupied grid may be obtained in the following manner, that is, the above-mentioned obtaining the target dynamic occupied grid corresponding to the target frame (for example, step S101 or step S201) may specifically include:

[0084] Step S1011: Determine the moving trajectory of the dynamic target.

[0085] Here, the moving trajectory of the dynamic target is determined based on the original point cloud data of N frames, and is used to represent the M1 first frames where the dynamic target is located, and the position of the dynamic target in each first frame.

[0086] Further, the M1 first frames are at least part of the N frames, for example, the M1 first frames are all frames in the N frames that contain the same dynamic target. Further, the target frame is any frame in the M1 first frames.

[0087] Here, it should be noted that the original point cloud data of the N frames are collected based on the radar equipment of the target vehicle.

[0088] Wherein, N is a natural number greater than or equal to 1; further, N is a natural number greater than or equal to 2. Correspondingly, M1 is a natural number greater than or equal to 1 and less than or equal to N; further, M1 is a natural number greater than or equal to 2 and less than or equal to N.

[0089] Step S1012: based on the movement trajectory of the dynamic target body and the original point cloud data of each first frame, obtaining the first initial point cloud data of the dynamic target body in each first frame.

[0090] That is to say, step S1012 can obtain M1 first initial point cloud data corresponding to the dynamic target body, thus providing strong support for the subsequent implementation of point cloud stitching and point cloud completion to obtain point cloud data corresponding to the dynamic target body with dense point cloud data.

[0091] Step S1013: Based on the first initial point cloud data of the dynamic target body in each first frame, a target dynamic occupation grid corresponding to the target frame is obtained.

[0092] In this way, the disclosed scheme provides a scheme for obtaining a grid refinement scheme for dynamically occupying a target, in which frame information containing dynamic target bodies is fully utilized, thereby effectively improving the accuracy of depth estimation of dynamic target bodies. At the same time, it also effectively solves the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance), thereby providing high-precision densified depth data for subsequent scenarios, such as autonomous driving, assisted driving, or subsequent model training, thereby laying the foundation for further enriching and improving user experience.

[0093] For example, in one example, the moving trajectory of the dynamic target may be obtained in the following manner, that is, the above-mentioned determination of the moving trajectory of the dynamic target (for example, the above-mentioned step S1011) may specifically include:

[0094] Step S1011-1: input the original point cloud data of the N frames acquired by the radar equipment of the target vehicle into the target detection model to obtain the target frame of the dynamic target body contained in each frame; the target frame is used to identify the position of the dynamic target body in the frame.

[0095] Here, in this example, the target frame can specifically indicate the category of the dynamic target body, the target center of the dynamic target body, the target orientation, the target size (such as length, width and height) and other related information.

[0096] It should be noted that the dynamic target bodies contained in different frames may be the same or different, and the disclosed solution does not impose any limitation on this.

[0097] It should be further explained that one frame of raw point cloud data may contain one or more dynamic target bodies. In this case, the target detection network can be used to obtain the target frame of each dynamic target body in the frame of raw point cloud data. In other words, the disclosed solution can detect and obtain the target frame of each dynamic target body contained in each frame of raw point cloud data.

[0098] In addition, it should be noted that, in one example, one target frame can be used to identify and locate one dynamic target, and different dynamic targets can be identified and located using different target frames, that is, the target frame corresponds to the dynamic target one by one. Alternatively, in another example, one target frame can identify and locate two or more dynamic targets, which can be determined based on the actual scene situation, and the disclosed solution does not limit this.

[0099] Step S1011 - 2 : Based on the target frame of the dynamic target object contained in each frame, the dynamic target object is tracked between frames to obtain the movement trajectory of the dynamic target object.

[0100] For example, in one example, the moving trajectory of the dynamic target body can be specifically M1 target frames.

[0101] In this way, the disclosed scheme provides a refined scheme for using the target detection model to quickly determine the position of the dynamic target (that is, the target frame) and then quickly obtain the moving trajectory of the dynamic target. This scheme is simple and efficient, and thus provides strong support for the subsequent rapid acquisition of high-precision dense depth maps.

[0102] Further, in an example, the first initial point cloud data of the dynamic target body in each first frame may be obtained in the following manner. Specifically, the above-mentioned first initial point cloud data of the dynamic target body in each first frame based on the movement trajectory of the dynamic target body and the original point cloud data of each first frame (for example, the above-mentioned step S1012) may specifically include:

[0103] Step S1012-1: Based on the feature information of the target frame of the dynamic target in the first frame (for example, the center and orientation of the dynamic target identified by the target frame), the first pose of the target frame in the first frame is obtained. In this way, the first pose of the target frame of the dynamic target in each first frame can be obtained.

[0104] It should be noted that the first frame can be understood as any frame in the N frames that contains the dynamic target body. At this time, the first position of each target frame in the M1 target frames can be obtained.

[0105] Step S1012-2: Based on the first pose of the target frame in the first frame and the size information of the target frame, the point cloud data of the dynamic target body in the first frame is cropped from the original point cloud data of the first frame to obtain the first initial point cloud data of the dynamic target body in each first frame.

[0106] That is to say, in this example, the first initial point cloud data of the dynamic target body in each of the M1 first frames can be obtained, that is, the point cloud data of the same dynamic target body in different frames can be obtained, that is, a total of M1 first initial point cloud data, which provides strong support for subsequent point cloud completion.

[0107] In this way, the disclosed scheme provides a specific scheme for obtaining point cloud data of the same dynamic target in different frames by utilizing the moving trajectory of the dynamic target. This effectively improves the accuracy of depth estimation of the dynamic target, provides strong support for subsequent point cloud completion and point cloud stitching to obtain dense point cloud data, and further provides strong support for effectively solving the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance).

[0108] Further, in one example, the target dynamic occupied grid corresponding to the target frame can be obtained in the following manner, that is, the target dynamic occupied grid corresponding to the target frame is obtained based on the first initial point cloud data of the dynamic target body in each first frame (for example, the above step S1013), including:

[0109] Step S1013 - 1 : converting the first initial point cloud data of the dynamic target in each first frame into a preset coordinate system to obtain the first initial point cloud data of the dynamic target in each first frame in the preset coordinate system.

[0110] It should be noted that in actual applications, when the target vehicle is in a moving state, the radar coordinate systems corresponding to the original point cloud data of different frames are different. In other words, the radar coordinate systems corresponding to the first initial point cloud data in different first frames are different. At this time, in order to improve the accuracy of the completed point cloud data, the first initial point cloud data corresponding to different first frames can be unified into the same coordinate system in this example.

[0111] Here, in a specific example, the preset coordinate system can be specifically a coordinate system constructed based on the dynamic target body, which can be called a dynamic target body coordinate system, or an obstacle coordinate system, thereby providing strong support for the subsequent splicing and completion of point cloud data.

[0112] It should be noted that the process of constructing the preset coordinate system can refer to the process of constructing the vehicle coordinate system above, which will not be repeated here. Moreover, it can be understood that the disclosed solution does not limit the process of constructing the preset coordinate system.

[0113] Step S1013 - 2 : Based on the first initial point cloud data of the dynamic target body in each first frame in the preset coordinate system, obtain the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system.

[0114] For example, in a specific example, the first target point cloud data may be obtained by point cloud stitching and point cloud completion. Specifically, the first initial point cloud data of the dynamic target body in each first frame based on the preset coordinate system described above, to obtain the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system, may specifically include:

[0115] Based on the first initial point cloud data of the dynamic target body in each first frame under the preset coordinate system, point cloud stitching and point cloud completion are performed on the dynamic target body in the target frame under the preset coordinate system to obtain the first target point cloud data of the dynamic target body in the target frame under the preset coordinate system.

[0116] It can be understood that the first target point cloud data obtained at this time is the completed point cloud data. Compared with the initial point cloud data of the dynamic target body, the first target point cloud data is denser. This provides strong support for the subsequent effective solution to the problem of inaccurate depth estimation caused by sparse point cloud (for example, sparse point cloud in the distance).

[0117] Step S1013 - 3 : voxelizing the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system to obtain a target dynamic occupation grid corresponding to the target frame in the preset coordinate system.

[0118] It is understandable that the target frame may contain multiple dynamic target bodies. At this time, for each dynamic target body, the disclosed solution can be used for processing to obtain dense point cloud data after point cloud completion, so as to obtain the target dynamic occupation grid corresponding to the target frame.

[0119] In this way, the disclosed scheme provides a specific scheme for obtaining the first target point cloud data of the dynamic target body in the target frame by utilizing the first initial point cloud data in other frames. For example, it provides a scheme for splicing and completing the point cloud data of the dynamic target body, and then obtaining the refined grid occupied by the target dynamically corresponding to the target frame. This scheme effectively improves the accuracy of depth estimation for the dynamic target body, and at the same time, it also effectively solves the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance), and provides strong support, thereby providing high-precision densified depth data for subsequent scenarios, such as autonomous driving, assisted driving, or subsequent model training, and thus provides strong support for enriching and improving user experience.

[0120] The following is a detailed description of the specific scheme for obtaining the target statically occupied grid:

[0121] In a specific example of the disclosed solution, the static target body can be determined in the following manner, that is, before obtaining the target static occupied grid corresponding to the target frame, the method further includes:

[0122] Determine the semantic category of points in each frame of N frames of original point cloud data;

[0123] Based on the semantic category of each point, a static target body in the target frame is determined; the target frame is any frame in the N frames.

[0124] For example, in one example, based on the semantic category of each point, all target objects except the dynamic target object are uniformly determined as static target objects.

[0125] In this way, the disclosed solution provides a specific solution for obtaining a static target body, which is simple and efficient, and thus provides strong support for the subsequent rapid acquisition of a high-precision densified depth image.

[0126] Further, in a specific example of the disclosed solution, the target static occupied grid may be obtained in the following manner, that is, the above-mentioned obtaining the target static occupied grid corresponding to the target frame (for example, step S102 or step S202) may specifically include:

[0127] Step S1021: Determine the second initial point cloud data in each of the M2 second frames where the static target object is located.

[0128] Further, the M2 second frames are at least part of the N frames, for example, the M2 second frames are all frames containing the same static target object in the N frames. Further, the target frame is any frame in the M2 first frames.

[0129] Here, it should be noted that the original point cloud data of the N frames are collected based on the radar equipment of the target vehicle.

[0130] Wherein, N is a natural number greater than or equal to 1; further, N is a natural number greater than or equal to 2. Correspondingly, M2 is a natural number greater than or equal to 1 and less than or equal to N; further, M2 is a natural number greater than or equal to 2 and less than or equal to N.

[0131] That is to say, step S1021 can obtain M2 second initial point cloud data corresponding to the static target body, thus providing strong support for the subsequent global filling of the point cloud to obtain point cloud data corresponding to the static target body with dense point cloud data.

[0132] Step S1022: Based on the second initial point cloud data of the static target body in each second frame, a target static occupation grid corresponding to the target frame is obtained.

[0133] In this way, the disclosed scheme provides a scheme for obtaining a grid refinement scheme for static occupancy of a target, in which frame information containing static target bodies is fully utilized, thereby effectively improving the accuracy of depth estimation for static target bodies. At the same time, it also effectively solves the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance), thereby providing high-precision densified depth data for subsequent scenarios, such as autonomous driving, assisted driving, or subsequent model training, thereby laying the foundation for further enriching and improving user experience.

[0134] Further, in an example, the target static occupancy grid may be obtained in the following manner. Specifically, the above-described obtaining the target static occupancy grid corresponding to the target frame based on the second initial point cloud data of the static target body in each second frame (for example, the above-described step S1022) may specifically include:

[0135] Step S1022 - 1 : converting the second initial point cloud data of the static target in each second frame into the world coordinate system to obtain the second initial point cloud data of the static target in each second frame in the world coordinate system.

[0136] It should be noted that in actual applications, when the target vehicle is in a moving state, the radar coordinate systems corresponding to the original point cloud data of different frames are different. In other words, the radar coordinate systems corresponding to the second initial point cloud data in different second frames are different. At this time, in order to improve the accuracy of the filled point cloud data, the second initial point cloud data corresponding to different second frames can be unified into the same coordinate system in this example.

[0137] Step S1022-2: Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, obtain the second target point cloud data of the static target body in the target frame in the world coordinate system.

[0138] For example, in a specific example, the point cloud global filling method can be specifically adopted to obtain the second target point cloud data. Specifically, the second initial point cloud data of the static target body in each second frame based on the world coordinate system described above to obtain the second target point cloud data of the static target body in the target frame in the world coordinate system can specifically include:

[0139] Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, the static target body in the target frame in the world coordinate system is globally filled to obtain the second target point cloud data of the static target body in the target frame in the world coordinate system.

[0140] It can be understood that the second target point cloud data obtained at this time is the globally filled point cloud data. Compared with the initial point cloud data of the static target body, the second target point cloud data is more global and denser. This provides strong support for the subsequent effective solution to the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance).

[0141] Step S1022-3: voxelize the second target point cloud data of the static target body in the target frame in the world coordinate system to obtain a target static occupancy grid corresponding to the target frame.

[0142] It is understandable that the target frame may contain multiple static target bodies. At this time, for each static target body, the disclosed solution can be used for processing to obtain dense point cloud data after point cloud filling, so as to obtain the target static occupation grid corresponding to the target frame.

[0143] In this way, the disclosed scheme provides a specific scheme for obtaining the second target point cloud data of the static target body in the target frame by utilizing the second initial point cloud data in other frames. For example, it provides a scheme for globally filling the point cloud data of the static target body and then obtaining the refined grid occupied by the target static body corresponding to the target frame. This scheme effectively improves the accuracy of depth estimation for the static target body. At the same time, it also effectively solves the problem of inaccurate depth estimation caused by sparse point clouds (for example, sparse point clouds in the distance) and provides strong support, thereby providing high-precision densified depth data for subsequent scenarios, such as autonomous driving, assisted driving, or subsequent model training, thereby providing strong support for enriching and improving user experience.

[0144] In a specific example of the disclosed solution, the total occupied grid may be obtained in the following manner. Specifically, the above-described method of obtaining the total occupied grid corresponding to the target frame in the vehicle coordinate system of the target vehicle based on the target static occupied grid and the target dynamic occupied grid (for example, step S103 or step S203) may specifically include:

[0145] Step S1031: unifying the target dynamically occupied grid and the target statically occupied grid into the world coordinate system to obtain the target dynamically occupied grid in the world coordinate system and the target statically occupied grid in the world coordinate system.

[0146] Step S1032: merging the target dynamic occupied grid in the world coordinate system and the target static occupied grid in the world coordinate system to obtain a total occupied grid in the world coordinate system.

[0147] Step S1033: converting the total occupied grid in the world coordinate system to the vehicle coordinate system of the target vehicle to obtain the total occupied grid in the vehicle coordinate system of the target vehicle corresponding to the target frame.

[0148] In this way, the disclosed solution provides a refinement solution for obtaining the total occupied grid, which is simple and efficient, and thus provides strong support for the subsequent rapid acquisition of a high-precision dense depth map.

[0149] The following is a further description of the disclosed solution with reference to specific examples, specifically:

[0150] The disclosed solution provides a solution for obtaining a high-precision dense depth map based on static scene voxel occupancy grid reconstruction technology, dynamic target completion technology, and depth completion network estimation in an autonomous driving scenario. The disclosed solution mainly uses trajectory-based voxel occupancy grid reconstruction technology for static scenes and targets (for example, scenes and targets that do not change or change very little over time), which solves the problem of inaccurate depth estimation caused by sparse distant targets and occluded targets; moreover, the disclosed solution also uses dynamic target completion technology and uses multi-frame information to complete the occupied voxels of the dynamic target body in the current frame, thereby solving the problem of dynamic target (also referred to as dynamic target body) depth estimation; further, the disclosed solution also solves the problem of occluded voxel conflict by judging visible voxels; finally, the processed occupied grid guidance model is used to complete high-precision dense depth estimation, thereby effectively improving the accuracy and robustness of depth estimation in various scenarios.

[0151] The following three parts further describe the disclosed solution in detail:

[0152] Part 1: Voxel Occupancy Mesh Reconstruction

[0153] 1. Framework

[0154] The voxel occupancy grid reconstruction part mainly includes four parts: pre-labeling, static occupancy grid reconstruction, dynamic occupancy grid completion reconstruction and frame-by-frame dynamic and static occupancy grid output.

[0155] (II) Pre-marking

[0156] like Figure 3 As shown, the main steps include:

[0157] Step 1.1: Simultaneous Localization and Mapping (SLAM); for example, based on multiple frames of raw point cloud data obtained from the vehicle (e.g., point cloud data collected by a lidar in the vehicle), images (e.g., images collected by an image acquisition component in the vehicle, such as a camera), and data detected by an inertial measurement unit (IMU), SLAM processing is performed on the multiple frames of raw point cloud data to obtain the pose of each frame of raw point cloud data in the world coordinate system.

[0158] It should be noted that in actual scenarios, since the vehicle is in a moving state, the lidar coordinate systems of different frames of point cloud data may be different. Based on this, in one example, the lidar coordinate system of the first frame of point cloud data in multiple frames of point cloud data can be determined as the world coordinate system, thus providing support for the unification of subsequent coordinate systems.

[0159] Step 1.2: Foreground detection: Input multiple frames of raw point cloud data into the target detection network to obtain the target frame of the obstacle (that is, the dynamic target body mentioned above) in each frame of the raw point cloud data.

[0160] Here, in this example, the target frame of the obstacle can specifically indicate information such as the category of the obstacle, the target center of the obstacle, the target orientation, and the length, width, and height of the obstacle.

[0161] It should be noted that the obstacles contained in different frames may be the same or different, and the disclosed solution does not impose any limitation on this.

[0162] It should be further explained that a frame of raw point cloud data may contain one or more obstacles. At this time, the target frame of each obstacle in the frame of raw point cloud data can be obtained by using the target detection network. In other words, the foreground detection stage can detect the target frame of each obstacle contained in each frame of raw point cloud data.

[0163] It should be noted that, in one example, one target frame can be used to identify and locate one obstacle, and different obstacles can be identified and located using different target frames, that is, the target frame corresponds to the obstacle one by one. Alternatively, in another example, one target frame can identify and locate two or more obstacles, which can be determined based on the actual scene situation, and the disclosed solution does not limit this.

[0164] Step 1.3: foreground association; based on the target frame of the obstacle in each frame of the original point cloud data obtained by foreground detection, the target frame of the obstacle is tracked and associated between frames to obtain the trajectory of each obstacle (which may be called the obstacle trajectory (corresponding to the movement trajectory of the dynamic target body described above). For example, in one example, the obstacle trajectory may be mainly composed of the target frames of the obstacle in different frames, that is, it may include a series of target frames).

[0165] For example, for the sake of ease of description, the frame where the obstacle is located can be called the first frame. Further, assuming that the same obstacle exists in M1 first frames, at this time, the obstacle trajectory can be mainly composed of the target frame identifying the obstacle in each of the M1 first frames, that is, composed of M1 target frames.

[0166] Furthermore, in this example, an identifier can be set for each obstacle track, for example, it can be recorded as track_id, and different obstacle tracks have different track_ids. Furthermore, in order to facilitate subsequent processing, all target frames in the same obstacle track can also be identified by track_id. In this case, all target frames in the same obstacle track have the same track_id.

[0167] Step 1.4: Semantic segmentation: Input multiple frames of point cloud raw data into the semantic segmentation network to obtain the semantic category of each point in each frame of point cloud raw data.

[0168] For example, in one example, the semantic categories may specifically include: ground, curb, green plants or walls, small vehicles, medium vehicles, large vehicles, etc. The disclosed solution does not impose specific restrictions on the semantic categories, and in actual applications, they can be set based on actual scene requirements.

[0169] (III) Dynamically occupied grid completion and reconstruction

[0170] It should be noted that, in the disclosed solution, the obstacles detected in the foreground may be potential dynamic obstacles, and therefore, this part is called dynamic occupied grid completion and reconstruction.

[0171] Furthermore, if Figure 4 As shown, the steps of dynamic occupied grid completion and reconstruction mainly include:

[0172] Step 2.1: Obstacle (i.e., the dynamic target body mentioned above) point cloud acquisition: For each obstacle trajectory, based on the relevant information of the target frame of each frame (i.e., the first frame mentioned above) contained in the obstacle trajectory (e.g., target center, target orientation, etc.), the target frame pose of each frame corresponding to the obstacle trajectory is obtained (corresponding to the first pose mentioned above, i.e., the pose of each target frame contained in the obstacle trajectory is obtained); further, based on the obtained target frame pose and the size of the target frame (e.g., length, width, and height), the initial obstacle point cloud data (i.e., the first initial point cloud data mentioned above) is cropped from the original point cloud data of the frame corresponding to the target frame.

[0173] That is to say, in this step, the obstacle trajectory and the position of the obstacle's target frame can be used to obtain the point cloud data corresponding to the same obstacle in different frames (for example, different first frames), which provides strong support for subsequent point cloud completion.

[0174] Step 2.2: Obstacle point cloud completion: After obtaining the initial obstacle point cloud data contained in each frame of each obstacle trajectory, for each obstacle trajectory, the target frame pose of each frame in the obstacle trajectory is used to convert the initial obstacle point cloud data of the frame from the lidar coordinate system to the preset obstacle coordinate system to obtain the initial obstacle point cloud data in the preset obstacle coordinate system; further, the initial obstacle point cloud data of different frames in the preset obstacle coordinate system are point cloud stitched and completed to obtain the target obstacle point cloud data for each frame (corresponding to the first target point cloud data described above).

[0175] At this time, the target obstacle point cloud data is a relatively complete obstacle point cloud.

[0176] That is to say, in this step, the point cloud data of the same obstacle in different frames (that is, the initial obstacle point cloud data mentioned above) can be used to complete the initial obstacle point cloud data corresponding to any frame containing the same obstacle, so as to obtain relatively complete point cloud data for the obstacle in the frame, that is, the target obstacle point cloud data.

[0177] Step 2.3: Voxelization of obstacle point cloud: Voxelize the completed point cloud data of each target obstacle in the preset obstacle coordinate system to obtain obstacle pixels, and then obtain the obstacle occupation grid corresponding to each frame after completion and reconstruction in the preset obstacle coordinate system (corresponding to the target dynamic occupation grid mentioned above).

[0178] (IV) Static Occupancy Grid Reconstruction

[0179] In the disclosed solution, the static occupied grid is mainly reconstructed relying on the background targets in all frames. Here, the background targets may specifically include: all point clouds except potential dynamic obstacles (for example, obstacles detected in the foreground), noise points, and ignored points, mainly including the ground, curbs, fences, green plants, walls, etc.

[0180] It can be understood that, compared with the obstacles or dynamic targets mentioned above, the background targets in this example can also be called static targets.

[0181] Furthermore, if Figure 5 As shown, the steps of static occupancy grid reconstruction mainly include:

[0182] Step 3.1: static point cloud acquisition (i.e., obtaining the initial point cloud data of the static target or static target body): based on the semantic category of each point in the original point cloud data in all frames, determine the points that do not belong to the dynamic target body. For example, points that do not belong to obstacles and noise, or points that do not need to be ignored, can be unified as static points to obtain a static target (also called a static target body); based on the original point cloud data of each frame, obtain the initial point cloud data of the static target body in each frame (which can be called initial static point cloud data, corresponding to the second initial point cloud data described above)

[0183] That is to say, in this step, the static target contained in each frame can be determined, and then the initial static point cloud data of the static target contained in each frame can be obtained. In other words, the point cloud data corresponding to the same static target in different frames (for example, different second frames) can be obtained, which provides strong support for subsequent point cloud filling.

[0184] For example, for ease of description, the frame where the static target object is located can be called the second frame, and it is assumed that the same static target object exists in M2 second frames. At this time, in this step, for the same static target object, M2 initial static point cloud data can be obtained.

[0185] It should be noted that the static target objects contained in different frames may be the same or different, and the disclosed solution does not impose any limitation on this.

[0186] Step 3.2: Global filling of static point cloud: The initial static point cloud data corresponding to each frame is converted from the lidar coordinate system to the world coordinate system to obtain the initial static point cloud data in the world coordinate system; further, the initial static point cloud data corresponding to different frames in the world coordinate system is used to fill the point cloud to obtain the target static point cloud data of each frame in the world coordinate system (corresponding to the second target point cloud data mentioned above).

[0187] At this point, the target static point cloud data is relatively complete point cloud data.

[0188] That is to say, in this step, the point cloud data of the same static target body in different frames (that is, the initial static point cloud data mentioned above) can be used to perform global point cloud filling on the initial static point cloud data corresponding to any frame containing the same static target body, so as to obtain relatively complete point cloud data for the static target body in the frame, that is, the second target point cloud data.

[0189] Step 3.2: Voxelization of static point cloud: Voxelization is performed on the target static point cloud data obtained after global filling to obtain the static occupancy grid corresponding to each frame in the world coordinate system (corresponding to the target static occupancy grid described above).

[0190] Here, it should be noted that, in one example, point cloud filling generally refers to filling missing parts or blank parts in existing point cloud data. Further, in another example, point cloud completion generally refers to generating a complete 3D model from very sparse or incomplete point cloud data.

[0191] (V) Frame-by-frame dynamic and static grid occupancy output

[0192] Step 4.1: Determine the target static occupied grid within the target range: According to the preset data usage requirements, set the target range of the single-frame dynamic and static occupied grid output, and process the static occupied grid corresponding to each frame to obtain the static occupied grid within the target range under the target perspective (such as Bird's Eye View (BEV)) for each frame.

[0193] Here, it should be noted that the vehicle perspective in this example can be understood as the perspective of an image acquisition component installed on the vehicle, such as a camera.

[0194] Step 4.2: Obtain the dynamic and static occupation grid (also called the total occupation grid, or global occupation grid): For the obstacle occupation grid determined in step 2.3, perform coordinate system transformation on each dynamic target body in the obstacle occupation grid corresponding to each frame, so as to transform it from the preset obstacle coordinate system to the world coordinate system, and fill it into the static occupation grid obtained in step 4.1 corresponding to the frame, so as to obtain the complete dynamic and static occupation network corresponding to the frame, which includes static target bodies and dynamic target bodies, that is, the total occupation grid.

[0195] Step 4.3: Output the dynamic and static occupancy grid: add voxel information such as voxel category and voxel movement speed to the voxels of the dynamic and static occupancy grid, convert the dynamic and static occupancy grid supplemented with the above voxel information from the world coordinate system to the vehicle coordinate system, and output it frame by frame.

[0196] Part 2: Visibility Judgment

[0197] The disclosed solution uses a voxel visibility determination method to remove the grid of voxels that are blocked from the vehicle's perspective (for example, the car's bird's-eye view), thereby solving the problem of providing the model with erroneous depth reference values ​​when multiple frames are superimposed.

[0198] It should be noted that before making a visibility judgment, voxels can be divided into two categories, for example, low class and non-low class. For example, in one example, low class voxels can mainly include: roads and roadside elements, etc., while non-low class voxels can mainly include obstacles, green plants, fences, walls, etc.

[0199] Furthermore, if Figure 6As shown, the steps of visibility judgment mainly include:

[0200] Step 5.1: Project all voxels in the dynamic and static occupied grid corresponding to the current frame onto the image captured by the vehicle's image acquisition component according to the center position of the voxels, and sort the voxels from near to far and from high to low according to projection features such as projection depth and projection height to obtain the sorting result.

[0201] Step 5.2: Determine the voxels to be processed from the sorting results in the order of high and near first, low and short second first; further, project the center point and vertices (e.g., eight vertices) of the voxels to be processed in the dynamic and static occupied grid corresponding to the current frame onto the image captured by the image acquisition component of the vehicle to obtain the projection points of the center point and the vertex, for example, for the center point and eight vertices, nine projection points can be obtained; further, for the two types of voxels, perform the following processing:

[0202] For non-low voxels, if any one of the nine projection points is not occupied after being projected onto the image captured by the image acquisition component (i.e., projected onto the view angle of the image acquisition component), then the voxel (the voxel may refer to the current voxel, i.e., the voxel currently being subjected to visibility judgment) is considered to be visible from the view angle of the corresponding image acquisition component. Otherwise, the voxel may be considered to be blocked from the view angle of the corresponding image acquisition component.

[0203] Here, the convex hull area corresponding to the projection points of the eight vertices on the image captured by the image acquisition component can be used as the projection area of ​​the voxel, and the occupancy state of the projection area is determined to be occupied; further, the elevation of the center position of the voxel is refreshed as the elevation of the projection area, and at the same time, the voxel category of the projection area is refreshed.

[0204] For low voxels, if any one of the nine obtained projection points is not occupied after being projected onto the image captured by the image acquisition component (that is, projected onto the viewing angle of the image acquisition component), or if all the projection points within the viewing angle of the image acquisition component are occupied by other low voxels and the center elevation of the voxel (the voxel may refer to the current voxel, that is, the voxel currently undergoing visibility judgment) is greater than the elevation of the projection point (for example, greater than the elevation of the projection point minus a fixed value (for example, 0.5 m)), then the voxel is considered to be visible in the viewing angle of the corresponding image acquisition component; otherwise, the voxel is considered to be obscured in the viewing angle of the corresponding image acquisition component.

[0205] Here, the convex hull area corresponding to the projection points of the eight vertices on the image captured by the image acquisition component can be used as the projection area of ​​the voxel, and the occupancy state of the projection area is determined to be occupied; further, the elevation of the center position of the voxel is refreshed as the elevation of the projection area, and at the same time, the voxel category of the projection area is refreshed.

[0206] In this way, the new dynamic and static positions corresponding to each frame can occupy the grid.

[0207] Part III: Depth Estimation Model

[0208] The disclosed solution may be based on an open source depth estimation model, for example, the depth estimation model may be specifically a long-short range recurrent updating network (LRRU). Figure 7 As shown, the image captured by the image acquisition device (that is, the camera image), the extrinsic parameters of the image acquisition device (that is, the camera extrinsic parameters), and the camera visible occupancy grid obtained above (that is, the new dynamic and static occupancy grid obtained after processing in the second part above) can be input into the depth estimation model (such as LRRU) to obtain a high-precision dense depth map.

[0209] Here, LRRU is a lightweight depth estimation model. Here, depth estimation may specifically refer to a process of estimating a dense depth map from sparse distance measurements.

[0210] Furthermore, the disclosed solution also provides a depth map generating device, such as Figure 8 As shown, including:

[0211] The grid determination unit 801 is used to obtain a target dynamic occupied grid corresponding to a target frame, and obtain a target static occupied grid corresponding to the target frame; wherein the target dynamic occupied grid is constructed based on a dynamic target body in the original point cloud data of the target frame; the target static occupied grid is constructed based on a static target body in the original point cloud data of the target frame; the target frame is acquired based on radar equipment of the target vehicle;

[0212] A grid processing unit 802 is used to obtain a total occupied grid corresponding to the target frame in a vehicle coordinate system of the target vehicle based on the target static occupied grid and the target dynamic occupied grid;

[0213] The image generating unit 803 is configured to obtain a target depth map corresponding to the target frame by using the total occupied grid.

[0214] In a specific example of the disclosed scheme, the grid processing unit is further used to perform visibility detection on voxels in the total occupied grid to obtain a target detection result corresponding to the voxel before using the total occupied grid to obtain a target depth map corresponding to the target frame; wherein the target detection result is used to indicate whether the voxel is visible from the vehicle perspective of the target vehicle; based on the target detection result, the grid of voxels that are invisible from the vehicle perspective is removed from the total occupied grid, and the total occupied grid is updated.

[0215] In a specific example of the disclosed solution, the image generation unit is specifically used to input at least the updated total occupied grid into a depth estimation model to obtain a target depth map corresponding to the target frame.

[0216] In a specific example of the disclosed solution, the grid processing unit is specifically used to:

[0217] Projecting the feature points of the voxels in the total occupied grid onto the image captured by the image acquisition component of the target vehicle to obtain projection points corresponding to the feature points;

[0218] Whether the projection point corresponding to the feature point is occupied on the image captured by the image acquisition component is detected, so as to obtain the target detection result based on the result of whether it is occupied.

[0219] In a specific example of the disclosed solution, the grid determination unit is specifically used to:

[0220] Determine the moving trajectory of the dynamic target; wherein the moving trajectory is determined based on the original point cloud data of N frames, and is used to represent the M1 first frames where the dynamic target is located, and the position of the dynamic target in each first frame; the M1 first frames are at least part of the N frames, and the target frame is any frame of the M1 first frames; the original point cloud data of the N frames is collected based on the radar equipment of the target vehicle;

[0221] Based on the movement trajectory of the dynamic target and the original point cloud data of each first frame, obtaining the first initial point cloud data of the dynamic target in each first frame;

[0222] Based on the first initial point cloud data of the dynamic target body in each first frame, a target dynamic occupation grid corresponding to the target frame is obtained.

[0223] In a specific example of the disclosed solution, the grid determination unit is specifically used to:

[0224] Converting the first initial point cloud data of the dynamic target in each first frame to a preset coordinate system to obtain the first initial point cloud data of the dynamic target in each first frame in the preset coordinate system;

[0225] Based on the first initial point cloud data of the dynamic target body in each first frame in the preset coordinate system, obtaining the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system;

[0226] The first target point cloud data of the dynamic target body in the target frame in the preset coordinate system is voxelized to obtain the target dynamic occupation grid corresponding to the target frame in the preset coordinate system.

[0227] In a specific example of the present disclosure, the grid determination unit is specifically used to: based on the first initial point cloud data of the dynamic target body in each first frame in the preset coordinate system, perform point cloud stitching and point cloud completion on the dynamic target body in the target frame in the preset coordinate system to obtain the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system.

[0228] In a specific example of the disclosed solution, the grid determination unit is specifically used to:

[0229] Inputting the N frames of original point cloud data acquired by the radar device of the target vehicle into the target detection model to obtain a target frame of the dynamic target body contained in each frame; the target frame is used to identify the position of the dynamic target body in the frame;

[0230] Based on the target frame of the dynamic target object contained in each frame, the dynamic target object is tracked between frames to obtain the movement trajectory of the dynamic target object.

[0231] In a specific example of the disclosed solution, the grid determination unit is specifically used to:

[0232] Based on feature information of a target frame of the dynamic target object in the first frame, obtaining a first position pose of the target frame in the first frame;

[0233] Based on the first pose of the target frame in the first frame and the size information of the target frame, the point cloud data of the dynamic target body in the first frame is cropped from the original point cloud data of the first frame to obtain the first initial point cloud data of the dynamic target body in each first frame.

[0234] In a specific example of the disclosed solution, the grid determination unit is further used to:

[0235] The semantic category of points in each frame of the original point cloud data of N frames is determined to determine the static target body in the target frame; the target frame is any frame in the N frames.

[0236] In a specific example of the disclosed solution, the grid determination unit is specifically used to:

[0237] Determine the second initial point cloud data in each of the M2 second frames where the static target object is located; the M2 second frames are at least part of the N frames; the target frame is any one of the M2 second frames;

[0238] Based on the second initial point cloud data of the static target body in each second frame, a target static occupancy grid corresponding to the target frame is obtained.

[0239] In a specific example of the disclosed solution, the grid determination unit is specifically used to:

[0240] Converting the second initial point cloud data of the static target in each second frame to the world coordinate system to obtain the second initial point cloud data of the static target in each second frame in the world coordinate system;

[0241] Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, second target point cloud data of the static target body in the target frame in the world coordinate system is obtained;

[0242] The second target point cloud data of the static target body in the target frame in the world coordinate system is voxelized to obtain a target static occupancy grid corresponding to the target frame.

[0243] In a specific example of the disclosed solution, the grid determination unit is specifically used to:

[0244] Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, the static target body in the target frame in the world coordinate system is globally filled to obtain the second target point cloud data of the static target body in the target frame in the world coordinate system.

[0245] In a specific example of the disclosed solution, the grid processing unit is specifically used to:

[0246] Unifying the target dynamic occupied grid and the target static occupied grid into the world coordinate system to obtain the target dynamic occupied grid in the world coordinate system and the target static occupied grid in the world coordinate system;

[0247] The target dynamic occupied grid in the world coordinate system and the target static occupied grid in the world coordinate system are merged to obtain the total occupied grid in the world coordinate system;

[0248] The total occupied grid in the world coordinate system is converted to the vehicle coordinate system of the target vehicle to obtain the total occupied grid in the vehicle coordinate system of the target vehicle corresponding to the target frame.

[0249] In a specific example of the disclosed solution, the grid processing unit is further used to: add the voxel information of each voxel in the total occupied grid to the total occupied grid to obtain the total occupied grid with the voxel information added;

[0250] The image generation unit is specifically used to obtain a target depth map corresponding to the target frame by using the total occupied grid to which the voxel information is added.

[0251] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0252] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0253] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0254] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0255] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0256] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0257] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the depth map generation method. For example, in some embodiments, the depth map generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the depth map generation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the depth map generation method in any other appropriate manner (e.g., by means of firmware).

[0258] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0259] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0260] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0261] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0262] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0263] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0264] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0265] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A depth map generation method, comprising: Obtain a target dynamic occupied grid corresponding to a target frame, and obtain a target static occupied grid corresponding to the target frame; wherein the target dynamic occupied grid is constructed based on a dynamic target body in the original point cloud data of the target frame; the target static occupied grid is constructed based on a static target body in the original point cloud data of the target frame; the target frame is acquired based on radar equipment of the target vehicle; Based on the target statically occupied grid and the target dynamically occupied grid, obtaining a total occupied grid corresponding to the target frame in the vehicle coordinate system of the target vehicle; The target depth map corresponding to the target frame is obtained by using the total occupied grid.

2. The method according to claim 1, before obtaining the target depth map corresponding to the target frame using the total occupied grid, the method further comprises: Performing visibility detection on the voxels in the total occupied grid to obtain a target detection result corresponding to the voxel; wherein the target detection result is used to indicate whether the voxel is visible from the vehicle perspective of the target vehicle; Based on the target detection result, grids of voxels that are not visible from the vehicle's perspective are removed from the total occupied grid, and the total occupied grid is updated.

3. The method according to claim 2, wherein: The obtaining a target depth map corresponding to the target frame by using the total occupied grid includes: At least the updated total occupied grid is input into a depth estimation model to obtain a target depth map corresponding to the target frame.

4. The method according to claim 2, wherein: The performing visibility detection on the voxels in the total occupied grid to obtain the target detection result corresponding to the voxels includes: Projecting the feature points of the voxels in the total occupied grid onto the image captured by the image acquisition component of the target vehicle to obtain projection points corresponding to the feature points; Whether the projection point corresponding to the feature point is occupied on the image captured by the image acquisition component is detected, so as to obtain the target detection result based on the result of whether it is occupied.

5. The method according to any one of claims 1 to 4, wherein: The step of obtaining a target dynamic occupation grid corresponding to the target frame includes: Determine the moving trajectory of the dynamic target; wherein the moving trajectory is determined based on the original point cloud data of N frames, and is used to represent the M1 first frames where the dynamic target is located, and the position of the dynamic target in each first frame; the M1 first frames are at least part of the N frames, and the target frame is any frame of the M1 first frames; the original point cloud data of the N frames is collected based on the radar equipment of the target vehicle; Based on the movement trajectory of the dynamic target and the original point cloud data of each first frame, obtaining the first initial point cloud data of the dynamic target in each first frame; Based on the first initial point cloud data of the dynamic target body in each first frame, a target dynamic occupation grid corresponding to the target frame is obtained.

6. The method according to claim 5, wherein: The step of obtaining a target dynamic occupied grid corresponding to a target frame based on the first initial point cloud data of the dynamic target body in each first frame includes: Converting the first initial point cloud data of the dynamic target in each first frame to a preset coordinate system to obtain the first initial point cloud data of the dynamic target in each first frame in the preset coordinate system; Based on the first initial point cloud data of the dynamic target body in each first frame in the preset coordinate system, obtaining the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system; The first target point cloud data of the dynamic target body in the target frame in the preset coordinate system is voxelized to obtain the target dynamic occupation grid corresponding to the target frame in the preset coordinate system.

7. The method according to claim 6, wherein: The step of obtaining the first target point cloud data of the dynamic target in the target frame in the preset coordinate system based on the first initial point cloud data of the dynamic target in each first frame in the preset coordinate system includes: Based on the first initial point cloud data of the dynamic target body in each first frame under the preset coordinate system, point cloud stitching and point cloud completion are performed on the dynamic target body in the target frame under the preset coordinate system to obtain the first target point cloud data of the dynamic target body in the target frame under the preset coordinate system.

8. The method according to claim 5, wherein: The determining the moving trajectory of the dynamic target body includes: Inputting the N frames of original point cloud data acquired by the radar device of the target vehicle into the target detection model to obtain a target frame of the dynamic target body contained in each frame; the target frame is used to identify the position of the dynamic target body in the frame; Based on the target frame of the dynamic target object contained in each frame, the dynamic target object is tracked between frames to obtain the movement trajectory of the dynamic target object.

9. The method according to claim 8, wherein: The step of obtaining the first initial point cloud data of the dynamic target in each first frame based on the moving trajectory of the dynamic target and the original point cloud data of each first frame includes: Based on feature information of a target frame of the dynamic target object in the first frame, obtaining a first position pose of the target frame in the first frame; Based on the first pose of the target frame in the first frame and the size information of the target frame, the point cloud data of the dynamic target body in the first frame is cropped from the original point cloud data of the first frame to obtain the first initial point cloud data of the dynamic target body in each first frame.

10. The method according to any one of claims 1 to 9, further comprising: Determine the semantic category of points in each frame of the N frames of original point cloud data to determine the static target body in the target frame; The target frame is any frame among the N frames.

11. The method according to claim 10, wherein: The obtaining of the target static occupied grid corresponding to the target frame includes: Determine the second initial point cloud data in each of the M2 second frames where the static target object is located; the M2 second frames are at least part of the N frames; the target frame is any one of the M2 second frames; Based on the second initial point cloud data of the static target body in each second frame, a target static occupancy grid corresponding to the target frame is obtained.

12. The method according to claim 11, wherein: The step of obtaining a target static occupancy grid corresponding to the target frame based on the second initial point cloud data of the static target body in each second frame includes: Converting the second initial point cloud data of the static target body in each second frame to the world coordinate system to obtain the second initial point cloud data of the static target body in each second frame in the world coordinate system; Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, obtaining the second target point cloud data of the static target body in the target frame in the world coordinate system; The second target point cloud data of the static target body in the target frame in the world coordinate system is voxelized to obtain a target static occupancy grid corresponding to the target frame.

13. The method according to claim 12, wherein: The step of obtaining the second target point cloud data of the static target body in the target frame in the world coordinate system based on the second initial point cloud data of the static target body in each second frame in the world coordinate system comprises: Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, the static target body in the target frame in the world coordinate system is globally filled to obtain the second target point cloud data of the static target body in the target frame in the world coordinate system.

14. The method according to any one of claims 1 to 4, wherein: The obtaining, based on the target statically occupied grid and the target dynamically occupied grid, a total occupied grid corresponding to the target frame in the vehicle coordinate system of the target vehicle comprises: Unifying the target dynamic occupied grid and the target static occupied grid into the world coordinate system to obtain the target dynamic occupied grid in the world coordinate system and the target static occupied grid in the world coordinate system; The target dynamic occupied grid in the world coordinate system and the target static occupied grid in the world coordinate system are merged to obtain the total occupied grid in the world coordinate system; The total occupied grid in the world coordinate system is converted to the vehicle coordinate system of the target vehicle to obtain the total occupied grid in the vehicle coordinate system of the target vehicle corresponding to the target frame.

15. The method according to claim 14, wherein: The obtaining a target depth map corresponding to the target frame by using the total occupied grid includes: Adding the voxel information of each voxel in the total occupied grid to the total occupied grid to obtain the total occupied grid with the voxel information added; The target depth map corresponding to the target frame is obtained by using the total occupied grid with added voxel information.

16. A depth map generating device, comprising: A grid determination unit is used to obtain a target dynamic occupied grid corresponding to a target frame, and obtain a target static occupied grid corresponding to the target frame; wherein the target dynamic occupied grid is constructed based on a dynamic target body in the original point cloud data of the target frame; the target static occupied grid is constructed based on a static target body in the original point cloud data of the target frame; the target frame is acquired based on radar equipment of a target vehicle; A grid processing unit, configured to obtain a total occupied grid corresponding to the target frame in a vehicle coordinate system of the target vehicle based on the target static occupied grid and the target dynamic occupied grid; An image generating unit is used to obtain a target depth map corresponding to the target frame by using the total occupied grid.

17. The device according to claim 16, wherein: The grid processing unit is further used to perform visibility detection on voxels in the total occupied grid before using the total occupied grid to obtain the target depth map corresponding to the target frame, so as to obtain a target detection result corresponding to the voxel; wherein the target detection result is used to indicate whether the voxel is visible from the vehicle perspective of the target vehicle; based on the target detection result, the grid of voxels that are not visible from the vehicle perspective is removed from the total occupied grid, and the total occupied grid is updated.

18. The device according to claim 17, wherein: The image generation unit is specifically configured to at least input the updated total occupied grid into a depth estimation model to obtain a target depth map corresponding to the target frame.

19. The device according to claim 17, wherein: The grid processing unit is specifically used for: Projecting the feature points of the voxels in the total occupied grid onto the image captured by the image acquisition component of the target vehicle to obtain projection points corresponding to the feature points; Whether the projection point corresponding to the feature point is occupied on the image captured by the image acquisition component is detected, so as to obtain the target detection result based on the result of whether it is occupied.

20. The device according to any one of claims 16 to 19, wherein: The grid determination unit is specifically used for: Determine the moving trajectory of the dynamic target; wherein the moving trajectory is determined based on the original point cloud data of N frames, and is used to represent the M1 first frames where the dynamic target is located, and the position of the dynamic target in each first frame; the M1 first frames are at least part of the N frames, and the target frame is any frame of the M1 first frames; the original point cloud data of the N frames is collected based on the radar equipment of the target vehicle; Based on the movement trajectory of the dynamic target and the original point cloud data of each first frame, obtaining the first initial point cloud data of the dynamic target in each first frame; Based on the first initial point cloud data of the dynamic target body in each first frame, a target dynamic occupation grid corresponding to the target frame is obtained.

21. The device according to claim 20, wherein: The grid determination unit is specifically used for: Converting the first initial point cloud data of the dynamic target in each first frame to a preset coordinate system to obtain the first initial point cloud data of the dynamic target in each first frame in the preset coordinate system; Based on the first initial point cloud data of the dynamic target body in each first frame in the preset coordinate system, obtaining the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system; The first target point cloud data of the dynamic target body in the target frame in the preset coordinate system is voxelized to obtain the target dynamic occupation grid corresponding to the target frame in the preset coordinate system.

22. The device according to claim 21, wherein The grid determination unit is specifically used to perform point cloud stitching and point cloud completion on the dynamic target body in the target frame in the preset coordinate system based on the first initial point cloud data of the dynamic target body in each first frame in the preset coordinate system, so as to obtain the first target point cloud data of the dynamic target body in the target frame in the preset coordinate system.

23. The device according to claim 20, wherein: The grid determination unit is specifically used for: Inputting the N frames of original point cloud data acquired by the radar device of the target vehicle into the target detection model to obtain a target frame of the dynamic target body contained in each frame; the target frame is used to identify the position of the dynamic target body in the frame; Based on the target frame of the dynamic target object contained in each frame, the dynamic target object is tracked between frames to obtain the movement trajectory of the dynamic target object.

24. The device according to claim 23, wherein: The grid determination unit is specifically used for: Based on feature information of a target frame of the dynamic target object in the first frame, obtaining a first position pose of the target frame in the first frame; Based on the first pose of the target frame in the first frame and the size information of the target frame, the point cloud data of the dynamic target body in the first frame is cropped from the original point cloud data of the first frame to obtain the first initial point cloud data of the dynamic target body in each first frame.

25. The device according to any one of claims 16 to 24, wherein: The grid determination unit is further used for: The semantic category of points in each frame of the original point cloud data of N frames is determined to determine the static target body in the target frame; the target frame is any frame in the N frames.

26. The device according to claim 25, wherein The grid determination unit is specifically used for: Determine the second initial point cloud data in each of the M2 second frames where the static target is located; the M2 second frames are at least part of the N frames; The target frame is any one of the M2 second frames; Based on the second initial point cloud data of the static target body in each second frame, a target static occupancy grid corresponding to the target frame is obtained.

27. The device according to claim 26, wherein: The grid determination unit is specifically used for: Converting the second initial point cloud data of the static target body in each second frame to the world coordinate system to obtain the second initial point cloud data of the static target body in each second frame in the world coordinate system; Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, obtaining the second target point cloud data of the static target body in the target frame in the world coordinate system; The second target point cloud data of the static target body in the target frame in the world coordinate system is voxelized to obtain a target static occupancy grid corresponding to the target frame.

28. The device according to claim 27, wherein The grid determination unit is specifically used for: Based on the second initial point cloud data of the static target body in each second frame in the world coordinate system, the static target body in the target frame in the world coordinate system is globally filled to obtain the second target point cloud data of the static target body in the target frame in the world coordinate system.

29. The device according to any one of claims 16 to 19, wherein: The grid processing unit is specifically used for: Unifying the target dynamic occupied grid and the target static occupied grid into the world coordinate system to obtain the target dynamic occupied grid in the world coordinate system and the target static occupied grid in the world coordinate system; The target dynamic occupied grid in the world coordinate system and the target static occupied grid in the world coordinate system are merged to obtain the total occupied grid in the world coordinate system; The total occupied grid in the world coordinate system is converted to the vehicle coordinate system of the target vehicle to obtain the total occupied grid in the vehicle coordinate system of the target vehicle corresponding to the target frame.

30. The device according to claim 29, wherein: The grid processing unit is further used to: add the voxel information of each voxel in the total occupied grid to the total occupied grid to obtain the total occupied grid with the voxel information added; The image generation unit is specifically used to obtain a target depth map corresponding to the target frame by using the total occupied grid to which the voxel information is added.

31. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

33. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images