Method and device for generating point cloud of three-dimensional object, computer equipment, readable storage medium and program product

Abstract

The invention relates to a method and device for generating a point cloud of a three-dimensional object, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an acquired to-be-processed image; obtaining a two-dimensional detection frame corresponding to each three-dimensional object based on the to-be-processed image; performing target segmentation on the to-be-processed image based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object; and obtaining dense depth information of the three-dimensional objects based on the two-dimensional segmentation result and the three-dimensional point clouds corresponding to the to-be-processed image so as to generate dense point clouds of the three-dimensional objects. By adopting the method, the real appearance of the object can be described, and display details are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating a point cloud of a three-dimensional object. Background Art

[0002] With the development of artificial intelligence technology, 3D reconstruction and rendering technology has emerged. By perceiving objects in the environment, reconstructing the 3D model of the object, and representing and rendering the 3D model in the form of point cloud, the real scene can be digitized and visualized. For example, in autonomous driving technology, the vehicle can perceive and display the vehicle and perceived traffic participants (including vulnerable traffic participants such as hot vehicles, pedestrians, non-motor vehicles, as well as obstacles and road edges, etc.) on the interactive screen, making it convenient for the driver or passenger to understand the current perception and working status of the intelligent driving system in a visualized way.

[0003] Most of the related technologies first conduct targeted training and recognition of the perceived objects based on the perception whitelist solution, and then match the pre-made object model for display. The display precision and properties of the recognized objects are often unable to be characterized. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for generating point clouds of three-dimensional objects, which can depict the real appearance of objects and improve display details in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a method for generating a point cloud of a three-dimensional object, the method comprising:

[0006] Acquire the collected image to be processed;

[0007] Acquire a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed;

[0008] Performing target segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each of the three-dimensional objects;

[0009] Based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, dense depth information of the three-dimensional object is obtained to generate a dense point cloud of each three-dimensional object.

[0010] In one of the embodiments, the acquiring of the two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed includes any one of the following methods:

[0011] Acquire each two-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed; or

[0012] Acquire each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, identify the vertices of each of the three-dimensional detection frames, project each corresponding vertex onto the image to be processed based on the camera parameters for collecting the image to be processed to obtain each first projection point, and obtain a two-dimensional detection frame corresponding to the corresponding three-dimensional object based on each of the first projection points; or

[0013] Acquire each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, select a target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, project the target point cloud onto the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain each second projection point, and obtain a corresponding two-dimensional detection frame corresponding to the three-dimensional object based on each second projection point.

[0014] In one of the embodiments, the object segmentation of the image to be processed based on the two-dimensional detection frame to obtain the two-dimensional segmentation result of each three-dimensional object includes any one of the following methods:

[0015] Inputting the two-dimensional detection frame and the image to be processed into a pre-trained segmentation model to obtain a two-dimensional segmentation result of each three-dimensional object; or

[0016] Acquire each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, select the target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, input the target point cloud, the two-dimensional detection frame and the image to be processed into the pre-trained segmentation model, and obtain the two-dimensional segmentation result of each three-dimensional object.

[0017] In one of the embodiments, the dense depth information is obtained by any of the following methods:

[0018] Acquire a three-dimensional point cloud corresponding to the image to be processed containing dense depth information output by a sensor, and extract the dense depth information of the three-dimensional object from the three-dimensional point cloud based on the two-dimensional segmentation result; or

[0019] Acquire each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, select a target point cloud belonging to the three-dimensional detection frame from the three-dimensional point cloud corresponding to the image to be processed, project the target point cloud onto the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain a sparse depth map, set the depth values ​​of the pixel points of the projection points of the non-target point cloud in the sparse depth map to the target values, and update the depth values ​​of the pixel points of the projection points of the non-target point cloud within the range of the two-dimensional segmentation result by interpolation to obtain dense depth information.

[0020] In one embodiment, the method further comprises:

[0021] Based on the camera parameters of the multiple cameras that collect the images to be processed, projecting the dense point cloud of the three-dimensional object generated based on the images to be processed collected by each of the cameras into a target coordinate system;

[0022] The three-dimensional objects are rendered based on the dense point clouds of the three-dimensional objects in the target coordinate system.

[0023] In one of the embodiments, after performing target segmentation on the image to be processed based on the two-dimensional detection frame to obtain the two-dimensional segmentation result of each three-dimensional object, the method further includes:

[0024] When the two-dimensional segmentation result indicates that the three-dimensional object is occluded, the two-dimensional segmentation result is input into a pre-trained completion model to obtain a two-dimensional segmentation result including complete appearance information.

[0025] In a second aspect, the present application further provides a device for generating a point cloud of a three-dimensional object, the device comprising:

[0026] An image acquisition module, used to acquire the collected image to be processed;

[0027] A two-dimensional detection frame acquisition module, used to acquire a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed;

[0028] A two-dimensional segmentation module, used to perform target segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object;

[0029] The dense point cloud generation module is used to obtain dense depth information of the three-dimensional object based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, so as to generate a dense point cloud of each three-dimensional object.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any one of the above embodiments when executing the computer program.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0032] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in any one of the above embodiments.

[0033] The above-mentioned method, apparatus, computer equipment, computer-readable storage medium and computer program product for generating a point cloud of a three-dimensional object obtain a collected image to be processed; obtain a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed; perform target segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object; based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, obtain dense depth information of the three-dimensional object to generate a dense point cloud of each three-dimensional object, so that the points of dense depth information are limited by the two-dimensional segmentation result to obtain a dense point cloud of the object, which can avoid the problem of inaccurate depth at the edge caused by densification of the entire image, and can portray the real appearance and details of the object, and improve the precision and accuracy of the three-dimensional object representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A diagram of an application environment of a method for generating a point cloud of a three-dimensional object in one embodiment;

[0036] Figure 2 is a flow chart of a method for generating a point cloud of a three-dimensional object in one embodiment;

[0037] Figure 3 is a flow chart of a method for generating a point cloud of a three-dimensional object in another embodiment;

[0038] Figure 4 is a structural block diagram of an apparatus for generating a point cloud of a three-dimensional object in one embodiment;

[0039] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It is understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various data, but these data are not limited by these terms. These terms are only used to distinguish one data from another data. For example, without departing from the scope of the present application, the projection point can be referred to as the first projection point or the second projection point. Both the first projection point and the second projection point are projection points, but they are not the same type of projection points.

[0041] The method for generating a point cloud of a three-dimensional object provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the image acquisition device is used to acquire the image to be processed, the depth information acquisition device is used to acquire the depth information, the perception system is respectively connected to the image acquisition device and the depth information acquisition device in communication, and the perception system is used to obtain the two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed, and to perform target segmentation on the image to be processed based on the two-dimensional detection frame to obtain the two-dimensional segmentation result of each three-dimensional object. The perception system is also used to obtain the dense depth information of the three-dimensional object based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, so as to generate a dense point cloud of each three-dimensional object.

[0042] Among them, the image acquisition device can be a multi-eye camera. The depth information acquisition device can be a radar device, such as a laser radar device or a millimeter wave radar device, which can collect sparse point clouds of three-dimensional objects, and sparse depth information can be obtained based on the sparse point clouds. The depth information acquisition device can also be a sensor, which can directly collect dense depth information of three-dimensional objects. The depth information acquisition device can also be a multi-eye camera, so that dense depth information can be estimated based on the image to be processed collected by the multi-eye camera. This application does not make specific limitations on image acquisition devices and depth information acquisition devices, and any device that can collect images and depth information can be used.

[0043] The perception system may be an autonomous driving perception system, which has the ability to detect a three-dimensional bounding box of an object. The specific process of the perception system detecting the three-dimensional bounding box of an object is not specifically limited here. The three-dimensional bounding box output by the perception system is used to obtain a corresponding two-dimensional bounding box based on the three-dimensional bounding box of the three-dimensional object. In other optional embodiments, the perception system may also directly perceive the two-dimensional detection box of the three-dimensional object, which is not specifically limited here.

[0044] The above method for generating a point cloud of a three-dimensional object limits the points of dense depth information through the two-dimensional segmentation results to obtain a dense point cloud of the object. This can avoid the problem of inaccurate depth at the edges caused by densification of the entire image, and can depict the true appearance and details of the object, thereby improving the refinement and accuracy of the three-dimensional object representation.

[0045] In an exemplary embodiment, Figure 2 As shown, a method for generating a point cloud of a three-dimensional object is provided, and the method is applied to Figure 1 The perception system in the example is used to illustrate, including the following steps S202 to S208. Among them:

[0046] S202: Acquire the collected image to be processed.

[0047] The image to be processed is an image captured by multiple cameras, such as an RGB image. Each camera perspective can correspond to at least one image to be processed, so that the corresponding image to be processed under each perspective can be processed in parallel to obtain a dense point cloud of the three-dimensional object under the corresponding perspective, and the dense point clouds of the three-dimensional object under each perspective are spliced ​​to obtain a dense point cloud of the three-dimensional object for easy rendering.

[0048] S204: Acquire a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed.

[0049] The perception system can process the image to be processed to obtain a two-dimensional detection frame of each three-dimensional object in the image to be processed.

[0050] The three-dimensional object can be any three-dimensional object in the scene. It should be noted that the "object" here refers to any tangible object in the three-dimensional scene, which may include a person, an animal, or an object. For example, in a driving scene, the three-dimensional object may include a vehicle, a pedestrian, an obstacle, etc., without specific limitation.

[0051] The two-dimensional detection frame is a two-dimensional bounding box of the three-dimensional object in the image to be processed, such as a minimum two-dimensional bounding box. Each three-dimensional object in the image to be processed corresponds to a two-dimensional bounding box. The two-dimensional bounding box includes at least four boundaries, and the three-dimensional object is located in the two-dimensional bounding box. Optionally, the two-dimensional bounding box can be a square or a rectangle, which is not specifically limited here.

[0052] S206: Performing object segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object.

[0053] The two-dimensional segmentation result is to segment the three-dimensional object from the image to be processed. Optionally, in the present application, the two-dimensional detection frame and the image to be processed are input into a pre-trained segmentation model to obtain the two-dimensional segmentation result of each three-dimensional object. Optionally, the pre-trained segmentation model can segment the image to be processed based on the two-dimensional detection frame to obtain the area to be processed, and then perform target segmentation on the area to be processed to obtain the two-dimensional segmentation result of the three-dimensional object.

[0054] S208: Based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, dense depth information of the three-dimensional object is obtained as a dense point cloud of the three-dimensional object.

[0055] The perception system can determine the points involved in dense calculation from the image to be processed based on the two-dimensional segmentation result, that is, determine the pixels involved in dense calculation based on the two-dimensional segmentation result, so as to obtain the dense depth information corresponding to these pixels, so as to obtain the depth map corresponding to the image to be processed, that is, the coordinates of the pixels in the image to be processed are used as the coordinates of the xy plane, and the dense depth information of these pixels are used as the z-axis coordinates, so as to obtain the depth map. The depth map is subsequently converted into a dense point cloud of a three-dimensional object. The depth value of each pixel in the depth map here represents the distance from the object to the xy plane of the camera, that is, the depth map can provide information about the distance between a certain point in the scene and the camera, so that the points in the depth map are projected onto the three-dimensional plane based on the parameter information of the camera, and a dense point cloud can be obtained. It should be noted that a technician in this field can choose the method of converting the depth map into a point cloud based on needs, and no specific limitation is made here.

[0056] In one of the optional embodiments, the method further includes: displaying each three-dimensional object based on the dense point cloud.

[0057] Since dense point clouds include more information about three-dimensional objects, displaying three-dimensional objects based on dense point clouds can replicate the state and appearance of objects in real scenes, thereby improving the fineness and properties of the display of three-dimensional objects.

[0058] The above-mentioned method for generating a point cloud of a three-dimensional object obtains a collected image to be processed; obtains a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed; performs target segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object; based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, obtains dense depth information of the three-dimensional object to generate a dense point cloud of each three-dimensional object, thereby limiting the points of dense depth information through the two-dimensional segmentation result to obtain a dense point cloud of the object, which can avoid the problem of inaccurate depth at the edge caused by densification of the entire image, and can portray the real appearance and details of the object, thereby improving the fineness and accuracy of the three-dimensional object representation.

[0059] In one of the optional embodiments, obtaining a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed includes any one of the following methods: obtaining each two-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed; or obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, identifying the vertices of each three-dimensional detection frame, projecting each corresponding vertex to the image to be processed based on the camera parameters of the camera that captures the image to be processed to obtain each first projection point, and obtaining the two-dimensional detection frame corresponding to the corresponding three-dimensional object based on each first projection point; or obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, selecting a target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, projecting the target point cloud to the image to be processed based on the camera parameters of the camera that captures the image to be processed to obtain each second projection point, and obtaining the two-dimensional detection frame corresponding to the corresponding three-dimensional object based on each second projection point.

[0060] The camera parameters include camera intrinsic parameters and extrinsic parameters. The camera intrinsic parameters describe the process of the camera projecting points in the three-dimensional world onto the two-dimensional image plane, mainly including focal length, principal point coordinates, and distortion coefficients. The camera extrinsic parameters describe the position and orientation of the camera relative to the world coordinate system, mainly including rotation matrices and translation vectors.

[0061] The two-dimensional detection frame may be obtained in the following three ways:

[0062] The first method: when the perception system is able to detect the two-dimensional detection frame of the three-dimensional object, the two-dimensional detection frame of the three-dimensional object is directly obtained by using the existing functions of the perception system.

[0063] When the perception system is able to detect a 3D bounding box of a 3D object, a 2D detection box of the 3D object is obtained based on the 3D bounding box. Specifically, there are two methods:

[0064] The second method is to obtain the three-dimensional detection frames of each three-dimensional object obtained by the perception system based on the image to be processed, and identify the vertices of each three-dimensional detection frame. Generally, the three-dimensional detection frame is a rectangular detection frame or a cube detection frame, and such a three-dimensional detection frame includes 8 vertices. For each three-dimensional object, the camera parameters of the camera that collects the image to be processed can be obtained. One point that needs to be explained is that the camera that collects the image to be processed needs to be a camera that can observe the three-dimensional object. The camera that can observe the three-dimensional object means that at least a part of the three-dimensional object can be observed from the camera's perspective. In this way, based on the obtained camera parameters, the vertices of the three-dimensional detection frame corresponding to the three-dimensional object can be projected into the corresponding image to be processed to obtain each first projection point, and finally, based on each first projection point, the corresponding two-dimensional detection frame of the corresponding three-dimensional object can be obtained. For example, the first projection points can be connected in sequence to obtain the two-dimensional detection frame corresponding to the three-dimensional object.

[0065] The third method: obtain each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, select the target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, the sparse point cloud can be acquired by radar equipment, take the three-dimensional detection frame as the boundary, for example, the position of the boundary of the three-dimensional detection frame can be acquired, and then based on the position of the boundary, the target point cloud located in the three-dimensional detection frame is acquired from the sparse point cloud, the target point cloud is also the point cloud of the three-dimensional object. For each three-dimensional object, the camera parameters of the camera that acquires the image to be processed can be acquired, wherein one point to be explained is that the camera that acquires the image to be processed needs to be able to observe the three-dimensional object. The camera that can observe the three-dimensional object means that at least a part of the three-dimensional object can be observed from the camera's perspective. In this way, based on the acquired camera parameters, the target point cloud can be projected into the corresponding image to be processed to obtain each second projection point, so that the two-dimensional projection frame of the three-dimensional object in the image to be processed can be obtained based on each second projection point. Among them, those skilled in the art can choose any point cloud to be projected into a two-dimensional plane to project the target point cloud into the image to be processed, and no specific limitation is made here.

[0066] In the above embodiments, the two-dimensional detection frame can be obtained in a variety of ways. Among them, the method of obtaining the two-dimensional detection frame by using the three-dimensional detection frame of the three-dimensional object and the sparse point cloud collected by the radar device can make full use of the sparse point cloud collected by the radar device, thereby making the two-dimensional detection frame more accurate.

[0067] In one of the optional embodiments, target segmentation is performed on the image to be processed based on the two-dimensional detection frame to obtain the two-dimensional segmentation results of each three-dimensional object, including any of the following methods: inputting the two-dimensional detection frame and the image to be processed into a pre-trained segmentation model to obtain the two-dimensional segmentation results of each three-dimensional object; or obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, selecting the target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, and inputting the target point cloud, the two-dimensional detection frame and the image to be processed into the pre-trained segmentation model to obtain the two-dimensional segmentation results of each three-dimensional object. The target point cloud is used to limit the two-dimensional segmentation results of the three-dimensional object.

[0068] There are two ways to obtain the two-dimensional segmentation results, which are analyzed as follows:

[0069] The first method is to use the two-dimensional detection frame and the image to be processed as the input of the pre-trained segmentation model, and obtain the two-dimensional segmentation result of each three-dimensional object through the pre-trained segmentation model. The pre-trained segmentation model can be a deep learning network with general object segmentation capabilities, such as SAM2, etc., and no specific restrictions are made on the pre-trained segmentation model.

[0070] The second method is to use the two-dimensional detection frame, the sparse point cloud of each three-dimensional object and the image to be processed as the input of the pre-trained segmentation model, and obtain the two-dimensional segmentation result of each three-dimensional object through the pre-trained segmentation model. The sparse point cloud of the three-dimensional object can be selected from the sparse point cloud corresponding to the image to be processed based on the three-dimensional detection frame of each three-dimensional object. Specifically, the sparse point cloud can be acquired by radar equipment, and the three-dimensional detection frame is used as the boundary. For example, the position of the boundary of the three-dimensional detection frame can be obtained, and then the target point cloud located in the three-dimensional detection frame is obtained from the sparse point cloud corresponding to the image to be processed based on the position of the boundary. The target point cloud is also the point cloud of the three-dimensional object. The target point cloud of the three-dimensional object is used as a supplementary input of the pre-trained segmentation model, which can correct the two-dimensional segmentation result obtained based on the two-dimensional detection frame and the image to be processed, and can achieve better results for objects with large areas or complex structures.

[0071] In the above embodiment, by introducing the segmentation model, a two-dimensional segmentation result can be automatically obtained, thereby improving processing efficiency.

[0072] In one of the optional embodiments, after performing target segmentation on the image to be processed based on the two-dimensional detection frame to obtain the two-dimensional segmentation results of each three-dimensional object, it also includes: when the two-dimensional segmentation result indicates that the three-dimensional object is occluded, inputting the two-dimensional segmentation result into a pre-trained completion model to obtain a two-dimensional segmentation result that includes complete appearance information.

[0073] Among them, for objects with occlusion, a pre-trained completion model can be used to complete the image of the two-dimensional segmentation result to provide more complete object appearance information, so that the two-dimensional segmentation result is more accurate.

[0074] In one of the optional embodiments, the dense depth information is obtained by any of the following methods: obtaining a three-dimensional point cloud corresponding to the image to be processed containing dense depth information output by the sensor, and extracting dense depth information of the three-dimensional object from the three-dimensional point cloud based on the two-dimensional segmentation result; or obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, selecting a target point cloud belonging to the three-dimensional detection frame from the three-dimensional point cloud corresponding to the image to be processed, projecting the target point cloud onto the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain a sparse depth map, setting the depth values ​​of the pixel points of the projection points of the non-target point cloud in the sparse depth map to the target value, and updating the depth values ​​of the pixel points of the projection points of the non-target point cloud within the range of the two-dimensional segmentation result by interpolation to obtain dense depth information.

[0075] The method for obtaining dense depth information may include but is not limited to the following two methods:

[0076] The first method is to obtain the 3D point cloud corresponding to the image to be processed containing dense depth information output by the sensor, and extract the dense depth information of the 3D object from the 3D point cloud based on the 2D segmentation result. The dense depth information estimated based on the image to be processed acquired by the camera is obtained. For example, when the camera is a multi-eye camera, the dense depth information of the 3D point can be calculated based on the 2D projection points of the 3D points observed by the cameras at different positions, using trigonometric relationships, etc., so that the depth information of each pixel can be calculated based on the image to be processed to obtain the dense depth information.

[0077] The second method: using the two-dimensional segmentation result of the three-dimensional object to obtain dense depth information, specifically including: first obtaining a sparse depth map based on the three-dimensional detection frame, specifically including: obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, selecting the target point cloud belonging to the three-dimensional detection frame from the three-dimensional point cloud corresponding to the image to be processed, the sparse point cloud can be acquired by a radar device, based on the camera parameters of the camera that acquires the image to be processed, wherein it should be noted that the camera that acquires the image to be processed needs to be able to observe the three-dimensional object, based on the camera parameters of the camera that acquires the image to be processed, the target point cloud is projected to the image to be processed, so that a sparse depth map can be obtained, wherein based on the camera parameters of the camera that can observe the three-dimensional object, the process of converting the point cloud into the depth map is the inverse process of converting the depth map into the point cloud, and the method of converting the point cloud into the depth map is not specifically limited here, and those skilled in the art can choose any method to project the target point cloud into the image to be processed to obtain a sparse depth map. Optionally, after selecting the target point cloud belonging to the three-dimensional detection frame from the sparse point cloud, it can also include filtering the points in the selected target point cloud to remove flying points. After projection, for points without depth values, the depth values ​​are set to the target value, for example, to 0. Then, based on the two-dimensional segmentation result, the points with depth values ​​equal to the target value are interpolated in the segmented area of ​​the three-dimensional object. Optionally, the interpolation processing corresponding to the difference algorithm can be estimated and completed using a deep learning network capable of estimating depth values, or can be used using traditional visual methods such as trilinear interpolation, which is not specifically limited here.

[0078] In the above embodiments, dense point cloud information of the corresponding object can be obtained in a variety of ways.

[0079] In one of the optional embodiments, the method also includes: acquiring camera parameters of multiple cameras of the image to be processed, projecting a dense point cloud of the three-dimensional object generated based on the image to be processed acquired by each camera to a target coordinate system; and rendering the three-dimensional object based on the dense point cloud of each three-dimensional object in the target coordinate system.

[0080] This step involves three-dimensional space fusion. By processing each image to be processed, a dense point cloud of each object under the corresponding viewing angle of each camera can be obtained and rendered for display.

[0081] In this way, the camera parameters of a camera that can observe three-dimensional objects can be used to project the dense point cloud into a unified target coordinate system, such as the world coordinate system. Finally, based on the dense point cloud of each three-dimensional object in the target coordinate system, the three-dimensional objects can be rendered, thereby achieving realistic rendering and display of the objects in the rendering environment, and realizing a more detailed visualization of the objects in the environment.

[0082] An exemplary application scenario of the above embodiment is a driving scenario. In this scenario, by processing the collected environmental images, three-dimensional objects in the scene, such as dense three-dimensional point clouds of road objects such as pedestrians and vehicles, are obtained. The generated three-dimensional point clouds can reflect the status of road objects, such as the state of a vehicle door open at a certain angle, the state of pedestrians walking or stopping, and rendering and displaying the dense three-dimensional point clouds can more intuitively display the state of the perceived objects to the vehicle, so as to make more accurate driving decisions.

[0083] Optionally, multiple point clouds obtained by back-projecting the same object under different cameras can be spliced ​​to obtain a final dense point cloud of the object.

[0084] For ease of understanding, combined Figure 3 As shown, Figure 3 This is a flowchart of a method for generating a point cloud of a three-dimensional object in another embodiment. In this embodiment, it is assumed that the perception system outputs a three-dimensional detection frame of the three-dimensional object, so that based on the three-dimensional detection frame, the target point cloud of each three-dimensional object can be obtained from the sparse point cloud output by the radar device, and then a camera that can observe the object is selected, and the target point cloud is projected into the corresponding image to be processed using the parameter information of the camera to obtain a series of second projection points, and the two-dimensional detection frame of the three-dimensional object can be obtained based on these second projection points.

[0085] The perception system inputs the two-dimensional detection box, the image to be processed, and the target point cloud of each three-dimensional object into a deep learning network with general object segmentation capabilities to obtain the two-dimensional segmentation results of the three-dimensional object.

[0086] For three-dimensional objects with occlusion, the deep learning-based image completion algorithm completes the two-dimensional segmentation results to obtain more complete object appearance information.

[0087] For each three-dimensional object, a sparse depth map can be obtained by projecting the target point cloud into the corresponding image to be processed. The depth values ​​of the points without depth values ​​in the sparse depth map are set to 0, and the two-dimensional segmentation results are used to interpolate the points with depth value 0 in the segmented area of ​​the three-dimensional object to obtain a dense depth map.

[0088] A dense point cloud is obtained based on the projection of the dense depth map and the camera parameter information.

[0089] Finally, the dense point cloud is projected into a unified target coordinate system, such as the world coordinate system, using the camera parameters of a camera that can observe three-dimensional objects. Finally, the three-dimensional objects are rendered based on the dense point clouds of each three-dimensional object in the target coordinate system, thereby achieving realistic rendering and display of the objects in the rendering environment.

[0090] In the above embodiments, objects such as vehicles, pedestrians, obstacles, etc. in the autonomous driving scene can be rendered, and the state and appearance of the objects in the real scene can be replicated.

[0091] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0092] Based on the same inventive concept, the embodiment of the present application also provides a device for generating a point cloud of a three-dimensional object for implementing the method for generating a point cloud of a three-dimensional object involved in the above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more embodiments of the device for generating a point cloud of a three-dimensional object provided below can refer to the limitations of the method for generating a point cloud of a three-dimensional object in the above text, and will not be repeated here.

[0093] In an exemplary embodiment, Figure 4 As shown, a device for generating a point cloud of a three-dimensional object is provided, comprising: an image acquisition module 401, a two-dimensional detection frame acquisition module 402, a two-dimensional segmentation module 403 and a dense point cloud generation module 404, wherein:

[0094] An image acquisition module 401 is used to acquire an image to be processed;

[0095] A two-dimensional detection frame acquisition module 402 is used to acquire a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed;

[0096] A two-dimensional segmentation module 403 is used to perform object segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object;

[0097] The dense point cloud generation module 404 is used to obtain dense depth information of the three-dimensional object based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, so as to generate a dense point cloud of each three-dimensional object.

[0098] In one of the optional embodiments, the above-mentioned two-dimensional detection frame acquisition module 402 is used to obtain the two-dimensional detection frame based on any one of the following methods: obtaining each two-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed; or obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, identifying the vertices of each of the three-dimensional detection frames, projecting the corresponding vertices to the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain each first projection point, and obtaining the corresponding two-dimensional detection frame of the three-dimensional object based on each first projection point; or obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, selecting a target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, projecting the target point cloud to the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain each second projection point, and obtaining the corresponding two-dimensional detection frame of the three-dimensional object based on each second projection point.

[0099] In one of the optional embodiments, the above-mentioned two-dimensional segmentation module 403 is used to obtain a two-dimensional segmentation result based on any one of the following methods: inputting the two-dimensional detection box and the image to be processed into a pre-trained segmentation model to obtain a two-dimensional segmentation result of each of the three-dimensional objects; or obtaining each three-dimensional detection box of each three-dimensional object perceived by the perception system based on the image to be processed, selecting a target point cloud belonging to the three-dimensional detection box from the sparse point cloud corresponding to the image to be processed, and inputting the target point cloud, the two-dimensional detection box and the image to be processed into a pre-trained segmentation model to obtain a two-dimensional segmentation result of each of the three-dimensional objects.

[0100] In one of the optional embodiments, the above-mentioned dense point cloud generation module 404 is used to obtain dense depth information based on any one of the following methods: obtaining a three-dimensional point cloud corresponding to the image to be processed containing dense depth information output by the sensor, and extracting dense depth information of the three-dimensional object from the three-dimensional point cloud based on the two-dimensional segmentation result; or obtaining each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, selecting a target point cloud belonging to the three-dimensional detection frame from the three-dimensional point cloud corresponding to the image to be processed, projecting the target point cloud to the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain a sparse depth map, setting the depth values ​​of the pixel points of the projection points of the non-target point cloud in the sparse depth map to the target value, and updating the depth values ​​of the pixel points of the projection points of the non-target point cloud within the range of the two-dimensional segmentation result by interpolation to obtain dense depth information.

[0101] In one of the optional embodiments, the above-mentioned device also includes: a display module, which is used to project the dense point cloud of the three-dimensional object generated based on the image to be processed captured by each camera to the target coordinate system based on the camera parameters of multiple cameras that capture the image to be processed; and render the three-dimensional object based on the dense point cloud of each of the three-dimensional objects in the target coordinate system.

[0102] In one of the optional embodiments, the above-mentioned device also includes: an image completion module, which is used to input the two-dimensional segmentation result into a pre-trained completion model when the two-dimensional segmentation result indicates that the three-dimensional object is occluded, so as to obtain a two-dimensional segmentation result including complete appearance information.

[0103] Each module in the above-mentioned device for generating a point cloud of a three-dimensional object can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0104] In an exemplary embodiment, a computer device is provided. The computer device may be a vehicle terminal, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for generating a point cloud of a three-dimensional object is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0105] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0108] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0109] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0110] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0111] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a point cloud of a three-dimensional object, characterized in that: The method comprises: Acquire the collected image to be processed; Acquire a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed; Performing target segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each of the three-dimensional objects; Based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, dense depth information of the three-dimensional object is obtained to generate a dense point cloud of each three-dimensional object.

2. The method according to claim 1, characterized in that The acquiring of the two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed includes any of the following methods: Acquire each two-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed; or Acquire each 3D detection frame of each 3D object perceived by the perception system based on the image to be processed, identify the vertices of each 3D detection frame, project each corresponding vertices onto the image to be processed based on the camera parameters of the camera that acquires the image to be processed to obtain each first projection point, and obtain a 2D detection frame corresponding to the corresponding 3D object based on each first projection point; or Acquire each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, select a target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, project the target point cloud onto the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain each second projection point, and obtain a corresponding two-dimensional detection frame corresponding to the three-dimensional object based on each second projection point.

3. The method according to claim 1, characterized in that The performing target segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object includes any one of the following methods: Inputting the two-dimensional detection frame and the image to be processed into a pre-trained segmentation model to obtain a two-dimensional segmentation result of each three-dimensional object; or Acquire each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, select the target point cloud belonging to the three-dimensional detection frame from the sparse point cloud corresponding to the image to be processed, input the target point cloud, the two-dimensional detection frame and the image to be processed into the pre-trained segmentation model, and obtain the two-dimensional segmentation result of each three-dimensional object.

4. The method according to claim 1, characterized in that: The dense depth information is obtained by any of the following methods: Acquire a three-dimensional point cloud corresponding to the image to be processed containing dense depth information output by a sensor, and extract dense depth information of the three-dimensional object from the three-dimensional point cloud based on the two-dimensional segmentation result; or Acquire each three-dimensional detection frame of each three-dimensional object perceived by the perception system based on the image to be processed, select a target point cloud belonging to the three-dimensional detection frame from the three-dimensional point cloud corresponding to the image to be processed, project the target point cloud onto the image to be processed based on the camera parameters of the camera that collects the image to be processed to obtain a sparse depth map, set the depth values ​​of the pixel points of the projection points of the non-target point cloud in the sparse depth map to the target values, and update the depth values ​​of the pixel points of the projection points of the non-target point cloud within the range of the two-dimensional segmentation result by interpolation to obtain dense depth information.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Based on the camera parameters of the multiple cameras that collect the images to be processed, projecting the dense point cloud of the three-dimensional object generated based on the images to be processed collected by each of the cameras into a target coordinate system; The three-dimensional objects are rendered based on the dense point clouds of the three-dimensional objects in the target coordinate system.

6. The method according to any one of claims 1 to 4, characterized in that: After performing target segmentation on the image to be processed based on the two-dimensional detection frame to obtain the two-dimensional segmentation result of each three-dimensional object, the method further includes: When the two-dimensional segmentation result indicates that the three-dimensional object is occluded, the two-dimensional segmentation result is input into a pre-trained completion model to obtain a two-dimensional segmentation result including complete appearance information.

7. A device for generating a point cloud of a three-dimensional object, characterized in that: The device comprises: An image acquisition module, used to acquire the collected image to be processed; A two-dimensional detection frame acquisition module, used to acquire a two-dimensional detection frame corresponding to each three-dimensional object based on the image to be processed; A two-dimensional segmentation module, used to perform target segmentation on the image to be processed based on the two-dimensional detection frame to obtain a two-dimensional segmentation result of each three-dimensional object; The dense point cloud generation module is used to obtain dense depth information of the three-dimensional object based on the two-dimensional segmentation result and the three-dimensional point cloud corresponding to the image to be processed, so as to generate a dense point cloud of each three-dimensional object.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images