Three-dimensional reconstruction method and related equipment

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method and related equipment. The method comprises the steps that radar point cloud frames and image data corresponding to the same scene are acquired; determining a target edge point in the radar point cloud frame; determining edge contour points in the image data, and determining normal vectors corresponding to the edge contour points; determining a normal projection error of the target edge point and the edge contour point based on the target edge point, the edge contour point and the normal vector; updating a camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose; and fusing the radar point cloud frame and the image data based on the target camera pose to obtain a reconstructed image. By means of the method, the precision and robustness of three-dimensional reconstruction can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional reconstruction, and in particular, to a three-dimensional reconstruction method and related equipment. Background Art

[0002] The LIVO (LiDAR-Inertial-Vision-Odometry) system combines the sensor fusion technology of LiDAR, inertial measurement unit and camera vision data, which can be used to estimate and track the position and posture of mobile devices or vehicles. In order to achieve high-precision positioning and map construction, it is usually necessary to fuse the three-dimensional point cloud geometry information of the LiDAR with the two-dimensional image texture information of the camera. However, due to the asynchronous data acquisition of the camera and LiDAR, the discrete characteristics of the radar data and other reasons, it is difficult for related technologies to achieve accurate fusion of radar data and image data, resulting in difficulty in accurately aligning the geometry and texture in the reconstructed data, thus affecting the accuracy and availability of positioning and map construction. Summary of the invention

[0003] The present application discloses a three-dimensional reconstruction method and related equipment, which can improve the fusion accuracy of radar data and image data, as well as the alignment accuracy of geometry and texture in the reconstructed data.

[0004] A first aspect of an embodiment of the present application provides a three-dimensional reconstruction method, including: obtaining a radar point cloud frame and image data corresponding to the same scene; determining a target edge point in the radar point cloud frame; determining an edge contour point in the image data, and determining a normal vector corresponding to the edge contour point; determining a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point and the normal vector; updating a camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose; and fusing the radar point cloud frame and the image data based on the target camera pose to obtain a reconstructed image.

[0005] According to an embodiment of the present application, determining the target edge point in the radar point cloud frame includes: determining the target edge point in the radar point cloud frame according to an intensity difference between any radar point and an adjacent radar point in the radar point cloud frame, wherein the adjacent radar point represents a radar point located on the same radar scan line as the any radar point.

[0006] According to an embodiment of the present application, the adjacent radar points include a first radar point and a second radar point respectively located on both sides of the any radar point, and determining the target edge point in the radar point cloud frame according to the intensity difference between any radar point and the adjacent radar points in the radar point cloud frame includes: determining a first intensity difference between the any radar point and the first radar point; determining a second intensity difference between the any radar point and the second radar point; if the first intensity difference is greater than a first threshold and the second intensity difference is less than a second threshold, determining that the any radar point and the first radar point are the target edge points.

[0007] According to an embodiment of the present application, the method further includes: determining the first threshold and the second threshold according to the intensity value of any radar point.

[0008] According to an embodiment of the present application, determining the target edge point in the radar point cloud frame includes: obtaining a fused point cloud of the radar point cloud frame and an adjacent radar point cloud frame; determining the pixel coordinates of a first projection point corresponding to any radar point in the fused point cloud in a pixel coordinate system, wherein the pixel coordinate system represents a coordinate system corresponding to the image data; based on the pixel coordinates, determining a first adjacent pixel of the first projection point in the pixel coordinate system, and determining a weight of the first adjacent pixel; based on the weight and an intensity value of any radar point, determining an intensity parameter of the first adjacent pixel; determining a first pixel value of the first adjacent pixel according to the intensity parameter, and obtaining a first image based on the first pixel value; determining a second adjacent pixel from the first adjacent pixel, determining a second pixel value of the second adjacent pixel according to a preset first value, and obtaining a second image based on the second pixel value; determining the target edge point based on the first edge point of the first image and the second edge point of the second image.

[0009] According to an embodiment of the present application, obtaining a fused point cloud of the radar point cloud frame and adjacent radar point cloud frames includes: projecting the radar point cloud frame and the adjacent radar point cloud frames to a world coordinate system, and fusing the radar point cloud frame and the adjacent radar point cloud frames in the world coordinate system to obtain the fused point cloud.

[0010] According to an embodiment of the present application, determining the pixel coordinates of the first projection point corresponding to any radar point in the fused point cloud in the pixel coordinate system includes: based on the camera pose corresponding to the image data, projecting the radar point in the fused point cloud to the camera coordinate system corresponding to the image data, and determining the coordinates of the first projection point of the radar point in the camera coordinate system; based on the intrinsic parameter matrix of the camera corresponding to the image data, converting the coordinates of the first projection point to the pixel coordinate system to obtain the pixel coordinates.

[0011] According to an embodiment of the present application, determining the first neighboring pixel point of the first projection point in the pixel coordinate system based on the pixel coordinates, and determining the weight of the first neighboring pixel point, includes: selecting a pixel point within a first range centered on the pixel coordinates as the first neighboring pixel point; determining an influence factor of any radar point on the first neighboring pixel point based on the coordinates of the first neighboring pixel point and the pixel coordinates, and determining the weight of the first neighboring pixel point based on the influence factor.

[0012] According to an embodiment of the present application, the method also includes: if there is a corresponding relationship between the first neighboring pixel point and multiple radar points in the fused point cloud, determining the influence factor of each radar point on the first neighboring pixel point; determining a first accumulated value of the influence factors of the multiple radar points on the first neighboring pixel point, and determining the weight of the first neighboring pixel point according to the first accumulated value.

[0013] According to an embodiment of the present application, determining the intensity parameter of the first adjacent pixel point based on the weight and the intensity value of any radar point includes: determining the intensity parameter of the first adjacent pixel point based on a ratio of the intensity value of any radar point to the weight.

[0014] According to an embodiment of the present application, the method further includes: if there is a corresponding relationship between the first adjacent pixel point and multiple radar points in the fused point cloud, determining a second accumulated value of the intensity values ​​of the multiple radar points; and determining an intensity parameter of the first adjacent pixel point based on a ratio of the second accumulated value to the weight.

[0015] According to an embodiment of the present application, determining the first pixel value of the first adjacent pixel point according to the intensity parameter and obtaining the first image based on the first pixel value includes: using the first pixel value to update the pixel value of the first adjacent pixel point in a preset image to obtain the first image, wherein the pixel value of the pixel point in the preset image is a preset second value.

[0016] According to an embodiment of the present application, determining the second adjacent pixel point among the first adjacent pixel points, determining the second pixel value of the second adjacent pixel point according to a preset first numerical value, and obtaining a second image based on the second pixel value, includes: among the first adjacent pixel points, selecting a pixel point located in a second range with the pixel coordinates as the center as the second adjacent pixel point; and using the second pixel value to update the pixel value of the second adjacent pixel point in the preset image to obtain the second image, wherein the pixel value of the pixel point in the preset image is the preset second numerical value.

[0017] According to an embodiment of the present application, determining the target edge point based on the first edge point of the first image and the second edge point of the second image includes: performing edge extraction on the first image to determine the first edge point; performing edge extraction on the second image to determine the second edge point; determining a third edge point based on the intersection of the first edge point and the second edge point; and determining the target edge point based on a radar point corresponding to the third edge point in the fused point cloud.

[0018] According to an embodiment of the present application, determining edge contour points in the image data and determining normal vectors corresponding to the edge contour points include: obtaining a grayscale image corresponding to the image data, performing edge extraction on the grayscale image to obtain the edge contour points; and determining the normal vectors corresponding to the edge contour points based on a principal component analysis algorithm.

[0019] According to an embodiment of the present application, based on the target edge point cloud, the edge contour point and the normal vector, determining the normal projection error between the target edge point and the edge contour point includes: projecting the target edge point cloud to a pixel coordinate system corresponding to the image data, and determining a second projection point of the radar point in the target edge point cloud in the pixel coordinate system; and determining the normal projection error based on the distance between the edge contour point and the second projection point in the direction of the normal vector.

[0020] According to an embodiment of the present application, updating the camera pose corresponding to the image data according to the normal projection error includes: updating the camera pose based on a preset optimization target, wherein the optimization target includes minimizing the normal projection error.

[0021] According to an embodiment of the present application, after updating the camera pose corresponding to the image data according to the normal projection error, the method further includes: determining a reprojection error between the radar point cloud frame and the image data, updating the camera pose according to the reprojection error, and obtaining the target camera pose.

[0022] A second aspect of an embodiment of the present application provides a three-dimensional reconstruction device, which includes: an acquisition module, used to acquire radar point cloud frames and image data corresponding to the same scene; a determination module, used to determine target edge points in the radar point cloud frames; determine edge contour points in the image data, and determine normal vectors corresponding to the edge contour points; based on the target edge points, the edge contour points and the normal vectors, determine the normal projection errors between the target edge points and the edge contour points; an update module, used to update the camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose; and a reconstruction module, used to fuse the radar point cloud frames and the image data based on the target camera pose to obtain a reconstructed image.

[0023] A third aspect of an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the three-dimensional reconstruction method.

[0024] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional reconstruction method is implemented.

[0025] The three-dimensional reconstruction method provided in the embodiment of the present application obtains radar point cloud frames and image data of the same scene, accurately locates the target edge points in the radar point cloud and the edge contour points and their normal vectors in the image, and then calculates the normal projection error between the two. This error is used to optimize and update the camera pose corresponding to the image data until the accurate target camera pose is obtained. Based on this precise pose information, the radar point cloud and image data are fused to generate detailed and accurate reconstructed images, providing strong support for the visualization and understanding of the three-dimensional environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of an application environment of a three-dimensional reconstruction method provided in one embodiment of the present application.

[0028] Figure 2 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided in one embodiment of the present application.

[0029] Figure 3This is an example diagram of adjacent radar points provided by an embodiment of the present application.

[0030] Figure 4 A flowchart of a method for determining target edge points provided in one embodiment of the present application.

[0031] Figure 5 This is an example diagram of a first image provided in an embodiment of the present application.

[0032] Figure 6 This is an example diagram of a first edge point provided in an embodiment of the present application.

[0033] Figure 7 This is an example diagram of target edge points provided in one embodiment of the present application.

[0034] Figure 8 A flowchart of a three-dimensional reconstruction method provided in another embodiment of the present application.

[0035] Fig. 9 A structural diagram of a three-dimensional reconstruction device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specified order or sequence.

[0038] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0039] 3D reconstruction technology has a wide range of applications in autonomous driving, robot navigation, augmented reality and other fields. In order to achieve high-precision 3D scene reconstruction, multi-sensor fusion systems such as LIVO (LiDAR-Inertial-Visual Odometry) are widely used. Such systems combine multi-modal data such as LiDAR, Inertial Measurement Unit (IMU) and camera to generate a 3D map with consistent geometry and texture.

[0040] However, existing technologies still face the following challenges in practical applications: Sensor extrinsic calibration error: The extrinsic calibration error between the camera and the lidar will result in the inability to accurately align the camera's two-dimensional pixels and three-dimensional geometric features in the initial state, thereby affecting the subsequent camera pose optimization and map reconstruction accuracy; Sensor data asynchrony problem: Due to the different data acquisition frequencies of the camera and the lidar, and the problem of timestamp asynchrony, it is difficult to directly align multimodal data, further increasing the difficulty of pose estimation; IMU accuracy limitation: In the filter-based LIVO system, IMU data is used for pre-integration to estimate the sensor pose, but the accuracy of the IMU decreases over time, and its discrete data characteristics will introduce cumulative errors, affecting the long-term stability of the system; Insufficient geometry and texture alignment: Existing methods usually rely on rigid registration algorithms (such as iterative closest point (ICP)) for geometric alignment of lidar point clouds, but due to the lack of accurate three-dimensional-two-dimensional (3D-2D) correspondence, the optimization effect of the camera pose is limited, resulting in poor geometry and texture consistency of the reconstructed map.

[0041] In response to the above problems, the existing technology has proposed a variety of solutions, but there are still obvious shortcomings. For example, the initial stationary assumption method assumes that the system is stationary in the initial state to achieve 3D-2D alignment, but this assumption is difficult to meet in practical applications (such as handheld scanning devices) and limits the applicability of the system. The method that requires hardware synchronization of data acquisition obtains accurate initial pose by hard synchronization of camera and lidar data, but this requires additional hardware support and has extremely high requirements for the accuracy of external parameter calibration. Some methods use offline processing (such as SFM (Structure from Motion) reconstruction algorithm) combined with point-surface ICP constraints to optimize geometry and texture consistency, but this method cannot meet real-time requirements and has low computational efficiency in large scenes.

[0042] In order to solve the above problems, the three-dimensional reconstruction method provided in the embodiment of the present application combines the edge features of the radar point cloud with the normal vector information of the image edge contour, calculates the normal projection error and optimizes the camera pose, thereby achieving high-precision alignment of geometry and texture, and improving the accuracy and robustness of three-dimensional reconstruction.

[0043] See also Figure 1 , is a schematic diagram of an application environment of a three-dimensional reconstruction method provided by an embodiment of the present application. Figure 1 As shown, the three-dimensional reconstruction method provided in the embodiment of the present application can be applied to an electronic device 10, and the electronic device 10 can be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, an energy storage device, a power distribution device, a vehicle-mounted device, a self-moving device, a scanner, and other electronic devices. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0044] like Figure 1 As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the I / O interface 104 through the bus 105.

[0045] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), Bluetooth (blue tooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0046] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0047] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 103. The non-volatile memory may include a disk storage device and a flash memory.

[0048] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions, and when the multiple instructions are executed by the processor 103, the three-dimensional reconstruction method executed on the electronic device 10 can be implemented.

[0049] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0050] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0051] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned three-dimensional reconstruction method.

[0052] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information or visualize information. In addition, the I / O interface 104 can also be used to connect various sensor devices, such as radar sensors, image sensors, etc., so as to obtain the required radar point cloud data, image data and other data.

[0053] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 10 .

[0054] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0055] The following will be a computer program product that implements the three-dimensional reconstruction method of the embodiment of the present application and runs on an electronic device (such as Figure 1 The electronic device 10 shown in FIG. Figure 2 FIG. 1 is a flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application. In an embodiment of the present application, the method includes the following steps: Step S201, obtaining radar point cloud frames and image data corresponding to the same scene.

[0056] In some embodiments, radar point cloud frames and image data of the same scene can be collected synchronously. Among them, the radar point cloud frame can be obtained from the radar sensor, and the radar point cloud frame contains the three-dimensional coordinates (x, y, z) and intensity values ​​(intensity) of each radar point. Among them, the value of the Z coordinate can represent the depth value of the radar point, and the depth value can indicate the distance between the radar sensor and the object, so as to help construct the three-dimensional shape of the object in the three-dimensional reconstruction process. The intensity value can indicate the reflection intensity of the laser beam on the surface of the measured object in the scene, and can reflect the physical properties of the object surface, such as roughness, tilt angle, and reflectivity, so that the reconstructed three-dimensional image can truly reflect the material and shape differences of the object surface.

[0057] Image data, such as color images or grayscale images, can be obtained from image sensors. Image data can contain appearance features of objects in the scene, such as color, brightness, and texture details. By aligning and fusing image data with radar point cloud frames, the texture information in the image can be mapped to the geometric surface corresponding to the radar point cloud frame, and a three-dimensional model can be reconstructed based on the combined effect of intensity values ​​and texture information.

[0058] Step S202, determining target edge points in the radar point cloud frame.

[0059] In some embodiments, the target edge point represents a radar point in the radar point cloud frame that has a significant difference in intensity value. Two different methods can be used to determine the target edge point in the radar point cloud frame, and the first method includes: determining the target edge point in the radar point cloud frame according to the intensity difference between any radar point and an adjacent radar point in the radar point cloud frame, wherein the adjacent radar point represents a radar point located on the same radar scan line as the any radar point.

[0060] In some embodiments, the adjacent radar points include a first radar point and a second radar point located on both sides of the any radar point, respectively, and the determining of the target edge point in the radar point cloud frame according to the intensity difference between any radar point and the adjacent radar points in the radar point cloud frame includes: determining a first intensity difference between the any radar point and the first radar point; determining a second intensity difference between the any radar point and the second radar point; if the first intensity difference is greater than a first threshold and the second intensity difference is less than a second threshold, determining that the any radar point and the first radar point are the target edge points. The first threshold and the second threshold can be determined according to the intensity value of the any radar point.

[0061] In one example, reference Figure 3As shown, it is an example diagram of adjacent radar points provided by an embodiment of the present application. For any radar point b in the radar point cloud frame, its adjacent radar points include a first radar point a and a second radar point c located on the same radar scan line (i.e., the same laser beam scan ring). The first intensity difference between point b and the first radar point a is calculated as ΔIab=|Ib-Ia|, where Ib represents the intensity value of point b and Ia represents the intensity value of point a; the second intensity difference between point b and the second radar point c is calculated as ΔIbc=|Ib-Ic|, where Ic represents the intensity value of point c.

[0062] If the first intensity difference and the second intensity difference meet the following two conditions: ΔIab>δmax, ΔIbc<δmin, then point b and the first radar point a can be determined as target edge points. δmax represents the first threshold, for example, δ max=0.4×Ib; δmin represents the second threshold, for example δ min=0.1×Ib. For example, if the intensity of point b is Ib=100, δmax=40, δmin=10. If ΔIab=50, ΔIbc=8, then a and b are determined as edge points. Among them, ΔIab>δmax, it can be determined that there is a significant intensity jump between the first radar point a and point b. ΔIbc<δmin, it can be determined that the intensity change between point b and the second radar point c is gentle. In this way, point b and the first radar point with a significant intensity difference can be determined as target edge points.

[0063] In one example, all radar points on each radar scan line of the radar point cloud frame can be traversed, and target edge points can be extracted according to the method provided in the above embodiment. For example, for points d and e on the scan line, if the corresponding conditions are met, such as ΔIde>δmax and ΔIef<δmin, then d and e can also be determined as target edge points.

[0064] Based on the above embodiments, dynamic threshold setting can be used to avoid the problem of insufficient adaptability of fixed thresholds to different reflection intensity scenes. Combined with the dual-condition judgment of the intensity difference of adjacent points, the real edge and the local intensity fluctuation caused by noise can be effectively distinguished, so that the target edge point can be accurately identified.

[0065] In some embodiments, a second method that can be used to determine the target edge point in the radar point cloud frame includes: obtaining a fused point cloud of the radar point cloud frame and the adjacent frame, and projecting the radar point in the fused point cloud into the pixel coordinate system of the image data to determine the corresponding first projection point and its pixel coordinates; finding the first adjacent pixel point of the first projection point according to the pixel coordinates and calculating its weight, combining the weight with the radar point intensity value to determine the intensity parameter and the first pixel value of the first adjacent pixel point to generate a first image; determining the second adjacent pixel point from the first adjacent pixel point, and setting its second pixel value according to a preset value to generate a second image; combining the edge points of the first image with the edge points of the second image to determine the target edge point. In one example, the specific implementation steps of the above method include the following: Figure 4 The process shown.

[0066] Based on the above embodiments, two different methods can be used to accurately determine the target edge points in the radar point cloud frame. These target edge points represent radar points with significant intensity changes in the radar point cloud frame, and they play a vital role in the 3D reconstruction process. Specifically, these target edge points can serve as key feature points to assist in achieving more accurate camera pose estimation, thereby significantly improving the overall effect of 3D reconstruction.

[0067] Step S203: determining edge contour points in the image data, and determining normal vectors corresponding to the edge contour points.

[0068] In some embodiments, when determining the edge contour points in the image data, a grayscale image corresponding to the image data may be obtained, and the edge contour points may be obtained by performing edge extraction on the grayscale image. In one example, the above-mentioned edge extraction is performed using an image processing algorithm to identify significant edge feature points in the image as edge contour points. For example, the image processing algorithm includes but is not limited to a Canny edge detector, a Sobel operator, or a Prewitt operator. In this way, the color information in the image data may be simplified through a grayscale image, thereby improving the accuracy of edge extraction of the image.

[0069] Based on the above embodiments, the target edge points in the radar points often correspond to the geometric edges or texture edges of the image data in the corresponding image data. By accurately identifying these edge contour points in the image data, the precise corresponding points of the edge feature points in the radar points in the image data can be further determined, thereby providing corresponding edge point pair information for subsequent three-dimensional reconstruction of radar point cloud frames and image data, which helps to achieve high-quality three-dimensional reconstruction.

[0070] In some embodiments, when determining the normal vector corresponding to the edge contour point, the normal vector corresponding to the edge contour point can be determined based on a principal component analysis algorithm. In one example, a local window centered on the edge contour point can be used, and the range within the window is used as the neighborhood of the edge contour point, and the pixel points within the window are used as the neighborhood points of the edge contour point. Based on the principal component analysis algorithm, two main directions within the neighborhood of the edge contour point are determined, including the direction along the edge (tangent vector) and the direction perpendicular to the edge (normal vector). The direction of the normal vector is obtained by calculating the covariance matrix of the two main directions and performing eigenvalue decomposition. The normal vector of this direction is normalized, for example, the length of the normal vector is adjusted to 1, so that it only represents the direction without containing length information.

[0071] Based on the above embodiments, by determining edge contour points in image data and their corresponding normal vectors, it is possible to more accurately estimate the shape and texture of the surface of an object in three-dimensional reconstruction, thereby achieving a better three-dimensional reconstruction effect.

[0072] Step S204: determining a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point and the normal vector.

[0073] In some embodiments, when determining the normal projection error between the target edge point and the edge contour point, the target edge point cloud may be projected to a pixel coordinate system corresponding to the image data to determine a second projection point of the radar point in the target edge point cloud in the pixel coordinate system. The normal projection error is determined based on the distance between the edge contour point and the second projection point in the normal vector direction.

[0074] In some embodiments, when determining the second projection point, since the target edge point in the fused point cloud has coordinates in the world coordinate system, the coordinates of the target edge point in the world coordinate system can be converted to the camera coordinate system according to the camera pose to obtain the coordinates of the target edge point in the camera coordinate system. Thereafter, the coordinates of the target edge point in the camera coordinate system can be converted to the pixel coordinate system according to the camera intrinsic parameters, thereby determining the second projection point of the radar point in the target edge point cloud in the pixel coordinate system, and the position of the second projection point can indicate the precise projection position of the target edge point on the image plane.

[0075] In some embodiments, when determining the normal projection error, the distance between each second projection point and the nearest edge contour point in the normal vector direction may be calculated using known edge contour points and their corresponding normal vector information. For example, the formula used may be exemplarily expressed as: , in, Represents the coordinates of the second projection point proj corresponding to the target edge point j, represents the coordinates of the i-th nearest edge contour point corresponding to the second projection point, represents the normal vector of the i-th nearest edge contour point, k represents the k nearest edge contour points corresponding to the second projection point. E represents the normal projection error between the target edge point j and the edge contour point, which can be used to indicate the inconsistency between the target edge point and the image edge contour point in the normal vector direction.

[0076] Based on the above embodiment, the calculation of the normal projection error not only considers the spatial position relationship between the target edge point and the edge contour point, but also makes full use of the normal vector information of the edge contour point, thereby providing a more accurate and comprehensive error evaluation. This error index can be used to measure the fusion quality between radar data and image data, guide subsequent 3D reconstruction tasks, and achieve high-quality 3D reconstruction.

[0077] Step S205: updating the camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose.

[0078] In some embodiments, updating the camera pose corresponding to the image data according to the normal projection error includes: updating the camera pose based on a preset optimization target, wherein the optimization target includes minimizing the normal projection error.

[0079] In one example, the core of the optimization goal of the normal projection error includes minimizing the normal projection error. For example, it is expected that by adjusting the camera pose, when the target edge point in the radar point cloud frame is projected onto the image data, the distance between the edge contour point in the image in the normal vector direction (i.e., the normal projection error) is minimized. This optimization goal can be achieved by using a preset optimization algorithm, such as an iterative closest point algorithm, a bundle adjustment algorithm, etc. These optimization algorithms can gradually adjust the camera pose in an iterative manner according to the current normal projection error until a solution that meets the preset conditions is reached, such as the preset condition that the normal projection error is less than a preset threshold or the number of iterations reaches an upper limit. Among them, in order to maintain the continuity and smoothness of the camera pose, so as to avoid image distortion or inaccurate three-dimensional reconstruction results caused by sudden changes in the pose, a preset regularization term or constraint condition can be introduced when updating the camera pose to ensure the stability and reliability of the optimization process.

[0080] Based on the above embodiment, the camera pose can be accurately updated according to the normal projection error, thereby obtaining a more accurate target camera pose. This step can provide more reliable data support for subsequent data fusion and 3D reconstruction tasks, thereby improving the performance and accuracy of the entire system.

[0081] In some embodiments, after updating the camera pose corresponding to the image data according to the normal projection error, the method further includes: determining a reprojection error between the radar point cloud frame and the image data, updating the camera pose according to the reprojection error, and obtaining the target camera pose.

[0082] In one example, after updating the camera pose corresponding to the image data according to the normal projection error, the calculated reprojection error can be used to set an optimization target for the reprojection error, and the camera pose can be further adjusted based on the optimization target for the reprojection error to obtain a target camera pose. The optimization target for the reprojection error includes minimizing the reprojection error, for example, making the radar point as close as possible to the corresponding point in the image when projected onto the image, so as to achieve accurate alignment of the three-dimensional-two-dimensional (3D-2D) correspondence.

[0083] Based on the above embodiments, the camera pose can be updated by combining the normal projection error and the reprojection error to obtain a more accurate and robust target camera pose, thereby improving the accuracy of tasks such as three-dimensional reconstruction.

[0084] Step S206: fusing the radar point cloud frame and the image data based on the target camera posture to obtain a reconstructed image.

[0085] In some embodiments, after obtaining the target camera pose, radar point cloud frame and image data, the target camera pose can be used to project the points in the radar point cloud frame onto the image plane of the camera. The projected radar point cloud data can be fused with the image data. For example, the fusion can be a simple superposition or a more complex processing, such as coloring or enhancing the image according to the depth value of the radar point. The fused data can be further processed to generate a reconstructed image. The reconstructed image can be an enhanced image that combines radar depth information and image texture information, or it can be a completely new three-dimensional reconstruction result extracted from radar and image data. The generated reconstructed image can undergo some post-processing steps, such as denoising, smoothing, enhancement, etc., to improve the quality and readability of the reconstructed image.

[0086] The three-dimensional reconstruction method provided in the embodiment of the present application obtains radar point cloud frames and image data of the same scene, accurately locates the target edge points in the radar point cloud and the edge contour points and their normal vectors in the image, and then calculates the normal projection error between the two. This error is used to optimize and update the camera pose corresponding to the image data until the accurate target camera pose is obtained. Based on this precise pose information, the radar point cloud and image data are fused to generate detailed and accurate reconstructed images, providing strong support for the visualization and understanding of the three-dimensional environment.

[0087] In some embodiments, reference Figure 4As shown, the method for determining the target edge point in the radar point cloud frame includes the following process: Step S401, obtaining a fused point cloud of the radar point cloud frame and adjacent radar point cloud frames.

[0088] In some embodiments, the time node of collecting the current radar point cloud frame can be determined, and multiple continuous radar point cloud frames obtained before the time node are used as adjacent radar point cloud frames of the current radar point cloud frame. For example, the first 20 radar point cloud frames of the current radar point cloud frame can be obtained as adjacent radar point cloud frames.

[0089] In some embodiments, in order to achieve effective fusion of different radar point cloud frames, a first transformation relationship between the radar coordinate system corresponding to the radar point cloud frame and the world coordinate system may be obtained, such as a first rotation matrix and a first translation vector. Using this first transformation relationship, the radar point cloud frame and the adjacent radar point cloud frame may be projected to the world coordinate system, and the radar point cloud frame and the adjacent radar point cloud frame may be fused in the world coordinate system to obtain the fused point cloud. In this way, each radar point in the fused point cloud has corresponding coordinates in the world coordinate system.

[0090] Based on the above embodiment, by acquiring the current radar point cloud frame and multiple adjacent radar point cloud frames, the deficiency of single-frame data can be avoided in the subsequent analysis process, and the integrity and accuracy of the data can be improved. The fusion of multiple frames of data can provide richer scene information, which helps to more accurately describe the shape, position, texture and other features of the object, and helps to achieve more accurate three-dimensional reconstruction.

[0091] In addition, since the radar coordinate systems corresponding to different radar point cloud frames may be different, directly fusing multiple frames of data in the radar coordinate system may lead to problems such as coordinate mismatch and data misalignment. By fusing multiple frames of radar data in a unified world coordinate system, problems such as inconsistent coordinate systems, error accumulation, and lack of global perspective can be eliminated, thereby improving the reliability and practicality of the fused data.

[0092] Step S402, determining the pixel coordinates of a first projection point corresponding to any radar point in the fused point cloud in a pixel coordinate system.

[0093] In some embodiments, the pixel coordinate system represents the coordinate system corresponding to the image data. In order to determine the pixel coordinates of the first projection point corresponding to any radar point in the fused point cloud in the pixel coordinate system, a projection transformation can be performed according to the camera pose and intrinsic parameter matrix corresponding to the image data.

[0094] First, based on the camera pose corresponding to the image data, the radar point in the fused point cloud can be projected to the camera coordinate system corresponding to the image data, and the coordinates of the first projection point of the radar point in the camera coordinate system can be determined. The camera pose can include the conversion relationship between the camera coordinate system and the world coordinate system, such as a translation vector and a rotation matrix. Based on the camera pose, the fused point cloud in the world coordinate system can be converted to the camera coordinate system, thereby obtaining the coordinates of the first projection point of the radar point in the fused point cloud in the camera coordinate system.

[0095] Afterwards, the coordinates of the first projection point can be converted to the pixel coordinate system based on the intrinsic parameter matrix of the camera corresponding to the image data to obtain the pixel coordinates. The intrinsic parameter matrix includes internal parameters such as the focal length and optical center position of the camera, which can reflect how the points in the camera coordinate system are mapped to the pixel coordinate system, so that the coordinates in the camera coordinate system can be converted to the pixel coordinate system based on the intrinsic parameter matrix to obtain the pixel coordinates.

[0096] In one example, when projecting a three-dimensional radar point to a two-dimensional pixel coordinate system, there may be a situation where multiple radar points with different depth values ​​are projected to the same pixel coordinate position (for example, the same pixel), resulting in the same pixel corresponding to multiple depth values ​​of multiple radar points, and occlusion relationships and data confusion between multiple radar points. In order to avoid this situation, for multiple radar points projected to the same pixel, only the data of the radar point with the smallest depth value greater than 0 can be retained. The radar point with the smallest depth value represents the point closest to the camera. Retaining such a radar point is more in line with the rule that closer objects will visually occlude farther objects in visual perception. At the same time, requiring the depth value to be greater than 0 can exclude invalid or abnormal depth values ​​caused by noise or errors.

[0097] Based on the above embodiment, the radar point in the fused point cloud can be projected from the world coordinate system to the camera coordinate system using the camera pose associated with the image data to obtain the coordinates of the first projection point of the radar point in the camera coordinate system. Subsequently, the coordinates of the first projection point are further converted to the pixel coordinate system with the help of the camera's intrinsic parameter matrix, so as to accurately obtain the pixel coordinates of the radar point in the pixel coordinate system, which can provide accurate coordinate information for subsequent edge point extraction. Among them, by adopting a depth-first strategy to deal with the depth information conflict problem in the projection process, not only can a more accurate and reliable projection result be obtained, but also strong support can be provided for subsequent image analysis and processing tasks.

[0098] Step S403: determining a first adjacent pixel point of the first projection point in the pixel coordinate system based on the pixel coordinates, and determining a weight of the first adjacent pixel point.

[0099] In some embodiments, the pixel coordinates may be used as the center to select pixels within a first range as the first adjacent pixel points. For example, the first range may be a range of 7×7 pixels centered on the first projection point, and the pixels within the range except the first projection point are used as the first adjacent pixel points.

[0100] In some embodiments, the influence factor of any radar point on the first neighboring pixel point can be determined according to the coordinates of the first neighboring pixel point and the pixel coordinates, and the weight of the first neighboring pixel point can be determined based on the influence factor.

[0101] In one example, for each first adjacent pixel point, the influence factor of the radar point corresponding to the first projection point on the first pixel point is calculated, and the influence factor can be determined according to the distance between the first adjacent pixel point and the first projection point. Specifically, the influence factor can be set as a function that decreases as the distance increases to reflect the attenuation characteristics of the influence of the radar point corresponding to the first projection point on the first adjacent pixel point. For example, a Gaussian function can be used to calculate the influence factor wx, and the influence factor wx = e^(-(ui-u)^2 / s - (vi-v)^2 / s), where (u,v) represents the pixel coordinates of the first projection point, (ui,vi) represents the pixel coordinates of the first adjacent pixel point i, and s represents a preset coefficient for controlling the speed of attenuation of the influence factor.

[0102] In one example, if the first adjacent pixel point corresponds to only one radar point in the fused point cloud, the weight of each first pixel point may be equal to the influence factor.

[0103] In some embodiments, if the first neighboring pixel point has a corresponding relationship with multiple radar points in the fused point cloud, determine the influence factor of each radar point on the first neighboring pixel point, determine a first cumulative value of the influence factors of the multiple radar points on the first neighboring pixel point, and determine the weight of the first neighboring pixel point according to the first cumulative value.

[0104] In one example, if multiple radar points are projected near a first adjacent pixel point, the first adjacent pixel point may belong to the adjacent range of each of the multiple first projection points corresponding to the multiple radar points. At this time, it can be determined that the first adjacent pixel point has a corresponding relationship with the multiple radar points in the fused point cloud. In order to determine the comprehensive influence of the multiple radar points on the first adjacent pixel point, the influence factor of each radar point on the pixel point can be calculated respectively, and these influence factors are accumulated to obtain a first accumulated value, and the final weight of the first adjacent pixel point is determined according to this accumulated value.

[0105] Based on the above embodiment, adjacent pixels within the first range are selected with the pixel coordinates of the first projection point as the center, and their weights are determined by calculating the influence factors according to the distances between these adjacent pixels and the first projection point. When an adjacent pixel corresponds to only one radar point, its weight is directly equal to the influence factor; if it corresponds to multiple radar points, its weight is determined by combining the cumulative values ​​of the influence factors of multiple radar points, so that the attenuation characteristics and comprehensive effects of the radar point on the adjacent pixel can be accurately measured.

[0106] Step S404: determining an intensity parameter of the first adjacent pixel point based on the weight and the intensity value of any radar point.

[0107] In some embodiments, if the first neighboring pixel point has a corresponding relationship with only one radar point in the fused point cloud, the intensity parameter of the first neighboring pixel point can be determined according to the ratio of the intensity value of any radar point to the weight. The intensity parameter can reflect the relative intensity of the neighboring pixel points when the radar point influences the weight of the neighboring pixel points.

[0108] In some embodiments, if the first adjacent pixel point has a corresponding relationship with multiple radar points in the fused point cloud, a second cumulative value of the intensity values ​​of the multiple radar points is determined; and an intensity parameter of the first adjacent pixel point is determined according to a ratio of the second cumulative value to the weight. Similar to the corresponding situation of a single radar point, the intensity parameter also reflects the relative intensity of the adjacent pixel points when considering the weight of the influence of multiple radar points on the adjacent pixel points.

[0109] Based on the above embodiment, the intensity parameters of adjacent pixel points can be calculated by comprehensively considering the radar point intensity value and the weights of adjacent pixel points. The intensity parameters obtained by this method can be used to perform subsequent image colorization, edge point extraction and other operations, thereby obtaining more accurate and reliable results.

[0110] Step S405: determining a first pixel value of the first adjacent pixel point according to the intensity parameter, and obtaining a first image based on the first pixel value.

[0111] In some embodiments, an intensity parameter can be used to determine the first pixel value of each first adjacent pixel point. The intensity parameter can be used as a key indicator here. Since it combines the intensity information of the radar point and the weight distribution of the pixel point, it can accurately reflect the actual intensity characteristics of the pixel point. For example, the first pixel value can be used to update the pixel value of the first adjacent pixel point in the preset image to obtain the first image. The pixel value of the pixel point in the preset image is a preset second value, such as 0, 255 or an intermediate grayscale value. In the following examples, the second value will be 255 representing black. For example Figure 5The figure shown is an example diagram of the first image provided in an embodiment of the present application.

[0112] Based on the above embodiment, the first image can not only present the projection position of the radar point with a pixel value position different from the second value, but also accurately present the intensity change of each pixel point caused by the influence of the projection position of the radar point through the difference in pixel values, which can provide a data basis for subsequent edge point extraction.

[0113] Step S406, determining a second adjacent pixel point from the first adjacent pixel points, determining a second pixel value of the second adjacent pixel point according to a preset first value, and obtaining a second image based on the second pixel value.

[0114] In some embodiments, among the first adjacent pixel points, with the pixel coordinate as the center, a pixel point located in a second range is selected as the second adjacent pixel point. For example, if the first range is 7×7, the second range may be 3×3, 5×5, etc. The pixel value of the second adjacent pixel point in the preset image is updated using the second pixel value to obtain the second image. The pixel value of the pixel point in the preset image is a preset second value.

[0115] Based on the above embodiment, the second adjacent pixel points can be screened out from the first adjacent pixel points, and the second pixel values ​​are assigned to the positions of these points in the preset image, thereby generating a second image. Compared with the first image, the second image can remove the effects of holes that may be caused by missing radar data or projection errors, and the overall quality and visual effect of the image are improved by filling the hole areas, and the image discontinuity caused by missing data is reduced.

[0116] Step S407: determining the target edge point based on the first edge point of the first image and the second edge point of the second image.

[0117] In some embodiments, edge extraction is performed on the first image to determine the first edge point. Figure 6 The figure shows an example of the position of the first edge point provided in an embodiment of the present application. The second image is subjected to edge extraction to determine the second edge point. In one example, an image processing algorithm is used to perform the above-mentioned edge extraction to identify significant edge features in the image. For example, the image processing algorithm includes but is not limited to the Canny edge detector, the Sobel operator or the Prewitt operator. Based on the intersection of the first edge point and the second edge point, the third edge point is determined. Based on the radar point corresponding to the third edge point in the fused point cloud, the target edge point is determined. For example Figure 7 As shown in the figure, it is an example diagram of the position of the target edge point provided in an embodiment of the present application.

[0118] Based on the above embodiment, the third edge point determined by the intersection of the first edge point set and the second edge point set includes points that appear on the edges of the first image and the second image at the same time, which can represent a more stable and reliable edge feature. In the fused point cloud, the target edge point is determined according to the radar point corresponding to the third edge point, and the edge feature point in the radar point can be accurately determined.

[0119] Figure 8 It is a flowchart of a three-dimensional reconstruction method provided by another embodiment of the present application. The three-dimensional reconstruction method includes: using the first frame image obtained by the camera as a reference or initial data, and performing intensity edge point line alignment based on the frame image and the corresponding radar data. Alternatively, if it is not the first frame image, the Lucas-Kanade optical flow method is used to track the feature points or edges in the image. Among them, when performing intensity edge point line alignment, the radar point cloud frame and the fused point cloud of the adjacent radar point cloud frame can be projected onto the image plane to obtain an intensity map corresponding to the fused point cloud in the pixel coordinate system corresponding to the image data, so as to compare and fuse with the image data corresponding to the radar data. The Canny edge detection algorithm is used to detect the edge in the intensity map to obtain an intensity edge map, for example, the intensity edge map contains the position of the intensity edge point cloud (such as the target edge point). Canny edge detection is performed on the RGB image (such as image data) taken by the camera to obtain a gradient map, which is used to indicate the rate of change of pixel values ​​in the image. The gradient map is searched for contours to obtain a contour map. Intensity edge point line alignment is performed based on the intensity edge point cloud map and the gradient map, and the image and point cloud are aligned to the same coordinate system. Among them, the forward propagation algorithm can be used to iteratively optimize the process of intensity edge point line registration. After the intensity edge point line registration, the reprojection error registration method can be used to further align the results of the intensity edge point line registration to achieve accurate alignment of three-dimensional-two-dimensional (3D-2D) data.

[0120] Based on the above embodiments, the initial frame image and radar data can be used to perform intensity edge point and line registration, or the Lucas-Kanade optical flow method can be used to track features in subsequent frames. During the registration process, the radar point cloud is fused and projected to the image plane, and the intensity edge map and image gradient map are obtained in combination with Canny edge detection, and then point and line registration is performed. The registration results are iteratively optimized through the forward propagation algorithm, and the reprojection error registration method is used to further accurately align, ensuring the accurate fusion of three-dimensional and two-dimensional data, achieving high-precision alignment of camera images and radar data, and improving the accuracy of three-dimensional reconstruction.

[0121] Fig. 9 is a structural diagram of a three-dimensional reconstruction device provided in one embodiment of the present application.

[0122] In some embodiments, the 3D reconstruction device 70 may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the 3D reconstruction device 70 may be stored in a memory of an electronic device and executed by at least one processor to perform (see Figure 2 Description) Function of three-dimensional reconstruction.

[0123] In this embodiment, the three-dimensional reconstruction device 70 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an acquisition module 701, a determination module 702, an update module 703, and a reconstruction module 704. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functional implementation of each module in the three-dimensional reconstruction device 70 can refer to the above definition of the three-dimensional reconstruction method, and will not be repeated here.

[0124] The acquisition module 701 is used to acquire radar point cloud frames and image data corresponding to the same scene.

[0125] The determination module 702 is used to determine the target edge point in the radar point cloud frame; determine the edge contour point in the image data, and determine the normal vector corresponding to the edge contour point; based on the target edge point, the edge contour point and the normal vector, determine the normal projection error between the target edge point and the edge contour point.

[0126] The updating module 703 is used to update the camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose.

[0127] The reconstruction module 704 is used to fuse the radar point cloud frame and the image data based on the target camera posture to obtain a reconstructed image.

[0128] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0129] The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0130] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0131] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specified application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specified application, but such implementation should not be considered to be beyond the scope of this application.

[0133] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Obtain radar point cloud frames and image data corresponding to the same scene; Determining target edge points in the radar point cloud frame; Determine edge contour points in the image data, and determine normal vectors corresponding to the edge contour points; Determining a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point and the normal vector; The camera pose corresponding to the image data is updated according to the normal projection error to obtain a target camera pose; The radar point cloud frame and the image data are fused based on the target camera posture to obtain a reconstructed image.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: Determining the target edge point in the radar point cloud frame comprises: A target edge point in the radar point cloud frame is determined according to an intensity difference between any radar point and an adjacent radar point in the radar point cloud frame, wherein the adjacent radar point represents a radar point located on the same radar scan line as the any radar point.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The adjacent radar points include a first radar point and a second radar point respectively located on both sides of the any radar point, and the determining of the target edge point in the radar point cloud frame according to the intensity difference between any radar point and the adjacent radar point in the radar point cloud frame includes: Determine a first intensity difference between the any radar point and the first radar point; Determine a second intensity difference between the any radar point and the second radar point; If the first intensity difference is greater than a first threshold and the second intensity difference is less than a second threshold, it is determined that any one of the radar points and the first radar point are the target edge points.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The method further comprises: The first threshold and the second threshold are determined according to the intensity value of any radar point.

5. The three-dimensional reconstruction method according to claim 1, characterized in that: Determining the target edge point in the radar point cloud frame comprises: Acquire a fused point cloud of the radar point cloud frame and adjacent radar point cloud frames; Determine the pixel coordinates of a first projection point corresponding to any radar point in the fused point cloud in a pixel coordinate system, wherein the pixel coordinate system represents a coordinate system corresponding to the image data; Based on the pixel coordinates, determining a first neighboring pixel point of the first projection point in the pixel coordinate system, and determining a weight of the first neighboring pixel point; Determining an intensity parameter of the first adjacent pixel point based on the weight and the intensity value of any radar point; Determine a first pixel value of the first adjacent pixel point according to the intensity parameter, and obtain a first image based on the first pixel value; Determine a second adjacent pixel point from the first adjacent pixel points, determine a second pixel value of the second adjacent pixel point according to a preset first value, and obtain a second image based on the second pixel value; The target edge point is determined based on the first edge point of the first image and the second edge point of the second image.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: The step of obtaining a fused point cloud of the radar point cloud frame and adjacent radar point cloud frames includes: The radar point cloud frame and adjacent radar point cloud frames are projected to a world coordinate system, and the radar point cloud frame and adjacent radar point cloud frames are fused in the world coordinate system to obtain the fused point cloud.

7. The three-dimensional reconstruction method according to claim 6, characterized in that: Determining the pixel coordinates of the first projection point corresponding to any radar point in the fused point cloud in the pixel coordinate system comprises: Based on the camera pose corresponding to the image data, projecting the radar point in the fused point cloud to the camera coordinate system corresponding to the image data, and determining the coordinates of a first projection point of the radar point in the camera coordinate system; Based on the intrinsic parameter matrix of the camera corresponding to the image data, the coordinates of the first projection point are converted to the pixel coordinate system to obtain the pixel coordinates.

8. The three-dimensional reconstruction method according to claim 5, characterized in that: The step of determining, based on the pixel coordinates, a first adjacent pixel point of the first projection point in the pixel coordinate system, and determining a weight of the first adjacent pixel point includes: Taking the pixel coordinates as the center, select a pixel point within a first range as the first adjacent pixel point; An influence factor of any radar point on the first neighboring pixel point is determined according to the coordinates of the first neighboring pixel point and the pixel coordinates, and a weight of the first neighboring pixel point is determined based on the influence factor.

9. The three-dimensional reconstruction method according to claim 8, characterized in that: The method further comprises: If there is a corresponding relationship between the first adjacent pixel point and multiple radar points in the fused point cloud, determine an influence factor of each radar point on the first adjacent pixel point; Determine a first accumulated value of influence factors of the multiple radar points on the first adjacent pixel point, and determine a weight of the first adjacent pixel point according to the first accumulated value.

10. The three-dimensional reconstruction method according to claim 5, characterized in that: The determining, based on the weight and the intensity value of any radar point, the intensity parameter of the first adjacent pixel point includes: An intensity parameter of the first adjacent pixel point is determined according to a ratio of the intensity value of any radar point to the weight.

11. The three-dimensional reconstruction method according to claim 10, characterized in that: The method further comprises: If there is a corresponding relationship between the first adjacent pixel point and the multiple radar points in the fused point cloud, determine a second accumulated value of the intensity values ​​of the multiple radar points; An intensity parameter of the first adjacent pixel is determined according to a ratio of the second accumulated value to the weight.

12. The three-dimensional reconstruction method according to claim 5, characterized in that: The determining a first pixel value of the first adjacent pixel point according to the intensity parameter, and obtaining a first image based on the first pixel value, comprises: The first pixel value is used to update the pixel value of the first adjacent pixel point in the preset image to obtain the first image, wherein the pixel value of the pixel point in the preset image is a preset second value.

13. The three-dimensional reconstruction method according to claim 5, characterized in that: The determining of a second adjacent pixel point among the first adjacent pixel points, determining a second pixel value of the second adjacent pixel point according to a preset first value, and obtaining a second image based on the second pixel value includes: Among the first adjacent pixel points, taking the pixel coordinates as the center, select pixel points located in a second range as the second adjacent pixel points; The pixel value of the second adjacent pixel point in the preset image is updated by using the second pixel value to obtain the second image, wherein the pixel value of the pixel point in the preset image is a preset second value.

14. The three-dimensional reconstruction method according to claim 5, characterized in that: The determining the target edge point based on the first edge point of the first image and the second edge point of the second image includes: Performing edge extraction on the first image to determine the first edge point; Performing edge extraction on the second image to determine the second edge point; Determine a third edge point according to the intersection of the first edge point and the second edge point; The target edge point is determined according to the radar point corresponding to the third edge point in the fused point cloud.

15. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of determining edge contour points in the image data and determining normal vectors corresponding to the edge contour points includes: Acquire a grayscale image corresponding to the image data, and perform edge extraction on the grayscale image to obtain the edge contour points; The normal vector corresponding to the edge contour point is determined based on a principal component analysis algorithm.

16. The three-dimensional reconstruction method according to claim 1, characterized in that: Determining a normal projection error between the target edge point and the edge contour point based on the target edge point cloud, the edge contour point and the normal vector includes: Projecting the target edge point cloud to a pixel coordinate system corresponding to the image data, and determining a second projection point of a radar point in the target edge point cloud in the pixel coordinate system; The normal projection error is determined based on the distance between the edge contour point and the second projection point in the normal vector direction.

17. The three-dimensional reconstruction method according to claim 1, characterized in that: The updating of the camera pose corresponding to the image data according to the normal projection error comprises: The camera pose is updated based on a preset optimization objective, wherein the optimization objective includes minimizing the normal projection error.

18. The three-dimensional reconstruction method according to claim 1, characterized in that: After updating the camera pose corresponding to the image data according to the normal projection error, the method further includes: A reprojection error between the radar point cloud frame and the image data is determined, and the camera pose is updated according to the reprojection error to obtain the target camera pose.

19. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the three-dimensional reconstruction method according to any one of claims 1 to 18.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 18 is implemented.

Images