3D Reconstruction Method and Related Devices

Abstract

The embodiment of the present application provides a three-dimensional reconstruction method and related equipment, the method comprising: obtaining radar point cloud frames and image data corresponding to the same scene; determining target edge points in the radar point cloud frames; determining edge contour points in the image data and determining normal vectors corresponding to the edge contour points; determining normal projection errors between the target edge points and the edge contour points based on the target edge points, the edge contour points, and the normal vectors; updating the camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose; and fusing the radar point cloud frames and the image data based on the target camera pose to obtain a reconstructed image. The above method can be used to improve the accuracy and robustness of three-dimensional reconstruction.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular, to a three-dimensional reconstruction method and related equipment. Background Art

[0002] The LIVO (LiDAR-Inertial-Vision-Odometry) system, also known as "LiDAR-Inertial-Visual Odometry" or "Light Detection and Ranging-Inertial Navigation-Visual Positioning System") system combines sensor fusion technology from LiDAR, an inertial measurement unit, and camera visual data to estimate and track the position and posture of mobile devices or vehicles. To achieve high-precision positioning and mapping, it is typically necessary to fuse the three-dimensional point cloud geometry of the LiDAR with the two-dimensional image texture information of the camera. However, due to the asynchronous data acquisition between the camera and LiDAR, the discrete nature of radar data, and other reasons, related technologies have difficulty achieving accurate fusion of radar data and image data. This makes it difficult to accurately align the geometry and texture of the reconstructed data, thus affecting the accuracy and usability of positioning and mapping. Summary of the Invention

[0003] The present application discloses a three-dimensional reconstruction method and related equipment, which can improve the fusion accuracy of radar data and image data, as well as the alignment accuracy of geometry and texture in the reconstructed data.

[0004] A first aspect of an embodiment of the present application provides a three-dimensional reconstruction method, including: obtaining a radar point cloud frame and image data corresponding to the same scene; determining a target edge point in the radar point cloud frame; determining an edge contour point in the image data, and determining a normal vector corresponding to the edge contour point; determining a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point, and the normal vector; updating a camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose; and fusing the radar point cloud frame and the image data based on the target camera pose to obtain a reconstructed image.

[0005] According to an embodiment of the present application, determining the target edge point in the radar point cloud frame includes: determining the target edge point in the radar point cloud frame based on an intensity difference between any radar point and an adjacent radar point in the radar point cloud frame, wherein the adjacent radar point represents a radar point located on the same radar scan line as the any radar point.

[0006] According to an embodiment of the present application, the adjacent radar points include a first radar point and a second radar point respectively located on both sides of the any radar point. Determining the target edge point in the radar point cloud frame based on the intensity difference between any radar point and the adjacent radar points in the radar point cloud frame includes: determining a first intensity difference between the any radar point and the first radar point; determining a second intensity difference between the any radar point and the second radar point; and if the first intensity difference is greater than a first threshold and the second intensity difference is less than a second threshold, determining that the any radar point and the first radar point are the target edge points.

[0007] According to an embodiment of the present application, the method further includes: determining the first threshold and the second threshold according to the intensity value of any radar point.

[0008] According to an embodiment of the present application, determining the target edge point in the radar point cloud frame includes: obtaining a fused point cloud of the radar point cloud frame and an adjacent radar point cloud frame; determining the pixel coordinates of a first projection point corresponding to any radar point in the fused point cloud in a pixel coordinate system, wherein the pixel coordinate system represents a coordinate system corresponding to the image data; based on the pixel coordinates, determining a first neighboring pixel point of the first projection point in the pixel coordinate system, and determining a weight of the first neighboring pixel point; determining an intensity parameter of the first neighboring pixel point based on the weight and an intensity value of the any radar point; determining a first pixel value of the first neighboring pixel point based on the intensity parameter, and obtaining a first image based on the first pixel value; determining a second neighboring pixel point from the first neighboring pixel point, determining a second pixel value of the second neighboring pixel point based on a preset first value, and obtaining a second image based on the second pixel value; and determining the target edge point based on the first edge point of the first image and the second edge point of the second image.

[0009] According to an embodiment of the present application, obtaining the fused point cloud of the radar point cloud frame and the adjacent radar point cloud frames includes: projecting the radar point cloud frame and the adjacent radar point cloud frames into a world coordinate system, and fusing the radar point cloud frame and the adjacent radar point cloud frames in the world coordinate system to obtain the fused point cloud.

[0010] According to an embodiment of the present application, determining the pixel coordinates of the first projection point corresponding to any radar point in the fused point cloud in the pixel coordinate system includes: projecting the radar point in the fused point cloud to the camera coordinate system corresponding to the image data based on the camera pose corresponding to the image data, and determining the coordinates of the first projection point of the radar point in the camera coordinate system; based on the intrinsic parameter matrix of the camera corresponding to the image data, converting the coordinates of the first projection point to the pixel coordinate system to obtain the pixel coordinates.

[0011] According to an embodiment of the present application, determining the first adjacent pixel point of the first projection point in the pixel coordinate system based on the pixel coordinates, and determining the weight of the first adjacent pixel point, includes: selecting pixels within a first range centered on the pixel coordinates as the first adjacent pixel points; determining an influence factor of any radar point on the first adjacent pixel point based on the coordinates of the first adjacent pixel point and the pixel coordinates, and determining the weight of the first adjacent pixel point based on the influence factor.

[0012] According to an embodiment of the present application, the method further includes: if the first adjacent pixel point has a corresponding relationship with multiple radar points in the fused point cloud, determining an influence factor of each radar point on the first adjacent pixel point; determining a first cumulative value of the influence factors of the multiple radar points on the first adjacent pixel point, and determining a weight of the first adjacent pixel point based on the first cumulative value.

[0013] According to an embodiment of the present application, determining the intensity parameter of the first adjacent pixel point based on the weight and the intensity value of any radar point includes: determining the intensity parameter of the first adjacent pixel point based on a ratio of the intensity value of any radar point to the weight.

[0014] According to an embodiment of the present application, the method further includes: if the first adjacent pixel point corresponds to multiple radar points in the fused point cloud, determining a second accumulated value of the intensity values of the multiple radar points; and determining an intensity parameter of the first adjacent pixel point based on a ratio of the second accumulated value to the weight.

[0015] According to an embodiment of the present application, determining the first pixel value of the first adjacent pixel point according to the intensity parameter and obtaining the first image based on the first pixel value includes: using the first pixel value to update the pixel value of the first adjacent pixel point in the preset image to obtain the first image, wherein the pixel value of the pixel point in the preset image is a preset second value.

[0016] According to an embodiment of the present application, determining the second adjacent pixel point among the first adjacent pixel points, determining the second pixel value of the second adjacent pixel point according to a preset first numerical value, and obtaining a second image based on the second pixel value includes: among the first adjacent pixel points, selecting a pixel point located within a second range with the pixel coordinates as the center as the second adjacent pixel point; using the second pixel value to update the pixel value of the second adjacent pixel point in the preset image to obtain the second image, wherein the pixel value of the pixel point in the preset image is the preset second numerical value.

[0017] According to an embodiment of the present application, determining the target edge point based on the first edge point of the first image and the second edge point of the second image includes: performing edge extraction on the first image to determine the first edge point; performing edge extraction on the second image to determine the second edge point; determining a third edge point based on the intersection of the first edge point and the second edge point; and determining the target edge point based on the radar point corresponding to the third edge point in the fused point cloud.

[0018] According to an embodiment of the present application, determining the edge contour points in the image data and determining the normal vectors corresponding to the edge contour points include: obtaining a grayscale image corresponding to the image data, performing edge extraction on the grayscale image to obtain the edge contour points; and determining the normal vectors corresponding to the edge contour points based on a principal component analysis algorithm.

[0019] According to an embodiment of the present application, determining a normal projection error between the target edge point and the edge contour point based on the target edge point cloud, the edge contour point, and the normal vector includes: projecting the target edge point cloud to a pixel coordinate system corresponding to the image data, and determining a second projection point of the radar point in the target edge point cloud in the pixel coordinate system; and determining the normal projection error based on a distance between the edge contour point and the second projection point in the direction of the normal vector.

[0020] According to an embodiment of the present application, updating the camera pose corresponding to the image data according to the normal projection error includes: updating the camera pose based on a preset optimization target, wherein the optimization target includes minimizing the normal projection error.

[0021] According to an embodiment of the present application, after updating the camera pose corresponding to the image data according to the normal projection error, the method further includes: determining the reprojection error between the radar point cloud frame and the image data, updating the camera pose according to the reprojection error, and obtaining the target camera pose.

[0022] A second aspect of an embodiment of the present application provides a three-dimensional reconstruction device, which includes: an acquisition module for acquiring radar point cloud frames and image data corresponding to the same scene; a determination module for determining target edge points in the radar point cloud frames; determining edge contour points in the image data, and determining normal vectors corresponding to the edge contour points; determining normal projection errors between the target edge points and the edge contour points based on the target edge points, the edge contour points, and the normal vectors; an update module for updating a camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose; and a reconstruction module for fusing the radar point cloud frames and the image data based on the target camera pose to obtain a reconstructed image.

[0023] A third aspect of an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the three-dimensional reconstruction method.

[0024] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional reconstruction method is implemented.

[0025] The 3D reconstruction method provided in the embodiments of the present application obtains radar point cloud frames and image data of the same scene, accurately locates the target edge points in the radar point cloud and the edge contour points and their normal vectors in the image, and then calculates the normal projection error between the two. This error is used to optimize and update the camera pose corresponding to the image data until the accurate target camera pose is obtained. Based on this precise pose information, the radar point cloud and image data are fused to generate detailed and accurate reconstructed images, providing strong support for the visualization and understanding of the 3D environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 A schematic diagram of an application environment of a three-dimensional reconstruction method provided in one embodiment of the present application.

[0028] Figure 2 A schematic flow chart of a three-dimensional reconstruction method provided in one embodiment of the present application.

[0029] Figure 3This is an example diagram of adjacent radar points provided by an embodiment of the present application.

[0030] Figure 4 A flowchart of a method for determining target edge points provided in one embodiment of the present application.

[0031] Figure 5 This is an example diagram of the first image provided in an embodiment of the present application.

[0032] Figure 6 This is an example diagram of a first edge point provided in one embodiment of the present application.

[0033] Figure 7 This is an example diagram of target edge points provided in one embodiment of the present application.

[0034] Figure 8 A flowchart of a three-dimensional reconstruction method provided in another embodiment of the present application.

[0035] Figure 9 A structural diagram of a three-dimensional reconstruction device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of this application clearer, this application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] It should be noted that, in this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The terms "first," "second," "third," "fourth," and so on (if any) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or precedence.

[0038] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner. The following embodiments and features in the embodiments may be combined with each other unless there is a conflict.

[0039] 3D reconstruction technology has widespread applications in autonomous driving, robotic navigation, augmented reality, and other fields. To achieve high-precision 3D scene reconstruction, multi-sensor fusion systems, such as LIVO (LiDAR-Inertial-Visual Odometry), are widely adopted. These systems combine multimodal data from LiDAR, an inertial measurement unit (IMU), and a camera to generate geometrically and texturally consistent 3D maps.

[0040] However, existing technologies still face the following challenges in practical applications: Sensor extrinsic calibration error: The extrinsic calibration error between the camera and the lidar will cause the camera's two-dimensional pixels and three-dimensional geometric features to be unable to be accurately aligned in the initial state, thereby affecting the subsequent camera pose optimization and map reconstruction accuracy; Sensor data asynchrony problem: Due to the different data acquisition frequencies of the camera and the lidar and the problem of timestamp asynchrony, it is difficult to directly align multimodal data, further increasing the difficulty of pose estimation; IMU accuracy limitation: In the filter-based LIVO system, IMU data is used for pre-integration to estimate the sensor pose, but the accuracy of the IMU will decrease over time, and its discrete data characteristics will introduce cumulative errors, affecting the long-term stability of the system; Insufficient geometric and texture alignment: Existing methods usually rely on rigid registration algorithms (such as the iterative closest point algorithm (ICP)) for geometric alignment of lidar point clouds, but due to the lack of accurate three-dimensional-two-dimensional (3D-2D) correspondence, the optimization effect of the camera pose is limited, resulting in poor geometric and texture consistency of the reconstructed map.

[0041] To address the above-mentioned issues, existing technologies have proposed a variety of solutions, but they still have obvious shortcomings. For example, the initial stationary assumption method achieves 3D-2D alignment by assuming that the system is stationary in its initial state, but this assumption is difficult to meet in practical applications (such as handheld scanning devices) and limits the applicability of the system. Methods that require hardware-synchronized data acquisition obtain accurate initial poses by hard-synchronizing camera and lidar data, but this requires additional hardware support and places extremely high demands on the accuracy of external parameter calibration. Some methods use offline processing (such as the SFM (Structure from Motion) reconstruction algorithm) combined with point-to-surface ICP constraints to optimize geometric and texture consistency, but this method cannot meet real-time requirements and has low computational efficiency in large scenes.

[0042] To solve the above problems, the three-dimensional reconstruction method provided in the embodiment of the present application combines the edge features of the radar point cloud with the normal vector information of the image edge contour, calculates the normal projection error and optimizes the camera pose, thereby achieving high-precision alignment of geometry and texture, and improving the accuracy and robustness of three-dimensional reconstruction.

[0043] See also Figure 1 , is a schematic diagram of an application environment of a three-dimensional reconstruction method provided by an embodiment of the present application. Figure 1 As shown, the three-dimensional reconstruction method provided in the embodiment of the present application can be applied to an electronic device 10, which can be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, an energy storage device, a power distribution device, a vehicle-mounted device, a self-moving device, a scanner, or other electronic device. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0044] like Figure 1 As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102, and the I / O interface 104 via the bus 105.

[0045] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB) and controller area network (CAN). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0046] Memory 102 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The RAM can be directly read and written by the processor 103 and can be used to store executable programs (e.g., machine instructions) for the operating system or other running programs, as well as user and application data. RAM may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0047] The non-volatile memory can also store executable programs and user and application data, etc., and can be pre-loaded into the random access memory for direct reading and writing by the processor 103. The non-volatile memory can include disk storage devices and flash memory.

[0048] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the three-dimensional reconstruction method executed on the electronic device 10 can be implemented.

[0049] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0050] The processor 103 may include one or more processing units. For example, the processor 103 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0051] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute the computer program stored in the memory 102 to implement the above-mentioned three-dimensional reconstruction method.

[0052] The I / O interface 104 provides a channel for user input and output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch screen device, and display screen, allowing users to enter information or visualize information. Furthermore, the I / O interface 104 can also be used to connect various sensor devices, such as radar sensors and image sensors, to obtain required radar point cloud data, image data, and other data.

[0053] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 10 .

[0054] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0055] The following describes a computer program product that implements the three-dimensional reconstruction method of the present application and runs on an electronic device (such as Figure 1 The electronic device 10 shown in FIG. Figure 2 FIG. 1 is a flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application. In one embodiment of the present application, the method includes the following steps:

[0056] Step S201: Acquire radar point cloud frames and image data corresponding to the same scene.

[0057] In some embodiments, radar point cloud frames and image data of the same scene can be collected simultaneously. A radar point cloud frame can be obtained from a radar sensor, and the radar point cloud frame contains the three-dimensional coordinates (x, y, z) and intensity value of each radar point. The Z coordinate value can represent the depth value of the radar point, which can indicate the distance between the radar sensor and the object, thereby helping to construct the three-dimensional shape of the object during the three-dimensional reconstruction process. The intensity value can indicate the reflection intensity of the laser beam on the surface of the measured object in the scene, and can reflect the physical properties of the object's surface, such as roughness, tilt angle, and reflectivity, so that the reconstructed three-dimensional image can truly reflect the material and shape differences of the object's surface.

[0058] Image data, such as color or grayscale images, can be acquired from an image sensor. This image data can contain appearance characteristics of objects in the scene, such as color, brightness, and texture details. By aligning and fusing this image data with radar point cloud frames, the texture information in the image can be mapped onto the geometric surface corresponding to the radar point cloud frame, allowing a 3D model to be reconstructed based on the combined intensity values and texture information.

[0059] Step S202: Determine target edge points in the radar point cloud frame.

[0060] In some embodiments, target edge points represent radar points in a radar point cloud frame that have significant intensity differences. Two different methods can be used to determine target edge points in the radar point cloud frame. The first method includes determining target edge points in the radar point cloud frame based on intensity differences between any radar point and adjacent radar points in the radar point cloud frame, where the adjacent radar points are radar points located on the same radar scan line as the radar point.

[0061] In some embodiments, the adjacent radar points include a first radar point and a second radar point located on either side of the any radar point, respectively. Determining the target edge point in the radar point cloud frame based on the intensity difference between any radar point and the adjacent radar points in the radar point cloud frame includes: determining a first intensity difference between the any radar point and the first radar point; determining a second intensity difference between the any radar point and the second radar point; and determining that the any radar point and the first radar point are the target edge points if the first intensity difference is greater than a first threshold and the second intensity difference is less than a second threshold. The first threshold and the second threshold can be determined based on the intensity value of the any radar point.

[0062] In one example, reference Figure 3The figure shows an example diagram of adjacent radar points provided by an embodiment of the present application. For any radar point b in a radar point cloud frame, its adjacent radar points include a first radar point a and a second radar point c located on the same radar scan line (i.e., the same laser beam scan circle). The first intensity difference between point b and the first radar point a is calculated as ΔIab=|Ib-Ia|, where Ib represents the intensity value of point b and Ia represents the intensity value of point a. The second intensity difference between point b and the second radar point c is calculated as ΔIbc=|Ib-Ic|, where Ic represents the intensity value of point c.

[0063] If the first intensity difference and the second intensity difference meet the following two conditions: ΔIab>δmax, ΔIbc<δmin, then point b and the first radar point a can be determined to be target edge points. δmax represents the first threshold, for example, δ max=0.4×Ib; δmin represents the second threshold, for example δ min = 0.1 × Ib. For example, if the intensity of point b is Ib = 100, δmax = 40 and δmin = 10. If ΔIab = 50 and ΔIbc = 8, then a and b are identified as edge points. If ΔIab > δmax, it can be determined that there is a significant intensity jump between the first radar point a and point b. If ΔIbc < δmin, it can be determined that the intensity change between point b and the second radar point c is smooth. Thus, point b and the first radar point, which have a significant intensity difference, can be identified as target edge points.

[0064] In one example, all radar points on each radar scan line of the radar point cloud frame can be traversed and target edge points can be extracted according to the method provided in the above embodiment. For example, for points d and e on the scan line, if the corresponding conditions are met, such as ΔIde>δmax and ΔIef<δmin, then d and e can also be determined as target edge points.

[0065] Based on the above embodiment, dynamic threshold setting can be used to avoid the problem of fixed thresholds being insufficiently adaptable to different reflection intensity scenarios. Combining the dual-condition judgment of adjacent point intensity differences can effectively distinguish between true edges and local intensity fluctuations caused by noise, thereby accurately identifying target edge points.

[0066] In some embodiments, a second method that can be used when determining target edge points in a radar point cloud frame includes: obtaining a fused point cloud of a radar point cloud frame and an adjacent frame, and projecting the radar point in the fused point cloud into the pixel coordinate system of the image data to determine its corresponding first projection point and its pixel coordinates; finding the first adjacent pixel point of the first projection point based on the pixel coordinates and calculating its weight, combining the weight with the radar point intensity value to determine the intensity parameter and the first pixel value of the first adjacent pixel point to generate a first image; determining the second adjacent pixel point from the first adjacent pixel point, and setting its second pixel value according to a preset value to generate a second image; combining the edge points of the first image with the edge points of the second image to determine the target edge point. In one example, the specific implementation steps of the above method include the following: Figure 4 The process shown.

[0067] Based on the above embodiments, two different methods can be used to accurately determine target edge points in radar point cloud frames. These target edge points represent radar points with significant intensity variations within the radar point cloud frames and play a crucial role in the 3D reconstruction process. Specifically, these target edge points can serve as key feature points, assisting in achieving more accurate camera pose estimation, thereby significantly improving the overall 3D reconstruction effect.

[0068] Step S203: determining edge contour points in the image data, and determining normal vectors corresponding to the edge contour points.

[0069] In some embodiments, when determining edge contour points in the image data, a grayscale image corresponding to the image data can be obtained, and edge extraction can be performed on the grayscale image to obtain the edge contour points. In one example, the edge extraction is performed using an image processing algorithm to identify significant edge feature points in the image as edge contour points. For example, image processing algorithms include, but are not limited to, the Canny edge detector, the Sobel operator, or the Prewitt operator. In this way, the color information in the image data can be simplified using the grayscale image, thereby improving the accuracy of edge extraction in the image.

[0070] Based on the above embodiment, target edge points in radar points often correspond to geometric edges or texture edges in the corresponding image data. By accurately identifying these edge contour points in the image data, the precise corresponding points of the edge feature points in the radar points in the image data can be further determined, thereby providing corresponding edge point pair information for subsequent 3D reconstruction of radar point cloud frames and image data, helping to achieve high-quality 3D reconstruction.

[0071] In some embodiments, when determining the normal vector corresponding to the edge contour point, the normal vector corresponding to the edge contour point can be determined based on a principal component analysis algorithm. In one example, a local window centered on the edge contour point can be used, and the range within the window is used as the neighborhood of the edge contour point, and the pixel points within the window are used as the neighborhood points of the edge contour point. Based on the principal component analysis algorithm, two main directions within the neighborhood of the edge contour point are determined, including the direction along the edge (tangent vector) and the direction perpendicular to the edge (normal vector). The direction of the normal vector is obtained by calculating the covariance matrix of the two main directions and performing eigenvalue decomposition. The normal vector of this direction is normalized, for example, the length of the normal vector is adjusted to 1, so that it only represents the direction but does not contain length information.

[0072] Based on the above embodiments, by determining edge contour points and their corresponding normal vectors in image data, it is possible to more accurately estimate the shape and texture of the surface of an object in three-dimensional reconstruction, thereby achieving better three-dimensional reconstruction effects.

[0073] Step S204 : determining a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point, and the normal vector.

[0074] In some embodiments, when determining the normal projection error between the target edge point and the edge contour point, the target edge point cloud may be projected into a pixel coordinate system corresponding to the image data to determine a second projection point of the radar point in the target edge point cloud in the pixel coordinate system. The normal projection error is determined based on the distance between the edge contour point and the second projection point in the direction of the normal vector.

[0075] In some embodiments, when determining the second projection point, since the target edge point in the fused point cloud has coordinates in the world coordinate system, the coordinates of the target edge point in the world coordinate system can be converted to the camera coordinate system based on the camera pose, thereby obtaining the coordinates of the target edge point in the camera coordinate system. Subsequently, the coordinates of the target edge point in the camera coordinate system can be converted to the pixel coordinate system based on the camera intrinsic parameters, thereby determining the second projection point of the radar point in the target edge point cloud in the pixel coordinate system. The position of the second projection point can indicate the precise projection position of the target edge point on the image plane.

[0076] In some embodiments, when determining the normal projection error, the distance between each second projection point and the nearest edge contour point in the normal vector direction can be calculated using known edge contour points and their corresponding normal vector information. For example, the formula used can be exemplarily expressed as:

[0077] ,

[0078] in, Represents the coordinates of the second projection point proj corresponding to the target edge point j, represents the coordinates of the i-th nearest edge contour point corresponding to the second projection point, represents the normal vector of the i-th nearest edge contour point, k represents the k nearest edge contour points corresponding to the second projection point. E represents the normal projection error between the target edge point j and the edge contour point, which can be used to indicate the inconsistency between the target edge point and the image edge contour point in the normal vector direction.

[0079] Based on the above embodiment, the calculation of the normal projection error not only considers the spatial relationship between the target edge points and the edge contour points, but also fully utilizes the normal vector information of the edge contour points, thereby providing a more accurate and comprehensive error assessment. This error metric can be used to measure the quality of the fusion between radar data and image data, guiding subsequent 3D reconstruction tasks and achieving high-quality 3D reconstruction.

[0080] Step S205 : updating the camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose.

[0081] In some embodiments, updating the camera pose corresponding to the image data according to the normal projection error includes: updating the camera pose based on a preset optimization target, wherein the optimization target includes minimizing the normal projection error.

[0082] In one example, the core optimization objective for the normal projection error involves minimizing the normal projection error. For example, the camera pose is adjusted so that, when target edge points in the radar point cloud frame are projected onto the image data, the distance from the edge contour points in the image in the normal direction (i.e., the normal projection error) is minimized. This optimization objective can be achieved using a pre-defined optimization algorithm, such as the iterative closest point algorithm or the bundle adjustment algorithm. These algorithms iteratively adjust the camera pose based on the current normal projection error until a solution is reached that satisfies pre-defined conditions, such as the normal projection error being less than a pre-defined threshold or the number of iterations reaching an upper limit. To maintain the continuity and smoothness of the camera pose and avoid image distortion or inaccurate 3D reconstruction results due to sudden changes in the pose, pre-defined regularization terms or constraints can be introduced during the camera pose update to ensure the stability and reliability of the optimization process.

[0083] Based on the above embodiment, the camera pose can be accurately updated based on the normal projection error, thereby obtaining a more accurate target camera pose. This step can provide more reliable data support for subsequent data fusion and 3D reconstruction tasks, thereby improving the performance and accuracy of the entire system.

[0084] In some embodiments, after updating the camera pose corresponding to the image data according to the normal projection error, the method further includes: determining a reprojection error between the radar point cloud frame and the image data, and updating the camera pose according to the reprojection error to obtain the target camera pose.

[0085] In one example, after updating the camera pose corresponding to the image data based on the normal projection error, the calculated reprojection error can be used to set an optimization objective for the reprojection error. The camera pose can then be further adjusted based on the optimization objective to obtain a target camera pose. The optimization objective for the reprojection error can include minimizing the reprojection error, for example, ensuring that radar points projected onto the image are as close as possible to corresponding points in the image, thereby achieving precise alignment of three-dimensional (3D-2D) correspondences.

[0086] Based on the above embodiments, the camera pose can be updated by combining the normal projection error and the reprojection error to obtain a more accurate and robust target camera pose, thereby improving the accuracy of tasks such as 3D reconstruction.

[0087] Step S206: fusing the radar point cloud frame and the image data based on the target camera pose to obtain a reconstructed image.

[0088] In some embodiments, after obtaining the target camera pose, radar point cloud frame, and image data, the target camera pose can be used to project the points in the radar point cloud frame onto the camera's image plane. The projected radar point cloud data can be fused with the image data. For example, the fusion can be a simple overlay or more complex processing, such as coloring or enhancing the image based on the depth value of the radar point. The fused data can be further processed to generate a reconstructed image. The reconstructed image can be an enhanced image that combines radar depth information and image texture information, or it can be a completely new 3D reconstruction result extracted from the radar and image data. The generated reconstructed image can undergo some post-processing steps, such as denoising, smoothing, and enhancement, to improve the quality and readability of the reconstructed image.

[0089] The 3D reconstruction method provided in the embodiments of the present application obtains radar point cloud frames and image data of the same scene, accurately locates the target edge points in the radar point cloud and the edge contour points and their normal vectors in the image, and then calculates the normal projection error between the two. This error is used to optimize and update the camera pose corresponding to the image data until the accurate target camera pose is obtained. Based on this precise pose information, the radar point cloud and image data are fused to generate detailed and accurate reconstructed images, providing strong support for the visualization and understanding of the 3D environment.

[0090] In some embodiments, reference Figure 4 As shown, the method for determining the target edge point in the radar point cloud frame includes the following process:

[0091] Step S401: Acquire a fused point cloud of the radar point cloud frame and adjacent radar point cloud frames.

[0092] In some embodiments, the time node at which the current radar point cloud frame was acquired can be determined, and multiple consecutive radar point cloud frames acquired before the time node can be used as adjacent radar point cloud frames to the current radar point cloud frame. For example, the 20 radar point cloud frames preceding the current radar point cloud frame can be used as adjacent radar point cloud frames.

[0093] In some embodiments, to effectively fuse different radar point cloud frames, a first transformation relationship between the radar coordinate system corresponding to the radar point cloud frame and the world coordinate system can be obtained, such as a first rotation matrix and a first translation vector. Using this first transformation relationship, the radar point cloud frame and adjacent radar point cloud frames can be projected into the world coordinate system, and then fused with the adjacent radar point cloud frames in the world coordinate system to obtain the fused point cloud. In this way, each radar point in the fused point cloud has corresponding coordinates in the world coordinate system.

[0094] Based on the above embodiment, by obtaining the current radar point cloud frame and multiple adjacent radar point cloud frames, the deficiency of single-frame data can be avoided in the subsequent analysis process, and the integrity and accuracy of the data can be improved. The fusion of multiple frames of data can provide richer scene information, help to more accurately describe the shape, position, texture and other features of the object, and help to achieve more accurate three-dimensional reconstruction.

[0095] Furthermore, because different radar point cloud frames may correspond to different radar coordinate systems, fusing multiple frames directly in the radar coordinate system can lead to coordinate mismatches and data dislocations. By fusing multiple frames of radar data in a unified world coordinate system, we can eliminate issues such as inconsistent coordinate systems, error accumulation, and a lack of a global perspective, improving the reliability and practicality of the fused data.

[0096] Step S402: Determine the pixel coordinates of a first projection point corresponding to any radar point in the fused point cloud in a pixel coordinate system.

[0097] In some embodiments, the pixel coordinate system represents the coordinate system corresponding to the image data. To determine the pixel coordinates of the first projection point corresponding to any radar point in the fused point cloud in the pixel coordinate system, a projection transformation can be performed based on the camera pose corresponding to the image data and the intrinsic parameter matrix.

[0098] First, based on the camera pose corresponding to the image data, the radar points in the fused point cloud can be projected into the camera coordinate system corresponding to the image data, and the coordinates of the first projection point of the radar points in the camera coordinate system can be determined. The camera pose may include the transformation relationship between the camera coordinate system and the world coordinate system, such as a translation vector and a rotation matrix. Based on the camera pose, the fused point cloud in the world coordinate system can be transformed into the camera coordinate system, thereby obtaining the coordinates of the first projection point of the radar points in the fused point cloud in the camera coordinate system.

[0099] Subsequently, the coordinates of the first projection point can be converted to the pixel coordinate system based on the intrinsic parameter matrix of the camera corresponding to the image data to obtain the pixel coordinates. The intrinsic parameter matrix includes internal parameters of the camera, such as the focal length and optical center position. These parameters can reflect how points in the camera coordinate system are mapped to the pixel coordinate system. Therefore, the coordinates in the camera coordinate system can be converted to the pixel coordinate system based on the intrinsic parameter matrix to obtain the pixel coordinates.

[0100] In one example, when projecting three-dimensional radar points onto a two-dimensional pixel coordinate system, multiple radar points with different depth values may be projected onto the same pixel coordinate location (e.g., the same pixel). This can result in multiple depth values corresponding to the same pixel, leading to occlusion and data confusion between the multiple radar points. To avoid this, for multiple radar points projected onto the same pixel, only the radar point with the smallest depth value greater than 0 is retained. The radar point with the smallest depth value represents the point closest to the camera, and retaining this radar point is more consistent with the principle of visual perception that closer objects visually occlude farther objects. Furthermore, requiring depth values to be greater than 0 eliminates invalid or anomalous depth values caused by noise or errors.

[0101] Based on the above embodiment, the radar points in the fused point cloud can be projected from the world coordinate system to the camera coordinate system using the camera pose associated with the image data, obtaining the coordinates of the first projection point of the radar point in the camera coordinate system. Subsequently, the coordinates of the first projection point are further converted to the pixel coordinate system using the camera's intrinsic parameter matrix, thereby accurately obtaining the pixel coordinates of the radar point in the pixel coordinate system, which can provide accurate coordinate information for subsequent edge point extraction. By adopting a depth-first strategy to address depth information conflicts during the projection process, not only can more accurate and reliable projection results be obtained, but it can also provide strong support for subsequent image analysis and processing tasks.

[0102] Step S403 : determining a first adjacent pixel point of the first projection point in the pixel coordinate system based on the pixel coordinates, and determining a weight of the first adjacent pixel point.

[0103] In some embodiments, pixels within a first range centered on the pixel coordinates may be selected as the first adjacent pixel points. For example, the first range may be a 7×7 pixel range centered on the first projection point, and pixels within the range excluding the first projection point may be selected as the first adjacent pixel points.

[0104] In some embodiments, an influence factor of any radar point on the first neighboring pixel point can be determined according to the coordinates of the first neighboring pixel point and the pixel coordinates, and a weight of the first neighboring pixel point can be determined based on the influence factor.

[0105] In one example, for each first adjacent pixel point, the influence factor of the radar point corresponding to the first projection point on the first pixel point is calculated. This influence factor can be determined based on the distance between the first adjacent pixel point and the first projection point. Specifically, the influence factor can be set as a function that decreases as the distance increases to reflect the attenuation characteristics of the influence of the radar point corresponding to the first projection point on the first adjacent pixel point. For example, a Gaussian function can be used to calculate the influence factor wx, and the influence factor wx = e^(-(ui-u)^2 / s - (vi-v)^2 / s), where (u,v) represents the pixel coordinates of the first projection point, (ui,vi) represents the pixel coordinates of the first adjacent pixel point i, and s represents a preset coefficient used to control the speed at which the influence factor decays.

[0106] In one example, if the first adjacent pixel point corresponds to only one radar point in the fused point cloud, the weight of each first pixel point may be set to be equal to the influence factor.

[0107] In some embodiments, if the first neighboring pixel corresponds to multiple radar points in the fused point cloud, an influence factor of each radar point on the first neighboring pixel is determined, a first cumulative value of the influence factors of the multiple radar points on the first neighboring pixel is determined, and a weight of the first neighboring pixel is determined based on the first cumulative value.

[0108] In one example, if multiple radar points are projected near a first-neighboring pixel, the first-neighboring pixel may be within the proximity of each of the multiple first projection points corresponding to the multiple radar points. In this case, it can be determined that the first-neighboring pixel corresponds to the multiple radar points in the fused point cloud. To determine the combined influence of the multiple radar points on the first-neighboring pixel, the influence factor of each radar point on the pixel can be calculated separately, and these influence factors can be accumulated to obtain a first accumulated value. This accumulated value is then used to determine the final weight of the first-neighboring pixel.

[0109] Based on the above embodiment, adjacent pixels within a first range are selected with the pixel coordinates of the first projection point as the center, and their weights are determined by calculating influence factors based on their distances from the first projection point. When a neighboring pixel corresponds to only one radar point, its weight is directly equal to the influence factor. If it corresponds to multiple radar points, its weight is determined by summing the influence factors of all of the radar points. This allows for accurate measurement of the attenuation characteristics and overall effect of the radar point's influence on the neighboring pixels.

[0110] Step S404: Determine the intensity parameter of the first adjacent pixel point based on the weight and the intensity value of any radar point.

[0111] In some embodiments, if a first neighboring pixel corresponds to only one radar point in the fused point cloud, an intensity parameter of the first neighboring pixel can be determined based on the ratio of the intensity value of the one radar point to the weight. The intensity parameter can reflect the relative intensity of the neighboring pixel, taking into account the weight of the radar point's influence on the neighboring pixel.

[0112] In some embodiments, if the first neighboring pixel corresponds to multiple radar points in the fused point cloud, a second accumulated value of the intensity values of the multiple radar points is determined; and an intensity parameter of the first neighboring pixel is determined based on the ratio of the second accumulated value to the weight. Similar to the case of a single radar point, the intensity parameter also reflects the relative intensity of the neighboring pixel, taking into account the weight of the influence of multiple radar points on the neighboring pixel.

[0113] Based on the above embodiment, the intensity parameters of adjacent pixel points can be calculated by comprehensively considering the radar point intensity value and the weights of adjacent pixel points. The intensity parameters obtained by this method can be used to perform subsequent operations such as image colorization and edge point extraction, thereby obtaining more accurate and reliable results.

[0114] Step S405 : determining a first pixel value of the first adjacent pixel point according to the intensity parameter, and obtaining a first image based on the first pixel value.

[0115] In some embodiments, the intensity parameter can be used to determine the first pixel value of each first adjacent pixel point. The intensity parameter can be used as a key indicator here. Since it combines the intensity information of the radar point and the weight distribution of the pixel point, it can accurately reflect the actual intensity characteristics of the pixel point. For example, the first pixel value can be used to update the pixel value of the first adjacent pixel point in the preset image to obtain the first image. The pixel value of the pixel point in the preset image is a preset second value, such as 0, 255 or an intermediate grayscale value. In the following examples, the second value will be 255 representing black. For example Figure 5, which is an example diagram of the first image provided in an embodiment of the present application.

[0116] Based on the above embodiment, the first image can not only present the projection position of the radar point at a pixel value position different from the second value, but also accurately present the intensity changes of each pixel point caused by the influence of the projection position of the radar point through the difference in pixel values, thereby providing a data basis for subsequent edge point extraction.

[0117] Step S406: determining a second adjacent pixel point from the first adjacent pixel points, determining a second pixel value of the second adjacent pixel point according to a preset first value, and obtaining a second image based on the second pixel value.

[0118] In some embodiments, among the first neighboring pixels, pixels within a second range centered on the pixel coordinates are selected as the second neighboring pixels. For example, if the first range is 7×7, the second range may be 3×3, 5×5, etc. The second pixel value is used to update the pixel value of the second neighboring pixel in the preset image to obtain the second image. The pixel value of the pixel in the preset image is the preset second value.

[0119] Based on the above embodiment, second adjacent pixels can be selected from the first adjacent pixels, and second pixel values can be assigned to the locations of these pixels in the preset image, thereby generating a second image. Compared to the first image, the second image can remove the effects of holes that may be caused by missing radar data or projection errors. By filling the holes, the overall image quality and visual effect are improved, and image discontinuities caused by missing data are reduced.

[0120] Step S407 : determining the target edge point based on the first edge point of the first image and the second edge point of the second image.

[0121] In some embodiments, edge extraction is performed on the first image to determine the first edge point. Figure 6 As shown, it is an example diagram of the position of the first edge point provided in an embodiment of the present application. Edge extraction is performed on the second image to determine the second edge point. In one example, the above-mentioned edge extraction is performed using an image processing algorithm to identify significant edge features in the image. For example, the image processing algorithm includes but is not limited to the Canny edge detector, the Sobel operator or the Prewitt operator. Based on the intersection of the first edge point and the second edge point, the third edge point is determined. Based on the radar point corresponding to the third edge point in the fused point cloud, the target edge point is determined. For example Figure 7 The figure shown is an example diagram of the position of the target edge point provided in an embodiment of the present application.

[0122] Based on the above embodiment, the third edge point determined by the intersection of the first and second edge point sets includes points that appear on the edges of both the first and second images, representing a more stable and reliable edge feature. Determining target edge points in the fused point cloud based on the radar points corresponding to the third edge point accurately identifies edge feature points within the radar points.

[0123] Figure 8 This is a flowchart of a three-dimensional reconstruction method provided by another embodiment of the present application. The three-dimensional reconstruction method includes: using the first frame image captured by the camera as a reference or initial data, and performing intensity edge point and line registration based on the frame image and the corresponding radar data. Alternatively, if the frame image is not the first frame image, the Lucas-Kanade optical flow method is used to track feature points or edges in the image. During the intensity edge point and line registration, the radar point cloud frame and the fused point cloud of the adjacent radar point cloud frames can be projected onto the image plane to obtain an intensity map corresponding to the fused point cloud in the pixel coordinate system corresponding to the image data, so as to be compared and fused with the image data corresponding to the radar data. Edges in the intensity map are detected using the Canny edge detection algorithm to obtain an intensity edge map. For example, the intensity edge map contains the locations of intensity edge point clouds (e.g., target edge points). Canny edge detection is performed on the RGB image captured by the camera (e.g., image data) to obtain a gradient map. The gradient map indicates the rate of change of pixel values in the image. Contours are searched on the gradient map to obtain a contour map. Intensity edge point and line registration is performed based on the intensity edge point cloud map and the gradient map to align the image and point cloud to the same coordinate system. The forward propagation algorithm can be used to iteratively optimize the intensity edge point and line registration process. After the intensity edge point and line registration, the reprojection error registration method can be used to further align the intensity edge point and line registration results to achieve accurate alignment of three-dimensional and two-dimensional (3D-2D) data.

[0124] Based on the above embodiments, the initial frame image can be used to perform intensity edge point and line registration with the radar data, or the Lucas-Kanade optical flow method can be used to track features in subsequent frames. During the registration process, the radar point cloud is fused and projected onto the image plane. Intensity edge maps and image gradient maps are then obtained using Canny edge detection, and point and line registration is then performed. The registration results are iteratively optimized using a forward propagation algorithm, and further refined using the reprojection error registration method. This ensures the precise fusion of 3D and 2D data, achieves high-precision alignment of camera images and radar data, and improves the accuracy of 3D reconstruction.

[0125] Figure 9 3D reconstruction device according to an embodiment of the present invention.

[0126] In some embodiments, the 3D reconstruction device 70 may include multiple functional modules composed of computer program segments. The computer programs of the various program segments in the 3D reconstruction device 70 may be stored in a memory of an electronic device and executed by at least one processor to perform (see Figure 2 Description) 3D reconstruction function.

[0127] In this embodiment, the 3D reconstruction device 70 can be divided into multiple functional modules based on the functions they perform. These modules may include an acquisition module 701, a determination module 702, an update module 703, and a reconstruction module 704. As used herein, a module refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and are stored in a memory. The functional implementation of each module in this embodiment can be found in the definition of the 3D reconstruction method above and will not be repeated here.

[0128] The acquisition module 701 is used to acquire radar point cloud frames and image data corresponding to the same scene.

[0129] The determination module 702 is configured to determine a target edge point in the radar point cloud frame; determine an edge contour point in the image data, and determine a normal vector corresponding to the edge contour point; and determine a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point, and the normal vector.

[0130] The updating module 703 is configured to update the camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose.

[0131] The reconstruction module 704 is configured to fuse the radar point cloud frame with the image data based on the target camera pose to obtain a reconstructed image.

[0132] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0133] The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc.

[0134] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0135] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specified application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specified application, but such implementation should not be considered beyond the scope of this application.

[0137] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal devices and methods can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0138] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Obtain radar point cloud frames and image data corresponding to the same scene; Determining a target edge point in the radar point cloud frame includes: obtaining a fused point cloud of the radar point cloud frame and an adjacent radar point cloud frame; determining the pixel coordinates of a first projection point corresponding to any radar point in the fused point cloud in a pixel coordinate system, wherein the pixel coordinate system represents a coordinate system corresponding to the image data; determining a first neighboring pixel point of the first projection point in the pixel coordinate system based on the pixel coordinates, and determining a weight of the first neighboring pixel point; determining an intensity parameter of the first neighboring pixel point based on the weight and an intensity value of the any radar point; determining a first pixel value of the first neighboring pixel point based on the intensity parameter, and obtaining a first image based on the first pixel value; determining a second neighboring pixel point from the first neighboring pixel point, determining a second pixel value of the second neighboring pixel point based on a preset first value, and obtaining a second image based on the second pixel value; determining the target edge point based on a first edge point of the first image and a second edge point of the second image; Determining edge contour points in the image data, and determining normal vectors corresponding to the edge contour points; Determining a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point, and the normal vector; updating the camera pose corresponding to the image data according to the normal projection error to obtain a target camera pose; The radar point cloud frame and the image data are fused based on the target camera pose to obtain a reconstructed image.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: Determining the target edge point in the radar point cloud frame includes: Determine a target edge point in the radar point cloud frame based on an intensity difference between any radar point and an adjacent radar point in the radar point cloud frame, wherein the adjacent radar point represents a radar point located on the same radar scan line as the any radar point.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The adjacent radar points include a first radar point and a second radar point respectively located on both sides of the any radar point, and determining the target edge point in the radar point cloud frame according to the intensity difference between any radar point and the adjacent radar point in the radar point cloud frame includes: determining a first intensity difference between the any radar point and the first radar point; determining a second intensity difference between the any radar point and the second radar point; If the first intensity difference is greater than a first threshold and the second intensity difference is less than a second threshold, it is determined that the any radar point and the first radar point are the target edge points.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The method further comprises: The first threshold and the second threshold are determined according to the intensity value of any radar point.

5. The three-dimensional reconstruction method according to claim 1, characterized in that: The obtaining of a fused point cloud of the radar point cloud frame and adjacent radar point cloud frames includes: The radar point cloud frame and adjacent radar point cloud frames are projected into a world coordinate system, and the radar point cloud frame and adjacent radar point cloud frames are fused in the world coordinate system to obtain the fused point cloud.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: Determining the pixel coordinates of a first projection point corresponding to any radar point in the fused point cloud in a pixel coordinate system includes: projecting the radar point in the fused point cloud to a camera coordinate system corresponding to the image data based on a camera pose corresponding to the image data, and determining coordinates of a first projection point of the radar point in the camera coordinate system; Based on the intrinsic parameter matrix of the camera corresponding to the image data, the coordinates of the first projection point are converted to the pixel coordinate system to obtain the pixel coordinates.

7. The three-dimensional reconstruction method according to claim 1, characterized in that: The determining, based on the pixel coordinates, a first adjacent pixel point of the first projection point in the pixel coordinate system, and determining a weight of the first adjacent pixel point includes: Taking the pixel coordinates as the center, select pixels within a first range as the first adjacent pixel points; An influence factor of any radar point on the first adjacent pixel point is determined according to the coordinates of the first adjacent pixel point and the pixel coordinates, and a weight of the first adjacent pixel point is determined based on the influence factor.

8. The three-dimensional reconstruction method according to claim 7, characterized in that: The method further comprises: If the first adjacent pixel point corresponds to multiple radar points in the fused point cloud, determining an influence factor of each radar point on the first adjacent pixel point; Determine a first accumulated value of influence factors of the multiple radar points on the first adjacent pixel point, and determine a weight of the first adjacent pixel point according to the first accumulated value.

9. The three-dimensional reconstruction method according to claim 1, characterized in that: The determining, based on the weight and the intensity value of any radar point, the intensity parameter of the first adjacent pixel point includes: An intensity parameter of the first adjacent pixel point is determined according to a ratio of the intensity value of any radar point to the weight.

10. The three-dimensional reconstruction method according to claim 9, characterized in that: The method further comprises: If the first adjacent pixel point corresponds to a plurality of radar points in the fused point cloud, determining a second accumulated value of the intensity values of the plurality of radar points; An intensity parameter of the first adjacent pixel point is determined according to a ratio of the second accumulated value to the weight.

11. The three-dimensional reconstruction method according to claim 1, wherein: Determining a first pixel value of the first adjacent pixel point according to the intensity parameter, and obtaining a first image based on the first pixel value includes: The first pixel value is used to update the pixel value of the first adjacent pixel point in the preset image to obtain the first image, wherein the pixel value of the pixel point in the preset image is a preset second value.

12. The three-dimensional reconstruction method according to claim 1, characterized in that: The determining of a second adjacent pixel point among the first adjacent pixel points, determining a second pixel value of the second adjacent pixel point according to a preset first value, and obtaining a second image based on the second pixel value includes: Among the first adjacent pixel points, taking the pixel coordinates as the center, select pixel points located in a second range as the second adjacent pixel points; The pixel value of the second adjacent pixel point in the preset image is updated using the second pixel value to obtain the second image, wherein the pixel value of the pixel point in the preset image is a preset second value.

13. The three-dimensional reconstruction method according to claim 1, wherein: The determining the target edge point based on the first edge point of the first image and the second edge point of the second image includes: performing edge extraction on the first image to determine the first edge point; performing edge extraction on the second image to determine the second edge points; Determining a third edge point according to the intersection of the first edge point and the second edge point; The target edge point is determined according to a radar point corresponding to the third edge point in the fused point cloud.

14. The three-dimensional reconstruction method according to claim 1, characterized in that: The determining of edge contour points in the image data and determining normal vectors corresponding to the edge contour points includes: Acquire a grayscale image corresponding to the image data, and perform edge extraction on the grayscale image to obtain the edge contour points; The normal vector corresponding to the edge contour point is determined based on a principal component analysis algorithm.

15. The three-dimensional reconstruction method according to claim 1, characterized in that: Determining a normal projection error between the target edge point and the edge contour point based on the target edge point, the edge contour point, and the normal vector includes: Projecting the target edge point to a pixel coordinate system corresponding to the image data, and determining a second projection point of the radar point in the target edge point in the pixel coordinate system; The normal projection error is determined based on the distance between the edge contour point and the second projection point in the normal vector direction.

16. The three-dimensional reconstruction method according to claim 1, characterized in that: The updating of the camera pose corresponding to the image data according to the normal projection error includes: The camera pose is updated based on a preset optimization objective, wherein the optimization objective includes minimizing the normal projection error.

17. The three-dimensional reconstruction method according to claim 1, characterized in that: After updating the camera pose corresponding to the image data according to the normal projection error, the method further includes: A reprojection error between the radar point cloud frame and the image data is determined, and the camera pose is updated according to the reprojection error to obtain the target camera pose.

18. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the three-dimensional reconstruction method according to any one of claims 1 to 17.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 17 is implemented.

Images