Visual three-dimensional reconstruction method and device and storage medium

Abstract

The invention provides a visual three-dimensional reconstruction method and device and a storage medium, and the method comprises the steps: taking data generated by a first non-visual sensor as supervision, and carrying out the motion structure recovery of visual data generated by a visual sensor; constructing joint observation by using the first non-visual sensor and the visual sensor, and screening out effective visual data; and performing position supervision on the screened effective visual data by using data generated by a second non-visual sensor, and optimizing visual three-dimensional reconstruction. According to the method, depth information supervision and position supervision are respectively realized by using data acquired by the first non-visual sensor, the second non-visual sensor and the visual sensor, visual three-dimensional reconstruction is performed on a high-speed and large-scale outdoor scene, the reconstructed scene is more accurate, the visual effect is more vivid, and the efficiency and effect of visual three-dimensional reconstruction are improved.

Description

Technical Field

[0001] The present application belongs to the technical field of autonomous driving scene reconstruction, and specifically relates to a visual three-dimensional reconstruction method, device and storage medium. Background Art

[0002] Visual 3D reconstruction is a very important task in autonomous driving environment perception. This task mainly involves identifying important road signs or environmental points for autonomous driving, such as vehicles, pedestrians, and non-motor vehicles, in the images captured by the camera.

[0003] The efficient update and iteration of autonomous driving environment perception algorithms depends on the cost compression of data annotation. The automatic annotation technology based on visual simulation can effectively reduce the cost of data annotation. Visual simulation uses cameras to shoot environmental information from multiple perspectives to obtain multi-perspective images; based on multi-perspective images and corresponding camera poses, visual three-dimensional reconstruction is performed to restore environmental information, and by reconstructing two-dimensional (2D) image information into three-dimensional (3D) scene point cloud information, three-dimensional data is provided for automatic annotation.

[0004] The visual 3D reconstruction methods in current related technologies have achieved good reconstruction results in closed and simple indoor environments. However, for the reconstruction of large-scale outdoor scenes, the algorithm is difficult to converge and the reconstruction effect is poor. Summary of the invention

[0005] The present application proposes a visual three-dimensional reconstruction method, device and storage medium, which can improve the technical problem of poor three-dimensional scene reconstruction effect in related technologies.

[0006] The first embodiment of the present application proposes a visual three-dimensional reconstruction method, including:

[0007] Using the data generated by the first non-visual sensor as supervision, performing motion structure recovery on the visual data generated by the visual sensor;

[0008] Use the first non-visual sensor and the visual sensor to build a joint observation and filter out effective visual data;

[0009] The data generated by the second non-visual sensor is used to supervise the position of the filtered valid visual data and optimize the visual 3D reconstruction.

[0010] In some embodiments of the present application, using data generated by a first non-visual sensor as supervision to perform motion structure recovery on visual data generated by a visual sensor includes:

[0011] Obtaining depth information generated by a first non-vision sensor;

[0012] Depth information is used as a supervision item to supervise the depth data obtained after motion structure recovery of visual data generated by a visual sensor.

[0013] In some embodiments of the present application, obtaining depth information generated by a first non-visual sensor includes: obtaining a plurality of non-visual point cloud frames generated by the first non-visual sensor, each non-visual point cloud frame corresponding to a timestamp;

[0014] Using depth information as a supervision item, supervise the depth data obtained after the motion structure recovery of the visual data generated by the visual sensor, including:

[0015] Use the transformation matrix to transform the non-visual point cloud frame into the visual coordinate system;

[0016] Acquire multiple visual point cloud frames obtained by performing motion structure recovery on visual data generated by a visual sensor, each visual point cloud frame corresponding to a timestamp;

[0017] Determine a depth information supervision data pair, the depth information supervision data pair comprising a converted non-visual point cloud frame and a visual point cloud frame with consistent timestamps; perform weighted averaging on the converted non-visual point cloud frame in the depth information supervision data pair and the visual point cloud frame in the depth information supervision data pair to generate a visual point cloud frame after depth information supervision.

[0018] In some embodiments of the present application, a first non-visual sensor is used to construct a joint observation with a visual sensor to screen out valid visual data, including:

[0019] Determine a joint observation data pair, the joint observation data pair comprising a non-visual point cloud frame with consistent timestamp and a visual point cloud frame supervised by depth information;

[0020] For each joint observation data pair, the observation matrix is ​​used to observe the overlap between the non-visual point cloud frame and the visual point cloud frame after depth information supervision. The visual point cloud frame after depth information supervision whose overlap meets the requirements is taken as a valid visual point cloud frame and determined as valid visual data.

[0021] In some embodiments of the present application, observing the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information includes:

[0022] Constructing a source grid according to the non-visual point cloud frame, determining the source cell state corresponding to each source grid point in the constructed source grid; constructing a target grid according to the visual point cloud frame supervised by the depth information, the constructed target grid includes each target grid point and each corresponding target cell; using the corresponding transformation matrix from the first non-visual coordinate system to the visual coordinate system as the observation matrix corresponding to the first non-visual sensor and the visual sensor; or

[0023] A source grid is constructed according to the visual point cloud frame supervised by depth information, and the source cell state corresponding to each source grid point in the constructed source grid is determined; a target grid is constructed according to the non-visual point cloud frame, and the constructed target grid includes each target grid point and the corresponding target cells; the transformation matrix from the corresponding visual coordinate system to the first non-visual coordinate system is used as the observation matrix corresponding to the first non-visual sensor and the visual sensor.

[0024] In some embodiments of the present application, observing the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information further includes:

[0025] Observe the states of each target cell corresponding to each pseudo target grid point in the target grid obtained after coordinate transformation of each source grid point corresponding to each source cell in the source grid whose state is occupied;

[0026] The number of occupied target cells corresponding to each observed pseudo-target grid point is counted, and the number is used to indicate the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information.

[0027] In some embodiments of the present application, data generated by a second non-visual sensor is used to perform position supervision on the filtered effective visual data and optimize visual three-dimensional reconstruction, including:

[0028] The data generated by the second non-visual sensor includes a navigation pose matrix and a navigation position corresponding to a timestamp;

[0029] The filtered valid visual data includes a camera pose matrix corresponding to a timestamp;

[0030] Use the navigation pose matrix to supervise the camera pose matrix consistent with its timestamp based on the navigation position and generate the corresponding residual;

[0031] Based on the residuals corresponding to each time stamp, an optimization task for the selected valid visual data is constructed, the optimization task is solved, and a camera pose matrix after position supervision corresponding to each time stamp is generated;

[0032] The visual 3D reconstruction is optimized according to the supervised camera pose matrix at each position to generate an optimized visual 3D reconstructed image.

[0033] In some embodiments of the present application, the method further includes: arranging the camera pose matrices corresponding to all valid visual point cloud frames in the order of timestamps, selecting a preset number of camera pose matrices adjacent to each other from the beginning to the end to form a relative motion estimation sequence, and determining an angle translation combination corresponding to the relative motion estimation sequence;

[0034] Determine that the valid visual point cloud frame corresponding to the camera pose matrix specified in the relative motion estimation sequence whose angle translation combination meets the requirements is the secondary observation valid visual point cloud frame.

[0035] In some embodiments of the present application, the method further includes: arranging the camera pose matrices corresponding to all valid visual point cloud frames in the order of timestamps, selecting a preset number of camera pose matrices adjacent to each other from the beginning to the end to form a relative motion estimation sequence, and determining each angle translation combination corresponding to each relative motion estimation sequence;

[0036] Taking the combination of translation amounts at various angles as the cost function, an optimization task for all valid visual point cloud frames is constructed. The optimization task is solved with the minimum cost function as the constraint, and the optimized camera pose matrix corresponding to each valid visual point cloud frame is generated.

[0037] In some embodiments of the present application, a preset number of camera pose matrices adjacent to each other are selected in sequence from the beginning to the end to form a relative motion estimation sequence, and an angle translation amount combination corresponding to the relative motion estimation sequence is determined, including:

[0038] A preset number of camera pose matrices adjacent to each other are selected in sequence from the beginning to the end to form a relative motion estimation sequence, wherein the preset number of camera pose matrices in the relative motion estimation sequence are respectively a first matrix, a middle matrix or a last matrix;

[0039] Determine the relative motion estimation matrices from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the next intermediate matrix, and from the last intermediate matrix to the tail matrix in sequence; multiply the relative motion matrices as the angle translation combination corresponding to the relative motion estimation sequence.

[0040] In some embodiments of the present application, the first non-visual sensor includes a radar device; the visual sensor includes an optical camera; and the second non-visual sensor includes a position sensor.

[0041] An embodiment of the second aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0042] The embodiment of the third aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method of the first aspect.

[0043] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0044] This method extracts robust feature point pairs from the image frames acquired by the camera, introduces sensors such as lidar and GNSS to realize depth information supervision and position supervision respectively, and performs visual 3D reconstruction of high-speed and large-scale outdoor scenes. The reconstructed scenes are more accurate and the visual effects are more realistic, which improves the efficiency and effect of visual 3D reconstruction.

[0045] These and other aspects of the present application will become more apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS

[0046] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0047] In the attached picture:

[0048] Figure 1 A flowchart of a visual three-dimensional reconstruction method provided by an embodiment of the present application is shown;

[0049] Figure 2A A flowchart of a visual three-dimensional reconstruction method provided by another embodiment of the present application is shown;

[0050] Figure 2B A flowchart of a visual three-dimensional reconstruction method provided by another embodiment of the present application is shown;

[0051] Figure 3 A schematic diagram showing the steps of constructing a relative motion frame sequence in an embodiment of the present application is shown;

[0052] Figure 4A An example of an image frame generated by a camera obtained in an embodiment of the present application is shown;

[0053] Figure 4B An example of a laser radar point cloud frame generated by a laser radar acquired by an embodiment of the present application is shown;

[0054] Figure 4C An example of a landmark point cloud formed after motion structure recovery is performed on multiple image frames within a preset time span in one embodiment of the present application is shown;

[0055] Figure 4D An example of a landmark point cloud formed after motion structure recovery is performed using depth information supervision for multiple image frames within a preset time span in one embodiment of the present application is shown;

[0056] Figure 4E Shows Figure 4D A partial enlarged view of the

[0057] Figure 4F An example of a point cloud frame of an environment point formed after motion structure recovery is performed on multiple image frames within a preset time span in one embodiment of the present application is shown;

[0058] Figure 4G An example of a landmark point cloud formed by performing joint observation on multiple image frames within a preset time span and then performing motion structure recovery in one embodiment of the present application is shown;

[0059] Figure 4H An example diagram of a landmark point cloud formed when the motion structure recovery for a large-scale and large-range scene in the related art is difficult to converge is shown;

[0060] Fig. 4I An example diagram of a landmark point cloud formed by performing motion structure recovery based on satellite navigation posture for a large-scale and large-range scene in one embodiment of the present application is shown;

[0061] Figure 5 A schematic diagram of the structure of a visual three-dimensional reconstruction device provided in one embodiment of the present application is shown;

[0062] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0063] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0065] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0066] The following describes the visual three-dimensional reconstruction method, device and storage medium proposed according to the embodiments of the present application in conjunction with the accompanying drawings.

[0067] In related technologies, autonomous driving vehicles usually use lidar to collect point clouds of the vehicle's surrounding environment, and identify mobile objects such as vehicles, pedestrians, non-motor vehicles, etc. that may exist in the vehicle's surrounding environment based on the collected point clouds, as well as stationary objects such as traffic facilities on the road and roadside buildings.

[0068] In related technologies, autonomous vehicles usually use optical cameras to collect the surrounding environment of the vehicle. For example, when the camera follows the movement of the autonomous vehicle, it continuously collects the surrounding environment at fixed time intervals and generates images. Based on the collected images, it identifies the possible moving objects such as vehicles, pedestrians, non-motor vehicles, and stationary objects such as traffic facilities on the road and roadside buildings in the surrounding environment of the vehicle.

[0069] Vision-based 3D scene reconstruction is a very important task in autonomous driving environment perception. The task of 3D scene reconstruction is mainly to identify important road signs or environmental points for autonomous driving in the images obtained by the camera, such as vehicles, pedestrians, non-motor vehicles, traffic facilities on the road, roadside buildings, etc.

[0070] Specifically, optical cameras acquire images from different angles and distances, and combine them with precise position information (such as absolute position) provided by position sensors, such as GNSS, inertial navigation or wheel speed meters, to achieve real-time positioning and motion estimation at high frame rates. High-precision three-dimensional maps can be reconstructed, and the positions and shapes of landmarks or environmental points can be updated in real time by reusing the maps.

[0071] Specifically, based on the image information obtained by the camera, methods for realizing motion estimation include Structure From Motion (SFM), such as utilizing the image differences between adjacent image frames to extract feature points for each image and perform feature point matching between frames; after constructing matching associated feature point pairs, triangulation solution is performed to obtain the depth information of the environment points and the camera pose matrix.

[0072] The visual 3D reconstruction method in the current related technology has achieved good reconstruction results in closed and simple indoor environments, but in the reconstruction of large-scale and large-scale autonomous driving scenes, there may be some problems such as Figure 4H The algorithm shown is difficult to converge, such as Figure 4C The problem with poor reconstruction shown.

[0073] Based on this, an embodiment of the present application provides a visual three-dimensional reconstruction method, which extracts robust feature point pairs in the image frames acquired by the camera, introduces sensors such as lidar and GNSS to respectively realize depth information supervision and position supervision, and performs visual three-dimensional reconstruction of high-speed and large-scale outdoor scenes. The reconstructed scene is more accurate and the visual effect is more realistic, thereby improving the efficiency and effect of visual three-dimensional reconstruction.

[0074] The execution subject of the embodiment of the present application can be a movable vehicle for visual three-dimensional reconstruction of high-speed, large-scale outdoor scenes. The movable vehicle can be a vehicle, a ship, an aircraft or a robot, etc. Usually, when the movable vehicle moves, low speed includes a moving speed of less than 20km / h, medium speed includes a moving speed of 20-60km / h, and high speed includes a moving speed of more than 60km / h. Taking 60km / h as an example, the moving distance per minute is 1km. When the movable vehicle moves at medium or high speed, it belongs to the scene of large-scale, large-scale autonomous driving environment perception and visual reconstruction.

[0075] The movable vehicle is usually provided with an environmental information acquisition device to acquire the external environmental information of the vehicle. In some embodiments, the environmental information acquisition device may include an optical camera, a radar device, and a position sensor. Among them, the optical camera may include an RGB (Red Green Blue) three-primary color camera sensor, an infrared camera sensor, etc. Among them, the radar device may include a laser radar (Light Detection and Ranging, LiDAR), a millimeter wave radar (Radio Detection and Ranging, Radar), an ultrasonic radar (Ultrasonic Sensor), a synthetic aperture radar (Synthetic Aperture Radar), etc. Among them, the position sensor may include a satellite navigation system, such as a global satellite navigation system GNSS (Global Navigation Satellite System), an inertial measurement unit (Inertial Measurement Unit, IMU), a gyroscope (Gyroscope), an accelerometer (Accelerometer), a magnetometer (Magnetometers), etc. The number of optical cameras, radar devices, and position sensors may be one or more.

[0076] By shooting the target area, the RGB camera can obtain Figure 4A The image frame with the coloring effect is shown. Furthermore, a point cloud of each landmark or environmental point in the target area can be generated based on the image frames acquired continuously in time. Usually, each image frame can record multiple environmental points, and each environmental point has data in four dimensions: x0, y0, z0 and pixel value, where x0, y0, z0 represent the three-dimensional coordinates of the environmental point in the camera coordinate system.

[0077] The laser radar emits laser pulses to the target area, and the laser pulses are reflected back when they encounter objects in the target area. Based on the reflected echo received by the laser radar, the following can be obtained: Figure 4BThe point cloud of each environmental point in the target area is shown. Usually, each laser radar point cloud frame records multiple environmental points, and each environmental point has data in four dimensions: x, y, z, and echo reflectivity, where x, y, z represent the three-dimensional coordinates of the environmental point in the laser radar coordinate system.

[0078] Typically, the target area may include, but is not limited to, one or more areas in front, left, right, or rear of the movable vehicle.

[0079] Typically, the movable vehicle has a lidar installed at one set position and an RGB camera installed at another set position.

[0080] See also Figure 1 , the visual three-dimensional reconstruction method may include the following steps S1 to S3.

[0081] S1: Using the data generated by the first non-visual sensor as supervision, performing motion structure recovery on the visual data generated by the visual sensor;

[0082] S2: Use the first non-visual sensor and the visual sensor to establish a joint observation and filter out valid visual data;

[0083] S3: Use data generated by a second non-visual sensor to perform position supervision on the filtered effective visual data and optimize visual three-dimensional reconstruction.

[0084] In the above, the data generated by the first non-visual sensor is used as supervision to perform motion structure recovery on the visual data generated by the visual sensor; the first non-visual sensor and the visual sensor are used to construct a joint observation to screen out valid visual data; the data generated by the second non-visual sensor is used to perform position supervision on the screened valid visual data and optimize the visual three-dimensional reconstruction.

[0085] In this way, the visual 3D reconstruction method performs visual 3D reconstruction on high-speed, large-scale outdoor scenes, reconstructs scenes more accurately, and has better and more realistic visual effects, thereby improving the efficiency and effect of visual 3D reconstruction.

[0086] In some embodiments of the present application, in the aforementioned step S1, the data generated by the first non-visual sensor is used as supervision to perform motion structure recovery on the visual data generated by the visual sensor, including: obtaining depth information generated by the first non-visual sensor; using the depth information as a supervision item to supervise the depth data obtained after motion structure recovery on the visual data generated by the visual sensor.

[0087] In some embodiments of the present application, the acquiring of the depth information generated by the first non-visual sensor includes: acquiring a plurality of non-visual point cloud frames generated by the first non-visual sensor, each non-visual point cloud frame corresponding to a timestamp and recording data of each environmental point, the environmental point data including position information of the environmental point in a non-visual coordinate system;

[0088] The using the depth information as a supervision item to supervise the depth data obtained after the motion structure recovery is performed on the visual data generated by the visual sensor, comprises:

[0089] Using a transformation matrix from a first non-visual coordinate system to a visual coordinate system corresponding to the non-visual point cloud frame, the position information of each environmental point recorded in the non-visual point cloud frame is converted to the visual coordinate system;

[0090] Acquire multiple visual point cloud frames obtained by performing motion structure recovery on visual data generated by a visual sensor, each visual point cloud frame corresponds to a timestamp and records data of each environmental point, wherein the environmental point data includes position information of the environmental point in a visual coordinate system;

[0091] Determine a depth information supervision data pair, wherein the depth information supervision data pair includes a converted non-visual point cloud frame and a visual point cloud frame with consistent timestamps; perform weighted averaging on the position information of the environmental points recorded in the converted non-visual point cloud frame in the depth information supervision data pair and the position information of the environmental points recorded in the visual point cloud frame in the depth information supervision data pair, generate the position information of the environmental points after depth information supervision, and record it in the visual point cloud frame after depth information supervision.

[0092] In some embodiments of the present application, in the aforementioned step S2, using the first non-visual sensor and the visual sensor to establish a joint observation and screen out valid visual data includes:

[0093] Determine a joint observation data pair, the joint observation data pair comprising a non-visual point cloud frame with consistent timestamps and a visual point cloud frame supervised by depth information;

[0094] For each of the joint observation data pairs, the observation matrix corresponding to the first non-visual sensor and the visual sensor is used to observe the overlap between the non-visual point cloud frame and the visual point cloud frame after depth information supervision, and the visual point cloud frame after depth information supervision whose overlapping situation meets the requirements is taken as a valid visual point cloud frame and determined as valid visual data.

[0095] It should be understood that the valid visual data corresponding to any timestamp may also include a camera pose matrix associated with the visual point cloud frame.

[0096] In some embodiments of the present application, observing the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information includes:

[0097] Constructing a source grid according to the non-visual point cloud frame, determining the source cell state corresponding to each source grid point in the constructed source grid; constructing a target grid according to the visual point cloud frame supervised by the depth information, wherein the constructed target grid includes each target grid point and each corresponding target cell; using the corresponding transformation matrix from the first non-visual coordinate system to the visual coordinate system as the observation matrix corresponding to the first non-visual sensor and the visual sensor; or

[0098] A source grid is constructed according to the visual point cloud frame supervised by depth information, and the source cell state corresponding to each source grid point in the constructed source grid is determined; a target grid is constructed according to the non-visual point cloud frame, and the constructed target grid includes each target grid point and the corresponding target cell; and a transformation matrix from the corresponding visual coordinate system to the first non-visual coordinate system is used as an observation matrix corresponding to the first non-visual sensor and the visual sensor.

[0099] In some embodiments of the present application, observing the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information further includes:

[0100] Observe the states of target cells in the target grid corresponding to the pseudo target grid points obtained by performing coordinate transformation on the source grid points corresponding to the source cells in the source grid whose states are occupied;

[0101] The number of target cells corresponding to the observed pseudo-target grid points that are occupied is counted, and the number is used to indicate the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information.

[0102] In some embodiments of the present application, in the aforementioned step S3, using the data generated by the second non-visual sensor to perform position supervision on the screened effective visual data and optimize the visual three-dimensional reconstruction includes:

[0103] The data generated by the second non-visual sensor includes a navigation pose matrix and a navigation position corresponding to a timestamp;

[0104] The filtered valid visual data includes a camera pose matrix corresponding to a timestamp;

[0105] Use the navigation pose matrix to supervise the camera pose matrix consistent with its timestamp based on the navigation position and generate the corresponding residual;

[0106] Based on the residuals corresponding to each time stamp, an optimization task for the filtered valid visual data is constructed, the optimization task is solved, and a camera pose matrix after position supervision corresponding to each time stamp is generated;

[0107] The visual 3D reconstruction is optimized according to the supervised camera pose matrix at each position to generate an optimized visual 3D reconstructed image.

[0108] In some embodiments of the present application, step S4 may also be included: arranging the camera pose matrices corresponding to all valid visual point cloud frames in the order of timestamps, selecting a preset number of camera pose matrices adjacent to each other from the beginning to the end to form a relative motion estimation sequence, and determining an angle translation amount combination corresponding to the relative motion estimation sequence;

[0109] Determine that the effective visual point cloud frame corresponding to the camera pose matrix specified in the relative motion estimation sequence in which the angle translation combination meets the requirements is the secondary observation effective visual point cloud frame.

[0110] In some embodiments of the present application, step S5 may also be included: arranging the camera pose matrices corresponding to all valid visual point cloud frames in the order of timestamps, selecting a preset number of camera pose matrices adjacent to each other from the beginning to the end to form a relative motion estimation sequence, and determining each angle translation combination corresponding to each relative motion estimation sequence;

[0111] Taking the combination of translation amounts at various angles as the cost function, an optimization task for all valid visual point cloud frames is constructed, and the optimization task is solved with the minimum cost function as the constraint to generate an optimized camera pose matrix corresponding to each valid visual point cloud frame.

[0112] In some embodiments of the present application, in the above step S4 or S5, selecting a preset number of camera pose matrices adjacent to each other from the beginning to the end to form a relative motion estimation sequence, and determining the angle translation amount combination corresponding to the relative motion estimation sequence includes:

[0113] Selecting a preset number of camera pose matrices adjacent to each other from the front to the back to form a relative motion estimation sequence, wherein the preset number of camera pose matrices in the relative motion estimation sequence are respectively a first matrix, a middle matrix or a last matrix;

[0114] Determine the relative motion estimation matrices from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the next intermediate matrix, and from the last intermediate matrix to the last matrix in turn; multiply the relative motion matrices as the angle translation combination corresponding to the relative motion estimation sequence. It should be understood that the "pointing" used to explain the relationship between matrices above defines the matrix operation relationship of the product of the inverse matrix of the previous pose matrix and the next pose matrix.

[0115] It should be understood that the first non-visual sensor includes any one of the aforementioned radar devices; the visual sensor includes any one of the aforementioned optical cameras; and the second non-visual sensor includes any one of the aforementioned position sensors.

[0116] See also Figure 2A The following uses a movable vehicle with a combination of a laser radar and an RGB camera as an example to illustrate the visual 3D reconstruction method of the present application embodiment. The visual 3D reconstruction method may include the following steps S10 to S50.

[0117] S10: Using each laser radar point cloud frame to perform depth information supervision on motion structure recovery of each image frame, and generating point cloud frames after depth information supervision corresponding to each image frame;

[0118] S20: constructing a joint observation for each depth information supervised point cloud frame and each lidar point cloud frame, and determining that the depth information supervised point cloud frames that meet the motion consistency check are each valid visual point cloud frame;

[0119] S30: constructing optimization tasks for all valid visual point cloud frames using the acquired camera pose matrices to generate optimized camera pose matrices corresponding to each valid visual point cloud frame;

[0120] S40: Obtaining each navigation pose matrix corresponding to the optimized camera pose matrix, constructing an optimization task for all navigation pose matrices and the optimized camera pose matrix, and generating a camera pose matrix after each pose supervision;

[0121] S50: Performing visual three-dimensional reconstruction on each of the valid visual point cloud frames using the camera pose matrices after the pose supervision to obtain a visual three-dimensional reconstructed image.

[0122] In the above, each laser radar point cloud frame is used to perform depth information supervision on the motion structure recovery of each image frame, and the point cloud frames after depth information supervision corresponding to each image frame are generated, which improves the accuracy of motion structure recovery and visual triangulation solution; a joint observation is constructed for each point cloud frame after depth information supervision and each laser radar point cloud frame, and each valid visual point cloud frame that meets the motion consistency check is determined, and each invalid visual point cloud frame is deleted, which is beneficial to reduce the time consumption and resource consumption required for subsequent motion structure recovery, and improve the accuracy and efficiency of scene reconstruction; the obtained camera pose matrices are used to construct optimization tasks for all valid visual point cloud frames, and the optimized camera pose matrices corresponding to each valid visual point cloud frame are generated, which is beneficial to improve the accuracy of motion structure recovery and visual triangulation solution; each navigation pose matrix corresponding to the optimized camera pose matrix is ​​obtained, and an optimization task is constructed for all navigation pose matrices and the optimized camera pose matrix to generate a camera pose matrix after each pose supervision, which is beneficial to the rapid convergence of scene reconstruction and improve the accuracy and efficiency of scene reconstruction.

[0123] Referring to the above description, the lidar generates a lidar point cloud frame sequence at a specified first frame rate, and the RGB camera generates an image frame sequence at a specified second frame rate. Specifically, the first frame rate can be greater than the second frame rate, equal to the second frame rate, or less than the second frame rate. The frame rate refers to the rate at which image frames or point cloud frames are generated, or the number of image frames or point cloud frames generated per unit time.

[0124] In order to use the LiDAR point cloud frames to perform deep information supervision on the motion structure recovery of image frames, the LiDAR point cloud frame sequence and the image frame sequence are first screened based on the timestamp, that is, the LiDAR point cloud frames and image frames that are consistent in time are screened out. The consistency here can include being completely identical in time or being the closest in time.

[0125] Specifically, the smaller value of the first frame rate and the second frame rate is used as the reference frame rate and the laser radar point cloud frames and image frames that are consistent in time are screened. For example, when the first frame rate of the laser radar point cloud frame is less than the second frame rate of the image frame, for each laser radar point cloud frame in the laser radar point cloud frame sequence, its first timestamp is determined, and the image frame corresponding to the second timestamp that is closest to the first timestamp in time (may be ahead or behind) is determined, until the image frames that are closest to each laser radar point cloud frame are determined.

[0126] Similarly, when the first frame rate of the lidar point cloud frame is greater than the second frame rate of the image frame, for each image frame in the image frame sequence, its first timestamp is determined, and the lidar point cloud frame corresponding to the second timestamp that is closest in time to the first timestamp (which may be ahead or behind) is determined, until the lidar point cloud frames that are closest to each image frame are determined.

[0127] Similarly, when the first frame rate of the laser radar point cloud frame is equal to the second frame rate of the image frame, it can be obtained by reference, and no further description is given. So far, each laser radar point cloud frame and each image frame that is consistent in time are screened out.

[0128] For the convenience of the following description, a key frame sequence can also be constructed, each key frame including a lidar point cloud frame and an image frame that is most adjacent to it in time.

[0129] In some embodiments of the present application, in the aforementioned step S10, each laser radar point cloud frame is used to perform depth information supervision on the motion structure recovery of each image frame, and a point cloud frame after depth information supervision corresponding to each image frame is generated. This can be specifically achieved through the operations of the following steps S11, S12 and S13.

[0130] S11: Processing the position information of the environment points in each laser radar point cloud frame into a position information reference value for image frame depth information supervision;

[0131] S12: Perform motion structure recovery on each image frame to generate each original point cloud frame corresponding to each image frame, such as Figure 4C As shown;

[0132] S13: using the position information reference value for depth information supervision of each image frame to process each original point cloud frame corresponding to each image frame, and generating each depth information supervised point cloud frame corresponding to each image frame.

[0133] The aforementioned steps S11 and S12 can be executed sequentially or in reverse order, and both can achieve the purpose of processing the position information of the environmental points in each lidar point cloud frame into a position information reference value for image frame depth information supervision and generating each original point cloud frame corresponding to each image frame.

[0134] Specifically, in the aforementioned step S12, each image frame is subjected to motion structure recovery to generate each original point cloud frame corresponding to each image frame, including using the first two adjacent image frames in the image frame sequence as the initial image pair, performing motion structure recovery on the image frame sequence, and generating each original point cloud frame corresponding to each image frame. In the above, each original point cloud frame records the three-dimensional coordinates (x0, y0, z0) of multiple environmental points in the camera coordinate system.

[0135] In the method of this embodiment, the position information of the environmental points in each lidar point cloud frame is processed into a position information reference value for image frame depth information supervision. The position information reference value is used as the depth information for the environmental points extracted from the lidar point cloud frame, thereby realizing lidar-based depth information supervision, which is beneficial to solving the problem of inaccurate recovery of depth information of each environmental point during motion structure recovery of image frames, and can improve the accuracy of depth information in each original point cloud frame corresponding to each image frame.

[0136] In some embodiments of the present application, in the aforementioned step S11, the processing of the position information of the environment point in each laser radar point cloud frame into a position information reference value for image frame depth information supervision includes:

[0137] Using the obtained transformation matrix from the laser radar coordinate system to the camera coordinate system, the environmental point position information recorded in each laser radar point cloud frame is transformed into the camera coordinate system, and used as the position information reference value for image frame depth information supervision;

[0138] Correspondingly, in the aforementioned step S13, the use of the position information reference value for depth information supervision of each image frame to process each original point cloud frame corresponding to each image frame to generate each point cloud frame after depth information supervision corresponding to each image frame, including: taking a weighted average of the position information reference value for depth information supervision of the image frame and the environmental point position information recorded in the original point cloud frame to generate the environmental point position information after depth information supervision, and recording it in each point cloud frame after depth information supervision corresponding to each image frame.

[0139] Specifically, the laser radar point cloud frame records multiple environmental points. Referring to the above description, the three-dimensional coordinates of any environmental point in the laser radar coordinate system are (x, y, z). By using the acquired transformation matrix from the laser radar coordinate system to the camera coordinate system, the three-dimensional coordinates of the environmental point in the laser radar coordinate system can be converted to new three-dimensional coordinates (x', y', z') in the camera coordinate system through coordinate system conversion. The new three-dimensional coordinates (x', y', z') here are used to introduce depth information supervision based on laser radar. In this way, the position information reference value for depth information supervision of the processed image frame is the coordinate value in the camera coordinate system.

[0140] Specifically, a weighted average is performed on the position information reference value for image frame depth information supervision and the environmental point position information recorded in the original point cloud frame, including: constructing a difference function to determine the weights corresponding to the position information reference value for each image frame depth information supervision and the environmental point position information recorded in each original point cloud frame.

[0141] Specifically, the difference function is used to calculate the environmental point X recorded in the lidar point cloud frame l The new pseudo environment point X after conversion to the camera coordinate system lc (which corresponds to the aforementioned new three-dimensional coordinates (x', y', z'), that is, the position information reference value for image frame depth information supervision), and the matching true environment point X recorded in the original point cloud frame corresponding to the image frame consistent in time c (which corresponds to the aforementioned three-dimensional coordinates (x0, y0, z0), that is, the relative distance between the environmental point position information recorded in the original point cloud frame).

[0142] Specifically, the difference function is recorded as f = Diff (X lc ,X c ), the distance threshold is denoted as θ d , X lc ={x,,y,,z,},X c ={x0,y0,z0}, such as,

[0143] Specifically, the distance threshold θ d The value of can be determined according to the installation position of the laser radar and the installation position of the camera, which will not be repeated here.

[0144] Note that for the aforementioned pseudo environment point X lc The weight value is recorded as α, for the true environment point X c The weight value is recorded as β, and α+β=1. Because the depth information obtained by the laser radar is very accurate, it can be used as the true value. Therefore, the following weight distribution method is determined.

[0145] If the aforementioned relative distance is less than the distance threshold, the weight value for the aforementioned pseudo-environment point can be 0.5, and the weight value for the true environment point can be 0.5, that is, the arithmetic mean of the position information reference value for image frame depth information supervision and the environment point position information recorded in the original point cloud frame is used as the environment point position information after depth information supervision. If the aforementioned relative distance is greater than at least an integer multiple of the distance threshold, the weight value for the aforementioned pseudo-environment point can be 1.0, and the weight value for the true environment point can be 0, that is, the position information reference value for image frame depth information supervision is used as the environment point position information after depth information supervision.

[0146] If the aforementioned relative distance is less than at least one integer multiple of the distance threshold and greater than the distance threshold, the weight value for the aforementioned pseudo environment point gradually increases from 0.5 to 1.0, and the weight value for the true environment point gradually decreases from 0.5 to 0.

[0147] In this way, in the camera coordinate system, the closer the pseudo environment point is to the matching true environment point recorded in the original point cloud frame corresponding to the image frame, the smaller the weight value for the pseudo environment point; the farther away, the larger the weight value for the pseudo environment point, so as to gradually increase the role of depth information supervision for the lidar.

[0148] In this way, the introduction of depth information supervision of LiDAR solves the problem of image blur acquired by the camera in high-speed and large-scale motion scenes. By introducing direct supervision based on LiDAR frame by frame, the depth information of each environmental point recorded in the point cloud frame after depth information supervision corresponding to each frame image is adjusted respectively.

[0149] Using the point cloud frames supervised by each depth information to recover the motion structure, after transforming the environment points from the camera coordinate system to the world coordinate system, the obtained landmark point cloud frame can be Figure 4D shown.

[0150] In some embodiments of the present application, in the aforementioned step S20, the point cloud frames after each depth information supervision and each lidar point cloud frame are constructed with joint observation to determine each valid visual point cloud frame that satisfies the motion consistency check, which can be specifically achieved through the operations of the following steps S21, S22 and S23.

[0151] S21: Use the transformation matrix from the lidar coordinate system to the camera coordinate system as the relative motion estimation used for joint observation;

[0152] S22: using the relative motion estimation, determining the overlap between each depth information supervised point cloud frame and each laser radar point cloud frame;

[0153] S23: The point cloud frames after depth information supervision whose overlapping conditions meet the requirements are determined as valid visual point cloud frames that meet the motion consistency check, wherein the number of valid visual point cloud frames is not greater than the number of point cloud frames after depth information supervision; the point cloud frames after depth information supervision whose overlapping conditions do not meet the requirements are determined as invalid visual point cloud frames that do not meet the motion consistency check.

[0154] In the above-mentioned step S21, the transformation matrix from the lidar coordinate system to the camera coordinate system can be used as the relative motion estimation used for joint observation, and the transformation matrix from the camera coordinate system to the lidar coordinate system can be used as the relative motion estimation used for joint observation, which has more intuitive physical meaning and higher accuracy.

[0155] It should be understood that the relative motion estimation used in the above joint observation is the relative motion estimation obtained by observing the camera image frame and the lidar point cloud frame, and is not the relative motion estimation obtained by observing two image frames that are adjacent in time, that is, the camera pose matrix described later.

[0156] In the above-mentioned step S22, in the process of determining the overlap between each point cloud frame after depth information supervision and each lidar point cloud frame, for each point cloud frame after depth information supervision and each lidar point cloud frame that are consistent in time, the relative motion estimation used is consistent in time with the point cloud frame after depth information supervision and the lidar point cloud frame, that is, they correspond one to one.

[0157] The relative motion estimation described above is used to determine the overlap between each point cloud frame after depth information supervision and each lidar point cloud frame, and the point cloud frames after depth information supervision whose overlapping conditions meet the requirements are determined as valid visual point cloud frames that meet the motion consistency check, thereby realizing the use of estimated relative motion; based on the overlap between the point cloud frame after depth information supervision and the lidar point cloud frame, it is determined whether the relative motion indicated by the point cloud frame after depth information supervision is valid, and then whether the point cloud frame after depth information supervision is valid.

[0158] In this way, invalid visual point cloud frames can be deleted. In the subsequent motion structure recovery, only all valid visual point cloud frames are used and invalid visual point cloud frames are discarded, which is beneficial to reduce the time and resource consumption required for motion structure recovery and improve the accuracy and efficiency of scene reconstruction.

[0159] In some embodiments of the present application, in the aforementioned step S22, the use of the relative motion estimation to determine the overlap between each depth information supervised point cloud frame and each lidar point cloud frame specifically includes:

[0160] A source grid is constructed using one of the point cloud frame supervised by the depth information and the laser radar point cloud frame, and a source cell state corresponding to each source position point in the source grid is determined; a target grid is constructed using the other of the laser radar point cloud frame and the point cloud frame supervised by the depth information, and the target grid includes each target position point and each target cell;

[0161] Using the relative motion estimation, each source position point corresponding to each source cell in the source grid whose status is occupied is converted to the target grid to generate each pseudo target position point; in the target grid, the status of each target cell corresponding to each pseudo target position point is determined;

[0162] The number of target cells corresponding to the pseudo target position points that are occupied is determined as the overlap between the point cloud frame supervised by the depth information and the laser radar point cloud frame.

[0163] The above can be to construct a source grid with a point cloud frame supervised by depth information, and determine the source cell state corresponding to each source position point in the source grid; construct a target grid with a lidar point cloud frame, and the target grid includes each target position point and each target cell; use the relative motion estimation to convert each source position point corresponding to each source cell in the source grid whose state is occupied into the target grid to generate each pseudo-target position point; in the target grid, determine the state of each target cell corresponding to each pseudo-target position point; determine the number of target cells whose state is occupied corresponding to each pseudo-target position point as the overlap between the point cloud frame supervised by depth information and the lidar point cloud frame.

[0164] In the above, the source position points corresponding to the occupied source cells in the source grid constructed by the point cloud frame supervised by the depth information from the image frame are converted to the target grids constructed by the lidar point cloud frame, and the status of each target cell corresponding to each pseudo-target position point is determined. When the state of each target cell corresponding to each of the pseudo-target position points is that the number of occupied cells is greater than a preset value, it can be determined that the state of each source cell in the source grid is consistent with the state of each target cell in the target grid, and then it can be determined that the relative motion information indicated by the environmental point position information after depth information supervision recorded in the point cloud frame after depth information supervision is valid, and then it can be determined that the point cloud frame after depth information supervision is a valid visual point cloud frame that meets the motion consistency check; and when the state of each target cell corresponding to each of the pseudo-target position points is that the number of occupied cells is not greater than a preset value, it can be determined that the state of each source cell in the source grid is inconsistent with the state of each target cell in the target grid, and then it can be determined that the relative motion information indicated by the environmental point position information after depth information supervision recorded in the point cloud frame after depth information supervision is invalid, and then it can be determined that the point cloud frame after depth information supervision is an invalid visual point cloud frame that does not meet the motion consistency check.

[0165] The above can also be done by constructing a source grid with a laser radar point cloud frame, determining the source cell state corresponding to each source position point in the source grid; constructing a target grid with a point cloud frame supervised by depth information, wherein the target grid includes each target position point and each target cell; using the relative motion estimation, converting each source position point corresponding to each source cell in the source grid whose state is occupied into the target grid to generate each pseudo target position point; determining the state of each target cell corresponding to each pseudo target position point in the target grid; determining the number of target cells whose state is occupied corresponding to each pseudo target position point as the overlap between the point cloud frame supervised by depth information and the laser radar point cloud frame. For details, please refer to the above implementation and will not be repeated here.

[0166] In this way, the lidar point cloud frame and the point cloud frame supervised by depth information are respectively constructed into source grids or target grids. The overlap of the lidar point cloud frame and the point cloud frame supervised by depth information can be discretized and quantitatively evaluated through the unit discretization method of the grid.

[0167] In this way, the source grid or target grid is constructed correspondingly for the lidar point cloud frame and the point cloud frame supervised by depth information, and the overlap of the lidar point cloud frame to the point cloud frame supervised by depth information and the overlap of the point cloud frame supervised by depth information to the lidar point cloud frame can be obtained respectively. In this way, through cross-validation, it is conducive to more comprehensively deleting invalid visual point cloud frames that do not meet the motion consistency check, which is conducive to reducing the time and resource consumption required for subsequent motion structure recovery, and improving the accuracy and efficiency of scene reconstruction.

[0168] Specifically, when constructing a source grid or a target grid with a point cloud frame or a lidar point cloud frame supervised by depth information, it can include: taking the origin of the camera coordinate system or the lidar coordinate system as the origin of the grid coordinate system and constructing the grid coordinate system, dividing the three-dimensional space into U1*V1*W1 source cells or U2*V2*W2 target cells, wherein the coordinates of the center point of each cell are the position or three-dimensional coordinates of the cell in the corresponding grid coordinate system, each cell is roughly a cuboid, and the spatial volume or spatial area corresponding to each cell is equal.

[0169] For a scene in which the number of cells included in the source grid and the target grid are different, the part of the intersection of the source grid and the target grid is obtained, and the overlap of the point cloud frame or the lidar point cloud frame after depth information supervision is determined.

[0170] When determining the state of the source cell or target cell in the source grid or the target grid, it can include: determining the number of environmental points falling into each cell in the grid coordinate system; when it is greater than a preset value, determining the cell value to be 1, indicating occupation; when it is not greater than a preset value, determining the cell value to be 0, indicating idle. For the case where the number of point clouds falling into the cell is unknown (for example, the divided cell exceeds the target environment), determining the cell value to be -1, indicating unknown. In this way, each cell in the grid takes a value of 1 for occupation, 0 for idle, and -1 for unknown.

[0171] Using the estimated relative motion, the center point coordinates of each cell in the source grid whose value is 1 or whose status is occupied are converted to the target grid coordinate system, the pseudo-target position point of the center point coordinates of the source cell whose status is occupied in the target grid coordinate system is determined, and the status or value of the target cell of the pseudo-target position point in the target grid coordinate system is checked.

[0172] In order to quantify the consistency of point cloud to grid occupancy in two coordinate systems, the indicator function of the cell is defined as follows: I(GR t (R, t));

[0173] Among them, I() is used to represent the indicator function. If GR t The value of (R, t) is true, i.e., in the target grid GR t In the coordinate system, the state of the cell at this position is occupied, then GR t (R, t) is 1, which is true), then the indication value is 1; otherwise, if the parameter GR t The value of (R, t) is false, e.g., in the target grid GR t In the coordinate system, the cell at this position is idle, then GR t(R, t) is 0, which is false), then the indication value is 0.

[0174] Alternatively, if the source grid GR s The cells whose value is 1 or whose status is occupied are transferred to the target grid GR t In the coordinate system, if the value of the corresponding cell is 1 or the state is occupied, then the parameter of the indicator function I() is true, and the indicator value is 1.

[0175] For the source raster GR s The cells whose values ​​are all 1 or whose status is occupied are recorded as Q1, and the cells whose values ​​are all 1 or occupied are recorded as Q1, respectively. t In the coordinate system, the state or value of the cell at this position, that is, the aforementioned indication value, can obtain Q2 indication values ​​with a value of 1, where Q2 is not greater than Q1.

[0176] Specifically, in order to quantify the consistency of the point cloud to the grid occupancy in the two coordinate systems, the grid consistency ratio r can be defined as c as follows:

[0177]

[0178] Referring to the above description, the quotient or ratio of Q2 to Q1 can be determined to obtain the consistency ratio r c Thus, the consistency ratio r c Measured the target grid GR t With the source grid GR s degree of consistency.

[0179] After executing step S20, the number M of valid visual point cloud frames is not greater than the number N of point cloud frames after depth information supervision. The laser radar point cloud frames processed in the following steps are consistent with the valid visual point cloud frames in time, so their number is also N.

[0180] In some embodiments of the present application, in the aforementioned step S22, after using the relative motion estimation to determine the overlap between the point cloud frames after each depth information supervision and each lidar point cloud frame, the following operations S100 and S200 may also be included.

[0181] S100: constructing relative motion frame sequences for the acquired point cloud frames after depth information supervision whose overlap conditions meet the requirements, and determining angle translation combinations corresponding to the relative motion frame sequences using the acquired camera pose matrices;

[0182] S200: Determine that the point cloud frame after the depth information supervision specified in the relative motion frame sequence for which the angle translation combination meets the requirements is a valid visual point cloud frame that meets the motion consistency check; determine that the point cloud frame after the depth information supervision specified in the relative motion frame sequence for which the angle translation combination does not meet the requirements is an invalid visual point cloud frame that does not meet the motion consistency check.

[0183] In the method of this embodiment, relative motion frame sequences are constructed for the point cloud frames after the acquired depth information supervision whose overlapping conditions meet the requirements, and the angle translation combinations corresponding to the relative motion frame sequences are determined using the acquired camera pose matrices; the point cloud frames after the specified depth information supervision in the relative motion frame sequence whose angle translation combinations meet the requirements are determined to be valid visual point cloud frames that meet the motion consistency check, and the point cloud frames after the specified depth information supervision in the relative motion frame sequence whose angle translation combinations do not meet the requirements are determined to be invalid visual point cloud frames that do not meet the motion consistency check. The invalid visual point cloud frames can be further deleted or screened out. During the subsequent motion structure recovery, only all valid visual point cloud frames are obtained without obtaining invalid visual point cloud frames, which is beneficial to reducing the time consumption and resource consumption required for motion structure recovery and improving the accuracy and efficiency of scene reconstruction.

[0184] Specifically, in the aforementioned step S100, the point cloud frames obtained after the depth information supervision whose overlapping conditions meet the requirements are constructed into each relative motion frame sequence, including: recording the multiple point cloud frames obtained after the depth information supervision whose overlapping conditions meet the requirements as multiple verification point cloud frames accordingly; constructing multiple verification point cloud frames that are adjacent in time into a relative motion frame sequence, and in the relative motion frame sequence, each verification point cloud frame is respectively a first frame, an intermediate frame and a last frame.

[0185] Correspondingly, in the aforementioned step S100, the use of the acquired camera pose matrices to determine the angular translation combinations corresponding to the relative motion frame sequences includes: after the acquired camera pose matrices are temporally matched with the verification point cloud frames, the relative motion matrices formed by two adjacent verification point cloud frames are determined from the first frame to the last frame and from the last frame to the first frame in the relative motion frame sequence, and the relative motion matrices are multiplied as the angular translation combinations corresponding to the relative motion frame sequence.

[0186] Specifically, Figure 3As shown, every 4 temporally adjacent check point cloud frames are constructed as a relative motion frame sequence, and the relative motion frame sequence includes 4 check point cloud frames, each check point cloud frame is a first frame, an intermediate frame and a last frame, and includes 2 intermediate frames. When the acquired check point cloud frames are 100 frames in total, every 4 temporally adjacent check point cloud frames are constructed as a relative motion frame sequence, and 87 relative motion frame sequences can be constructed cumulatively. It should be understood that the relative motion frame sequence can include a number of point cloud frames greater than or equal to 3.

[0187] Specifically, the acquired camera pose matrices T are made to correspond to each verification point cloud frame in time, including acquiring each camera pose matrix that is consistent with each verification point cloud frame in time, and making each verification point cloud frame correspond to each camera pose matrix one by one.

[0188] Specifically, obtaining each camera pose matrix that is consistent in time for each verification point cloud frame includes: performing motion structure recovery on each image frame in the aforementioned step S10 to generate each camera pose matrix T corresponding to each image frame; obtaining each camera pose matrix T that is consistent in time with each verification point cloud frame. That is, each camera extrinsic parameter matrix T is obtained by solving the process of performing motion structure recovery on each image frame. Specifically, for each image frame, triangulation is performed on the image frames adjacent to it to obtain the camera pose information corresponding to this image frame. Usually, the pose obtained by triangulation is at the initial moment of the camera coordinate system. After the coordinate system is converted, the camera pose matrix in the world coordinate system is obtained, which is recorded as T. wi , where w is used to indicate the world coordinate system and i is used to indicate the i-th moment. That is, at each moment i corresponding to each image frame, there is a camera pose matrix T i , and the pose camera matrix records the rotation matrix R and the translation vector t, where the rotation matrix R is a 3*3 dimensional matrix and the translation vector t is a single value, T i It is a 4*4 dimensional matrix.

[0189] Furthermore, performing motion structure recovery on each image frame in the aforementioned step S10 to generate each camera pose matrix corresponding to each image frame may also include: obtaining the satellite navigation pose corresponding to each image frame, and weighting the satellite navigation pose and the camera pose information obtained by triangulation solution as the camera pose matrix T corresponding to each image frame.

[0190] Specifically, determining the relative motion matrix formed by two adjacent check point cloud frames includes: determining the previous camera pose matrix T corresponding to the previous check point cloud frame in time i-1 , determine the subsequent camera pose matrix T corresponding to the subsequent verification point cloud frame in time i , determine the inverse matrix T of the front camera pose matrix -1 i-1With the camera pose matrix T behind i The product of the two adjacent check point cloud frames is the relative motion matrix T i-1,i .

[0191] Above, the relative motion matrix T i-1,i It is used to indicate the relative motion between the pose of the image frame or point cloud frame corresponding to time i-1 and the pose of the image frame or point cloud frame corresponding to time i. Figure 4C As shown in the short lines displayed on the outside, these short lines are arranged in chronological order and roughly form a multi-circular outline. This is because the device is constantly moving, so the pose camera matrix corresponding to each image frame is constantly changing, and the relative motion between adjacent image frames is also constantly changing.

[0192] Correspondingly, each relative motion matrix formed by two adjacent check point cloud frames is determined from the first frame to the last frame and from the last frame to the first frame in the relative motion frame sequence, including: respectively determining each two adjacent check point cloud frames from the first frame to the last frame and from the last frame to the first frame in the relative motion frame sequence; and respectively determining each relative motion matrix formed by two adjacent check point cloud frames.

[0193] Correspondingly, the method of multiplying the relative motion matrices as an angle translation combination corresponding to the relative motion frame sequence includes:

[0194] The relative motion matrices are multiplied in sequence from the first frame to the last frame and from the last frame to the first frame in the relative motion frame sequence to obtain the cumulative product of the relative motion matrices as the angle translation combination corresponding to the relative motion frame sequence.

[0195] Specifically, from Figure 3 In the above formula, four adjacent check point cloud frames can be determined, and four relative motion matrices can be formed accordingly, which can be recorded as: T i,j , T j,k , T k,l , T l,i The angle translation combination corresponding to the relative motion frame sequence satisfies T ij T jk T kl T li ∈SE(3). SE(3) is also called Euclidean Transformation or Rigid Transformation. In theory, it is a 4*4 dimensional unit matrix. In practice, it comes from the inconsistency of using images obtained by the camera to observe the motion or posture of the movable vehicle. The angle translation combination is usually not a unit matrix.

[0196] By constructing a relative motion frame sequence and determining the angle translation combination corresponding to each relative motion frame sequence, it is possible to verify whether the camera pose matrices obtained by triangulating each image frame are consistent with each other. This is because the local information corresponding to each point cloud frame in the relative motion frame sequence will not be affected by the cumulative drift, so the result of the joint observation is more accurate.

[0197] Specifically, the angle translation combination includes a rotation matrix R and a translation vector t. Accordingly, in the aforementioned step S200, determining that the angle translation combination satisfies the specified depth information supervision point cloud frame in the relative motion frame sequence is a valid visual point cloud frame that satisfies the motion consistency check includes:

[0198] When the rotation matrix R in the angle translation combination is not greater than a preset angle, and the translation vector t is not greater than a preset translation, it is determined that the angle translation combination meets the requirements;

[0199] It is determined that the first point cloud frame supervised by depth information in the relative motion frame sequence in which the angle translation combination meets the requirements is a valid visual point cloud frame that meets the motion consistency check.

[0200] Accordingly, in the aforementioned step S200, determining that the point cloud frame supervised by the depth information specified in the relative motion frame sequence whose angle translation combination does not meet the requirement is an invalid visual point cloud frame that does not meet the motion consistency check includes:

[0201] The rotation matrix R in the angle translation combination is greater than the preset angle θ R , and the translation vector t is greater than the preset translation amount θ T , determining that the angle translation combination does not meet the requirement;

[0202] It is determined that the first point cloud frame after depth information supervision in the relative motion frame sequence whose angle translation combination does not meet the requirements is an invalid visual point cloud frame that does not meet the motion consistency check.

[0203] In this way, a motion consistency check is performed on the relative motion frame sequence to check whether the camera pose matrices obtained by triangulating each image frame are consistent in observation; and the point cloud frame after depth information supervision corresponding to the inconsistent camera pose matrix is ​​determined to be an invalid point cloud frame, and the camera pose matrix is ​​an invalid camera pose matrix; and the point cloud frame after depth information supervision corresponding to the consistent camera pose matrix is ​​determined to be a valid point cloud frame, and the camera pose matrix is ​​a valid camera pose matrix.

[0204] Specifically, when determining the aforementioned preset angle θ R , and the preset translation amount θ TWhen performing the motion consistency check on the relative motion frame sequence using the aforementioned method, a set of thresholds can be selected (usually smaller values) to determine whether each relative motion frame sequence passes the check by using the set of thresholds; when the number of relative motion frame sequences that pass the check does not meet the preset value, the threshold is increased until the increased threshold can be used to determine whether the number of relative motion frame sequences that pass the check meets the preset value when the motion consistency check on the relative motion frame sequence using the aforementioned method is performed using the specified number of check point cloud frames.

[0205] As described above, after executing steps S100 and S200 , the number H of valid visual point cloud frames is not greater than the number M mentioned above.

[0206] At this point, the effective visual point cloud frames are used to recover the motion structure. After the environmental points are transformed from the camera coordinate system to the world coordinate system, the obtained landmark point cloud frame can be Figure 4D shown.

[0207] In some embodiments of the present application, in step S30, the optimization task of constructing an optimization task for all valid visual point cloud frames using the acquired camera pose matrices to generate optimized camera pose matrices corresponding to each valid visual point cloud frame can be specifically achieved through the operations of the following steps S31, S32 and S33.

[0208] S31: constructing relative motion frame sequences for the acquired valid visual point cloud frames, and determining angle translation combinations corresponding to the relative motion frame sequences using the acquired camera pose matrices;

[0209] S32: combining the angle translations corresponding to the relative motion frame sequences as relative motion sub-item residuals;

[0210] Constructing an optimization task, wherein the optimization variables of the optimization task include the pose matrices of each camera, the cost function of the optimization task includes the cumulative sum of all relative motion sub-item residuals, and the objective function of the optimization task includes minimizing the cost function;

[0211] S33: Solve the optimization task and generate an optimized camera pose matrix corresponding to each valid visual point cloud frame.

[0212] In the aforementioned step S31, relative motion frame sequences are constructed for the acquired valid visual point cloud frames, and the angle translation combinations corresponding to the relative motion frame sequences are determined using the acquired camera pose matrices, including: recording the acquired valid visual point cloud frames correspondingly as multiple verification point cloud frames; and implementing the operations of the multiple verification point cloud frames correspondingly with reference to the aforementioned step S100, which will not be repeated here.

[0213] In this way, the step S31 or step S100 respectively and correspondingly realizes using all acquired valid visual point cloud frames as multiple verification point cloud frames and using all acquired point cloud frames after depth information supervision whose overlapping conditions meet the requirements as multiple verification point cloud frames, which will not be repeated.

[0214] In the above, the angle translation corresponding to each relative motion frame sequence is combined as the relative motion sub-item residual; an optimization task is constructed, wherein the optimization variables of the optimization task include the camera pose matrices, the cost function of the optimization task includes the cumulative sum of all relative motion sub-item residuals, and the objective function of the optimization task includes minimizing the cost function; the optimization task is solved to generate the optimized camera pose matrix corresponding to each valid visual point cloud frame. In this way, all relative motion frame sequences are included in the optimization task, and through multiple rounds of optimization, the camera pose matrices corresponding to the minimum cost function are solved and used as the optimized camera pose matrix corresponding to each valid visual point cloud frame.

[0215] Specifically, in the process of solving the optimization task, the angle translation combination corresponding to each relative motion frame sequence is used as the relative motion sub-item residual as the cost function in each round of iteration, and the cumulative sum of all relative motion sub-item residuals is derived by the solver for multiple rounds. In multiple rounds, the angle translation combination corresponding to each relative motion frame sequence will be updated. In this way, the corresponding camera pose matrices when the cost function is minimized can be solved and used as the optimized camera pose matrices corresponding to each valid visual point cloud frame.

[0216] Refer to the above description, and Figure 3 The optimization task corresponding to the relative motion frame sequence shown can be expressed as follows:

[0217]

[0218] Among them, after performing logarithmic operation on the 4*4 matrix, the specified matrix norm is solved. Specifically, the specified matrix norm can be 0 norm, 1 norm, 2 norm or infinite norm, and the 2 norm is used in the above formula.

[0219] As described above, after executing step S30, the number H of valid visual point cloud frames remains unchanged. By optimizing the camera pose matrix corresponding to each valid visual point cloud frame, the consistency of the relative motion recorded in the optimized camera pose matrix is ​​improved, which is beneficial to improving the accuracy of scene reconstruction.

[0220] It should be understood that when solving the above nonlinear optimization tasks, particle swarm optimization, genetic algorithms, simulated annealing, neural networks and other solution strategies that are conducive to rapid convergence or searching for the global optimal solution can also be used.

[0221] against Figure 4FAfter executing step S30, the optimized camera pose matrix is ​​used to perform visual 3D reconstruction on each of the valid visual point cloud frames to obtain a visual 3D reconstructed image, such as Figure 4G As shown. Figure 4F compared to, Figure 4G The reconstruction quality of the displayed scenes is significantly improved.

[0222] The above illustrates the steps of implementing visual 3D reconstruction for image frames and laser radar frames synchronously acquired by a camera and a laser radar in combination with the accompanying drawings. For scenes where multiple cameras and multiple laser radars synchronously acquire image frames and laser radar frames, one camera and one laser radar are considered as a combination, and multiple cameras and multiple laser radars are respectively considered as multiple combinations. In each combination, the aforementioned visual 3D reconstruction method is independently implemented, and no further description is given.

[0223] Figure 4H The phenomenon of inaccurate scale of three-dimensional reconstruction of large-scale autonomous driving scenes in related technologies is shown. In some embodiments of the present application, in step S40, the acquisition of each navigation pose matrix corresponding to the optimized camera pose matrix, the construction of an optimization task for all navigation pose matrices and the optimized camera pose matrix, and the generation of a camera pose matrix after each pose supervision can be specifically achieved through the operations of the following steps S41, S42 and S43.

[0224] S41: Determine the deviation between the optimized camera pose matrix and each navigation pose matrix as a sub-item pose residual;

[0225] S42: constructing a nonlinear pose optimization task, wherein the optimization variables of the nonlinear pose optimization task include the optimized camera pose matrix matrix, the cost function of the nonlinear pose optimization task includes the weighted average of each sub-item pose residual and each sub-item pose weight, and the objective function of the nonlinear pose optimization task includes minimizing the cost function;

[0226] S43: Solve the nonlinear pose optimization task and generate a camera pose matrix after each pose is supervised.

[0227] In the above, in step S41, the deviation between the optimized camera pose matrix and each navigation pose matrix is ​​determined as the sub-item pose residual, including: obtaining each navigation pose matrix corresponding to the optimized camera pose matrix, wherein each optimized camera pose matrix is ​​consistent with each navigation pose matrix in time, such as corresponding to time i respectively; respectively determining the deviation between the optimized camera pose matrix and each navigation pose matrix, and using them as the sub-item pose residual.

[0228] Referring to the above description, after executing step S30, the number H of valid visual point cloud frames remains unchanged, and accordingly, the number of optimized camera pose matrices is also H. Accordingly, in step S41, it is determined that the number of deviations between the optimized camera pose matrix and each navigation pose matrix is ​​also H, so the number of sub-item pose residuals is also H.

[0229] As above, a nonlinear pose optimization task is constructed, wherein the optimization variables of the nonlinear pose optimization task include the optimized camera pose matrix, the cost function of the nonlinear pose optimization task includes the weighted average of each sub-item pose residual and each sub-item pose weight, and the objective function of the nonlinear pose optimization task includes minimizing the cost function; solving the nonlinear pose optimization task, generating the camera pose matrix after each pose supervision. In this way, all sub-item pose residuals are included in the optimization task, and through multiple rounds of optimization, the camera pose matrix after each pose supervision corresponding to the minimum cost function is obtained.

[0230] In this way, the pose information of the navigation pose matrix is ​​used to supervise the camera pose information obtained by the aforementioned image motion structure recovery. By minimizing the deviation of the pose center constraint, the camera pose matrix after supervision corresponding to each pose when the cost function is minimized can be solved. The camera pose matrix after supervision of each pose can be used for image motion structure recovery to further improve the accuracy of three-dimensional scene reconstruction. In this way, the problem of inaccurate scale of three-dimensional reconstruction of large-scale autonomous driving scenes can be solved.

[0231] In some embodiments of the present application, the step S40 also includes step S44: determining a matching similarity transformation matrix between each of the optimized camera pose matrices and each navigation pose matrix; using the matching similarity transformation matrix, transforming each of the optimized camera pose matrices into a pseudo-navigation pose matrix; determining the deviation between each pseudo-navigation pose matrix and each navigation pose matrix as the pose weight of each sub-item.

[0232] Specifically, the RANSAC algorithm is used to estimate the matching similarity transformation matrix. For example, the optimized camera pose matrix and the matching point pairs of each navigation pose matrix are calculated, and the RANSAC algorithm is used to find the matching similarity transformation matrix. In this way, the similarity transformation is achieved.

[0233] Accordingly, the estimated matching similarity transformation matrix is ​​applied to transform the optimized camera pose matrix into a pseudo navigation pose matrix to align it with each navigation pose matrix. The residual, i.e., the difference between the aligned SFM position and GNSS position, is calculated, i.e., the deviation between each pseudo navigation pose matrix and each navigation pose matrix is ​​determined and used as the pose weight of each sub-item.

[0234] Preferably, the deviations between the pseudo-navigation pose matrices and the navigation pose matrices can also be sorted, and the median value can be determined as the weight of the robust fitting error, that is, the median value can be used as the weight shared by the aforementioned sub-item pose residuals.

[0235] In this way, similarity transformation and robust fitting are used to obtain the residuals and weights of each sub-item, and the weighted average of the residuals of each sub-item posture and the weights of each sub-item posture is introduced into the nonlinear optimization problem as the cost function. The optimizer can adjust the rotation matrix R and the translation vector t recorded in the camera matrix of each posture according to the residual to minimize the deviation of the posture center constraint. The camera pose matrix after supervision corresponding to each posture when the cost function is minimized can be solved, and the camera pose matrix after supervision of each posture can be used for image motion structure recovery to further improve the accuracy of three-dimensional scene reconstruction, which is beneficial to improve the accuracy of the three-dimensional reconstruction scale.

[0236] In this way, the deviation of the posture is recorded in the sub-item residual, and the residual is optimized with the help of the optimizer, so as to achieve supervision of the posture information by minimizing the deviation of the posture center constraint.

[0237] In some embodiments of the present application, in the above-mentioned step S40, in the process of solving the nonlinear pose optimization task, in each intermediate round of iteration, the pose residual of each sub-item is the deviation between the optimized camera pose matrix generated in the previous round and each navigation pose matrix; the weight of each sub-item pose is determined according to the optimized camera pose matrix generated in the previous round and each navigation pose matrix; in the last round of iteration, the generated optimized camera pose matrix is ​​used as the camera pose matrix after supervision of each pose.

[0238] It should be understood that when solving the above nonlinear optimization tasks, particle swarm optimization, genetic algorithms, simulated annealing, neural networks and other solution strategies that are conducive to rapid convergence or searching for the global optimal solution can also be used.

[0239] After executing step S40, the camera pose matrix after each pose supervision can be used to perform visual 3D reconstruction on each valid visual point cloud frame to obtain a visual 3D reconstructed image, such as Fig. 4I As shown. Figure 4H Compared with the related art shown in FIG. Fig. 4I The reconstruction quality of the displayed scenes is significantly improved.

[0240] like Figure 2B As shown, the embodiment of the present application also provides another visual three-dimensional reconstruction method, which is implemented by the operations of the following steps S110, S120, S130, S140, and S150.

[0241] S110: timestamp screening and constructing a key frame sequence.

[0242] Specifically, the LiDAR frame sequence and the image frame sequence are timestamped and a key frame sequence is constructed.

[0243] like

[0244] The LiDAR frame sequence (frame rate is 10 Hz) and the image frame sequence (frame rate is 30 Hz) are screened based on the timestamp. The image frame with the nearest timestamp is selected for each LiDAR frame to construct a key frame sequence (frame rate is 10 Hz). Each key frame K i Each contains a LiDAR frame and an image frame The key frame, the LiDAR frame and the image frame correspond to key frame time i respectively, and the three are consistent in time.

[0245] S120: Use LiDAR to supervise the depth information of the key frame image sequence SFM process to improve the solution accuracy of motion structure recovery.

[0246] Specifically, for each key frame time i, LiDAR measures the state of the environment point in the laser radar coordinate system (including depth information and position information), and transforms the pose from the laser radar coordinate system to the camera coordinate system by T lc The position information of the environmental point in the laser radar coordinate system is transferred to the camera coordinate system as the initial reference value X of the state quantity of the visual SFM l The state quantity X is obtained by performing visual triangulation on the image frame sequence c . Define the difference function f = Diff(X l , X c ), define the threshold θ d When the result of the difference function is less than or equal to the threshold θ d , then the final state quantity is X c and X l Take the mean; if the result of the difference function is greater than the threshold θ d , then the final state quantity only retains X l In this way, the position information of the environment point in the image frame is adjusted.

[0247] S130: construct a joint observation of the landmark point cloud and the lidar point cloud obtained by the key frame image SFM, and select effective features of motion consistency.

[0248] Specifically, for all key frames in the key frame sequence The current image frame The sparse point cloud of the landmark obtained after triangulation of the previous image frame is The overlap between the LiDAR point clouds is calculated and inconsistent point clouds are excluded.

[0249] Specifically, the source grid GR is constructed according to the corresponding point cloud s and the target grid GR t Each cell in the grid takes the value 1 for occupied, 0 for idle, and -1 for unknown. Then, for GR s Each occupied cell in is transformed into GR using the estimated relative motion (R, t) t coordinate system (introducing the transformation of the known quantity LiDAR coordinate system to the camera coordinate system) and checking the cell state at that position. In order to quantify the consistency, the consistency ratio r is defined c as follows:

[0250]

[0251] Where I() is an indicator function, which is 1 if the argument is true and 0 otherwise. c Measuring GR t With GR s degree of consistency.

[0252] In addition, cross-validation is performed by exchanging the target and source rasters and recalculating the consistency ratio. s If the relative motion is invalid, the current image frame and lidar point cloud frame are deleted from the keyframe.

[0253] Specifically, it also includes: defining two groups of key frame pairs with a key frame interval of 4 as a small cycle, and verifying the observation consistency by checking the small cycle, wherein the local information represented by the small cycle will not be affected by the cumulative drift.

[0254] Combined relative motion T ij T jk T kl T li ∈SE(3) and check the resulting angle and translation. c and p c As the thresholds of angle and translation, the inspection results of small loops are marked as passed or failed.

[0255] Specifically, when determining the thresholds of angle and translation, for each relative motion, track its success rate T s , defined as Where N p is the number of checks passed, N i is the number of inspections involved, where T s Below the threshold value θ t Relative motion is considered invalid. If there is still invalid motion, the threshold θ is increased. t .

[0256] By incorporating multiple global small loops into nonlinear optimization and constructing nonlinear optimization equations, we can obtain:

[0257]

[0258] Where Γ is the cost function. Specifically, the overall feature screening is optimized by minimizing the cost function.

[0259] S140: Use GNSS for absolute position monitoring.

[0260] S150: Backend optimization for supervised 3D visual reconstruction.

[0261] Specifically, the pose information of the Global Navigation Satellite System (GNSS) is used to supervise the pose information obtained by SFM to solve the problem of inaccurate scale of 3D reconstruction in large-scale autonomous driving scenes. Residuals are obtained using techniques such as similarity transformation and robust fitting, and the residuals are added to the optimization problem. The optimizer can adjust the rotation and translation parameters of the camera based on these residuals to minimize the deviation of the pose center constraint. The specific methods include:

[0262] The RANSAC algorithm is used to estimate the similarity transformation matrix, and the best similarity transformation matrix is ​​found by calculating the matching point pairs of the SFM and GNSS results. The estimated similarity transformation matrix is ​​applied to transform the SFM position to align it with the GNSS position. Based on the aligned SFM position and GNSS position, the residual, that is, the difference between the two, is calculated; the residuals are sorted and the median value is obtained as the weight of the robust fitting error.

[0263] The rotation and translation vectors of the camera at each moment are used to calculate the difference between them and the GNSS position as the deviation of the attitude center position; the calculated weight is passed to the backend optimization, the weight is multiplied by the deviation of the attitude center position, and the result is stored in the residuals. The residuals are optimized with the help of the optimizer, and the supervision of the posture information is achieved by minimizing the deviation of the attitude center constraint.

[0264] In this way, since SFM visual triangulation solution is affected by factors such as image quality and calibration results, and is prone to inaccurate depth recovery, the introduction of direct depth measurement supervision of LiDAR is conducive to improving the accuracy of SFM visual triangulation solution.

[0265] In this way, since there are many invalid visual features in SFM, it is difficult to screen out invalid feature points only by visual means, and the participation of invalid feature points in matching and reconstruction increases time and resource consumption and is not beneficial to accuracy. LiDAR is introduced for joint observation to realize the screening of invalid features.

[0266] In this way, the visual reconstruction of large-scale and large-scene scenes lacks the constraint of absolute scale information and is prone to convergence failure. The introduction of GNSS constraints is conducive to reconstruction convergence.

[0267] The present application also provides a visual 3D reconstruction device 500 for executing the visual 3D reconstruction method provided in any of the above embodiments. Figure 5 As shown, the device comprises:

[0268] The depth information supervision processing module 501 is used to perform depth information supervision on the motion structure recovery of each image frame using each laser radar point cloud frame, and generate each depth information supervised point cloud frame corresponding to each image frame;

[0269] The effective visual point cloud frame screening module 502 is used to construct a joint observation of each point cloud frame after depth information supervision and each laser radar point cloud frame, and determine each effective visual point cloud frame that meets the motion consistency check;

[0270] The camera pose matrix optimization module 503 is used to construct an optimization task for all valid visual point cloud frames using the acquired camera pose matrices to generate optimized camera pose matrices corresponding to each valid visual point cloud frame;

[0271] The camera pose matrix pose supervision module 504 is used to obtain each navigation pose matrix corresponding to the optimized camera pose matrix, construct an optimization task for all navigation pose matrices and the optimized camera pose matrix, and generate a camera pose matrix after each pose supervision;

[0272] The visual 3D reconstruction module 505 is used to perform visual 3D reconstruction on each of the valid visual point cloud frames using the camera pose matrix after each pose supervision to obtain a visual 3D reconstructed image.

[0273] The visual three-dimensional reconstruction device provided in the embodiment of the present application and the visual three-dimensional reconstruction method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0274] The present application also provides an electronic device to perform the above-mentioned visual three-dimensional reconstruction method. Figure 6 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 6As shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802 and a communication interface 803, and the processor 800, the communication interface 803 and the memory 801 are connected via the bus 802; the memory 801 stores a computer program that can be run on the processor 800, and when the processor 800 runs the computer program, it executes the visual three-dimensional reconstruction method provided by any of the aforementioned embodiments of the present application.

[0275] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 803 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0276] The bus 802 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs, and the processor 800 executes the programs after receiving the execution instructions. The visual three-dimensional reconstruction method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 800, or implemented by the processor 800.

[0277] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 800. The above processor 800 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0278] The electronic device provided in the embodiment of the present application and the visual three-dimensional reconstruction method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0279] The present application also provides a computer-readable storage medium corresponding to the visual three-dimensional reconstruction method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the visual three-dimensional reconstruction method provided by any of the aforementioned embodiments will be executed.

[0280] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0281] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the visual three-dimensional reconstruction method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0282] It should be noted that:

[0283] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0284] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0285] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0286] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A visual three-dimensional reconstruction method, characterized in that: include: Using the data generated by the first non-visual sensor as supervision, performing motion structure recovery on the visual data generated by the visual sensor; Use the first non-visual sensor and the visual sensor to establish a joint observation and filter out valid visual data; The data generated by the second non-visual sensor is used to perform position supervision on the filtered effective visual data and optimize the visual three-dimensional reconstruction.

2. The method according to claim 1, characterized in that The method of using the data generated by the first non-visual sensor as supervision to perform motion structure recovery on the visual data generated by the visual sensor includes: Obtaining depth information generated by a first non-vision sensor; The depth information is used as a supervision item to supervise the depth data obtained after motion structure recovery is performed on the visual data generated by the visual sensor.

3. The method according to claim 2, characterized in that The obtaining of the depth information generated by the first non-visual sensor includes: obtaining a plurality of non-visual point cloud frames generated by the first non-visual sensor, each non-visual point cloud frame corresponding to a timestamp; The using the depth information as a supervision item to supervise the depth data obtained after the motion structure recovery of the visual data generated by the visual sensor includes: Using a transformation matrix, transforming the non-visual point cloud frame into a visual coordinate system; Acquire multiple visual point cloud frames obtained by performing motion structure recovery on visual data generated by a visual sensor, each visual point cloud frame corresponding to a timestamp; Determine a depth information supervision data pair, wherein the depth information supervision data pair includes a converted non-visual point cloud frame and a visual point cloud frame with consistent timestamps; perform weighted averaging on the converted non-visual point cloud frame in the depth information supervision data pair and the visual point cloud frame in the depth information supervision data pair to generate a visual point cloud frame after depth information supervision.

4. The method according to claim 1, characterized in that Using the first non-visual sensor and the visual sensor to construct a joint observation and filter out valid visual data includes: Determine a joint observation data pair, the joint observation data pair comprising a non-visual point cloud frame with consistent timestamps and a visual point cloud frame supervised by depth information; For each of the joint observation data pairs, the observation matrix is ​​used to observe the overlap between the non-visual point cloud frame and the visual point cloud frame after depth information supervision, and the visual point cloud frame after depth information supervision whose overlap meets the requirements is taken as a valid visual point cloud frame and determined as valid visual data.

5. The method according to claim 4, characterized in that The observed overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information includes: Constructing a source grid according to the non-visual point cloud frame, determining the source cell state corresponding to each source grid point in the constructed source grid; constructing a target grid according to the visual point cloud frame supervised by the depth information, wherein the constructed target grid includes each target grid point and each corresponding target cell; using the corresponding transformation matrix from the first non-visual coordinate system to the visual coordinate system as the observation matrix corresponding to the first non-visual sensor and the visual sensor; or A source grid is constructed according to the visual point cloud frame supervised by depth information, and the source cell state corresponding to each source grid point in the constructed source grid is determined; a target grid is constructed according to the non-visual point cloud frame, and the constructed target grid includes each target grid point and the corresponding target cell; and a transformation matrix from the corresponding visual coordinate system to the first non-visual coordinate system is used as an observation matrix corresponding to the first non-visual sensor and the visual sensor.

6. The method according to claim 5, characterized in that The observing the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information also includes: Observe the states of target cells in the target grid corresponding to the pseudo target grid points obtained by performing coordinate transformation on the source grid points corresponding to the source cells in the source grid whose states are occupied; The number of target cells corresponding to the observed pseudo-target grid points that are occupied is counted, and the number is used to indicate the overlap between the non-visual point cloud frame and the visual point cloud frame supervised by the depth information.

7. The method according to claim 1, characterized in that Using data generated by a second non-visual sensor to perform position supervision on the filtered effective visual data and optimize visual three-dimensional reconstruction, including: The data generated by the second non-visual sensor includes a navigation pose matrix and a navigation position corresponding to a timestamp; The filtered valid visual data includes a camera pose matrix corresponding to a timestamp; Use the navigation pose matrix to supervise the camera pose matrix consistent with its timestamp based on the navigation position and generate the corresponding residual; Based on the residuals corresponding to each time stamp, an optimization task for the filtered valid visual data is constructed, the optimization task is solved, and a camera pose matrix after position supervision corresponding to each time stamp is generated; The visual 3D reconstruction is optimized according to the supervised camera pose matrix at each position to generate an optimized visual 3D reconstructed image.

8. The method according to claim 4, characterized in that Also includes: Arrange the camera pose matrices corresponding to all valid visual point cloud frames in the order of timestamps, select a preset number of camera pose matrices adjacent to each other from the beginning to the end to form a relative motion estimation sequence, and determine the angle translation combination corresponding to the relative motion estimation sequence; Determine that the effective visual point cloud frame corresponding to the camera pose matrix specified in the relative motion estimation sequence in which the angle translation combination meets the requirements is the secondary observation effective visual point cloud frame.

9. The method according to claim 4, characterized in that Also includes: Arrange the camera pose matrices corresponding to all valid visual point cloud frames in the order of timestamps, select a preset number of adjacent camera pose matrices from the beginning to the end to form a relative motion estimation sequence, and determine the angle translation combinations corresponding to each relative motion estimation sequence; Taking the combination of translation amounts at various angles as the cost function, an optimization task for all valid visual point cloud frames is constructed, and the optimization task is solved with the minimum cost function as the constraint to generate an optimized camera pose matrix corresponding to each valid visual point cloud frame.

10. The method according to claim 8 or 9, characterized in that The step of selecting a preset number of camera pose matrices adjacent to each other from the beginning to the end to form a relative motion estimation sequence, and determining an angle translation amount combination corresponding to the relative motion estimation sequence, comprises: Selecting a preset number of camera pose matrices adjacent to each other from the front to the back to form a relative motion estimation sequence, wherein the preset number of camera pose matrices in the relative motion estimation sequence are respectively a first matrix, a middle matrix or a last matrix; Determine the relative motion estimation matrices from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the next intermediate matrix, and from the last intermediate matrix to the tail matrix in sequence; multiply the relative motion matrices as the angle translation combination corresponding to the relative motion estimation sequence.

11. The method according to any one of claims 1 to 9, characterized in that The first non-visual sensor includes a radar device; the visual sensor includes an optical camera; and the second non-visual sensor includes a position sensor.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 11.

Images