Method and device for screening point cloud, equipment and storage medium

Abstract

The invention provides a point cloud screening method and device and a storage medium, and the method comprises the steps: carrying out the motion structure recovery calculation based on an image frame collected by a visual sensor, and obtaining a plurality of visual point cloud frames; and based on the plurality of visual point cloud frames and the non-visual point cloud frames collected by the first non-visual sensor, screening out effective visual point cloud frames from the plurality of visual point cloud frames. According to the invention, based on combined observation of the first non-visual sensor and the visual sensor, invalid feature points in the visual data obtained by motion structure recovery are reduced. And secondary screening can be carried out based on a camera pose matrix corresponding to the screened effective visual point cloud frame. A camera pose matrix can also be optimized, pose supervision is performed on the camera pose matrix based on a navigation pose, and depth supervision is performed on motion structure recovery based on a first non-visual sensor. The calculation amount and calculation resource consumption of subsequent three-dimensional scene reconstruction are reduced, and the efficiency and accuracy of three-dimensional scene reconstruction are improved.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and specifically relates to a method, device, equipment and storage medium for screening point clouds. Background Art

[0002] In the scenario of autonomous driving, the vehicle uses a camera to capture image frames of the surrounding environment, and performs triangulation based on continuous image frames to recover the structure from motion, and convert the two-dimensional image into a three-dimensional point cloud. The accuracy of the three-dimensional point cloud has a great impact on the accuracy of three-dimensional scene reconstruction.

[0003] Related technologies: There are many invalid point clouds in the point cloud obtained by motion structure recovery. The invalid point clouds participate in the subsequent three-dimensional scene reconstruction, which increases the time and resource consumption and reduces the accuracy of three-dimensional scene reconstruction. Summary of the invention

[0004] The present application proposes a method, device, equipment and storage medium for screening point clouds, which can alleviate the technical problem of a large number of invalid point clouds in the point clouds obtained by motion structure recovery in the related art.

[0005] The first embodiment of the present application provides a method for screening a point cloud, comprising:

[0006] Based on the image frames collected by the visual sensor, motion structure recovery is performed to obtain multiple visual point cloud frames;

[0007] Based on the multiple visual point cloud frames and the non-visual point cloud frames acquired by the first non-visual sensor, valid visual point cloud frames are screened out from the multiple visual point cloud frames.

[0008] In some embodiments of the present application, the filtering out valid visual point cloud frames from the multiple visual point cloud frames based on the multiple visual point cloud frames and the non-visual point cloud frames collected by the first non-visual sensor includes:

[0009] Based on the multiple visual point cloud frames and the non-visual point cloud frames collected by the first non-visual sensor, a plurality of joint observation data pairs are constructed; the joint observation data pairs include a non-visual point cloud frame and a visual point cloud frame with consistent timestamps;

[0010] For each of the joint observation data pairs, detecting the overlap between the non-visual point cloud frame and the visual point cloud frame in the joint observation data pair;

[0011] The visual point cloud frames whose overlapping conditions meet the requirements are determined as valid visual point cloud frames.

[0012] In some embodiments of the present application, the detecting the overlap between the non-visual point cloud frame and the visual point cloud frame in the joint observation data pair includes:

[0013] Construct a source grid according to a first point cloud frame in a first joint observation data pair, and determine a source cell state corresponding to each source grid point in the source grid; the first joint observation data pair is any data pair in the plurality of joint observation data pairs, and the first point cloud frame is one of a non-visual point cloud frame and a visual point cloud frame in the first joint observation data pair;

[0014] Constructing a target grid according to a second point cloud frame in the first joint observation data pair, wherein the target grid includes each target grid point and each corresponding target cell; the second point cloud frame is another point cloud frame in the first joint observation data pair except the first point cloud frame;

[0015] Based on the source cell states corresponding to the source grid points and the target cell states corresponding to the target grid points, the overlap between the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair is detected through a preset observation matrix; the preset observation matrix is ​​a transformation matrix between the coordinate system of the first non-visual sensor and the coordinate system of the visual sensor.

[0016] In some embodiments of the present application, based on the source cell states corresponding to the source grid points and the target cell states corresponding to the target grid points, detecting the overlap between the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair through a preset observation matrix includes:

[0017] Through the preset observation matrix, coordinate transformation is performed on each source grid point corresponding to each source cell in the source grid whose state is occupied, so as to obtain each pseudo target grid point corresponding to each source grid point whose state is occupied;

[0018] Respectively obtain the states of each target cell corresponding to each pseudo target grid point in the target grid;

[0019] The number of occupied target cells corresponding to the pseudo target grid points is counted, and the number is used to indicate an overlap between the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair.

[0020] In some embodiments of the present application, the method further includes:

[0021] Sort the camera pose matrices corresponding to each valid visual point cloud frame in the order of timestamps;

[0022] From the sequence of camera pose matrices obtained by sorting, a preset number of continuous camera pose matrices are selected to form a relative motion estimation sequence, and the angle translation combinations corresponding to each relative motion estimation sequence are determined respectively;

[0023] The valid visual point cloud frame corresponding to the camera pose matrix specified in the relative motion estimation sequence that satisfies the angle translation combination is determined as the valid visual point cloud frame for secondary observation.

[0024] In some embodiments of the present application, the method further includes:

[0025] Sort the camera pose matrices corresponding to each valid visual point cloud frame in the order of timestamps;

[0026] From the sequence of camera pose matrices obtained by sorting, a preset number of continuous camera pose matrices are selected to form a relative motion estimation sequence, and the angle translation combinations corresponding to each relative motion estimation sequence are determined respectively;

[0027] Taking the combination of each angle translation as the cost function, an optimization task for each valid visual point cloud frame is constructed, and the optimization task is solved with the minimum cost function as the constraint to generate an optimized camera pose matrix corresponding to each valid visual point cloud frame.

[0028] In some embodiments of the present application, the preset number of camera pose matrices in the relative motion estimation sequence include a first matrix, an intermediate matrix, and a last matrix; and the step of respectively determining the angle translation combinations corresponding to each relative motion estimation sequence includes:

[0029] Determine in sequence the relative motion estimation matrices from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the adjacent next intermediate matrix, and from the last intermediate matrix to the tail matrix;

[0030] The relative motion estimation matrices are multiplied to obtain an angle translation combination corresponding to the relative motion estimation sequence.

[0031] In some embodiments of the present application, the method further includes:

[0032] Using the optimized camera pose matrix corresponding to each valid visual point cloud frame, visual 3D reconstruction is performed on each valid visual point cloud frame to obtain a visual 3D reconstructed image.

[0033] An embodiment of a second aspect of the present application provides a movable vehicle, comprising: a first non-visual sensor, a visual sensor, and a controller;

[0034] The controller is used to execute the method for screening point clouds described in the first aspect based on the image frames collected by the visual sensor and the non-visual point cloud frames collected by the first non-visual sensor.

[0035] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0036] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0037] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0038] In an embodiment of the present application, joint observation is performed based on the first non-visual sensor and the visual sensor to reduce invalid feature points in the visual data obtained by motion structure recovery, reduce the amount of calculation and computing resource consumption for subsequent three-dimensional scene reconstruction based on the visual data restored by motion structure, and improve the efficiency and accuracy of three-dimensional scene reconstruction.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0041] In the attached picture:

[0042] Figure 1 A flow chart showing a method for screening point clouds provided by an embodiment of the present application is shown;

[0043] Figure 2A A flow chart showing a method for screening point clouds provided by another embodiment of the present application is shown;

[0044] Figure 2B A flowchart showing a method for reconstructing a three-dimensional scene provided by another embodiment of the present application is shown;

[0045] Figure 3 A schematic diagram showing the steps of constructing a relative motion estimation sequence in an embodiment of the present application is shown;

[0046] Figure 4A An example of an image frame generated by a camera obtained in an embodiment of the present application is shown;

[0047] Figure 4BAn example of a laser radar point cloud frame generated by a laser radar acquired by an embodiment of the present application is shown;

[0048] Figure 4C An example of a landmark point cloud formed after motion structure recovery is performed on multiple image frames within a preset time span in one embodiment of the present application is shown;

[0049] Figure 4D An example of a landmark point cloud formed after motion structure recovery is performed using depth information supervision for multiple image frames within a preset time span in one embodiment of the present application is shown;

[0050] Figure 4E Shows Figure 4D A partial enlarged view of the

[0051] Figure 4F An example of a point cloud frame of an environment point formed after motion structure recovery is performed on multiple image frames within a preset time span in one embodiment of the present application is shown;

[0052] Figure 4G An example of a landmark point cloud formed by performing joint observation on multiple image frames within a preset time span and then performing motion structure recovery in one embodiment of the present application is shown;

[0053] Figure 4H An example diagram of a landmark point cloud formed when the motion structure recovery for a large-scale and large-range scene in the related art is difficult to converge is shown;

[0054] Fig. 4I An example diagram of a landmark point cloud formed by performing motion structure recovery based on satellite navigation posture for a large-scale and large-range scene in one embodiment of the present application is shown;

[0055] Figure 5 A schematic diagram of the structure of a point cloud screening device provided by an embodiment of the present application is shown;

[0056] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0057] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0059] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0060] The following describes a method, device, equipment and storage medium for screening point clouds according to an embodiment of the present application in conjunction with the accompanying drawings.

[0061] Vision-based 3D scene reconstruction is a very important task in autonomous driving environment perception. The task of 3D scene reconstruction is mainly to identify important road signs or environmental points for autonomous driving in the images obtained by the camera, such as vehicles, pedestrians, non-motor vehicles, traffic facilities on the road, roadside buildings, etc.

[0062] In related technologies, optical cameras acquire images from different angles and distances, and combine them with precise position information (such as absolute position) provided by position sensors, such as GNSS (Global Navigation Satellite System), inertial navigation or wheel speed meters, to achieve real-time positioning and motion estimation at high frame rates. High-precision three-dimensional maps can be reconstructed, and the positions and shapes of landmarks or environmental points can be updated in real time by reusing the maps.

[0063] Based on the images captured by the camera, the algorithms for motion estimation include Structure From Motion (SFM), such as extracting feature points from each image using the image differences between adjacent image frames, and matching feature points between frames. After constructing the matching feature point pairs, triangulation is performed to obtain the depth information of the environment points and the camera pose matrix.

[0064] At present, there are many invalid point clouds in the point clouds obtained by motion structure recovery in related technologies. The invalid point clouds participate in the subsequent three-dimensional scene reconstruction, which increases the time and resource consumption and reduces the accuracy of three-dimensional scene reconstruction.

[0065] Based on this, an embodiment of the present application provides a method for screening point clouds, which introduces a first non-visual sensor, constructs a joint observation based on the first non-visual sensor and the visual sensor, and screens effective point cloud frames for the visual point cloud frames obtained by motion structure recovery. The three-dimensional scene reconstructed based on the screened effective point cloud frames is more accurate and the visual effect is more realistic, which improves the accuracy of the point cloud frames used for three-dimensional scene reconstruction and helps to improve the efficiency and effect of subsequent three-dimensional scene reconstruction.

[0066] The execution subject of the embodiment of the present application can be a movable vehicle for indoor sports scenes or for high-speed, large-scale outdoor scenes. The movable vehicle can be a vehicle, a ship, an aircraft or a robot, etc. When the movable vehicle moves, low speed can include a moving speed of less than 20km / h, medium speed can include a moving speed of 20-60km / h, and high speed can include a moving speed of more than 60km / h. Taking 60km / h as an example, the moving distance per minute is 1km. When the movable vehicle moves at medium or high speed, it belongs to the scene of large-scale, large-scale autonomous driving environment perception and visual reconstruction. In practical applications, there can be other ways of dividing low speed, medium speed and high speed, which are not limited by the embodiment of the present application.

[0067] The movable vehicle is usually provided with an environmental information acquisition device to acquire the external environmental information of the vehicle. In some embodiments, the environmental information acquisition device can be classified into a visual sensor and a non-visual sensor. Among them, the visual sensor can include but is not limited to an optical camera, and the non-visual sensor can include but is not limited to a radar device and a position sensor.

[0068] Among them, the optical camera may include but is not limited to RGB (Red Green Blue) three primary color camera, infrared camera, etc. Among them, the radar device may include but is not limited to LiDAR (Light Detection and Ranging), Radar (Radio Detection and Ranging, Radar), Ultrasonic Sensor, Synthetic Aperture Radar, etc. Position sensors may include but are not limited to satellite navigation systems (such as GNSS), Inertial Measurement Unit (Inertial Measurement Unit, IMU), Gyroscope, Accelerometer, Magnetometer, etc. The number of visual sensors and non-visual sensors set on the movable carrier can be one or more.

[0069] The visual sensor set on the movable vehicle is explained by taking an RGB camera as an example. The RGB camera can obtain the following information by shooting the target area: Figure 4A The image frame with the coloring effect is shown. Further, based on the image frames acquired continuously in time, the point cloud of each landmark or environmental point in the target area can be generated by performing inter-frame calculation through SFM to obtain the three-dimensional coordinates of each environmental point.

[0070] The non-visual sensor installed on the movable vehicle is explained by taking the laser radar as an example. The laser radar emits laser pulses to the target area. The laser pulses are reflected back when they encounter objects in the target area. Based on the reflected echo received by the laser radar, the following information can be obtained: Figure 4B Point clouds of each environment point in the target area shown. Usually each LiDAR point cloud frame can include multiple environment points, each of which has data in four dimensions: x, y, z, and echo reflectivity, where x, y, z represent the three-dimensional coordinates of the environment point in the LiDAR coordinate system.

[0071] The target area may include but is not limited to one or more areas in front, left, right or rear of the movable vehicle. In the embodiment of the present application, a laser radar and an RGB camera are installed on the movable vehicle as an example for detailed description.

[0072] See also Figure 1 , the method for screening point clouds may include the following steps S1 to S2.

[0073] S1: Based on the image frames collected by the visual sensor, motion structure recovery is performed to obtain multiple visual point cloud frames.

[0074] S2: Based on the multiple visual point cloud frames and the non-visual point cloud frames acquired by the first non-visual sensor, select a valid visual point cloud frame from the multiple visual point cloud frames.

[0075] The executor of the embodiment of the present application is a movable carrier, such as a vehicle, a ship, an aircraft, etc. The above-mentioned first non-visual sensor can be a laser radar, a millimeter-wave radar, an ultrasonic radar, etc. The first non-visual sensor arranged on the movable carrier transmits a detection signal to the target area around the movable carrier, and obtains a point cloud of the target area based on the reflected detection signal. In the embodiment of the present application, the data collected by the first non-visual sensor is referred to as a non-visual point cloud, and the non-visual point cloud includes multiple non-visual point cloud frames. The first non-visual sensor detects the target area according to a certain sampling frequency, so each non-visual point cloud frame collected corresponds to a timestamp.

[0076] The above-mentioned visual sensor can be an RGB camera, etc. A plurality of visual sensors can be arranged on the movable carrier, and a movable carrier is taken as a vehicle, and a visual sensor is an RGB camera as an example. One or more RGB cameras can be arranged at the front, rear, left, right, etc. positions of the vehicle. The visual data collected by the visual sensor may include an image of the target area. The visual sensor also shoots the target area at a certain sampling frequency, so the data collected by the visual sensor may include multiple image frames, and each image frame corresponds to a timestamp.

[0077] In the embodiment of the present application, a first non-visual sensor is introduced, and based on the data collected by the first non-visual sensor, the visual point cloud frames obtained by performing motion structure recovery and solving based on the data collected by the visual sensor are screened, and invalid visual point cloud frames and valid visual point cloud frames can be accurately distinguished, invalid visual point cloud frames are eliminated, and valid point cloud frames are retained. Thereby reducing the number of point cloud frames involved in the subsequent three-dimensional scene reconstruction, and improving the accuracy of these point cloud frames, thereby improving the accuracy of three-dimensional scene reconstruction, and reducing the time and computing resources consumed by three-dimensional scene reconstruction.

[0078] In some embodiments of the present application, the screening process of step S2 may specifically include: constructing multiple joint observation data pairs based on multiple visual point cloud frames and non-visual point cloud frames collected by the first non-visual sensor; the joint observation data pair includes a non-visual point cloud frame and a visual point cloud frame with consistent timestamps; for each joint observation data pair, detecting the overlap between the non-visual point cloud frame and the visual point cloud frame in the joint observation data pair; and determining the visual point cloud frame whose overlap meets the requirements as a valid visual point cloud frame.

[0079] The above-mentioned timestamp consistency can be understood as that the timestamp of the non-visual point cloud frame belonging to the same joint observation data pair is the same as the timestamp of the visual point cloud frame. Alternatively, the timestamp of the non-visual point cloud frame belonging to the same joint observation data pair is the closest to the timestamp of the visual point cloud frame. Among them, the closest means that among the multiple visual point cloud frames obtained by performing motion structure recovery and solution on the image frames captured by the visual sensor, the time difference between the timestamp of the visual point cloud frame belonging to the same joint observation data pair as the non-visual point cloud frame and the timestamp of the non-visual point cloud frame is the smallest.

[0080] In the related art, there are many invalid features in the visual data after the motion structure is restored. It is difficult to filter out invalid feature points only by visual means, and the participation of invalid feature points in matching and reconstruction increases time consumption and resource consumption without benefiting the accuracy. The embodiment of the present application introduces a first non-visual sensor, and through the joint observation of the first non-visual sensor and the visual sensor, it is possible to filter out invalid features, improve the accuracy of three-dimensional reconstruction, and reduce the time consumption and computing resource consumption of the three-dimensional reconstruction process.

[0081] In some embodiments of the present application, the above-mentioned overlapping detection process may include: constructing a source grid based on the first point cloud frame in the first joint observation data pair, and determining the source cell state corresponding to each source grid point in the source grid; the first joint observation data pair is any data pair among multiple joint observation data pairs, and the first point cloud frame is one of the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair; constructing a target grid based on the second point cloud frame in the first joint observation data pair, and the constructed target grid includes each target grid point and the corresponding target cells; the second point cloud frame is another point cloud frame other than the first point cloud frame in the first joint observation data pair; based on the source cell state corresponding to each source grid point and the target cell state corresponding to each target grid point, detecting the overlapping situation of the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair through a preset observation matrix; the preset observation matrix is ​​the transformation matrix between the coordinate system of the first non-visual sensor and the coordinate system of the visual sensor.

[0082] In some embodiments, the first point cloud frame is a non-visual point cloud frame in the first joint observation data pair, and the second point cloud frame is a visual point cloud frame in the first joint observation data pair. In this embodiment, the preset observation matrix is ​​a transformation matrix for converting from the coordinate system of the first non-visual sensor to the coordinate system of the visual sensor.

[0083] In other embodiments, the first point cloud frame is a visual point cloud frame in the first joint observation data pair, and the second point cloud frame is a non-visual point cloud frame in the first joint observation data pair. In this embodiment, the preset observation matrix is ​​a transformation matrix for converting from the coordinate system of the visual sensor to the coordinate system of the first non-visual sensor.

[0084] The above-mentioned source grid and target grid construction methods can be performed through point cloud rasterization processing, which will not be described in detail in the embodiments of the present application.

[0085] By rasterizing the non-visual point cloud frames and the visual point cloud frames belonging to the same joint observation data pair, the overlap between the non-visual point cloud frames and the visual point cloud frames is determined based on the state of each grid cell in the two grids. Determining the overlap by grid cell can improve the accuracy of the determined overlap, and thus can accurately eliminate invalid visual data and retain valid visual data.

[0086] In some other embodiments of the present application, the overlap of the two point cloud frames can be determined by taking the first point cloud frame as the non-visual point cloud frame in the first joint observation data pair and the second point cloud frame as the visual point cloud frame in the first joint observation data pair. Also, the overlap of the two point cloud frames can be determined by taking the first point cloud frame as the visual point cloud frame in the first joint observation data pair and the second point cloud frame as the non-visual point cloud frame in the first joint observation data pair. The overlap determined by the two methods is cross-validated to further improve the accuracy of screening valid visual point cloud frames.

[0087] In some embodiments of the present application, the above-mentioned process based on grid detection overlap may specifically include: through a preset observation matrix, coordinate transformation of each source grid point corresponding to each source cell in the source grid whose status is occupied is performed respectively, to obtain each pseudo-target grid point corresponding to each source grid point whose status is occupied; respectively obtain the status of each target cell corresponding to each pseudo-target grid point in the target grid; count the number of target cells corresponding to each pseudo-target grid point whose status is occupied, and use the number to indicate the overlap between the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair.

[0088] The state of a cell in the grid may include occupied or unoccupied. A cell in the occupied state is used to indicate that there is an obstacle at the grid point in the cell, and unoccupied is used to indicate that there is no obstacle at the grid point in the cell.

[0089] In the above embodiment, the occupied grid points in the source grid are transformed by using a preset observation matrix, which is equivalent to transforming the occupied source grid points to the same coordinate system as the target grid. In this way, in the same coordinate system, the pseudo-target grid points obtained by transforming the occupied source grid points have corresponding target cells in the target grid, and the number of occupied cells in the target cells corresponding to the pseudo-target grid points can characterize the overlap between the non-visual point cloud frame and the visual point cloud frame in the same joint observation data pair. The above-mentioned overlap can be determined by coordinate system conversion and simple calculations, with a small amount of calculation, and high calculation efficiency and accuracy.

[0090] After detecting the overlap of two point cloud frames in the joint observation data pair in the above manner, if the overlap value indicated by the overlap is greater than a preset threshold, the visual point cloud frame in the joint observation data pair is determined as a valid visual point cloud frame. Otherwise, the visual point cloud frame in the joint observation data pair is determined as an invalid visual point cloud frame and is discarded.

[0091] In some embodiments of the present application, each valid visual data screened out by the above-mentioned embodiments may include, in addition to a valid visual point cloud frame, a camera pose matrix associated with the visual point cloud frame. The visual point cloud frame is obtained by solving an image frame acquired by a visual sensor, and the camera pose matrix associated with the visual point cloud frame may be a camera pose matrix when the visual sensor acquires an image frame for solving the visual point cloud frame.

[0092] In an embodiment of the present application, after obtaining multiple valid visual point cloud frames in the above manner, the camera pose matrices corresponding to each valid visual point cloud frame can also be sorted in the order of timestamps; from the sorted camera pose matrix sequence, a continuous preset number of camera pose matrices are selected to form a relative motion estimation sequence, and the angle translation combinations corresponding to each relative motion estimation sequence are determined respectively; the valid visual point cloud frames corresponding to the camera pose matrices specified in the relative motion estimation sequence whose angle translation combinations meet the requirements are determined as the valid visual point cloud frames for secondary observation.

[0093] The preset number may be 3, 4, 6 or 8, etc. The number of relative motion estimation sequences constructed in the above manner may be one or more.

[0094] The process of determining the angle translation combination corresponding to the above-mentioned relative motion estimation sequence may include: a preset number of camera pose matrices in the relative motion estimation sequence include a first matrix, an intermediate matrix and a last matrix; determining in sequence the relative motion estimation matrices pointing from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the adjacent next intermediate matrix, and from the last intermediate matrix to the tail matrix; and multiplying each relative motion estimation matrix as the angle translation combination corresponding to the relative motion estimation sequence.

[0095] The “pointing” used to illustrate the relationship between matrices above defines the matrix operation relationship of the product of the inverse matrix of the previous camera pose matrix and the next camera pose matrix.

[0096] The first matrix, the middle matrix and the last matrix are the divisions of the arrangement positions of the multiple camera pose matrices in the relative motion estimation sequence. The first matrix can be the first or the first certain number of matrices in the relative motion estimation sequence, and the last matrix can be the last or the last certain number of matrices in the relative motion estimation sequence. The middle matrix refers to the matrix between the first matrix and the last matrix in the relative motion estimation sequence.

[0097] The above relative motion estimation matrices are all calculated based on two adjacent camera pose matrices, and are used to reflect the relative motion between two adjacent camera pose matrices. The above relative motion estimation matrices are determined in sequence from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the adjacent next intermediate matrix, and from the last intermediate matrix to the tail matrix. Specifically, it means that the product between the matrix of the first matrix and the front intermediate matrix is ​​calculated, and the product is a relative motion estimation matrix. Similarly, the product between the inverse matrix of the front intermediate matrix and another intermediate matrix adjacent to the intermediate matrix is ​​calculated to obtain a relative motion estimation matrix, and multiple relative motion estimation matrices are calculated in sequence in this way.

[0098] The above-mentioned angle translation combination includes angle and translation, wherein the angle can be a rotation matrix and the translation can be a translation vector. In the embodiment of the present application, an angle threshold and a translation threshold are pre-configured in the movable vehicle. After determining the angle translation combination corresponding to the relative motion estimation sequence in the above manner, the angle in the combination is compared with the pre-configured angle threshold, and the translation in the combination is compared with the pre-configured translation threshold.

[0099] The above angle translation combination satisfies the requirement that the angle in the combination satisfies the magnitude relationship between the angle threshold and the translation amount satisfies the magnitude relationship between the translation amount threshold. For example, the angle is less than or equal to the angle threshold, and the translation amount is less than or equal to the translation amount threshold.

[0100] The camera pose matrix specified in the relative motion estimation sequence whose angle translation combination meets the requirements may be all or part of the camera pose matrices in the relative motion estimation sequence. For example, the specified camera pose matrix may be the first camera pose matrix in the relative motion estimation sequence whose angle translation combination meets the requirements. The valid visual point cloud frames corresponding to these camera pose matrices are determined as valid visual point cloud frames for secondary observation. Among them, the valid visual point cloud frames for secondary observation represent valid visual point cloud frames obtained by further screening based on the camera pose matrix on the basis of the valid visual point cloud frames screened out by the joint observation in step S2. The accuracy of the valid visual point cloud frames is further improved, which helps to improve the accuracy of the subsequent three-dimensional reconstruction scene.

[0101] In some embodiments of the present application, the camera pose matrices corresponding to each valid visual point cloud frame can also be sorted in order of timestamps; from the sorted camera pose matrix sequence, a continuous preset number of camera pose matrices are selected to form a relative motion estimation sequence, and the angle translation combinations corresponding to each relative motion estimation sequence are determined respectively; using each angle translation combination as a cost function, an optimization task for each valid visual point cloud frame is constructed, and the optimization task is solved with the minimum cost function as a constraint to generate an optimized camera pose matrix corresponding to each valid visual point cloud frame.

[0102] The determination method of the above relative motion estimation sequence and its corresponding angle translation amount combination is the same as the determination method described above, which will not be repeated here.

[0103] In this embodiment, the combination of each angle translation amount is used as the cost function to construct an optimization task for each valid visual point cloud frame, and the optimization task is solved with the minimum cost function as the constraint to generate an optimized camera pose matrix corresponding to each valid visual point cloud frame. Subsequently, based on the optimized camera pose matrix, each valid visual point cloud frame is reconstructed into a three-dimensional scene to obtain a three-dimensional scene reconstructed image. The accuracy of the three-dimensional scene reconstructed image can be improved, so that the reconstructed image is more in line with the real scene situation.

[0104] In other embodiments of the present application, after the effective visual data is screened out through the above embodiments, the camera pose matrix corresponding to the screened effective visual data can also be supervised using the data collected by the second non-visual sensor. The second non-visual sensor can include but is not limited to GNSS.

[0105] Specifically, the data collected by the second non-visual sensor include a navigation pose matrix and a navigation position corresponding to a timestamp; the screened out valid visual data include a camera pose matrix corresponding to a timestamp; the navigation pose matrix is ​​used to perform supervision on the camera pose matrix consistent with its timestamp based on the navigation position to generate corresponding residuals; based on the residuals corresponding to each timestamp, an optimization task for the screened out valid visual data is constructed, the optimization task is solved, and a camera pose matrix after pose supervision corresponding to each timestamp is generated.

[0106] Subsequently, based on the camera pose matrix after pose supervision, the three-dimensional scene is reconstructed for each valid visual point cloud frame to obtain a three-dimensional scene reconstructed image, which can further improve the accuracy of the reconstructed image.

[0107] In some other embodiments of the present application, before performing joint observation to screen valid visual point cloud frames in step S2, the data collected by the first non-visual sensor can also be used to perform depth supervision on the visual point cloud frames obtained by performing motion structure recovery based on the visual data collected by the visual sensor. Specifically, based on multiple non-visual point cloud frames collected by the first non-visual sensor, the depth information of the environmental points in each non-visual point cloud frame is obtained respectively; the depth information is used as a supervision item to perform motion structure recovery on the visual data collected by the visual sensor.

[0108] In the above-mentioned manner, in the motion structure recovery, the data collected by the first non-visual sensor is introduced to perform depth information supervision, which is beneficial to improving the accuracy of the motion structure recovery solution, and further improving the accuracy of the visual three-dimensional reconstruction.

[0109] In some embodiments of the present application, the process of obtaining the depth information of the environmental points in each non-visual point cloud frame is the same. The embodiment of the present application takes the process of obtaining the depth information of the environmental points in the first non-visual point cloud frame as an example to explain the process in detail. Among them, the first non-visual point cloud frame is any one of the multiple non-visual point cloud frames collected by the first non-visual sensor. Specifically, from the first non-visual point cloud frame collected by the first non-visual sensor, the position information of the environmental points in the first non-visual point cloud frame is obtained; the position information of the environmental points is converted from the coordinate system of the first non-visual sensor to the coordinate system of the visual sensor to obtain the depth information of the environmental points.

[0110] For each other visual point cloud frame collected by the first non-visual sensor, the depth information of the environment point in each other visual point cloud frame is obtained respectively in the above manner.

[0111] In some embodiments of the present application, the obtained depth information is used as a supervision item to perform motion structure recovery on the visual data collected by the visual sensor, including: performing motion structure recovery and solving the image frames included in the visual data collected by the visual sensor to obtain multiple visual point cloud frames; constructing multiple supervision data pairs based on each non-visual point cloud frame and the multiple visual point cloud frames; wherein the supervision data pair includes a non-visual point cloud frame and a visual point cloud frame with consistent timestamps; and performing weighted averaging on the depth information of the environmental points in the non-visual point cloud frame included in the same supervision data pair and the depth information of the same environmental points in the visual point cloud frame to obtain a visual point cloud frame supervised based on the depth information.

[0112] The above-mentioned timestamp consistency can be understood as that the timestamp of the non-visual point cloud frame belonging to the same supervised data pair is the same as the timestamp of the visual point cloud frame. Alternatively, the timestamp of the non-visual point cloud frame belonging to the same supervised data pair is the most adjacent to the timestamp of the visual point cloud frame. Among them, the most adjacent means that among the multiple visual point cloud frames obtained by performing motion structure recovery and solution on the image frames collected by the visual sensor, the time difference between the timestamp of the visual point cloud frame belonging to the same supervised data pair as the non-visual point cloud frame and the timestamp of the non-visual point cloud frame is the smallest.

[0113] Since the motion structure recovery solution is affected by the uniform speed of the image quality and calibration results collected by the visual sensor, it is easy to have the problem of inaccurate depth recovery. Through the above method, the depth information of the environmental points in the non-visual point cloud frame with the same timestamp is weighted averaged with the depth information of the same environmental points in the visual point cloud frame obtained by motion structure recovery, so as to introduce depth supervision based on non-visual sensors in motion structure recovery, which helps to improve the problem of inaccurate depth recovery in motion structure recovery and improve the accuracy of motion structure recovery.

[0114] After deep supervision is performed in the above manner, the depth information in the visual point cloud frame obtained by motion structure recovery is more accurate. On this basis, the effective visual point cloud frames are screened through joint observation in step S2, which can improve the accuracy of the effective visual point cloud frames finally screened out.

[0115] In the embodiment of the present application, joint observation is performed based on the first non-visual sensor and the visual sensor to reduce invalid feature points in the visual data obtained by motion structure recovery, reduce the amount of calculation and computing resource consumption of subsequent three-dimensional scene reconstruction based on the visual data restored by motion structure, and improve the efficiency and accuracy of three-dimensional scene reconstruction. Furthermore, the first non-visual sensor is introduced to monitor the depth information of motion structure recovery, alleviate the problem of inaccurate depth recovery in motion structure recovery, and improve the accuracy of motion structure recovery triangulation solution.

[0116] In some other embodiments of the present application, taking the first visual sensor as a laser radar and the visual sensor as an RGB camera as an example, the data collected by the laser radar is a laser radar point cloud frame of the target area, and the RGB camera collects image frames of the target area, and motion structure recovery is performed based on continuous image frames to obtain a point cloud frame of the target area. Figure 2A , the method for screening point clouds may include the following steps S10 to S30.

[0117] S10: Based on the image frames captured by the camera, a motion structure recovery solution is performed to obtain multiple visual point cloud frames.

[0118] S20: constructing a joint observation for the above-mentioned multiple visual point cloud frames and each lidar point cloud frame, and determining that the visual point cloud frames that meet the motion consistency check are each valid visual point cloud frame.

[0119] S30: Perform secondary screening on each valid visual point cloud frame using the acquired camera pose matrices to obtain a final valid visual point cloud frame.

[0120] The visual point cloud frames obtained by motion structure recovery and the laser radar point cloud frames are jointly observed to determine the valid visual point cloud frames that meet the motion consistency check and delete the invalid visual point cloud frames, which is conducive to reducing the time and resource consumption required for subsequent 3D scene reconstruction and improving the accuracy and efficiency of scene reconstruction. All valid visual point cloud frames are further screened using the acquired camera pose matrices, which further improves the accuracy of valid visual point cloud frame screening.

[0121] In some embodiments of the present application, in the aforementioned step S20, determining each valid visual point cloud frame that satisfies the motion consistency check can be specifically implemented through the operations of the following steps S21 and S22.

[0122] S21: Based on a preset observation matrix, determine the overlap between each lidar point cloud frame and the visual point cloud frame obtained by motion structure recovery.

[0123] S22: Determine the visual point cloud frames whose overlapping conditions meet the requirements as valid visual point cloud frames that meet the motion consistency check, wherein the number of valid visual point cloud frames is not greater than the number of visual point cloud frames obtained by motion structure recovery; determine the visual point cloud frames whose overlapping conditions do not meet the requirements as invalid visual point cloud frames that do not meet the motion consistency check.

[0124] In the aforementioned step S21, the transformation matrix from the lidar coordinate system to the camera coordinate system can be used as the preset observation matrix, or the transformation matrix from the camera coordinate system to the lidar coordinate system can be used as the preset observation matrix, which has more intuitive physical meaning and higher accuracy.

[0125] In the above, a preset observation matrix is ​​used to determine the overlap between the visual point cloud frames after motion structure restoration and the lidar point cloud frames, and the visual point cloud frames that meet the overlap requirements are determined as valid visual point cloud frames that meet the motion consistency check. This realizes the overlap between the visual point cloud frames and the lidar point cloud frames obtained by motion structure restoration to determine whether the relative motion indicated by the visual point cloud frames obtained by motion structure restoration is valid, and then to determine whether the visual point cloud frames are valid.

[0126] In this way, invalid visual point cloud frames can be deleted. In the subsequent 3D scene reconstruction, only all valid visual point cloud frames are used and invalid visual point cloud frames are discarded, which is beneficial to reduce the time and resource consumption required for 3D scene reconstruction and improve the accuracy and efficiency of scene reconstruction.

[0127] In some embodiments of the present application, in the aforementioned step S22, the overlap between each point cloud frame obtained by motion structure recovery and each lidar point cloud frame is determined, specifically including: constructing a source grid with one of the visual point cloud frame and the lidar point cloud frame obtained by motion structure recovery, and determining the state of the source cell corresponding to each source position point in the source grid; constructing a target grid with the other of the lidar point cloud frame and the visual point cloud frame obtained by motion structure recovery, the target grid including each target position point and each target cell; using a preset observation matrix, converting each source position point corresponding to each source cell in the source grid whose state is occupied to the target grid, generating each pseudo-target position point; in the target grid, determining the state of each target cell corresponding to each pseudo-target position point; determining the number of occupied target cells corresponding to each pseudo-target position point as the overlap between the visual point cloud frame obtained by motion structure recovery and the lidar point cloud frame.

[0128] In the case where the source grid is constructed by the visual point cloud frame obtained by motion structure recovery and the target grid is constructed by the laser radar point cloud frame, when the state of each target cell corresponding to each pseudo target position point is that the number of occupied cells is greater than a preset value, it can be determined that the state of each source cell in the source grid is consistent with the state of each target cell in the target grid, and then it can be determined that the relative motion information indicated by the visual point cloud frame obtained by motion structure recovery is valid, and then it can be determined that the visual point cloud frame obtained by motion structure recovery is a valid visual point cloud frame that meets the motion consistency check. When the state of each target cell corresponding to each pseudo target position point is that the number of occupied cells is not greater than a preset value, it can be determined that the state of each source cell in the source grid is inconsistent with the state of each target cell in the target grid, and then it can be determined that the relative motion information indicated by the visual point cloud frame obtained by motion structure recovery is invalid, and then it can be determined that the visual point cloud frame obtained by motion structure recovery is an invalid visual point cloud frame that does not meet the motion consistency check.

[0129] When the source grid is constructed using the lidar point cloud frame and the target grid is constructed using the visual point cloud frame obtained by motion structure recovery, the judgment process of whether the visual point cloud frame obtained by motion structure recovery is valid is the same as the above judgment process and will not be repeated here.

[0130] In the embodiment of the present application, the overlap between the laser radar point cloud frame and the visual point cloud frame obtained by motion structure recovery can be discretized and quantitatively evaluated by means of a unitized discretization method such as a grid. The source grid or the target grid is constructed correspondingly to the laser radar point cloud frame and the visual point cloud frame obtained by motion structure recovery, respectively, and the overlap between the laser radar point cloud frame and the visual point cloud frame obtained by motion structure recovery, as well as the overlap between the visual point cloud frame obtained by motion structure recovery and the laser radar point cloud frame can be obtained grid by grid. In this way, through cross-validation, it is beneficial to more comprehensively delete invalid visual point cloud frames that do not meet the motion consistency check, which is beneficial to reduce the time consumption and resource consumption required for subsequent motion structure recovery, and improve the accuracy and efficiency of scene reconstruction.

[0131] Specifically, constructing a source grid or a target grid with a visual point cloud frame or a lidar point cloud frame obtained by motion structure restoration can include: taking the origin of the camera coordinate system or the lidar coordinate system as the origin of the grid coordinate system and constructing the grid coordinate system, dividing the three-dimensional space into U1*V1*W1 source cells or U2*V2*W2 target cells, wherein the coordinates of the center point of each cell are the position or three-dimensional coordinates of the cell in the corresponding grid coordinate system, and each cell can be roughly a rectangular parallelepiped, and the spatial volume or spatial area corresponding to each cell is equal.

[0132] For scenes where the number of cells included in the source grid and the target grid are different, the intersection of the source grid and the target grid is obtained, and the overlap of the visual point cloud frame or the lidar point cloud frame obtained by motion structure recovery is determined.

[0133] When determining the state of a source cell or a target cell in a source grid or a target grid, it may include: determining the number of environmental points falling into each cell in the grid coordinate system. When the number of environmental points falling into is greater than a preset value, the state value of the cell is determined to be 1, indicating occupation; when it is not greater than a preset value, the state value of the cell is determined to be 0, indicating idle. For the case where the number of environmental points falling into a cell is unknown (e.g., the divided cell is outside the target environment), the state value of the cell is determined to be -1, indicating unknown. In this way, the state value of each cell in the grid is 1 for occupation, 0 for idle, and -1 for unknown.

[0134] Using the preset observation matrix, the center point coordinates of each cell in the source grid whose value is 1 or whose status is occupied are converted to the target grid coordinate system, and the pseudo-target position point of the center point coordinates of the source cell whose status is occupied in the target grid coordinate system is determined, and the status or value of the target cell where the pseudo-target position point is located in the target grid coordinate system is checked.

[0135] In order to quantify the consistency of the grid occupancy of environmental points in the two coordinate systems, the indicator function of the cell is defined as follows: I(GR t (R, t));

[0136] Where I() represents the indicator function. If GR t The value of (R, t) is true, i.e., in the target grid GR t In the coordinate system, the state of the cell at this position is occupied, then GR t (R, t) is 1, which is true, indicating that the function takes the value of 1. Otherwise, if the parameter GR t The value of (R, t) is false, e.g., in the target grid GR t In the coordinate system, the cell at this position is idle, then GR t The value of (R, t) is 0, which is false, indicating that the function takes the value of 0.

[0137] Alternatively, if the source grid GR s The cells whose value is 1 or whose status is occupied are transferred to the target grid GR t In the coordinate system, if the value of the corresponding cell is 1 or the state is occupied, then the parameter of the indicator function I() is true, and the indicator function takes the value of 1.

[0138] For the source raster GR s The cells whose values ​​are all 1 or whose status is occupied are recorded as Q1, and the cells whose values ​​are all 1 or occupied are recorded as Q1, respectively. t In the coordinate system, the state or value of the cell at this position, that is, the value of the aforementioned indicator function, can obtain Q2 indicator values ​​with a value of 1, where Q2 is not greater than Q1.

[0139] Specifically, in order to quantify the consistency of the point cloud to the grid occupancy in the two coordinate systems, the grid consistency ratio r can be defined as c as follows:

[0140]

[0141] Referring to the above description, the quotient or ratio of Q2 to Q1 can be determined to obtain the consistency ratio r c Thus, the consistency ratio r c Measured the target grid GR t With the source grid GR s degree of consistency.

[0142] After step S20 is executed, the number M of valid visual point cloud frames is not greater than the number N of visual point cloud frames recovered by motion structure. The laser radar point cloud frames processed in the following steps are consistent with the valid visual point cloud frames in time, so their number is also M.

[0143] In some embodiments of the present application, in the aforementioned step S22, after determining the overlap between each visual point cloud frame and each lidar point cloud frame, the following operations S100 and S200 may also be included.

[0144] S100: constructing relative motion estimation sequences for the acquired visual point cloud frames whose overlap conditions meet the requirements, and determining angle translation combinations corresponding to the relative motion estimation continuous current sequences using the acquired camera pose matrices;

[0145] S200: Determine that the visual point cloud frames specified in the relative motion frame sequence whose angle translation combination meets the requirements are valid visual point cloud frames that meet the motion consistency check; determine that the visual point cloud frames specified in the relative motion frame sequence whose angle translation combination does not meet the requirements are invalid visual point cloud frames that do not meet the motion consistency check.

[0146] In the above method, invalid visual point cloud frames can be further deleted or screened out. In the subsequent three-dimensional scene reconstruction, only all valid visual point cloud frames are used without invalid visual point cloud frames, which is beneficial to reduce the time and resource consumption required for three-dimensional scene reconstruction and improve the accuracy and efficiency of scene reconstruction.

[0147] Specifically, in the aforementioned step S100, relative motion estimation sequences are constructed for the acquired visual point cloud frames whose overlapping conditions meet the requirements, including: recording the multiple visual point cloud frames whose overlapping conditions meet the requirements correspondingly as multiple verification point cloud frames; constructing multiple verification point cloud frames that are adjacent in time into a relative motion estimation sequence, and within the relative motion estimation sequence, each verification point cloud frame is divided into a first frame, an intermediate frame and a last frame.

[0148] Correspondingly, in the aforementioned step S100, the obtained camera pose matrices are used to determine the angle translation combinations corresponding to the relative motion estimation sequences, including: after the obtained camera pose matrices are corresponded to the verification point cloud frames in time, the relative motion matrices formed by two adjacent verification point cloud frames are determined from the first frame to the last frame and from the last frame to the first frame in the relative motion estimation sequence, and the relative motion matrices are multiplied as the angle translation combinations corresponding to the relative motion estimation sequence.

[0149] Specifically, Figure 3As shown, every 4 temporally adjacent check point cloud frames are constructed as a relative motion estimation sequence, and the relative motion estimation sequence includes 4 check point cloud frames, each of which is a first frame, an intermediate frame, and a last frame, and includes 2 intermediate frames. When the acquired check point cloud frames total 100 frames, every 4 temporally adjacent check point cloud frames are constructed as a relative motion estimation sequence, and 87 relative motion estimation sequences can be constructed cumulatively. It should be understood that the relative motion estimation sequence can include a number of point cloud frames greater than or equal to 3.

[0150] Specifically, each acquired camera pose matrix T is made to correspond to each verification point cloud frame in time, including acquiring each camera pose matrix that is consistent with each verification point cloud frame in time, and making each verification point cloud frame correspond to each camera pose matrix one by one.

[0151] Specifically, obtaining each camera pose matrix that is consistent in time for each verification point cloud frame includes: performing motion structure recovery on each image frame in the aforementioned step S10 to generate each camera pose matrix T corresponding to each image frame; and obtaining each camera pose matrix T that is consistent in time with each verification point cloud frame. That is, each camera pose matrix T is obtained by solving in the process of performing motion structure recovery on each image frame.

[0152] For each image frame, triangulate the adjacent image frames to obtain the camera pose information corresponding to this image frame. The pose obtained by triangulation is the pose of the camera coordinate system at the initial moment. After the coordinate system conversion, the camera pose matrix in the world coordinate system is obtained, which is recorded as T wi , where w is used to indicate the world coordinate system and i is used to indicate the i-th moment. That is, at each moment i corresponding to each image frame, there is a camera pose matrix T i , and the camera pose matrix includes a rotation matrix R and a translation vector t, where the rotation matrix R can be a 3*3 dimensional matrix and the translation vector t can be a single value, T i It can be a matrix of 4*4 dimensions.

[0153] Furthermore, performing motion structure recovery on each image frame in the aforementioned step S10 to generate each camera pose matrix corresponding to each image frame may also include: obtaining the satellite navigation pose corresponding to each image frame, and weighting the satellite navigation pose and the camera pose information obtained by triangulation solution as the camera pose matrix T corresponding to each image frame.

[0154] Specifically, determining the relative motion matrix formed by two adjacent check point cloud frames includes: determining the previous camera pose matrix T corresponding to the previous check point cloud frame in time i-1 , determine the subsequent camera pose matrix T corresponding to the subsequent verification point cloud frame in time i. Determine the inverse matrix T of the previous camera pose matrix -1 i-1 With the camera pose matrix T behind i The product of the two adjacent check point cloud frames is the relative motion matrix T i-1,i .

[0155] Above, the relative motion matrix T i-1,i It is used to indicate the relative motion between the pose of the image frame or point cloud frame corresponding to time i-1 and the pose of the image frame or point cloud frame corresponding to time i. Figure 4C As shown in the short lines displayed on the outside, these short lines are arranged in chronological order and roughly form a multi-circular outline. This is because the device is constantly moving, so the pose camera matrix corresponding to each image frame is constantly changing, and the relative motion between adjacent image frames is also constantly changing.

[0156] Correspondingly, the relative motion matrices formed by two adjacent check point cloud frames are determined from the first frame to the last frame and from the last frame to the first frame in the relative motion estimation sequence, including: respectively determining two adjacent check point cloud frames from the first frame to the last frame and from the last frame to the first frame in the relative motion estimation sequence; and respectively determining the relative motion matrices formed by two adjacent check point cloud frames.

[0157] Accordingly, the relative motion matrices are multiplied to obtain an angle translation combination corresponding to the relative motion estimation sequence, including:

[0158] The relative motion matrices are multiplied in sequence from the first frame to the last frame and from the last frame to the first frame in the relative motion estimation sequence to obtain the cumulative product of the relative motion matrices as the angle translation combination corresponding to the relative motion estimation sequence.

[0159] Specifically, from Figure 3 Four adjacent check point cloud frames can be determined, and four relative motion matrices are formed accordingly, which can be recorded as: T i,j 、T j,k 、T k,l 、T l,i The angle translation combination corresponding to the relative motion estimation sequence satisfies T ii T jk T kl T li ∈SE(3). SE(3) is also called Euclidean Transformation or Rigid Transformation. In theory, it is a 4*4 dimensional unit matrix. In practice, it comes from the inconsistency of using images obtained by the camera to observe the motion or posture of the movable vehicle. The angle translation combination is usually not a unit matrix.

[0160] By constructing a relative motion estimation sequence and determining the angle translation combination corresponding to each relative motion estimation sequence, it is possible to verify whether the camera pose matrices obtained by triangulating each image frame are consistent with each other. This is because the local information corresponding to each point cloud frame in the relative motion estimation sequence will not be affected by the cumulative drift, so the result of the joint observation is more accurate.

[0161] Specifically, the angle translation combination includes a rotation matrix R and a translation vector t. Accordingly, in the aforementioned step S200, determining that the specified visual point cloud frame in the relative motion estimation sequence that satisfies the angle translation combination as a valid visual point cloud frame that satisfies the motion consistency check includes:

[0162] When the rotation matrix R in the angle translation combination is not greater than the preset angle and the translation vector t is not greater than the preset translation, it is determined that the angle translation combination meets the requirements; the first visual point cloud frame in the relative motion estimation sequence in which the angle translation combination meets the requirements is determined to be a valid visual point cloud frame that meets the motion consistency check.

[0163] Accordingly, in the aforementioned step S200, determining that the visual point cloud frame specified in the relative motion estimation sequence whose angle translation combination does not meet the requirement is an invalid visual point cloud frame that does not meet the motion consistency check includes:

[0164] The rotation matrix R in the angle translation combination is greater than the preset angle θ R , and the translation vector t is greater than the preset translation amount θ T When , it is determined that the angle translation combination does not meet the requirements; it is determined that the first visual point cloud frame in the relative motion estimation sequence whose angle translation combination does not meet the requirements is an invalid visual point cloud frame that does not meet the motion consistency check.

[0165] The relative motion estimation sequence is subjected to motion consistency check to check whether the camera pose matrices obtained by triangulating each image frame are consistent in observation. The visual point cloud frame corresponding to the inconsistent camera pose matrix is ​​determined to be an invalid point cloud frame, and the camera pose matrix is ​​an invalid camera pose matrix. The visual point cloud frame corresponding to the consistent camera pose matrix is ​​determined to be a valid point cloud frame, and the camera pose matrix is ​​a valid camera pose matrix.

[0166] Specifically, when determining the aforementioned preset angle θ R , and the preset translation amount θ TWhen the relative motion estimation sequence is subjected to the motion consistency check using the aforementioned method, a set of threshold values ​​(usually a smaller value) can be selected to determine a specified number of check point cloud frames respectively using the set of threshold values. When the relative motion estimation sequence is subjected to the motion consistency check using the aforementioned method, whether each relative motion estimation sequence passes the check. When the number of relative motion estimation sequences that pass the check does not meet the preset number value, the threshold value is increased until the specified number of check point cloud frames can be determined using the increased threshold value. When the relative motion estimation sequence is subjected to the motion consistency check using the aforementioned method, the number of relative motion estimation sequences that pass the check meets the preset number value.

[0167] As described above, after executing steps S100 and S200 , the number H of valid visual point cloud frames is not greater than the number M mentioned above.

[0168] At this point, the three-dimensional scene is reconstructed using each valid visual point cloud frame. After the environmental points are transformed from the camera coordinate system to the world coordinate system, the obtained landmark point cloud frame can be Figure 4D As shown, Figure 4E for Figure 4D A partial enlarged view of the .

[0169] In some embodiments of the present application, before performing 3D scene reconstruction, each camera pose matrix obtained can also be used to construct an optimization task for each valid visual point cloud frame to generate an optimized camera pose matrix corresponding to each valid visual point cloud frame. Using the optimized camera pose matrix, 3D scene reconstruction is performed on each valid visual point cloud frame to obtain a 3D scene reconstructed image, further improving the accuracy of 3D scene reconstruction.

[0170] The above-mentioned optimization of the camera pose matrix can be specifically achieved through the operations of the following steps S31, S32 and S33.

[0171] S31: constructing relative motion estimation sequences for the acquired valid visual point cloud frames, and determining angle translation combinations corresponding to the relative motion estimation sequences using the acquired camera pose matrices;

[0172] S32: The angle translations corresponding to the relative motion estimation sequences are combined as the relative motion sub-item residuals to construct an optimization task, wherein the optimization variables of the optimization task include the pose matrices of each camera, the cost function of the optimization task includes the cumulative sum of all relative motion sub-item residuals, and the objective function of the optimization task includes minimizing the cost function.

[0173] S33: Solve the optimization task and generate an optimized camera pose matrix corresponding to each valid visual point cloud frame.

[0174] The operation of the aforementioned step S31 is the same as the operation of the aforementioned step S100 and will not be repeated here.

[0175] In the above way, all relative motion estimation sequences are included in the optimization task. Through multiple rounds of optimization, the corresponding camera pose matrices when the cost function is minimized are solved and used as the optimized camera pose matrices corresponding to each valid visual point cloud frame.

[0176] Specifically, in the process of solving the optimization task, the angle translation combination corresponding to each relative motion estimation sequence is used as the relative motion sub-item residual as the cost function in each round of iteration, and the cumulative sum of all relative motion sub-item residuals is derived by the solver for multiple rounds. In multiple rounds, the angle translation combination corresponding to each relative motion estimation sequence will be updated. In this way, the corresponding camera pose matrices when the cost function is minimized can be solved and used as the optimized camera pose matrices corresponding to each valid visual point cloud frame.

[0177] Refer to the above description, and Figure 3 The optimization task corresponding to the relative motion estimation sequence shown can be expressed as follows:

[0178]

[0179] After performing logarithmic operations on the 4*4 matrix, the specified matrix norm is solved. The specified matrix norm can be 0 norm, 1 norm, 2 norm, or infinite norm, etc. The above formula is 2 norm.

[0180] By optimizing the camera pose matrix corresponding to each valid visual point cloud frame, the consistency of the relative motion recorded by the optimized camera pose matrix is ​​improved, which is beneficial to improving the accuracy of scene reconstruction.

[0181] It should be understood that when solving the above nonlinear optimization tasks, particle swarm optimization, genetic algorithms, simulated annealing, neural networks and other solution strategies that are conducive to rapid convergence or searching for the global optimal solution can also be used.

[0182] against Figure 4F The landmark point cloud image shown in FIG. 1 uses the optimized camera pose matrix to reconstruct the three-dimensional scene for each valid visual point cloud frame to obtain a three-dimensional scene reconstructed image, as shown in FIG. Figure 4G As shown. Figure 4F compared to, Figure 4G The reconstruction quality of the displayed scenes is significantly improved.

[0183] The above, combined with the accompanying drawings, illustrates the process of implementing point cloud screening and 3D scene reconstruction for image frames and lidar point cloud frames synchronously acquired by a camera and a lidar. For scenes where multiple cameras and multiple lidars synchronously acquire image frames and lidar point cloud frames, one camera and one lidar are considered as a combination, and multiple cameras and multiple lidars are considered as multiple combinations respectively. In each combination, the aforementioned method is independently implemented and will not be described in detail.

[0184] On the basis of the above-mentioned optimization of the camera pose matrix corresponding to the effective visual point cloud frame, each navigation pose matrix corresponding to the optimized camera pose matrix can also be obtained, and the optimization task is constructed for all navigation pose matrices and the optimized camera pose matrix to generate the camera pose matrix after each pose supervision. Then, the three-dimensional scene is reconstructed for each effective visual point cloud frame using each camera pose matrix after pose supervision to obtain a three-dimensional scene reconstructed image, which is conducive to the rapid convergence of scene reconstruction and improves the accuracy and efficiency of scene reconstruction.

[0185] Figure 4H This shows the phenomenon that the scale of the three-dimensional reconstruction of large-scale autonomous driving scenes in the related art is inaccurate. The above-mentioned posture supervision of the camera posture matrix can be specifically implemented through the operations of the following steps S41, S42 and S43.

[0186] S41: Determine the deviation between the optimized camera pose matrix and each navigation pose matrix as a sub-item pose residual.

[0187] S42: Construct a nonlinear pose optimization task, wherein the optimization variables of the nonlinear pose optimization task include the optimized camera pose matrix matrix, the cost function of the nonlinear pose optimization task includes the weighted average of each sub-item pose residual and each sub-item pose weight, and the objective function of the nonlinear pose optimization task includes minimizing the cost function.

[0188] S43: Solve the nonlinear pose optimization task and generate the camera pose matrix after each pose supervision.

[0189] In step S41, the deviation between the optimized camera pose matrix and each navigation pose matrix is ​​determined as the sub-item pose residual, including: obtaining each navigation pose matrix corresponding to the optimized camera pose matrix, wherein each optimized camera pose matrix is ​​consistent with each navigation pose matrix in time, such as corresponding to time i respectively; respectively determining the deviation between the optimized camera pose matrix and each navigation pose matrix and using them as the sub-item pose residual.

[0190] After step S30 is executed, the number H of valid visual point cloud frames remains unchanged, and accordingly, the number of optimized camera pose matrices is also H. Correspondingly, in step S41, it is determined that the number of deviations between the optimized camera pose matrix and each navigation pose matrix is ​​also H, so the number of sub-item pose residuals is also H. All sub-item pose residuals are included in the optimization task, and through multiple rounds of optimization, the camera pose matrices corresponding to each pose supervision when the cost function is minimized are solved.

[0191] The pose information of the navigation pose matrix is ​​used to supervise the camera pose information. By minimizing the deviation of the pose center constraint, the camera pose matrix after supervision corresponding to each pose when the cost function is minimized can be solved. The camera pose matrix after supervision for each pose can be used for three-dimensional scene reconstruction to further improve the reconstruction accuracy and alleviate the problem of inaccurate scale of three-dimensional reconstruction of large-scale autonomous driving scenes.

[0192] In some embodiments of the present application, the previous step S40 uses the optimized camera pose matrices to reconstruct the three-dimensional scene for each valid visual point cloud frame to obtain a three-dimensional scene reconstructed image, and also includes step S44: determining the matching similarity transformation matrix between each optimized camera pose matrix and each navigation pose matrix; using the matching similarity transformation matrix, transforming each optimized camera pose matrix into a pseudo-navigation pose matrix; determining the deviation between each pseudo-navigation pose matrix and each navigation pose matrix as the pose weight of each sub-item.

[0193] Specifically, the RANSAC (Random sample consensus) algorithm is used to estimate the matching similarity transformation matrix. For example, the optimized camera pose matrix and the matching point pairs of each navigation pose matrix are calculated, and the RANSAC algorithm is used to find the matching similarity transformation matrix to achieve similarity transformation.

[0194] Accordingly, the estimated matching similarity transformation matrix is ​​applied to transform the optimized camera pose matrix into a pseudo navigation pose matrix to align it with each navigation pose matrix. The residual, i.e., the difference between the aligned SFM (Structure from Motion) position and GNSS position, is calculated, i.e., the deviation between each pseudo navigation pose matrix and each navigation pose matrix is ​​determined and used as the pose weight of each sub-item.

[0195] The deviations between the pseudo-navigation pose matrices and the navigation pose matrices may also be sorted to determine the median value as the weight of the robust fitting error, that is, the median value is used as the weight shared by the aforementioned sub-item pose residuals.

[0196] In this way, similarity transformation and robust fitting are used to obtain the residuals and weights of each sub-item, and the weighted average of the residuals of each sub-item pose and the weights of each sub-item pose is introduced into the nonlinear optimization problem as the cost function. The optimizer can adjust the rotation matrix R and the translation vector t recorded in the camera matrix of each pose according to the residual to minimize the deviation of the pose center constraint. The camera pose matrix after supervision corresponding to each pose when the cost function is minimized can be solved, and the camera pose matrix after supervision of each pose can be used for three-dimensional scene reconstruction to further improve the accuracy of three-dimensional scene reconstruction.

[0197] The deviation of the posture is recorded in the sub-item residual, and the residual is optimized with the help of the optimizer. The supervision of the posture information is achieved by minimizing the deviation of the posture center constraint.

[0198] In some embodiments of the present application, in the process of solving the nonlinear pose optimization task, in each intermediate round of iteration, the pose residual of each sub-item is the deviation between the optimized camera pose matrix generated in the previous round and each navigation pose matrix. The pose weight of each sub-item is determined according to the optimized camera pose matrix generated in the previous round and each navigation pose matrix. In the last round of iteration, the generated optimized camera pose matrix is ​​used as the camera pose matrix after each pose supervision.

[0199] When solving the above nonlinear optimization tasks, you can also use particle swarm optimization, genetic algorithms, simulated annealing, neural networks and other solution strategies that are conducive to rapid convergence or searching for the global optimal solution.

[0200] After executing step S40, the camera pose matrix after each pose supervision can be used to reconstruct the three-dimensional scene for each valid visual point cloud frame to obtain a three-dimensional scene reconstructed image, such as Fig. 4I As shown. Figure 4H Compared with the situation in the related art, Fig. 4I The reconstruction quality of the displayed scenes is significantly improved.

[0201] In some embodiments of the present application, considering that the motion structure recovery is affected by factors such as image quality and calibration results, it is easy to have the problem of inaccurate depth recovery. Therefore, in step S10, when performing motion structure recovery, direct depth measurement of the laser radar can be introduced to perform depth supervision on the motion structure recovery to improve the accuracy of motion structure recovery.

[0202] In an embodiment of the present application, the laser radar collects a laser radar point cloud frame sequence at a specified first frame rate, and the RGB camera collects an image frame sequence at a specified second frame rate. Specifically, the first frame rate may be greater than, equal to, or less than the second frame rate. The frame rate refers to the rate at which image frames or point cloud frames are collected, or the number of image frames or point cloud frames collected per unit time.

[0203] In order to use the LiDAR point cloud frames to perform deep information supervision on the motion structure recovery of image frames, the LiDAR point cloud frame sequence and the image frame sequence are first screened based on the timestamp, that is, the LiDAR point cloud frames and image frames that are consistent in time are screened out. The consistency here can include being completely identical in time or being the closest in time.

[0204] Specifically, the smaller value of the first frame rate and the second frame rate is used as the reference frame rate, and the laser radar point cloud frames and image frames that are consistent in time are screened. For example, when the first frame rate of the laser radar point cloud frame is less than the second frame rate of the image frame, for each laser radar point cloud frame in the laser radar point cloud frame sequence, its first timestamp is determined, and the image frame corresponding to the second timestamp that is closest to the first timestamp in time (may be equal, ahead or behind) is determined, until the image frames that are closest to each laser radar point cloud frame are determined.

[0205] Similarly, when the first frame rate of the lidar point cloud frame is greater than the second frame rate of the image frame, for each image frame in the image frame sequence, its first timestamp is determined, and the lidar point cloud frame corresponding to the second timestamp that is closest in time to the first timestamp (which may be equal, ahead of, or behind) is determined, until the lidar point cloud frames that are closest to each image frame are determined.

[0206] Similarly, when the first frame rate of the laser radar point cloud frame is equal to the second frame rate of the image frame, it can be obtained by referring to the above method, and no further description is given. So far, each laser radar point cloud frame and each image frame that is consistent in time are screened out.

[0207] For the convenience of the following description, a key frame sequence may also be constructed. The key frame sequence includes multiple key frames, each of which includes a lidar point cloud frame and an image frame that is most adjacent to it in time.

[0208] In some embodiments of the present application, each laser radar point cloud frame is used to perform depth information supervision on the motion structure recovery of each image frame, and point cloud frames after depth information supervision corresponding to each image frame are generated. This can be specifically achieved through the operations of the following steps S11, S12 and S13.

[0209] S11: Process the position information of the environmental points in each lidar point cloud frame into a position information reference value for image frame depth information supervision.

[0210] S12: Perform motion structure recovery on each image frame to generate each original point cloud frame corresponding to each image frame, such as Figure 4C shown.

[0211] S13: Use the position information reference value for depth information supervision of each image frame to process each original point cloud frame, and generate a point cloud frame after depth information supervision corresponding to each image frame.

[0212] The aforementioned steps S11 and S12 can be executed sequentially or in reverse order, and both can achieve the purpose of processing the position information of the environmental points in each lidar point cloud frame into a position information reference value for image frame depth information supervision and generating each original point cloud frame.

[0213] Specifically, in the aforementioned step S12, each image frame is subjected to motion structure recovery to generate each original point cloud frame corresponding to each image frame, including using the first two adjacent image frames in the image frame sequence as the initial image pair, performing motion structure recovery on the image frame sequence, and generating each original point cloud frame corresponding to each image frame. Each original point cloud frame includes the three-dimensional coordinates (x0, y0, z0) of multiple environmental points in the camera coordinate system.

[0214] In this embodiment, the difference between the above-mentioned position information reference value and the self-lidar position information can be used as the depth information for the environmental point extracted from the self-lidar point cloud frame. Among them, the self-lidar position information can be the position information of the environmental point in the laser radar point cloud frame. The depth information supervision based on the laser radar is realized, which is conducive to solving the problem of inaccurate depth information recovery of each environmental point in the process of motion structure recovery of the image frame, and can improve the accuracy of the depth information in each original point cloud frame corresponding to each image frame.

[0215] In some embodiments of the present application, in the aforementioned step S11, the position information of the environmental points in each lidar point cloud frame is processed into a position information reference value for supervising the depth information of the image frame, including: using the transformation matrix from the lidar coordinate system to the camera coordinate system, transforming the position information of the environmental points in each lidar point cloud frame to the camera coordinate system, and using it as the position information reference value for supervising the depth information of the image frame.

[0216] Correspondingly, in the aforementioned step S13, the position information reference value for depth information supervision of each image frame is used to process each original point cloud frame corresponding to each image frame to generate each point cloud frame after depth information supervision corresponding to each image frame, including: taking a weighted average of the position information reference value for depth information supervision of the image frame and the position information of the environmental points recorded in the original point cloud frame to generate the position information of the environmental points after depth information supervision, and record it in each point cloud frame after depth information supervision corresponding to each image frame.

[0217] Specifically, the laser radar point cloud frame includes multiple environmental points. The three-dimensional coordinates of any environmental point in the laser radar coordinate system are (x, y, z). By using the transformation matrix from the laser radar coordinate system to the camera coordinate system, the three-dimensional coordinates of the environmental point in the laser radar coordinate system can be converted to new three-dimensional coordinates (x', y', z') in the camera coordinate system through coordinate system conversion. The new three-dimensional coordinates (x', y', z') here are used to introduce depth information supervision based on laser radar. In this way, the position information reference value used for depth information supervision of the processed image frame is the coordinate value in the camera coordinate system.

[0218] Specifically, a weighted average is performed on the position information reference value used for depth information supervision of the image frame and the environmental point position information recorded in the original point cloud frame, including: constructing a difference function to determine the weights corresponding to the position information reference value used for depth information supervision of each image frame and the environmental point position information recorded in each original point cloud frame.

[0219] The above difference function is used to measure the environmental point X in the lidar point cloud frame l The new pseudo environment point X after conversion to the camera coordinate system lc (which corresponds to the aforementioned new three-dimensional coordinates (x', y', z'), that is, the position information reference value used for image frame depth information supervision), and the matching true environment point X recorded in the original point cloud frame corresponding to the image frame consistent in time c (which corresponds to the aforementioned three-dimensional coordinates (x0, y0, z0), that is, the relative distance between the location information of the environmental point in the original point cloud frame).

[0220] The difference function can be written as f = Diff(X lc ,X c ), the distance threshold is denoted as θ d , X lc ={x',y',z'},X c ={x0,y0,z0}, such as, Among them, the distance threshold θ d The value of can be determined according to the installation position of the laser radar and the installation position of the camera, which will not be repeated here.

[0221] Note that for the aforementioned pseudo environment point X lc The weight value is recorded as α, for the true environment point X c The weight value is recorded as β, and α+β=1. Because the depth information obtained by the laser radar is very accurate, it can be used as the true value. Therefore, the following weight distribution method is determined.

[0222] If the aforementioned relative distance is less than the distance threshold, the weight value for the aforementioned pseudo-environment point can be 0.5, and the weight value for the true environment point can be 0.5, that is, the arithmetic mean of the position information reference value used for image frame depth information supervision and the environment point position information recorded in the original point cloud frame is used as the environment point position information after depth information supervision. If the aforementioned relative distance is greater than at least an integer multiple of the distance threshold, the weight value for the aforementioned pseudo-environment point can be 1.0, and the weight value for the true environment point can be 0, that is, the position information reference value used for image frame depth information supervision is used as the environment point position information after depth information supervision.

[0223] If the aforementioned relative distance is less than at least one integer multiple of the distance threshold and greater than the distance threshold, the weight value for the aforementioned pseudo environment point gradually increases from 0.5 to 1.0, and the weight value for the true environment point gradually decreases from 0.5 to 0.

[0224] In this way, in the camera coordinate system, the closer the pseudo environment point is to the matching true environment point recorded in the original point cloud frame corresponding to the image frame, the smaller the weight value for the pseudo environment point; the farther away, the larger the weight value for the pseudo environment point, so as to gradually increase the role of depth information supervision for the lidar.

[0225] The introduction of LiDAR depth information supervision solves the problem of blurred images acquired by the camera in high-speed, large-scale motion scenes. By introducing direct supervision based on LiDAR frame by frame, the depth information of each environmental point in the supervised point cloud frame corresponding to each frame image is adjusted respectively.

[0226] The 3D scene is reconstructed using the visual point cloud frames supervised by the depth information. After the environmental points are transformed from the camera coordinate system to the world coordinate system, the obtained landmark point cloud frames can be Figure 4D shown.

[0227] Based on the visual point cloud frames supervised by each depth information, performing the operation of screening valid visual point cloud frames in steps S20-S30 can further improve the accuracy of the final valid visual point cloud frames, thereby improving the accuracy of three-dimensional scene reconstruction.

[0228] In an embodiment of the present application, joint observation is performed based on a laser radar and a camera to reduce invalid feature points in the visual data obtained by motion structure recovery, reduce the amount of calculation and computing resource consumption for subsequent three-dimensional scene reconstruction, and improve the efficiency and accuracy of three-dimensional scene reconstruction. It is also possible to perform secondary screening of valid visual point cloud frames based on the camera pose matrix corresponding to the screened valid visual point cloud frames to further improve the accuracy of effective feature screening. It is also possible to optimize the camera pose matrix corresponding to the valid visual point cloud frame, and perform pose supervision on the camera pose matrix corresponding to the valid visual point cloud frame based on the navigation pose to improve the accuracy of the camera pose matrix, and use the highly accurate camera pose to charge the screened valid visual point cloud frame for three-dimensional scene, so that the reconstruction accuracy is very high. Furthermore, a laser radar is introduced to perform depth information supervision on motion structure recovery, alleviate the problem of inaccurate depth recovery in motion structure recovery, and improve the accuracy of triangulation solution for motion structure recovery.

[0229] like Figure 2B As shown, an embodiment of the present application also provides a method for reconstructing a three-dimensional scene, which includes the method for screening point clouds in the above-mentioned embodiment. The method for reconstructing a three-dimensional scene is implemented by the operations of the following steps S110, S120, S130, S140, and S150.

[0230] S110: timestamp screening and constructing a key frame sequence.

[0231] Specifically, the timestamps of the lidar frame sequence and the image frame sequence are filtered to construct a key frame sequence.

[0232] like

[0233] The lidar frame sequence (such as the frame rate of 10Hz) and the image frame sequence (such as the frame rate of 30Hz) are screened based on the timestamp, and the image frame with the nearest timestamp is selected for each lidar point cloud frame to construct a key frame sequence. Each key frame K i Contains a LiDAR point cloud frame and an image frame The key frame, the lidar point cloud frame and the image frame correspond to key frame time i respectively, and the three are consistent in time.

[0234] S120: Depth recovery based on LiDAR detection depth supervision SFM.

[0235] That is, the SFM process of the key frame image sequence uses lidar to supervise the depth information and improve the solution accuracy of motion structure recovery.

[0236] Specifically, for each key frame time i, the state of the environment point in the laser radar coordinate system (including depth information and position information) is measured, and the pose transformation T from the laser radar coordinate system to the camera coordinate system is performed.lc The position information of the environmental point in the laser radar coordinate system is transferred to the camera coordinate system as the initial reference value X of the state quantity of the visual SFM l The state quantity X is obtained by performing visual triangulation on the image frame sequence c .

[0237] Define the difference function f = Diff(X l , X c ), define the threshold θ d When the result of the difference function is less than or equal to the threshold θ d , then the final state quantity is x c and X1 take the mean; if the result of the difference function is greater than the threshold θ d , then the final state quantity only retains X l In this way, the position information of the environment point in the image frame is adjusted.

[0238] S130: Construct joint observations and screen consistent and effective features.

[0239] The landmark point cloud and lidar point cloud obtained by SFM of the key frame image are jointly observed to screen effective features of motion consistency.

[0240] Specifically, for all key frames in the key frame sequence The current image frame The sparse point cloud of the landmark obtained after triangulation of the previous image frame is The overlap between the LiDAR point clouds is calculated and inconsistent point clouds are excluded.

[0241] Construct source grid GR according to the corresponding point cloud s and the target grid GR t Each cell in the grid takes the value 1 for occupied, 0 for idle, and -1 for unknown. Then, for GR s Each occupied cell in is transformed into GR using the estimated relative motion (R, t) t coordinate system (introducing the transformation of the known quantity LiDAR coordinate system to the camera coordinate system) and checking the cell state at that position. In order to quantify the consistency, the consistency ratio r is defined c as follows:

[0242]

[0243] Where I() is an indicator function, which is 1 if the argument is true and 0 otherwise. c Measuring GR t With GR s degree of consistency.

[0244] We also perform cross-validation by exchanging the target and source rasters and recalculating the consistency ratio. s If the relative motion is invalid, the current image frame and lidar point cloud frame are deleted from the keyframe.

[0245] Specifically, it also includes: defining two sets of key frame pairs with a key frame interval of a preset number of frames as a small cycle, and verifying the observation consistency by checking the small cycle, wherein the local information represented by the small cycle will not be affected by the cumulative drift. The preset number of frames can be 4, 5 or 6, etc.

[0246] Combined relative motion T ij T jk T kl T li ∈SE(3) and check the resulting angle and translation. c and p c As the thresholds of angle and translation, the inspection results of small loops are marked as passed or failed.

[0247] Specifically, when determining the thresholds of angle and translation, for each relative motion, track its success rate T s , defined as Where N p is the number of checks passed, N i is the number of inspections involved, where T s Below the threshold value θ t Relative motion is considered invalid. If there is still invalid motion, the threshold θ is increased. t .

[0248] By incorporating multiple global small loops into nonlinear optimization and constructing nonlinear optimization equations, we can obtain:

[0249]

[0250] Where Γ is the cost function. Specifically, the overall feature screening is optimized by minimizing the cost function.

[0251] S140: GNSS absolute pose supervision.

[0252] S150: Backend optimization based on supervision.

[0253] Specifically, the pose information of GNSS is used to supervise the pose information obtained by SFM to solve the problem of inaccurate scale of 3D reconstruction in large-scale autonomous driving scenes. Residuals are obtained using techniques such as similarity transformation and robust fitting, and the residuals are added to the optimization problem. The optimizer can adjust the rotation and translation parameters of the camera based on these residuals to minimize the deviation of the pose center constraint. The specific methods include:

[0254] The RANSAC algorithm is used to estimate the similarity transformation matrix, and the best similarity transformation matrix is ​​found by calculating the matching point pairs of the SFM and GNSS results. The estimated similarity transformation matrix is ​​applied to transform the SFM position to align it with the GNSS position. Based on the aligned SFM position and GNSS position, the residual, that is, the difference between the two, is calculated; the residuals are sorted and the median value is obtained as the weight of the robust fitting error.

[0255] The rotation and translation vectors of the camera at each moment are used to calculate the difference between them and the GNSS position as the deviation of the attitude center position; the calculated weight is passed to the backend optimization, the weight is multiplied by the deviation of the attitude center position, and the result is stored in the residuals. The residuals are optimized with the help of the optimizer, and the supervision of the posture information is achieved by minimizing the deviation of the attitude center constraint.

[0256] In this way, the SFM visual triangulation solution is affected by factors such as image quality and calibration results, and is prone to inaccurate depth recovery. The introduction of direct depth measurement supervision by LiDAR is conducive to improving the accuracy of SFM visual triangulation solution. As there are many invalid features in SFM vision, it is difficult to screen invalid feature points only by visual means, and the participation of invalid feature points in matching and reconstruction increases time and resource consumption and is not beneficial to accuracy. The introduction of LiDAR for joint observation can realize the screening of invalid features. In large-scale and large-scene visual reconstruction, the lack of absolute scale information constraints is prone to convergence failure. The introduction of GNSS constraints is conducive to reconstruction convergence.

[0257] The present application also provides a point cloud screening device 500, which is used to perform the point cloud screening method provided in any of the above embodiments. Figure 5 As shown, the device comprises:

[0258] The motion structure recovery module 501 is used to perform motion structure recovery and calculation based on the image frames collected by the visual sensor to obtain multiple visual point cloud frames;

[0259] The effective point cloud screening module 502 is used to screen out effective visual point cloud frames from the multiple visual point cloud frames based on the multiple visual point cloud frames and the non-visual point cloud frames collected by the first non-visual sensor.

[0260] The effective point cloud screening module 502 is used to construct multiple joint observation data pairs based on multiple visual point cloud frames and non-visual point cloud frames collected by the first non-visual sensor; the joint observation data pair includes a non-visual point cloud frame and a visual point cloud frame with consistent timestamps; for each joint observation data pair, the overlap between the non-visual point cloud frame and the visual point cloud frame in the joint observation data pair is detected; and the visual point cloud frame whose overlap meets the requirements is determined as a valid visual point cloud frame.

[0261] The effective point cloud screening module 502 is used to construct a source grid according to the first point cloud frame in the first joint observation data pair, and determine the source cell state corresponding to each source grid point in the source grid; the first joint observation data pair is any data pair in multiple joint observation data pairs, and the first point cloud frame is one of the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair; the target grid is constructed according to the second point cloud frame in the first joint observation data pair, and the target grid includes each target grid point and the corresponding target cells; the second point cloud frame is another point cloud frame in the first joint observation data pair except the first point cloud frame; based on the source cell state corresponding to each source grid point and the target cell state corresponding to each target grid point, the overlap of the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair is detected through a preset observation matrix; the preset observation matrix is ​​a transformation matrix between the coordinate system of the first non-visual sensor and the coordinate system of the visual sensor.

[0262] The effective point cloud screening module 502 is used to perform coordinate transformation on each source grid point corresponding to each source cell in the source grid whose status is occupied through a preset observation matrix, so as to obtain each pseudo-target grid point corresponding to each source grid point whose status is occupied; obtain the status of each target cell corresponding to each pseudo-target grid point in the target grid; count the number of target cells corresponding to each pseudo-target grid point whose status is occupied, and use the number to indicate the overlap between the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair.

[0263] The valid point cloud screening module 502 is used to sort the camera pose matrices corresponding to each valid visual point cloud frame in the order of timestamps; select a preset number of continuous camera pose matrices from the sorted camera pose matrix sequence to form a relative motion estimation sequence, and determine the angle translation combination corresponding to each relative motion estimation sequence; and determine the valid visual point cloud frame corresponding to the camera pose matrix specified in the relative motion estimation sequence whose angle translation combination meets the requirements as the valid visual point cloud frame for secondary observation.

[0264] The device also includes: a camera pose optimization module, which is used to sort the camera pose matrices corresponding to each valid visual point cloud frame in the order of timestamps; from the sorted camera pose matrix sequence, a preset number of continuous camera pose matrices are selected to form a relative motion estimation sequence, and the angle translation combination corresponding to each relative motion estimation sequence is determined respectively; using each angle translation combination as a cost function, constructing an optimization task for each valid visual point cloud frame, solving the optimization task with the minimum cost function as a constraint, and generating an optimized camera pose matrix corresponding to each valid visual point cloud frame.

[0265] A preset number of camera pose matrices in the relative motion estimation sequence include a first matrix, an intermediate matrix and a last matrix; the effective point cloud screening module 502 or the camera pose optimization module is used to sequentially determine the relative motion estimation matrices pointing from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the adjacent next intermediate matrix, and from the last intermediate matrix to the tail matrix; and the relative motion estimation matrices are multiplied as an angular translation combination corresponding to the relative motion estimation sequence.

[0266] The device also includes: a three-dimensional reconstruction module, which is used to perform visual three-dimensional reconstruction on each valid visual point cloud frame by using the optimized camera posture matrix corresponding to each valid visual point cloud frame to obtain a visual three-dimensional reconstructed image.

[0267] The point cloud screening device provided in the embodiment of the present application and the method for screening point clouds provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0268] An embodiment of the present application also provides a movable vehicle, including: a first non-visual sensor, a visual sensor and a controller; the controller is used to execute the method for screening point clouds provided in any of the above embodiments based on image frames captured by the visual sensor and non-visual point cloud frames captured by the first non-visual sensor.

[0269] The movable carrier provided in the embodiment of the present application and the method for screening point clouds provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0270] The present application also provides an electronic device to perform the above method for screening point clouds. Figure 6 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 6As shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802 and a communication interface 803, wherein the processor 800, the communication interface 803 and the memory 801 are connected via the bus 802; the memory 801 stores a computer program that can be run on the processor 800, and when the processor 800 runs the computer program, the method for screening point clouds provided in any of the aforementioned embodiments of the present application is executed.

[0271] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 803 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0272] The bus 802 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs, and the processor 800 executes the program after receiving an execution instruction. The method for screening point clouds disclosed in any implementation of the embodiment of the present application may be applied to the processor 800, or implemented by the processor 800.

[0273] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 800. The above processor 800 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0274] The electronic device provided in the embodiment of the present application and the method for screening point clouds provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0275] The present application also provides a computer-readable storage medium corresponding to the method for screening point clouds provided in the above-mentioned embodiment. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method for screening point clouds provided in any of the aforementioned embodiments is executed.

[0276] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0277] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method for screening point clouds provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0278] It should be noted that:

[0279] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0280] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0281] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0282] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for screening point clouds, characterized in that: include: Based on the image frames collected by the visual sensor, motion structure recovery is performed to obtain multiple visual point cloud frames; Based on the multiple visual point cloud frames and the non-visual point cloud frames acquired by the first non-visual sensor, valid visual point cloud frames are screened out from the multiple visual point cloud frames.

2. The method according to claim 1, characterized in that The method of selecting a valid visual point cloud frame from the multiple visual point cloud frames based on the multiple visual point cloud frames and the non-visual point cloud frames collected by the first non-visual sensor includes: Based on the multiple visual point cloud frames and the non-visual point cloud frames collected by the first non-visual sensor, a plurality of joint observation data pairs are constructed; the joint observation data pairs include a non-visual point cloud frame and a visual point cloud frame with consistent timestamps; For each of the joint observation data pairs, detecting the overlap between the non-visual point cloud frame and the visual point cloud frame in the joint observation data pair; The visual point cloud frames whose overlapping conditions meet the requirements are determined as valid visual point cloud frames.

3. The method according to claim 2, characterized in that The detecting the overlap between the non-visual point cloud frame and the visual point cloud frame in the joint observation data pair includes: Construct a source grid according to a first point cloud frame in a first joint observation data pair, and determine a source cell state corresponding to each source grid point in the source grid; the first joint observation data pair is any data pair in the plurality of joint observation data pairs, and the first point cloud frame is one of a non-visual point cloud frame and a visual point cloud frame in the first joint observation data pair; Constructing a target grid according to a second point cloud frame in the first joint observation data pair, wherein the target grid includes each target grid point and each corresponding target cell; the second point cloud frame is another point cloud frame in the first joint observation data pair except the first point cloud frame; Based on the source cell states corresponding to the source grid points and the target cell states corresponding to the target grid points, the overlap between the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair is detected through a preset observation matrix; the preset observation matrix is ​​a transformation matrix between the coordinate system of the first non-visual sensor and the coordinate system of the visual sensor.

4. The method according to claim 3, characterized in that The detecting, based on the source cell states corresponding to the source grid points and the target cell states corresponding to the target grid points, the overlapping of the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair by using a preset observation matrix comprises: Through the preset observation matrix, coordinate transformation is performed on each source grid point corresponding to each source cell in the source grid whose state is occupied, so as to obtain each pseudo target grid point corresponding to each source grid point whose state is occupied; Respectively obtain the states of each target cell corresponding to each pseudo target grid point in the target grid; The number of occupied target cells corresponding to the pseudo target grid points is counted, and the number is used to indicate an overlap between the non-visual point cloud frame and the visual point cloud frame in the first joint observation data pair.

5. The method according to claim 1, characterized in that The method further comprises: Sort the camera pose matrices corresponding to each valid visual point cloud frame in the order of timestamps; From the sequence of camera pose matrices obtained by sorting, a preset number of continuous camera pose matrices are selected to form a relative motion estimation sequence, and the angle translation combinations corresponding to each relative motion estimation sequence are determined respectively; The valid visual point cloud frame corresponding to the camera pose matrix specified in the relative motion estimation sequence that satisfies the angle translation combination is determined as the valid visual point cloud frame for secondary observation.

6. The method according to claim 1, characterized in that The method further comprises: Sort the camera pose matrices corresponding to each valid visual point cloud frame in the order of timestamps; From the sequence of camera pose matrices obtained by sorting, a preset number of continuous camera pose matrices are selected to form a relative motion estimation sequence, and the angle translation combinations corresponding to each relative motion estimation sequence are determined respectively; Taking the combination of each angle translation as the cost function, an optimization task for each valid visual point cloud frame is constructed, and the optimization task is solved with the minimum cost function as the constraint to generate an optimized camera pose matrix corresponding to each valid visual point cloud frame.

7. The method according to claim 5 or 6, characterized in that: The preset number of camera pose matrices in the relative motion estimation sequence include a first matrix, a middle matrix and a last matrix; The step of respectively determining the angle translation combinations corresponding to the relative motion estimation sequences comprises: Determine in sequence the relative motion estimation matrices from the first matrix to the front intermediate matrix, from the previous intermediate matrix to the adjacent next intermediate matrix, and from the last intermediate matrix to the tail matrix; The relative motion estimation matrices are multiplied to obtain an angle translation combination corresponding to the relative motion estimation sequence.

8. The method according to claim 6, characterized in that The method further comprises: Using the optimized camera pose matrix corresponding to each valid visual point cloud frame, visual 3D reconstruction is performed on each valid visual point cloud frame to obtain a visual 3D reconstructed image.

9. A movable carrier, characterized in that: include: a first non-visual sensor, a visual sensor, and a controller; The controller is used to execute the method for screening point clouds according to any one of claims 1 to 8 based on the image frames collected by the visual sensor and the non-visual point cloud frames collected by the first non-visual sensor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images