Method and device for recovering depth of visual feature points, electronic equipment and storage medium

Abstract

The present disclosure relates to a method and device for depth recovery of visual feature points, electronic equipment and storage medium. The method comprises: acquiring a visual image frame and a laser point cloud frame, the visual image frame comprising a plurality of visual feature points, and the laser point cloud frame comprising a plurality of laser points; establishing a corresponding relationship between the visual feature points in the visual image frame and the laser points in the laser point cloud frame, the corresponding relationship comprising N laser points corresponding to each visual feature point, the N laser points corresponding to the visual feature points forming a target point cloud of the visual feature points, N being a positive integer greater than or equal to 5; screening the target point cloud of the visual feature points to obtain a smooth target point cloud meeting a smooth condition; and performing depth calculation based on the smooth target point cloud meeting the smooth condition to obtain a laser depth of the visual feature points. The method can filter out the target point cloud comprising sharp laser points with large curvature, and improve the accuracy of the laser depth of the visual feature points obtained finally.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a method, apparatus, electronic device, and storage medium for depth recovery of visual feature points. Background Technology

[0002] Localization technology is a core technology in autonomous driving, and Simultaneous Localization and Mapping (SLAM) is one of the main methods for achieving localization in autonomous driving. During the localization and mapping process of visual SLAM, it is necessary to recover the coordinate information of each real road sign in the surrounding environment, which requires accurate depth recovery of visual feature points in the visual image.

[0003] Existing methods for recovering depth from visual feature points typically include triangulation using a monocular camera, stereo matching algorithms using a binocular camera, or infrared sensing methods using an RGBD camera. Among these, triangulation using a monocular camera is widely used. This method involves acquiring visual image frames and laser point cloud frames, establishing the association between visual feature points in the visual image frames and laser points in the laser point cloud frames, identifying multiple laser points associated with each visual feature point, and then performing plane fitting on these multiple laser points. Depth calculation can be performed based on the fitted plane obtained after plane fitting. However, in the process of implementing this invention, the inventors discovered that some of the multiple laser points associated with each visual feature point are sharp laser points with large curvatures, which affects the accuracy of the depth calculation results. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method, apparatus, electronic device, and storage medium for depth recovery of visual feature points.

[0005] In a first aspect, this disclosure provides a depth recovery method for visual feature points, the method comprising:

[0006] Acquire visual image frames and laser point cloud frames, wherein the visual image frames include multiple visual feature points and the laser point cloud frames include multiple laser points;

[0007] Establish a correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5.

[0008] The target point cloud of the visual feature points is filtered to obtain a smooth target point cloud that meets the smoothing conditions.

[0009] Depth calculation is performed based on the smooth target point cloud that meets the smoothing conditions to obtain the laser depth of the visual feature points.

[0010] Secondly, this disclosure also provides a monocular vision initialization method, including:

[0011] Identify the reference visual image frame among multiple visual image frames used for monocular vision initialization;

[0012] Based on the depth recovery method of visual feature points as described above, the laser depth of the first visual feature point that can be depth recovered in the plurality of visual image frames is determined.

[0013] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, the pose estimation result of the other visual image frames is determined. Based on the pose estimation result of the other visual image frames, the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame is determined by triangulation. The second visual feature point is a visual feature point that cannot be depth recovered.

[0014] Thirdly, this disclosure also provides a depth recovery device for visual feature points, comprising:

[0015] The first acquisition module is used to acquire visual image frames and laser point cloud frames. Therefore, the visual image frame includes multiple visual feature points, and the laser point cloud frame includes multiple laser points.

[0016] The correspondence establishment module is used to establish the correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5.

[0017] The filtering module is used to filter the target point cloud of the visual feature points to obtain a smooth target point cloud that meets the smoothing conditions.

[0018] The depth calculation module is used to perform depth calculation based on the smooth target point cloud that meets the smoothing conditions, so as to obtain the laser depth of the visual feature points.

[0019] Fourthly, this disclosure also provides a monocular vision initialization device, comprising:

[0020] The reference frame determination module is used to determine the reference visual image frame among multiple visual image frames used for monocular vision initialization.

[0021] A laser depth acquisition module is used in the device described above for depth recovery of visual feature points to determine the laser depth of a first visual feature point in the plurality of visual image frames that can be depth recovered.

[0022] The triangulation depth acquisition module is used to determine the pose estimation result of the other visual image frames based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, and to determine the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame using the triangulation method based on the pose estimation result of the other visual image frames. The second visual feature point is a visual feature point that cannot be depth recovered.

[0023] Fifthly, this disclosure also provides an electronic device, the electronic device comprising:

[0024] One or more processors;

[0025] Storage device for storing one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the depth recovery method or monocular vision initialization method for visual feature points as described above.

[0027] In a sixth aspect, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the depth recovery method or monocular vision initialization method for visual feature points as described above.

[0028] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0029] The technical solution provided in this embodiment filters the target point cloud of visual feature points to obtain a smooth target point cloud that meets the smoothing conditions. This can filter out target point clouds that include sharp laser points with large curvature. In subsequent calculations, target point clouds that include sharp laser points with large curvature will not be used, and will not be affected by sharp laser points. Therefore, the accuracy of the laser depth of the final visual feature points can be improved. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart of a depth recovery method for visual feature points provided in this embodiment of the disclosure;

[0033] Figure 2 A schematic diagram of the local curvature of a point cloud provided in an embodiment of this disclosure;

[0034] Figure 3 for Figure 1 A flowchart of one implementation method of S120;

[0035] Figure 4 A schematic diagram of a normalized plane provided in an embodiment of this disclosure;

[0036] Figure 5 A schematic diagram illustrating the acquisition of laser point cloud frames by an acquisition device according to an embodiment of this disclosure;

[0037] Figure 6 A flowchart of a method for implementing S140 provided in an embodiment of this disclosure;

[0038] Figure 7 A flowchart of a method for implementing S210 provided in an embodiment of this disclosure;

[0039] Figure 8 and Figure 9 Schematic diagrams of two local surfaces fitted from smooth target point clouds provided in embodiments of this disclosure;

[0040] Figure 10 A flowchart of another method for implementing S140 provided in this disclosure embodiment;

[0041] Figure 11 A schematic diagram illustrating the principle of determining the laser depth of a visual feature point based on the normal vector of the laser point plane, as provided in this embodiment of the disclosure;

[0042] Figure 12 A flowchart of a monocular vision initialization method provided in this embodiment of the disclosure;

[0043] Figure 13 A schematic diagram of a sliding window provided in an embodiment of this disclosure;

[0044] Figure 14 This is a schematic diagram of the structure of a depth recovery device for visual feature points according to an embodiment of the present disclosure;

[0045] Figure 15 This is a schematic diagram of the structure of a monocular vision initialization device according to an embodiment of the present disclosure;

[0046] Figure 16 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0047] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0048] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0049] Figure 1 This flowchart illustrates a depth recovery method for visual feature points provided in an embodiment of the present disclosure. The method can be executed by a depth recovery device for visual feature points, which can be implemented in software and / or hardware. This device can be configured in an electronic device, including but not limited to smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, smart home devices, and vehicles. Figure 1 As shown, the method may specifically include:

[0050] S110. Acquire visual image frames and laser point cloud frames. The visual image frames include multiple visual feature points, and the laser point cloud frames include multiple laser points.

[0051] The device that acquires visual image frames and laser point cloud frames is called the acquisition device. The acquisition device is equipped with a camera and a lidar sensor. The relative positions of the camera and lidar sensor within the acquisition device can be fixed. The process by which the acquisition device acquires visual image frames and laser point cloud frames is called the acquisition process. During the acquisition process, objects in the environment surrounding the acquisition device are called target objects.

[0052] Visual feature points are points in a visual image frame that correspond to the points that make up the target object. They are the image representation of the points that make up the target object and carry coordinate information in the camera coordinate system. In autonomous driving scenarios, target objects can be guardrails, lane lines, and buildings around the road, etc.

[0053] A laser point is an echo signal formed when the laser emitted by a lidar is reflected after encountering a target object. It carries coordinate information in the lidar coordinate system. The laser point corresponds to the incident point on the target object where the echo signal is formed (i.e., the laser point itself).

[0054] The visual image frames and laser point cloud frames obtained in this step have a corresponding relationship.

[0055] In one embodiment, one visual image frame corresponds to multiple laser point cloud frames. Specifically, the absolute value of the difference between the acquisition time of the corresponding visual image frame and the acquisition time of each of the multiple laser point cloud frames is less than or equal to a set time threshold. That is, within the allowable error range, it can be approximately considered that the corresponding visual image frame and the multiple laser point cloud frames were acquired in the same extremely short time period. Within this extremely short time period, although the acquisition device is displaced, the displacement is extremely short and can be ignored within the allowable error range.

[0056] In one embodiment, after performing this step, optionally, each frame of the laser point cloud is preprocessed to filter out invalid points at infinite depth, sparse outliers far from the main body of the point cloud, and laser measurement noise points; and each frame of the visual image is subjected to distortion correction to compensate for the positional shift of image pixels caused by lens imaging. This setup helps to ensure that the final laser depth is accurate.

[0057] S120. Establish the correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5.

[0058] Since visual feature points correspond to specific points that constitute the target object, laser points also correspond to specific points that constitute the target object. The purpose of this step is to establish a correspondence between visual feature points and laser points that correspond to the same points constituting the target object.

[0059] For example, the visual image frame includes visual feature point A and visual feature B, where visual feature point A corresponds to the start point of the lane line and visual feature B corresponds to the end point of the lane line. The laser point cloud frame includes laser points 1-20, where laser points 1-5 correspond to the start point of the lane line and laser points 10-20 correspond to the end point of the lane line. During this step, a correspondence is established between visual feature point A and laser points 1-5, and a correspondence is established between visual feature point B and laser points 10-20. Then, laser points 1-5 constitute the target point cloud of visual feature point A, and laser points 10-20 constitute the target point cloud of visual feature point B.

[0060] S130. Filter the target point cloud of visual feature points to obtain a smooth target point cloud that meets the smoothing conditions.

[0061] Smooth target point cloud refers to a point cloud that does not include sharp laser points with large curvature.

[0062] In practice, there are various specific conditions that can be used as smoothing conditions, and this application does not limit them. In one embodiment, the implementation method of this step can be set to include: calculating the local curvature of the point cloud corresponding to each laser point in the target point cloud; if the local point cloud curvature corresponding to at least 5 laser points is less than a first set threshold, then the target point cloud meets the smoothing condition.

[0063] For example, laser points 101-200 constitute the target point cloud of visual feature point C. The local curvature of the point cloud corresponding to each laser point in laser points 101-200 is calculated. If the local curvature of the point cloud corresponding to 5 laser points in laser points 101-200 is less than a first preset threshold, then the target point cloud constituted by laser points 101-200 is considered to meet the smoothing condition. Subsequently, the laser depth of visual feature point A will be calculated based on the coordinate information carried by at least some of the laser points in laser points 101-200.

[0064] Further, the local curvature of the point cloud corresponding to each laser point in the target point cloud is calculated, including:

[0065] .

[0066] in, c This refers to the local curvature of the point cloud at the current laser point. The set of laser points corresponding to the current laser point. This indicates the number of laser points in the laser point set. Indicates the current depth of the laser point. This indicates the depth of other laser points in the laser point set. If the local curvature of the point cloud corresponding to a certain laser point is currently being calculated, that laser point is the current laser point. The depth of the current laser point. The depth of the current laser point can be calculated based on its coordinates in the lidar coordinate system. The depth of other laser points in the laser point set can be calculated based on their coordinates in the lidar coordinate system.

[0067] Figure 2 This is a schematic diagram of the local curvature of a point cloud provided as an embodiment of this disclosure. See also... Figure 2 For any laser point, the corresponding local curvature of the point cloud c The larger the value, the coarser the point cloud surrounding the laser point; c The smaller the value, the smoother the point cloud around the laser point.

[0068] When determining the set of laser points corresponding to the current laser point, optionally, associated laser points associated with the current laser point are determined; the current laser point and the associated laser points associated with the current laser point are used together as elements in the set of laser points corresponding to the current laser point to form the set of laser points corresponding to the current laser point.

[0069] There are various methods for determining associated laser points related to the current laser point, and this application does not limit this method. For example, a method for determining associated laser points related to the current laser point includes: selecting a predetermined number of laser points located on the same laser scanning line as the current laser point and on either side of the current laser point, and using these selected laser points as associated laser points related to the current laser point. Here, the laser scanning line is a linear scanning beam formed by the rotation of the lidar.

[0070] S140. Perform depth calculation based on the smooth target point cloud that meets the smoothing conditions to obtain the laser depth of visual feature points.

[0071] The above technical solution filters the target point cloud of visual feature points to obtain a smooth target point cloud that meets the smoothing condition. This means that the target point cloud, including sharp laser points with large curvature, is filtered out. In subsequent calculations, the target point cloud containing sharp laser points with large curvature will not be used, and it will not be affected by sharp laser points. Therefore, the accuracy of the laser depth of the final visual feature points can be improved.

[0072] In the above technical solutions, there are various methods for implementing S120, and this application does not limit them. For example, Figure 3 for Figure 1 A flowchart of one implementation method of S120 is shown, which corresponds to the case where the acquired laser point cloud frames are multiple laser point cloud frames acquired at different times. The method for implementing S120 includes:

[0073] S121. Convert multiple laser point cloud frames collected at different times in the laser coordinate system to the world coordinate system.

[0074] S122. Accumulate multiple laser point cloud frames in the world coordinate system to obtain the accumulated local point cloud map.

[0075] S123. Obtain the local point cloud map in the camera coordinate system at the acquisition time of the visual image frame based on the accumulated local point cloud map.

[0076] S124. Project the visual feature points in the visual image frame and the laser points in the local point cloud map under the camera coordinate system at the acquisition time of the visual image frame onto the camera normalization plane.

[0077] S125. Obtain multiple laser points corresponding to each visual feature point, wherein the distance between the multiple laser points and the visual feature point on the normalization plane is less than or equal to a second set threshold.

[0078] For example, the extrinsic parameters of the lidar and the camera are known to be... . Visual image frames acquired at different times and Time to Corresponding to the laser point cloud frames collected at any time.

[0079] Suppose The laser point cloud frame at time is ,in, g ≤ i ≤ h , ig=KT, K positive integer, T This refers to the time interval for the lidar to periodically acquire laser point cloud frames. By interpolating the pose of the visual odometry, we can obtain... Transformation matrix from camera coordinate system to world coordinate system Laser point cloud frame from The lidar coordinate system transformation at time t is... The process of transforming the camera coordinate system at a given time point to the world coordinate system can be represented as follows:

[0080] .

[0081] in, That is The laser point cloud frames are located in the world coordinate system at any given time. All laser point cloud frames within a given time period (i.e., all laser point cloud frames corresponding to the same visual image frame) are transformed to the world coordinate system according to the above process and then accumulated together to form a local point cloud map.

[0082] .

[0083] for Visual image frames at any given moment, through Transformation matrix from camera coordinate system to world coordinate system Local point cloud map Transform to Camera coordinates at time

[0084] ,

[0085] This allows us to obtain a local map of the laser point cloud in the camera coordinate system at any given time. This means obtaining the coordinate information of each laser point in the camera coordinate system.

[0086] Will Visual feature points in a visual image frame at a given time and a local map of the laser point cloud in the camera coordinate system. All laser points are projected onto the camera's normalized plane. Specifically, regardless of whether... The visual feature points in the visual image frame at any given time are still represented by a local map of the laser point cloud in the camera coordinate system. The laser point in the image is the projection object. Since the coordinate information of the projection object in the camera coordinate system is known, the coordinate values ​​of the projection object on each coordinate axis in the camera coordinate system are divided by the value of the last dimension of the coordinate information (i.e., the depth of the point from the camera imaging plane). The resulting coordinate information is the coordinate information of the projection object on the camera normalized plane, which is equivalent to projecting the projection object onto the camera normalized plane.

[0087] In one embodiment, for any visual feature point, a neighborhood of that visual feature point is defined on the camera normalization plane, centered on its projection and with a second predetermined threshold as its side length. Each visual feature point corresponds to a laser point projected into its neighborhood. Figure 4 This is a schematic diagram of a normalized plane provided for an embodiment of this disclosure. For example, see [link to relevant documentation]. Figure 4 The normalized plane includes the projection a of the visual feature point and the projection b of the laser point. Figure 4 In the diagram, a bold square frame e is drawn outside the projection of each visual feature point a. This bold square frame e represents the boundary line of the neighborhood range of that visual feature point. Each visual feature point a corresponds to a laser point b projected into its neighborhood range.

[0088] In the above technical solution, multiple laser point cloud frames in the world coordinate system are accumulated to obtain an accumulated local point cloud map. Subsequently, the depth of the visual feature points is recovered by data association between the local point cloud map and visual feature points. Compared with techniques that only use a single laser point cloud frame from a time adjacent to the acquisition time of the visual image frame to recover the depth of visual feature points, the local point cloud map in this application solves the problem of sparse single-frame laser point clouds, increases the point cloud density in the camera coordinate system, and expands the projection area of ​​the point cloud on the camera normalized plane, making the depth recovery results of visual feature points more accurate and reliable.

[0089] In some embodiments, S125 may be replaced by: determining laser points whose distance from the visual feature point on the normalization plane is less than or equal to a second set threshold, as neighborhood laser points of the visual feature point; and determining laser points that meet the foreground conditions among the neighborhood laser points as N laser points corresponding to the visual feature point.

[0090] Assumption Visual image frames acquired at different times and Time to Corresponding to the laser point cloud frames collected at any time. g ≤ i ≤ h , ig=KT, K positive integer, T This refers to the time interval for the lidar to periodically acquire laser point cloud frames. When calculating the laser depth of visual feature points in the constantly acquired visual image frames, it is necessary to rely on Time to Laser points in laser point cloud frames collected at any time.

[0091] Figure 5 This is a schematic diagram illustrating the acquisition of laser point cloud frames by an acquisition device according to an embodiment of this disclosure. See also... Figure 5 ,exist At time t, part P1 of object P is in a visible (i.e., not occluded) state. The constantly acquired laser point cloud frames include laser points formed by partial reflection from P1. Then, over time, the acquisition device moves continuously. At time 1, part P1 of object P is occluded by object Q. The laser point cloud frames acquired at any time do not include laser points formed by partial reflection from P1, but do include laser points formed by reflection from object Q.

[0092] because At that moment, part P1 was obscured. The visual image frames acquired at each moment do not include visual feature points corresponding to part P1, but do include visual feature points corresponding to object Q. However, during the execution of S122, [the visual feature points will be...]. Laser point cloud frames collected in real time and The laser point cloud frames acquired at different times are accumulated, and the resulting accumulated local point cloud map includes laser points formed by partial reflection from P1. Consequently, the neighborhood laser points of the visual feature point corresponding to object Q include those formed by partial reflection from P1. Clearly, calculating the laser depth of the visual feature point corresponding to object Q based on the laser points formed by partial reflection from P1 would lead to inaccurate calculations. The foreground condition in the above technical solution is used to filter out laser points formed by the occluded object (such as...). Figure 4 The laser point formed by partial reflection of P1 solves the problem of the invisible occluded point cloud having a depth value after multiple frames of laser point cloud are superimposed, which can make the laser depth recovered in the end accurate.

[0093] Optionally, the laser points that meet the foreground conditions in the neighborhood laser points are identified as N laser points corresponding to the visual feature points, including: determining the Euclidean distance from each laser point in the neighborhood laser points to the camera optical center; determining the minimum Euclidean distance based on the Euclidean distance from each laser point to the camera optical center; and identifying the laser points whose difference between the Euclidean distance from each laser point to the camera optical center and the minimum Euclidean distance is less than or equal to a third preset threshold as N laser points corresponding to the visual feature points.

[0094] Optionally, the minimum Euclidean distance can be determined based on the Euclidean distances from each laser point to the camera optical center. Specifically, this can include: sorting the Euclidean distances from each laser point to the camera optical center in ascending order to form an Euclidean distance queue; and taking the Euclidean distance at the first position in the Euclidean distance queue as the minimum Euclidean distance.

[0095] This application does not limit the specific value of the third set threshold. For example, the third set threshold can be set to 2m.

[0096] Since occluded objects are often relatively far from the camera, a third set threshold is established to select N laser points corresponding to visual feature points where the difference between the Euclidean distance to the camera's optical center and the minimum Euclidean distance is less than or equal to the threshold. This effectively eliminates laser points formed by reflections from occluded objects, thus ensuring the accuracy of the laser depth at the final visual feature points.

[0097] Based on the above technical solution, there are various methods that can be used to implement S140, and this application does not limit them. Figure 6 A flowchart illustrating a method for implementing S140, provided as an embodiment of this disclosure. See also... Figure 6 The method for implementing S140 includes:

[0098] S210. Determine whether the smoothed target point cloud meets the first plane fitting condition.

[0099] S220. When the smooth target point cloud meets the first plane fitting condition, perform depth calculation based on the smooth target point cloud.

[0100] If a smooth target point cloud meets the first plane fitting condition, it means that a plane can be fitted based on the smooth target point cloud. If a smooth target point cloud does not meet the first plane fitting condition, it means that a plane cannot be fitted based on the smooth target point cloud.

[0101] Since depth calculation is performed based on the smooth target point cloud when the smooth target point cloud meets the first plane fitting condition, the influence of unreasonable plane fitting results on the depth recovery of visual feature points can be avoided, and the accuracy of the laser depth information of the final visual feature points can be ensured.

[0102] Figure 7 A flowchart illustrating a method for implementing S210, provided as an embodiment of this disclosure. See also... Figure 7 The method for implementing S210 includes:

[0103] S211. Construct a local surface based on N laser points in the smooth target point cloud.

[0104] S212. Obtain the covariance matrix of the local surface.

[0105] S213. Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues ​​of the first eigenvector, the second eigenvector, and the third eigenvector arranged in order of size. If the ratio of the eigenvalue of the first eigenvector to the eigenvalue of the second eigenvector is less than the fourth set threshold, then the smoothed target point cloud meets the first plane fitting condition.

[0106] For example, Figure 8 and Figure 9 Schematic diagrams of two local surfaces fitted from smooth target point clouds provided in embodiments of this disclosure. See also Figure 8 and Figure 9 Suppose that k laser points in a smooth target point cloud constitute a local surface P of the point cloud. The covariance matrix of the local surface reflects the local geometric characteristics of the point cloud. The local surface covariance matrix of the point cloud is defined as follows:

[0107] .

[0108] in The geometric center of a local surface is represented by the following form:

[0109]

[0110] For covariance matrix Eigenvalue decomposition yields three eigenvectors. , and And the eigenvalues ​​corresponding to the three eigenvectors, arranged in ascending order from smallest to largest. , and eigenvectors and These represent the normal direction and principal direction of the local surface P, respectively.

[0111] Let the selected smooth laser points form a local surface set. The set is calculated based on the aforementioned local surface covariance matrix. Calculate the covariance matrix of all points within the local surface and perform eigenvalue decomposition on the covariance matrix. Analyze the results of the eigenvalue decomposition of the local surface covariance matrix, and if the intermediate eigenvalues... Much larger than the smallest eigenvalue This indicates that the smallest eigenvalue is... corresponding feature vector The magnitude of the eigenvector is much smaller than the magnitudes of the other two eigenvectors. This local surface conforms to... Figure 9 Given the planar geometric features, it can be assumed that all laser points within the local surface can be fitted into a plane.

[0112] Figure 10 A flowchart illustrating another method for implementing S140, provided as an embodiment of this disclosure. See also... Figure 10 The method for implementing S140 includes:

[0113] S310. The laser point plane is obtained by fitting the N laser points that meet the smoothing conditions corresponding to the visual feature points.

[0114] S320. Determine the normal vector of the laser point plane.

[0115] For example, in the camera coordinate system, the five laser points corresponding to a certain visual feature point that meet the smoothing condition are represented as follows: According to spatial geometry, the general equation of a plane in three-dimensional space can be expressed as:

[0116]

[0117] Where vector It is the normal vector of the plane. D Generally acceptable Five smooth laser points Substituting the general equations of a plane in three-dimensional space into these equations, we can form an overdetermined system of linear equations.

[0118] .

[0119] Where the coefficient matrix constant matrix The plane normal vector can be obtained by performing QR decomposition (i.e., orthogonal trigonometric decomposition) on the overdetermined linear equation system. At this point, the laser point plane equation and its normal vector for depth recovery of visual feature points are obtained. If the coordinates of any one of the five laser points satisfy...

[0120]

[0121] This indicates that the plane fitting effect is poor, and no interpolation is performed to restore the depth of visual feature points. It sets a threshold.

[0122] S330. Determine the laser depth of the visual feature point based on the normal vector of the laser point plane.

[0123] Alternatively, the laser depth of the visual feature point can be obtained according to the following formula. d :

[0124]

[0125] in, Let x be the normal vector of the laser point plane, and let x be the coordinates of the visual feature point on the camera normalized plane. Let be the coordinates of one of the laser points in the camera coordinate system.

[0126] The reason why the laser depth of visual feature points can be calculated using the above formula is:

[0127] The depth of a visual feature point is defined as the coordinate value of the intersection point of the ray connecting the camera's optical center and the visual feature point with the laser point plane on the z-axis of the camera coordinate system. Figure 11 This is a schematic diagram illustrating the principle of determining the laser depth of a visual feature point based on the normal vector of the laser point plane, as provided in an embodiment of this disclosure. See also... Figure 11 Let the optical center of the camera be... The coordinates of the visual feature points on the normalized plane are

[0128] .

[0129] The coordinates of any three laser points on the laser point plane in the camera coordinate system are as follows: , and spatial point Rays emitted from the optical center of the camera The intersection with the plane of the laser point is the line segment. The projection length on the z-axis of the camera coordinate system is the visual feature point. The laser depth value. Known vector. Since the scalar product of any vector in the plane and the normal vector of the plane is 0, we can obtain...

[0130] .

[0131] Transforming the above formula yields the laser depth of the visual feature point. d The solution formula is the equation. .

[0132] The coordinates of the spatial points corresponding to the visual feature points in the camera coordinate system are: .

[0133] It should be noted that combining the above-mentioned technical features results in the visual feature point depth recovery method provided in this application including one or more screening steps. These could include screening steps using smoothing conditions, foreground conditions, or first plane fitting conditions. Due to the effect of one or more screening steps, only a portion of the visual feature points in the visual image frame can have their laser depth recovered based on the visual feature point depth recovery method provided in this application. Therefore, a depth channel with two attributes can be set for the visual feature points, referred to as laser depth and triangulation depth, respectively. For visual feature points whose laser depth can be obtained using any of the methods provided in this disclosure, their laser depth is equal to the final calculation result obtained according to any of the methods provided in this disclosure, indicating that the laser depth of the feature point is valid. For visual feature points whose laser depth cannot be obtained using any of the methods provided in this disclosure, their laser depth is set to -1, indicating that the laser depth of the feature point is invalid, and in this case, only triangulation can be used to recover its triangulation depth.

[0134] The method of recovering the triangulation depth of visual feature points using triangulation is an existing technology and will not be described in detail here.

[0135] Figure 12 This is a flowchart illustrating a monocular vision initialization method provided in an embodiment of this disclosure. The method can be executed by a monocular vision initialization device, which can be implemented in software and / or hardware. This device can be configured in electronic devices, such as smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, smart home devices, and vehicles. See also... Figure 12 The method includes:

[0136] S510. Determine the reference visual image frame among the multiple visual image frames for monocular vision initialization.

[0137] Because the visual image frames include visual feature points whose laser depth cannot be obtained using any of the methods provided in this disclosure, triangulation is required to recover the triangulated depth of such visual feature points. Triangulation requires estimating the pose changes of the acquisition device at different visual image frame acquisition times. A reference visual image frame is used as a reference object to estimate the pose changes of the acquisition device at acquisition times among the multiple visual image frames used for monocular vision initialization, excluding the reference visual image frame.

[0138] There are various ways to implement this step, and this application does not limit this one. For example, any one of a plurality of visual image frames can be selected as the reference visual image frame.

[0139] In one embodiment, the implementation method of this step includes: performing feature recognition on each visual image frame in the sliding window and the current visual image frame; if the number of visual feature points continuously tracked from the Lth visual image frame to the current image frame is greater than or equal to a first preset value, and the disparity from the Lth visual image frame to the current visual image frame is greater than or equal to a second preset value of pixels, then the Lth visual image frame is determined as a reference visual image frame.

[0140] The sliding window is used to perform preliminary screening of the visual image frame sequence to obtain visual image frames used as initialization for monocular vision. Figure 13 This is a schematic diagram of a sliding window provided in an embodiment of the present disclosure. The visual image frames within the sliding window are acquired consecutively in time, and the maximum number of visual image frames it can hold is fixed. When the number of visual image frames already held in the sliding window equals its maximum capacity, each time a new visual image frame (whose acquisition time is later than that of any visual image frame currently held in the sliding window) is added to the sliding window, the visual image frame with the earliest acquisition time needs to be removed from the sliding window.

[0141] The current image frame is the visual image frame acquired latest within the sliding window. In the following text, the current image frame will be referred to simply as the current frame.

[0142] See also Figure 13 If the latest frame (i.e., frame 10) within the sliding window is the current frame, and the oldest frame is frame 0, then starting from frame 0, the feature correspondence between each frame and the current frame is checked. If more than 30 visual feature points are continuously tracked between a frame and the current frame, and the disparity between this frame and the current frame is greater than 20 pixels, then it is considered that there is stable feature tracking from this frame to the current frame and that there is sufficient disparity between the two frames. In this case, this frame is selected as the reference visual image frame for visual initialization.

[0143] S520. Based on the depth recovery method for visual feature points provided in the embodiments of this disclosure, determine the laser depth of the first visual feature point that can be depth recovered in multiple visual image frames.

[0144] S530. Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature point of other visual image frames, determine the pose estimation result of other visual image frames. Based on the pose estimation result of other visual image frames, use triangulation to determine the triangulation depth of the second visual feature point in other visual image frames that is observed in the same way as the reference visual image frame. The second visual feature point is a visual feature point that cannot be depth recovered.

[0145] Optionally, assuming the Lth visual image frame is the reference visual image frame, the other visual image frames include the current visual image frame, X first visual image frames between the reference visual image frame L and the current visual image frame, and Y second visual image frames between frame 0 and the reference visual image frame L. The implementation method of this step includes: determining the pose estimation result of the current visual image frame based on the laser depth of the first visual feature points of the reference visual image frame and the laser depth of the first visual feature points of the current visual image frame; and determining the triangulation depth of the second visual feature points in the current visual image frame and the reference visual image frame using triangulation based on the pose estimation result of the current visual image frame; determining the triangulation depth of the second visual feature points in the current visual image frame and the reference visual image frame based on the laser depth of the first visual feature points of the reference visual image frame and the laser depth of the first visual feature points of the X first visual image frames from frame L+1 to the current visual image frame; and determining the pose estimation result of the X first visual image frames from frame L+1 to the current visual image frame based on the laser depth of the first visual feature points of the reference visual image frame and the laser depth of the first visual feature points of the X first visual image frames from frame L+1 to the current visual image frame. Based on the pose estimation results of the visual image frames, the triangulation depth of the second visual feature points observed jointly with the reference visual image frame in the X first visual image frames from frame L+1 to the current visual image frame is determined sequentially using triangulation. Based on the laser depth of the first visual feature points in the reference visual image frame and the laser depth of the first visual feature points in the Y second visual image frames from frame L-1 to frame 0, the pose estimation results of the Y second visual image frames from frame L-1 to frame 0 are determined sequentially. Based on the pose estimation results of the Y second visual image frames from frame L-1 to frame 0, the triangulation depth of the second visual feature points observed jointly with the reference visual image frame in the Y second visual image frames from frame L-1 to frame 0 is determined sequentially using triangulation.

[0146] Optionally, determining the pose estimation results for other visual image frames can include using the Perspective-n-Point algorithm. Compared to epipolar geometric constraints, the Perspective-n-Point algorithm avoids scale uncertainty and scale drift, resulting in pose estimation results without scale scaling. Since the pose estimation results are not scale-scaled, the triangulation depth of the second visual feature points recovered using triangulation is more accurate, improving the initialization accuracy, speed, and robustness.

[0147] See also Figure 13For example, starting from frame L+1 and looping towards the current frame, the PnP algorithm is used to estimate the pose between frames L+n (n=1, 2, 3...) and the reference frame. Based on the estimated pose, triangulation is used to recover the triangulation depth of the second visual feature points jointly observed in frames L+n and the current frame. This process is repeated until the frame preceding the current frame. A new loop is then set up, and similar steps are repeated from frame Ln (n=1, 2, 3...) towards frame 0 until the poses of all frames within the sliding window relative to the reference frame are estimated and all second visual feature points are triangulated. Finally, bundle adjustment is performed on all frames within the sliding window to re-optimize the poses of all frames and the triangulation depth of all feature points.

[0148] The monocular vision initialization method provided in this application is based on the depth recovery method for any visual feature point provided in the embodiments of this disclosure, and combines it with existing triangulation methods to perform depth recovery on all visual feature points in a visual image frame. Since the depth information obtained through the depth recovery method for any visual feature point provided in the embodiments of this disclosure is relatively accurate, the final initialization result also has high precision and robustness.

[0149] The monocular vision initialization method described above does not require specific ambient light levels or light direction during execution, and can be applied to outdoor scenes with ample sunlight.

[0150] Using the technical solution provided in this application, only a monocular camera is needed in the localization process of a known visual map. At this time, the binocular camera is redundant, and the binocular camera has the problem of periodic replacement and recalibration. However, the system that fuses a monocular camera with LiDAR does not cause a waste of sensor configuration.

[0151] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0152] Figure 14 This is a schematic diagram of the structure of a depth recovery device for visual feature points according to an embodiment of this disclosure. The depth recovery device for visual feature points provided in this embodiment can be used in electronic devices. See also... Figure 14 The depth recovery device for visual feature points specifically includes:

[0153] The first acquisition module 610 is used to acquire visual image frames and laser point cloud frames. Therefore, the visual image frame includes multiple visual feature points, and the laser point cloud frame includes multiple laser points.

[0154] The correspondence establishment module 620 is used to establish the correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5.

[0155] The filtering module 630 is used to filter the target point cloud of the visual feature points to obtain a smooth target point cloud that meets the smoothing conditions.

[0156] The depth calculation module 640 is used to perform depth calculation based on the smooth target point cloud that meets the smoothing conditions, so as to obtain the laser depth of the visual feature points.

[0157] Furthermore, the filtering module 630 is used for:

[0158] Calculate the local curvature of the point cloud corresponding to each laser point in the target point cloud;

[0159] If the curvature of the local point cloud corresponding to at least 5 laser points is less than the first set threshold, then the target point cloud meets the smoothing condition.

[0160] Furthermore, the filtering module 630 is used for:

[0161]

[0162] in, c This refers to the local curvature of the point cloud at the current laser point. The set of laser points corresponding to the current laser point. This indicates the number of laser points in the laser point set. This indicates the depth of the current laser point. This indicates the depth of other laser points in the laser point set.

[0163] Furthermore, the acquired laser point cloud frames are multiple laser point cloud frames collected at different times, and the device also includes an accumulation module;

[0164] The accumulation module is used for:

[0165] The multiple laser point cloud frames acquired at different times in the laser coordinate system are transformed to the world coordinate system.

[0166] Multiple laser point cloud frames in the world coordinate system are accumulated to obtain an accumulated local point cloud map;

[0167] The correspondence establishment module is used for:

[0168] Based on the accumulated local point cloud map, obtain the local point cloud map in the camera coordinate system at the acquisition time of the visual image frame;

[0169] The visual feature points in the visual image frame and the laser points in the local point cloud map in the camera coordinate system at the acquisition time of the visual image frame are all projected onto the camera normalization plane.

[0170] Multiple laser points are obtained corresponding to each visual feature point, and the distance between the multiple laser points and the visual feature point on the normalization plane is less than or equal to a second set threshold.

[0171] Furthermore, the correspondence establishment module is used for:

[0172] Laser points whose distance from the visual feature point on the normalization plane is less than or equal to a second set threshold are identified as neighborhood laser points of the visual feature point.

[0173] Among the neighborhood laser points, laser points that meet the foreground conditions are identified as N laser points corresponding to the visual feature points.

[0174] Furthermore, the correspondence establishment module is used for:

[0175] Determine the Euclidean distance from each laser point in the neighborhood laser point to the camera optical center;

[0176] The minimum Euclidean distance is determined based on the Euclidean distance from each laser point to the optical center of the camera.

[0177] The laser points whose difference between the Euclidean distance to the optical center of the camera and the minimum Euclidean distance is less than or equal to a third set threshold are identified as N laser points corresponding to the visual feature points.

[0178] Furthermore, the depth calculation module is used for:

[0179] Determine whether the smoothed target point cloud meets the first plane fitting condition;

[0180] And when the smoothed target point cloud meets the first plane fitting condition, depth calculation is performed based on the smoothed target point cloud.

[0181] Furthermore, the depth calculation module is used for:

[0182] A local surface is constructed based on N laser points in the smoothed target point cloud;

[0183] Obtain the covariance matrix of the local surface;

[0184] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain the eigenvalues ​​of the first eigenvector, the second eigenvector, and the third eigenvector arranged in order of size. If the ratio of the eigenvalue of the first eigenvector to the eigenvalue of the second eigenvector is less than a fourth set threshold, then the smoothed target point cloud meets the first plane fitting condition.

[0185] Furthermore, the depth calculation module is used for:

[0186] The laser point plane is obtained by fitting N laser points that meet the smoothing conditions corresponding to the visual feature points.

[0187] Determine the normal vector of the laser point plane;

[0188] The laser depth of the visual feature point is determined based on the normal vector of the laser point plane.

[0189] Furthermore, the depth calculation module is used for:

[0190] The laser depth of the visual feature point is obtained according to the following formula. d :

[0191]

[0192] in, Let x be the normal vector of the laser point plane, and let x be the coordinates of the visual feature point on the camera normalized plane. Let be the coordinates of one of the laser points in the camera coordinate system.

[0193] The depth recovery device for visual feature points provided in this disclosure can execute the steps in the depth recovery method for visual feature points provided in this disclosure, and has the execution steps and beneficial effects, which will not be described in detail here.

[0194] Figure 15 This is a schematic diagram of the structure of a monocular vision initialization device according to an embodiment of this disclosure. The monocular vision initialization device provided in this embodiment can be used in electronic devices. See also... Figure 15 The monocular vision initialization device specifically includes:

[0195] The reference frame determination module 710 is used to determine the reference visual image frame among multiple visual image frames used for monocular vision initialization.

[0196] The laser depth acquisition module 720 is used to determine the laser depth of the first visual feature point that can be depth recovered in the plurality of visual image frames based on the depth recovery device of the visual feature points described above.

[0197] The triangulation depth acquisition module 730 is used to determine the pose estimation result of the other visual image frames based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, and to determine the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame using the triangulation method based on the pose estimation result of the other visual image frames. The second visual feature point is a visual feature point that cannot be depth recovered.

[0198] Furthermore, the reference frame determination module is used for:

[0199] Feature recognition is performed on each visual image frame in the sliding window and the current visual image frame. If the number of continuously tracked visual feature points from the Lth visual image frame to the current image frame is greater than or equal to a first preset value, and the disparity from the Lth visual image frame to the current visual image frame is greater than or equal to a second preset value of pixels, then the Lth visual image frame is determined as the reference visual image frame.

[0200] Furthermore, the other visual image frames include the current visual image frame, X first visual image frames between the reference visual image frame L and the current visual image frame, and Y second visual image frames between frame 0 and the reference visual image frame L;

[0201] The triangulation depth acquisition module is used for:

[0202] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature point of the current visual image frame, the pose estimation result of the current visual image frame is determined, and based on the pose estimation result of the current visual image frame, the triangulation depth of the second visual feature point observed in the current visual image frame and the reference visual image frame is determined by the triangulation method.

[0203] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature point of the X first visual image frames from the L+1 frame to the current visual image frame, the pose estimation results of the X first visual image frames from the L+1 frame to the current visual image frame are determined sequentially. Based on the pose estimation results of the X first visual image frames from the L+1 frame to the current visual image frame, the triangulation depth of the second visual feature point observed in the X first visual image frames from the L+1 frame to the current visual image frame and the reference visual image frame is determined sequentially using the triangulation method.

[0204] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature point of the Y second visual image frames from frame L-1 to frame 0, the pose estimation results of the Y second visual image frames from frame L-1 to frame 0 are determined sequentially. Based on the pose estimation results of the Y second visual image frames from frame L-1 to frame 0, the triangulation depth of the second visual feature point observed in the Y second visual image frames from frame L-1 to frame 0 and the reference visual image frame is determined sequentially using the triangulation method.

[0205] Furthermore, the triangulation depth acquisition module is used to determine the pose estimation results of the other visual image frames using the Perspective-n-Point algorithm.

[0206] The monocular vision initialization device provided in this disclosure can execute the steps in the monocular vision initialization method provided in this disclosure, and has the functions of execution steps and beneficial effects, which will not be described in detail here.

[0207] Figure 16 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 16 The diagram illustrates a structural schematic suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 16 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0208] like Figure 16 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003 to implement a depth recovery method or a monocular vision initialization method for visual feature points as described in the embodiments of this disclosure. The RAM 1003 also stores various programs and information required for the operation of the electronic device 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0209] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to exchange information with other devices wirelessly or via wired communication. Although Figure 16 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0210] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the depth recovery method or monocular vision initialization method for visual feature points as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0211] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include information signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated information signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0212] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0213] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0214] Acquire visual image frames and laser point cloud frames, wherein the visual image frames include multiple visual feature points and the laser point cloud frames include multiple laser points;

[0215] Establish a correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5.

[0216] The target point cloud of the visual feature points is filtered to obtain a smooth target point cloud that meets the smoothing conditions.

[0217] Depth calculation is performed based on the smooth target point cloud that meets the smoothing conditions to obtain the laser depth of the visual feature points.

[0218] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0219] Identify the reference visual image frame among multiple visual image frames used for monocular vision initialization;

[0220] Based on the depth recovery method of the above visual feature points, the laser depth of the first visual feature point that can be depth recovered in the plurality of visual image frames is determined.

[0221] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, the pose estimation result of the other visual image frames is determined. Based on the pose estimation result of the other visual image frames, the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame is determined by triangulation. The second visual feature point is a visual feature point that cannot be depth recovered.

[0222] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0223] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0224] The units described in the embodiments of this disclosure can be implemented in software or in hardware.

[0225] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0226] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0227] One or more processors;

[0228] Memory, used to store one or more programs;

[0229] When the one or more programs are executed by the one or more processors, the one or more processors implement the depth recovery method or monocular vision initialization method for visual feature points as provided in any of the present disclosure.

[0230] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a depth recovery method or a monocular vision initialization method for visual feature points as described in any of the present disclosure.

[0231] This disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the depth recovery method or monocular vision initialization method for visual feature points as described above.

[0232] Solution 1: A depth recovery method for visual feature points, the method comprising:

[0233] Acquire visual image frames and laser point cloud frames, wherein the visual image frames include multiple visual feature points and the laser point cloud frames include multiple laser points;

[0234] Establish a correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5.

[0235] The target point cloud of the visual feature points is filtered to obtain a smooth target point cloud that meets the smoothing conditions.

[0236] Depth calculation is performed based on the smooth target point cloud that meets the smoothing conditions to obtain the laser depth of the visual feature points.

[0237] Solution 2: According to the method described in Solution 1, the target point cloud of the visual feature points is filtered to obtain a smooth target point cloud that meets the smoothing condition, including:

[0238] Calculate the local curvature of the point cloud corresponding to each laser point in the target point cloud;

[0239] If the curvature of the local point cloud corresponding to at least 5 laser points is less than the first set threshold, then the target point cloud meets the smoothing condition.

[0240] Option 3: According to the method described in Option 2, the calculation of the local curvature of the point cloud corresponding to each laser point in the target point cloud includes:

[0241]

[0242] in, c This refers to the local curvature of the point cloud at the current laser point. The set of laser points corresponding to the current laser point. This indicates the number of laser points in the laser point set. This indicates the depth of the current laser point. This indicates the depth of other laser points in the laser point set.

[0243] Option 4: According to the method described in Option 1, the obtained laser point cloud frame is multiple laser point cloud frames acquired at different times, and the method further includes:

[0244] The multiple laser point cloud frames acquired at different times in the laser coordinate system are transformed to the world coordinate system.

[0245] Multiple laser point cloud frames in the world coordinate system are accumulated to obtain an accumulated local point cloud map;

[0246] Establishing the correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame includes:

[0247] Based on the accumulated local point cloud map, obtain the local point cloud map in the camera coordinate system at the acquisition time of the visual image frame;

[0248] The visual feature points in the visual image frame and the laser points in the local point cloud map in the camera coordinate system at the acquisition time of the visual image frame are all projected onto the camera normalization plane.

[0249] Multiple laser points are obtained corresponding to each visual feature point, and the distance between the multiple laser points and the visual feature point on the normalization plane is less than or equal to a second set threshold.

[0250] Option 5: According to the method described in Option 4, obtaining N laser points corresponding to each visual feature point includes:

[0251] Laser points whose distance from the visual feature point on the normalization plane is less than or equal to a second set threshold are identified as neighborhood laser points of the visual feature point.

[0252] Among the neighborhood laser points, laser points that meet the foreground conditions are identified as N laser points corresponding to the visual feature points.

[0253] Option 6: According to the method described in Option 5, the step of determining the laser points that meet the foreground conditions among the neighboring laser points as N laser points corresponding to the visual feature points includes:

[0254] Determine the Euclidean distance from each laser point in the neighborhood laser point to the camera optical center;

[0255] The minimum Euclidean distance is determined based on the Euclidean distance from each laser point to the optical center of the camera.

[0256] The laser points whose difference between the Euclidean distance to the optical center of the camera and the minimum Euclidean distance is less than or equal to a third set threshold are identified as N laser points corresponding to the visual feature points.

[0257] Solution 7: According to the method described in Solution 1, the depth calculation based on the smoothed target point cloud that meets the smoothing conditions includes:

[0258] Determine whether the smoothed target point cloud meets the first plane fitting condition;

[0259] And when the smoothed target point cloud meets the first plane fitting condition, depth calculation is performed based on the smoothed target point cloud.

[0260] Solution 8: According to the method described in Solution 7, determining whether the smoothed target point cloud meets the first plane fitting condition includes:

[0261] A local surface is constructed based on N laser points in the smoothed target point cloud;

[0262] Obtain the covariance matrix of the local surface;

[0263] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain the eigenvalues ​​of the first eigenvector, the second eigenvector, and the third eigenvector arranged in order of size. If the ratio of the eigenvalue of the first eigenvector to the eigenvalue of the second eigenvector is less than a fourth set threshold, then the smoothed target point cloud meets the first plane fitting condition.

[0264] Solution 9: According to the method described in Solution 1, the depth calculation based on the smoothed target point cloud that meets the smoothing conditions includes:

[0265] The laser point plane is obtained by fitting N laser points in the smooth target point cloud that meet the smoothing conditions corresponding to the visual feature points.

[0266] Determine the normal vector of the laser point plane;

[0267] The laser depth of the visual feature point is determined based on the normal vector of the laser point plane.

[0268] Solution 10: According to the method described in Solution 9, determining the depth of the visual feature point based on the normal vector of the laser point plane includes:

[0269] The laser depth of the visual feature point is obtained according to the following formula. d :

[0270]

[0271] in, Let x be the normal vector of the laser point plane, and let x be the coordinates of the visual feature point on the camera normalized plane. Let be the coordinates of one of the laser points in the camera coordinate system.

[0272] Option 11: A monocular vision initialization method, comprising:

[0273] Identify the reference visual image frame among multiple visual image frames used for monocular vision initialization;

[0274] Based on the method described in any of schemes 1-10, determine the laser depth of the first visual feature point in the plurality of visual image frames that can be depth recovered;

[0275] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, the pose estimation result of the other visual image frames is determined. Based on the pose estimation result of the other visual image frames, the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame is determined by triangulation. The second visual feature point is a visual feature point that cannot be depth recovered.

[0276] Solution 12: According to the method described in Solution 11, determining the reference visual image frame among the multiple visual image frames used for monocular vision initialization includes:

[0277] Feature recognition is performed on each visual image frame in the sliding window and the current visual image frame. If the number of continuously tracked visual feature points from the Lth visual image frame to the current image frame is greater than or equal to a first preset value, and the disparity from the Lth visual image frame to the current visual image frame is greater than or equal to a second preset value of pixels, then the Lth visual image frame is determined as the reference visual image frame.

[0278] Scheme 13: According to the method described in Scheme 12, the other visual image frames include the current visual image frame, X first visual image frames between the reference visual image frame L and the current visual image frame, and Y second visual image frames between the 0th frame and the reference visual image frame L.

[0279] The laser depth of the first visual feature point in the reference visual image frame and the laser depth of the first visual feature points in other visual image frames are used to determine the pose estimation result of the other visual image frames. Based on the pose estimation result of the other visual image frames, the triangulation depth of the second visual feature point in the other visual image frames that is jointly observed with the reference visual image frame is determined using triangulation, including:

[0280] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature point of the current visual image frame, the pose estimation result of the current visual image frame is determined, and based on the pose estimation result of the current visual image frame, the triangulation depth of the second visual feature point observed in the current visual image frame and the reference visual image frame is determined by the triangulation method.

[0281] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature point of the X first visual image frames from the L+1 frame to the current visual image frame, the pose estimation results of the X first visual image frames from the L+1 frame to the current visual image frame are determined sequentially. Based on the pose estimation results of the X first visual image frames from the L+1 frame to the current visual image frame, the triangulation depth of the second visual feature point observed in the X first visual image frames from the L+1 frame to the current visual image frame and the reference visual image frame is determined sequentially using the triangulation method.

[0282] Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature point of the Y second visual image frames from frame L-1 to frame 0, the pose estimation results of the Y second visual image frames from frame L-1 to frame 0 are determined sequentially. Based on the pose estimation results of the Y second visual image frames from frame L-1 to frame 0, the triangulation depth of the second visual feature point observed in the Y second visual image frames from frame L-1 to frame 0 and the reference visual image frame is determined sequentially using the triangulation method.

[0283] Solution 14: According to the method described in Solution 11, determining the pose estimation results of the other visual image frames includes:

[0284] The pose estimation results of the other visual image frames are determined using the Perspective-n-Point algorithm.

[0285] Solution 15: A depth recovery device for visual feature points, comprising:

[0286] The first acquisition module is used to acquire visual image frames and laser point cloud frames. Therefore, the visual image frame includes multiple visual feature points, and the laser point cloud frame includes multiple laser points.

[0287] The correspondence establishment module is used to establish the correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5.

[0288] The filtering module is used to filter the target point cloud of the visual feature points to obtain a smooth target point cloud that meets the smoothing conditions.

[0289] The depth calculation module is used to perform depth calculation based on the smooth target point cloud that meets the smoothing conditions, so as to obtain the laser depth of the visual feature points.

[0290] Option 16: A monocular vision initialization device, comprising:

[0291] The reference frame determination module is used to determine the reference visual image frame among multiple visual image frames used for monocular vision initialization.

[0292] A laser depth acquisition module is used to determine the laser depth of a first visual feature point that can be depth recovered in the plurality of visual image frames based on the depth recovery device of the visual feature points described in Scheme 15.

[0293] The triangulation depth acquisition module is used to determine the pose estimation result of the other visual image frames based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, and to determine the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame using the triangulation method based on the pose estimation result of the other visual image frames. The second visual feature point is a visual feature point that cannot be depth recovered.

[0294] Solution 17: An electronic device, characterized in that the electronic device comprises:

[0295] One or more processors;

[0296] Storage device for storing one or more programs;

[0297] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of schemes 1-10 or 11-14.

[0298] Scheme 18: A computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method as described in any one of Schemes 1-10 or Schemes 11-14.

[0299] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A depth recovery method for visual feature points, characterized in that, The method includes: Acquire visual image frames and laser point cloud frames, wherein the visual image frames include multiple visual feature points and the laser point cloud frames include multiple laser points; Establish a correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5. The target point cloud of the visual feature points is filtered to obtain a smooth target point cloud that meets the smoothing conditions. Depth calculation is performed on the smooth target point cloud that meets the smoothing conditions to obtain the laser depth of the visual feature points; The step of filtering the target point cloud of the visual feature points to obtain a smooth target point cloud that meets the smoothing conditions includes: Calculate the local curvature of the point cloud corresponding to each laser point in the target point cloud; If the curvature of the local point cloud corresponding to at least 5 laser points is less than the first set threshold, then the target point cloud meets the smoothing condition.

2. The method according to claim 1, characterized in that, The calculation of the local curvature of the point cloud corresponding to each laser point in the target point cloud includes: in, c This refers to the local curvature of the point cloud at the current laser point. The set of laser points corresponding to the current laser point. This indicates the number of laser points in the laser point set. This indicates the depth of the current laser point. This indicates the depth of other laser points in the laser point set.

3. The method according to claim 1, characterized in that, The acquired laser point cloud frames are multiple laser point cloud frames collected at different times, and the method further includes: The multiple laser point cloud frames acquired at different times in the laser coordinate system are transformed to the world coordinate system. Multiple laser point cloud frames in the world coordinate system are accumulated to obtain an accumulated local point cloud map; Establishing the correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame includes: Based on the accumulated local point cloud map, obtain the local point cloud map in the camera coordinate system at the acquisition time of the visual image frame; The visual feature points in the visual image frame and the laser points in the local point cloud map in the camera coordinate system at the acquisition time of the visual image frame are all projected onto the camera normalization plane. Multiple laser points are obtained corresponding to each visual feature point, and the distance between the multiple laser points and the visual feature point on the normalization plane is less than or equal to a second set threshold.

4. The method according to claim 3, characterized in that, The process of obtaining N laser points corresponding to each visual feature point includes: Laser points whose distance from the visual feature point on the normalization plane is less than or equal to a second set threshold are identified as neighborhood laser points of the visual feature point. Among the neighborhood laser points, laser points that meet the foreground conditions are identified as N laser points corresponding to the visual feature points.

5. A monocular vision initialization method, characterized in that, include: Identify the reference visual image frame among multiple visual image frames used for monocular vision initialization; Based on the method according to any one of claims 1-4, determine the laser depth of the first visual feature point in the plurality of visual image frames that can be depth recovered; Based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, the pose estimation result of the other visual image frames is determined. Based on the pose estimation result of the other visual image frames, the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame is determined by triangulation. The second visual feature point is a visual feature point that cannot be depth recovered.

6. A depth recovery device for visual feature points, characterized in that, include: The first acquisition module is used to acquire visual image frames and laser point cloud frames. Therefore, the visual image frame includes multiple visual feature points, and the laser point cloud frame includes multiple laser points. The correspondence establishment module is used to establish the correspondence between visual feature points in the visual image frame and laser points in the laser point cloud frame. The correspondence includes N laser points corresponding to each visual feature point. The corresponding N laser points constitute the target point cloud of the visual feature point, where N is a positive integer greater than or equal to 5. The filtering module is used to filter the target point cloud of the visual feature points to obtain a smooth target point cloud that meets the smoothing conditions. The depth calculation module is used to perform depth calculation based on the smooth target point cloud that meets the smoothing conditions, so as to obtain the laser depth of the visual feature points; The filtering module is used to: calculate the local curvature of the point cloud corresponding to each laser point in the target point cloud; if the local curvature of the point cloud corresponding to at least 5 laser points is less than a first set threshold, then the target point cloud meets the smoothing condition.

7. A monocular vision initialization device, characterized in that, include: The reference frame determination module is used to determine the reference visual image frame among multiple visual image frames used for monocular vision initialization. A laser depth acquisition module is used to determine the laser depth of a first visual feature point that can be depth recovered in the plurality of visual image frames based on the depth recovery device of the visual feature point as described in claim 6. The triangulation depth acquisition module is used to determine the pose estimation result of the other visual image frames based on the laser depth of the first visual feature point of the reference visual image frame and the laser depth of the first visual feature points of other visual image frames, and to determine the triangulation depth of the second visual feature point in the other visual image frames that is observed in the reference visual image frame using the triangulation method based on the pose estimation result of the other visual image frames. The second visual feature point is a visual feature point that cannot be depth recovered.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-4 or claim 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4 or claim 5.

Images