Multi-sensor fusion algorithm based on improved feature point extraction
Through improved feature point extraction algorithm and multi-sensor fusion technology, the problem of visual inertial front-end mischecking edge points as corner points in complex environments is solved, the accuracy and robustness of visual feature detection are improved, and a stable three-dimensional point cloud image is generated.
Patent Information
- Application Number
- CN202411729116.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-07
AI Technical Summary
In complex environments, the front end of the visual inertia in multi-sensor fusion is prone to misdetect edge points as corner points, resulting in positioning drift of the sensor in situations of unstable information, thereby reducing the accuracy and robustness of visual feature detection.
Using a multi-sensor fusion algorithm based on improved feature point extraction, by obtaining the grayscale intensity of the current frame image and the next frame image of the visual camera, calculating the change amount and weighted change amount of the grayscale intensity, performing angular processing to determine the feature value, filter the corner point features, and fuse it with the radar image to generate a three-dimensional point cloud image.
It significantly improves the detection stability and accuracy of visual feature points, reduces positioning drift, and enhances the stability and robustness of the algorithm in complex environments.
Smart Images
Figure CN119577684A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of heterogeneous multi-robot collaborative technology enabled by a large language model, and in particular to a multi-sensor fusion algorithm based on improved feature point extraction. Background Art
[0002] Multi-sensor fusion technology, especially Simultaneous Localization And Mapping (SLAM), has become the key to achieving autonomous navigation in complex environments. By fusing data from multiple sensors, SLAM technology has made significant progress in real-time state estimation and environmental mapping, overcoming the limitations of single sensors (such as lidar or cameras). Lidar is good at capturing external contour information, but its performance is limited in environments lacking structural features. Visual SLAM performs better in environments with rich environments, but motion and lighting affect its stability and robustness. Combining lidar, visual sensors and IMU to improve mapping stability and accuracy has become the current trend. IMU not only corrects lidar data distortion, but also provides continuous posture and position information when features are missing, enhancing the system's environmental adaptability. Summary of the invention
[0003] The embodiments of the present invention provide a multi-sensor fusion algorithm based on improved feature point extraction, which at least solves the technical problems that in complex environments, the visual inertial front end in multi-sensor fusion misdetects edge points as corner points, and the positioning drift of the sensor under unstable information conditions, resulting in low accuracy and poor robustness of visual feature detection.
[0004] According to one aspect of an embodiment of the present invention, a multi-sensor fusion algorithm based on improved feature point extraction is provided. The method may include: obtaining the grayscale intensity corresponding to the center point of the window of the current frame image of the visual camera and the grayscale intensity corresponding to the center point of the window of the next frame image; determining the change of the grayscale intensity based on the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image; determining the weighted change of the grayscale intensity based on the window function of the next frame image and the change of the grayscale intensity; performing angular processing on the autocorrelation matrix in the weighted change of the grayscale intensity to obtain the first eigenvalue and the second eigenvalue; determining whether the center point of the window of the next frame image is a corner point based on the first eigenvalue and the second eigenvalue; extracting the corner point feature when the center point of the window of the next frame image is a corner point; obtaining the radar image corresponding to the next frame image of the visual camera taken by the radar; obtaining a three-dimensional point cloud image based on the corner point features of the radar image and the next frame image.
[0005] Optionally, the expression for determining the change in grayscale intensity based on the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image is:
[0006] in, is the change in grayscale intensity, is the grayscale intensity corresponding to the center point of the window of the next frame image, It is the grayscale intensity corresponding to the center point of the window of the current frame image.
[0007] Optionally, the process of determining the weighted change amount of gray intensity based on the window function and the change amount of gray intensity of the next frame image is:
[0008] in, is the window function of the next frame image, is the weighted change of grayscale intensity; The formula for determining the weighted change of gray intensity is expanded by Taylor expansion, and the approximate expression of the weighted change of gray intensity is obtained as follows:
[0009] in, is the autocorrelation matrix, The u and v in it represent the horizontal and vertical movement of the window of the next frame image respectively.
[0010] Optionally, the expression of the autocorrelation matrix is:
[0011] in, The center point of the window for the next frame of image The gray intensity at The center point of the window for the next frame of image The grayscale intensity at .
[0012] Optionally, the autocorrelation matrix in the weighted variation of the grayscale intensity is subjected to angularization processing to obtain the expressions of the first eigenvalue and the second eigenvalue: =
[0013] in, is an orthogonal matrix, is the inverse matrix of the orthogonal matrix, and are the first and second eigenvalues respectively.
[0014] Optionally, determining whether the window center point of the next frame image is a corner point based on the first eigenvalue and the second eigenvalue includes: when the first eigenvalue and the second eigenvalue are both relatively large, if the minimum value of the first eigenvalue and the second eigenvalue is greater than a preset threshold, then the window center point of the next frame image is a corner point; when the first eigenvalue is much larger than the second eigenvalue or the second eigenvalue is much larger than the first eigenvalue, then the window center point of the next frame image is an edge point; when the first eigenvalue and the second eigenvalue are both relatively small, then the window center point of the next frame image is a plane point.
[0015] Beneficial effects of the present invention: The present invention proposes a multi-sensor fusion algorithm based on improved feature point extraction. By fusing the visual inertial system with the lidar inertial system, a threshold is set for the visual feature points to screen corner points, and corner point features are obtained. The corner point features are fused with a picture taken by the radar at the same time as the corner point features to obtain a three-dimensional point cloud image. The algorithm significantly improves the stability and accuracy of detection. This method effectively solves the problem that the visual inertial front end easily misidentifies edge points as corner points during the multi-sensor fusion process in complex environments, reduces the positioning drift caused by unstable sensor information, and thus improves the stability and robustness of the algorithm in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flow chart of a multi-sensor fusion algorithm based on improved feature point extraction according to an embodiment of the present invention; Figure 2 This is a corner point classification diagram for feature value judgment according to an embodiment of the present invention. DETAILED DESCRIPTION
[0017] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0019] Example 1 According to an embodiment of the present invention, a multi-sensor fusion algorithm based on improved feature point extraction is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.
[0020] Figure 1 is a flow chart of a multi-sensor fusion algorithm based on improved feature point extraction according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: Step S101, obtaining the grayscale intensity corresponding to the center point of the window of the current frame image and the grayscale intensity corresponding to the center point of the window of the next frame image of the visual camera.
[0021] In the technical solution provided in the above step S101 of the present invention, for example, when a vehicle is driving on a road, a visual camera captures a current frame image, and a window is set for the current frame image, and the center of the window is located at the coordinates of the current frame image (grayscale image). The grayscale intensity of the center point of the window is The method for obtaining the grayscale intensity corresponding to the center point of the window of the next frame image is the same as that of the current frame. The current frame image is the image obtained at the first moment, and the next frame image is the image obtained at the second moment.
[0022] Step S102, determining the change amount of the grayscale intensity based on the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image.
[0023] In the technical solution provided in the above step S102 of the present invention, the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image are calculated to obtain the change in grayscale intensity.
[0024] Step S103, determining a weighted change in gray intensity based on the window function and the change in gray intensity of the next frame of image.
[0025] In the technical solution provided in the above step S103 of the present invention, the window function and the change amount of the gray intensity of the next frame image are calculated to obtain the weighted change amount of the gray intensity.
[0026] Step S104, performing angularization processing on the autocorrelation matrix in the weighted variation of the gray intensity to obtain a first eigenvalue and a second eigenvalue.
[0027] In the technical solution provided in the above step S104 of the present invention, the autocorrelation matrix in the weighted variation of the gray intensity is subjected to angularization processing to obtain the first eigenvalue and the second eigenvalue.
[0028] Step S105: determining whether the window center point of the next frame image is a corner point based on the first eigenvalue and the second eigenvalue.
[0029] In the technical solution provided in the above step S105 of the present invention, it is determined whether the window center point of the next frame image is a corner point based on the first eigenvalue and the second eigenvalue.
[0030] Step S106, when the window center point of the next frame image is a corner point, extract the corner point feature.
[0031] In the technical solution provided in the above step S106 of the present invention, if the center point of the window of the next frame image is a corner point, the corner point feature is extracted; if the center point of the window of the next frame image is not a corner point, the corner point feature is not extracted.
[0032] Step S107, obtaining a radar image corresponding to the next frame image of the visual camera taken by the radar.
[0033] In the technical solution provided in the above step S107 of the present invention, a radar image taken by the radar at the second moment is obtained.
[0034] Step S108, obtaining a three-dimensional point cloud image based on the corner point features of the radar image and the next frame image.
[0035] In the technical solution provided in the above step S108 of the present invention, the corner point features of the radar image and the next frame image are fused to obtain a three-dimensional point cloud image.
[0036] The above method of this embodiment is further introduced below.
[0037] As an optional implementation manner, in step S102, the expression for determining the change amount of the grayscale intensity based on the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image is:
[0038] in, is the change in grayscale intensity, is the grayscale intensity corresponding to the center point of the window of the next frame image, It is the grayscale intensity corresponding to the center point of the window of the current frame image.
[0039] In this embodiment, the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image are calculated to obtain the change in grayscale intensity.
[0040] As an optional embodiment, in step S102, the process of determining the weighted change amount of gray intensity based on the window function and the change amount of gray intensity of the next frame image is:
[0041] in, is the window function of the next frame image, is the weighted change of grayscale intensity; The formula for determining the weighted change of gray intensity is expanded by Taylor expansion, and the approximate expression of the weighted change of gray intensity is obtained as follows:
[0042] Among them, The autocorrelation matrix, The u and v in it represent the horizontal and vertical movement of the window of the next frame image respectively.
[0043] As an optional implementation manner, the expression of the autocorrelation matrix is:
[0044] in, The center point of the window for the next frame of image The gray intensity at The center point of the window for the next frame of image The grayscale intensity at .
[0045] As an optional implementation manner, in step S104, the autocorrelation matrix in the weighted variation of the gray intensity is subjected to angularization processing to obtain the expressions of the first eigenvalue and the second eigenvalue: =
[0046] in, is an orthogonal matrix, is the inverse matrix of the orthogonal matrix, and are the first and second eigenvalues respectively.
[0047] As an optional implementation method, step S105, based on the first eigenvalue and the second eigenvalue, determines whether the window center point of the next frame image is a corner point, including: when the first eigenvalue and the second eigenvalue are both relatively large, if the minimum value of the first eigenvalue and the second eigenvalue is greater than a preset threshold, then the window center point of the next frame image is a corner point; when the first eigenvalue is much larger than the second eigenvalue or the second eigenvalue is much larger than the first eigenvalue, then the window center point of the next frame image is an edge point; when the first eigenvalue and the second eigenvalue are both relatively small, then the window center point of the next frame image is a plane point.
[0048] In this embodiment, Figure 2 is a corner point classification diagram for feature value judgment according to an embodiment of the present invention, such as Figure 2 As shown, when the first eigenvalue and the second eigenvalue are both relatively large, if the minimum value of the first eigenvalue and the second eigenvalue is greater than the preset threshold, the center point of the window of the next frame image is a corner point, that is, Figure 2 The gray part in ; when the first eigenvalue is much larger than the second eigenvalue or the second eigenvalue is much larger than the first eigenvalue, the center point of the window of the next frame image is the edge point, that is, Figure 2 The blue part in the image; when the first eigenvalue and the second eigenvalue are both small, the center point of the window of the next frame image is a plane point, that is, Figure 2 The orange part.
[0049] In an embodiment of the present invention, the grayscale intensity corresponding to the center point of the window of the current frame image of the visual camera and the grayscale intensity corresponding to the center point of the window of the next frame image are obtained; based on the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image, the change amount of the grayscale intensity is determined; based on the window function of the next frame image and the change amount of the grayscale intensity, the weighted change amount of the grayscale intensity is determined; the autocorrelation matrix in the weighted change amount of the grayscale intensity is angularized to obtain the first eigenvalue and the second eigenvalue; based on the first eigenvalue and the second eigenvalue, the center point of the window of the next frame image is determined. Whether it is a corner point; when the center point of the window of the next frame of image is a corner point, extract the corner point features; obtain the radar image corresponding to the next frame of image taken by the visual camera of the radar; based on the corner point features of the radar image and the next frame of image, obtain the three-dimensional point cloud image, which solves the technical problems of low accuracy and poor robustness of visual feature detection caused by the visual inertial front end misdetecting edge points as corner points in multi-sensor fusion in complex environments and the positioning drift of sensors under unstable information conditions. It achieves the technical effect of screening corner points by setting thresholds for visual feature points in complex environments, thereby improving the accuracy and robustness of visual feature detection.
[0050] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0051] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0052] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0053] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0054] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0055] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A multi-sensor fusion algorithm based on improved feature point extraction, characterized in that: include: Obtain the grayscale intensity corresponding to the center point of the window of the current frame image of the visual camera and the grayscale intensity corresponding to the center point of the window of the next frame image; Determine the change amount of the grayscale intensity based on the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image; Determine a weighted change in grayscale intensity based on a window function and a change in grayscale intensity of a next frame of image; Performing angular processing on the autocorrelation matrix in the weighted variation of the gray intensity to obtain the first eigenvalue and the second eigenvalue; Based on the first eigenvalue and the second eigenvalue, determining whether the center point of the window of the next frame image is a corner point; When the center point of the window of the next frame image is a corner point, the corner point features are extracted; Obtain a radar image corresponding to the next frame of the visual camera captured by the radar; Based on the corner point features of the radar image and the next frame image, a three-dimensional point cloud image is obtained.
2. The method according to claim 1, characterized in that The expression for determining the change in grayscale intensity based on the grayscale intensity corresponding to the center point of the window of the next frame image and the grayscale intensity corresponding to the center point of the window of the current frame image is: in, is the change in grayscale intensity, is the grayscale intensity corresponding to the center point of the window of the next frame image, It is the grayscale intensity corresponding to the center point of the window of the current frame image.
3. The method according to claim 1, characterized in that The process of determining the weighted change in gray intensity based on the window function and the change in gray intensity of the next frame image is as follows: in, is the window function of the next frame image, is the weighted change of grayscale intensity; The formula for determining the weighted change of gray intensity is expanded by Taylor expansion, and the approximate expression of the weighted change of gray intensity is obtained as follows: in, is the autocorrelation matrix, The u and v in it represent the horizontal and vertical movement of the window of the next frame image respectively.
4. The method according to claim 3, characterized in that The expression of the autocorrelation matrix is: in, The center point of the window for the next frame of image The gray intensity at The center point of the window for the next frame of image The grayscale intensity at .
5. The method according to claim 1, characterized in that The autocorrelation matrix in the weighted variation of the gray intensity is subjected to angularization processing to obtain the expressions of the first eigenvalue and the second eigenvalue: = in, is an orthogonal matrix, is the inverse matrix of the orthogonal matrix, and are the first and second eigenvalues respectively.
6. The method according to claim 1, characterized in that The step of determining whether the window center point of the next frame image is a corner point based on the first eigenvalue and the second eigenvalue comprises: When both the first eigenvalue and the second eigenvalue are relatively large, if the minimum value of the first eigenvalue and the second eigenvalue is greater than a preset threshold, the center point of the window of the next frame image is a corner point; When the first eigenvalue is much larger than the second eigenvalue or the second eigenvalue is much larger than the first eigenvalue, the center point of the window of the next frame image is an edge point; When the first eigenvalue and the second eigenvalue are both relatively small, the center point of the window of the next frame image is a plane point.
7. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.
8. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.
9. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.