Aerial target three-dimensional reconstruction method and system based on monocular vision
By using geometric feature points of aerial targets for monocular vision, the problem of high computing power consumption by monocular vision reconstruction in the existing technology is solved, more efficient target tracking and reconstruction is achieved, and the hardware of the intelligent driving system is reduced. cost.
Patent Information
- Application Number
- CN202410918544.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-26
AI Technical Summary
The existing three-dimensional target reconstruction method based on monocular vision consumes a lot of computing power on the vehicle side and cannot meet the real-time needs of intelligent driving systems.
The geometric feature points of the aerial target are used to replace the traditional ORB feature points. The external contour of a single frame aerial target is extracted through OpenCV, the target center point and key point position are calculated, the target tracking is performed based on the pixel overlap, and the Kalman filtering and sliding window are used. Optimize the position and realize three-dimensional reconstruction.
It effectively reduces the computing power consumption in the target tracking and reconstruction process, improves the reconstruction speed and accuracy, reduces the occupation of the body's computing power resources, and reduces the application cost of the smart driving system.
Smart Images

Figure CN119027579A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of three-dimensional reconstruction, and in particular relates to a three-dimensional reconstruction method and system for aerial targets based on monocular vision. Background Art
[0002] Most current smart driving solutions in urban areas rely on high-precision electronic maps, but high-precision maps that are surveyed at high cost may not meet the application requirements of all scenarios due to factors such as map range and update frequency. Using the vehicle's existing sensors to continuously build and update maps during vehicle operation can make up for the shortcomings of high-precision maps in terms of both cost and computing power. Therefore, using monocular cameras and other non-visual sensors for mapping is a feasible technical solution.
[0003] At present, most monocular vision mapping solutions are based on feature point detection and optical flow tracking. These methods are robust and accurate (about decimeter-level accuracy outdoors), but they still consume a lot of computing power. Even with faster ORB feature points (generally depending on the image size and the area of interest, an average of 8K DMIPS can be achieved in actual applications), this cannot be effectively improved, which is unacceptable for the precious vehicle-side computing power. In the existing VSLAM-based monocular vision reconstruction solution, ORB feature points are still used for feature extraction, matching, and tracking. Once the number of reconstructed objects increases, the target proportion increases, or the image is large, the vehicle-side computing power will be greatly consumed. Summary of the invention
[0004] In view of this, an embodiment of the present invention provides a method and system for three-dimensional reconstruction of aerial targets based on monocular vision, which is used to solve the problem that the existing three-dimensional target reconstruction will greatly consume the computing power on the vehicle side.
[0005] In a first aspect of an embodiment of the present invention, a method for three-dimensional reconstruction of an aerial target based on monocular vision is provided, comprising:
[0006] The center point of the aerial target and the key points on the target contour are taken as feature points. After extracting the external contour of the aerial target in a single frame through OpenCV, the positions of the target center point and key points are calculated.
[0007] Based on the target center point and key point positions, the target pixel overlap between two adjacent frames of images is calculated. When the pixel overlap exceeds a preset threshold, the target is tracked between the two adjacent frames of images.
[0008] The pixels of the same type of targets in a single frame image are divided, and the target pixels whose overlap with the target pixels of the previous frame image exceeds a preset threshold are assigned the same number, and the target pixels whose overlap with the target pixels of the previous frame image does not exceed the preset threshold are assigned a new number;
[0009] The geometric feature points of targets with the same number are matched one by one. After correcting the monocular camera pose through Kalman filtering, the initial position of each target between adjacent frames is obtained based on triangulation, and the initial position of the target is optimized through a sliding window.
[0010] The aerial target is reconstructed in three dimensions based on the optimized target position.
[0011] In a second aspect of an embodiment of the present invention, a system for reconstructing aerial targets based on monocular vision is provided, comprising:
[0012] The feature point extraction module is used to take the center point of the aerial target and the key points on the target contour as feature points. After extracting the external contour of the aerial target in a single frame through OpenCV, the position of the target center point and key points is calculated;
[0013] The target tracking module is used to calculate the target pixel overlap between two adjacent frames of images based on the target center point and key point positions. When the pixel overlap exceeds a preset threshold, the target is tracked between the two adjacent frames of images.
[0014] The pixels of the same type of targets in a single frame image are divided, and the target pixels whose overlap with the target pixels of the previous frame image exceeds a preset threshold are assigned the same number, and the target pixels whose overlap with the target pixels of the previous frame image does not exceed the preset threshold are assigned a new number;
[0015] The position optimization module is used to make one-to-one correspondence between the geometric feature points of targets with the same number, correct the monocular camera posture through Kalman filtering, obtain the initial position of each target between adjacent frames based on triangulation, and optimize the initial position of the target through a sliding window;
[0016] The target reconstruction module is used to perform three-dimensional reconstruction of the aerial target based on the optimized target position.
[0017] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect of the embodiment of the present invention when executing the computer program.
[0018] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect of the embodiment of the present invention are implemented.
[0019] In an embodiment of the present invention, the geometric feature points of aerial targets are used to replace traditional ORB feature points to track and reconstruct the target. This can not only effectively reduce the computing power consumed in the target tracking and reconstruction process and reduce the amount of calculation for target tracking and matching, but also ensure the target reconstruction accuracy to a certain extent and improve the target reconstruction speed, thereby reducing the occupancy of the vehicle body computing resources and reducing the application cost of the intelligent driving system. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 A schematic flow chart of a method for three-dimensional reconstruction of an aerial target based on monocular vision provided by one embodiment of the present invention;
[0022] Figure 2 A schematic diagram of geometric feature points provided by an embodiment of the present invention;
[0023] Figure 3 A schematic diagram of the structure of a monocular vision-based aerial target 3D reconstruction system provided by one embodiment of the present invention;
[0024] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] It should be understood that the term "including" and other similar expressions in the specification or claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, such as a process, method, system, or device including a series of steps or units is not limited to the listed steps or units. In addition, "first" and "second" are used to distinguish different objects, not to describe a specific order.
[0027] It is understandable that with the improvement of vehicle perception, positioning, and planning and control capabilities, the requirements of intelligent driving systems for maps are getting lower and lower, mainly in terms of elements, scope, and accuracy. The reduction in element requirements means that the intelligent driving system no longer needs very complete traffic element information, only some key elements are needed; the reduction in scope requirements means that the system no longer needs to have centimeter-level high-precision maps available at all times for positioning; the reduction in accuracy requirements means that the system no longer needs centimeter-level accuracy, only decimeter-level or even meter-level accuracy for some elements. The position accuracy of aerial targets (such as signs and traffic lights) in intelligent driving is not critical to the intelligent driving system, and it is sufficient to be at the meter level. Therefore, there is no need to spend a lot of computing power to reconstruct aerial elements with high precision.
[0028] For vehicles equipped with a monocular forward-looking camera and a pose estimation system, the vehicle will synchronously obtain the position and pose estimation of the positioning system and the video stream of the monocular camera when the vehicle moves. Each frame of the video has a timestamp synchronized with the positioning. These images can obtain semantic segmentation results through perception. Given the internal and external parameters of the camera, it is only necessary to reconstruct the aerial targets based on the perception results, that is, to reconstruct the aerial signs, traffic lights, etc. However, the existing feature point extraction requires a large amount of computing power due to the use of ORB feature points or SI FT feature points. For example, for a 640X480 image, the extraction time of ORB feature points is about 3ms, plus feature point matching, it is about 8ms. In actual large-scale outdoor scenes, the image resolution is higher, and the time consumption of feature point extraction and matching may exceed 30ms, which will completely occupy an x86 core. If it is on an embedded platform, a single core cannot meet the real-time requirements. SI FT feature points consume more performance and are more difficult to meet real-time requirements.
[0029] In view of this, please refer to Figure 1 , a flow chart of a method for three-dimensional reconstruction of an aerial target based on monocular vision provided by an embodiment of the present invention includes:
[0030] S101, taking the center point of the aerial target and the key points on the target contour as feature points, extracting the circumscribed contour of the aerial target in a single frame through OpenCV, and calculating the positions of the target center point and the key points;
[0031] The aerial target may be a non-ground traffic element such as a traffic light or a sign. The center point of the target is the center point of the target outline, such as the center point of a circular sign or a rectangular traffic light.
[0032] The key points on the target contour are the corner points or center points on the edge lines of a rectangle, the corner points or center points on the edge lines of a triangle, and the equally divided points on the edge lines of a circle.
[0033] The key points on the target contour can be the corner points or edge center points of a rectangle, triangle, or polygon, or the equally divided points on a circle, such as Figure 2 As shown, the center point on the rectangular edge and the equally divided points on the circular edge can be used as key points. In this embodiment, directly replacing the ORB feature points with geometric feature points such as key points on the target contour and the target center point can significantly reduce the amount of calculation in the feature matching and target tracking process, and reduce the consumption of computing resources.
[0034] The target image captured by the monocular camera is processed through OpenCV, the circumscribed contour of the identified target is extracted, and the coordinate positions of the center point and key points of the aerial target in the image are calculated in the image coordinate system.
[0035] S102, based on the target center point and the key point position, calculating the target pixel overlap between two adjacent frames of images, and when the pixel overlap exceeds a preset threshold, performing target tracking on the two adjacent frames of images;
[0036] The pixels of the same type of targets in a single frame image are divided, and the target pixels whose overlap with the target pixels of the previous frame image exceeds a preset threshold are assigned the same number, and the target pixels whose overlap with the target pixels of the previous frame image does not exceed the preset threshold are assigned a new number;
[0037] Obtain the same type of targets in the previous and next frames, and based on the positions of the key points and center points in the targets, calculate the pixel overlap of the same type of targets in two adjacent frames, and determine whether the targets in the two frames are the same target based on the pixel overlap.
[0038] Specifically, the target pixel overlap is calculated according to the formula:
[0039]
[0040] In the formula, A and B represent the pixel sub-regions of two overlapping elements of the same type in the image coordinate system in the previous and next frame images, respectively. The numerator A∩B represents the overlapping pixel area between objects of the same type, and the denominator A∪B represents the union area of the two pixel regions. The obtained ROI is between 0 and 1.0.
[0041] In a short time interval, the camera's observation of the same object will not be very different. Considering factors such as the vehicle's speed, camera resolution, and target distance, the preset threshold can be determined through multiple tests. Preferably, the preset threshold is 0.7.
[0042] Among them, the projection pixels of the same type of targets in a single frame image are divided by the DBSCAN clustering algorithm.
[0043] When tracking targets, they are first grouped by type, that is, clustered within the group according to the distance reachable algorithm (DBSCAN). The projection pixels of multiple targets of the same type in one frame of image can be separated. The separated pixel areas are numbered, and the same number is set if the ROI value of the target in the subsequent image frames reaches the threshold. In the two adjacent frames of images, the same aerial targets are determined to be the same and set to the same number, and real-time tracking is performed. For targets whose overlap does not meet the requirements, a new number is set.
[0044] S103, one-to-one correspondence is performed for the geometric feature points of the targets with the same number, and after correcting the monocular camera posture through Kalman filtering, the initial position of each target between adjacent frames is obtained based on triangulation, and the initial position of the target is optimized through a sliding window;
[0045] After associating the geometric feature points (i.e., center points, key points) observed multiple times for the same target, the camera pose is corrected based on the observed target pose difference between the previous and next frames and the estimated vehicle pose (IMU), and the target position is obtained by triangulation. The triangulation is a process of calculating the absolute 3D coordinates of the feature point in the world coordinate system through the observation of the same point by multiple frames of cameras.
[0046] The obtained initial target position is optimized by a sliding window optimizer, and the window position is adjusted by optimization to obtain a more accurate target position after optimization.
[0047] S104: Perform three-dimensional reconstruction of the aerial target based on the optimized target position.
[0048] Based on the optimized target position, that is, the position of the target feature points, three-dimensional reconstruction can be directly performed in the high-precision map.
[0049] In this embodiment, replacing traditional ORB feature points with target geometric feature points can not only effectively reduce the amount of calculation in the target tracking and reconstruction process and reduce the computing power occupied by the vehicle side, but also ensure the accuracy and speed of target reconstruction, thereby reducing the hardware cost of vehicle-side target reconstruction and reducing the actual cost of intelligent driving.
[0050] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0051] Figure 3 A schematic diagram of the structure of a monocular vision-based aerial target 3D reconstruction system provided by an embodiment of the present invention, the system comprising:
[0052] The feature point extraction module 310 is used to take the center point of the aerial target and the key points on the target contour as feature points, extract the circumscribed contour of the aerial target in a single frame through OpenCV, and calculate the positions of the target center point and the key points;
[0053] The key points on the target contour are the corner points or center points on the edge lines of a rectangle, the corner points or center points on the edge lines of a triangle, and the equally divided points on the edge lines of a circle.
[0054] The target tracking module 320 is used to calculate the target pixel overlap between two adjacent frames of images based on the target center point and the key point position, and when the pixel overlap exceeds a preset threshold, the target is tracked between the two adjacent frames of images;
[0055] The pixels of the same type of targets in a single frame image are divided, and the target pixels whose overlap with the target pixels of the previous frame image exceeds a preset threshold are assigned the same number, and the target pixels whose overlap with the target pixels of the previous frame image does not exceed the preset threshold are assigned a new number;
[0056] Specifically, the calculation of the target pixel overlap of two adjacent frames of images includes:
[0057] Calculate the target pixel overlap according to the formula:
[0058]
[0059] Where A and B represent the pixel areas of the same type of aerial targets in the image coordinate system in the previous and next two adjacent frames of images, respectively.
[0060] Among them, the projection pixels of the same type of targets in a single frame image are divided by the DBSCAN clustering algorithm.
[0061] The position optimization module 330 is used to make a one-to-one correspondence between the geometric feature points of the targets with the same number, correct the monocular camera posture through Kalman filtering, obtain the initial position of each target between adjacent frames based on triangulation, and optimize the initial position of the target through a sliding window;
[0062] The target reconstruction module 340 is used to perform three-dimensional reconstruction of the aerial target based on the optimized target position.
[0063] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system and module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0064] Figure 4 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is used to realize three-dimensional reconstruction of aerial targets based on a monocular camera. Figure 4 As shown, the electronic device 4 of this embodiment includes: a memory 410, a processor 420 and a system bus 430, wherein the memory 410 includes an executable program 4101 stored thereon, and those skilled in the art can understand that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0065] Combine the following Figure 4 A detailed introduction to the various components of electronic equipment:
[0066] The memory 410 can be used to store software programs and modules, and the processor 420 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 410. The memory 410 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as cache data), etc. In addition, the memory 410 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0067] The memory 410 includes an executable program 4101 of the network request method, and the executable program 4101 can be divided into one or more modules / units, which are stored in the memory 410 and executed by the processor 420 to perform three-dimensional reconstruction of aerial targets, etc. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the executable program 4101 in the electronic device 4. For example, the executable program 4101 can be divided into functional modules such as a feature point extraction module, a target tracking module, a position optimization module, and a target reconstruction module.
[0068] The processor 420 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 410, and calling data stored in the memory 410, so as to monitor the overall status of the electronic device. Optionally, the processor 420 may include one or more processing units; preferably, the processor 420 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 420.
[0069] The system bus 430 is used to connect the various functional components inside the computer, and can transmit data information, address information, and control information. Its types can be, for example, PCI bus, ISA bus, CAN bus, etc. The instructions of the processor 420 are transmitted to the memory 410 through the bus, and the memory 410 feeds back data to the processor 420. The system bus 430 is responsible for the data and instruction exchange between the processor 420 and the memory 410. Of course, the system bus 430 can also be connected to other devices, such as network interfaces, display devices, etc.
[0070] In the embodiment of the present invention, the executable program executed by the processor 420 included in the electronic device includes:
[0071] The center point of the aerial target and the key points on the target contour are taken as feature points. After extracting the external contour of the aerial target in a single frame through OpenCV, the positions of the target center point and key points are calculated.
[0072] Based on the target center point and key point positions, the target pixel overlap between two adjacent frames of images is calculated. When the pixel overlap exceeds a preset threshold, the target is tracked between the two adjacent frames of images.
[0073] The pixels of the same type of targets in a single frame image are divided, and the target pixels whose overlap with the target pixels of the previous frame image exceeds a preset threshold are assigned the same number, and the target pixels whose overlap with the target pixels of the previous frame image does not exceed the preset threshold are assigned a new number;
[0074] The geometric feature points of targets with the same number are matched one by one. After correcting the monocular camera pose through Kalman filtering, the initial position of each target between adjacent frames is obtained based on triangulation, and the initial position of the target is optimized through a sliding window.
[0075] The aerial target is reconstructed in three dimensions based on the optimized target position.
[0076] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0077] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0078] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for three-dimensional reconstruction of aerial targets based on monocular vision, characterized in that: include: The center point of the aerial target and the key points on the target contour are taken as feature points. After extracting the external contour of the aerial target in a single frame through OpenCV, the positions of the target center point and key points are calculated. Based on the target center point and key point positions, the target pixel overlap between two adjacent frames of images is calculated. When the pixel overlap exceeds a preset threshold, the target is tracked between the two adjacent frames of images. The pixels of the same type of targets in a single frame image are divided, and the target pixels whose overlap with the target pixels of the previous frame image exceeds a preset threshold are assigned the same number, and the target pixels whose overlap with the target pixels of the previous frame image does not exceed the preset threshold are assigned a new number; The geometric feature points of targets with the same number are matched one by one. After correcting the monocular camera pose through Kalman filtering, the initial position of each target between adjacent frames is obtained based on triangulation, and the initial position of the target is optimized through a sliding window. The aerial target is reconstructed in three dimensions based on the optimized target position.
2. The method according to claim 1, characterized in that The key points on the target contour are the corner points or center points on the edge lines of a rectangle, the corner points or center points on the edge lines of a triangle, and the equally divided points on the edge lines of a circle.
3. The method according to claim 1, characterized in that The calculation of the target pixel overlap between two adjacent frames of images comprises: Calculate the target pixel overlap according to the formula: Where A and B represent the pixel areas of the same type of aerial targets in the image coordinate system in the previous and next two adjacent frames of images, respectively.
4. The method according to claim 1, characterized in that: The dividing of pixels of the same type of objects in a single frame image comprises: The projected pixels of the same type of targets in a single frame image are divided using the DBSCAN clustering algorithm.
5. A three-dimensional reconstruction system of aerial targets based on monocular vision, characterized in that: include: The feature point extraction module is used to take the center point of the aerial target and the key points on the target contour as feature points. After extracting the external contour of the aerial target in a single frame through OpenCV, the position of the target center point and key points is calculated; The target tracking module is used to calculate the target pixel overlap between two adjacent frames of images based on the target center point and key point positions. When the pixel overlap exceeds a preset threshold, the target is tracked between the two adjacent frames of images. The pixels of the same type of targets in a single frame image are divided, and the target pixels whose overlap with the target pixels of the previous frame image exceeds a preset threshold are assigned the same number, and the target pixels whose overlap with the target pixels of the previous frame image does not exceed the preset threshold are assigned a new number; The position optimization module is used to make one-to-one correspondence between the geometric feature points of targets with the same number, correct the monocular camera posture through Kalman filtering, obtain the initial position of each target between adjacent frames based on triangulation, and optimize the initial position of the target through a sliding window; The target reconstruction module is used to perform three-dimensional reconstruction of the aerial target based on the optimized target position.
6. The system according to claim 5, characterized in that The key points on the target contour are the corner points or center points on the edge lines of a rectangle, the corner points or center points on the edge lines of a triangle, and the equally divided points on the edge lines of a circle.
7. The system according to claim 5, characterized in that The calculation of the target pixel overlap between two adjacent frames of images comprises: Calculate the target pixel overlap according to the formula: Where A and B represent the pixel areas of the same type of aerial targets in the image coordinate system in the previous and next two adjacent frames of images, respectively.
8. The system according to claim 5, characterized in that The dividing of pixels of the same type of objects in a single frame image comprises: The projected pixels of the same type of targets in a single frame image are divided using the DBSCAN clustering algorithm.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a method for three-dimensional reconstruction of aerial targets based on monocular vision as described in any one of claims 1 to 4 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps of a method for three-dimensional reconstruction of aerial targets based on monocular vision as described in any one of claims 1 to 4 are implemented.
