Long and short focus binocular vision radar target fusion method and device and storage medium

Abstract

The invention discloses a long-focus and short-focus binocular vision radar target fusion method, which is characterized in that feature extraction is carried out in a long-focus boundary area range, matching with feature points extracted by a short focus is carried out, a conversion relation is automatically obtained, and meanwhile, the calculated amount is saved, and feature special area extraction and matching do not need to be carried out in a long-focus total image range. Similarly, since the target in the long-focus center area is certainly more reliable than the short-focus corresponding area, the target is only matched with the short-focus target in the long-focus boundary area, the number of the targets needing iou matching is reduced, the fusion effect is improved, and meanwhile computing resources are saved.

Description

Technical Field

[0001] The invention relates to a long- and short-focus binocular vision radar target fusion method, equipment and storage medium. Background Art

[0002] With the development of science and technology, various sensor technologies continue to emerge, such as millimeter wave radar, lidar, camera, ultrasonic sensor, etc. Each sensor has its advantages and limitations, and the data types, accuracy, update frequency and noise characteristics they collect are different. As the complexity of the tasks to be performed gradually increases, the data of a single sensor may not be sufficient to meet the needs of the system, especially in complex environments. Multi-sensor fusion technology combines the advantages of different sensors by combining data from different sensors to improve the overall performance and accuracy of the system. Multi-sensor fusion is of great significance in many fields, including autonomous driving, autonomous navigation of mobile robots, military, medical and other fields. Therefore, how to fuse the collected information of multiple sensors has become a technical problem that needs to be solved urgently.

[0003] The patent with the publication number CN116188770A discloses a method for fusion of long-focus and short-focus cameras. The similarity of long-focus and short-focus images is calculated using the features of SSIM and grayscale histogram to obtain the regions of interest of long-focus and short-focus targets, and then determine whether the long-focus target is within the region of interest. If it is within the region, it is switched to short focus and matched with the short-focus target. However, the features of the image are easily affected by factors that affect image quality, such as lighting and weather, thus affecting the generation of the region of interest. If the generation is inaccurate, missed matches and mismatches may occur.

[0004] The patent with publication number CN114089329A discloses a target detection method based on the fusion of long- and short-focus cameras and millimeter-wave radars. The long- and short-focus images are calibrated in advance, and the long-focus image is converted to the short-focus image for IOU matching and fusion. However, the proposal does not take into account that if the target overlap on the same camera is relatively high, even if it is converted to another camera, even different targets will have a high IOU, resulting in mismatching. And only the depth information of the radar is used, and the radar target and the visual target are not fused.

[0005] The patent with publication number CN117523514A discloses a radar visual fusion data target detection method and system based on cross-attention. The patent obtains visual images and point cloud feature maps to form a sample data set, and divides the sample data set into a training set, a validation set, and a test set in a certain proportion; constructs a target detection model based on the cross-attention mechanism; sets the optimizer, learning rate strategy, and loss function to obtain a trained target detection model for fusion. This method requires synchronization and integration of different types of sensor data, which increases the complexity of data processing, and data loss or information redundancy may occur during the fusion process, which will affect the final fusion effect. Summary of the invention

[0006] The main purpose of the present invention is to provide a long- and short-focus binocular vision radar target fusion method, device and storage medium, aiming to solve the above technical problems.

[0007] To achieve the above object, the present invention provides a long- and short-focus binocular vision radar target fusion method.

[0008] The long- and short-focus binocular vision radar target fusion method comprises the following steps: Acquire long-focus image and short-focus image; Delineate reliable areas of telephoto images; Detect feature points outside the reliable area in the telephoto image, match them with the feature points detected in the short-focus image to obtain a matching point set, and obtain a conversion relationship through the matching point set; According to the conversion relationship, the target at the boundary of the reliable area of ​​the long-focus image is converted into the short-focus image for image fusion.

[0009] In one embodiment, after the step of converting the target at the boundary of the reliable area of ​​the long-focus image into the short-focus image for image fusion according to the conversion relationship, the long-focus and short-focus binocular vision radar target fusion method further includes: The target in the reliable area of ​​the long-focus image is directly replaced with the target in the corresponding short-focus image area.

[0010] In one embodiment, the step of detecting feature points outside the reliable area in the telephoto image and matching them with feature points detected in the short-focus image to obtain a matching point set, and deriving a conversion relationship through the matching point set includes: The homography transformation matrix of the image is estimated and fitted by the least squares method, and the conversion relationship between the long-focus image and the short-focus image is obtained.

[0011] In one embodiment, before the step of defining a reliable area of ​​a long-focus image, the long-focus and short-focus binocular vision radar target fusion method includes: By judging the overlap between the two targets in the long-focus image and the short-focus image, it is determined whether the two targets are blocked. If so, the radar point cloud data is combined and the radar point cloud is converted into the corresponding image target frame through the conversion relationship to obtain the depth information of the target.

[0012] In one embodiment, after the step of converting the radar point cloud into the corresponding image target frame through the conversion relationship to obtain the depth information of the target, the long- and short-focus binocular vision radar target fusion method includes: Project the radar target onto the short focus image to obtain the radar target position frame on the short focus image; If the projected radar target position frame does not overlap with the position frames of all visual targets, the radar target is added to the fusion target; If the converted radar target position frame overlaps with any camera target position frame, the radar target is filtered out.

[0013] In one embodiment, the step of determining whether two targets are blocked by determining the overlap between the two targets in the long-focus image and the short-focus image is specifically as follows: The judgment is made by calculating the overlap of two targets. If the overlap of two targets is greater than a preset threshold, the two targets are determined to be in an occlusion situation.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a long-short focus binocular vision radar target fusion device, which includes: a memory, a processor, and a long-short focus binocular vision radar target fusion program stored in the memory and executable on the processor. When the long-short focus binocular vision radar target fusion program is executed by the processor, the steps of the long-short focus binocular vision radar target fusion method as described above are implemented.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a long- and short-focus binocular vision radar target fusion program is stored. When the long- and short-focus binocular vision radar target fusion program is executed by a processor, the steps of the long- and short-focus binocular vision radar target fusion method as described above are implemented.

[0016] Beneficial effects that can be achieved by the present invention: A long- and short-focus binocular vision radar target fusion method proposed in an embodiment of the present invention automatically obtains the conversion relationship by extracting features within the long-focus boundary area and matching them with the feature points extracted with the short-focus. This saves the amount of calculation because it is not necessary to extract and match the feature special area within the full long-focus image. Similarly, since the target in the long-focus center area must be more reliable than the short-focus corresponding area, it is only matched with the short-focus target in the long-focus boundary area, reducing the number of targets that need to be iou matched, improving the fusion effect, and saving computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the structure of a device in a hardware operating environment involved in an embodiment of the present invention; Figure 2 It is a schematic flow chart of the target fusion method of the long- and short-focus binocular vision radar of the present invention; Figure 3 It is a comparison diagram of the long-focus image and the short-focus image of the present invention.

[0018] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0020] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0021] The terminal of the embodiment of the present invention can be a PC, or it can be a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a portable computer, or other portable terminal devices with display function.

[0022] like Figure 1As shown, the terminal may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0023] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal can also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0024] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0025] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a long- and short-focus binocular vision radar target fusion program.

[0026] exist Figure 1In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and perform data communication with the client; and the processor 1001 can be used to call the long-short focal length binocular vision radar target fusion program stored in the memory 1005, and perform the following operations: Acquire long-focus image and short-focus image; Delineate reliable areas of telephoto images; Detect feature points outside the reliable area in the telephoto image, match them with the feature points detected in the short-focus image to obtain a matching point set, and obtain a conversion relationship through the matching point set; According to the conversion relationship, the target at the boundary of the reliable area of ​​the long-focus image is converted into the short-focus image for image fusion.

[0027] Further, the processor 1001 may call the long- and short-focus binocular vision radar target fusion program stored in the memory 1005, and further perform the following operations: The target in the reliable area of ​​the long-focus image is directly replaced with the target in the corresponding short-focus image area.

[0028] Further, the processor 1001 may call the long- and short-focus binocular vision radar target fusion program stored in the memory 1005, and further perform the following operations: The homography transformation matrix of the image is estimated and fitted by the least squares method, and the conversion relationship between the long-focus image and the short-focus image is obtained.

[0029] Further, the processor 1001 may call the long- and short-focus binocular vision radar target fusion program stored in the memory 1005, and further perform the following operations: By judging the overlap between the two targets in the long-focus image and the short-focus image, it is determined whether the two targets are blocked. If so, the radar point cloud data is combined and the radar point cloud is converted into the corresponding image target frame through the conversion relationship to obtain the depth information of the target.

[0030] Further, the processor 1001 may call the long- and short-focus binocular vision radar target fusion program stored in the memory 1005, and further perform the following operations: Project the radar target onto the short focus image to obtain the radar target position frame on the short focus image; If the projected radar target position frame does not overlap with the position frames of all visual targets, the radar target is added to the fusion target; If the converted radar target position frame overlaps with any camera target position frame, the radar target is filtered out.

[0031] Further, the processor 1001 may call the long- and short-focus binocular vision radar target fusion program stored in the memory 1005, and further perform the following operations: The overlap between two targets is calculated. If the overlap between two targets is greater than a preset threshold, the two targets are considered to be blocked. The specific embodiments of the data storage device of the present invention are basically the same as the embodiments of the data storage method described below, and will not be described in detail here.

[0032] Reference Figure 2 The present invention provides a long- and short-focus binocular vision radar target fusion method, the long- and short-focus binocular vision radar target fusion method comprising: Acquire long-focus image and short-focus image; Delineate reliable areas of telephoto images; Detect feature points outside the reliable area in the telephoto image, match them with the feature points detected in the short-focus image to obtain a matching point set, and obtain a conversion relationship through the matching point set; According to the conversion relationship, the target at the boundary of the reliable area of ​​the long-focus image is converted into the short-focus image for image fusion.

[0033] First, a reliable area of ​​the telephoto image is delineated, usually the central area of ​​the telephoto image. Feature points outside the central area of ​​the telephoto image are detected and then matched with the feature points detected on the short-focus image to obtain a series of matching point sets. These points are used to estimate the conversion relationship between the long-focus and short-focus images. Compared with the existing fusion method, this method can automatically obtain the conversion relationship between the long-focus and short-focus images without the need for prior manual calibration, and does not require feature point extraction and matching in the central area of ​​the telephoto image, which reduces a lot of calculations.

[0034] Then, target detection is performed on the two images, and the targets outside the telephoto center area, that is, the targets at the boundary, are projected onto the short focus according to the conversion relationship for matching and fusion. The targets in the telephoto center area can directly replace the targets in the corresponding short focus area. Compared with the existing fusion method, a more reliable fusion target can be obtained, and the number of targets matched by IOU is reduced, saving the amount of calculation.

[0035] For the occluded targets on the same road, the corresponding radar point cloud depth information and its own pixel coordinates are used to convert it into 3D space, and matching fusion is performed through the point set matching method. Compared with the existing fusion method, it can well avoid the IOU matching error problem caused by occlusion, and after obtaining the fused visual target, the point cloud data is clustered to obtain the radar target, and the clustered radar target is projected onto the short-focus image and fused with the visual target. The flowchart of this application is as follows Figure 2 shown.

[0036] This application first calculates the central area and boundary area of ​​the telephoto image according to the size of the telephoto image, such as Figure 3 The left image below shows the image taken by the short-focus camera, and the right image shows the image taken by the long-focus camera. The red box on the left shows the range of the long-focus image mapped to the corresponding range on the short-focus, and the green box shows the range of the center area of ​​the long-focus range mapped to the corresponding range on the short-focus.

[0037] For the central area of ​​the telephoto image, no feature extraction and matching are performed, and target matching and fusion will not be performed in this area later. This application only extracts feature points in the telephoto boundary area and matches them with feature points on the short-focus image. The feature point calculation method can use orb features or other features, and then use brute force search to match feature points. After obtaining a series of feature point matching point sets, the homography transformation matrix of the image can be estimated and fitted by the least squares method to obtain the conversion relationship between the long and short focus images.

[0038] After obtaining the conversion relationship between the long-focus and short-focus images, the target at the long-focus boundary can be converted to the short-focus. Because the target at the long-focus boundary may be incomplete, while the target at the center is complete and must be more reliable than the target in the corresponding area of ​​the short-focus, the present application only converts the target at the long-focus boundary to the short-focus for iou matching, thereby reducing the number of targets that need to be iou matched.

[0039] For target matching at the boundary, this application compensates for the inaccuracy of IOU matching on the image by matching the target point set in 3D space. If on the telephoto image, the previous target blocks the next target, and the position frames of the two targets overlap very highly, after the telephoto target is projected onto the short-focus image, due to the influence of the conversion error, the position frames of the two telephoto targets projected onto the short-focus image may be very close to the positions of the two corresponding targets on the short-focus image, and it is easy to have mismatching, resulting in problems with the fusion of the long-focus and short-focus targets. Therefore, this application first determines whether two targets on the same image are occluded. The judgment can be made by calculating the iou values ​​of the two targets. If the iou thresholds of the two targets are greater than a certain threshold, occlusion occurs between the two targets.

[0040] After determining the occluded target, combined with the radar point cloud data, the radar point cloud is converted into the corresponding image target frame through the pre-calibrated conversion relationship to obtain the depth information of the target. With the pixel position and depth information, the target frame can be projected into the 3D world. Although the positions of the targets on the image are overlapping, there will be a certain gap in the positions in the 3D space. Then, the matching relationship between the occluded targets is obtained through point set matching to improve the accuracy of target fusion. Since the long focus will definitely be clearer than the short focus, for the target in the center of the long focus, the target in the corresponding area of ​​the short focus can be directly replaced.

[0041] After obtaining the fused visual target, in order to obtain a more complete fused target, the radar point cloud is clustered and extracted to remove the radar target. Since the position frame of the visual target in this application is based on the short-focus image coordinate system, the radar target is projected onto the short focus to obtain the radar target position frame on the short-focus image. If the projected radar target position frame does not overlap with the position frames of all visual targets, and the clustered target confidence is relatively high, the radar target is added to the fused target. If the converted radar target position frame overlaps with any camera target position frame, the radar target is filtered out. Finally, the fusion of radar and visual targets is completed to obtain a complete fused target.

[0042] In combination with the above embodiments, the present application proposes a long-short focal binocular vision radar target fusion method, which performs feature extraction within the long focal boundary area, matches the feature points extracted with the short focal, and automatically obtains the conversion relationship while saving the amount of calculation, because it is not necessary to extract and match the feature special area within the long focal full image. Similarly, since the target in the long focal center area must be more reliable than the short focal corresponding area, the short focal target is only matched within the long focal boundary area, reducing the number of targets that need to be matched with IOU, improving the fusion effect, and saving computing resources. For the occluded target, due to the camera perspective relationship, two targets will overlap, resulting in the position frames of these overlapping targets being very close after projection, and it is easy to match errors. The present invention restores the position center of the overlapping target in the real world through the depth information of the radar point cloud, and solves the problem of overlapping target matching errors on the image by using the point set matching method. In order to obtain a more complete fusion result, the radar point cloud is clustered to obtain the radar target, projected onto the short focal image, and it is determined whether there is an intersection with the visual target. If there is no intersection and the confidence of the radar target is relatively high, it is added to the fusion target.

[0043] In summary, the present invention combines image IOU matching and 3D space point set matching to perform fusion, utilizes radar point cloud depth information, projects overlapping targets occluded in the image into 3D space to obtain the position center, and performs matching between point sets to make up for the shortcomings of image IOU matching.

[0044] The present invention matches feature points and boundary targets by dividing regions, sets a long-focus center region and a boundary region, directly replaces the short-focus target in the center region, and matches and fuses with the short-focus target in the boundary region.

[0045] The present invention projects the radar target onto the short-focus image, determines whether it overlaps with the visual target position frame, decides whether to add the radar target, and fuses it with the visual target to obtain a more complete fusion result.

[0046] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0047] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0048] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0049] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A long- and short-focus binocular vision radar target fusion method, characterized in that: The long- and short-focus binocular vision radar target fusion method comprises the following steps: Acquire long-focus image and short-focus image; Delineate reliable areas of telephoto images; Detect feature points outside the reliable area in the telephoto image, match them with the feature points detected in the short-focus image to obtain a matching point set, and obtain a conversion relationship through the matching point set; According to the conversion relationship, the target at the boundary of the reliable area of ​​the long-focus image is converted into the short-focus image for image fusion.

2. The long- and short-focus binocular vision radar target fusion method as claimed in claim 1, characterized in that: After the step of converting the target at the boundary of the long-focus image reliable area into the short-focus image for image fusion according to the conversion relationship, the long-focus and short-focus binocular vision radar target fusion method further includes: The target in the reliable area of ​​the long-focus image is directly replaced with the target in the corresponding short-focus image area.

3. The long- and short-focus binocular vision radar target fusion method as claimed in claim 1, characterized in that: The step of detecting feature points outside the reliable area in the long-focus image and matching them with feature points detected in the short-focus image to obtain a matching point set, and obtaining a conversion relationship through the matching point set includes: The homography transformation matrix of the image is estimated and fitted by the least squares method, and the conversion relationship between the long-focus image and the short-focus image is obtained.

4. The long- and short-focus binocular vision radar target fusion method according to claim 1, characterized in that: Before the step of defining a reliable area of ​​the telephoto image, the method for fusion of long- and short-focus binocular vision radar targets includes: By judging the overlap between the two targets in the long-focus image and the short-focus image, it is determined whether the two targets are blocked. If so, the radar point cloud data is combined and the radar point cloud is converted into the corresponding image target frame through the conversion relationship to obtain the depth information of the target.

5. The long- and short-focus binocular vision radar target fusion method according to claim 4, characterized in that: After the step of converting the radar point cloud into the corresponding image target frame through the conversion relationship to obtain the depth information of the target, the long- and short-focus binocular vision radar target fusion method includes: Project the radar target onto the short focus image to obtain the radar target position frame on the short focus image; If the projected radar target position frame does not overlap with the position frames of all visual targets, the radar target is added to the fusion target; If the converted radar target position frame overlaps with any camera target position frame, the radar target is filtered out.

6. The long- and short-focus binocular vision radar target fusion method according to claim 4, characterized in that: The step of determining whether the two targets are blocked by determining the overlap between the two targets in the long-focus image and the short-focus image is specifically as follows: The judgment is made by calculating the overlap of two targets. If the overlap of two targets is greater than a preset threshold, the two targets are determined to be in an occlusion situation.

7. A long- and short-focus binocular vision radar target fusion device, characterized in that: The long- and short-focus binocular vision radar target fusion device includes: a memory, a processor, and a long- and short-focus binocular vision radar target fusion program stored in the memory and executable on the processor. When the long- and short-focus binocular vision radar target fusion program is executed by the processor, the steps of the long- and short-focus binocular vision radar target fusion method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a long- and short-focus binocular vision radar target fusion program, and when the long- and short-focus binocular vision radar target fusion program is executed by the processor, the steps of the long- and short-focus binocular vision radar target fusion method according to any one of claims 1 to 6 are implemented.

Images