RabeX-ray Fusion Method, System, Device and Media Based on Dual-Spectrum Field Alignment

Abstract

This invention relates to the field of road traffic technology, specifically disclosing a radar-video fusion method, system, device, and medium based on dual-spectral field-of-view alignment. The method includes: calibrating the latitude and longitude of visible light data from a first region acquired by a visible light camera after field-of-view alignment and thermal imaging data from a first region acquired by a thermal imaging camera after field-of-view alignment, obtaining a first calibration conversion relationship; calibrating the radar data from a second region acquired by a millimeter-wave radar with the visible light data from the first region, obtaining a second calibration conversion relationship; and fusing the target visible light data, target thermal imaging data, and target radar data of a target region within a preset time period based on the first and second calibration conversion relationships, obtaining radar-video fused data. This invention improves the accuracy of radar data and video data fusion through dual-spectral field-of-view alignment, providing high-confidence data assurance for machine vision inspection.

Description

Technical Field

[0001] This invention relates to the field of road traffic technology, and in particular to a radar-visual fusion method, system, device and medium based on dual-spectral field-of-view alignment. Background Technology

[0002] Currently, the varying widths and heights of targets in dual-spectrum imaging lead to accuracy deviations. Furthermore, in multi-target scenarios, the inconsistent overall field of view results in varying numbers of targets within the field of view, causing discrepancies in target data and introducing uncertainty into subsequent processing. In addition, dual-spectrum calibration is mostly performed manually, which introduces a degree of subjectivity, making it impossible to guarantee the accuracy of radar data and video data fusion.

[0003] Therefore, there is an urgent need to provide a technical solution to address the above problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method, system, device, and medium for radar-visual fusion based on dual-spectral field-of-view alignment.

[0005] In a first aspect, the present invention provides a radar-visual fusion method based on dual-spectral field-of-view alignment, the technical solution of which is as follows:

[0006] The visible light data of the first region acquired by the visible light camera after field of view alignment and the thermal imaging data of the first region acquired by the thermal imaging camera after field of view alignment are calibrated in latitude and longitude to obtain the first calibration conversion relationship;

[0007] The radar data of the second region acquired by the millimeter-wave radar and the visible light data of the first region acquired by the visible light camera after the field of view are aligned are calibrated to obtain a second calibration conversion relationship;

[0008] Based on the first calibration conversion relationship and the second calibration conversion relationship, the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period are fused to obtain the radar-visual fusion data of the target area within the preset time period; wherein, the target area is the common area of ​​the first area and the second area.

[0009] The beneficial effects of the radar-visual fusion method based on dual-spectral field-of-view alignment of the present invention are as follows:

[0010] The method of this invention improves the accuracy of radar data and video data fusion by using dual-spectral field-of-view alignment, providing high-confidence data assurance for machine vision inspection.

[0011] Based on the above scheme, the radar-visual fusion method based on dual-spectral field alignment of the present invention can be further improved as follows.

[0012] In one alternative approach, it also includes:

[0013] A visible light camera is used to acquire an original visible light image containing the first region, and a thermal imaging camera is used to acquire an original thermal imaging image containing the first region.

[0014] The original visible light image and the original thermal imaging image are respectively processed to be semi-transparent to obtain the target visible light image and the target thermal imaging image;

[0015] Based on the target visible light image and the target thermal imaging image, the field of view of the visible light camera and the thermal imaging camera are aligned.

[0016] In one alternative approach, it also includes:

[0017] The second region is determined within the coverage area based on RTK information in the coverage area of ​​the millimeter-wave radar.

[0018] In one alternative approach, it also includes:

[0019] Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein, the target area is the lane line area.

[0020] Secondly, the present invention provides a radar-visual fusion system based on dual-spectral field-of-view alignment, the technical solution of which is as follows:

[0021] Includes: a first calibration module, a second calibration module, and a radar-visual fusion module;

[0022] The first calibration module is used to: perform latitude and longitude calibration on the visible light data of the first region acquired by the visible light camera after field of view alignment and the thermal imaging data of the first region acquired by the thermal imaging camera after field of view alignment, so as to obtain the first calibration conversion relationship;

[0023] The second calibration module is used to: calibrate the radar data of the second region acquired by the millimeter-wave radar and the visible light data of the first region acquired by the visible light camera after the field of view is aligned, so as to obtain a second calibration conversion relationship;

[0024] The radar-visual fusion module is used to: fuse the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period based on the first calibration conversion relationship and the second calibration conversion relationship, to obtain the radar-visual fusion data of the target area within the preset time period; wherein, the target area is: the common area of ​​the first area and the second area.

[0025] The beneficial effects of the radar-visual fusion system based on dual-spectral field-of-view alignment of the present invention are as follows:

[0026] The system of this invention improves the accuracy of radar data and video data fusion by using dual-spectral field-of-view alignment, providing high-confidence data assurance for machine vision inspection.

[0027] Based on the above scheme, the radar-visual fusion system based on dual-spectral field alignment of the present invention can be further improved as follows.

[0028] In one alternative embodiment, the module further includes a field-of-view alignment module; the field-of-view alignment module is used for:

[0029] Using the visible light camera, a raw visible light image containing the first region is acquired; and using the thermal imaging camera, a raw thermal imaging image containing the first region is acquired.

[0030] The original visible light image and the original thermal imaging image are respectively processed to be semi-transparent to obtain the target visible light image and the target thermal imaging image;

[0031] Based on the target visible light image and the target thermal imaging image, the field of view of the visible light camera and the thermal imaging camera are aligned.

[0032] In an alternative embodiment, the method further includes: a determining module; the determining module is used to:

[0033] The second region is determined within the coverage area based on RTK information in the coverage area of ​​the millimeter-wave radar.

[0034] In an alternative embodiment, it further includes: a monitoring module; the monitoring module is used for:

[0035] Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein, the target area is the lane line area.

[0036] Thirdly, the technical solution of an electronic device according to the present invention is as follows:

[0037] It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the radar-visual fusion method based on dual-spectral field alignment as described in this invention.

[0038] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows:

[0039] The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the radar-visual fusion method based on dual-spectral field alignment of the present invention.

[0040] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0041] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 This is a flowchart illustrating an embodiment of a radar-visual fusion method based on dual-spectral field-of-view alignment according to the present invention.

[0043] Figure 2 This is a schematic diagram of a semi-transparent process;

[0044] Figure 3 This is a schematic diagram of radar calibration.

[0045] Figure 4 This is a schematic diagram of an embodiment of a radar-visual fusion system based on dual-spectral field-of-view alignment according to the present invention.

[0046] Figure 5 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0047] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0048] Figure 1This diagram illustrates a flowchart of an embodiment of a radar-visual fusion method based on dual-spectral field-of-view alignment provided by the present invention. This method can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the radar-visual fusion method based on dual-spectral field-of-view alignment by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps:

[0049] S1. The visible light data of the first region acquired by the visible light camera after field of view alignment and the thermal imaging data of the first region acquired by the thermal imaging camera after field of view alignment are calibrated in latitude and longitude to obtain the first calibration conversion relationship.

[0050] The first region is the area captured (acquired) by the visible light camera and the thermal imaging camera. In road traffic, the first region can be a one-way lane area on a section of a highway, or it can be adjusted according to the actual situation; there are no restrictions here. Visible light data includes multiple targets in the visible light image, and thermal imaging data includes multiple targets in the thermal imaging image, such as lane lines, vehicles, trees, etc. The first calibration conversion relationship is the calibration (latitude and longitude) conversion relationship between the visible light image and the thermal imaging image regarding the same target object. The first calibration conversion relationship is used to match and fuse the visible light image and the thermal imaging image.

[0051] S2. The radar data of the second region acquired by the millimeter-wave radar and the visible light data of the first region acquired by the visible light camera after the field of view is aligned are calibrated to obtain a second calibration conversion relationship.

[0052] The second region is the imaging (acquisition) area determined by the millimeter-wave radar. The second region is of the same type as the first region, and the size of the area it encompasses can be the same or different; no restrictions are set here. The second calibration conversion relationship is the calibration conversion relationship between radar data and visible light images regarding the same target object. This second calibration conversion relationship is used to match and fuse radar data and visible light images.

[0053] S3. Based on the first calibration conversion relationship and the second calibration conversion relationship, the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period are fused to obtain the radar-visual fusion data of the target area within the preset time period.

[0054] The target area is the shared area of ​​the first and second areas. The preset time period can be set according to actual conditions, such as 5s, 10s, etc. The target visible light data, target thermal imaging data, and target radar data are the visible light data, thermal imaging data, and radar data within the target area. The radar-visual fusion data is the data obtained by associating and fusing the target visible light data, target thermal imaging data, and target radar data based on the first calibration conversion relationship and the second calibration conversion relationship.

[0055] It should be noted that in this embodiment, the visible light camera, thermal imaging camera, and millimeter-wave radar are integrated into one device.

[0056] In one alternative approach, it also includes:

[0057] A visible light camera is used to acquire a raw visible light image containing the first region, and a thermal imaging camera is used to acquire a raw thermal imaging image containing the first region.

[0058] Among them, the original visible light image is a visible light image directly acquired using a visible light camera, and the original thermal image is a thermal image directly acquired using a thermal imaging camera.

[0059] The original visible light image and the original thermal imaging image are processed to be semi-transparent to obtain the target visible light image and the target thermal imaging image.

[0060] The visible light image and thermal image of the target are semi-transparent images. The semi-transparency processing method is to set the transparency of the original image to a preset value, which is 50% by default, but can be adjusted according to actual needs.

[0061] Based on the target visible light image and the target thermal imaging image, the field of view of the visible light camera and the thermal imaging camera are aligned.

[0062] Among them, such as Figure 2 As shown, by superimposing the semi-transparent images, the field of view alignment can be finely adjusted. The alignment process can be achieved by adjusting the up, down, left, and right directions and zooming to make the fields of view of the two-light images (visible light images and thermal imaging images) completely consistent.

[0063] In one alternative approach, it also includes:

[0064] The second region is determined within the coverage area based on RTK information in the coverage area of ​​the millimeter-wave radar.

[0065] Among them, such as Figure 3 As shown, multiple RTK information points exist within the coverage area of ​​the millimeter-wave radar. Figure 3 It contains four RTK information items. Based on the RTK information present in the coverage area, the second area is defined, namely: the lane line area defined by the millimeter-wave radar.

[0066] In one alternative approach, it also includes:

[0067] Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained.

[0068] The target area is the lane line area. Event monitoring information includes, but is not limited to: traffic congestion, vehicles driving in the wrong direction, parking, littering, vehicles leaving the area, pedestrian crossing, high temperature targets, lane icing, fog, and low visibility.

[0069] Taking vehicle reversing as an example, multiple frames of radar-visual fusion data of the target area within a preset time period are acquired. Machine vision algorithms are then used to detect vehicle trajectory changes within the target area from these frames, and the presence of a vehicle reversing is determined based on these trajectories. For instance, if the correct driving direction in the target area is from bottom to top, but the vehicle trajectory changes from top to bottom, a vehicle reversing event is identified, and an event report is generated.

[0070] The technical solution in this embodiment improves the accuracy of radar data and video data fusion by using dual-spectral field-of-view alignment, providing high-confidence data assurance for machine vision inspection.

[0071] Figure 4 A schematic diagram of an embodiment of a radar-visual fusion system 200 based on dual-spectral field-of-view alignment provided by the present invention is shown. Figure 4 As shown, the system 200 includes: a first calibration module 210, a second calibration module 220, and a radar-visual fusion module 230;

[0072] The first calibration module 210 is used to: perform latitude and longitude calibration on the visible light data of the first region acquired by the visible light camera after field of view alignment and the thermal imaging data of the first region acquired by the thermal imaging camera after field of view alignment, so as to obtain a first calibration conversion relationship;

[0073] The second calibration module 220 is used to: calibrate the radar data of the second region acquired by the millimeter-wave radar and the visible light data of the first region acquired by the visible light camera after the field of view is aligned, and obtain a second calibration conversion relationship;

[0074] The radar-visual fusion module 230 is used to: fuse the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period based on the first calibration conversion relationship and the second calibration conversion relationship, to obtain the radar-visual fusion data of the target area within the preset time period; wherein, the target area is: the common area of ​​the first area and the second area.

[0075] In one alternative embodiment, the module further includes a field-of-view alignment module; the field-of-view alignment module is used for:

[0076] Using the visible light camera, a raw visible light image containing the first region is acquired; and using the thermal imaging camera, a raw thermal imaging image containing the first region is acquired.

[0077] The original visible light image and the original thermal imaging image are respectively processed to be semi-transparent to obtain the target visible light image and the target thermal imaging image;

[0078] Based on the target visible light image and the target thermal imaging image, the field of view of the visible light camera and the thermal imaging camera are aligned.

[0079] In an alternative embodiment, the method further includes: a determining module; the determining module is used to:

[0080] The second region is determined within the coverage area based on RTK information in the coverage area of ​​the millimeter-wave radar.

[0081] In an alternative embodiment, it further includes: a monitoring module; the monitoring module is used for:

[0082] Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein, the target area is the lane line area.

[0083] The technical solution in this embodiment improves the accuracy of radar data and video data fusion by using dual-spectral field-of-view alignment, providing high-confidence data assurance for machine vision inspection.

[0084] The parameters and steps for implementing the corresponding functions of each module in the radar-visual fusion system 200 based on dual-spectral field alignment in this embodiment can be referred to the parameters and steps in the embodiments of the radar-visual fusion method based on dual-spectral field alignment above, and will not be repeated here.

[0085] like Figure 5As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320 coupled to a memory 310. The memory 310 stores at least one computer program 330, which is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above-mentioned laser-visual fusion methods based on dual-spectral field-of-view alignment. Specifically:

[0086] The electronic device 300 can vary considerably due to differences in configuration or performance. It may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310. The one or more memories 310 store at least one computer program 330, which is loaded and executed by the one or more processors 320 to enable the electronic device 300 to implement any of the dual-spectral field-of-view alignment-based radar-visual fusion methods provided in the above embodiments. Of course, the electronic device 300 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be elaborated upon here.

[0087] An embodiment of the present invention provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-described laser-visual fusion methods based on dual-spectral field alignment.

[0088] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0089] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the aforementioned laser-visual fusion methods based on dual-spectral field-of-view alignment.

[0090] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0091] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product in one or more computer-readable media containing computer-readable program code.

[0092] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0093] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for radar and visual fusion based on dual-spectrum field of view alignment, characterized in that, The method comprises the following steps: acquiring, by using a visible light camera, an original visible light image containing a first region, and acquiring, by using a thermal imaging camera, an original thermal imaging image containing the first region; respectively performing semi-transparent processing on the original visible light image and the original thermal imaging image to obtain a target visible light image and a target thermal imaging image; aligning fields of view of the visible light camera and the thermal imaging camera based on the target visible light image and the target thermal imaging image; wherein the fields of view are adjusted to be completely consistent by adjusting directions and zooming up, down, left and right through superimposition of the semi-transparent processed images; performing longitude and latitude calibration on visible light data and thermal imaging data of the first region after the field of view alignment to obtain a first calibration conversion relationship; determining a second region in a coverage area of a millimeter wave radar according to RTK information in the coverage area, and performing calibration on radar data of the second region and visible light data of the first region to obtain a second calibration conversion relationship; performing fusion on target visible light data, target thermal imaging data and target radar data of a target region in a preset time period based on the first calibration conversion relationship and the second calibration conversion relationship to obtain radar and visible light fusion data of the target region in the preset time period; wherein the target region is a common region of the first region and the second region; the first calibration conversion relationship is a calibration conversion relationship between a visible light image and a thermal imaging image about a same target object, and is used for matching and fusing the visible light image and the thermal imaging image; the second calibration conversion relationship is a calibration conversion relationship between radar data and a visible light image about a same target object, and is used for matching and fusing the radar data and the visible light image; obtaining event monitoring information of the target region in the preset time period based on the radar and visible light fusion data; wherein the target region is a lane line region.

2. A dual-spectrum field-of-view alignment based electro-optical fusion system, characterized in that, The method comprises the following steps: a field of view alignment module, a first calibration module, a second calibration module, a radar and visible light fusion module and a monitoring module; the field of view alignment module is used for acquiring, by using a visible light camera, an original visible light image containing a first region, and acquiring, by using a thermal imaging camera, an original thermal imaging image containing the first region; performing semi-transparent processing on the original visible light image and the original thermal imaging image to obtain a target visible light image and a target thermal imaging image; and aligning fields of view of the visible light camera and the thermal imaging camera based on the target visible light image and the target thermal imaging image; wherein the fields of view are adjusted to be completely consistent by adjusting directions and zooming up, down, left and right through superimposition of the semi-transparent processed images; the first calibration module is used for performing longitude and latitude calibration on visible light data and thermal imaging data of the first region after the field of view alignment to obtain a first calibration conversion relationship; The second calibration module is configured to: determine a second region in a coverage area of the millimeter wave radar according to RTK information in the coverage area, and calibrate radar data of the second region and visible light data of the first region to obtain a second calibration conversion relationship; The radar and visible light fusion module is configured to: based on the first calibration conversion relationship and the second calibration conversion relationship, fuse target visible light data, target thermal imaging data and target radar data of a target region in a preset time period to obtain radar and visible light fusion data of the target region in the preset time period; wherein the target region is a common region of the first region and the second region; the first calibration conversion relationship is a calibration conversion relationship between visible light images and thermal imaging images about a same target object, and is used for matching and fusing the visible light images and the thermal imaging images; and the second calibration conversion relationship is a calibration conversion relationship between radar data and visible light images about the same target object, and is used for matching and fusing the radar data and the visible light images. The monitoring module is configured to: based on the radar and visible light fusion data, obtain event monitoring information of the target region in the preset time period; and the target region is a lane line region.

3. An electronic device, comprising: The electronic device includes a processor coupled with a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the dual-spectrum field alignment-based radar and visible light fusion method of claim 1.

4. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor to enable the computer readable storage medium to implement the dual-spectrum field alignment-based radar and visible light fusion method of claim 1.

Images