Leighting fusion method, system and equipment based on dual-spectrum view field alignment and medium

Abstract

The invention relates to the technical field of road traffic, and particularly discloses a thundersight fusion method, system and device based on dual-spectrum view field alignment and a medium. Comprising the following steps: carrying out latitude and longitude calibration on visible light data of a first area acquired by a visible light camera after view field alignment and thermal imaging data of the first area acquired by a thermal imaging camera after view field alignment to obtain a first calibration conversion relation; calibrating radar data of a second area acquired by the millimeter wave radar and visible light data of the first area to obtain a second calibration conversion relation; and based on the first calibration conversion relation and the second calibration conversion relation, fusing the target visible light data, the target thermal imaging data and the target radar data of the target area in the preset time period to obtain thunder-vision fusion data. According to the invention, through a double-spectrum view field alignment mode, the fusion accuracy of the radar data and the video data is improved, and a high-confidence data guarantee is provided for machine vision detection.

Description

Background Art

[0002] At present, the width and height of the target in dual-light imaging are different, which leads to accuracy deviation. In multi-target scenes, the overall range of the field of view is inconsistent, which leads to inconsistent number of targets in the field of view, resulting in target data differences, which brings uncertainty to subsequent processing. In addition, dual-spectrum calibration mostly adopts manual calibration, which has certain subjectivity, and the accuracy of radar data and video data fusion cannot be guaranteed.

[0003] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a radar-vision fusion method, system, device and medium based on dual-spectral field of view alignment.

[0005] In a first aspect, the present invention provides a radar-vision fusion method based on dual-spectral field of view alignment, and the technical solution of the method is as follows:

[0006] Calibrate the longitude and latitude of the visible light data of the first area collected by the visible light camera after the field of view is aligned and the thermal imaging data of the first area collected by the thermal imaging camera after the field of view is aligned to obtain a first calibration conversion relationship;

[0007] Calibrate the radar data of the second area collected by the millimeter-wave radar and the visible light data of the first area collected by the visible light camera after the field of view is aligned to obtain a second calibration conversion relationship;

[0008] Based on the first calibration conversion relationship and the second calibration conversion relationship, the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period are fused to obtain the radar-visual fusion data of the target area within the preset time period; wherein the target area is: the common area of ​​the first area and the second area.

[0009] The beneficial effects of the radar-vision fusion method based on dual-spectrum field of view alignment of the present invention are as follows:

[0010] The method of the present invention improves the accuracy of the fusion of radar data and video data by aligning the dual-spectral field of view, and provides high-confidence data guarantee for machine vision detection.

[0011] On the basis of the above scheme, the radar-vision fusion method based on dual-spectral field of view alignment of the present invention can also be improved as follows.

[0012] In an optional manner, the method further includes:

[0013] Using a visible light camera, collecting an original visible light image including the first area, and using the thermal imaging camera, collecting an original thermal imaging image including the first area;

[0014] Performing semi-transparent processing on the original visible light image and the original thermal imaging image respectively to obtain a target visible light image and a target thermal imaging image;

[0015] Based on the target visible light image and the target thermal imaging image, the visible light camera and the thermal imaging camera are aligned in field of view.

[0016] In an optional manner, the method further includes:

[0017] The second area is determined in the coverage area according to the RTK information in the coverage area of ​​the millimeter wave radar.

[0018] In an optional manner, the method further includes:

[0019] Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein the target area is a lane line area.

[0020] In a second aspect, the present invention provides a radar-vision fusion system based on dual-spectral field of view alignment, and the technical solution of the system is as follows:

[0021] It includes: a first calibration module, a second calibration module and a radar-vision fusion module;

[0022] The first calibration module is used to: perform longitude and latitude calibration on visible light data of the first area collected by the visible light camera after the field of view is aligned and thermal imaging data of the first area collected by the thermal imaging camera after the field of view is aligned, to obtain a first calibration conversion relationship;

[0023] The second calibration module is used to calibrate the radar data of the second area collected by the millimeter wave radar and the visible light data of the first area collected by the visible light camera after the field of view is aligned, so as to obtain a second calibration conversion relationship;

[0024] The radar-visual fusion module is used to: fuse the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period based on the first calibration conversion relationship and the second calibration conversion relationship, so as to obtain the radar-visual fusion data of the target area within the preset time period; wherein the target area is: the common area of ​​the first area and the second area.

[0025] The beneficial effects of the radar-vision fusion system based on dual-spectrum field alignment of the present invention are as follows:

[0026] The system of the present invention improves the accuracy of the fusion of radar data and video data by aligning the dual-spectral field of view, providing high-confidence data guarantee for machine vision detection.

[0027] On the basis of the above scheme, the radar-vision fusion system based on dual-spectral field of view alignment of the present invention can also be improved as follows.

[0028] In an optional manner, the device further includes: a field of view alignment module; the field of view alignment module is used to:

[0029] Using the visible light camera, collecting an original visible light image including the first area, and using the thermal imaging camera, collecting an original thermal imaging image including the first area;

[0030] Performing semi-transparency processing on the original visible light image and the original thermal imaging image respectively to obtain a target visible light image and a target thermal imaging image;

[0031] Based on the target visible light image and the target thermal imaging image, the visible light camera and the thermal imaging camera are aligned in field of view.

[0032] In an optional manner, the method further includes: a determination module; the determination module is used to:

[0033] The second area is determined in the coverage area according to the RTK information in the coverage area of ​​the millimeter wave radar.

[0034] In an optional manner, the system further includes: a monitoring module; the monitoring module is used to:

[0035] Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein the target area is a lane line area.

[0036] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0037] The invention comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the radar-vision fusion method based on dual-spectral field of view alignment of the present invention are realized.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0039] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the radar-vision fusion method based on dual-spectral field of view alignment of the present invention.

[0040] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0042] Figure 1 It is a flow chart of an embodiment of a method for radar-vision fusion based on dual-spectral field of view alignment of the present invention;

[0043] Figure 2 This is a schematic diagram of semi-transparency processing;

[0044] Figure 3 This is a schematic diagram of radar calibration;

[0045] Figure 4 It is a structural schematic diagram of an embodiment of a radar-vision fusion system based on dual-spectral field of view alignment of the present invention;

[0046] Figure 5 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0047] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0048] Figure 1 A flow chart of an embodiment of a method for radar-vision fusion based on dual-spectral field of view alignment provided by the present invention is shown. The method for radar-vision fusion based on dual-spectral field of view alignment can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the method for radar-vision fusion based on dual-spectral field of view alignment by means of a processor calling computer-readable instructions stored in a memory. As Figure 1 As shown, the following steps are included:

[0049] S1. Calibrate the longitude and latitude of visible light data of a first area collected by a visible light camera after field of view alignment and the thermal imaging data of the first area collected by a thermal imaging camera after field of view alignment to obtain a first calibration conversion relationship.

[0050] Among them, the first area is the shooting (collection) area determined by the visible light camera and the thermal imaging camera. In road traffic, the first area can be a one-way lane line area of ​​a certain section on a highway, and can also be adjusted according to actual conditions, without any restrictions. Visible light data includes multiple targets in visible light images, and thermal imaging data includes multiple targets in thermal imaging images, such as lane lines, vehicles, trees, etc. The first calibration conversion relationship is: the calibration (latitude and longitude) conversion relationship between the visible light image and the thermal imaging image for the same target object, and the first calibration conversion relationship is used to match and fuse the visible light image and the thermal imaging image.

[0051] S2. Calibrate the radar data of the second area collected by the millimeter-wave radar and the visible light data of the first area collected by the visible light camera after the field of view is aligned to obtain a second calibration conversion relationship.

[0052] The second area is the shooting (collection) area determined by the millimeter wave radar. The second area is of the same type as the first area, and the size of the area included can be the same or different, without limitation. The second calibration conversion relationship is: the calibration conversion relationship between radar data and visible light image for the same target object, and the second calibration conversion relationship is used to match and fuse radar data and visible light image.

[0053] S3. Based on the first calibration conversion relationship and the second calibration conversion relationship, the target visible light data, the target thermal imaging data and the target radar data of the target area within a preset time period are fused to obtain the radar-visual fusion data of the target area within the preset time period.

[0054] The target area is the common area of ​​the first area and the second area. The length of the preset time period can be set according to actual conditions, such as 5s, 10s, etc. The target visible light data, target thermal imaging data and target radar data are the visible light data, thermal imaging data and radar data in the target area. The radar vision fusion data is the data after the target visible light data, target thermal imaging data and target radar data are associated and fused based on the first calibration conversion relationship and the second calibration conversion relationship.

[0055] It should be noted that the visible light camera, thermal imaging camera and millimeter wave radar in this embodiment are integrated into one device.

[0056] In an optional manner, the method further includes:

[0057] An original visible light image including the first area is acquired by using a visible light camera, and an original thermal imaging image including the first area is acquired by using the thermal imaging camera.

[0058] The original visible light image is a visible light image directly captured by a visible light camera, and the original thermal imaging image is a thermal imaging image directly captured by a thermal imaging camera.

[0059] The original visible light image and the original thermal imaging image are respectively subjected to semi-transparent processing to obtain a target visible light image and a target thermal imaging image.

[0060] The target visible light image and the target thermal imaging image are semi-transparent images. The semi-transparent processing method is: the transparency of the original image is set to a preset value, which is 50% by default and can also be adjusted according to actual needs.

[0061] Based on the target visible light image and the target thermal imaging image, the visible light camera and the thermal imaging camera are aligned in field of view.

[0062] Among them, Figure 2 As shown, by superimposing the semi-transparent processed images, the field of view alignment can be finely adjusted. The alignment process can make the field of view of the dual-light image (visible light image and thermal imaging image) completely consistent through up, down, left, and right direction adjustment and zooming.

[0063] In an optional manner, the method further includes:

[0064] The second area is determined in the coverage area according to the RTK information in the coverage area of ​​the millimeter wave radar.

[0065] Among them, Figure 3 As shown, there are multiple RTK information in the coverage area of ​​the millimeter wave radar ( Figure 3 The second area is defined based on the RTK information in the coverage area, namely, the lane line area defined by the millimeter-wave radar.

[0066] In an optional manner, the method further includes:

[0067] Based on the radar and vision fusion data, event monitoring information of the target area within the preset time period is obtained.

[0068] The target area is the lane line area. Event monitoring information includes, but is not limited to, monitoring information of events such as vehicle congestion, vehicle reversal, parking, spilled objects, vehicle departure, pedestrian crossing, high temperature target, lane icing, fog and low visibility.

[0069] Taking the vehicle's reverse driving as an example, multiple frames of radar-visual fusion data of the target area within a preset time period are obtained, and the vehicle's changing trajectory in the target area is detected using the machine vision algorithm in the multiple frames of radar-visual fusion data, and whether the vehicle is reverse driving is determined based on the vehicle's changing trajectory. For example, if the correct driving direction in the target area is from bottom to top, and the vehicle's changing trajectory is from top to bottom, it is determined that there is a vehicle reverse driving event and an event report is generated.

[0070] The technical solution of this embodiment improves the accuracy of the fusion of radar data and video data by aligning the dual spectral fields of view, providing high-confidence data assurance for machine vision detection.

[0071] Figure 4 FIG. 2 shows a schematic diagram of a structure of an embodiment of a radar-vision fusion system 200 based on dual-spectrum field alignment provided by the present invention. Figure 4 As shown, the system 200 includes: a first calibration module 210, a second calibration module 220 and a radar-visual fusion module 230;

[0072] The first calibration module 210 is used to: perform longitude and latitude calibration on visible light data of the first area collected by the visible light camera after the field of view is aligned and thermal imaging data of the first area collected by the thermal imaging camera after the field of view is aligned, to obtain a first calibration conversion relationship;

[0073] The second calibration module 220 is used to calibrate the radar data of the second area collected by the millimeter wave radar and the visible light data of the first area collected by the visible light camera after the field of view is aligned, so as to obtain a second calibration conversion relationship;

[0074] The radar-visual fusion module 230 is used to: fuse the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period based on the first calibration conversion relationship and the second calibration conversion relationship, so as to obtain the radar-visual fusion data of the target area within the preset time period; wherein the target area is: the common area of ​​the first area and the second area.

[0075] In an optional manner, the device further includes: a field of view alignment module; the field of view alignment module is used to:

[0076] Using the visible light camera, collecting an original visible light image including the first area, and using the thermal imaging camera, collecting an original thermal imaging image including the first area;

[0077] Performing semi-transparency processing on the original visible light image and the original thermal imaging image respectively to obtain a target visible light image and a target thermal imaging image;

[0078] Based on the target visible light image and the target thermal imaging image, the visible light camera and the thermal imaging camera are aligned in field of view.

[0079] In an optional manner, the method further includes: a determination module; the determination module is used to:

[0080] The second area is determined in the coverage area according to the RTK information in the coverage area of ​​the millimeter wave radar.

[0081] In an optional manner, the system further includes: a monitoring module; the monitoring module is used to:

[0082] Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein the target area is a lane line area.

[0083] The technical solution of this embodiment improves the accuracy of the fusion of radar data and video data by aligning the dual spectral fields of view, providing high-confidence data assurance for machine vision detection.

[0084] The above-mentioned parameters and steps for each module to implement corresponding functions in the radar-vision fusion system 200 based on dual-spectral field of view alignment of this embodiment can refer to the parameters and steps in the embodiment of the radar-vision fusion method based on dual-spectral field of view alignment above, and will not be repeated here.

[0085] like Figure 5 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above-mentioned radar-visual fusion methods based on dual-spectral field of view alignment, specifically:

[0086] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any of the radar-visual fusion methods based on dual-spectral field of view alignment provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.

[0087] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned radar-vision fusion methods based on dual-spectral field of view alignment.

[0088] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0089] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned radar-vision fusion methods based on dual-spectral field of view alignment.

[0090] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0091] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0092] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0093] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for radar-visual fusion based on dual-spectral field of view alignment, characterized in that: include: Calibrate the longitude and latitude of the visible light data of the first area collected by the visible light camera after the field of view is aligned and the thermal imaging data of the first area collected by the thermal imaging camera after the field of view is aligned to obtain a first calibration conversion relationship; Calibrate the radar data of the second area collected by the millimeter-wave radar and the visible light data of the first area collected by the visible light camera after the field of view is aligned to obtain a second calibration conversion relationship; Based on the first calibration conversion relationship and the second calibration conversion relationship, the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period are fused to obtain the radar-visual fusion data of the target area within the preset time period; wherein the target area is: the common area of ​​the first area and the second area.

2. The method for radar-visual fusion based on dual-spectral field of view alignment according to claim 1 is characterized in that: Also includes: Using a visible light camera, collecting an original visible light image including the first area, and using the thermal imaging camera, collecting an original thermal imaging image including the first area; Performing semi-transparent processing on the original visible light image and the original thermal imaging image respectively to obtain a target visible light image and a target thermal imaging image; Based on the target visible light image and the target thermal imaging image, the visible light camera and the thermal imaging camera are aligned in field of view.

3. The method for radar-visual fusion based on dual-spectral field of view alignment according to claim 1 is characterized in that: Also includes: The second area is determined in the coverage area according to the RTK information in the coverage area of ​​the millimeter wave radar.

4. The method for radar-visual fusion based on dual-spectral field of view alignment according to any one of claims 1 to 3, characterized in that: Also includes: Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein the target area is a lane line area.

5. A radar-vision fusion system based on dual-spectral field of view alignment, characterized in that: include: A first calibration module, a second calibration module and a radar-vision fusion module; The first calibration module is used to: perform longitude and latitude calibration on visible light data of the first area collected by the visible light camera after the field of view is aligned and thermal imaging data of the first area collected by the thermal imaging camera after the field of view is aligned, to obtain a first calibration conversion relationship; The second calibration module is used to calibrate the radar data of the second area collected by the millimeter wave radar and the visible light data of the first area collected by the visible light camera after the field of view is aligned, so as to obtain a second calibration conversion relationship; The radar-visual fusion module is used to: fuse the target visible light data, target thermal imaging data and target radar data of the target area within a preset time period based on the first calibration conversion relationship and the second calibration conversion relationship, so as to obtain the radar-visual fusion data of the target area within the preset time period; wherein the target area is: the common area of ​​the first area and the second area.

6. The radar-vision fusion system based on dual-spectral field of view alignment according to claim 5 is characterized in that: Also includes: Field of view alignment module; The field of view alignment module is used for: Using the visible light camera, collecting an original visible light image including the first area, and using the thermal imaging camera, collecting an original thermal imaging image including the first area; Performing semi-transparency processing on the original visible light image and the original thermal imaging image respectively to obtain a target visible light image and a target thermal imaging image; Based on the target visible light image and the target thermal imaging image, the visible light camera and the thermal imaging camera are aligned in field of view.

7. The radar-vision fusion system based on dual-spectral field of view alignment according to claim 5 is characterized in that: Also includes: Determine module; the determination module is used to: The second area is determined in the coverage area according to the RTK information in the coverage area of ​​the millimeter wave radar.

8. The radar-vision fusion system based on dual-spectral field of view alignment according to any one of claims 5 to 7, characterized in that: Also includes: Monitoring module; the monitoring module is used to: Based on the radar-visual fusion data, event monitoring information of the target area within the preset time period is obtained; wherein the target area is a lane line area.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the radar-vision fusion method based on dual-spectral field of view alignment as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that the computer-readable storage medium implements the radar-vision fusion method based on dual-spectral field of view alignment as described in any one of claims 1 to 4.

Images