A multi-modal data acquisition system and synchronous acquisition method

Abstract

The present invention discloses a multimodal data acquisition system, including a visible light camera array data acquisition subsystem composed of n×n visible light cameras, an infrared camera data acquisition subsystem composed of m infrared cameras, and a millimeter wave radar data acquisition subsystem composed of p millimeter wave radars, wherein n, m, and p are natural numbers greater than or equal to 1. The present invention also discloses a multimodal data synchronous acquisition method, including: building a multimodal data acquisition system; calibrating external parameters between each data acquisition subsystem to obtain a multimodal data acquisition system with unified external parameters; each data acquisition subsystem obtains a time-series synchronized multimodal data acquisition system by combining synchronous triggering hard synchronization and alignment and registration soft synchronization in the time dimension, and acquires data of different modes; and fuses data of different modes. The present invention can be used for applications such as image quality enhancement and target detection under severe weather conditions such as dense smoke, dense fog, rain and snow.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-modal data synchronous acquisition and fusion system and method. Background Art

[0002] In the application fields of daily life, such as security monitoring and autonomous driving, visible light cameras will lose their function under extremely bad weather conditions such as thick smoke, dense fog, rain and snow. They cannot obtain all the texture and color information of the scene and the imaging quality is not high. This makes all-weather monitoring of natural scenes particularly difficult. Therefore, it is particularly important to improve the image quality of visible light cameras under bad weather conditions such as thick smoke, dense fog, rain and snow.

[0003] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0004] In order to solve the problem of image quality enhancement of visible light cameras under severe weather conditions such as thick smoke, thick fog, rain and snow, the present invention proposes a multimodal data acquisition system and a synchronous acquisition method, which can be used for applications such as image quality enhancement and target detection under severe weather conditions such as thick smoke, thick fog, rain and snow.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention discloses a multimodal data acquisition system, comprising a visible light camera array data acquisition subsystem consisting of n×n visible light cameras, an infrared camera data acquisition subsystem consisting of m infrared cameras, and a millimeter wave radar data acquisition subsystem consisting of p millimeter wave radars, wherein n, m, and p are natural numbers greater than or equal to 1, respectively.

[0007] The present invention also discloses a method for synchronously collecting multimodal data, comprising the following steps:

[0008] Z1: Build the above multimodal data acquisition system;

[0009] Z2: Calibrate external parameters between various data acquisition subsystems to obtain a multi-modal data acquisition system with unified external parameters;

[0010] Z3: Each data acquisition subsystem obtains a time-series synchronized multi-modal data acquisition system by combining synchronous triggering hard synchronization and alignment and registration soft synchronization in the time dimension, and acquires data of different modalities;

[0011] Z4: Fuse data from different modalities to obtain quality-enhanced images.

[0012] Preferably, the visible light camera array data acquisition subsystem is used to collect texture and color information of the scene; the infrared camera data acquisition subsystem is used to collect detail information of the scene; and the millimeter wave radar data acquisition subsystem is used to collect depth information of the scene.

[0013] Preferably, step Z2 specifically includes: taking the visible light camera array data acquisition subsystem as a reference benchmark, obtaining the relative position relationship between each data acquisition subsystem through self-calibration of the visible light camera array data acquisition subsystem, joint calibration of external parameters between the visible light camera array data acquisition subsystem and the millimeter wave radar data acquisition subsystem, and feature point matching and registration between the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem, so as to obtain a multimodal data acquisition system with unified external parameters.

[0014] Preferably, the joint calibration of external parameters between the visible light camera array data acquisition subsystem and the millimeter wave radar data acquisition subsystem specifically includes: placing radar calibration objects in the scene, calculating the global homography matrix of the real three-dimensional points on the plane scanned by the radar and the corresponding pixels, and completing the solution of the relative position relationship between the radar and the visible light camera.

[0015] Preferably, the feature point matching and registration between the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem specifically includes: scaling the image according to the proportional relationship between the images acquired by the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem, extracting feature points and constructing descriptors, and then using RANSAC to remove external points and using the remaining internal points to solve the homography matrix to complete image mapping, thereby achieving calibration between the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem.

[0016] Preferably, the synchronous triggering hard synchronization specifically includes: receiving a synchronization signal through a universal input and output interface, and acquiring a rising or falling edge of the synchronization signal to trigger shooting to realize synchronous acquisition.

[0017] Preferably, the synchronization signal is transmitted via a coaxial cable, and a square wave signal generator is used as a trigger source for triggering each data acquisition subsystem to shoot, wherein the synchronization signal is transmitted via a coaxial cable in the following manner: the synchronization signal is encoded and transmitted, a single-ended input to differential signal circuit module is used to convert the trigger signal into a differential signal and the signal is transmitted via a twisted pair cable, and a differential input to single-ended signal circuit module is used to restore the signal at the receiving end of each data acquisition subsystem.

[0018] Preferably, the field synchronization signal of the infrared camera data acquisition subsystem is used as the trigger source for shooting of each data acquisition subsystem, and the field synchronization signal is converted into a trigger signal acceptable to other data acquisition subsystems through a frequency conversion module. The trigger signal is transmitted to each data acquisition subsystem to complete hard synchronization acquisition.

[0019] Preferably, the alignment and registration soft synchronization of each data acquisition subsystem in the time dimension specifically includes: recording the data acquisition time of each data acquisition subsystem of the multimodal data acquisition system in the form of a system timestamp according to possible data frame loss and start frame alignment problems during actual acquisition, so as to align the data timestamps between each data acquisition subsystem, and thus obtain a multimodal synchronous acquisition system with time synchronization.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: the multimodal data acquisition system and synchronous acquisition method proposed in the present invention construct a multimodal data acquisition system consisting of a visible light camera array data acquisition subsystem composed of n×n visible light cameras, an infrared camera data acquisition subsystem composed of m infrared cameras, and a millimeter wave radar data acquisition subsystem composed of p millimeter wave radars, and external parameter calibration is performed between each data acquisition subsystem to determine the relative position relationship, thereby obtaining the mapping relationship of data between each data acquisition subsystem; each data acquisition subsystem is obtained by combining synchronous triggering hard synchronization and alignment and registration soft synchronization in the time dimension to obtain a time-series synchronized multimodal data acquisition system. Finally, the millimeter wave radar data acquisition subsystem is used to provide depth information, the infrared camera data acquisition subsystem is used to provide contour information, and data fusion is performed with the visible light camera array data acquisition subsystem, which can make full use of the complementary advantages between different modal data to achieve image quality enhancement, target detection and other applications under severe weather conditions such as dense smoke, dense fog, rain and snow. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a multi-modal data synchronous acquisition method according to a preferred embodiment of the present invention;

[0022] Figure 2 It is a unified schematic diagram of external parameters of a multimodal data acquisition system according to a preferred embodiment of the present invention;

[0023] Figure 3 It is a timing synchronization schematic diagram of a multimodal data acquisition system according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the present invention and the embodiments are explained in more detail below in conjunction with the accompanying drawings. The accompanying drawings are only used as examples rather than as limitations of the present invention. Any changes or substitutions made by any technician familiar with the technical field within the technical scope of the present invention should be covered within the protection scope of the present invention.

[0025] A preferred embodiment of the present invention discloses a multimodal data acquisition system, which is composed of a visible light camera array data acquisition subsystem consisting of n×n visible light cameras, an infrared camera data acquisition subsystem consisting of m infrared cameras, and a millimeter wave radar data acquisition subsystem consisting of p millimeter wave radars.

[0026] like Figure 1 As shown, a multi-modal synchronous data acquisition method according to a preferred embodiment of the present invention comprises the following steps:

[0027] Z1: Build a multi-modal synchronous data acquisition system;

[0028] A multimodal data acquisition system is composed of a visible light camera array data acquisition subsystem consisting of n×n visible light cameras, an infrared camera data acquisition subsystem consisting of m infrared cameras, and a millimeter-wave radar data acquisition subsystem consisting of p millimeter-wave radars, wherein sensors within each data acquisition subsystem and between each data acquisition subsystem are freely placed without any restrictions, and n, m, and p are natural numbers greater than or equal to 1, respectively.

[0029] Among them, the visible light camera array data acquisition subsystem is used to collect the texture and color information of the scene; the infrared camera data acquisition subsystem is used to collect detailed information such as the outline of the scene; and the millimeter wave radar data acquisition subsystem is used to collect the depth information of the scene.

[0030] Z2: Perform external parameter calibration between the multimodal data acquisition subsystems to determine the relative position relationship, thereby obtaining a multimodal data acquisition system 100 with unified external parameters;

[0031] Among them, the multi-modal data synchronous acquisition system 100 with unified external parameters takes the visible light camera array data acquisition subsystem 10 as a reference benchmark, obtains the relative position relationship between each data acquisition subsystem through self-calibration of the visible light camera array data acquisition subsystem 10, joint calibration of external parameters between the visible light camera array data acquisition subsystem 10 and the millimeter wave radar data acquisition subsystem 30, and feature point matching and registration between the visible light camera array data acquisition subsystem 10 and the infrared camera data acquisition subsystem 20, and then obtains the multi-modal data acquisition system 100 with unified external parameters, as shown in the schematic diagram. Figure 2 shown.

[0032] In the camera array self-calibration, a calibration method combining static calibration and dynamic calibration is adopted: the static calibration method adopts Zhang Zhengyou's plane calibration method, which calibrates the internal and external parameters between each camera by placing a checkerboard in front of the camera array; the dynamic calibration method is for dynamically changing scenes, using the results of static calibration as the initial estimate, extracting and matching feature points of the images collected by the camera array, inferring the real three-dimensional point coordinates and reprojecting them to remove external points with large errors, and using the internal points to find the refined camera internal and external parameters by minimizing the reprojection error, so as to complete the high-precision calibration of the camera array in complex scenes.

[0033] The joint calibration of external parameters between the visible light camera array data acquisition subsystem and the millimeter wave radar data acquisition subsystem means: by placing radar calibration objects in the scene, calculating the global homography matrix of the real three-dimensional points and corresponding pixels of the plane scanned by the radar, the relative position relationship between the radar and the camera is solved.

[0034] In the registration part of infrared image and visible light image, due to the inconsistency of the focal length and pixel size of the infrared camera, the size of each pixel in the image captured by the infrared camera and the visible light camera is different. Therefore, in the present invention, it is necessary to first scale the image according to the proportional relationship between the two, and then extract the feature points and construct the descriptor, and then use RANSAC to remove the external points, and use the remaining internal points to solve the homography matrix, so as to complete the image mapping and realize the calibration between the visible light camera and the infrared camera.

[0035] In the above, through the pairwise calibration of external parameters between each subsystem, a multimodal data acquisition system with unified external parameters is obtained.

[0036] Z3: A multi-modal data acquisition system with synchronous triggering and time-alignment between multi-modal subsystems to obtain time-series synchronization;

[0037] Specifically, the multimodal data synchronous acquisition system uses a combination of hard synchronization and soft synchronization to achieve time-series synchronized multimodal data acquisition, that is, the multimodal data acquisition subsystems are synchronized by a combination of hard synchronization triggered by synchronization and soft synchronization of alignment and registration in the time dimension to obtain a time-series synchronized multimodal data acquisition system, as shown in the schematic diagram. Figure 3 shown.

[0038] The hard synchronization uses a synchronization signal to trigger each data acquisition subsystem. Each subsystem adopts a trigger acquisition mode, receives the synchronization signal through a general input and output interface (GPIO), obtains the rising or falling edge of the synchronization signal, and triggers the shooting to achieve synchronous acquisition. In the multimodal data synchronous acquisition system, a square wave signal generator is used as a trigger source, and the synchronization signal is transmitted through a coaxial cable. At the same time, in order to improve the anti-interference ability of the signal during outdoor and long-distance transmission, the multimodal data synchronous acquisition system encodes and transmits the synchronization signal, and uses a single-ended input to differential signal circuit module to convert the trigger signal into a differential signal and transmit it through a twisted pair. The signal is restored at the receiving end of each data acquisition subsystem using a differential input to single-ended signal circuit module. In some embodiments, if there is a situation in which the data acquisition subsystem cannot trigger acquisition due to its own hardware limitations, its output synchronization signal can be considered as the trigger source of the system. For example, the field synchronization signal of the infrared camera data acquisition system is used as the trigger source of the entire system, and the field synchronization signal is converted into a trigger signal acceptable to other subsystems through a frequency conversion module. By transmitting the trigger signal to each data acquisition subsystem, the hard synchronization acquisition is finally completed.

[0039] The soft synchronization of the multi-mode data acquisition system records the data acquisition time of each data acquisition subsystem of the multi-modal data acquisition system in the form of a system timestamp based on the data frame loss and start frame alignment problems that may exist during actual system acquisition. This is to align the data timestamps between the data acquisition subsystems, and thus obtain a multi-modal synchronous acquisition system with timing synchronization.

[0040] Z4: Fusion of data features from different modalities is used to ultimately achieve image quality enhancement and target detection for visible light cameras in severe weather conditions such as heavy smoke, fog, rain and snow.

[0041] The multimodal data fusion method is applied to the multimodal data acquisition system to achieve a variety of applications such as image quality enhancement and target detection. Under severe weather conditions such as thick smoke, dense fog, rain and snow, a large amount of Mie scattering occurs in the visible light band in the air, resulting in a sharp decrease in visible light visibility. It is difficult for infrared cameras to obtain texture and color information on the surface of objects, but they have a strong ability to image through smoke and water vapor. At the same time, they are more sensitive to heat sources such as people and cars. The infrared camera data acquisition subsystem is used to provide significant object contour information to the visible light camera array data acquisition subsystem. The image fusion method is used to fuse the significant object contour information in the infrared image with the color information of the visible light image to achieve image quality enhancement. At the same time, due to the depth of field problem of the scene during data acquisition, all objects cannot be clearly imaged, but the visible light camera array refocusing method can make them focused. In a certain depth plane, objects in the plane are made clear, but it is a question that needs to be clarified which specific depth layer should be focused on. The millimeter wave radar data acquisition subsystem is used to provide the visible light camera array data acquisition subsystem with accurate imaging plane depth information for refocusing, and at the same time correct the visible light image fog transmittance map, so as to achieve the improvement of the image quality of the visible light camera under severe weather conditions such as thick smoke, thick fog, rain and snow; at the same time, the traditional target detection network methods such as yolo and centernet are used for the multimodal data fusion results, or the infrared image and visible light image features are fused and the target is detected in the deep neural network, which can also improve the target detection accuracy, so as to realize target detection and other related applications.

[0042] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for synchronously collecting multimodal data, characterized in that: The following steps are involved: Z1: Build a multimodal data acquisition system, which includes a visible light camera array data acquisition subsystem consisting of n×n visible light cameras, an infrared camera data acquisition subsystem consisting of m infrared cameras, and a millimeter wave radar data acquisition subsystem consisting of p millimeter wave radars, where n, m, and p are natural numbers greater than or equal to 1 respectively; Z2: external parameter calibration is performed between each data acquisition subsystem to obtain a multimodal data acquisition system with unified external parameters: taking the visible light camera array data acquisition subsystem as a reference benchmark, the relative position relationship between each data acquisition subsystem is obtained through self-calibration of the visible light camera array data acquisition subsystem, joint external parameter calibration between the visible light camera array data acquisition subsystem and the millimeter wave radar data acquisition subsystem, and feature point matching and registration between the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem, so as to obtain a multimodal data acquisition system with unified external parameters; Z3: Each data acquisition subsystem obtains a time-series synchronized multi-modal data acquisition system by combining synchronous triggering hard synchronization and alignment and registration soft synchronization in the time dimension, and acquires data of different modalities; Z4: Fuse data from different modalities to obtain quality-enhanced images.

2. The method for synchronously collecting multimodal data according to claim 1, characterized in that: The visible light camera array data acquisition subsystem is used to collect texture and color information of the scene; the infrared camera data acquisition subsystem is used to collect detail information of the scene; and the millimeter wave radar data acquisition subsystem is used to collect depth information of the scene.

3. The method for synchronously collecting multimodal data according to claim 1, characterized in that: The external parameter joint calibration between the visible light camera array data acquisition subsystem and the millimeter wave radar data acquisition subsystem specifically includes: by placing radar calibration objects in the scene, calculating the global homography matrix of the real three-dimensional points and corresponding pixels of the plane scanned by the radar, and completing the solution of the relative position relationship between the radar and the visible light camera.

4. The method for synchronously collecting multimodal data according to claim 1, characterized in that: The feature point matching and registration between the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem specifically includes: scaling the image according to the proportional relationship between the images acquired by the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem, extracting feature points and constructing descriptors, and then removing external points using RANSAC, and solving the homography matrix using the remaining internal points to complete image mapping, thereby realizing calibration between the visible light camera array data acquisition subsystem and the infrared camera data acquisition subsystem.

5. The method for synchronously collecting multimodal data according to claim 1, characterized in that: The synchronous trigger hard synchronization specifically includes: receiving the synchronization signal through the general input and output interface, obtaining the rising or falling edge of the synchronization signal to trigger shooting to achieve synchronous acquisition.

6. The method for synchronously collecting multimodal data according to claim 5, characterized in that: The synchronization signal is transmitted through a coaxial cable, and a square wave signal generator is used as a trigger source for triggering each data acquisition subsystem to shoot. The synchronization signal is transmitted through a coaxial cable in the following ways: the synchronization signal is encoded and transmitted, and the trigger signal is converted into a differential signal using a single-ended input to differential signal circuit module and transmitted through a twisted pair cable. The signal is restored at the receiving end of each data acquisition subsystem using a differential input to single-ended signal circuit module.

7. The method for synchronously collecting multimodal data according to claim 1, characterized in that: The field synchronization signal of the infrared camera data acquisition subsystem is used as the trigger source for shooting of each data acquisition subsystem. The field synchronization signal is converted into a trigger signal acceptable to other data acquisition subsystems through a frequency conversion module. The trigger signal is transmitted to each data acquisition subsystem to complete hard synchronization acquisition.

8. The method for synchronously collecting multimodal data according to any one of claims 1 to 7, characterized in that: The alignment and soft synchronization of each data acquisition subsystem in the time dimension specifically includes: according to the possible data frame loss and start frame alignment problems during actual acquisition, the data acquisition time of each data acquisition subsystem of the multimodal data acquisition system is recorded in the form of system timestamps to align the data timestamps between each data acquisition subsystem, thereby obtaining a multimodal synchronous acquisition system with time synchronization.

Images