A multi-modal image data acquisition system

Abstract

The application provides a multi-modal image data acquisition system, and belongs to the technical field of multi-modal images.The system comprises a visible light camera, a near-infrared camera, a medium infrared camera, a far-infrared camera, a four-in-one pod, a data storage unit, a servo control unit, a multi-rotor unmanned aerial vehicle and a ground command control station.The cameras are used to acquire image data; the four-in-one pod is used to mount and fix the visible light camera, the near-infrared camera, the medium infrared camera and the far-infrared camera, so that the target scale in different modal images acquired can be ensured to be consistent; the data storage unit is used to store the acquired multi-modal images; the servo control unit is used to control the azimuth and pitch adjustment of the cameras, so that multiple imaging under different imaging angles can be realized; the multi-rotor unmanned aerial vehicle is used to carry the cameras; and the ground command control station provides a task planning function.The application has the beneficial effect of solving the technical problems of multi-modal image data being difficult to match and being difficult to collect in large quantities.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal image, and particularly relates to a multi-modal image data acquisition system. BACKGROUND

[0002] Modality is a form of things, which refers to the description of things from a particular angle. Multi-modal image usually contains two or more than two single modal images, which classifies and explains image data from the angle of light wave band, for example: visible light image and near-infrared image form double modal image data; visible light image, near-infrared image and mid-infrared image form multi-modal image data; visible light image, near-infrared image, mid-infrared image and far-infrared image form multi-modal image data.

[0003] In computer vision, the complementary characteristics of multi-modal image data can make the feature more comprehensive and stereoscopic. Although multi-modal image data has significant advantages in describing features, it still faces great challenges. First, it is difficult to obtain high-quality and large-scale multi-modal image data; second, multi-modal image data needs to be matched into pairs for research, and the absence of any modal data will lead to a decrease in available images. Therefore, it is urgent to carry out research on multi-modal image data acquisition technology. SUMMARY

[0004] The application provides a multi-modal image data acquisition system, which aims to meet the demand of multi-modal image data acquisition, realize multiple imaging under different imaging angles, and solve the problems of difficult matching and large-scale collection of multi-modal image data.

[0005] The system comprises a visible light camera, a near-infrared camera, a mid-infrared camera, a far-infrared camera, a four-in-one pod, a data storage unit, a servo control unit, a multi-rotor unmanned aerial vehicle and a ground command control station.

[0006] The visible light camera is used to acquire visible light images, the near-infrared camera is used to acquire near-infrared images, the mid-infrared camera is used to acquire mid-infrared images, and the far-infrared camera is used to acquire far-infrared images; the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera all image the same scene to acquire images of the same scene under four different modalities, and the acquired images under the four different modalities have consistent target scales.

[0007] The four-in-one pod is used to install and fix the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera; the four-in-one pod is embedded with a detection, recognition and tracking module, which adopts edge computing and artificial intelligence algorithm to detect, recognize and track the target, and simultaneously links the servo control unit and each camera in the tracking process.

[0008] The data storage unit is used for storing the acquired visible light image, near-infrared image, mid-infrared image and far-infrared image;

[0009] The servo control unit controls and drives the visible light camera, near-infrared camera, mid-infrared camera and far-infrared camera to carry out azimuth and pitch adjustment, so as to realize multiple imaging under different imaging angles.

[0010] The multi-rotor unmanned aerial vehicle is used for carrying the visible light camera, near-infrared camera, mid-infrared camera, far-infrared camera, four-in-one pod, data storage unit and servo control unit, and indirectly changes the imaging distance by adjusting the flight height of the multi-rotor unmanned aerial vehicle, so as to change the image target scale and realize multi-modal and multi-scale target image acquisition.

[0011] The ground command control station provides task planning, manipulation management, recording and playback, comprehensive information display, flight path display, intelligence processing and real-time communication functions, and is connected with the multi-rotor unmanned aerial vehicle through a radio measurement control and information transmission subsystem.

[0012] In an embodiment, the image target scale is calculated in the following manner:

[0013]

[0014] Wherein, s represents the target scale; o b represents the target length; f represents the camera focal length; p represents the pixel size; d b represents the imaging distance; l d represents the number of long-side pixels of the detector.

[0015] In an embodiment, the four-in-one pod is precisely machined from aviation aluminum alloy, has the characteristics of vibration absorption, corrosion resistance, dust and rain prevention, and is suitable for field working environment.

[0016] In an embodiment, the interface between each device on the multi-rotor unmanned aerial vehicle and the multi-rotor unmanned aerial vehicle is a unified and standardized interface; the multi-rotor unmanned aerial vehicle includes a fuselage, a motor, an electronic speed controller, a paddle, a landing gear, a flight control assembly and a navigation assembly, and has the functions of sensing and adjusting its own attitude, speed and height.

[0017] In an embodiment, the response band of the visible light camera covers 0.4 μm-1 μm, the response band of the near-infrared camera covers 1 μm-2.5 μm, the response band of the mid-infrared camera covers 3 μm-5 μm, and the response band of the near-infrared camera covers 8 μm-14 μm.

[0018] In an embodiment, the azimuth angle adjustment range of the visible light camera, near-infrared camera, mid-infrared camera and far-infrared camera is 0°-360°, and the pitch angle adjustment range is 0°-180°.

[0019] In an embodiment, the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera adopt a fixed focus design or a zoom design.

[0020] In an embodiment, the azimuth adjustment range of the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera is 0°-360°. The pitch adjustment range of the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera is 0°-180°.

[0021] In an embodiment, the minimum flight height of the multi-rotor unmanned aerial vehicle is 2.65m, and the maximum flight height is 90.6m.

[0022] The beneficial effects of the present application include: (1) the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera are embedded in the four-in-one intelligent pod, which ensures the field of view consistency of the real multi-modal data, and solves the problem of difficult matching of multi-modal image data; (2) multi-modal image data of different scenes, different azimuths, different pitches and different target scales can be obtained, which ensures the diversity of multi-modal data and solves the problem of difficult large-scale collection of multi-modal image data. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the following description of the specific embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0024] Figure 1 The figure is a schematic diagram of the composition of the multi-modal image data acquisition system according to the embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0026] The present application proposes a multi-modal image data acquisition system, which aims to meet the needs of multi-modal image data acquisition. It includes a visible light camera, a near-infrared camera, a mid-infrared camera, a far-infrared camera, a four-in-one pod, a data storage unit, a servo control unit, a multi-rotor unmanned aerial vehicle, and a ground command and control station. Among them:

[0027] The visible light camera is used to acquire visible light images, the near-infrared camera is used to acquire near-infrared images, the mid-infrared camera is used to acquire mid-infrared images, and the far-infrared camera is used to acquire far-infrared images; the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera all image the same scene to acquire images of the same scene in four different modalities, and the acquired images in the four different modalities have consistent target scales;

[0028] The four-in-one pod is used to mount and fix the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera; the four-in-one pod is embedded with a detection, recognition, and tracking module, which uses edge computing and artificial intelligence algorithms to detect, recognize, and track targets, and simultaneously links the servo control unit and the cameras during tracking;

[0029] The data storage unit is used to store the acquired visible light images, near-infrared images, mid-infrared images, and far-infrared images;

[0030] The servo control unit controls and drives the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera to adjust the azimuth and pitch, thereby achieving multiple imaging at different imaging angles;

[0031] The multi-rotor unmanned aerial vehicle is used to carry the visible light camera, the near-infrared camera, the mid-infrared camera, the far-infrared camera, the four-in-one pod, the data storage unit, and the servo control unit, and indirectly changes the imaging distance by adjusting the flight height of the multi-rotor unmanned aerial vehicle, thereby changing the target scale of the images and achieving multi-modal and multi-scale target image acquisition;

[0032] The ground command and control station provides task planning, manipulation management, recording and playback, comprehensive information display, flight path display, intelligence processing, and real-time communication functions, and is connected to the multi-rotor unmanned aerial vehicle through a radio measurement and control and information transmission subsystem.

[0033] As shown in Figure 1 The system includes a visible light camera, a near-infrared camera, a mid-infrared camera, a far-infrared camera, a four-in-one pod, a data storage unit, a servo control unit, a multi-rotor unmanned aerial vehicle, and a ground command and control station.

[0034] The visible light camera is used to acquire visible light image data, the near-infrared camera is used to acquire near-infrared image data, the mid-infrared camera is used to acquire mid-infrared image data, and the far-infrared camera is used to acquire far-infrared image data.

[0035] The four-in-one pod is used for mounting and fixing the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera, and ensures that the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera all image the same scene, so that the target scale in the different modal images obtained is consistent. The calculation formula of the scale is as follows:

[0036]

[0037] In the formula, s represents the target scale; o b represents the target length; f represents the camera focal length; p represents the pixel size; d b represents the imaging distance; l d represents the number of long-side pixels of the detector.

[0038] The four-in-one pod is precisely machined from aviation aluminum alloy, has the characteristics of vibration absorption, corrosion resistance, dust and rain prevention, and can adapt to harsh field working environment.

[0039] The four-in-one pod is embedded with a detection, recognition and tracking module at the same time, which uses advanced edge computing and artificial intelligence algorithms to realize the detection, recognition and tracking of the target, and needs to be linked with the servo and the camera during the tracking process.

[0040] The data storage unit is used for storing the visible light image, the near-infrared image, the mid-infrared image and the far-infrared image obtained.

[0041] The servo control unit controls and drives the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera to adjust the azimuth and the pitch, so as to realize multiple imaging under different imaging angles.

[0042] The multi-rotor unmanned aerial vehicle includes a fuselage, a motor, an electronic speed controller, a paddle, a landing gear, a flight control assembly and a navigation assembly, and has the ability to perceive and stably adjust its attitude, speed and height.

[0043] The multi-rotor unmanned aerial vehicle is used for carrying the visible light camera, the near-infrared camera, the mid-infrared camera, the far-infrared camera, the four-in-one pod, the data storage unit and the servo control unit, and the imaging distance can be indirectly changed by adjusting the flight height of the unmanned aerial vehicle, so that the target scale in the image also changes, and multi-modal and multi-scale target image acquisition is realized. In addition, the interfaces of all devices and the unmanned aerial vehicle are unified and standardized.

[0044] The ground command and control station provides functions such as task planning, manipulation management, recording and playback, comprehensive information display, flight path display, intelligence processing and real-time communication, and is connected with the multi-rotor unmanned aerial vehicle through a radio measurement and control and information transmission subsystem.

[0045] The visible light camera response wavelength band covers 0.4 μm-1 μm. The near-infrared camera response wavelength band covers 1 μm-2.5 μm. The mid-infrared camera response wavelength band covers 3 μm-5 μm. The far-infrared camera response wavelength band covers 8 μm-14 μm.

[0046] The azimuth angle adjustment range of the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera is 0°-360°. The pitch angle adjustment range of the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera is 0°-180°.

[0047] The visible light camera adopts a fixed focus design, and the focal length is 55.2 mm. The near-infrared camera adopts a fixed focus design, and the focal length is 120 mm. The mid-infrared camera adopts a fixed focus design, and the focal length is 120 mm. The far-infrared camera adopts a fixed focus design, and the focal length is 96 mm.

[0048] The visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera can also adopt a zoom design. In the case that the flight height of the multi-rotor unmanned aerial vehicle is unchanged, multi-modal multi-scale target image acquisition can also be achieved.

[0049] The visible light camera, the near-infrared camera, the mid-infrared camera, the far-infrared camera and the four-in-one pod are carried on the multi-rotor unmanned aerial vehicle, and the minimum flight height of the multi-rotor unmanned aerial vehicle is 2.65 m, and the maximum flight height of the unmanned aerial vehicle is 90.6 m.

[0050] The resolution of the visible light camera is 1280×1024, and the pixel size is 3.45 μm. The resolution of the near-infrared camera is 640×512, and the pixel size is 15 μm. The resolution of the mid-infrared camera is 640×512, and the pixel size is 15 μm. The resolution of the far-infrared camera is 640×512, and the pixel size is 12 μm.

[0051] Note that the technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure. The above-described embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present patent should be subject to the appended claims.

Claims

1. A multi-modality image data acquisition system, characterized by, The system comprises a visible light camera, a near-infrared camera, a mid-infrared camera, a far-infrared camera, a four-in-one pod, a data storage unit, a servo control unit, a multi-rotor unmanned aerial vehicle, and a ground command control station. The visible light camera is used to acquire visible light images, the near-infrared camera is used to acquire near-infrared images, the mid-infrared camera is used to acquire mid-infrared images, and the far-infrared camera is used to acquire far-infrared images; the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera all image the same scene to acquire images of the same scene in four different modalities, and the acquired images in the four different modalities have consistent target scales; The four-in-one pod is used to mount and fix the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera; the four-in-one pod is embedded with a detection, recognition, and tracking module that uses edge computing and artificial intelligence algorithms to detect, recognize, and track targets while linking the servo control unit and the cameras during tracking; The data storage unit is used to store the acquired visible light images, near-infrared images, mid-infrared images, and far-infrared images; The servo control unit controls and drives the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera to adjust the azimuth and pitch, thereby achieving multiple imaging at different imaging angles; The multi-rotor unmanned aerial vehicle is used to carry the visible light camera, the near-infrared camera, the mid-infrared camera, the far-infrared camera, the four-in-one pod, the data storage unit, and the servo control unit, and indirectly changes the imaging distance by adjusting the flight height of the multi-rotor unmanned aerial vehicle, thereby changing the target scale of the images and achieving multi-modal and multi-scale target image acquisition; The ground command control station provides task planning, manipulation management, recording and playback, comprehensive information display, flight path display, intelligence processing, and real-time communication functions, and is connected to the multi-rotor unmanned aerial vehicle through a radio measurement and control and information transmission subsystem. The calculation method of the image target scale is as follows: ， wherein: represents the target scale; represents the target length; represents the camera focal length; represents the pixel size; represents the imaging distance; represents the number of long-side pixels of the detector; The resolution of the visible light camera is 1280x1024, the pixel size is 3.45μm, the resolution of the near-infrared camera is 640x512, the pixel size is 15μm, the resolution of the mid-infrared camera is 640x512, the pixel size is 15μm, and the resolution of the far-infrared camera is 640x512, the pixel size is 12μm.

2. The multi-modality image data acquisition system of claim 1, wherein, The four-in-one pod is made of aviation aluminum alloy and has the characteristics of vibration absorption, corrosion resistance, dust and rain prevention, and is suitable for outdoor working environment.

3. A multi-modality image data acquisition system according to claim 2, wherein, The interfaces between the devices on the multi-rotor unmanned aerial vehicle and the multi-rotor unmanned aerial vehicle are unified and standardized; the multi-rotor unmanned aerial vehicle includes a fuselage, a motor, an electronic speed controller, a propeller, a landing gear, a flight control component, and a navigation component, and has the functions of sensing, adjusting its attitude, speed, and height.

4. A multi-modality image data acquisition system according to claim 3, wherein, The response band of the visible light camera covers 0.4μm~1μm, the response band of the near-infrared camera covers 1μm~2.5μm, the response band of the mid-infrared camera covers 3μm~5μm, and the response band of the near-infrared camera covers 8μm~14μm.

5. A multi-modality image data acquisition system according to claim 4, wherein, The azimuth adjustment range of the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera is 0°-360°, and the pitch adjustment range is 0°-180°.

6. A multi-modality image data acquisition system according to claim 5, wherein, The visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera adopt a fixed focus design or a zoom design.

7. A multi-modality image data acquisition system according to claim 6, wherein, The azimuth adjustment range of the visible light camera, the near-infrared camera, the mid-infrared camera and the far-infrared camera is 0°-360°, and the pitch adjustment range is 0°-180°.

8. A multi-modality image data acquisition system according to claim 7, wherein, The minimum flight height of the multi-rotor unmanned aerial vehicle is 2.65 m, and the maximum flight height is 90.6 m.

Images