Multi-source target detection method and device

Abstract

The invention discloses a multi-source target detection method and device, and relates to the field of spectral detectors, a hyperspectral imaging module comprises a movable imaging lens protruding out of a mounting shell, an imaging spectrometer communicated with the imaging lens, and a detector mounted at the output part of the imaging spectrometer; the RGB camera and the imaging lens are located in the same acquisition direction, and a lens of the RGB camera is embedded into the surface of the shell to acquire an RGB image; the push-sweeping module is mounted on the inner wall of the mounting shell and comprises a driving motor, a sliding rail and a push-sweeping platform; the push-broom platform is mounted on the slide rail, the imaging lens is embedded in a mounting hole of the push-broom platform, and light enters through a slit at the entrance of the spectrometer; the push-scan platform is controlled based on the driving motor and moves along the sliding rail, and a hyperspectral image is obtained through push-scan. According to the scheme, the problems of poor precision and difficulty in correction of aircraft push-scan imaging are solved through the built-in push-scan module, and image type diversification is realized through the built-in multifunctional camera module.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of spectral detectors, and particularly to a multi-source target detection method and device. Background Art

[0002] The working principle of a hyperspectral detector is that the probe obtains target light, which is converted into an electrical signal by a photoelectric conversion device, and then becomes a digital signal by an A / D (digital / analog) conversion device and enters a computer. It combines imaging technology and spectral technology, and while detecting the spatial characteristics of an object, it disperses each spatial pixel to form a continuous spectrum of dozens to hundreds of narrow bands, with a very high spectral resolution (generally less than 10 nanometers), and is suitable for fine remote sensing quantitative analysis. In the forestry field, the hyperspectral detector forms a "spectral fingerprint" by capturing the reflection or absorption characteristics of vegetation in specific bands, and combines algorithm models to invert the physiological state of the vegetation. In the environmental monitoring field, the hyperspectral detector can identify the types of surface pollutants through spectral differences, and is suitable for environmental monitoring in complex scenarios, such as identifying the types of surface pollutants through spectral differences and assisting in judging the health status of crops. In the agricultural field, hyperspectral imaging technology can assist in judging the health status of crops and help farmers better manage farmland. For example, by analyzing the spectral characteristics of crops, the nutritional status and pest and disease conditions of crops can be judged, and corresponding management measures can be taken.

[0003] In some specific situations, target detection requires combining spectral analysis and image data analysis, and detecting in combination with aerial photography requires large-area image data analysis. However, it is difficult for a simple spectral device carried by an aircraft to perform pushbroom imaging to achieve accurate positioning and regional image stitching. The accuracy of spectral data is difficult to guarantee, and the target area cannot be intuitively identified solely based on the single obtained spectral data, and the diversification of target image types cannot be achieved. Summary of the Invention

[0004] The embodiments of the present application provide a multi-source target detection method and device to solve the problems of low imaging accuracy and data reliability of a single spectral detector in large-field pushbroom imaging, as well as the single image type.

[0005] On the one hand, the present application provides a multi-source target detection device, including an installation shell, a hyperspectral imaging module, an RGB camera module, and a pushbroom module; The hyperspectral imaging module includes a movable imaging lens protruding outside the installation shell, an imaging spectrometer communicated with the imaging lens inside the installation shell, and a detector installed at the output part of the imaging spectrometer; The RGB camera module is located inside the installation shell, and the RGB camera therein is in the same acquisition direction as the imaging lens, and the lens of the RGB camera is embedded in the surface of the shell to acquire RGB images; The push-sweeping module is installed on the inner wall of the installation shell and includes a driving motor, a slide rail, and a push-sweeping platform; the push-sweeping platform is installed on the slide rail, and the imaging lens is embedded in the mounting hole of the push-sweeping platform. Light enters through the slit at the entrance of the spectrometer; the push-sweeping platform is controlled based on the driving motor and moves along the slide rail to push-sweep and obtain hyperspectral images.

[0006] Specifically, the imaging spectrometer is a reflection-transmission integrated structure fixed in the shell. The entrances of the reflection part and the transmission part form a preset included angle, and the detector is installed at the exit of the transmission part; The entrance of the reflection part is vertically downward, receives the light incident from the imaging lens through the slit, and enters the transmission part and the detector after reflection.

[0007] Specifically, a processor module and a control main board are also arranged inside the installation shell; the control main board is installed in the space between the reflection part and the transmission part of the imaging spectrometer; The processor module is installed on the side of the transmission part of the imaging spectrometer, and the driving motor is located in the space below the transmission part of the imaging spectrometer and the detector.

[0008] Specifically, a thermal imaging camera is arranged on the side of the reflection part of the imaging spectrometer. The thermal imaging lens is embedded in the surface of the shell to obtain a thermal imaging map according to the control; the RGB camera is arranged between the processor module and the thermal imaging camera.

[0009] Specifically, the push-sweeping module further includes a push rod and a driver. One end of the push rod is connected to the output shaft of the driving motor, and the other end is connected to the push-sweeping platform; The driver is located on the side of the driving motor and controls the driving motor to perform the push-sweeping operation.

[0010] Specifically, a pair of limit pieces distributed along the moving direction of the slide rail are also arranged on the push-sweeping platform. A pair of photoelectric limit sensors matching the limit pieces are installed on the imaging spectrometer above the push-sweeping platform; the push-sweeping module controls the push-sweeping distance by detecting the two limit pieces.

[0011] Specifically, an adjustment hole with an opening size larger than the lens barrel diameter is opened at the interference fit place between the imaging lens and the installation shell. The imaging lens extends into the adjustment hole and is fitted into the mounting hole on the push-sweeping platform; A flexible dust-proof sheet that wraps the imaging lens and is connected to the push-sweeping platform and the adjustment hole is arranged in the adjustment hole.

[0012] On the other hand, the present application provides a multi-source target detection method. The method is used for an aerial vehicle carrying a multi-source target detection device, and the method includes: Start the RGB camera module to continuously capture the RGB image video stream. When flying to the first aerial photography point, obtain the first RGB image, identify the first ROI area and mark it; Switch to the push-broom imaging mode and start the hyperspectral imaging module to capture the hyperspectral image of the first ROI area; in the push-broom imaging mode, the push-broom module controls the imaging lens to move along the slide rail direction and captures several frames of the first hyperspectral images according to the instruction; After the push-broom is completed, start the RGB camera module to capture the intermediate RGB image, fly to the second aerial photography point to obtain the second RGB image and mark the second ROI area; Determine the camera attitude angle according to the second RGB image and the intermediate RGB image, switch to the push-broom imaging mode, and capture several frames of the second hyperspectral images of the second ROI area according to the instruction; Stitch and fuse the first hyperspectral image and the second hyperspectral image according to the camera attitude angle and the intermediate RGB image.

[0013] Specifically, in the push-broom mode, start the thermal imaging camera and capture the thermal imaging map of the corresponding target area according to the instruction. After moving to the adjacent aerial photography point, stitch and fuse the thermal imaging maps of two non-identical target areas according to the camera attitude angle and the intermediate RGB image.

[0014] Specifically, the imaging field of view of the RGB camera module is larger than that of the hyperspectral imaging module, and the imaging field of view of the thermal imaging camera is between the imaging field of view of the RGB camera module and the hyperspectral imaging module; At the distance between adjacent two aerial photography points, there is an overlap between the two ROI areas obtained by the hyperspectral imaging module under the corresponding imaging field of view; The processor module determines the camera attitude angle according to the overlapping area between the second RGB image and the intermediate RGB image.

[0015] The beneficial effects brought by the technical solution provided by the embodiment of the present application at least include: the technical effects brought by the technical solution of the present application include: the detection device and the image fusion and correction method can perform real-time correction on the system attitude based on the gimbal, and can reflect the fusion of hyperspectral images, spectra, thermal imaging (temperature), etc. of the photographed area with better accuracy, providing better support for later analysis. The RGB camera is used to solve the problem of hyperspectral image distortion. The RGB camera and the hyperspectral camera are designed on the same platform to make the alignment angles of the two consistent and achieve simultaneous imaging; the pixel-level image alignment and high-matching method can accurately register the hyperspectral image and the area array camera image, so as to achieve the purpose of image fusion and distortion correction.

[0016] In addition, the additionally installed thermal imaging camera in this solution can also collect thermal imaging data of the target area and assist in image fusion correction. Inside the entire device, multiple camera modules, a processor, a control main board, and a push-scan module for controlling push-scan operations are integrated into a limited space. By means of modules such as RGB cameras, stabilized gimbals, and lens push-scan imaging structures, the capabilities of remote sensing image, spectrum, temperature, and other signal detectors with high-precision and high-efficiency acquisition are realized; problems such as large accuracy differences and difficult corrections in the push-scan imaging mode achieved by aircraft are overcome; the imaging spatial resolution is improved, and the advantages of diverse types of target images in the same area are synchronously reflected, providing technical support for scientific research and industrial applications. Brief Description of the Drawings

[0017] Figure 1 is a schematic structural diagram of the hyperspectral imaging module provided by an embodiment of the present application; Figure 2 is a schematic structural diagram and a sectional view of the imaging spectrometer provided by an embodiment of the present application; Figure 3 is a schematic overall structural diagram of the multi-source target detection device provided by an embodiment of the present application; Figure 4 is a schematic diagram of the lens installation of each camera module; Figure 5 is a schematic overall structural diagram of the multi-source target detection device from another perspective; Figure 6 shows a sectional structural diagram of the push-scan module; Figure 7 shows a partially enlarged sectional view of the push-scan module and the imaging lens; Figure 8 is a flowchart of the plating method for the multi-source target detection device provided by the present application; Figure 9 is a schematic diagram of the stitching of RGB images and hyperspectral images taken at adjacent flight points.

[0018] Reference Numerals: 1, mounting shell; 2, imaging spectrometer; 3, detector; 4, imaging lens; 5, thermal imaging camera; 6, RGB camera; 7, processor module; 8, control main board; 9, drive motor; 10, slide rail; 11, flexible dust-proof sheet; 12, distal photoelectric limit sensor; 13, distal limit sheet; 14, proximal photoelectric limit sensor; 15, proximal limit sheet; 16, push-scan platform; 17, push rod; 18, driver; 19, adjustment hole. Detailed Embodiments

[0019] To make the objectives, technical solutions, and advantages of the present application clearer, the embodiments of the present application will be described in further detail below in conjunction with the accompanying drawings.

[0020] Figure 1It is a schematic structural diagram of the hyperspectral imaging module provided by an embodiment of the present application. This device is highly integrated inside a closed mounting shell 1, and all the acquisition cameras are concentrated on one mounting surface. When the device is installed on an aircraft, the acquisition surface should be vertically downward. Inside the mounting shell 1, a hyperspectral imaging module, an RGB camera module, a push-sweep module, etc. are provided.

[0021] Figure 1 Only the shell surface of the housing in which various lens modules are embedded is shown. The hyperspectral imaging module includes a movable imaging lens 4 protruding outside the mounting shell 1, an imaging spectrometer 2 inside the mounting shell 1 that communicates with the imaging lens 4, and a detector 3 installed at the output part of the imaging spectrometer 2.

[0022] In this solution, the structure of the imaging spectrometer 2 can be referred to Figure 2 as shown. The lower arrow indicates the light entry direction, and this position corresponds to the imaging lens 4. The right arrow indicates the light output direction, and this position corresponds to the installation of the detector 3. During installation, the light entry direction needs to be set vertically downward to adapt to the aerial photography perspective.

[0023] Figure 3 It is a schematic overall structure diagram of the multi-source target detection device provided by an embodiment of the present application. The RGB camera module therein is also located inside the mounting shell 1. Figure 4 It is a schematic diagram of the lens installation of each camera module. The RGB camera 6 and the imaging lens 4 are in the same acquisition direction. The lens of the RGB camera 6 is embedded in the housing surface to acquire RGB images according to instructions. It should be noted in this structure that the imaging lens 4 of the hyperspectral imaging module is detachable, so it is relatively arranged outside the housing, and the spectral principle also determines that the barrel length cannot be placed inside the housing. The introduction of the RGB camera 6 is essentially to facilitate the real-time acquisition of color RGB images for auxiliary positioning, as well as the auxiliary stitching and correction processing of hyperspectral images, which will be specifically described in the method embodiment.

[0024] Figure 5 It is a schematic overall structure diagram of the multi-source target detection device from another perspective. The push-sweep module therein is also installed on the inner wall of the mounting shell 1 and is located on the shell surface close to various lenses. This module includes a driving motor 9, a slide rail 10, and a push-sweep platform 16. The slide rail 10 and the driving motor 9 are fixed to the housing, and the push-sweep platform 16 is installed on the slide rail 10 and moves along the slide rail 10 under the control of the driving motor 9 to push-sweep and acquire hyperspectral images.

[0025] In the structure of the present application, the push-sweep platform 16 is used to carry and move the imaging lens 4, and also serves to connect the imaging lens 4 to the imaging spectrometer 2 inside the housing. Since the collected light ultimately needs to enter the detector 3, the push-sweep platform 16 needs to be provided with an installation hole (in a cylindrical shape) that can embed the integrated imaging lens 4. The integrated imaging lens 4 is embedded in the installation hole of the push-sweep platform 16, and the light enters through the slit at the entrance of the spectrometer 2. Figure 2 The circled part in the imaging spectrometer 2 is the slit entrance. It should be noted that the internal detailed structure of the imaging spectrometer 2 is not within the scope of protection of the present application and will not be elaborated here.

[0026] Since this multi-source target detection device needs to integrate functions such as control, acquisition, and data processing, it is necessary to integrate numerous components and processing modules. Moreover, since the device also needs to be installed on an aircraft, it is necessary to make reasonable use of the spatial layout. For this reason, the present application designs the overall structure of the imaging spectrometer 2 as a reflection-transmission integrated structure and fixes it inside the housing.

[0027] As Figure 2 shown, the overall imaging spectrometer 2 is divided into a cylindrical reflection part and a transmission part. The entrances of the reflection part and the transmission part form a preset angle. The detector 3 is installed at the outlet of the transmission part. In a possible implementation manner, the preset angle is set to 45 degrees. In this way, the entrance of the reflection part is set vertically downward, and the light incident from the imaging lens 4 is received through the slit, and after reflection, it enters the transmission part and the detector 3. The detector 3 and the transmission part also present a 45-degree angle relative to the housing plane, and some other modules can be built in the two formed angular spaces. In the present application, the push-sweep module is set in the space below the transmission part of the imaging spectrometer 2.

[0028] In the installation housing 1 of this device, a processor module 7 and a control main board 8 are also provided. The control main board 8 is installed in the (45-degree) space between the reflection part and the transmission part of the imaging spectrometer 2. The processor module 7 is installed on the side of the transmission part of the imaging spectrometer 2, and the drive motor 9 of the push-sweep module is located in the space below the transmission part of the imaging spectrometer 2 and the detector 3.

[0029] As Figure 3 and Figure 4 shown, a thermal imaging camera 5 is provided on the side of the reflection part of the imaging spectrometer 2. The thermal imaging lens is embedded in the housing surface, and a thermal imaging map is obtained according to the control. The RGB camera 6 is arranged between the processor module 7 and the thermal imaging camera 5. The three lenses form a triangular positioning relationship, and the viewing angle size of each lens is adjusted according to the actual situation.

[0030] To facilitate the description of the relationship between the imaging spectrometer 2 and the imaging lens 4, Figure 6 the cross-sectional structure diagram of the push-sweep module is shown, Figure 7Shows a partially enlarged sectional view of the push-sweep module and the imaging lens. The most important aspects of camera push-sweeping are stability and sealing. Since the imaging camera 4 can move while the imaging spectrometer 2 is fixed, the push-sweep platform 16 connecting the two is extremely important. In this structure, an adjustment hole 19 with an opening size larger than the lens barrel diameter is provided at the interference fit between the imaging lens 4 and the mounting shell 1. The imaging lens 4 extends into the adjustment hole 19 and is fitted into the mounting hole on the push-sweep platform 16. To achieve a dust-proof and sealing effect, a flexible dust-proof sheet 11 that wraps the imaging lens 4 and is connected to the push-sweep platform 16 and the adjustment hole 19 is provided in the adjustment hole 19. Refer to Figure 7 It is shown that the flexible dust-proof sheet 11 is designed with a laminated structure similar to fish scales and has good stretching and ductility. The adjustment hole 19 can be designed as a rectangular structure, and the length direction of the rectangle is the moving direction of the imaging lens 4. The rectangular adjustment hole 19 can also be designed in a sunken groove mode, that is, there are protruding bearing ends around the rectangle. Both ends of the flexible dust-proof sheet 11 are fixed to the bearing end positions in the width direction of the rectangle, and the bearing ends in the length direction of the rectangle are used to support the flexible dust-proof sheet 11. This flexible structure can ensure the maintenance of sealing during the movement process and isolate external light from entering the imaging spectrometer 2.

[0031] From Figure 5 it can be seen that the push-sweep module further includes a push rod 17 and a driver 18. One end of the push rod 17 is connected to the output shaft of the driving motor 9, and the other end is connected to the push-sweep platform 16. The driver 18 is located on the side of the driving motor 9. When the driver 18 receives an instruction sent by the processor module 7, it controls the driving motor 9 to operate. Under the action of the push rod 17, the push-sweep platform 16 is pulled to perform a push-sweep operation along the direction of the slide rail 10.

[0032] Since the imaging lens 4 is embedded in the push-sweep platform 16, this structure, under one-dimensional electric control drive, causes a relative movement trend between the rear focal plane lens of the imaging lens 4 and the entrance slit of the imaging spectrometer 2, thereby achieving push-sweep imaging. The distance between the imaging lens 4 and the slit can be determined by the object distance, lens focal length, etc. The standard imaging relationship is: the distance from the imaging lens to the rear focal length imaging slit is 17.532 mm. When the focal length of the imaging lens is determined, the linear field of view of its shooting target area can be basically determined, that is, the field of view that the slit length can shoot. Under the drive of the above one-dimensional electric control platform, the effective movement stroke of the imaging lens 4 is 1 cm. This movement stroke determines the scanning field of view width of the system. At a specified height or distance, the target area (area) of the system shooting a single-scene target can be determined based on the above two field of view dimensions.

[0033] The effective travel distance of the imaging lens 4 is determined by sensor detection. A pair of limit plates distributed along the moving direction of the slide rail 10 are arranged on the push-sweep platform 16, namely, the near-end limit plate 15 and the far-end limit plate 13. A pair of photoelectric limit sensors matching the limit plates, namely, the far-end photoelectric limit sensor 12 and the near-end photoelectric limit sensor 14 are installed on the imaging spectrometer 2 located above the push-sweep platform 16. When the push-sweep platform 16 drives the far-end limit plate 13 to move to the far-end photoelectric limit sensor 12, it stops moving away from the drive motor 9; when the push-sweep platform 16 drives the near-end limit plate 15 to move to the near-end photoelectric limit sensor 14, it stops moving toward the drive motor 9. The maximum travel at both ends is set to 1 cm.

[0034] In the entire imaging system, the built-in thermal imaging camera is used to obtain the thermal radiation image information of the object being photographed, and to obtain and record the temperature and other information of the thermal effect of the target area through this thermal imaging camera.

[0035] Although the fields of view formed by the above three detection modules are different, the final data calibration and correction need to be achieved in coordination with each other, integrating the images of the three detection sources in time and space dimensions to obtain more relevant information of the target.

[0036] The layout of the hyperspectral imaging module lens, RGB camera lens, and thermal imaging camera lens is designed in consideration of their respective sizes, field of view, and other factors; at the same time, it also takes into account factors such as the order of data collection and the time node of data collection. In the entire imaging system, the built-in hyperspectral imaging module, RGB camera, thermal imaging module, microprocessor, electronic control components, etc. are all in a static state, that is, only the imaging lens of the entire system is in relative motion, and other components will not move at all; so the center of gravity of the entire system is relatively stable, and compared with the imaging lens being static and the built-in imaging spectrometer module being in motion, the stability will be better.

[0037] When the above-mentioned imaging lens moves with a stroke of 1 cm, the center of gravity of the entire system will only change slightly. During this process, the three-dimensional stabilization gimbal of the aircraft will perform attitude correction in real time based on each unit step of the one-dimensional motion platform. In this way, in each fine movement state, the drone-mounted imaging system is statically related to the shooting target. This can well ensure attitude accuracy and image quality. Compared with the aircraft push-broom imaging method, it greatly reduces the difficulty of later data processing and there is no need to calibrate POS information every time, which has obvious advantages.

[0038] Figure 8The figure shows a flowchart of the multi-source target detection method provided by an embodiment of the present application, which is used for an aerial vehicle equipped with a multi-source target detection device, and includes the following steps: S1. Start the RGB camera module to capture the RGB image video stream in real time. When flying to the first aerial photography point, obtain the first RGB image, identify the first ROI area and mark it; In this embodiment, the device can be built-in an RGB camera with 8 million pixels, and its field of view angle can reach 75 degrees. The RGB camera can be set to the frame shooting mode and can work in the video stream and shooting modes. When the hyperspectral image is not collected, the RGB image is transmitted back to the ground in the video stream mode in real time. When the aircraft flies to the first aerial photography point (preset aerial photography position) and it is determined that the hyperspectral camera is to take a picture, the RGB camera is still transmitting the monitored picture, and the two are independent threads. After the image of this scene is taken, the system moves to the next flight point under the corresponding instruction and continues to perform repetitive work.

[0039] When arriving at the aerial photography point, it is first necessary to control the RGB camera to take the first RGB image of this position and field of view. This image is mainly used to identify the (first) ROI area of interest. Specifically, the ROI area of the RGB camera can be set through an algorithm so that the area to be taken by the hyperspectral camera can be accurately marked. The purpose is to be able to observe and mark the ROI area in real time through this RGB camera when the hyperspectral image is not obtained during aerial operations, so as to accurately determine the area to be taken and achieve the effect of "what you see is what you get".

[0040] S2. Switch to the push-broom imaging mode and start the hyperspectral imaging module to capture the hyperspectral image of the first ROI area; in the push-broom imaging mode, the push-broom module controls the imaging lens to move along the slide rail direction and captures a number of frames of the first hyperspectral image according to the instruction; When the system receives the instruction to switch to the push-broom mode, randomly start the hyperspectral imaging module to capture the hyperspectral image of the previously determined first ROI area. In the push-broom imaging mode, the encoder is used to control the motor to move at a specified speed and frequency, control the imaging lens to move along the slide rail direction, and capture a number of frames of the first hyperspectral image according to the instruction. When capturing the spectral image, the acquisition and transmission of the RGB image can also be started synchronously to facilitate the ground workstation to observe the real-time data.

[0041] S3. After the push-broom is completed, start the RGB camera module to capture the intermediate RGB image, fly to the second aerial photography point to obtain the second RGB image and mark the second ROI area; The RGB camera will be triggered at the moment when the hyperspectral pushbroom imaging is completed, and an RGB image will be taken and saved for reference in subsequent stitching. In theory, the aircraft is stationary at the first aerial photography point, and only the tiny travel difference in the pushbroom mode is moving, which does not affect the shooting effect of the RGB camera. However, to ensure accuracy, this application still intercepts a frame of the normal recording stage after the pushbroom is completed and marks it as the intermediate RGB image.

[0042] Immediately control the aircraft to move to the second aerial photography point, obtain the second RGB image and mark the second ROI area therein. The two ROI areas before and after this process are not exactly the same and usually contain an overlapping area, so as to facilitate the subsequent data stitching of the large area to be collected. Since the spectral data is a series of band data, it is impossible to stitch the images from different perspectives through image recognition technology. Therefore, this application introduces the RGB image and indirectly aligns and stitches the spectral images based on the recognizability of the RGB image.

[0043] S4. Determine the camera attitude angle according to the second RGB image and the intermediate RGB image, and after switching to the pushbroom imaging mode, take several frames of second hyperspectral images of the second ROI area according to the instruction; Figure 9 It is a stitching schematic diagram of the RGB images and hyperspectral images taken at adjacent flight points. The system conducts shooting strictly according to the flight point trajectory, and there is a 25% overlapping area between adjacent flight points. The existence of this overlapping area is for stitching the taken hyperspectral images, and of course, it can be adjusted according to the actual situation in different scenarios. On the one hand, the RGB camera serves the purpose of observation. On the other hand, the above-mentioned taken RGB color images are used to provide reference for subsequent data stitching and correction, etc.

[0044] Since the coordinates of the two aerial photography points are determined, when moving to the new aerial photography point, the second RGB image needs to contain part of the picture of the intermediate RGB image, that is, under the distance between adjacent aerial photography points, there is an overlap between the two ROI areas obtained by the hyperspectral imaging module in the corresponding imaging field of view. The processor module determines the camera attitude angle according to the overlapping area between the second RGB image and the intermediate RGB image.

[0045] S5. Stitch and fuse the first hyperspectral image and the second hyperspectral image according to the camera attitude angle and the intermediate RGB image.

[0046] The camera pose perspective is actually the matrix transformation formula for image stitching and fusion. When the position of the ROI area in the RGB image is determined, the first stitching matrix of the corresponding waypoint RGB image and the hyperspectral image can be determined. Align the overlapping areas of two RGB images, and the second stitching matrix of adjacent RGB images can be determined according to the coordinate relationship. Then, the stitching matrix relationship between two hyperspectral images can be calculated by combining the first and second stitching matrices, and then fusion stitching can be directly performed.

[0047] In some embodiments, for the accuracy and data richness of stitching, a thermal imaging camera is also activated to capture a thermal image in the push-broom mode. The thermal image can also be used to assist in the fusion stitching of the hyperspectral image, or the thermal imaging data of a specific target area can be obtained separately, and information such as temperature can be obtained and recorded through this thermal image. Therefore, the present application can also set the imaging field of view of the RGB camera module to be larger than that of the hyperspectral imaging module, and the imaging field of view of the thermal imaging camera is between the imaging field of view of the RGB camera module and the hyperspectral imaging module. That is, in terms of the picture content, the ROI area Thermal imaging target area RGB area. In this way, the stitching matrix relationship of the hyperspectral image can be calculated separately according to the RGB image and the thermal image, and then the final stitching matrix relationship of the hyperspectral image can be calculated by weighted calculation according to the set weight ratio, and then fusion stitching can be directly performed.

[0048] In summary, the technical effects brought by the technical solution of the present application include: the detection device and the image fusion correction method can perform real-time correction of the system pose based on the pan-tilt, and can better reflect the fusion of hyperspectral images, spectra, thermal imaging (temperature), etc. of the photographed area with higher accuracy, providing better support for later analysis. The RGB camera is used to solve the problem of hyperspectral image distortion. The RGB camera and the hyperspectral camera are designed on the same platform to make the alignment angles of the two consistent and achieve simultaneous imaging. Using the pixel-level high-matching method for image alignment, the hyperspectral image and the area array camera image can be accurately registered, so as to achieve the purpose of image fusion and distortion correction.

[0049] In addition, the additionally added thermal imaging camera in this solution can also realize the acquisition of thermal imaging data of the target area and assist in image fusion correction. Multiple camera modules, a processor, a control main board, and a push-broom module for controlling the push-broom operation are integrated into a limited space inside the entire device. With the help of modules such as the RGB camera, the stabilized pan-tilt, and the lens push-broom imaging structure, the ability of a remote sensing image, spectrum, temperature, etc. source detector with high precision and high efficiency can be achieved; problems such as large accuracy difference and difficult correction in the push-broom imaging mode realized by an aircraft are overcome; the imaging spatial resolution is improved, and the advantage of diverse target image types in the same domain is synchronously reflected, providing technical support for scientific research and industrial applications.

[0050] This specific embodiment is only an interpretation of the present invention and not a limitation thereof. After reading this specification, those skilled in the art may make modifications to this embodiment that do not contribute creatively as needed, but as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A multi-source target detection device, characterized in that, It includes an installation shell (1), a hyperspectral imaging module, an RGB camera module, and a push-sweep module; The hyperspectral imaging module includes a movable imaging lens (4) protruding outside the installation shell (1), an imaging spectrometer (2) inside the installation shell (1) communicating with the imaging lens (4), and a detector (3) installed at the output part of the imaging spectrometer (2); The RGB camera module is located inside the installation shell (1), and the RGB camera (6) therein and the imaging lens (4) are in the same acquisition direction, and the lens of the RGB camera (6) is embedded in the surface of the shell to acquire RGB images; The push-sweep module is installed on the inner wall of the installation shell (1) and includes a driving motor (9), a slide rail (10), and a push-sweep platform (16); the push-sweep platform (16) is installed on the slide rail (10), the imaging lens (4) is embedded in the installation hole of the push-sweep platform (16), and light enters through the slit at the entrance of the spectrometer (2); the push-sweep platform (16) is controlled based on the driving motor (9) and moves along the slide rail (10) to push-sweep and acquire hyperspectral images.

2. The multi-source target detection device according to claim 1, wherein The imaging spectrometer (2) is a reflection-transmission integrated structure fixed inside the shell, the entrances of the reflection part and the transmission part form a preset included angle, and the detector (3) is installed at the exit of the transmission part; The entrance of the reflection part is vertically downward, receives the light incident from the imaging lens (4) through the slit, and after reflection, enters the transmission part and the detector (3).

3. The multi-source target detection device according to claim 2, characterized in that, A processor module (7) and a control main board (8) are also provided inside the installation shell (1); the control main board (8) is installed in the space between the reflection part and the transmission part of the imaging spectrometer (2); The processor module (7) is installed on the side of the transmission part of the imaging spectrometer (2), and the driving motor (9) is located in the space below the transmission part of the imaging spectrometer (2) and the detector (3).

4. The multi-source target detection device according to claim 3, wherein A thermal imaging camera (5) is provided on the side of the reflection part of the imaging spectrometer (2), the thermal imaging lens is embedded in the surface of the shell, and a thermal imaging map is obtained according to the control; the RGB camera (6) is arranged between the processor module (7) and the thermal imaging camera (5).

5. The multi-source target detection device according to claim 2, characterized in that, The push-sweep module further includes a push rod (17) and a driver (18), one end of the push rod (17) is connected to the output shaft of the driving motor (9), and the other end is connected to the push-sweep platform (16); The driver (18) is located on the side of the driving motor (9) and controls the driving motor (9) to perform the push-sweep operation.

6. The multi-source target detection device according to any one of claims 1-5, characterized in that, A pair of limit pieces distributed along the moving direction of the slide rail (10) are also provided on the push-sweep platform (16), and a pair of photoelectric limit sensors matching the limit pieces are installed on the imaging spectrometer (2) above the push-sweep platform (16); the push-sweep module controls the push-sweep distance by detecting the two limit pieces.

7. The multi-source target detection device according to any one of claims 1-5, characterized in that, An adjustment hole (19) with an opening size larger than the lens barrel diameter is opened at the interference fit place between the imaging lens (4) and the installation shell (1), the imaging lens (4) extends into the adjustment hole (19) and is fitted into the installation hole on the push-sweep platform (16); A flexible dust-proof sheet (11) that wraps the imaging lens (4) and is connected to the push-sweep platform (16) and the adjustment hole (19) is provided inside the adjustment hole (19).

8. A multi-source target detection method, characterized in that, The method is used for an aerial vehicle equipped with a multi-source target detection device, and the method includes: Start the RGB camera module to capture an RGB image video stream in real time. When flying to the first aerial photography point, obtain the first RGB image, identify the first ROI area and mark it; Switch to the push-broom imaging mode, and start the hyperspectral imaging module to capture the hyperspectral image of the first ROI area; in the push-broom imaging mode, the push-broom module controls the imaging lens to move along the slide rail direction and capture a number of frames of the first hyperspectral image according to the instruction; After the push-broom is completed, start the RGB camera module to capture the middle RGB image, fly to the second aerial photography point to obtain the second RGB image and mark the second ROI area; Determine the camera attitude perspective according to the second RGB image and the middle RGB image, and after switching to the push-broom imaging mode, capture a number of frames of the second hyperspectral image of the second ROI area according to the instruction; Stitch and fuse the first hyperspectral image and the second hyperspectral image according to the camera attitude perspective and the middle RGB image.

9. The multi-source target detection method according to claim 8, wherein In the push-broom mode, start the thermal imaging camera and capture the thermal imaging map of the corresponding target area according to the instruction. After moving to the adjacent aerial photography point, stitch and fuse the thermal imaging maps of two non-identical target areas according to the camera attitude perspective and the middle RGB image.

10. The multi-source target detection method according to claim 9, wherein, The imaging field of view of the RGB camera module is larger than that of the hyperspectral imaging module, and the imaging field of view of the thermal imaging camera is between the imaging field of view of the RGB camera module and the hyperspectral imaging module; Under the distance between two adjacent aerial photography points, there is an overlap between the two ROI areas obtained by the hyperspectral imaging module under the corresponding imaging field of view; The processor module determines the camera attitude perspective according to the overlapping area between the second RGB image and the middle RGB image.

Images