Algorithm-based multi-view strapdown multi-source image recognition equipment

Abstract

The invention relates to the technical field of detection equipment, and provides algorithm-based multi-view strapdown type multi-source image recognition equipment, which comprises a strapdown integrated guide module, the strapdown integrated guide module is mounted on the image recognition equipment through a gyroscope of the strapdown integrated guide module, and the strapdown integrated guide module comprises a Hi3559A processing module, a multi-view strapdown type multi-source image recognition module and a multi-view strapdown type multi-source image recognition module, the Hi3559A processing module integrates a multi-source signal fusion module, an image information processing module, a control module, an operation module and a multi-core heterogeneous processing module; a strapdown integrated guide head, wherein the strapdown integrated guide head can be independently and detachably mounted on the identification equipment; it can be understood that the method simulates the detection and attack mechanism of an eagle and tenon to the ground, is combined with the intelligent target detection and tracking, guidance information extraction, guidance law, control law and other elements of the unmanned aerial vehicle, enlarges the reconnaissance area, stabilizes the tracking target, and improves the strike precision; on the basis of a strapdown guidance mode, an optical path and processing model coupled with a reconnaissance route and an attack trajectory is designed, and the problems that an existing strapdown guidance system is insufficient in information acquisition, and a target is prone to being lost in the guidance stage are solved. A multi-task network model is utilized to realize a target identification and tracking integrated method, and the problems of low identification speed, small sensing range and difficulty in target search are solved.

Description

Technical Field

[0001] The present invention relates to the field of detection equipment, and more specifically, to an algorithm-based multi-eye strapdown multi-source image recognition device. Background Art

[0002] The detection equipment currently carried by aircraft can be divided into single-mode and multi-mode according to the information source. The sensors used by single-mode include visible light cameras, infrared cameras, 4D imaging millimeter wave radars, lidars, etc. Multi-mode mostly use a combination of infrared, visible light, and laser detection methods, all of which have obvious limitations. Single-mode detection equipment has a lower cost, but the detection range is smaller and is constrained by the working area of ​​each sensor. Multi-mode sensors are mostly gimbal-type structures, which use mechanical gimbals to align the data of multiple sensors, effectively improving the perception range and expanding the use scenarios. However, the cost and price are high, which does not conform to the development trend of future aircraft clusters; during the flight detection process of aircraft, the situation changes rapidly and there are many gray areas, which requires the aircraft to have wide-area reconnaissance and detection capabilities;

[0003] First, the Hawkeye is a strapdown imaging system, and its eyeballs do not have the ability to rotate. Secondly, a single Hawkeye imaging system can be regarded as a combination of a wide-angle camera and two telephoto cameras. Based on the above characteristics of the Hawkeye imaging system, the present invention proposes a multi-source heterogeneous strapdown reconnaissance imaging platform device based on the bionic Hawkeye structure. At present, the cost and performance of detection equipment are the two major factors restricting the development of aircraft image reconnaissance systems;

[0004] The first is the cost of the guidance system. According to statistics, the cost of the image system and algorithm module of the high-end UAV image detection system accounts for more than two-thirds of the total cost of the whole machine. Therefore, reducing the cost of the guidance system is the key direction to reduce the cost of the UAV image detection system. The second is the performance of the UAV image detection system. At present, micro-aircraft mostly use a composite reconnaissance and navigation integrated guidance system based on vision. Its structural forms mainly include platform type and strapdown type. Among them, the platform type is more mature and widely used, but it has problems such as difficulty in miniaturization, high cost, complex structure, and small instantaneous field of view. Although the strapdown seeker has a lower cost, it has problems such as small detection range, easy loss of targets, and insufficient detection accuracy.

[0005] In order to solve the above problems, this application proposes an algorithm-based multi-eye strapdown multi-source image recognition device. Summary of the invention

[0006] The purpose of the present invention is to provide an algorithm-based multi-eye strapdown multi-source image recognition device, which solves the problems in the prior art by combining the technical features of unmanned aerial vehicles with the field of view features of falcons.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] The algorithm-based multi-eye strapdown multi-source image recognition device comprises: a strapdown integrated guidance module, the strapdown integrated guidance module is installed on the image recognition device through its own gyroscope, the strapdown integrated guidance module comprises: a Hi 3559A processing module, the Hi The 3559A processing module integrates multi-source signal fusion, image information processing, control module, computing module, and multi-core heterogeneous processing module; a strapdown integrated seeker, which can be independently detached and installed on the identification device. As an independent detachable and separately used product original, the strapdown integrated seeker can effectively reduce the production cost, secondary development cost, and use cost of the current guidance system. The strapdown integrated seeker includes: a strapdown seeker, which is used to track the target; an optoelectronic seeker, which is used for precision guidance; a heterogeneous imaging sensor, which is used for the optimization of the strapdown integrated guidance module; multi-source heterogeneous detection and tracking, which refers to the imaging system with multiple lenses, multi-source information sensors, infrared, and visible light information sensors; a telephoto camera and two wide-angle cameras symmetrically distributed on both sides of the telephoto camera, and an infrared camera;

[0009] In the above technical scheme, firstly, the multi-source image reconnaissance demand of UAV is taken as input, and the design technology of the coordination between the multi-source image reconnaissance demand of UAV and the workflow is studied to establish a generalized design model of strapdown optoelectronic guidance head that conforms to the use process of conventional unmanned aerial vehicles. The flight dynamics model and the initial design of the heterogeneous imaging sensor are taken as input to carry out the coupling design of the multi-source image reconnaissance process and the detection range of the UAV, and the reconnaissance and attack trajectories as well as the optimized optical axis distribution and field of view angle of the heterogeneous imaging sensor are obtained. The optical axis distribution and field of view angle of the heterogeneous imaging sensor are iteratively optimized and designed through the multi-source heterogeneous simultaneous detection and tracking technology, and the design result that meets the use requirements is finally obtained;

[0010] Combined with the UAV reconnaissance and detection workflow, a detailed analysis is conducted on the constraints of optical path design, sensitive unit scanning process, digital processing, etc. caused by demand differences, so as to establish a design and processing model; combined with the motion characteristics and expression features of small and weak targets, a study of multi-source heterogeneous information fusion processing algorithms is carried out. Improve the reliability of synchronous detection and tracking of small and weak targets in complex environments, and reduce the probability of target loss caused by multi-lens switching; optimize the iterative heterogeneous imaging sensor optical axis distribution and field of view as input, design each imaging sensor in a compact and miniaturized manner, optimize the multi-channel image processing software framework, adjust the front-end optical path balance during multi-channel processing, and complete the integrated design and testing of the guidance system.

[0011] The Hi 3559A processing module can comprehensively process image information, IMU information, airspeed meter information, global satellite navigation information, etc., and control the body through motors and servos to achieve complete bionic heterogeneous guidance functions.

[0012] The intelligent algorithms of deep learning in multi-source heterogeneous detection and tracking are target tracking technology based on infrared and visible light feature fusion, small target detection technology for wide-angle cameras, and long-term tracking technology for sequence images. The target tracking technology based on the features of infrared and visible light images effectively utilizes multi-source input and can achieve accurate target tracking in all weather conditions.

[0013] The visible light infrared image can distinguish the target from its background based on the radiation difference.

[0014] The multi-source heterogeneous imaging sensors can achieve simultaneous tracking of targets through their respective characteristics.

[0015] Beneficial effects of the present invention:

[0016] 1. The cost of the present invention through multiple high-resolution lenses and ISP modules is much lower than that of the gyro-stabilized platform based on inertial sensors, so the overall consumables cost is much lower than that of the platform seeker within the same sensing range;

[0017] 2. The platform seeker itself needs high-precision gyro and sensor parameter correction, which makes the manufacturing process more complicated. The strapdown guidance system can be installed in any direction and only needs complete self-initialization at power-on to ensure the normal operation of the guidance system, thus significantly reducing the manufacturing cost.

[0018] 3. The platform seeker usually has built-in motors, gyroscopes, servo control systems and other structures. This structure needs to perform some data interaction when connected to the flight platform to achieve the best use effect. This increases the R&D cost brought about by secondary development. The guidance system proposed in this solution can work independently and undergo secondary development, so the overall R&D cost is lower;

[0019] 4. Due to its delicate guidance and control mechanical mechanism, the platform seeker is usually severely damaged after completing a single flight or simulation mission. Its overload resistance is weak and it is difficult to recover for secondary use, which brings higher costs for research and development and use. However, this solution can fix the guidance system and the processing board during installation and add protection and buffer mechanisms, which can increase the possibility of equipment recovery and secondary use and reduce the overall use cost during cluster testing;

[0020] In summary, compared with the platform guidance system with the same sensing range, the present invention can significantly reduce material costs, research and development costs, production costs, use costs and other costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a composition diagram of the strapdown integrated guidance system of the present invention;

[0023] Figure 2 This is a flow chart of information processing of the image detection device in the present invention;

[0024] Figure 3 This is the schematic diagram of the bionic heterogeneous strapdown guidance system based on H i3559A;

[0025] Figure 4 This is a schematic diagram of a multi-source intelligent algorithm based on deep learning;

[0026] Figure 5 This is a schematic diagram of the field of view combining the characteristics of the drone and the falcon's monocular imaging structure;

[0027] Figure 6 It is a schematic diagram of infrared integration technology based on complementary imaging information;

[0028] Figure 7 This is a schematic diagram of the target recognition and detection algorithm;

[0029] Figure 8 This is the tracker structure diagram; DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] As shown in the figure:

[0032] The present embodiment provides an algorithm-based multi-eye strapdown multi-source image recognition device, including: a strapdown integrated guidance module, which is installed on the image recognition device through its own gyroscope, and the strapdown integrated guidance module includes: a Hi 3559A processing module, which integrates multi-source signal fusion, image information processing, control module, operation module, and multi-core heterogeneous processing module; a strapdown integrated guidance head, which can be independently detached and installed on the recognition device, and the strapdown integrated guidance head includes: a strapdown guidance head, which is used to track the target; an optoelectronic guidance head, which is used for precision guidance; a heterogeneous imaging sensor, which is used for the optimization of the strapdown integrated guidance module; multi-source heterogeneous detection and tracking, and the multi-source heterogeneous detection and tracking module refers to an imaging system with multiple lenses, multi-source information sensors, infrared, and visible light information sensors, so that it can meet the current multi-class computer vision field algorithms and improve the applicability of the aircraft in complex environments;

[0033] Specifically, in order to effectively process the input image information and run the intelligent algorithm based on deep learning. They are target tracking technology based on the fusion of infrared and visible light features, small target detection technology for wide-angle cameras, and long-term tracking technology for sequential images. The target tracking technology based on the features of infrared and visible light images effectively utilizes multi-source input and can achieve accurate target tracking in all weather conditions. As for the problem of the relatively smaller target size caused by the wide-angle camera, a small target detection method based on the characteristics of dynamic and static targets is proposed to improve the detection accuracy and detection effect of small targets. In addition, on the basis that the network can effectively identify the target, in order to achieve long-term target tracking, a long-term tracking technology for sequential images is proposed, which combines the tracking and detection methods, and can achieve target re-capture when the aircraft shakes violently, yaws or rolls, and achieve long-term target tracking.

[0034] As an embodiment of the present invention, Figure 2 As shown in the figure, considering the all-weather use of micro-aircraft, in the process of integrating and implementing this technology, low-light imaging sensors should be used instead of general visible light imaging sensors. Considering the low cost issue, a forward-looking infrared sensor is used to provide the target's heat source information during the target search and target attack phases. At the same time, considering the massive amount of information collected by multiple sensors working simultaneously, higher requirements are placed on the core processing chip of the detection device. This technology intends to use a multi-core heterogeneous processing chip that can simultaneously process multi-channel visual sensor data as the computing core of the detection device to achieve the design indicators.

[0035] As an embodiment of the present invention, the Hi3559A processing module can comprehensively process image information, IMU information, airspeed meter information, global satellite navigation information, etc., and control the body through motors and steering gears to achieve a complete bionic heterogeneous guidance function;

[0036] Specifically, Figure 3 As shown in the figure, the proposed multi-source heterogeneous strapdown guidance system schematic diagram, the core processor of the module is composed of a quad-core CPU based on the ARM architecture, the GPU uses the embedded GPU chip of NVID IA NX, and a 6-way camera input interface and a 12-line M IPI interface are set up. A single data channel can provide 1.5GB / s image data transmission bandwidth for the multi-source heterogeneous structure. In addition, the module adopts an external ISP method to integrate multi-source image stitching, gamma correction, black level compensation, lens correction, bad pixel correction, anti-mosaic color interpolation, denoising, automatic white balance, color correction, color space correction, color noise reduction, electronic image stabilization, gyro image stabilization, dynamic compensation and other functions, so that the ISP module can provide sufficient computing power and transmission bandwidth for the guidance system. In addition, the CPU and GPU modules selected by the mainboard can support the various algorithms proposed in the first two key technologies, support the deployment of deep learning algorithms, and also provide a platform intelligent data interface.

[0037] As an implementation method of the present invention, the intelligent algorithms for deep learning in multi-source heterogeneous detection and tracking are target tracking technology based on infrared and visible light feature fusion, small target detection technology for wide-angle cameras, and long-term tracking technology for sequence images. The target tracking technology based on the features of infrared and visible light images effectively utilizes multi-source input and can achieve accurate target tracking in all weather conditions;

[0038] As an embodiment of the present invention, the visible light infrared image can distinguish the target from its background based on the radiation difference;

[0039] Specifically, visible infrared images work well in all-weather and all-day / night conditions. In contrast, visible images can provide texture details with high spatial resolution and clarity in a manner consistent with the human visual system. The two types of images are fused to combine the advantages of thermal radiation information in infrared images and detailed texture information in visible light images.

[0040] As an embodiment of the present invention, the multi-source heterogeneous imaging sensors can achieve simultaneous tracking of the target through their respective characteristics;

[0041] Specifically, this technology takes infrared integration technology as the starting point, and proposes a feature-enhanced target tracking network based on the twin network and feature-level image fusion method to achieve an all-weather, high-accuracy target tracking method. The specific methods of this technology are as follows Figure 6 shown.

[0042] As an embodiment of the present invention, the strapdown integrated seeker has a telephoto camera, two wide-angle cameras symmetrically distributed on both sides of the telephoto camera, and an infrared camera;

[0043] Specifically, Figure 5 The figure shows the eagle-eye monocular imaging field of view range obtained by the present invention in combination with a drone and based on the characteristics of the eagle imaging system.

[0044] For the small target detection technology of wide-angle camera, the method combining single sample learning and optical flow can be used to realize accurate recognition of moving targets in the image. The specific process is as follows: Figure 7 As shown;

[0045] Initialization stage: In this stage, the target sample needs to be input and the target sample is intercepted according to its bounding box. The filled image is input into the Backbone module for feature extraction. This module can use a variety of general network structures such as ResNet, ResNeXt, AlexNet, etc. to extract the target weight;

[0046] In the working stage, the images between two consecutive frames are firstly extracted using the sparse LK optical flow method to obtain the positions of each bounding box in the scene. Then, the number of targets obtained is used as the number of categories required by the K-means algorithm to perform clustering to obtain the center position and size of each target. After the features at the target are extracted, they are compared with the sample features for convolution similarity, and the target position with the highest similarity is used as the final position point using NMS to complete the final target detection.

[0047] The long-term tracking technology of sequence images first borrows the SimSTA tracker structure. After deep feature extraction, the extracted features are re-tested twice to determine whether the extracted features are credible, and the scores of the two comparison functions are summed to obtain the final confidence score. At the same time, the target motion trajectory evaluation and spatiotemporal feature constraint methods are introduced to effectively prevent the tracker from failing when the aircraft is blocked by some objects. Its network structure is as follows: Figure 8 As shown;

[0048] According to the historical trajectory of the aircraft in the time domain, the possible location of the current aircraft is predicted. When the re-detection is completed, the corresponding image in the cache is re-tracked using the frame tracking method, and the predicted position is compared with the features at the tracking position, and the tracking result is updated to obtain the current optimal solution, thus realizing long-term tracking of the flying target.

[0049] It can be understood that the present invention imitates the detection and attack mechanism of falcons against the ground, and is combined with the elements of the UAV's intelligent target detection and tracking, guidance information extraction, guidance law, control law, etc., to increase the reconnaissance area, stably track the target, and improve the strike accuracy; based on the strapdown guidance method, an optical path and processing model coupled with the reconnaissance route and attack trajectory is designed to solve the current problems of insufficient information acquisition of the strapdown guidance system and easy loss of targets in the guidance stage; using a multi-task network model, an integrated target recognition and tracking method is realized to solve the problems of slow recognition speed, small perception range, and difficult target search.

[0050] In the description of the present invention, unless otherwise specified, "plurality" means two or more than two; it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inside", "around" and the like indicate orientation or positional relationship, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0051] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Algorithm-based multi-eye strapdown multi-source image recognition device, characterized in that: include: A strapdown integrated guidance module, wherein the strapdown integrated guidance module is installed on the image recognition device through its own gyroscope, and the strapdown integrated guidance module comprises: Hi 3559A processing module, the Hi 3559A processing module integrates multi-source signal fusion, image information processing, control module, computing module, and multi-core heterogeneous processing module; The strapdown integrated seeker can be independently detachably mounted on the identification device, and the strapdown integrated seeker includes: A strapdown seeker, wherein the strapdown seeker is used to track a target; An optoelectronic guidance head, wherein the optoelectronic guidance head is used for precision guidance; A heterogeneous imaging sensor, wherein the heterogeneous imaging sensor is used for optimization of a strapdown integrated guidance module; Multi-source heterogeneous detection and tracking, the multi-source heterogeneous detection and tracking module refers to the imaging system multi-lens, multi-source information sensor, infrared, visible light information sensor; A telephoto camera and two wide-angle cameras symmetrically distributed on both sides of the telephoto camera, as well as an infrared camera.

2. The algorithm-based multi-eye strapdown multi-source image recognition device according to claim 1, characterized in that: The Hi 3559A processing module is capable of comprehensively processing image information, IMU information, airspeed meter information, global satellite navigation information, etc., and controls the aircraft body through motors and servos to achieve a complete bionic heterogeneous guidance function.

3. The algorithm-based multi-eye strapdown multi-source image recognition device according to claim 1, characterized in that: The intelligent algorithms of deep learning in multi-source heterogeneous detection and tracking are target tracking technology based on infrared and visible light feature fusion, small target detection technology for wide-angle cameras, and long-term tracking technology for sequence images. The target tracking technology based on the features of infrared and visible light images effectively utilizes multi-source input and can achieve accurate target tracking in all weather conditions.

4. The algorithm-based multi-eye strapdown multi-source image recognition device according to claim 3, characterized in that: Through the use of weak target detection technology tailored to the characteristics of dynamic and static targets, the accuracy and effect of small target detection are ensured, and long-term target tracking is achieved based on the network's ability to effectively identify targets.

5. The algorithm-based multi-eye strapdown multi-source image recognition device according to claim 3, characterized in that: Combining tracking and detection methods can achieve target recapture and long-term target tracking when the aircraft shakes, yaws or rolls violently.

6. The algorithm-based multi-eye strapdown multi-source image recognition device according to claim 1, characterized in that: The visible light infrared image can distinguish the target from its background based on the radiation difference.

7. The algorithm-based multi-eye strapdown multi-source image recognition device according to claim 1, characterized in that: The multi-source heterogeneous imaging sensors can achieve simultaneous tracking of targets through their respective characteristics.

8. The algorithm-based multi-eye strapdown multi-source image recognition device according to claim 7, characterized in that: The multi-source heterogeneous imaging sensor takes infrared integration technology as a starting point, and proposes a feature-enhanced target tracking network based on a twin network and a feature-level image fusion method to achieve all-weather, high-accuracy target tracking.

Images