Ultra-long-distance visible infrared multi-target ecological monitoring system

Abstract

The invention discloses an ultra-long-distance visible infrared multi-target ecological monitoring system, which realizes beyond-visual-range all-weather scene monitoring, moving target capturing, and rapid identification and positioning of moving targets, hot spots and the like through a long-zoom high-sensitivity infrared detection system, a long-zoom high-resolution visible light detection system and a dynamic target autonomous tracking algorithm. And the monitoring requirements of hour-level time resolution, sub-meter-level spatial resolution and sub-pixel-level high-precision geometric positioning are met. And reference information is provided for all-day key ecological targets, human activities and ecological abnormity monitoring. And the business operation mode requirements of accurate monitoring, anomaly analysis, intelligent early warning and quick response of ecological protection are effectively met.

Description

Technical Field

[0001] The invention relates to an ultra-long-distance visible infrared multi-target ecological monitoring system, belonging to the technical field of ecological remote sensing monitoring. Background Art

[0002] Wildlife is an important biological group in ecosystem research and protection management. It plays a significant role in regulating the structure and function of ecosystems and maintaining the healthy balance of ecosystems. Among the many monitoring data, image data has attracted more and more attention from researchers due to its intuitive, efficient and rich content, and has become an important part of wildlife monitoring. Scientific monitoring data is the prerequisite for wildlife research, protection and management decisions, but due to the limitations of traditional monitoring technology, the diversity characteristics of wildlife and their balance mechanism with the environment and ecosystem have not received sufficient attention and research. The current wildlife image acquisition methods mainly include manual acquisition and intelligent camera acquisition, but they all have their own defects, such as long acquisition cycle and high labor cost.

[0003] Wildlife monitoring is a hot research area at home and abroad, and there is a lack of research on long-term dynamic monitoring of wild animals in China. In summary, there are mainly the following ways to monitor wild animals: 1. Artificial field surveys, specific methods include interception method, fixed-point counting method, encounter rate method, questionnaire survey method, etc. This method has a limited monitoring range, high labor intensity, high risk, and cannot achieve all-weather monitoring; 2. Global Positioning System (GPS) collar method. Compared with traditional radio collars, GPS collars provide better spatial resolution and determine location under wider time and space conditions, but this monitoring method is not user-friendly and cannot obtain animal image information; 3. Offline near-infrared camera method. The image data taken by the infrared camera is stored in a local large-capacity SD memory card. Researchers have to go to the site to retrieve the S at regular intervals. D card, read out data, this method has the disadvantages of data collection lag, long monitoring cycle, high labor intensity, high risk, etc. Some infrared cameras are combined with 3G network, and the collected image data is sent to the user's mobile phone through 3G network. This method relies on 3G mobile phone signal coverage, and there is no mobile phone signal in the core area of ​​some national nature reserves; 4. Wireless camera, wireless broadband microwave transmission image, this method requires city power supply, and because of the high deployment, it is easy to be blocked by the canopy, it is difficult to obtain images of wild animals in the forest, and the cost of this equipment is relatively high; 5. Satellite remote sensing monitoring, satellite remote sensing comprehensive monitoring is difficult to accurately measure local microscopic information, and there is a scanning cycle, and it cannot be monitored in real time. It can be seen that the traditional wildlife protection monitoring method is not conducive to researchers to fully, accurately and timely grasp the status of wildlife resources and the dynamic changes of wildlife resources. Summary of the invention

[0004] The technical problem solved by the present invention is: in view of the defects of the wild animal monitoring method in the current prior art, an ultra-long-distance visible infrared multi-target ecological monitoring system is proposed.

[0005] The present invention solves the above technical problems by the following technical solutions:

[0006] An ultra-long-range visible infrared multi-target ecological monitoring system includes a visible light camera module, a medium-wave infrared tracking camera module, a heavy-load two-dimensional precision turntable, a servo control module, and a target real-time detection and tracking processing unit, wherein:

[0007] The visible light camera module uses a long focal length zoom fog-penetrating lens to conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during the daytime, and sends visible band image data to the target real-time detection and tracking processing unit;

[0008] The medium-wave infrared tracking camera module uses a large-aperture, long-focal-length zoom lens to conduct wide-area surveillance and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring day and night, and sends infrared band image data to the target real-time detection and tracking processing unit;

[0009] The heavy-loaded two-dimensional precision turntable is used to carry the visible light camera module and the medium-wave infrared tracking camera module. According to the turntable control instructions sent by the servo control unit, the visible light camera module and the medium-wave infrared tracking camera module are controlled to perform azimuth rotation control and pitch rotation control according to the preset angles and specified positions respectively;

[0010] The servo control unit generates a turntable control instruction and sends it to the heavy-loaded two-dimensional precision turntable according to the received positioning information of the dynamic target, the working status of the visible light camera module and the medium-wave infrared tracking camera module; at the same time, it receives new target information in real time and adjusts the turntable control instruction in real time; at the same time, it sends a cooling control instruction to the medium-wave infrared tracking camera module to provide a cooling function for the medium-wave infrared tracking camera module;

[0011] The target real-time detection and tracking processing unit receives the dual-band image data sent by the visible light camera module and the medium-wave infrared tracking camera module, performs infrared and visible light image registration and image compensation processing at different resolutions, and autonomously tracks and identifies and locates the dynamic targets in the image content according to the image content after registration and image compensation processing, and displays the positioning information of the dynamic targets to the user and sends it to the servo control unit at the same time.

[0012] The visible light camera module includes a visible window, a visible lens, a visible detector, and a visible video circuit unit;

[0013] The visible lens uses a long focal length zoom fog-penetrating lens, and the visible detector has a sensitivity of 2k×2k pixels, which can conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during daytime;

[0014] The visible window is used to protect the visible lens from contamination. The visible detector collects data of the visible band image collected by the visible lens and sends it to the target real-time detection and tracking processing unit through the visible video circuit unit.

[0015] The medium-wave infrared tracking camera module includes a medium-wave infrared window, a medium-wave infrared lens, a medium-wave infrared detector, a medium-wave infrared refrigerator, a refrigerator drive unit, and a medium-wave infrared video circuit unit;

[0016] The medium-wave infrared lens uses a large-aperture, long-focal-length zoom lens, and the medium-wave infrared detector uses a cooled, 640×512-pixel, high-sensitivity medium-wave infrared detector, which can conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during the day and night.

[0017] The medium-wave infrared window is used to protect the medium-wave infrared lens from contamination. The medium-wave infrared detector collects data of the infrared band image collected by the medium-wave infrared lens and sends it to the target real-time detection and tracking processing unit through the medium-wave infrared video circuit unit;

[0018] The medium-wave infrared detector is cooled by a medium-wave infrared refrigerator and driven and controlled by a refrigerator drive unit; the cooling control command sent by the refrigerator drive unit is issued by a servo control unit.

[0019] The heavy-load two-dimensional precision turntable includes an azimuth rotation mechanism, a pitch rotation mechanism, a turntable base, and a load mounting closed structure;

[0020] The visible light camera module and the medium-wave infrared tracking camera module are both arranged in a payload-mounted enclosed structure; the azimuth rotation mechanism and the pitch rotation mechanism are arranged on the turntable base, and are used to realize the positioning rotation of the preset angle and the specified position of the circumferential scanning according to the turntable control instructions sent by the servo control unit; the pitch rotation mechanism is used to realize the positioning rotation of the specified pitch angle range and the specified position according to the turntable control instructions sent by the servo control unit.

[0021] The azimuth rotation mechanism and the pitch rotation mechanism form a double-frame structure on the turntable base through the azimuth frame and the pitch frame. The azimuth frame and the pitch frame are both arranged on the turntable base through mounting flanges. The azimuth frame is correspondingly provided with an azimuth axis system, and the pitch frame is correspondingly provided with a pitch axis system. Both are driven by DC torque motors to realize the positioning rotation of the azimuth frame in the circumferential direction and the pitch frame in the pitch direction at preset angles and designated positions.

[0022] The load installation closed structure is a waterproof and dustproof structure, and is internally provided with a temperature control unit and a high-precision measurement unit; the temperature control unit is provided with a heating circuit and a temperature measurement circuit to ensure the temperature environment in the load installation closed structure, and to protect the visible window and the medium-wave infrared window during non-working hours;

[0023] The high-precision measurement unit is used to measure the turntable attitude and rotation angular velocity of the azimuth rotation mechanism and the pitch rotation mechanism, and the measured data is used to locate the target and determine the load's wind disturbance resistance capability.

[0024] Dust covers and electric wipers are provided outside the visible window and the medium-wave infrared window. The dust covers have a switch function and are closed during non-working hours to protect the optical windows. The electric wipers are used to clean the visible window and the medium-wave infrared window in rainy and foggy weather to ensure imaging quality.

[0025] The target real-time detection and tracking processing unit runs an image acquisition processing program, and performs registration processing and target detection processing on the latest two frames of images in the dual-band data respectively; the ecological monitoring system also includes a main control machine data display and storage unit. After the target real-time detection and tracking processing unit receives the original image or the image of the marked detection target, it displays it in the main window and thumbnail window on the main control machine data storage unit. The main window displays a large field of view infrared image by default, and the thumbnail window displays a visible image.

[0026] The data display storage unit of the host control machine switches the display window through user control, and realizes the storage and playback processing of local image data according to user needs; if image data playback is required, the data cache is read from the local image data at fixed intervals.

[0027] The target real-time detection and tracking processing unit runs an image acquisition processing program and performs image registration on the dual-band image data, and performs image motion estimation processing and image motion compensation processing on the dual-band image data, specifically:

[0028] The image motion parameters of the dual-band image data are calculated using the block grayscale matching method, and a reference image is selected based on the obtained image motion information. The remaining dual-band image data to be registered are compared with the reference image. The dual-band image data to be registered are transformed according to the comparison results to obtain the image content after registration and image compensation.

[0029] The image content after the registration and image compensation processing is processed by frame difference detection to obtain differential image data, and the differential image data is segmented by grayscale threshold to select suspected targets, and the suspected targets are subjected to morphological processing and Blob analysis filtering processing, and the target shape size, speed size and speed change are used as constraints to calculate the association cost matrix, and the trajectory association is performed through Kalman filtering, and the trajectory point position, speed, and observation number information are updated, and the trajectory is confirmed according to the target related information to eliminate the target trajectory that is incorrectly associated or lost for a long time, and determine the stable tracking target.

[0030] The advantages of the present invention compared with the prior art are:

[0031] (1) The present invention provides an ultra-long-range visible infrared multi-target ecological monitoring system that can cooperate with wireless image sensor network technology to monitor wildlife images and achieve all-day wide-area online ecological monitoring. It can provide excellent views in difficult conditions such as complete darkness, dazzling light, and persistent haze, and accurately capture video and image records of any wildlife encounters. High-definition thermal imaging retains the details of moving objects to the greatest extent, and based on the composite target recognition algorithm, it achieves all-day, highly environmentally adaptable ecological target monitoring capabilities. It can accurately detect wild animals, collect high-definition images, and the detection range can cover 360° horizontally, and achieve effective capture, which can greatly improve the ecosystem monitoring capabilities;

[0032] (2) The present invention realizes ultra-long-range visible infrared multi-target ecological monitoring payload through camera modules and tracking modules. The cameras all adopt a transmission imaging system to ensure high imaging quality. The moving target recognition algorithm based on morphological processing is used to identify moving targets under complex backgrounds, so that the system has wide-area target monitoring and multi-target tracking functions in various application environments. The system can be used for wildlife monitoring in nature reserves, forest fire monitoring, coastal and border security and alert, etc. The development of this project can solve the current problems of low accuracy of satellite remote sensing monitoring of ecological elements, few special equipment for ecological remote sensing monitoring, and low performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the multi-objective ecological monitoring system provided for the invention;

[0034] Figure 2 A schematic diagram of the multi-objective ecological monitoring process provided for the invention;

[0035] Figure 3 A schematic diagram of image acquisition and display provided for the invention;

[0036] Figure 4 A schematic diagram of the image registration process provided for the invention;

[0037] Figure 5A schematic diagram of the target detection process provided for the invention;

[0038] Figure 6 A schematic diagram of the trajectory association process provided for the invention;

[0039] Figure 7 A schematic diagram of a multi-target polling tracking process provided for the invention;

[0040] Figure 8 A schematic diagram of the fixed-point scanning panoramic image stitching algorithm provided for the invention; DETAILED DESCRIPTION

[0041] An ultra-long-range visible infrared multi-target ecological monitoring system, such as Figure 1 As shown in the figure, through the long-zoom high-sensitivity infrared detection system, the long-zoom high-resolution visible light detection system and the dynamic target autonomous tracking algorithm, over-the-horizon all-weather scene monitoring, moving target capture, and rapid identification and positioning of moving targets and hot spots are achieved. It meets the monitoring needs of hourly time resolution, sub-meter spatial resolution, and sub-pixel high-precision geometric positioning. It provides reference information for all-day key ecological targets, human activities, and ecological anomaly monitoring. It effectively solves the business operation mode requirements of precise monitoring of ecological protection, anomaly analysis, intelligent early warning, and rapid response.

[0042] The ultra-long-distance visible infrared multi-target ecological monitoring system includes a visible light camera module, a medium-wave infrared tracking camera module, a heavy-load two-dimensional precision turntable, a servo control module, and a target real-time detection and tracking processing unit, among which:

[0043] The visible light camera module uses a long focal length zoom fog-penetrating lens to conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during the daytime, and sends visible band image data to the target real-time detection and tracking processing unit;

[0044] The medium-wave infrared tracking camera module uses a large-aperture, long-focal-length zoom lens to conduct wide-area surveillance and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring day and night, and sends infrared band image data to the target real-time detection and tracking processing unit;

[0045] The heavy-loaded two-dimensional precision turntable is used to carry the visible light camera module and the medium-wave infrared tracking camera module. According to the turntable control instructions sent by the servo control unit, the visible light camera module and the medium-wave infrared tracking camera module are controlled to perform azimuth rotation control and pitch rotation control according to the preset angles and specified positions respectively;

[0046] The servo control unit generates a turntable control instruction and sends it to the heavy-loaded two-dimensional precision turntable according to the received positioning information of the dynamic target, the working status of the visible light camera module and the medium-wave infrared tracking camera module; at the same time, it receives new target information in real time and adjusts the turntable control instruction in real time; at the same time, it sends a cooling control instruction to the medium-wave infrared tracking camera module to provide a cooling function for the medium-wave infrared tracking camera module;

[0047] The target real-time detection and tracking processing unit receives the dual-band image data sent by the visible light camera module and the medium-wave infrared tracking camera module, performs infrared and visible light image registration and image compensation processing at different resolutions, and autonomously tracks and identifies and locates the dynamic targets in the image content according to the image content after registration and image compensation processing, and displays the positioning information of the dynamic targets to the user and sends it to the servo control unit at the same time.

[0048] The visible light camera module includes a visible window, a visible lens, a visible detector, and a visible video circuit unit;

[0049] The visible lens uses a long focal length zoom fog-penetrating lens, and the visible detector has a sensitivity of 2k×2k pixels, which can conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during daytime;

[0050] The visible window is used to protect the visible lens from contamination. The visible detector collects data of the visible band image collected by the visible lens and sends it to the target real-time detection and tracking processing unit through the visible video circuit unit.

[0051] The medium-wave infrared tracking camera module includes a medium-wave infrared window, a medium-wave infrared lens, a medium-wave infrared detector, a medium-wave infrared refrigerator, a refrigerator drive unit, and a medium-wave infrared video circuit unit;

[0052] The medium-wave infrared lens uses a large-aperture, long-focal-length zoom lens, and the medium-wave infrared detector uses a cooled, 640×512-pixel, high-sensitivity medium-wave infrared detector, which can conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during the day and night.

[0053] The medium-wave infrared window is used to protect the medium-wave infrared lens from contamination. The medium-wave infrared detector collects data of the infrared band image collected by the medium-wave infrared lens and sends it to the target real-time detection and tracking processing unit through the medium-wave infrared video circuit unit;

[0054] The medium-wave infrared detector is cooled by a medium-wave infrared refrigerator and driven and controlled by a refrigerator drive unit; the cooling control command sent by the refrigerator drive unit is issued by a servo control unit.

[0055] The heavy-load two-dimensional precision turntable includes an azimuth rotation mechanism, a pitch rotation mechanism, a turntable base, and a load installation closed structure;

[0056] The visible light camera module and the medium-wave infrared tracking camera module are both arranged in a payload-mounted enclosed structure; the azimuth rotation mechanism and the pitch rotation mechanism are arranged on the turntable base, and are used to realize the positioning rotation of the preset angle and the specified position of the circumferential scanning according to the turntable control instructions sent by the servo control unit; the pitch rotation mechanism is used to realize the positioning rotation of the specified pitch angle range and the specified position according to the turntable control instructions sent by the servo control unit.

[0057] The azimuth rotation mechanism and the pitch rotation mechanism form a double-frame structure on the turntable base through the azimuth frame and the pitch frame. The azimuth frame and the pitch frame are both set on the turntable base through mounting flanges. The azimuth frame is correspondingly provided with an azimuth axis system, and the pitch frame is correspondingly provided with a pitch axis system. Both are driven by DC torque motors to achieve positioning rotation at preset angles and specified positions in the circumferential direction of the azimuth frame and the pitch direction of the pitch frame.

[0058] The load installation closed structure is waterproof and dustproof, and is equipped with a temperature control unit and a high-precision measurement unit. The temperature control unit is equipped with a heating circuit and a temperature measurement circuit to ensure the temperature environment inside the load installation closed structure and protect the visible window and medium-wave infrared window during non-working hours.

[0059] The high-precision measurement unit is used to measure the turntable attitude and rotation angular velocity of the azimuth rotation mechanism and the pitch rotation mechanism, and the measured data is used to locate the target and determine the load's wind disturbance resistance capability.

[0060] Dust covers and electric wipers are installed outside the visible window and medium-wave infrared window. The dust cover has a switch function and is closed during non-working hours to protect the optical window. The electric wipers are used to clean the visible window and medium-wave infrared window in rainy and foggy weather to ensure imaging quality.

[0061] The target real-time detection and tracking processing unit runs the image acquisition processing program. The acquisition and display design is as follows: Figure 3 As shown, the registration processing and target detection processing of the latest two frames of images are performed respectively in the dual-band data; the ecological monitoring system also includes a main control machine data display storage unit. After the target real-time detection and tracking processing unit receives the original image or the image of the marked detection target, it is displayed in the main window and thumbnail window on the main control machine data storage unit. The main window displays a large field of view infrared image by default, and the thumbnail window displays a visible image.

[0062] The data display storage unit of the main control machine switches the display window through user control, and realizes the storage and playback processing of local image data according to user needs; if image data playback is required, the data cache is read from the local image data at fixed intervals.

[0063] like Figure 2 As shown, the target real-time detection and tracking processing unit runs the image acquisition processing program and performs image registration on the dual-band image data, and performs image motion estimation processing and image motion compensation processing on the dual-band image data, specifically:

[0064] The image motion parameters of the dual-band image data are calculated using the block grayscale matching method, and a reference image is selected based on the obtained image motion information. The remaining dual-band image data to be registered are compared with the reference image. The dual-band image data to be registered are transformed according to the comparison results to obtain the image content after registration and image compensation.

[0065] The image content after registration and image compensation is processed by frame difference detection to obtain differential image data. The differential image data is segmented by grayscale threshold to select suspected targets. The suspected targets are subjected to morphological processing and Blob analysis filtering. The target shape size, speed size and speed change are used as constraints to calculate the association cost matrix. The trajectory is associated through Kalman filtering, and the trajectory point position, speed and number of observations are updated. The trajectory is confirmed according to the target-related information to eliminate the target trajectory that is incorrectly associated or lost for a long time, and to determine a stable tracking target.

[0066] The following is further described in conjunction with the accompanying drawings and preferred embodiments of the specification:

[0067] In the current embodiment, the ultra-long-range visible infrared multi-target ecological monitoring payload is composed of a visible light camera module, a medium-wave infrared tracking camera module, a heavy-duty two-dimensional precision turntable, and a target detection and tracking module. Both visible and infrared cameras use a transmission imaging system to ensure high imaging quality, and use a moving target recognition algorithm based on morphological processing to identify moving targets under complex backgrounds. It combines intelligent detection algorithms and high-precision tracking control algorithms to achieve an integrated design of detection, inspection, checking, and tracking. The product is loaded on a heavy-duty two-dimensional turntable, which effectively isolates the influence of airflow, and has the ability to drive the detection optical axis to track mobile targets, so that the measured target is always within the detection field of view. The system has wide-area target monitoring and multi-target tracking functions for a variety of application environments.

[0068] The specific design of the embodiment is as follows:

[0069] The visible light camera module consists of a window, lens, detector and video circuit unit. It uses a high-sensitivity visible near-infrared detector with 2k×2k pixels and a long-focal-length zoom fog-penetrating lens to achieve wide-area monitoring and high-definition recognition of key ecological targets (birds, large wild animals on the ground), human activities (human interference), and ecological anomaly monitoring (vegetation, etc.) during the day.

[0070] The medium-wave infrared tracking camera module consists of an infrared window, lens, detector, refrigerator, video circuit unit and refrigerator drive unit. It uses a cooled 640×512 pixel high-sensitivity medium-wave infrared detector and a large-aperture long-focal-length zoom lens to achieve wide-area monitoring and high-definition identification of key ecological targets (birds, large wild animals on the ground), human activities (human interference), and ecological anomaly monitoring (vegetation, etc.) day and night.

[0071] The heavy-loaded two-dimensional precision turntable consists of an azimuth rotation mechanism, a pitch rotation mechanism, and a base and load mounting closed structure. The azimuth rotation mechanism can achieve 360° rapid scanning and rapid positioning and pointing to a specified position. The pitch mechanism has an adjustment range of -15° to +70° and rapid positioning and pointing to a specified position, meeting the needs of large-scale target search and surveillance and rapid tracking of typical targets.

[0072] The enclosed structure is waterproof and dustproof, providing a good working environment for visible and infrared cameras, and meeting the normal working needs of the payload in adverse weather conditions. The high-precision measurement unit can achieve fast and precise measurement of the turntable attitude and angular velocity, which is used to achieve rapid positioning of the target and the payload's ability to resist wind disturbances.

[0073] The temperature control unit is mainly composed of two parts: a heating circuit and a temperature measurement circuit, which ensures that the temperature in the closed load compartment is maintained at a certain level, ensuring the normal operation of the circuit products, while avoiding frost and condensation that may cause product performance degradation.

[0074] The dust cover has a switch function and can be closed during non-working hours to protect the optical window. The electric wiper can be used to clean the window in rainy and foggy weather to ensure that the camera has good imaging quality.

[0075] The servo control unit is used to precisely control the drive of the high-precision multi-target fast tracking two-dimensional turntable, drive the visible light camera and infrared camera payload azimuth rotation, complete the large field of view imaging movement, and can also change the visual axis direction according to remote instructions to achieve scanning imaging of specific areas. At the same time, it provides high-precision pointing information in the pitch and azimuth directions in real time to provide position information for target positioning. As the carrier of the push-scan imaging device, the optical turntable has a large rotation range, low torque disturbance, and high-speed stability, which are the basis for stable imaging.

[0076] The main functions of the target real-time detection and tracking processing unit include receiving visible and infrared dual-band image data, completing infrared and visible light image stitching and image fusion at different resolutions, and further extracting feature points of images with different resolutions based on traditional image stitching according to the different characteristics of the two images. The difference of the images is then matched to achieve panoramic image registration. Motion target detection and tracking based on infrared and visible light image sequences, such as Figure 5As shown in the figure, the dynamic target autonomous tracking algorithm and the slant range positioning algorithm based on template recognition are combined to realize the rapid identification and positioning of moving targets, hot spots, etc. The thermal radiation characteristics of the target are detected to overcome the influence of lighting and weather conditions on the imaging effect, and all-weather scene monitoring is realized. Infrared data is combined with the dynamic target autonomous tracking algorithm to realize the capture of moving targets beyond the line of sight. Infrared data is combined with the slant range positioning algorithm based on template recognition, and the distribution of control points in the scene is used to realize sub-pixel-level high-precision geometric positioning, and high-precision tracking and positioning of moving targets are realized.

[0077] Furthermore, the visible light camera module consists of a window, a lens, a detector and a video circuit unit. It uses a high-sensitivity visible near-infrared detector with 2k×2k pixels, and cooperates with a long-focal-length zoom fog-penetrating lens to achieve wide-area monitoring and high-definition recognition of key ecological targets (birds, large wild animals on the ground), human activities (human interference), and ecological anomaly monitoring (vegetation, etc.) during the day. Requirement.

[0078] The medium-wave infrared tracking camera module consists of an external window, an infrared lens, an infrared detector, a refrigerator, an infrared video circuit unit and a refrigerator drive unit. It uses a cooled 640×512 pixel high-sensitivity medium-wave infrared detector and a large-aperture long-focal-length zoom lens to achieve wide-area monitoring and high-definition identification of key ecological targets (birds, large wild animals on the ground), human activities (human interference), and ecological anomaly monitoring (vegetation, etc.) day and night.

[0079] The heavy-loaded two-dimensional precision turntable consists of an azimuth rotation mechanism, a pitch rotation mechanism, and a base and load mounting closed structure. The azimuth rotation mechanism can achieve 360° rapid scanning and rapid positioning and pointing to a specified position. The pitch mechanism has an adjustment range of -15° to +70° and rapid positioning and pointing to a specified position, meeting the needs of large-scale target search and surveillance and rapid tracking of typical targets.

[0080] The pitch assembly includes: infrared light window, visible light window, angle measuring element, motor and driving component, pitch frame and pitch axis system. The azimuth assembly includes: angle measuring element, motor and driving component, azimuth frame and azimuth axis system.

[0081] The two-axis two-frame platform consists of an azimuth frame and a pitch frame, and the two axes of the frame are closed-loop controlled by a DC torque motor. The pitch frame is loaded with optical loads, and achieves angular velocity stability and angular rotation under the support of the horizontal axis system. The azimuth frame is connected to the mounting flange, and achieves angular velocity stability and angular rotation under the support of the vertical axis system. A DC torque motor is used as the actuator for azimuth and pitch movements. The heavy-duty turntable adopts an integrated optical turntable with a horizontal U-shaped structure, which integrates the rotation axis systems in both azimuth and pitch directions. The U-shaped frame is equipped with a visible light camera and an infrared camera, and the bottom is fixedly installed with the monitoring tower, which has outstanding advantages such as high stability, low disturbance, and long life. It can meet the following requirements: 360° free rotation in azimuth, a wide range of rotation angles of -15° to 70° in pitch, and no blind spots in the field of view; the azimuth and pitch have a maximum angular velocity of 60° / s and 30° / s 2 The maximum angular acceleration allows for rapid target tracking; the azimuth and pitch stability accuracy of 0.05mrad provides more stable imaging.

[0082] The enclosed structure is waterproof and dustproof, providing a good working environment for visible and infrared cameras and meeting the normal working needs of the payload in adverse weather conditions. The high-precision measurement unit can realize fast and precise measurement of the turntable attitude and angular velocity, which is used to achieve rapid positioning of the target and the ability of the payload to resist wind disturbance. The temperature control unit is mainly composed of a heating circuit and a temperature measurement circuit, which ensures that the temperature in the enclosed payload compartment is maintained at a certain level, ensures the normal operation of the circuit product, and avoids frost and condensation that may cause product performance degradation. The dust cover has a switch function, which can be closed during non-working hours to protect the optical window. The electric wiper can be used to clean the window in rainy and foggy weather to ensure that the camera has good imaging quality.

[0083] The servo control unit is used to precisely control the high-precision multi-target fast tracking two-dimensional turntable, drive the visible camera and infrared camera payload azimuth rotation, complete the large field of view scanning imaging movement, and can also change the visual axis direction according to remote instructions to achieve scanning imaging of specific areas. Provide high-precision pointing information in pitch and azimuth directions in real time, and provide position information for target positioning. As a mounting platform for imaging equipment, the optical turntable has a large rotation range, low torque disturbance, and high-speed stability, which are the basis for stable imaging.

[0084] The real-time target detection and tracking processing unit runs the image acquisition processing program and performs image registration on the dual-band image data. Through the combination of software and hardware, it continuously collects and updates images, displays them on the interface, and saves data according to interactive operations or program logic. The program maintains a dynamically updated image cache queue. In each detection cycle, the latest two frames of images are used in turn for registration, difference, morphological processing, and Blob analysis to preliminarily extract suspected target information. Then, the Kalman filter is used to associate the target trajectory, and the distance cost matrix is ​​calculated to associate the trajectory with the latest target, and the tracking trajectory chain is updated. After confirming the target trajectory, the next step is to start tracking and obtaining evidence or continue scanning imaging according to the selected mode, and continue the detection process after the turntable is in place.

[0085] The processing program uses an independent thread to maintain a set of image cache circular queues for each camera to ensure fixed-period image refresh and temporary storage of spare images to be detected. At fixed intervals, the latest image frame is obtained from the camera and added to the end of the program image cache queue.

[0086] In terms of image display, the original image or the marked image after the target is detected is determined according to the display category identifier. The display identifier is controlled by the main thread detection process. The image display is designed with multiple thumbnail windows and the main display window. The images of two cameras are refreshed and displayed in real time in three thumbnail windows. The main window displays the infrared image with a large field of view by default. You can switch the main display channel by double-clicking the thumbnail window with the left mouse button.

[0087] If you choose to play back local video, you can select the playback file through the interface interaction, the program will update the data source of the thumbnail display internally, and then read data from the video source to the image cache at fixed intervals.

[0088] In case of jitter in continuous multi-frame images, image registration is required before detection to suppress the impact of jitter. Image registration can first achieve stable display of the video and improve the viewing experience; secondly, in order to reduce the static background image motion caused by the platform, it supports subsequent differential and background modeling work; finally, based on the continuous image motion results, the zero position of the platform is estimated under random jitter, so as to accurately obtain the actual angle of the target.

[0089] During the registration process, the amplitude of the platform jitter is much smaller than the monitoring distance, resulting in a relatively small image shift; the image shift is mainly in the horizontal and pitch directions, the overall image information between adjacent frames remains basically unchanged, and the rotation and perspective changes are negligible.

[0090] Image registration can be divided into two steps: image motion estimation and image motion compensation. In this project, the image motion is calculated by using the block grayscale template matching method under the approximation of the adjacent frame translation motion model. Then, based on the image motion information, the image to be registered is transformed relative to the reference image, including whole pixel translation and sub-pixel filter kernel deconvolution. Figure 4As shown;

[0091] The frame difference method is selected as the preliminary detection method to adapt to platform motion and local background changes. For the differential image, the suspected target is selected through grayscale threshold segmentation, and then further filtered through morphological processing and Blob analysis. Again, the target shape size, speed size, and speed change are used as constraints to calculate the association cost matrix, and the trajectory is associated through Kalman filtering, and the trajectory point position, speed, number of observations and other information are updated. Finally, the final trajectory is confirmed by the size, speed and visibility of the target, and the target trajectory that is incorrectly associated or lost for a long time is eliminated.

[0092] When performing trajectory association, the process is as follows Figure 6 As shown. First, a rough detection target array is established based on the Blob analysis results. Then, based on the target array and trajectory array, the cost matrix of trajectory association is calculated based on the distance, speed and scale difference. The trajectory array is updated according to the cost matrix, including updating the trajectory end node, trajectory life and speed information. Trajectories that have been unrelated for a long time are eliminated, and new trajectories are created for unrelated new targets. The tracking life, displacement and speed of all targets are judged, and the targets that are stably tracked are predicted.

[0093] Target tracking requires scheduling the turntable to move at a certain horizontal and vertical speed in the direction of intersection with the moving target so that the target is located exactly in the center of the field of view after a certain period of time. Then, the turntable is adjusted to keep consistent with the target's movement speed for continuous tracking.

[0094] Taking single target tracking as an example, before starting tracking, the pitch angle and azimuth of the current turntable are known, and the pitch angle and azimuth of the current target relative to the turntable zero position, as well as the angular velocity of the target in the horizontal and vertical directions can be obtained according to the detection results. In order to achieve fast tracking, a larger turntable angular velocity is set, and the turntable speed, rotation time and expected intersection position for tracking are calculated, and then the turntable is controlled to execute according to the results.

[0095] Multiple targets may be detected at the same time at a gaze position, and multiple targets need to be polled and tracked, such as Figure 7 As shown in the figure, multi-target tracking can be decomposed into tracking individual targets one by one, and the turntable control method remains unchanged. The main problem lies in the tracking order of multiple targets. Since the possible position and speed of the target are unpredictable, in the absence of target identification features and priority standards, in order to cover more and faster targets to be tracked, the real-time fastest priority principle is adopted.

[0096] Each time the next target is decided, the turntable rotation time required for each target to be tracked is calculated first, and the one with the shortest time is selected as the next tracking target. After each target is tracked, the current turntable direction and other target positions are updated, and then the next round of planning begins. When some targets move very fast, the turntable tracking time is long, and the interactive position is far from the current gaze position. Therefore, a tracking range limit is set. When the next tracking position exceeds the limit, it is skipped. This polling cycle continues until all targets are tracked. Figure 7 As shown;

[0097] Image stitching is a technique that stitches together several overlapping images into a seamless panorama or high-resolution image. Image registration and image fusion are key technologies for image stitching. When the panoramic synthesis input video recording of this project is used, the camera is fixed on a fixed turntable and rotates continuously to the right to form an image, and the video scene keeps moving in one direction. The video is a circular scan of a distant scene by the camera, and the transformation relationship between adjacent images is approximately a translation, and is mainly a horizontal translation. The panoramic stitching algorithm is implemented as follows, and the flow chart is as follows: Figure 8 As shown:

[0098] Read the image sequence from the video and set the first frame image as the current reference image;

[0099] The image shift between the nth frame and the n+1th frame is calculated in sequence, and the accumulated image shift of the nth frame image relative to the reference image is calculated at the same time;

[0100] Determine whether the accumulated image displacement is greater than 1 pixel; if so, start stitching;

[0101] According to the accumulated image shift of the latest frame image relative to the reference image, the integer image shift and the decimal image shift are distinguished. Based on the decimal image shift, an inverse filtering template is constructed to compensate for the sub-pixel image shift in the image filtering;

[0102] Calculate the new panoramic image area based on integer image shift and dynamically expand the panoramic image size;

[0103] In order to solve the obvious splicing traces caused by the difference in inter-frame responses, the splicing area is expanded, the newly added part is quantitatively expanded, and the overlapping part is weightedly fused;

[0104] Set the latest frame as the reference image, read the next frame, and repeat 2-5 until the video ends;

[0105] Save the panoramic image as a local file.

[0106] Sub-pixel image motion compensation can effectively reduce the "ghosting" that may exist at the edges of overlapping areas. By expanding weighted fusion of newly added stitching areas, the problem of excessive discontinuity between images can be improved.

[0107] The ecological monitoring system is used to perform high-resolution optical imaging of the measured area under daytime or nighttime lighting. It also uses a long-focal-length infrared lens in conjunction with a high-sensitivity infrared detector to receive infrared spectral radiation information of the target background, which is then output to a data receiving, processing and display terminal after digital conversion, to achieve real-time detection of key ecological targets.

[0108] The target real-time detection and tracking processing unit is capable of receiving data and analyzing and processing real-time data, fusion display of visible and infrared dual-band images, and completing real-time detection, tracking and positioning of key ecological targets.

[0109] The processing program used by the target real-time detection and tracking processing unit combines the dynamic target autonomous tracking algorithm with the slant range positioning algorithm based on template recognition to achieve rapid identification and positioning of moving targets, hot spots, etc. Infrared data is combined with the dynamic target autonomous tracking algorithm to achieve over-the-horizon moving target capture. Infrared data is combined with the slant range positioning algorithm based on template recognition, and the distribution of control points in the scene is used to achieve sub-pixel-level high-precision geometric positioning, achieving high-precision tracking and positioning of moving targets.

[0110] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

[0111] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. An ultra-long-range visible infrared multi-target ecological monitoring system, characterized by: It includes visible light camera module, medium wave infrared tracking camera module, heavy-load two-dimensional precision turntable, servo control module, and target real-time detection and tracking processing unit, among which: The visible light camera module uses a long focal length zoom fog-penetrating lens to conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during the daytime, and sends visible band image data to the target real-time detection and tracking processing unit; The medium-wave infrared tracking camera module uses a large-aperture, long-focal-length zoom lens to conduct wide-area surveillance and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring day and night, and sends infrared band image data to the target real-time detection and tracking processing unit; The heavy-loaded two-dimensional precision turntable is used to carry the visible light camera module and the medium-wave infrared tracking camera module. According to the turntable control instructions sent by the servo control unit, the visible light camera module and the medium-wave infrared tracking camera module are controlled to perform azimuth rotation control and pitch rotation control according to the preset angles and specified positions respectively; The servo control unit generates a turntable control instruction and sends it to the heavy-loaded two-dimensional precision turntable according to the received positioning information of the dynamic target, the working status of the visible light camera module and the medium-wave infrared tracking camera module; at the same time, it receives new target information in real time and adjusts the turntable control instruction in real time; at the same time, it sends a cooling control instruction to the medium-wave infrared tracking camera module to provide a cooling function for the medium-wave infrared tracking camera module; The target real-time detection and tracking processing unit receives the dual-band image data sent by the visible light camera module and the medium-wave infrared tracking camera module, performs infrared and visible light image registration and image compensation processing at different resolutions, and autonomously tracks and identifies and locates the dynamic targets in the image content according to the image content after registration and image compensation processing, and displays the positioning information of the dynamic targets to the user and sends it to the servo control unit at the same time.

2. The ultra-long-range visible infrared multi-target ecological monitoring system according to claim 1 is characterized by: The visible light camera module includes a visible window, a visible lens, a visible detector, and a visible video circuit unit; The visible lens uses a long focal length zoom fog-penetrating lens, and the visible detector has a sensitivity of 2k×2k pixels, which can conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during daytime; The visible window is used to protect the visible lens from contamination. The visible detector collects data on the visible band image collected by the visible lens and sends it to the target real-time detection and tracking processing unit through the visible video circuit unit.

3. The ultra-long-range visible infrared multi-target ecological monitoring system according to claim 1 is characterized by: The medium-wave infrared tracking camera module includes a medium-wave infrared window, a medium-wave infrared lens, a medium-wave infrared detector, a medium-wave infrared refrigerator, a refrigerator drive unit, and a medium-wave infrared video circuit unit; The medium-wave infrared lens uses a large-aperture, long-focal-length zoom lens, and the medium-wave infrared detector uses a cooled, 640×512-pixel, high-sensitivity medium-wave infrared detector, which can conduct wide-area monitoring and high-definition identification of key ecological targets, human activities, and ecological anomaly monitoring during day and night periods; The medium-wave infrared window is used to protect the medium-wave infrared lens from contamination. The medium-wave infrared detector collects data of the infrared band image collected by the medium-wave infrared lens and sends it to the target real-time detection and tracking processing unit through the medium-wave infrared video circuit unit; The medium-wave infrared detector is cooled by a medium-wave infrared refrigerator and driven and controlled by a refrigerator drive unit; the cooling control command sent by the refrigerator drive unit is issued by a servo control unit.

4. The ultra-long-range visible infrared multi-target ecological monitoring system according to claim 1 is characterized by: The heavy-load two-dimensional precision turntable includes an azimuth rotation mechanism, a pitch rotation mechanism, a turntable base, and a load mounting closed structure; The visible light camera module and the medium-wave infrared tracking camera module are both arranged in a payload-mounted enclosed structure; the azimuth rotation mechanism and the pitch rotation mechanism are arranged on the turntable base, and are used to realize the positioning rotation of the preset angle and the specified position of the circumferential scanning according to the turntable control instructions sent by the servo control unit; the pitch rotation mechanism is used to realize the positioning rotation of the specified pitch angle range and the specified position according to the turntable control instructions sent by the servo control unit.

5. The ultra-long-range visible infrared multi-target ecological monitoring system according to claim 4 is characterized by: The azimuth rotation mechanism and the pitch rotation mechanism form a double-frame structure on the turntable base through the azimuth frame and the pitch frame. The azimuth frame and the pitch frame are both arranged on the turntable base through mounting flanges. The azimuth frame is correspondingly provided with an azimuth axis system, and the pitch frame is correspondingly provided with a pitch axis system. Both are driven by DC torque motors to realize the positioning rotation of the azimuth frame in the circumferential direction and the pitch frame in the pitch direction at preset angles and designated positions.

6. The ultra-long-range visible infrared multi-target ecological monitoring system according to claim 3 is characterized by: The load installation closed structure is a waterproof and dustproof structure, and is internally provided with a temperature control unit and a high-precision measurement unit; the temperature control unit is provided with a heating circuit and a temperature measurement circuit to ensure the temperature environment in the load installation closed structure, and to protect the visible window and the medium-wave infrared window during non-working hours; The high-precision measurement unit is used to measure the turntable attitude and rotation angular velocity of the azimuth rotation mechanism and the pitch rotation mechanism, and the measured data is used to locate the target and determine the load's wind disturbance resistance capability.

7. The ultra-long-range visible infrared multi-target ecological monitoring system according to claim 6 is characterized by: Dust covers and electric wipers are provided outside the visible window and the medium-wave infrared window. The dust covers have a switch function and are closed during non-working hours to protect the optical windows. The electric wipers are used to clean the visible window and the medium-wave infrared window in rainy and foggy weather to ensure imaging quality.

8. The ultra-long-distance visible infrared multi-target ecological monitoring system according to claim 6 is characterized by: The target real-time detection and tracking processing unit runs an image acquisition processing program, and performs registration processing and target detection processing on the latest two frames of images in the dual-band data respectively; the ecological monitoring system also includes a main control machine data display and storage unit. After the target real-time detection and tracking processing unit receives the original image or the image of the marked detection target, it displays it in the main window and thumbnail window on the main control machine data storage unit. The main window displays a large field of view infrared image by default, and the thumbnail window displays a visible image.

9. The ultra-long-range visible infrared multi-target ecological monitoring system according to claim 8, characterized in that: The data display storage unit of the host control machine switches the display window through user control, and realizes the storage and playback processing of local image data according to user needs; if image data playback is required, the data cache is read from the local image data at fixed intervals.

10. The ultra-long-distance visible infrared multi-target ecological monitoring system according to claim 8, characterized in that: The target real-time detection and tracking processing unit runs an image acquisition processing program and performs image registration on the dual-band image data, and performs image motion estimation processing and image motion compensation processing on the dual-band image data, specifically: The image motion parameters of the dual-band image data are calculated using the block grayscale matching method, and a reference image is selected based on the obtained image motion information. The remaining dual-band image data to be registered are compared with the reference image. The dual-band image data to be registered are transformed according to the comparison results to obtain the image content after registration and image compensation.

11. The ultra-long-distance visible infrared multi-target ecological monitoring system according to claim 10, characterized in that: The image content after the registration and image compensation processing is processed by frame difference detection to obtain differential image data, and the differential image data is segmented by grayscale threshold to select suspected targets, and the suspected targets are subjected to morphological processing and Blob analysis filtering processing, and the target shape size, speed size and speed change are used as constraints to calculate the association cost matrix, and the trajectory association is performed through Kalman filtering, and the trajectory point position, speed, and observation number information are updated, and the trajectory is confirmed according to the target related information to eliminate the target trajectory that is incorrectly associated or lost for a long time, and determine the stable tracking target.

Images