Intelligent ground reconnaissance method for people over annulus

Abstract

The invention discloses an over-the-ground intelligent reconnaissance method for people in a ring, aiming at the phenomena of misrecognition, disordered recognized target management, unstable recognition and tracking and the like in the reconnaissance process, a process control and task management strategy is designed, and a novel intelligent reconnaissance method is formed by combining a target detection and tracking algorithm and key operation of people in the ring. According to the process control and task management strategy, the intelligent and efficient advantages of a deep learning target detection algorithm can be exerted, people can intervene in key decisions, and misrecognition caused by uninterpretability of deep learning and target management chaos and unstable recognition tracking caused by insufficient algorithm migration capability are avoided. The air-to-ground intelligent reconnaissance method is provided on the basis of achieving operation safety, and air-to-ground reconnaissance tasks can be continuously and stably executed in various complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent reconnaissance decision-making, and relates to an air-to-ground intelligent reconnaissance target information method based on computer vision for target detection and tracking, in-the-loop process control, and task management cooperation to achieve operational safety. More specifically, it relates to an air-to-ground intelligent reconnaissance method with humans in the loop. Background Art

[0002] Unmanned aerial vehicles (UAVs) have become an important force for land reconnaissance, surveillance, and intelligence collection due to their low cost, convenient operation, and large air-to-ground field of view. They are equipped with modules such as sensors, data links, and control systems. Existing fully autonomous reconnaissance UAVs carry different types of sensors, such as radars, short-wave collectors, optoelectronics, etc. Different sensors have different application scenarios and their respective advantages. With the development of artificial intelligence and the popularization of high-performance deep learning, the data collected by different sensors is combined with target detection and tracking algorithms based on deep learning models to achieve intelligent air-to-ground reconnaissance, reducing the burden on UAV pilots to control UAVs. However, deep learning algorithms have the problems of being unexplainable and unstable, and there are inherent uncontrollable problems such as misidentification, chaotic target management, and unstable identification and tracking during the reconnaissance process. In addition, during air-to-ground reconnaissance, the ground background is much more complex than the sky and ocean backgrounds, which will exacerbate the problems of misidentification and missed identification. These problems can lead to the uncontrollability of fully autonomous reconnaissance UAVs and cause operational safety issues. Summary of the Invention

[0003] To solve the above problems, the present invention realizes functional tasks through target detection and tracking algorithms, and a small number of key decisions are issued by humans through a ground station. At the same time, process control and task management strategies are designed to balance the intelligent execution of reconnaissance tasks and controllable and safe target detection and management.

[0004] An air-to-ground intelligent reconnaissance method with humans in the loop includes the following steps:

[0005] The first step is to prepare a task module, a reconnaissance module, and a ground station, and configure the corresponding software, hardware, and communication systems;

[0006] The second step is to enter the mission area. The on-board task module issues a scanning instruction to the reconnaissance module. While the reconnaissance module controls the optoelectronic seeker to perform intelligent search, it simultaneously transmits the compressed video in real time. When a target is detected during the search, the reconnaissance module performs multi-target management and selects the target with the highest threat level, adjusts the seeker servo to ensure stable tracking of the target, and reports it to the ground station through the data link;

[0007] In the third step, the remote operator makes key decisions such as target ignoring, switching, and non-operation (waiting for the algorithm to track stably) at the ground station based on the video and target information transmitted in real time by the reconnaissance system. The reconnaissance module adopts different strategies according to whether the seeker is in the tracking state, whether there is a target to be tracked in the field of view, and whether there are other targets in the field of view, combined with the decision of the operator, and then adopts different intelligent execution methods to complete the tracking task.

[0008] In the fourth step, after the reconnaissance module determines that the tracking is stable according to the stable tracking strategy, the ground station operator confirms the target. The reconnaissance module performs laser ranging and target position calculation according to the positioning strategy and reports relevant information. If the target is lost during the tracking process, the reconnaissance module controls the seeker to first turn off the laser ranging and then re-enter the scanning mode to search for new targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of an intelligent air-to-ground reconnaissance method with human-in-the-loop of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be further described in detail below in conjunction with the drawings and preferred embodiments.

[0011] As Figure 1 shown, an intelligent air-to-ground reconnaissance method with human-in-the-loop includes the following steps:

[0012] In the first step, a hardware device carrying the task module and the reconnaissance module software is installed behind the optoelectronic seeker of the unmanned aerial vehicle (UAV) and connected through a communication cable. The UAV is equipped with an airborne data link for communication with the ground station, and software for issuing key decision instructions is deployed on the ground station.

[0013] In the second step, after the UAV enters the mission area, the airborne task module issues a scanning instruction to the reconnaissance module. The reconnaissance module controls the optoelectronic seeker servo to perform scanning in the left-right and pitch directions through a communication cable (network cable or serial cable). The seeker transmits the original video to the reconnaissance module through a video transmission cable, and the reconnaissance module compresses it and pushes it to the ground station in UDP format for real-time monitoring by the operator.

[0014] If a target appears in the field of view during the scanning process, the following multi-target management strategy is adopted: The target detection and tracking algorithm built into the reconnaissance module will judge whether there are multiple targets in the field of view by identifying the number of targets. The target detection and tracking algorithm selects YOLOv8. If it is a single target, the reconnaissance module inputs the coordinate frame output by the target detection and tracking algorithm into the seeker to correct the servo to ensure stable tracking. If there are multiple targets, the coordinate frame of the target with the highest threat level is selected and input into the seeker to correct the servo. The function of correcting the servo is to keep the target at the center of the field of view. If the detected target coordinate frame has a width of w and a height of h, then the target with the largest w*h is selected as the target with the highest threat level.

[0015] The reconnaissance module superimposes the target coordinate box and the target number on the video transmitted to the ground station, and reports the detection and tracking situation.

[0016] In the third step, if the real-time video transmitted back shows that the seeker is not in the tracking state, the ground station operator does not perform any operation and waits for the mission and the reconnaissance module to search, identify and track the target by themselves.

[0017] If the seeker is already in the tracking state and the target being tracked is the target that the operator wants to track, the ground station operator does not perform any operation, and the reconnaissance module continues to track until it is stable.

[0018] If the seeker is already in the tracking state and the target being tracked is not the target that the operator wants to track (the algorithm will select the target with the highest threat level for tracking, but if the operator wants to track other targets, the operator's priority should be the highest), the following two strategies are adopted: (1) If there is no target that the operator wants to track in the field of view, send a target ignore instruction with the target number to the reconnaissance module through the ground station. The reconnaissance module turns off the seeker laser and adds the target selected by the operator to the blacklist. After that, even if the algorithm identifies the target, it will not perform tracking and target reporting. At this time, if there are no other targets in the field of view, the reconnaissance module enters the scanning state to search for targets again. If there are other targets in the field of view, select the target with the highest threat level except the ignored target for tracking. (2) If there is a target that the operator wants to track in the field of view, send a target switching instruction with the target number to the reconnaissance module through the ground station. When the reconnaissance module executes this task, it judges whether the current tracked target number is the same as the target number to be switched. If they are the same, directly track the target to be switched. If they are different, first turn off the laser and then perform the target switching.

[0019] In the fourth step, after the reconnaissance module determines that it has stably tracked the target, it notifies the ground station. The following strategy is used to determine whether it is stable tracking: Based on the field of view image resolution of 1920*1080, if the center position of the target coordinate box in 20 consecutive frames differs from the previous frame by no more than 70 pixels, it is considered stable tracking.

[0020] Due to the misidentification phenomenon of existing target detection algorithms, for safety considerations, after the ground station receives the stable tracking information from the reconnaissance module, the following positioning strategy is adopted: The operator can judge whether to turn on the laser positioning. If it is confirmed to be turned on, the operator issues a target confirmation instruction. The reconnaissance module controls the seeker to turn on the laser to complete ranging. After the seeker completes the target positioning position calculation, it reports to the reconnaissance module. The reconnaissance module superimposes the positioning information on the video and streams it to the ground station, and at the same time sends the content of the positioning information to the ground station.

[0021] If the target is lost during the tracking process, the reconnaissance module controls the seeker to first turn off the laser rangefinder and then re-enter the scanning mode to search for a new target.

Claims

1. An intelligent air-to-ground reconnaissance method for a person in the loop, characterized in that, It includes the following steps: In the first step, a hardware device carrying the software of the mission module and the reconnaissance module is installed behind the optoelectronic seeker of the UAV and connected through a communication cable. The UAV is equipped with an airborne data link for communication with the ground station, and software for issuing key decision-making instructions is deployed on the ground station; In the second step, after the UAV enters the mission area, the airborne mission module issues a scanning instruction to the reconnaissance module. The reconnaissance module controls the optoelectronic seeker servo to scan in the left-right and pitch directions through a communication cable, i.e., a network cable or a serial cable. The seeker transmits the original video to the reconnaissance module through a video transmission cable. After compression, the reconnaissance module pushes the stream to the ground station via the data link for real-time monitoring by personnel; If a target appears in the field of view during the scanning process, the target detection and tracking algorithm built into the reconnaissance module determines whether there are multiple targets in the field of view by identifying the number of targets. The target detection and tracking algorithm selects YOLOv8; if it is a single target, the reconnaissance module inputs the coordinate frame output by the target detection and tracking algorithm into the seeker to correct the servo to ensure stable tracking. If there are multiple targets, the coordinate frame of the target with the highest threat level is selected and input into the seeker to correct the servo. The function of correcting the servo is to keep the target at the center of the field of view; if the width of the detected target coordinate frame is w and the height is h, then the target with the largest w*h is selected as the target with the highest threat level; The reconnaissance module superimposes the target coordinate frame and the target number on the video transmitted to the ground station and reports the detection and tracking situation; In the third step, if the real-time video transmitted back shows that the seeker is not in the tracking state, the ground station operator does not perform any operation and waits for the mission and reconnaissance modules to search, identify, and track the target by themselves; If the seeker is already in the tracking state and the target being tracked is the target that the operator wants to track, the ground station operator does not perform any operation, and the reconnaissance module continues to track until it is stable; If the seeker is already in the tracking state and the target being tracked is not the target that the operator wants to track, the algorithm will select the target with the highest threat level for tracking. However, if the operator wants to track other targets, the operator's priority should be the highest, and the following two strategies are adopted: First, if the target that the operator wants to track does not exist in the field of view, a target ignore instruction with the target number is sent to the reconnaissance module through the ground station. The reconnaissance module turns off the seeker laser and adds the target selected by the operator to the blacklist. After that, even if the algorithm identifies the target, it will not perform tracking and target reporting; at this time, if there are no other targets in the field of view, the reconnaissance module enters the scanning state to search for targets again. If there are other targets in the field of view, the target with the highest threat level except the ignored target is selected for tracking; Second, if the target that the operator wants to track exists in the field of view, a target switching instruction with the target number is sent to the reconnaissance module through the ground station. When the reconnaissance module executes this task, it judges whether the current tracking target number is the same as the target number to be switched. If they are the same, it directly tracks the target to be switched. If they are different, it first turns off the laser and then performs the target switching; In the fourth step, after the reconnaissance module determines that it has stably tracked the target, it notifies the ground station; Whether stable tracking is determined by the following strategy: Based on the field of view image resolution of 1920*1080, if the difference in the center position of the target coordinate box between 20 consecutive frames and the previous frame does not exceed 70 pixels, it is considered stable tracking; Due to the misidentification phenomenon existing in the existing target detection algorithms, for safety considerations, after the ground station receives the stable tracking information from the reconnaissance module, the operator can judge whether to activate laser positioning. If it is confirmed to be activated, the operator issues a target confirmation command. The reconnaissance module controls the seeker to turn on the laser to complete ranging. After the seeker completes the calculation of the target positioning position, it reports to the reconnaissance module. The reconnaissance module superimposes the positioning information on the video and streams it to the ground station, and at the same time sends the content of the positioning information to the ground station; If the target is lost during the tracking process, the reconnaissance module controls the seeker to first turn off the laser ranging and then re-enter the scanning mode to search for new targets.

Images