An unmanned aerial vehicle automatic searching and tracking system based on AI image recognition technology

Abstract

The application relates to the technical field of image recognition, in particular to an unmanned aerial vehicle automatic search and tracking system based on AI image recognition technology, which comprises a collection module, an identification module, a residual error module, a control module and a recapture module. The collection module generates flight observation records, the identification module determines target frame positions, area ratios and identification confidence, the residual error module generates a visual axis retention residual error chain, the control module determines an expected reappearance window and generates a gimbal swing instruction and a flight control adjustment instruction, and the recapture module determines a recapture target frame and outputs a tracking state. In the application, the target frame result is associated with the gimbal attitude, the body attitude and the flight state according to the image collection time, the visual axis constraint before target disappearance is formed, the comparison area is limited by the expected reappearance window when the confidence decreases, the window position is converted into the gimbal and flight control instructions, the full-screen search after the shielding is reduced, and the continuous transmission between the image recognition result and the tracking control is maintained.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an automatic search and tracking system for drones based on AI image recognition technology. Background Technology

[0002] Image recognition technology involves computer vision processing of images or video frames acquired by camera devices, including target detection, feature extraction, image classification, and image matching. Traditional AI-based image recognition-based drone automatic search and tracking systems involve equipping the drone with a visible light camera, image acquisition interface, onboard processor, and memory. The camera continuously acquires aerial image frames, and the processor performs preprocessing, candidate region extraction, target category recognition, target bounding box coordinate output, and inter-frame target matching on the image frames. The recognition results are then transmitted to the flight control unit or ground station display system.

[0003] Existing UAV search and tracking systems typically rely on cameras to continuously acquire aerial images and onboard processors to perform target bounding box recognition and frame-to-frame matching. When a target briefly enters a vegetation or building obstruction area, the bounding box disappears from the image. The flight control unit can only maintain or stop following based on the recognition result of the current frame. There is a lack of continuous constraints on the gimbal attitude and the change in the line of sight before the target disappears. When the target turns or the UAV's attitude is disturbed, the single-frame matching result is difficult to support the linkage control of heading speed and gimbal angle. The reacquisition retrieval range expands, irrelevant image areas are involved in the comparison, and the tracking status is prone to interruption. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides an automatic search and tracking system for drones based on AI image recognition technology.

[0005] On the one hand, an automatic search and tracking system for drones based on AI image recognition technology is provided, the system including: The acquisition module is used to acquire aerial image frames, image acquisition time, gimbal azimuth angle, gimbal pitch angle, UAV altitude, UAV heading angle, aircraft roll angle and aircraft pitch angle, and generate flight observation records according to the image acquisition time. The recognition module, connected to the acquisition module, is used to divide the aerial image frame into image regions, extract the convolutional feature vectors of each image region, compare the convolutional feature vectors with the target image feature vectors, and determine the center position of the target box, the area ratio of the target box, and the target recognition confidence. The residual module, connected to the recognition module, is used to calculate the lateral offset change, longitudinal offset change, scale change, gimbal azimuth change, and gimbal pitch change based on the continuous flight observation records, the center position of the target frame, and the area ratio of the target frame, and to generate a line-of-sight-maintaining residual chain. The control module, connected to the residual module, is used to determine the expected reproduction window based on the line-of-sight residual chain when the target recognition confidence is lower than the recapture confidence threshold determined by the average confidence of consecutive valid frames, and to generate gimbal sway commands and flight control adjustment commands according to the expected reproduction window. The recapture module, connected to the control module, is used to receive the expected reproduction window, extract the window feature vector from the image region covered by the expected reproduction window, compare the window feature vector with the target image feature vector, determine the recapture target box based on the recapture confidence threshold, and output the tracking status.

[0006] As a further embodiment of the present invention, the acquisition module includes an image acquisition submodule, an attitude acquisition submodule, and a record generation submodule; The image acquisition submodule obtains aerial image frames and image acquisition time; The attitude acquisition submodule acquires the gimbal azimuth angle, gimbal pitch angle, UAV altitude, UAV heading angle, aircraft roll angle, and aircraft pitch angle within the acquisition period corresponding to the image acquisition time. The record generation submodule establishes a mapping relationship between aerial image frames and attitude and altitude data within the corresponding acquisition period to generate flight observation records.

[0007] As a further aspect of the present invention, the identification module includes a region segmentation submodule, a feature extraction submodule, and a target comparison submodule; The region division submodule divides the aerial image frame into multiple non-overlapping image regions according to the pixel row and column positions of the aerial image frame; The feature extraction submodule obtains the edge direction distribution, color channel mean, and texture change intensity of each image region and generates a convolutional feature vector; The target comparison submodule compares the convolutional feature vector with the target image feature vector to determine the center position of the target box, the area ratio of the target box, and the target recognition confidence.

[0008] As a further embodiment of the present invention, the residual module includes a record filtering submodule, an offset calculation submodule, and a residual generation submodule; The record filtering submodule filters valid observation records from the continuous flight observation records, including the center position of the target frame and the area ratio of the target frame. The offset calculation submodule calculates the lateral offset change, longitudinal offset change, scale change, gimbal azimuth change, and gimbal pitch change between adjacent valid observation records according to the image acquisition time sequence. The residual generation submodule arranges the changes in sequence according to the image acquisition time to generate a view axis-preserving residual chain.

[0009] As a further aspect of the present invention, the residual module is also used for: The center position of the target frame in two adjacent flight observation records is compared horizontally and vertically with the center position of the aerial image frame, respectively, to obtain the horizontal and vertical line-of-sight offsets corresponding to the two flight observation records. The difference between adjacent lateral and longitudinal view axis offsets is calculated to generate lateral offset change and longitudinal offset change.

[0010] As a further aspect of the present invention, the residual module is also used for: The gimbal azimuth change is compared with the UAV heading angle change, and the gimbal pitch change is compared with the airframe pitch angle change to obtain the gimbal-airframe compensation deviation. When the gimbal body compensation deviation exceeds the attitude compensation threshold determined by the average of continuous effective compensation deviations, the lateral offset change and longitudinal offset change of the corresponding acquisition period are marked as attitude disturbance residuals.

[0011] As a further aspect of the present invention, the control module includes a threshold determination submodule, a window determination submodule, and an instruction generation submodule; The threshold determination submodule determines the recapture confidence threshold based on the average confidence score of consecutive valid frames; When the target recognition confidence level is lower than the recapture confidence threshold, the window determination submodule determines the expected reproduction window based on the line-of-sight preservation residual chain. The instruction generation submodule generates gimbal sway instructions and flight control adjustment instructions based on the positional relationship between the expected reproduction window and the center of the aerial image frame.

[0012] As a further aspect of the present invention, the control module is also used for: Read the lateral and longitudinal offset changes in the view-axis holding residual chain, and determine the directional consistency of each lateral offset change and each longitudinal offset change according to the image acquisition time sequence. When both the horizontal and vertical directions meet the continuity and consistency condition, the center position of the expected reproduction window is determined based on the center position of the nearest target box and the direction of offset change, and the expected reproduction window is generated.

[0013] As a further embodiment of the present invention, the recapture module includes a window receiving submodule, a window feature submodule, a recapture comparison submodule, and a status output submodule; The window receiving submodule receives the expected reproduction window and determines the image area covered by the expected reproduction window; The window feature submodule extracts the window feature vector of the image region; The recapture comparison submodule compares the window feature vector with the target image feature vector and generates a window matching confidence score. The status output submodule determines the recapture target box and outputs the tracking status based on the window matching confidence and recapture confidence threshold.

[0014] As a further aspect of the present invention, the process of generating the window feature vector specifically includes: Obtain the brightness value, color channel value, and edge direction value of each pixel within the image area covered by the expected reproduction window; The brightness value of each pixel is compared with the brightness value of its neighboring pixels to determine the edge change location, and the distribution number of the edge change locations in the horizontal and vertical directions is counted. The distribution of brightness variation, color mean, and edge direction are statistically analyzed according to the pixel row and column order. The distribution quantity, the brightness variation distribution, the color mean distribution, and the edge direction distribution are written into the window feature sequence according to a fixed feature order to generate the window feature vector.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: By establishing a correspondence between the target bounding box center position, target bounding box area ratio, and target recognition confidence level, and the gimbal azimuth angle, gimbal pitch angle, UAV altitude, UAV heading angle, and aircraft attitude at the same image acquisition time, the horizontal offset change, vertical offset change, scale change, and gimbal attitude change are continuously calculated to form a line-of-sight maintenance residual chain. When the target recognition confidence level is lower than the recapture confidence threshold, the expected reproduction window is limited according to the residual chain, so that the recapture comparison is concentrated on the image area where the target may reproduce. Simultaneously, gimbal sway commands and flight control adjustment commands are generated, forming a continuous control link from image recognition results to flight execution, reducing invalid comparisons after occlusion, and maintaining the target tracking state. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a diagram illustrating the overall architecture of the UAV automatic search and tracking system of the present invention. Figure 2 This is a flowchart of the aerial image and attitude acquisition and observation record generation process of the present invention; Figure 3This is a flowchart of the image region feature extraction and target recognition process of the present invention; Figure 4 This is a flowchart of the line-of-sight retention residual chain generation and attitude perturbation marking process of the present invention; Figure 5 This is a flowchart of the recapture threshold determination and expected reproduction window control of the present invention; Figure 6 This is a flowchart of the window feature extraction and target recapture process of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0019] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0020] This embodiment provides an automated search and tracking system for unmanned aerial vehicles (UAVs) based on AI image recognition technology. This system is suitable for autonomous searching and continuous tracking of targets without external beacons in scenarios such as field search and rescue, emergency patrols, public security, and target inspection. During actual operation, the UAV continuously acquires aerial images via its onboard camera, adjusts the camera's shooting direction via a gimbal mechanism, maintains the UAV's attitude and flight status via flight control components, and completes target recognition, line-of-sight change recording, recapture window determination, and tracking status output via onboard computing components.

[0021] In complex environments, the tracked target may briefly enter areas obscured by trees, buildings, smoke, or terrain, causing the target bounding box in the aerial image to temporarily disappear. If the decision to continue tracking is based solely on whether the target is detected in the current frame, it can easily lead to a lack of continuous data for gimbal control and flight control. This embodiment correlates changes in image position, target bounding box size, gimbal attitude, and aircraft attitude before the target disappears. This allows for the determination of the expected recapture window based on the trend of line-of-sight changes even when the target is temporarily invisible, and target recapture can be performed within that window.

[0022] Please see Figure 1 The overall architecture of the UAV automatic search and tracking system of this invention includes a data acquisition module, an identification module, a residual module, a control module, and a reacquisition module. The data acquisition module is connected to the identification module, the identification module is connected to the residual module, the residual module is connected to the control module, and the control module is connected to the reacquisition module. The data acquisition module is used to generate flight observation records, the identification module is used to generate the target frame center position, the target frame area ratio, and the target recognition confidence level, the residual module is used to generate the line-of-sight persistence residual chain, the control module is used to generate the expected reproduction window, gimbal sway commands, and flight control adjustment commands, and the reacquisition module is used to generate the reacquisition target frame and tracking status.

[0023] The flight observation record is a data set indexed by image acquisition time. Its content includes aerial image frames, gimbal azimuth, gimbal pitch, UAV altitude, UAV heading, roll, and pitch. An aerial image frame represents a single frame acquired by the UAV camera within the current acquisition cycle. The image acquisition time indicates the time stamp when the frame was acquired or stably received. The gimbal azimuth represents the camera's rotation relative to the UAV in the lateral direction. The gimbal pitch represents the camera's rotation relative to the UAV in the vertical direction. The UAV altitude represents the UAV's altitude relative to the operational reference plane. The UAV heading represents the UAV's orientation relative to the operational reference direction in the forward direction. The roll and pitch angles represent the UAV's tilt in the lateral and longitudinal directions, respectively.

[0024] Please see Figure 2 The acquisition module includes an image acquisition submodule, an attitude acquisition submodule, and a record generation submodule. The image acquisition submodule connects to the UAV camera device to receive aerial image frames and writes the image acquisition time when the aerial image frame is acquired or stabilized. The attitude acquisition submodule connects to the gimbal mechanism, flight control components, attitude detection components, and altitude detection components to acquire the gimbal azimuth angle, gimbal pitch angle, UAV altitude, UAV heading angle, roll angle, and pitch angle within the acquisition period corresponding to the image acquisition time. The record generation submodule receives the aerial image frames and image acquisition time output from the image acquisition submodule, as well as the attitude and altitude data output from the attitude acquisition submodule, and establishes a mapping relationship between the two according to the same acquisition period to generate a flight observation record.

[0025] During the specific operation of the acquisition module, the image acquisition submodule adds an image acquisition time to each aerial image frame received. The attitude acquisition submodule uses this image acquisition time as a reference to read the gimbal azimuth, gimbal pitch, UAV altitude, UAV heading, roll, and pitch angles within the same acquisition cycle. If multiple sets of attitude and altitude data exist within the same acquisition cycle, the attitude acquisition submodule selects the data closest to the image acquisition time as the attitude and altitude data corresponding to that aerial image frame. If an aerial image frame is not received completely, the record generation submodule marks that acquisition cycle as an image missing state. If the attitude and altitude data within an acquisition cycle cannot establish a correspondence with an aerial image frame, the record generation submodule marks that acquisition cycle as an observation incomplete state. Flight observation records marked as image missing or observation incomplete states are not considered valid observation records for subsequent residual calculations.

[0026] Through the aforementioned acquisition structure, aerial image frames can be correlated with gimbal attitude, aircraft attitude, and UAV altitude according to the same time reference, avoiding time mismatches between subsequent changes in target bounding box position and UAV attitude. This flight observation record provides image input for the recognition module and also provides the attitude basis required for target line-of-sight changes for the residual module.

[0027] Please see Figure 3 The recognition module includes a region segmentation submodule, a feature extraction submodule, and a target comparison submodule. The region segmentation submodule receives the aerial image frame output by the acquisition module and divides it into multiple non-overlapping image regions according to the pixel row and column positions. Each image region is a local area within the aerial image frame, and its row and column positions within the entire frame are preserved. The feature extraction submodule, connected to the region segmentation submodule, obtains the edge direction distribution, color channel mean, and texture variation intensity of each image region, and generates convolutional feature vectors accordingly. The target comparison submodule, also connected to the feature extraction submodule, compares the similarity between the convolutional feature vectors and the target image feature vectors to determine the target bounding box center position, target bounding box area ratio, and target recognition confidence level.

[0028] The convolutional feature vector is a data sequence formed by writing the edge direction distribution, color channel mean, and texture change intensity within the image region in a fixed feature order. The edge direction distribution comes from the statistical results of the brightness change direction of adjacent pixels in the image region, the color channel mean comes from the overall color state of different color channels in the image region, and the texture change intensity comes from the local brightness fluctuations and edge density in the image region. The target image feature vector comes from the target image of the target to be tracked or the confirmed target box. If the target image is uploaded by the ground station, the recognition module performs feature extraction on the target image with the same aperture as the aerial image frame to form the target image feature vector. If the target image feature vector is generated from the confirmed target box, the recognition module extracts features from the target box region in a stable tracking state and updates the extraction result to the target image feature vector corresponding to the current target.

[0029] In the specific operation of the recognition module, the region segmentation submodule first divides the aerial image frame into several local image regions, enabling subsequent target comparisons to be performed around these local areas, rather than directly comparing the entire aerial image frame indiscriminately. The feature extraction submodule reads the brightness state, color channel state, and brightness variations between adjacent pixels for each image region. For the same image region, the feature extraction submodule first organizes the edge direction distribution based on the brightness variation direction, then organizes the color channel mean based on the overall state of each color channel, and finally organizes the texture variation intensity based on local brightness fluctuations and edge density. After the above data is written in a fixed feature order, a convolutional feature vector corresponding to that image region is formed.

[0030] The target comparison submodule reads the convolutional feature vectors of each image region and the feature vector of the target image. During similarity comparison, the submodule compares the closeness between the convolutional feature vectors and the target image feature vectors in the same feature order, and converts the comparison results into target recognition confidence scores. Target recognition confidence scores represent the degree of reliability in matching the features of the current image region with those of the target image. If the target recognition confidence score of an image region meets the target locking condition, the target comparison submodule identifies that image region as a candidate target bounding box region. If adjacent candidate target bounding box regions belong to the same target, the submodule merges them according to the positional relationship between the candidate regions to obtain the target bounding box boundary, and then determines the center position and area ratio of the target bounding box based on the boundary.

[0031] The target locking condition is not an arbitrarily set single-frame judgment condition, but is jointly determined by the current target image feature vector and the target recognition confidence in consecutive valid aerial image frames. For the same image region, the target comparison submodule arranges the target recognition confidence within the continuous acquisition period according to the image acquisition time. Only when the image region meets the target locking requirements in consecutive valid aerial image frames will it output the target box center position, target box area ratio, and target recognition confidence. If the target image feature vector has not yet been established, the target comparison submodule outputs the target feature missing status and does not output the target box result. If the aerial image frame has overexposure, severe blur, or image missing status, the target comparison submodule will not consider the recognition result of that frame as a valid target box result.

[0032] Through the aforementioned recognition structure, the target recognition result not only provides information on the existence of the target, but also its position in the image, the scale it occupies, and the confidence level of the recognition. The center position of the target bounding box is used by the residual module to calculate the target's offset relative to the image center; the area ratio of the target bounding box is used by the residual module to determine the target's distance; and the target recognition confidence level is used by the control module to determine whether to enter the recapture control state.

[0033] Please see Figure 4 The residual module includes a record filtering submodule, an offset calculation submodule, and a residual generation submodule. The record filtering submodule receives continuous flight observation records and filters out valid observation records that have the target box center position and target box area ratio. The offset calculation submodule, connected to the record filtering submodule, is used to calculate the lateral offset change, longitudinal offset change, scale change, gimbal azimuth change, and gimbal pitch change between adjacent valid observation records according to the image acquisition time sequence. The residual generation submodule, also connected to the offset calculation submodule, is used to arrange the changes according to the image acquisition time sequence to generate a line-of-sight-preserving residual chain.

[0034] A valid observation record refers to a flight observation record that simultaneously contains aerial image frames, image acquisition time, attitude and altitude data, target bounding box center position, and target bounding box area ratio. If a flight observation record lacks the target bounding box center position, it indicates that no target bounding box suitable for tracking was formed during that acquisition period, and this flight observation record will not be included in residual calculation. If a flight observation record lacks attitude and altitude data, it indicates that the target bounding box result does not correspond to the shooting attitude at the time, and this flight observation record will also not be included in residual calculation.

[0035] The offset calculation submodule reads the target frame center position from adjacent valid observation records and compares it laterally and vertically with the center position of the aerial image frame, respectively, to obtain the lateral and vertical line-of-sight offsets corresponding to the two valid observation records. The lateral line-of-sight offset represents the deviation of the target frame center from the image center in the left-right direction, and the vertical line-of-sight offset represents the deviation of the target frame center from the image center in the up-down direction. The offset calculation submodule then calculates the difference between adjacent lateral line-of-sight offsets to generate the lateral offset change, and calculates the difference between adjacent vertical line-of-sight offsets to generate the vertical offset change.

[0036] Simultaneously, the offset calculation submodule reads the target frame area ratio, gimbal azimuth, and gimbal pitch angle from adjacent valid observation records. Changes in the area ratio of adjacent target frames form a scale change, representing the target's distance trend in the image. The offset calculation submodule also reads the UAV altitude from adjacent valid observation records, comparing the scale change with the UAV altitude change. When the scale change indicates the target is moving away while the UAV altitude remains relatively constant, the scale change is retained to distinguish between the target's own distance changes and scale changes caused by the UAV's ascent and descent. Changes in adjacent gimbal azimuth angles form a gimbal azimuth change, and changes in adjacent gimbal pitch angles form a gimbal pitch change. The residual generation submodule associates the lateral offset change, longitudinal offset change, scale change, gimbal azimuth change, and gimbal pitch change as residual nodes based on the same image acquisition time relationship. Multiple residual nodes are then connected in the order of image acquisition time to form a line-of-sight-preserving residual chain.

[0037] The line-of-sight residual chain records the continuous trend of target changes relative to the camera's line of sight before the target disappears. This chain does not directly represent the target's geographic coordinates, but rather the target's changes relative to the center of the aerial view, changes in the target frame size, and changes in gimbal attitude. Since the UAV itself experiences heading and pitch changes during flight, the residual module also compensates for the relationship between gimbal attitude changes and overall aircraft attitude changes. Specifically, the residual module compares the difference between the gimbal azimuth change and the UAV heading angle change, and compares the difference between the gimbal pitch change and the overall aircraft pitch angle change, to obtain the gimbal-aircraft compensation deviation.

[0038] The attitude compensation threshold is determined by the average of continuous effective compensation deviations. The average of continuous effective compensation deviations is derived from recent valid observation records not marked as attitude anomalies, representing the reference compensation relationship between gimbal attitude changes and aircraft attitude changes under stable flight conditions. When the gimbal / aircraft compensation deviation exceeds the attitude compensation threshold, the residual module marks the lateral and longitudinal offset changes corresponding to the acquisition period as attitude disturbance residuals. These attitude disturbance residuals are not removed from the line-of-sight preservation residual chain but are treated as low-confidence residual nodes when subsequently determining the expected recurrence window, to reduce the impact of aircraft disturbances on the determination of the target disappearance direction.

[0039] When the UAV first enters the operational area or the target is first locked, continuous valid observation records may not yet be sufficient to form a stable line-of-sight (LOS)-maintaining residual chain. At this time, the residual module retains the acquired valid observation records and outputs a status indicating that the residual chain is not yet established. Once continuous valid observation records can form adjacent difference relationships, the residual module begins generating the LOS-maintaining residual chain. If aerial image frames are missing or attitude data is abnormal, the residual module skips flight observation records that cannot form valid difference relationships and continues to use valid observation records that can establish a temporal sequence relationship to generate residual nodes.

[0040] Through the aforementioned residual structure, the changes in image offset, target bounding box scale, and gimbal attitude before the target disappears are organized into a line-of-sight-maintaining residual chain that can be invoked by the control module. This chain provides continuous basis for the reacquisition direction after a short-term target occlusion, preventing the control module from deciding whether to continue tracking solely based on whether the target is detected in the current aerial image frame.

[0041] Please see Figure 5 The control module includes a threshold determination submodule, a window determination submodule, and a command generation submodule. The threshold determination submodule determines the recapture confidence threshold based on the average confidence level of consecutive valid frames. The window determination submodule, connected to the threshold determination submodule and the residual module, determines the expected reproduction window based on the line-of-sight preservation residual chain when the target recognition confidence level is lower than the recapture confidence threshold. The command generation submodule, connected to the window determination submodule, generates gimbal sway commands and flight control adjustment commands based on the positional relationship of the expected reproduction window relative to the center of the aerial image frame.

[0042] The average confidence score of consecutive valid frames is derived from the target recognition confidence scores in recent valid observation records. The threshold determination submodule reads the target recognition confidence scores from the consecutive valid observation records that are not affected by attitude perturbation residuals, and forms a confidence reference state for the current target under the current shooting conditions based on these target recognition confidence scores. The recapture confidence threshold is determined by this confidence reference state. If the number of consecutive valid frames is insufficient, the threshold determination submodule uses the confidence reference state saved during the target locking phase as a temporary threshold source; once the number of consecutive valid frames meets the stable operating conditions, it switches to the recapture confidence threshold determined by the average confidence score of consecutive valid frames. Thus, usable recapture decision boundaries can be obtained both when the target is first locked and during stable tracking.

[0043] When the target recognition confidence level is lower than the recapture confidence threshold, the window determination submodule enters the recapture window determination state. The window determination submodule reads the lateral and longitudinal offset changes in the view-axis preservation residual chain and determines the directional consistency of each lateral and longitudinal offset change according to the image acquisition time sequence. Directional consistency means that the target's offset direction in the nearest valid residual node has not reversed, and the offset directions can form a continuous extension relationship. When both the lateral and longitudinal directions meet the continuous consistency condition, the window determination submodule determines the expected center position of the recapture window based on the center position of the nearest target bounding box and the offset change direction.

[0044] The expected reappearance window is a local area in the aerial image frame that the target may reappear after a short period of occlusion. When determining this window, the window determination submodule considers the nearest target bounding box center position, lateral offset changes, vertical offset changes, target bounding box area ratio, and attitude perturbation residual markers. If the target bounding box area ratio shows a shrinking trend before the target disappears, it indicates that the target may be moving away from the image. The window determination submodule ensures that the expected reappearance window extends along the direction of target disappearance. If a residual node is marked as an attitude perturbation residual, the window determination submodule reduces the influence of this residual node on the window orientation determination, preventing aircraft attitude perturbations from causing the window to deviate from the target's likely reappearance area.

[0045] The command generation submodule generates control commands based on the positional relationship of the expected reproduction window relative to the center of the aerial image frame. If the center of the expected reproduction window is located to the left or right of the center of the frame, the command generation submodule generates a gimbal azimuth swing command in the corresponding direction. If the center of the expected reproduction window is located above or below the center of the frame, the command generation submodule generates a gimbal pitch swing command in the corresponding direction. If the scale change reflects that the target is moving away from the image, the command generation submodule generates flight control adjustment commands based on the current flight status of the UAV, enabling the UAV to perform a limited-range search along the possible reproduction direction of the target. The gimbal swing command is output to the gimbal mechanism, and the flight control adjustment command is output to the flight control unit.

[0046] When the target recognition confidence level continuously falls below the recapture confidence threshold, the control module records the low-confidence frame state. This low-confidence frame state is used to determine whether the expected reproduction window needs to be updated. During window updates, the control module does not directly expand the search range to the entire aerial image frame. Instead, it adjusts the window range based on the center position of the existing expected reproduction window, combined with the target bounding box area ratio, attitude perturbation residual markers, and the boundaries of the aerial image frame. If the expected reproduction window is close to the edge of the aerial image frame, the control module restricts the window to the recognizable area to prevent the recapture module from reading invalid regions outside the frame.

[0047] Through the aforementioned control structure, when the target recognition confidence decreases, the control module can utilize the line-of-sight residual chain to determine the local area where the target may reappear, and convert this local area into gimbal sway commands and flight control adjustment commands. In this way, even when the target is temporarily invisible, the UAV and gimbal can still perform continuous control around the line-of-sight trend before the target disappeared.

[0048] Please see Figure 6 The recapture module includes a window receiving submodule, a window feature submodule, a recapture comparison submodule, and a status output submodule. The window receiving submodule, connected to the control module, receives the expected recapture window and determines the image region covered by it. The window feature submodule, connected to the window receiving submodule, extracts the window feature vector of the image region. The recapture comparison submodule, connected to the window feature submodule, compares the window feature vector with the target image feature vector and generates a window matching confidence score. The status output submodule, connected to the recapture comparison submodule, determines the recapture target box based on the window matching confidence score and a recapture confidence threshold, and outputs the tracking status.

[0049] The window receiving submodule reads the center position and window boundaries of the expected reproduction window and maps the window boundaries to the pixel rows and columns of the current aerial image frame. If the expected reproduction window extends beyond the aerial image frame boundaries, the window receiving submodule crops the excess portion within the aerial image frame boundaries and writes this cropping status to the window boundary. The window boundary cropping status is then sent to the window feature submodule along with the corresponding image region, enabling the window feature submodule to identify whether the current window has incomplete boundaries.

[0050] The window feature submodule reads the brightness value, color channel value, and edge direction value of each pixel within the image area covered by the expected reproduction window. The brightness value represents the pixel's lightness / darkness, the color channel value represents the pixel's color state in different color channels, and the edge direction value represents the category of brightness change direction between the pixel and its neighbors. The window feature submodule compares the brightness value of each pixel with the brightness values ​​of its neighbors to determine the location of edge changes and counts the distribution of edge change locations in the horizontal and vertical directions. Simultaneously, the window feature submodule organizes the brightness change distribution, color mean distribution, and edge direction distribution according to pixel row and column order, and writes the distribution counts, brightness change distributions, color mean distributions, and edge direction distributions into the window feature sequence according to a fixed feature order, generating a window feature vector.

[0051] The recognition module and the recapture module each write corresponding feature vectors using a fixed feature order, and the target image feature vectors are used to generate corresponding comparison benchmarks according to the feature extraction standards of the recognition module and the recapture module, respectively. In other words, the recapture comparison submodule compares the window feature vector with the target image feature vector generated according to the recapture module's standards, and the recognition module compares the convolutional feature vector with the target image feature vector generated according to the recognition module's standards. This ensures that the same target undergoes feature arrangement and comparison according to a unified standard in both the recognition and recapture stages, preventing distortion of matching results due to inconsistent feature arrangement.

[0052] The recapture comparison submodule compares the window feature vector with the target image feature vector item by item, and generates a window matching confidence score based on the similarity between the two. If the image region corresponding to the window feature vector has a window boundary cropping state, the recapture comparison submodule only compares the valid window region that has not been cropped, and writes the cropping state into the window matching record. The window matching confidence score is used to represent the degree of confidence in the match between the candidate target and the target image features within the expected reproduction window.

[0053] The status output submodule determines the recapture target box based on the window matching confidence score and the recapture confidence threshold. Specifically, after identifying candidate target boxes within the expected reproduction window, the status output submodule determines whether the center position of the candidate target box is within the coverage area of ​​the expected reproduction window, and whether the area ratio of the candidate target box maintains a consistent scale relationship with the area ratio of the last effective target box before occlusion. When the window matching confidence score meets the recapture confidence threshold, the center position of the candidate target box is within the expected reproduction window, and the area ratio of the candidate target box matches the scale state of the target box before occlusion, the status output submodule determines the candidate target box as the recapture target box and outputs the tracking recovery status.

[0054] If multiple candidate bounding boxes exist within the expected reproduction window, the status output submodule sequentially determines them based on window matching confidence, the positional relationship between the center of the candidate bounding box and the center of the expected reproduction window, and the scale relationship between the area ratio of the candidate bounding box and the area ratio of the last effective bounding box before occlusion. If multiple candidate bounding boxes meet the recapture confidence threshold, the status output submodule prioritizes selecting the candidate bounding box whose center is within the expected reproduction window and whose scale relationship matches the state of the bounding box before occlusion as the recapture target box. If a unique recapture target box still cannot be determined, the status output submodule outputs a recapture waiting state and continues to receive the expected reproduction window and aerial image frames for verification in the next acquisition cycle.

[0055] The tracking status includes normal tracking, recapture waiting, resumed tracking, and search terminated status. Normal tracking indicates that the identification module can stably output target boxes. Recapture waiting indicates that a recaptured target box has not yet been confirmed within the expected reproduction window, but the conditions for continued verification are still retained. Resumed tracking indicates that the recaptured target box has been confirmed, and subsequent processing will continue as in normal tracking. Search terminated status indicates that a recaptured target box cannot be confirmed within the continuous acquisition cycle, and the expected reproduction window can no longer provide a valid recapture range. When outputting the search terminated status, the status output submodule writes the flight observation record corresponding to the last valid target box into the tracking status for mission recording, playback, and subsequent processing.

[0056] When aerial image frames are missing, the entire expected reproduction window is outside the frame, the target image feature vector is missing, or the recapture confidence threshold has not been established, the recapture module does not output the recapture target bounding box. When aerial image frames are missing, the recapture module waits for the next aerial image frame. When the entire expected reproduction window is outside the frame, the recapture module outputs an abnormal window boundary status and causes the control module to re-determine the expected reproduction window. When the target image feature vector is missing, the recapture module outputs a target feature missing status and causes the recognition module to re-establish the target image feature vector. When the recapture confidence threshold has not been established, the recapture module waits for the control module to provide a temporary or stable threshold source.

[0057] In this embodiment, the flight observation records output by the acquisition module provide input to both the identification module and the residual module. The target bounding box center position, target bounding box area ratio, and target identification confidence level output by the identification module provide input to both the residual module and the control module. The line-of-sight persistence residual chain output by the residual module provides a basis for the control module to determine the expected reproduction window. The expected reproduction window output by the control module provides a basis for the reacquisition module to limit the reacquisition range. The tracking status output by the reacquisition module provides a status basis for subsequent tracking control. The above data transmission relationship enables target identification, line-of-sight persistence, window control, and target reacquisition to form a continuous link in the same UAV tracking task.

[0058] Through the above structural setup, aerial image frames, target bounding boxes, gimbal attitude, aircraft attitude, and flight status can be correlated according to image acquisition time. When the target is briefly occluded, the line-of-sight residual chain recorded by the residual module can continue to provide directional guidance for the expected reproduction window. The control module can convert the expected reproduction window into gimbal sway commands and flight control adjustment commands. The reacquisition module can perform window feature extraction and target verification within the image area covered by the expected reproduction window. Therefore, this embodiment can maintain the continuity of tracking control when the image recognition result drops briefly and reduce the entry of irrelevant image areas into the reacquisition comparison process.

[0059] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An unmanned aerial vehicle automatic search and tracking system based on AI image recognition technology, characterized in that, The system includes: The acquisition module is used to acquire aerial image frames, image acquisition time, gimbal azimuth angle, gimbal pitch angle, UAV altitude, UAV heading angle, aircraft roll angle and aircraft pitch angle, and generate flight observation records according to the image acquisition time. The recognition module, connected to the acquisition module, is used to divide the aerial image frame into image regions, extract the convolutional feature vectors of each image region, compare the convolutional feature vectors with the target image feature vectors, and determine the center position of the target box, the area ratio of the target box, and the target recognition confidence. The residual module, connected to the recognition module, is used to calculate the lateral offset change, longitudinal offset change, scale change, gimbal azimuth change, and gimbal pitch change based on the continuous flight observation records, the center position of the target frame, and the area ratio of the target frame, and to generate a line-of-sight-maintaining residual chain. The control module, connected to the residual module, is used to determine the expected reproduction window based on the line-of-sight residual chain when the target recognition confidence is lower than the recapture confidence threshold determined by the average confidence of consecutive valid frames, and to generate gimbal sway commands and flight control adjustment commands according to the expected reproduction window. The recapture module, connected to the control module, is used to receive the expected reproduction window, extract the window feature vector from the image region covered by the expected reproduction window, compare the window feature vector with the target image feature vector, determine the recapture target box based on the recapture confidence threshold, and output the tracking status. 2.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 1, wherein The acquisition module includes an image acquisition submodule, an attitude acquisition submodule, and a record generation submodule; The image acquisition submodule obtains aerial image frames and image acquisition time; The attitude acquisition submodule acquires the gimbal azimuth angle, gimbal pitch angle, UAV altitude, UAV heading angle, aircraft roll angle, and aircraft pitch angle within the acquisition period corresponding to the image acquisition time. The record generation submodule establishes a mapping relationship between aerial image frames and attitude and altitude data within the corresponding acquisition period to generate flight observation records. 3.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 1, wherein The identification module includes a region segmentation submodule, a feature extraction submodule, and a target comparison submodule; The region division submodule divides the aerial image frame into multiple non-overlapping image regions according to the pixel row and column positions of the aerial image frame; The feature extraction submodule obtains the edge direction distribution, color channel mean, and texture change intensity of each image region and generates a convolutional feature vector; The target comparison submodule compares the convolutional feature vector with the target image feature vector to determine the center position of the target box, the area ratio of the target box, and the target recognition confidence. 4.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 1, wherein The residual module includes a record filtering submodule, an offset calculation submodule, and a residual generation submodule; The record filtering submodule filters valid observation records from the continuous flight observation records, including the center position of the target frame and the area ratio of the target frame. The offset calculation submodule calculates the lateral offset change, longitudinal offset change, scale change, gimbal azimuth change, and gimbal pitch change between adjacent valid observation records according to the image acquisition time sequence. The residual generation submodule arranges the changes in sequence according to the image acquisition time to generate a view axis-preserving residual chain. 5.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 4, wherein The residual module is also used for: The center position of the target frame in two adjacent flight observation records is compared horizontally and vertically with the center position of the aerial image frame, respectively, to obtain the horizontal and vertical line-of-sight offsets corresponding to the two flight observation records. The difference between adjacent lateral and longitudinal view axis offsets is calculated to generate lateral offset change and longitudinal offset change. 6.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 4, wherein The residual module is also used for: The gimbal azimuth change is compared with the UAV heading angle change, and the gimbal pitch change is compared with the airframe pitch angle change to obtain the gimbal-airframe compensation deviation. When the gimbal body compensation deviation exceeds the attitude compensation threshold determined by the average of continuous effective compensation deviations, the lateral offset change and longitudinal offset change of the corresponding acquisition period are marked as attitude disturbance residuals. 7.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 1, wherein The control module includes a threshold determination submodule, a window determination submodule, and an instruction generation submodule; The threshold determination submodule determines the recapture confidence threshold based on the average confidence score of consecutive valid frames; When the target recognition confidence level is lower than the recapture confidence threshold, the window determination submodule determines the expected reproduction window based on the line-of-sight preservation residual chain. The instruction generation submodule generates gimbal sway instructions and flight control adjustment instructions based on the positional relationship between the expected reproduction window and the center of the aerial image frame. 8.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 7, wherein, The control module is also used for: Read the lateral and longitudinal offset changes in the view-axis holding residual chain, and determine the directional consistency of each lateral offset change and each longitudinal offset change according to the image acquisition time sequence. When both the horizontal and vertical directions meet the continuity and consistency condition, the center position of the expected reproduction window is determined based on the center position of the nearest target box and the direction of offset change, and the expected reproduction window is generated. 9.The AI image recognition technology-based unmanned aerial vehicle automatic search and tracking system of claim 1, wherein The recapture module includes a window receiving submodule, a window feature submodule, a recapture comparison submodule, and a status output submodule. The window receiving submodule receives the expected reproduction window and determines the image area covered by the expected reproduction window; The window feature submodule extracts the window feature vector of the image region; The recapture comparison submodule compares the window feature vector with the target image feature vector and generates a window matching confidence score. The status output submodule determines the recapture target box and outputs the tracking status based on the window matching confidence and recapture confidence threshold.

10. The drone automatic search and tracking system based on AI image recognition technology according to claim 9, characterized in that, The process of generating the window feature vector specifically includes: Obtain the brightness value, color channel value, and edge direction value of each pixel within the image area covered by the expected reproduction window; The brightness value of each pixel is compared with the brightness value of its neighboring pixels to determine the edge change location, and the distribution number of the edge change locations in the horizontal and vertical directions is counted. The distribution of brightness variation, color mean, and edge direction are statistically analyzed according to the pixel row and column order. The distribution quantity, the brightness variation distribution, the color mean distribution, and the edge direction distribution are written into the window feature sequence according to a fixed feature order to generate the window feature vector.

Images