An unmanned aerial vehicle target person back following method, system, device and storage medium based on natural language conditional constraints

Abstract

The application discloses a kind of unmanned aerial vehicle target person rear following method, system, equipment and storage medium based on natural language condition restriction, it is related to unmanned aerial vehicle autonomous flight control, computer vision and multi-modal intelligent perception technical field.The method includes: through unmanned aerial vehicle vision sensor, candidate character target is detected to environment image acquisition;According to natural language description, the semantic matching of candidate character target area is carried out, and target person is determined in combination with continuity information and position prior information;Establish the appearance memory of target person, keep target identity consistency under the condition of occlusion, detection frame fluctuation and tracking mark change;The orientation state estimation of target person is carried out, and forward probability, backward probability, left probability and right probability are output, and rear relationship confidence is obtained according to backward probability;According to target position deviation, target scale, target locking state and rear relationship confidence, relationship state machine is constructed, and flight process is divided into approach state, round rear acquisition state and rear stable following state;According to current state, corresponding flight control instruction is generated, and unmanned aerial vehicle is driven to realize the rear following of specified target person.The application can solve the problems that it is difficult to specify target according to language, identity is easy to drift, lack of rear relationship modeling and control stage division is not clear in prior art, with good interactivity, stability and engineering application value.

Description

Technical Field

[0001] This invention relates to the fields of autonomous flight control of unmanned aerial vehicles (UAVs), computer vision, multimodal artificial intelligence and intelligent following technology, and in particular to a method, system, device and storage medium for UAV target person following based on natural language condition constraints. Background Technology

[0002] With the development of drone platforms, small airborne vision sensors, and intelligent control technologies, vision-based people-following drone systems have increasingly broad application prospects in scenarios such as security patrols, intelligent companionship, interactive services, and automatic follow-up photography. In existing technologies, common people-following solutions typically achieve drone tracking of target individuals through target detection, single-target tracking, and image-bias-based visual servo control.

[0003] However, existing technologies have the following main shortcomings: First, the target designation capability is insufficient. Existing character-following systems typically can only follow the most prominent character in the current field of view, or rely on manual pre-selection of targets, lacking the ability to distinguish and lock onto specific characters in multi-person scenes based on natural language descriptions. For example, when multiple people are present in a scene, the system struggles to accurately select targets based on natural language descriptions such as "the person wearing purple clothes," "the person with the backpack," or "the person on the left."

[0004] Second, the ability to maintain target identity is insufficient. Most existing systems rely on single-frame detection results, short-term tracking results, or tracking markers to maintain target identity. When short-term occlusion, fluctuations in the target detection box, partial disappearance of the target, multiple people crossing over, or changes in the tracking marker occur, problems such as target drift, identity mismatch, and incorrect target reselection are prone to occur, making it difficult to guarantee identity consistency during long-term continuous tracking.

[0005] Third, there is a lack of relationship-level modeling capabilities. Existing technologies typically focus on target-level information such as "whether a person is detected," "whether the person is centered in the frame," and "whether the distance between the person and the drone is appropriate," while rarely explicitly modeling the relative relationship between the drone and the target person. In particular, for the practically significant relationship constraint of "following behind the target person," existing methods usually lack dedicated state estimation and relationship preservation mechanisms, making it difficult to meet the requirements of applications such as security escort, interactive patrol, and rear-view filming for following relationships.

[0006] Fourth, the control strategy lacks a phased constraint mechanism. Existing follow-up control mostly adopts a uniform visual error closed-loop control logic, which makes it difficult to distinguish the differentiated control requirements of the UAV at different stages, such as approaching the target, flanking to obtain the target's rear position, and stably maintaining the rear-view relationship. When the target turns around, changes direction, performs zigzag movement, or is briefly obstructed, a single control strategy can easily lead to frequent UAV shaking, unstable positioning, loss of relationship, or misjudgment switching.

[0007] Fifth, insufficient adaptability to complex dynamic scenarios. In scenarios with multiple interferences, frequent changes in the target's path, significant changes in the target's posture, and rapid changes in the target's scale and perspective, existing technologies often struggle to simultaneously ensure target selection, identity stability, and maintenance of the relationship between the target and the target, resulting in decreased stability during the following process.

[0008] Therefore, there is an urgent need for a method, system, device, and storage medium for UAV target person back-following that can specify the target person based on natural language in multi-person scenarios, maintain the consistency of target identity when target detection and tracking are unstable, explicitly estimate the back relationship of the target person, and perform phased flight control based on the relationship state machine.

[0009] Therefore, there is an urgent need for a drone-based method and system for following a person's back, which can integrate natural language target designation, identity persistence, back relationship judgment, and relationship-constrained flight control. Summary of the Invention

[0010] Purpose of the invention The purpose of this invention is to provide a method, system, device, and storage medium for drone target person back-following based on natural language condition constraints, in order to solve the following problems existing in the prior art: Existing person-following drones can usually only perform ordinary following of prominent persons in the image or pre-selected persons, and it is difficult to lock onto a specified target in a multi-person scene based on natural language description; identity drift is prone to occur when the detection box fluctuates, is occluded, or the tracking marker changes; existing technologies usually only focus on person target detection and position tracking, lacking explicit modeling and continuous maintenance of the relationship state of "being behind the target person"; at the same time, existing control strategies mostly adopt a single closed-loop logic, which is difficult to adapt to the control requirements of different stages such as approach, flanking acquisition, and stable back-following.

[0011] To this end, this invention proposes an integrated UAV back-following scheme that combines language-guided target locking, appearance memory identity preservation, orientation and back relationship estimation, and relational state machine constraint control, thereby achieving stable, continuous, and interpretable back-following of a designated target person.

[0012] Technical solution To achieve the above objectives, the present invention adopts the following technical solution: A method for drone-based target person tracking based on natural language condition constraints includes the following steps: 1. The system uses a visual sensor mounted on a drone to collect environmental images, detect human targets in the environmental images, and obtain multiple candidate target regions and their location information, scale information, detection confidence information, and optional tracking marker information. 2. Receive natural language description information input by the user, encode the natural language description information into text, extract image features of the target region of each candidate object, and determine the target person from multiple candidates based on the semantic matching degree between the candidate region and the natural language description, combined with the continuity information and location prior information of the candidate target. 3. Establish appearance memory for the identified target person. In subsequent consecutive image frames, based on the similarity between the current candidate target and the historical appearance memory, as well as the target continuity information, the target identity is continuously maintained to suppress target drift under conditions of occlusion, detection jitter, and tracking changes in the identifier. 4. Estimate the orientation state of the target person, output the forward probability, backward probability, left probability and right probability of the target person relative to the drone's view, and obtain the back relationship confidence based on the backward probability; 5. Based on the confidence level of the target's back relationship, the target position deviation, the target scale information, the target lock status, and the continuous and stable counting information, construct a relationship state machine to determine the current following stage. The relationship state machine includes at least the approach state, the flanking acquisition state, and the back stable following state. 6. Generate differentiated flight control commands based on the current state of the relational state machine. In the approach state, control the drone to approach the target. In the behind-the-back acquisition state, control the drone to adjust the lateral offset and relative position to obtain the target's rear relationship. In the stable follow-behind state, control the drone to maintain a safe follow-behind relationship with the target person. 7. Continuously update the perception results, relationship status, and control results to form an integrated closed loop of language constraint target selection, identity maintenance, relationship estimation, and control execution, so as to achieve the following of the designated target person.

[0013] Further technical solutions Based on the above technical solution, preferably, the process of determining the target person includes: 1) Calculate the text semantic matching score between each candidate object target region and the natural language description; 2) Calculate the continuity score between candidate targets and historically locked targets; 3) Calculate the prior scores of the candidate target relative to the positional words in the linguistic description; 4) The text semantic matching score, continuity score, and location prior score are weighted and fused to obtain the comprehensive score of the candidate target; 5) Select the person with the highest overall score as the current target person.

[0014] Beneficial effects Compared with the prior art, the present invention has at least the following beneficial effects:

[0015] In multi-person scenarios, the system can locate a specific target person based on natural language descriptions such as color, clothing, and relative position, thereby improving the system's human-computer interaction capabilities and task flexibility.

[0016] By employing appearance memory and continuity fusion mechanisms, the target identity can remain consistent even under conditions such as occlusion, detection of jitter, and tracking of changes in identifiers, thereby reducing the risk of misfollowing.

[0017] It can not only detect and follow people, but also estimate and maintain the relationship between the drone and the target person, such as whether the drone is behind the target person, thereby improving the ability to understand the relationship in the task of following people.

[0018] By switching between states such as approach, flanking acquisition, and stable following behind, differentiated flight control can be achieved, which can improve the stability of the system in complex path, turning, obstruction and disturbance scenarios.

[0019] This invention is compatible with various target detection models, language coding models, appearance feature extraction models, and flight control platforms, and is suitable for a variety of application scenarios such as security patrol, personnel escort, intelligent tracking, and human-machine collaboration. Attached Figure Description

[0020] Figure 1 This is a block diagram of the overall structure of a drone-based target person following system based on natural language condition constraints according to the present invention.

[0021] Figure 2 This is a flowchart illustrating a method for drone-based target person following method based on natural language condition constraints according to the present invention.

[0022] Figure 3 This is a schematic diagram of the workflow for persistently maintaining the target identity and generating the relationship state in this invention.

[0023] Figure 4 This is a schematic diagram of the relational state machine and hierarchical control logic in this invention. Detailed Implementation

[0024] The present invention will be further described below with reference to specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] This embodiment provides a drone-based target person back-following system based on natural language condition constraints. The system includes a visual perception module, a language-guided target locking module, an appearance memory and identity persistence module, an orientation and back-relationship estimation module, a relationship state machine module, and a relationship constraint control module.

[0026] The visual perception module is deployed on the UAV's onboard computing platform or an edge computing platform connected to the UAV. It processes the video stream captured by the UAV's forward-looking camera in real time and outputs candidate target areas. The language-guided target locking module receives natural language descriptions input by the user, such as "follow the person wearing purple clothes" or "follow the person with the backpack," and combines them with the image features of the candidate areas for matching and scoring, thereby locking onto the target person in a multi-person scene.

[0027] The appearance memory and identity persistence module establishes historical appearance memories for locked targets, maintaining target identity consistency based on appearance similarity and positional continuity under conditions of detection frame drift, short-term occlusion, and changes in tracking markers. The orientation and back view relationship estimation module estimates the orientation of the target relative to the UAV's viewpoint and outputs a back view relationship confidence score. The relationship state machine module determines the current flight phase based on target size, lateral deviation, target lock status, and back view relationship confidence score. The relationship constraint control module generates forward, lateral, and yaw control commands based on the state machine output and sends them to the flight control system for execution.

[0028] In this embodiment, the system first extracts multiple candidate object bounding boxes from the current image frame using a person detection model. For each candidate object bounding box, the system crops the corresponding image region and extracts the region's image features. Simultaneously, the system performs text encoding on the user-input natural language description information to obtain a text feature vector.

[0029] Subsequently, the system calculates the semantic similarity between each candidate object target region and text features to form a text matching score. To avoid false detections caused by relying solely on single-frame text matching, the system further introduces a candidate target continuity score and a positional prior score. The candidate target continuity score describes the center position, scale, and historical locking consistency between the current candidate target and the target locked at the previous time step; the positional prior score handles positional constraints such as "left," "right," and "middle" in the linguistic description.

[0030] Finally, the system weights and fuses the text matching score, continuity score, and location prior score to obtain a comprehensive score, and identifies the candidate with the highest comprehensive score as the current target person. This method enables language condition locking for a specific person in a multi-person scenario.

[0031] In this embodiment, for target individuals already identified through language conditions, the system establishes appearance memory features. These appearance memory features can be obtained from the target individual region through an appearance feature extraction network. During the initial identification phase, the system uses the first reliable target region features as the initial memory. In subsequent processing, the system only updates the appearance memory using an exponential sliding update method when the current candidate target meets conditions such as high detection confidence, reasonable language matching, and good target continuity.

[0032] The following situations occur: the target detection box drifts slightly; the tracking marker changes; the target is briefly occluded and then reappears; The system performs a similarity match between the appearance features of the current candidate target and historical appearance memories, and combines this with target continuity information to determine whether the current candidate target is still the original locked target. If both similarity and continuity conditions are met, the target's identity remains unchanged; otherwise, its priority as the current target is reduced.

[0033] Through the above mechanism, the system can maintain the consistency of the target identity in multi-person scenarios, reducing the common problem of accidental switching in ordinary tracking methods.

[0034] In this embodiment, after locking onto the target person, the system estimates the orientation state of the target person's region. Orientation state estimation can be achieved by combining image feature branches and pose feature branches.

[0035] The system employs two branches: an image feature branch to extract overall appearance and contour information from the image of the person; and a pose feature branch to extract structured features such as visibility features of key human points, shoulder width features, hip width features, differences between left and right key points, and the aspect ratio of the target. After fusing these two types of features, the system outputs the forward, backward, left, and right probabilities of the target person relative to the drone's viewpoint.

[0036] In this implementation, the backward probability is used as the confidence level of the back-to-back relationship to characterize the credibility of the relationship between the current drone and the target person. The higher the confidence level of the back-to-back relationship, the closer the current viewpoint is to the state where the target person is facing away from the drone; the lower the confidence level of the back-to-back relationship, the more likely the current viewpoint is biased towards the front, side, or an unclear relationship.

[0037] In this embodiment, the system constructs a relational state machine to manage the current flight phase of the UAV. The relational state machine includes at least an approach state, a behind-the-back acquisition state, and a stable following state. 1. Approaching state

[0038] When the target person has been locked but the distance between the drone and the target is too far, the system enters the approach state. In this state, the control module controls the drone to move closer to the target person based on the target's lateral error and scale error in the image, gradually bringing the target closer to the center of the image. 2. Obtain the state by flanking.

[0039] When the drone approaches the target person, but the back-to-back relationship has not yet been established or is unstable, the system enters the rear-view acquisition state. In this state, the control module generates yaw control, forward control, and lateral control commands based on the back-to-back relationship confidence, lateral deviation, and preset lateral offset, driving the drone to move to the side and rear of the target person to gradually establish the target's rear-view relationship. 3. Stable back support

[0040] When the system determines that the confidence level of the target person's back relationship is higher than the set entry threshold, the target's lateral error is within an acceptable range, and the target's size is within the set following range, the system enters a stable back-following state. In this state, the control module reduces the forward and lateral control gains, focusing on maintaining the relative rear position and safe distance between the drone and the target person.

[0041] In addition, the relational state machine is equipped with an exit threshold and an anomaly counter. When the confidence of the back relation decreases, the target scale deviates significantly, or the target lock fails, the system retreats from the stable back-following state to the around-to-acquire state or the near state to re-establish the back relation. This state switching method with hysteresis can reduce the impact of instantaneous fluctuations on the control state.

[0042] In this embodiment, the system operates continuously in cycles of video frames. Within each cycle, the following are executed sequentially: image acquisition and candidate target detection; voice-guided target locking; target appearance memory matching and identity update; orientation and back-relationship estimation; state machine update; and control command generation and transmission.

[0043] After receiving the target position error, target size, back-to-back relationship confidence, target lock status, and relationship state machine results at the current moment, the controller generates corresponding flight control commands. After the UAV executes these control commands, its new image observation results are fed back to the perception module, thus forming a continuous closed loop. In practical applications, when the target performs straight-line movement, turns, short-term turns, or complex path movements, the system can switch between different stages through the state machine to achieve a continuous control process from ordinary approach to back-to-back relationship establishment and then to stable back-to-back following.

[0044] This invention can be applied to, but is not limited to, the following scenarios: Security patrol scenario: The drone locks onto the designated patrol personnel based on natural language descriptions and remains in a safe area behind them to conduct accompanying patrols.

[0045] Intelligent companionship scenario: Users specify the target person to be accompanied via voice. The drone selects the correct target in a multi-person scenario and provides stable companionship from behind the target.

[0046] Interactive tracking shooting scenario: The drone locks onto the subject based on natural language, maintains the shooting relationship from behind, reduces frontal interference and obstruction, and improves the shooting experience.

[0047] The above-described embodiments are merely preferred embodiments of the present invention. For those skilled in the art, various equivalent transformations, substitutions, and improvements can be made without departing from the principles and concepts of the present invention, and all such equivalent transformations, substitutions, and improvements should fall within the protection scope of the present invention.

Claims

1. A method for drone-based target person tracking based on natural language condition constraints, characterized in that, Includes the following steps: Step 1: The visual sensor on the drone collects environmental images, detects human targets in the environmental images, and obtains at least one candidate human target region and its corresponding position parameters, scale parameters and confidence parameters. Step 2: Obtain the natural language description information input by the user, encode the text features of the natural language description information, extract image features from each candidate target region, and determine the target person based on the semantic matching degree between each candidate target region and the natural language description information, combined with the continuity information and location prior information of the candidate targets. Step 3: Establish an appearance memory for the target person. In subsequent consecutive image frames, based on the similarity between the current candidate target and the historical appearance memory, as well as the target continuity information, the identity of the target person is continuously maintained. Step 4: Estimate the orientation of the target person to obtain the forward probability, backward probability, left probability and right probability of the target person relative to the drone's view, and determine the confidence level of the target person's back relationship based on the backward probability. Step 5: Based on the back relationship confidence, the positional deviation of the target person in the image, the target scale parameters, and the target identity preservation state, construct a relationship state machine to determine the current following stage of the UAV. The relationship state machine includes at least the approach state, the flanking acquisition state, and the back stable following state. Step 6: Generate corresponding flight control commands based on the current state of the relational state machine to drive the UAV to perform approach control, flanking control, or stable follow control on the target person. Step 7: During the flight of the drone, the detection results, identity preservation results, orientation state estimation results, and relational state machine results of the target person are updated in real time, thereby realizing the natural language conditional constraint back-following of the target person.

2. The method for drone-based target person following method based on natural language condition constraints according to claim 1, characterized in that, Step 2, in which the target person is identified, includes: Calculate the text semantic matching score, target continuity score, and location prior score for each candidate object's target region. The text semantic matching score, target continuity score, and location prior score are weighted and fused to obtain a comprehensive score for the candidate target; The candidate with the highest overall score is identified as the target person.

3. The method for drone target person following method based on natural language condition constraints according to claim 1, characterized in that, Step 3, which involves maintaining the identity of the target person, includes: Extract the appearance feature vector of the current candidate target; The appearance feature vector is matched with the target appearance features in historical appearance memory to obtain an appearance similarity score; Calculate the target continuity score by combining the target's historical center location, historical tracking markers, and the current candidate target location; Based on the appearance similarity score and the target continuity score, it is determined whether the current candidate target is still the target person; The appearance memory is established and maintained by combining initialization update and exponential sliding update to maintain the consistency of the target identity when the target is temporarily occluded, the detection box fluctuates, or the tracking marker changes.

4. The method for drone-based target person following method based on natural language condition constraints according to claim 1, characterized in that, The orientation state estimation in step 4 is based on the image features and / or human pose features of the target person region, and outputs the forward probability, backward probability, left probability and right probability of the target person relative to the drone's view, and uses the backward probability as the confidence of the back relationship; wherein, the human pose features include at least one of the following: head key point visibility features, shoulder key point visibility features, hip key point visibility features, shoulder width features, hip width features, left and right key point difference features, target box aspect ratio features and motion trend features.

5. The method for drone target person following method based on natural language condition constraints according to claim 1, characterized in that, The state transition rules of the relational state machine in step 5 include: When the target has been identified and the target size has reached the approach threshold, switch from the approach state to the flanking acquisition state. When the confidence level of the back-to-back relationship is higher than the entry threshold, the target lateral deviation is less than the set threshold, and the target scale is within the following scale range, the state switches from the back-to-back acquisition state to the back-to-back stable following state. When the confidence level of the back-to-back relationship is lower than the exit threshold, the target lateral deviation is greater than the loss threshold, or the target scale deviates from the following scale range, the back-to-back stable following state is switched to the back-to-back acquisition state. The relational state machine employs a hysteresis switching mechanism that separates entry and exit conditions, and combines a continuous stable frame counter and a continuous abnormal frame counter to reduce frequent state jitter.

6. The method for drone-based target person following method based on natural language condition constraints according to claim 1, characterized in that, Step 6 includes: In the approach phase, the drone is controlled to perform approach control based on the lateral error of the target person in the image and the target scale error. In the flanking acquisition state, the control drone performs lateral flanking adjustment and forward approach control based on the confidence level of the target person's back relationship, lateral error and target offset. In the back-stabilized following state, the control drone performs back-stabilized following control based on the target person's lateral error, target size error and back-relation confidence. The flight control commands in the behind-the-back acquisition state include yaw control components, forward velocity control components, and lateral velocity control components. The lateral velocity control components are used to drive the UAV to shift behind the target person in order to establish a back-to-back relationship. The flight control commands in the back-to-back stable following state include target distance maintenance control, lateral error suppression control, and back-to-back relationship maintenance control, so that the UAV remains in a safe area behind the target person.

7. The method for drone-based target person following method based on natural language condition constraints according to claim 1, characterized in that, When the target's identity is detected to be locked, the confidence of the back-to-back relationship decreases, or the target disappears briefly, the relationship state machine reverts from the stable back-to-back following state to the around-to-back acquisition state or the approach state in order to re-establish the back-to-back following relationship.

8. A drone-based target person tracking system based on natural language condition constraints, characterized in that, include: The visual perception module is used to acquire environmental images and detect candidate objects. The language-guided target locking module is used to identify the target person from multiple candidate targets based on natural language description information; The appearance memory and identity retention module is used to establish the appearance memory of the target person and maintain the consistency of the target's identity in consecutive image frames; The orientation and back-back relationship estimation module is used to estimate the forward, backward, left, and right probabilities of the target person relative to the drone's viewpoint, and output the back-back relationship confidence score. The relational state machine module is used to determine the current following phase of the UAV based on the back relation confidence, target position deviation, target scale parameters, and target identity status. The relationship constraint control module is used to generate corresponding flight control commands based on the current following stage, so as to drive the drone to follow the target person's back.

9. The system according to claim 8, characterized in that, The language-guided target locking module is used to fuse and score text semantic matching score, continuity score and location prior score to determine the target person; The orientation and back-back relationship estimation module is used to output forward probability, backward probability, left probability, and right probability; The relational state machine module includes at least an approach state submodule, a behind-the-back acquisition state submodule, and a back-stable following state submodule, and realizes state switching based on hysteresis threshold and stable counting mechanism.

10. An electronic device, characterized in that, The device includes at least one processor and at least one memory, wherein the at least one memory stores a computer program that, when executed by the at least one processor, causes the electronic device to perform the method as described in any one of claims 1 to 7.

Images