Person interaction detection method based on open vocabularies in unmanned aerial vehicle scene

Abstract

The invention provides a character interaction detection method based on open vocabularies in an unmanned aerial vehicle scene, and the method comprises the steps: extracting global features of an image through employing a pre-trained CLIP visual encoder, segmenting the image into a plurality of image blocks, and carrying out the coding of the image blocks; an MLR module is introduced to extract global context information, character interaction is decoded from a multi-level feature map, and when bipartite graph matching is carried out between a prediction result and a real result, a loss function is designed to guide a low-level feature map to correspond to a character pair with a small distance and guide a high-level feature map to correspond to a character pair with a large distance; a large language model is used for generating human body part state description related to character interaction in an image, character interaction category names and the state description of the related human body parts are coded into text embedding, and the embedding is combined with the output of a character interaction decoder and global context information extracted by an MLR module to obtain an instance-level detection score. According to the invention, the model can better adapt to interaction detection requirements under different distances.

Description

Technical Field

[0001] This application relates to the fields of human interaction detection and recognition, and intelligent monitoring, and particularly relates to a method for detecting human interaction based on open vocabulary in an unmanned aerial vehicle (UAV) scenario. Background Art

[0002] In traditional object detection tasks, models usually need to be trained and evaluated on a finite, pre-defined set of categories. However, there are a large number of object categories in the real world, which limits the generality and flexibility of traditional object detection models in practical applications. Open-vocabulary object detection is an emerging computer vision task that aims to detect objects in images, even if the categories of these objects are not explicitly labeled or recognized during the training phase. The key technologies of open-vocabulary object detection include multi-modal feature fusion, cross-modal attention mechanisms, and dynamic vocabulary generation, etc. The development of these technologies not only promotes the research of open-vocabulary object detection, but also provides new ideas and methods for human interaction detection in UAV scenarios.

[0003] The human interaction detection technology is an interdisciplinary subject of object detection, behavior recognition, and visual relationship detection, aiming to identify the mutual relationship between people and objects in specific application scenarios. Human interaction detection methods can be divided into two categories: global instance-based and local instance-based, and are further subdivided according to the differences in the visual features used, including fusing spatial position information, fusing appearance information, and fusing human pose information. In addition, the applications of zero-shot learning, weakly supervised learning, and Transformer models in human interaction detection have also been widely discussed.

[0004] The development of UAV technology has brought new challenges and opportunities to the field of computer vision. UAVs have the characteristics of small size, strong flexibility, wide aerial photography vision, etc., and are widely used in multiple industries, such as police patrol, urban traffic supervision, weather monitoring, etc. With the development of deep learning technology, the object detection technology in the UAV field has been significantly improved and strengthened. Especially in the open-vocabulary scenario, the method for detecting human interaction in UAVs is particularly important, which can identify and understand the complex interaction relationship between people and objects.

[0005] The emergence of large-scale vision-language models has brought a brand-new idea to the open-vocabulary detection task in UAV scenarios. The multi-modal algorithm model learns the joint embedding space of images and texts from large-scale image-text datasets and shows powerful capabilities in image classification tasks. Applying the pre-trained vision-language model as an external knowledge base to the open-vocabulary object detector helps to improve the model generalization ability and predict unknown categories.

[0006] In addition, by leveraging the extensive world knowledge of large language models and using this knowledge to generate language descriptions of the states of human body parts, the prompting ability of language can better reduce the interference brought by complex environments, thereby guiding the model to more accurately provide object detection results and human interaction relationships in the picture.

[0007] In summary, traditional methods usually use feature maps of the same level to model human interactions at different distances, which may lead to poor performance in identifying long-distance interactions. In addition, these methods mainly rely on class names and ignore the rich context information provided by language. Therefore, innovative methods need to be introduced to improve and optimize the open-vocabulary based human interaction detection model to better meet the needs of drone intelligent monitoring and detection in various environments. Summary of the Invention

[0008] In view of the above technical problems, it is necessary to provide a method for detecting human interactions based on open vocabulary in a drone scenario, which can detect and classify untrained objects, identify human interaction relationships, reduce the impact of complex shooting scene conditions of drones on the detection and classification results, achieve zero-shot prediction of unknown classes, and can gradually learn new unknown classes, reducing the cost of manual annotation.

[0009] Technical Solution: To achieve the above invention purpose, the present invention proposes a method for detecting human interactions based on open vocabulary in a drone scenario, and the method includes the following steps:

[0010] S1. Subdivide the data set into a training set, a validation set, and a test set, and perform resampling and normalization processing on the data set;

[0011] S2. Use a pre-trained CLIP visual encoder to extract the global features of the image. The visual encoder divides the image into multiple image patches and encodes each image patch to generate a multi-level feature map;

[0012] S3. After feature extraction, introduce an MLR module to extract global context information. At the same time, decode human interactions from the multi-level feature map in parallel. When performing bipartite graph matching between the prediction result and the ground truth result, a unique loss function is designed to guide the low-level feature map to correspond to human pairs at smaller distances and the high-level feature map to correspond to human pairs at larger distances;

[0013] S4. Use a large language model to generate descriptions of the states of human body parts related to human interactions in the image, encode the human interaction class names and the state descriptions of relevant human body parts into text embeddings, and combine these embeddings with the output of the human interaction decoder and the global context information extracted by the MLR module to obtain instance-level detection scores.

[0014] Furthermore, the specific method of S1 is as follows:

[0015] Divide the original dataset into a training set, a validation set, and a test set according to a specified ratio. Ensure that the division of the dataset is reasonable to guarantee the representativeness and independence of the data sources for model training, validation, and testing. Define the interaction as a quadruple (b o , b h , o, a), where respectively represent the bounding boxes of the detected person and object instances. respectively represent the object and action categories, where and respectively represent the human action and object sets. The goal of open-vocabulary person interaction detection is to accurately detect a wide range of interactions. This requires the ability to recognize interactions not encountered during the training phase, including unseen objects, actions, and their various combinations.

[0016] Furthermore, the specific method of S2 is as follows:

[0017] Utilize the generalization ability of CLIP for classification tasks to construct an end-to-end open-vocabulary person interaction detector, and convert person interaction detection into an end-to-end set matching problem similar to DETR, thus eliminating the need for manual components such as anchor generation. Given an image I, first, the pre-trained CLIP visual encoder E V extracts the global context representation F img :

[0018] F img = E V (I) (1)

[0019] where, represents a series of feature embeddings of I, and then the Transformer-based decoder D is used to decode the person interactions in I. Specifically, the projected context features and the person interaction query Q = (q1, q2,..., q M ) are used as inputs, and the output of the last layer of D is used as the representation of the interaction:

[0020] H = D(Q, F img ) (2)

[0021] where, in the cross-attention mechanism of the person interaction decoder D, Q is used as the query, the projected context representation is used as the key-value, H = (h1, h2,..., h M ), where M represents the number of HOI queries. Then, the context features, the person interaction query Q, and the feature embeddings F img of the image are provided to two different head networks: 1) a bounding box regressor P bbox, which predicts a confidence score c and the bounding boxes of the interacting person and object (b h , b o ). 2) The linear projection P that maps features to the visual and text joint space cls . Calculate its similarity with the text encoder features for interaction recognition, and finally calculate the box regression loss L b , the intersection over union loss L iou and the interaction classification loss L cls . The matching cost formula is:

[0022]

[0023] The visual encoder E v divides the fixed-resolution image I into small patches and projects these patches as a sequence. Assume where is the i-th patch of E V . Let the input be X0, and E V encodes the image through the self-attention mechanism of each layer:

[0024]

[0025] where X i is the encoded feature map after the i-th patch of E V . Then, multi-level feature maps are used to parallelly decode human interactions:

[0026]

[0027] where represents the set of meaningful levels. Then, the decoded human interaction hidden state is sent to the bounding box regressor and the interaction classifier to predict the bounding boxes of human pairs and interaction categories respectively.

[0028] Furthermore, specifically in S3:

[0029] In the bipartite matching between the N prediction sets of each image and the set of ground-truth human interactions , a unique loss is designed to guide H decoded from the low-level feature map i to simulate human interactions with relatively small center distances, and to guide the decoding from the high-level feature map to simulate human interactions with relatively large center distances, so as to reduce the performance gap between human interactions at different distances. Among them, the predicted set of human interactions includes the bounding boxes of people and objects, the classes of people and objects, the categories of interactions, and confidence scores; the set of ground-truth human interactions y includes the bounding boxes of people and objects, the classes of people and objects, and the categories of interactions for each ground-truth human interaction. Specifically, in The normalized feature layer level and y j A new constraint loss is designed between the person distances

[0030]

[0031] where is The normalized feature layer level, with subscript i corresponding to the i-th prediction in the prediction set, and g(y j ) is the normalized distance between the center of the person annotated in y j and the object, with subscript j corresponding to the j-th true person interaction in the true label set. This design guides the model to classify person interactions with different distances into separate groups and uses specific feature maps customized for the features of each group to process the person interactions in each group.

[0032] To ensure that the model can correctly select the appropriate feature layer level for person interactions at different distances, a conditional matching mechanism is introduced. Conditional matching is achieved by adjusting the matching cost, which not only considers the standard matching loss L cost , but also includes an additional constraint loss L d :

[0033] L = L cost + L d (7)

[0034] L cost includes the bounding box regression loss, the intersection over union loss, and the interaction classification loss, while L d is the newly introduced loss term used to guide different feature levels to focus on different types of person interactions.

[0035] For each predicted person interaction and true person interaction, an N×M cost matrix is constructed, where N is the number of predictions and M is the number of true labels. Then the constraint loss L d is added, and the Hungarian algorithm is used to solve the optimal matching of the cost matrix to obtain the matching relationship between each predicted person interaction and true person interaction. Then, the loss between the predicted person interaction and the true person interaction of the model is calculated according to the matching result, and finally the model parameters are updated through backpropagation.

[0036] The global maximum pooling operator and the concatenation of all feature vectors at different levels are used to obtain the global image embedding e global . Then the global image embedding is input into two branches, where the text MLR branch aligns e global with the text embeddings of different classes generated by the CLIP text encoder, while the visual MLR branch extracts the global image embedding of the CLIP image encoder as gglobal 。

[0037] The text MLR aims to align the image embeddings with the corresponding class text embeddings generated by the CLIP text encoder, so as to classify each image according to the CLIP text embeddings. First, the linear layer f(·) is used to globally project the image embedding e into a new feature space, that is, the learned text embedding space, to obtain e text = f(e global ). Then, a template filled with class names is fed into the CLIP text encoder to obtain the text embedding t of a specific class.

[0038] The classification score for each class c of image i is calculated using cosine similarity:

[0039]

[0040] At the same time, the MLR ranking loss is adopted to increase the scores of positive classes (i.e., the classes that appear in the image) and reduce the negative classes (i.e., the classes that the image does not contain). The ranking loss is defined as:

[0041]

[0042] where N(i) represents the image-level label of image i, and s i,n and s i,p represent the classification scores for the positive and negative classes with respect to the image, respectively.

[0043] The visual MLR branch extracts knowledge from the CLIP image encoder to achieve better zero-shot recognition performance. This is because the image encoder of CLIP has been trained on a large number of images using corresponding text supervision. This enables it to extract more generalized image embeddings, which include features of basic classes and new classes, as well as important background environments. To extract these generalized embeddings, another linear layer f(·) is used to convert the global image embedding into a learned image embedding Then, the L1 loss is used to minimize the distance between e image and the embedding of the CLIP image encoder:

[0044]

[0045] where IE(i) represents the embedding of image i from the CLIP image encoder. Since IE(i) contains rich new class knowledge and important context features, the learned e image can well complement e text in the text MLR to detect new object classes, especially those that require recognizing the global image context.

[0046] Furthermore, the method of S4 is as follows:

[0047] To address the limited distinction between human interaction names, semantic information provided by descriptions of body parts related to each human interaction is proposed. This means that for each type of human interaction, not only its name but also the state of the relevant human body parts needs to be encoded. Given a human interaction category defined as an action-object pair, denoted as (act i ,obj i ), a sentence is formed using the globally learnable context token Ctx.

[0048] At the beginning of the sentence, several prefix tokens Ctx pre and several learnable connection tokens Ctx con are used to automatically determine how to connect the category names of the action and the object. Then, through the pre-trained CLIP text encoder E T a text embedding is generated for each HOI label:

[0049] T hoi = E T (Ctx pre ,act i ,Ctx con ,obj i ) (11)

[0050] The embedding results of all human interaction names are denoted as where N represents the number of human interaction triple categories, and C t represents the text embedding dimension of the CLIP text encoder.

[0051] In addition, to encode the description Si of the human body part state for each HOI, the CLIP text encoder E T is used:

[0052] T b = E T (S i ) (12)

[0053] where, represents the embedding of the body part descriptions of all human interactions, and combining the output logarithms of T hoi and T b gives the final interaction prediction:

[0054]

[0055] where h j represents the hidden state of the human interaction, p j represents the corresponding interaction prediction, α hoi and α bThey are learnable scalars used to balance the weights of the person interaction embedding and the body part embedding respectively.

[0056] Given a test image the model produces a set of instance predictions {(b, p ovod ) j}, where b j are the bounding box coordinates, denotes the classification probability score for the j-th instance of all C = |C test | classes. Then the MLR module predicts the image-level classification score Although the MLR module cannot localize objects, it provides scene context information and prior knowledge about the types of objects that may exist in the entire image. This context information enhances the region-level detection performance, especially when detecting targets under visual interference. The following weighted geometric mean is used to combine the image-level score p mmlr and the instance-level score :

[0057]

[0058] where denotes the context-aware detection score for the j-th instance, ⊙ denotes element-wise multiplication, and γ is a hyperparameter that balances these two scores.

[0059] Advantageous effects. Compared with the prior art, the technical solution of the present invention has the following advantageous technical effects:

[0060] (1) The present invention designs a person interaction detection framework based on open vocabulary. By using a large language model to generate descriptions of the human body parts participating in person interaction in the training images, and using the pre-trained vision-language model CLIP to convert visual information into text information, and taking both as external references, it can detect objects that have not been trained and achieve zero-shot prediction for unknown classes. The present invention introduces an additional constraint loss function to guide the model to match local interactions with a closer person-object distance with low-level feature maps and global interactions with a farther person-object distance with high-level feature maps, so that the model can better adapt to the interaction detection requirements at different distances;

[0061] (2) The present invention designs a person interaction detection framework based on open vocabulary in the UAV scenario. The global features of the image are extracted by the MLR module, and multi-label classification information is learned using two branches: text and vision. The text branch aligns the image features with the CLIP text embeddings to learn image classification. The vision branch aligns the image features with the CLIP image embeddings to learn image classification and improve zero-shot recognition ability. The image-level classification scores are fused with the instance-level scores predicted using a large language model and the pre-trained vision-language model CLIP to obtain scene-aware detection scores, and the scene-aware detection scores are used for object detection to improve the detection performance, especially when detecting objects under visual interferences such as small size, blur, or occlusion. Description of the Drawings

[0062] Figure 1 It is a flowchart of the method for the person interaction detection framework based on open vocabulary in the UAV scenario. Detailed Embodiment

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] As Figure 1 shown, the present invention proposes a method for a person interaction detection framework based on open vocabulary in the UAV scenario. The method includes the following steps:

[0065] Step S1: Divide the original dataset into a training set, a validation set, and a test set according to a specified ratio. Ensure that the division of the dataset is reasonable to ensure that the data sources for model training, validation, and testing are representative and independent. Divide the original dataset into a training set, a validation set, and a test set according to a specified ratio. Ensure that the division of the dataset is reasonable to ensure that the data sources for model training, validation, and testing are representative and independent. Define the interaction as a quadruple (b o ,b h ,o,a), where respectively represent the bounding boxes of the detected person and object instances. respectively represent the object and action categories, where and respectively represent the human action and object set.

[0066] Step S2: Utilize the generalization ability of CLIP for classification tasks to construct an end-to-end open vocabulary person interaction detector, and convert the person interaction detection into an end-to-end set matching problem similar to DETR, thus eliminating the need for manual components such as anchor generation. Given an image I, first, the pre-trained CLIP vision encoder E VExtract the global context representation F img :

[0067] F img = E V (I)(1)

[0068] Among them, represents a series of feature embeddings of I, and then uses the Transformer-based decoder D to decode the human interactions in I. Specifically, the projected context features and the human interaction query Q = (q1, q2,..., q M ) are used as inputs, and the output of the last layer of D is used as the representation of the interaction:

[0069] H = D(Q, F img )(2)

[0070] Among them, in the cross-attention mechanism of the human interaction decoder D, Q is used as the query, the projected context representation is used as the key value, H = (h1, h2,..., h M ), where M represents the number of HOI queries. Then the context features, the human interaction query Q, and the feature embeddings F of the image img are provided to two different head networks: 1) A bounding box regressor P bbox , which predicts a confidence score c and the bounding boxes (b n , b o ) of the interacting person and object. 2) A linear projection P cls that maps the features to the visual and text joint space. Calculate its similarity with the features of the text encoder for interaction recognition, and finally calculate the box regression loss L b , the intersection over union loss L iou and the interaction classification loss L cls . The matching cost formula is:

[0071]

[0072] The visual encoder E v divides the image I with a fixed resolution into small patches and projects these image patches as a sequence. Assume where is the i-th patch of E V . Let the input be X0, and E V encodes the image through the self-attention mechanism of each layer:

[0073]

[0074] where X i is the encoded feature map after the i-th patch of E V . Then use the multi-level feature maps to parallelly decode the human interactions:

[0075]

[0076] wherein represents a set of meaningful levels. Then, the decoded hidden state of the human interaction is fed into the bounding box regressor and the interaction classifier to predict the bounding boxes of the human pairs and the interaction categories respectively.

[0077] Step S3: In the bipartite matching between the N prediction sets of each image and the set of true human interactions a unique loss is designed to guide the H decoded from the low-level feature map i to simulate human interactions with relatively small center distances, and guide the decoding from the high-level feature map to simulate human interactions with relatively large center distances, so as to reduce the performance gap between human interactions at different distances. Among them, the predicted set of human interactions includes the bounding boxes of people and objects, the classes of people and objects, the categories of interactions, and the confidence scores; the set of true human interactions y includes the bounding boxes of people and objects, the classes of people and objects, and the categories of interactions for each true human interaction. Specifically, a new constraint loss is designed between the normalized feature map level of j and the human distances in y

[0078]

[0079] wherein is the normalized feature map level of j , the subscript i corresponds to the i-th prediction in the prediction set, and g(y j ) is the normalized distance between the centers of the person and the object labeled in y

[0080] This design guides the model to classify human interactions with different distances into separate groups and uses specific feature maps customized for the features of each group to process the human interactions in each group. cost , and also includes an additional constraint loss L d :

[0081] L = L cost + L d (7)

[0082] L costIt includes the bounding box regression loss, the intersection over union loss, and the interaction classification loss, L d is a newly introduced loss term used to guide different feature levels to focus on different types of human interactions.

[0083] For each predicted human interaction and the ground truth human interaction, an N×M cost matrix is constructed, where N is the number of predictions and M is the number of ground truth labels. Then the constraint loss L d is added, and the Hungarian algorithm is used to solve the optimal matching of the cost matrix to obtain the matching relationship between each predicted human interaction and the ground truth human interaction. Then, the loss between the predicted human interaction and the ground truth human interaction of the model is calculated according to the matching result, and finally the model parameters are updated through backpropagation.

[0084] Text MLR aims to align the image embedding with the corresponding class text embedding generated by the CLIP text encoder so as to classify each image according to the CLIP text embedding. First, the image embedding e is globally projected into a new feature space, i.e., the learned text embedding space, using a linear layer f(·) to obtain e text = f(e global ). Then a template filled with the class name is fed into the CLIP text encoder to obtain the class-specific text embedding t.

[0085] The classification score for each class c of image i is calculated using cosine similarity:

[0086]

[0087] At the same time, the MLR ranking loss is adopted to increase the scores of positive classes (i.e., the classes that appear in the image) and reduce the negative classes (i.e., the classes that the image does not contain), and the ranking loss is defined as:

[0088]

[0089] where N(i) represents the image-level label of image i, s i,n and s i,p represent the classification scores for the positive and negative classes with respect to the image, respectively.

[0090] The visual MLR branch extracts knowledge from the CLIP image encoder to achieve better zero-shot recognition performance. This is because the image encoder of CLIP has been trained on a large number of images using the corresponding text supervision. This enables it to extract more generalized image embeddings, which include features of basic classes and new classes, as well as important background environments. To extract these generalized embeddings, another linear layer f(·) is used to convert the global image embedding into a learned image embedding Then the L1 loss is used to minimize e imageDistance between the embeddings of the CLIP image encoder:

[0091]

[0092] where IE(i) represents the embedding of image i from the CLIP image encoder. Since IE(i) contains rich new category knowledge and important context features, the learned e image can well complement the e in the text MLR text to detect new object categories, especially those that require identifying the global image context.

[0093] Step S4: Use several prefix tokens Ctx at the beginning of the sentence pre and several learnable connection tokens Ctx con to automatically determine how to connect the class names of actions and objects. Then, through the pre-trained CLIP text encoder E T generate text embeddings for each HOI label:

[0094] T hoi = E T (Ctx pre , act i , Ctx con , obj i ) (11)

[0095] The embedding results of all human interaction names are denoted as where N represents the number of human interaction triple categories, and C t represents the text embedding dimension of the CLIP text encoder.

[0096] In addition, to encode the human body part state description Si of each HOI, use the CLIP text encoder E T :

[0097] T b = E T (S i ) (12)

[0098] where represents the embedding of all human interaction body part descriptions, and combining the output logarithms of T hoi and T b obtains the final interaction prediction:

[0099]

[0100] where h j represents the hidden state of the human interaction, p j represents the corresponding interaction prediction, and α hoi and αb They are learnable scalars used to balance the weights of the person interaction embedding and the body part embedding respectively.

[0101] Given a test image the model generates a set of instance predictions {(b, p ovod ) j}, where b j are the bounding box coordinates, denotes the classification probability score of the j-th instance of all C = |C test | classes. Then the MLR module predicts the image-level classification score Although the MLR module cannot localize the objects, it provides scene context information and prior knowledge about the types of objects that may exist in the whole image. This context information enhances the region-level detection performance, especially when detecting targets under visual interference. The image-level score p mmlr and the instance-level score are combined using the following weighted geometric mean:

[0102]

[0103] where denotes the context-aware detection score of the j-th instance, ⊙ denotes element-wise multiplication, and γ is a hyperparameter that balances these two scores.

[0104] S5. Evaluate and validate the model using the test set;

[0105] The above embodiments are only for illustrating the technical concept and features of the present invention, and the purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. It should not be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for detecting human interaction based on open vocabulary in the scenario of unmanned aerial vehicles, characterized in that, It includes the following steps: S1. Use a pre-trained CLIP visual encoder to extract the global features of the image. The visual encoder divides the image into multiple image patches and encodes each image patch to generate a multi-level feature map; S2. After feature extraction, introduce an MLR module to extract global context information. At the same time, decode human interactions from the multi-level feature map in parallel. When performing bipartite graph matching between the prediction result and the ground truth result, design a loss function to guide the low-level feature map to correspond to human pairs with a smaller distance, and the high-level feature map to correspond to human pairs with a larger distance; S3. Use a large language model to generate a description of the state of human body parts related to the human interaction in the image. Encode the human interaction category name and the state description of the human body parts into text embeddings, and combine the embeddings with the output of the human interaction decoder and the global context information extracted by the MLR module to obtain the instance-level detection score.

2. The method for detecting human interaction based on open vocabulary in a drone scenario according to claim 1, wherein The specific method of S1 is as follows: Construct an end-to-end open-vocabulary human interaction detector using the generalization ability of CLIP for classification tasks, and convert human interaction detection into an end-to-end set matching problem similar to DETR. Given an image I, first, the pre-trained CLIP visual encoder E V extracts the global context representation F img : F img = E V (I) (1) Among them, represents a series of feature embeddings of I, and then uses a Transformer-based decoder D to decode the human interactions in I, using the projected context features and the human interaction query Q = (q1, q2,..., q M ) as the input, and the output of the last layer of D is used as the representation of the interaction: H = D(Q,F img ) (2) Among them, in the cross-attention mechanism of the human interaction decoder D, Q is used as the query, and the projected context representation is used as the key-value, H = (h1, h2,..., h M ), where M represents the number of HOI queries.

3. The method for detecting human interaction based on open vocabulary in an unmanned aerial vehicle scenario according to claim 2, wherein Embed the context features, the person interaction query Q, and the features of the image F img Provide to two different head networks: 1) A bounding box regressor P bbox , which predicts a confidence score c and a bounding box b of the interacting person and object h , b o ; 2) The linear projection P that maps features to the visual and text joint space cls , calculates its similarity with the text encoder features for interactive recognition, and finally calculates the box regression loss L b , the intersection over union loss L iou and the interaction classification loss L cls , and the matching cost formula is: Visual encoder E v Divides the fixed-resolution image I into small patches and projects these patches as a sequence, assuming where is the i-th patch of E V Let the input be X0, and E V Encodes the image through the self-attention mechanism of each layer: where X i is E V the encoded feature map after the i-th block, and then use multi-level feature maps to parallelly decode human interactions: Among them represents a set of meaningful levels, and then the decoded hidden state of the human interaction is sent to a bounding box regressor and an interaction classifier to predict the bounding boxes of human pairs and interaction categories respectively.

4. The method for detecting human interaction based on open vocabulary in a drone scenario according to claim 1, characterized in that Specifically in S2: Among the N prediction sets of each image and the ground-truth human interaction set in the bipartite matching, a design loss is used to guide H decoded from the low-level feature map i to simulate human interactions with relatively small center distances and vice versa, in order to reduce the performance gap between human interactions at different distances. A new constraint loss is designed between the normalized horizontal index of j and the human distance of y where is 's normalized horizontal index, g(y j ) is the L2 distance between the center of the person and the object labeled in y j . The guiding model classifies person-object interactions with different distances into separate groups and uses specific feature maps customized for the features of each group to process the person-object interactions in each group. After the model encodes the feature maps, an MLR module is introduced. The MLR module contains two branches, text and vision. By extracting global context information as an external cue, it combines with the human body part descriptions related to the person-object interactions generated by the large language model and the output of the person-object interaction decoder Text MLR aims to align the image embedding with the corresponding category text embedding generated by the CLIP text encoder for classifying each image according to the CLIP text embedding. Introduce a visual MLR branch. The visual MLR branch extracts knowledge from the CLIP image encoder. Since the image encoder of CLIP has been trained on a large number of images using the corresponding text supervision, it can extract more generalized image embeddings, including features of basic categories and new categories, as well as important background environments.

5. The method for detecting human interaction based on open vocabulary in a drone scenario according to claim 1, wherein The method of S3 is as follows: Given a human-object interaction category defined as an action-object pair, denoted as (act i ,obj i ), a sentence is formed using the globally learnable context token Ctx, with several prefix tokens Ctx pre and several learnable connection tokens Ctx con at the beginning of the sentence to automatically determine how to connect the category names of actions and objects, and then a text embedding is generated for each HOI label through the pre-trained CLIP text encoder E T : T hoi = E T (Ctx pre , act i , Ctx con , obj i ) (11) The embedding results of all character interaction names are denoted as where N represents the number of character interaction triple categories, and C t represents the text embedding dimension of the CLIP text encoder.

6. A method for detecting human interaction based on open vocabulary in a drone scenario according to claim 5, characterized in that To encode the human body part state description Si for each HOI, the CLIP text encoder E is utilized T : T b = E T (S i ) (12) Among them, represents the embedding of the body part descriptions of all human interactions, combined with T hoi and the output logarithm of T b to obtain the final interaction prediction: where h j represents the hidden state of human interaction, p j represents the corresponding interaction prediction, α hoi and α b are learnable scalars used to balance the weights of human interaction embeddings and body part embeddings respectively; Text MLR and the visual MLR branch complement each other in identifying basic categories and new categories. Combine the classification scores of the text MLR and the visual MLR branch to form a multi-modal MLR to identify new objects and basic objects from a global perspective.

Images