Lightweight scene graph generation method for unmanned aerial vehicle platform

Abstract

The invention discloses a lightweight scene graph generation method for an unmanned aerial vehicle platform, and the method comprises the steps: directly decoding a subject-object pair which may have a predicate relation from an image through designing a lightweight model based on a Transform encoder-decoder architecture; and adopting an integrated learning technology, re-sampling from an original training set to generate a plurality of sub-training sets, training a plurality of predicate relationship classifiers, and integrating and constructing a predicate relationship classifier with higher robustness. And integrating the obtained multiple groups of subject-predicate-object triples to generate a scene graph. The problem of real-time and efficient scene understanding in a resource limited environment is solved for the scene graph generation requirement of an unmanned aerial vehicle platform, the generated scene graph is rich in semantics and clear in structure, the intelligent sensing and decision-making capability of the unmanned aerial vehicle in a complex dynamic environment is remarkably enhanced, and the method has high practical value and popularization prospects.

Description

Technical Field

[0001] The present invention relates to the field of scene understanding, and in particular to a lightweight scene graph generation method for unmanned aerial vehicle platforms. Background Art

[0002] With the rapid development of drone technology, drones are increasingly being used in a wide range of fields, including agricultural monitoring, disaster assessment, urban planning, and security patrols. In these scenarios, drones often need to process complex visual information, such as detecting and identifying objects, analyzing scene content, and inferring relationships between objects. However, traditional image analysis methods, which primarily focus on object detection and classification, lack a deep understanding of the semantic information and relational structures within a scene, making them incapable of meeting the demands of these complex tasks.

[0003] Scene graph generation, an emerging computer vision task, aims to generate a semantically structured graph, or scene graph, from an input image. A scene graph provides a structured representation of image content by modeling the subject and object targets in the image as graph nodes and representing the relationships between them as edges. It not only helps terminal computing devices identify objects in an image but also infers the relationships between these objects, thereby supporting higher-level scene understanding and reasoning capabilities. This technology is of great value in various drone applications, such as target detection and tracking, mission planning, environmental perception, and disaster assessment, significantly enhancing the intelligent capabilities of drones in complex scenarios.

[0004] Existing scene graph generation methods typically rely on complex deep neural network structures, which require high computational resources, are difficult to deploy, and have low operational efficiency, making them unsuitable for UAV platforms with limited computing power. Because UAVs must operate in highly dynamic and resource-constrained environments, they place higher demands on lightweight and real-time algorithms. Furthermore, UAVs often need to process large-scale, multi-scale scenes during flight, and the scene graph generation process must balance the expression of spatial scope and local details. Therefore, integrating scene graph generation tasks with UAV vision applications requires not only optimizing the lightweight design of the algorithm, but also maintaining the ability to accurately model semantic relationships in the scene within limited computing resources to meet the diverse application needs of UAVs. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this paper proposes an efficient scene graph generation method for drone applications. Combining lightweight entity pair detection technology with an ensemble learning-based predicate relationship classifier optimization method, this method can rapidly decode subject-object pairs that may have predicate relationships from drone images. It also improves the accuracy of predicate relationship prediction by optimizing long-tail distribution issues. This method has strong environmental adaptability and operates stably in diverse scenarios, supporting drones in autonomous flight, target recognition, and intelligent decision-making for complex tasks.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A lightweight scene graph generation method for unmanned aerial vehicle (UAV) platforms is proposed. By designing a lightweight model based on the Transformer encoder-decoder architecture, subject-object pairs that may have predicate relationships are directly decoded from images. Ensemble learning technology is used to resample the original training set to generate multiple sub-training sets, train multiple predicate relationship classifiers, and integrate them to construct a more robust predicate relationship classifier. The obtained multiple sets of subject-predicate-object triples are integrated to generate a scene graph.

[0008] Furthermore, the method comprises the following steps:

[0009] Step 1: A 3-channel image I∈R taken by a drone 3×H×W , where H and W are the height and width of the image respectively, and an initial feature map of the image is obtained using the ResNet50 backbone network Where C represents the number of feature channels, H0 and W0 are the height and width of the feature map; the projection layer is used to reduce the dimension of the feature map channel to obtain d represents the number of feature channels after dimensionality reduction;

[0010] Step 2: Expand f0 into a one-dimensional sequence Z0 of shape d×H0×W0, add a position code pos to Z0, and encode it using an encoder consisting of a multi-layer multi-head attention and feedforward neural network;

[0011] Z=MultiAtt(Q=Z0+pos, K=Z0+pos, V=Z0);

[0012] Among them, the single-head attention module is:

[0013]

[0014] Among them, d k The dimensions are Q, K, and V. MultiAtt(Q,K,V) multi-head attention is composed of multiple single-head attentions added together.

[0015] Step 3: Set the subject query embedding Q s ∈R N×d , object query is embedded as Q o ∈R N×d , set an embedding for all target queries and a subject-object relationship embedded in E so ∈R N×d ; where N is the upper bound of the number of subject and object pairs, d is the dimension, and is calculated using self-attention;

[0016] [Q s ,Q o ]=MultiAtt(Q,k,[Q s ,Q o ]);

[0017] Where, Q=K=[Q s +E so ,Q o +E so ], [] means concatenating two embeddings;

[0018] Step 4: Q s ,Q o Perform cross attention calculation with the feature Z obtained by the encoder and retain the attention weights (Ms, Mo);

[0019] Q s ,M s =MultiAtt(Q=Q s +E so ,K=Z+pos,V=Z);

[0020] Q o ,M o =MultiAtt(Q=Q o +E so ,K=Z+pos,V=Z);

[0021] Step 5: Query the subject and object Q s ,Q o and Q e , do cross attention calculation:

[0022] Q s =MultiAtt(Q=Q s +E so ,K=Q e ,V=Q e );

[0023] Q o =MultiAtt(Q=Q o +E so ,K=Qe ,V=Q e );

[0024] Step 6: Repeat steps 3, 4, and 5 J times, where J is a positive integer, to obtain the attention weights M of the subject and object. s ,M o and feature Q s ,Q o ;

[0025] Step 7: Feature Q s Through the Linear layer s Predict the subject category, MLP s The layer predicts the target frame of the subject; feature Q o Through the Linear layer o Predict the category of the object, MLP o The layer predicts the target box of the object;

[0026] Step 8: Set the weight M s and M o Splice and resize, input into the convolutional network to form an auxiliary feature M A , M s and Q s ,Q o After splicing, a predicate relation classifier MLP is formed by a multi-layer perceptron r Get the predicate relationship classification results;

[0027] Step 9: Output a fixed number N of subject-object pair prediction results and where c sub and c obj denote the subject and object target categories, respectively, b sub and b obj Represent the target frame positions of the subject and object respectively, and Represent the prediction results of the subject and object respectively, and the true label y is filled with the background label φ <background-background>Make its quantity consistent with the predicted result;

[0028] Step 10: Calculate the total matching loss C of the subject and object so :

[0029]

[0030] in, are the matching losses for subject and object respectively;

[0031] The matching loss is

[0032]

[0033] in, is the category matching loss, is the target box matching loss;

[0034] Class matching loss Including positive and negative category loss terms:

[0035]

[0036] in

[0037]

[0038] Among them, α, γ, and ε are hyperparameters. Represents the probability of the category;

[0039] The target box matching loss is

[0040]

[0041] in, is the L1 loss, L GIoU is the generalized IoU loss;

[0042] Step 11: Use the Hungarian algorithm to minimize the matching loss C so Match the label with the most recent prediction and use the matching results of the label and prediction to calculate the loss:

[0043]

[0044] Among them, L cls is the cross entropy loss between the subject and object true labels and the closest predicted categories after matching, L box It is the target box position loss, including L1 loss and generalized IoU loss;

[0045] Step 12: Output the prediction results of a fixed number N of subject-verb-object triples and Calculate the triplet matching loss matrix C tri :

[0046]

[0047] in, and The matching losses for subject, object, and relation classification are respectively, and the Hungarian algorithm is used to minimize the matching loss c tri Match the label and the closest predicted category; calculate the cross entropy loss between the label predicate relationship category and the closest predicted category

[0048]

[0049] Step 13: Multiple predicate relation classifiers with different weights are trained by resampling the original dataset to generate multiple training sets. For each input subject and predicate feature, the multiple classifiers respectively generate a predicted predicate relation category and its confidence level. Next, the predicate relations with the most occurrences are screened based on a voting mechanism. These candidate predicate relations are then screened again, and the predicate relation with the highest confidence level is selected as the final predicate relation classification result.

[0050] Step 14: Repeat steps 1 to 13. and When the loss is less than the specified minimum value, the calculation ends;

[0051] Step 15: For the input drone-photographed image, select the subject-verb-object triples whose confidence level in the output method is greater than the specified value. and Integrate multiple triplets to generate a scene graph.

[0052] The technical concept of this invention is as follows: Aiming at the real-time scene understanding requirements in UAV applications, a lightweight and efficient scene graph generation method is implemented through a Transformer-based subject-object pair prediction model and an ensemble learning-optimized predicate relation classifier, which can provide intelligent support for autonomous flight, target recognition, and complex mission decision-making of UAVs in dynamic environments.

[0053] The beneficial effects of the present invention are: through a lightweight scene graph generation model and an integrated learning optimization method, the ability of the UAV to generate scene graphs in real time in complex environments is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of a lightweight scene graph generation method for UAV platforms; DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to the accompanying drawings.

[0056] Reference Figure 1 ,A lightweight scene graph generation method for UAV platforms, including the following steps:

[0057] Step 1: A 3-channel image I∈R taken by a drone 3×H×W , where H and W are the height and width of the image respectively, and an initial feature map of the image is obtained using the ResNet50 backbone network Where C represents the number of feature channels, H0 and W0 are the height and width of the feature map; the projection layer is used to reduce the dimension of the feature map channel to obtain d represents the number of feature channels after dimensionality reduction;

[0058] Step 2: Expand f0 into a one-dimensional sequence Z0 of shape d×H0×W0, add a position code pos to Z0, and encode it using an encoder consisting of a multi-layer multi-head attention and feedforward neural network;

[0059] Z=MultiAtt(Q=Z0+pos, K=Z0+pos, V=Z0);

[0060] Among them, the single-head attention module is:

[0061]

[0062] Among them, d k The dimensions are Q, K, and V. MultiAtt(Q,K,V) multi-head attention is composed of multiple single-head attentions added together.

[0063] Step 3: Set the subject query embedding Q s ∈R N×d , object query is embedded as Q o ∈R N×d , set an embedding for all target queries and a subject-object relationship embedded in E so ∈R N×d ; where N is the upper bound of the number of subject and object pairs, d is the dimension, and is calculated using self-attention;

[0064] [Q s ,Q o ]=MultiAtt(Q,K,[Q s ,Q o ]);

[0065] Where, Q=K=[Q s +E so ,Q o +E so ], [] means concatenating two embeddings;

[0066] Step 4: Q s ,Q o Perform cross attention calculation with the feature Z obtained by the encoder and retain the attention weights (Ms, Mo);

[0067] Q s ,M s =MultiAtt(Q=Q s +E so ,K=Z+pos,V=Z);

[0068] Q o ,M o =MultiAtt(Q=Q o +E so ,K=Z+pos,V=Z);

[0069] Step 5: Query the subject and object Q s ,Q o and Q e , do cross attention calculation:

[0070] Q s =MultiAtt(Q=Q s +E so ,K=Q e ,V=Q e );

[0071] Q o =MultiAtt(Q=Q o +E so ,K=Q e ,V=Q e );

[0072] Step 6: Repeat steps 3, 4, and 5 8 times to obtain the attention weights M of the subject and object. s ,M o and feature Q s ,Q o ;

[0073] Step 7: Feature Q s Through the Linear layer s Predict the subject category, MLP s The layer predicts the target frame of the subject; feature Q o Through the Linear layer o Predict the category of the object, MLP o The layer predicts the target box of the object;

[0074] Step 8: Set the weight M s and M o Splice and resize, input into the convolutional network to form an auxiliary feature M A , M s and Q s ,Q o After splicing, a predicate relation classifier MLP is formed by a multi-layer perceptron r Get the predicate relationship classification results;

[0075] Step 9: Output a fixed number N of subject-object pair prediction results and where c sub and c obj denote the subject and object target categories, respectively, b sub and b obj Represent the target frame positions of the subject and object respectively, and Represent the prediction results of the subject and object respectively, and the true label y is filled with the background label φ <background-background>Make its quantity consistent with the predicted result;

[0076] Step 10: Calculate the total matching loss C of the subject and object so :

[0077]

[0078] in, are the matching losses for subject and object respectively;

[0079] The matching loss is

[0080]

[0081] in, is the category matching loss, is the target box matching loss;

[0082] Class matching loss Including positive and negative category loss terms:

[0083]

[0084] in

[0085]

[0086] Among them, α, γ, and ε are hyperparameters. Represents the probability of the category;

[0087] The target box matching loss is

[0088]

[0089] in, is the L1 loss, L GIoU is the generalized IoU loss;

[0090] Step 11: Use the Hungarian algorithm to minimize the matching loss C so Match the label with the most recent prediction and use the matching results of the label and prediction to calculate the loss:

[0091]

[0092] Among them, L cls is the cross entropy loss between the subject and object true labels and the closest predicted categories after matching, L box It is the target box position loss, including L1 loss and generalized IoU loss;

[0093] Step 12: Output the prediction results of a fixed number N of subject-verb-object triples and Calculate the triplet matching loss matrix C tri :

[0094]

[0095] in, and The matching losses for subject, object, and relation classification are respectively, and the Hungarian algorithm is used to minimize the matching loss c tri Match the label and the closest predicted category; calculate the cross entropy loss between the label predicate relationship category and the closest predicted category

[0096] Step 13: Multiple predicate relation classifiers with different weights are trained by resampling the original dataset to generate multiple training sets. For each input subject and predicate feature, the multiple classifiers respectively generate a predicted predicate relation category and its confidence level. Next, the predicate relations with the most occurrences are screened based on a voting mechanism. These candidate predicate relations are then screened again, and the predicate relation with the highest confidence level is selected as the final predicate relation classification result.

[0097] Step 14: Repeat steps 1 to 13. and When the value is less than 12.0, the calculation ends;

[0098] Step 15: For the input drone-photographed image, select the subject-verb-object triples with a confidence score greater than 0.7 in the method output. and Integrate multiple triplets to generate a scene graph.

[0099] In this example, a drone-captured image of a city street is input. The system automatically extracts image features, detects the subject "pedestrian" and the object "bicycle," and extracts their bounding boxes. Using a multi-classifier voting mechanism, it identifies the relationship between them as "riding." Finally, it outputs multiple sets of subject-verb-object triples similar to (pedestrian-rider-bicycle), which are then integrated to generate the corresponding scene graph.

[0100] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A lightweight scene graph generation method for UAV platforms, characterized by: By designing a lightweight model based on the Transformer encoder-decoder architecture, we can directly decode subject-object pairs that may have predicate relationships from images. We also use ensemble learning techniques to resample the original training set to generate multiple sub-training sets, train multiple predicate relationship classifiers, and integrate them to construct a more robust predicate relationship classifier. The obtained multiple groups of subject-verb-object triples are integrated to generate a scene graph.

2. A lightweight scene graph generation method for an unmanned aerial vehicle platform according to claim 1, characterized in that: The method comprises the following steps: Step 1: A 3-channel image I∈R taken by a drone 3×H×W , where H and W are the height and width of the image respectively, and an initial feature map of the image is obtained using the ResNet50 backbone network Where C represents the number of feature channels, H0 and W0 are the height and width of the feature map; the projection layer is used to reduce the dimension of the feature map channel to obtain d represents the number of feature channels after dimensionality reduction; Step 2: Expand f0 into a one-dimensional sequence Z0 of shape d×H0×W0, add a position code pos to Z0, and encode it using an encoder consisting of a multi-layer multi-head attention and feedforward neural network; Z=MultiAtt(Q=Z0+pos, K=Z0+pos, V=Z0); Among them, the single-head attention module is: Among them, d k The dimensions are Q, K, and V. MultiAtt(Q,K,V) multi-head attention is composed of multiple single-head attentions added together. Step 3: Set the subject query embedding Q s ∈R N×d , object query is embedded as Q o ∈R N×d , set an embedding for all target queries and a subject-object relationship embedded in E so ∈R N×d ; where N is the upper bound of the number of subject and object pairs, d is the dimension, and self-attention is used for calculation: [Q s ,Q o ]=MultiAtt(Q,K,[Q s ,Q o ]); Where, Q=K=[Q s +E so ,Q o +E so ], [] means concatenating two embeddings; Step 4: Q s ,Q o Perform cross attention calculation with the feature Z obtained by the encoder and retain the attention weights (Ms, Mo); Q s ,M s =MultiAtt(Q=Q s +E so ,K=Z+pos,V=Z); Q o ,M o =MultiAtt(Q=Q o +E so ,K=Z+pos,V=Z); Step 5: Query the subject and object Q s ,Q o and Q e , do cross attention calculation; Q s =MultiAtt(Q=Q s +E so ,K=Q e ,V=Q e ); Q o =MultiAtt(Q=Q o +E so ,K=Q e ,V=Q e ); Step 6: Repeat steps 3, 4, and 5 J times, where J is a positive integer, to obtain the attention weights M of the subject and object. s ,M o and feature Q s ,Q o ; Step 7: Feature Q s Through the Linear layer s Predict the subject category, MLP s The layer predicts the target frame of the subject; feature Q o Through the Linear layer o Predict the category of the object, MLP o The layer predicts the target box of the object; Step 8: Set the weight M s and M o Splice and resize, input into the convolutional network to form an auxiliary feature M A , M s and Q s ,Q o After splicing, a predicate relation classifier MLP is formed by a multi-layer perceptron r Get the predicate relationship classification results; Step 9: Output a fixed number N of subject-object pair prediction results and where c sub and c obj denote the subject and object target categories, respectively, b sub and b obj Represent the target frame positions of the subject and object respectively, and Represent the prediction results of the subject and object respectively, and the true label y is filled with the background label φ <background-background> Make its quantity consistent with the predicted result;< / background-background> Step 10: Calculate the total matching loss C of the subject and object so : in, are the matching losses for subject and object respectively; The matching loss is in, is the category matching loss, is the target box matching loss; Class matching loss Including positive and negative category loss terms: in Among them, α, γ, and ε are hyperparameters. Represents the probability of the category; The target box matching loss is in, is the L1 loss, L GIoU is the generalized IoU loss; Step 11: Use the Hungarian algorithm to minimize the matching loss C so Match the label with the most recent prediction and use the matching results of the label and prediction to calculate the loss: Among them, L cls is the cross entropy loss between the subject and object true labels and the closest predicted categories after matching, L box It is the target box position loss, including L1 loss and generalized IoU loss; Step 12: Output the prediction results of a fixed number N of subject-verb-object triples and Calculate the triplet matching loss matrix C tri : in, and The matching losses for subject, object, and relation classification are respectively, and the Hungarian algorithm is used to minimize the matching loss c tri Match the label and the closest predicted category; calculate the cross entropy loss between the label predicate relationship category and the closest predicted category Step 13: Multiple predicate relation classifiers with different weights are trained by resampling the original dataset to generate multiple training sets. For each input subject and predicate feature, the multiple classifiers respectively generate a predicted predicate relation category and its confidence level. Next, the predicate relations with the most occurrences are screened based on a voting mechanism. These candidate predicate relations are then screened again, and the predicate relation with the highest confidence level is selected as the final predicate relation classification result. Step 14: Repeat steps 1 to 13. and When the loss is less than the specified minimum value, the calculation ends; Step 15: For the input drone-photographed image, select the subject-verb-object triples whose confidence level in the output method is greater than the specified value. and Integrate multiple triplets to generate a scene graph.

Images