Target detection method, device and equipment for aerial image of unmanned aerial vehicle, and medium

Abstract

The invention discloses a target detection method, device and equipment for an aerial image of an unmanned aerial vehicle and a medium, and relates to the field of target detection.The target detection method for the aerial image of the unmanned aerial vehicle comprises the steps that feature extraction processing is conducted on the target aerial image, and visual features of the target aerial image are obtained; determining a target node matched with the visual feature according to the visual feature of the target aerial image and the entity object feature of each node in a knowledge graph; performing feature fusion on the visual feature of the target aerial image and the node feature of the target node to obtain a fusion feature of the target aerial image; wherein the node features of each node in the knowledge graph comprise association relationship features between entity objects corresponding to each node and entity object features of the entity objects; and target detection is carried out on the target aerial image based on the fusion features of the target aerial image and the knowledge graph, and the method is beneficial for improving the generalization ability and accuracy of unmanned aerial vehicle aerial target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection, and in particular to a method, apparatus, equipment and medium for target detection in aerial images taken by unmanned aerial vehicles. Background Technology

[0002] Currently, traditional drone aerial target detection methods mainly include data-driven deep learning and image processing methods. Among them, data-driven deep learning only focuses on the internal features of the image and ignores prior knowledge such as the drone's aerial perspective and target scale differences. It relies on a large amount of labeled drone aerial data, and its generalization ability is insufficient when encountering new scenes or target types not included in the training data. Image processing methods have a high false detection rate in scenes with dense targets and severe occlusion, and a low recall rate for small targets.

[0003] Therefore, how to detect targets in UAV aerial photography and improve the generalization ability and accuracy of UAV aerial target detection has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a target detection method, apparatus, device, and medium for drone aerial images, so as to improve the generalization ability and accuracy of drone aerial target detection.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A target detection method for drone aerial images includes: performing feature extraction processing on the target aerial image to obtain visual features of the target aerial image; determining target nodes matching the visual features based on the visual features of the target aerial image and entity object features of each node in a knowledge graph; fusing the visual features of the target aerial image with the node features of the target node to obtain fused features of the target aerial image; wherein, the node features of each node in the knowledge graph include: the association relationship features between entity objects corresponding to each node and the entity object features of the entity objects; and performing target detection on the target aerial image based on the fused features of the target aerial image and the knowledge graph.

[0006] In one optional embodiment of this application, determining the target node matching the visual features based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph includes: calculating the correlation between the visual features of each spatial region in the target aerial image and the entity object features of each node in the knowledge graph to determine the correlation between the visual features of each spatial region and each node in the knowledge graph; and determining the target node matching the visual features of the spatial region based on the correlation.

[0007] In one optional embodiment of this application, the relevance is determined by the following formula: ; in, Indicates the relevance; The relevance is calculated using weights; This represents the transformation matrix that maps the visual features to the latent space; The transformation matrix represents the mapping of the features of the entity object to the causal space; Represents the visual features of the i-th spatial region; The entity object features representing the k-th node of the spatial map; This indicates the bias in the correlation calculation.

[0008] In one optional embodiment of this application, the step of fusing the visual features of the target aerial image with the node features of the target node to obtain the fused features of the target aerial image includes: determining at least one target node that matches the visual features of the target aerial image; and fusing the visual features of the target aerial image with the node features of the target node according to the correlation between each target node and the visual features to obtain the fused features of the target aerial image.

[0009] In one optional embodiment of this application, the fusion features of the target aerial image are obtained by the following formula: ; in, This represents the fusion feature corresponding to the i-th image region of the target aerial image; The visual features corresponding to the i-th image region of the target aerial image; The similarity between the fusion feature corresponding to the i-th image region and the entity object feature of the j-th target node in the knowledge graph is represented; N represents the number of target nodes. This represents the node characteristics of the j-th target node; It represents the linear transformation matrix that transforms the dimension of the target node's node feature space to the visual feature space.

[0010] In one optional embodiment of this application, the step of performing target detection on the target aerial image based on the fusion features of the target aerial image and the knowledge graph includes: inputting the fusion features of the target aerial image as a new node of the knowledge graph into a pre-trained target detection model, so as to perform target detection on the target aerial image by combining the fusion features and the association relationship features between the entity objects corresponding to each node in the knowledge graph through the target detection model, and obtaining the target detection result of the target aerial image.

[0011] In one optional embodiment of this application, the method further includes: inputting the target detection result of the target aerial image into a pre-constructed Bayesian network, so as to optimize the target detection result by combining the conditional probability of the target detection results of different entity objects through the Bayesian network, and obtaining an optimized target detection result.

[0012] Compared with existing technologies, the target detection method for UAV aerial images provided by this invention fuses the visual features of the target aerial image with the node features of the knowledge graph to obtain fused features. The visual features of the target aerial image are enhanced by the node features of the knowledge graph. Combined with the fused image of the target aerial image and the knowledge graph, the target aerial image is detected. This method combines target detection of UAV aerial images with the knowledge graph through feature fusion. In the target detection process, the association relationship between different entity objects in the knowledge graph is used as a constraint, which helps to improve the accuracy and generalization ability of target detection.

[0013] The present invention also provides a target detection device for drone aerial images, comprising: The feature extraction unit is used to perform feature extraction processing on the target aerial image to obtain the visual features of the target aerial image.

[0014] The feature matching unit is used to determine the target node that matches the visual features based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph.

[0015] The feature fusion unit is used to fuse the visual features of the target aerial image with the node features of the target node to obtain the fused features of the target aerial image; wherein, the node features of each node in the knowledge graph include: the association relationship features between the entity objects corresponding to each node and the entity object features of the entity objects.

[0016] The target detection unit is used to perform target detection on the target aerial image based on the fusion features of the target aerial image and the knowledge graph.

[0017] Compared with the prior art, the beneficial effects of the target detection device for UAV aerial images provided by the present invention are the same as those described in the technical solution of the target detection method for UAV aerial images, and will not be repeated here.

[0018] The present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to execute the target detection method for UAV aerial images by running the instructions in the memory.

[0019] Compared with the prior art, the beneficial effects of the electronic device provided by the present invention are the same as those of the target detection method for UAV aerial images described in the above technical solution, and will not be repeated here.

[0020] The present invention also provides a computer storage medium storing instructions, which, when executed, implement the above-described target detection method for UAV aerial images.

[0021] Compared with the prior art, the beneficial effects of the computer storage medium provided by the present invention are the same as those of the target detection method for UAV aerial images described in the above technical solution, and will not be repeated here. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 A flowchart illustrating the target detection method for drone aerial images provided in this application embodiment.

[0023] Figure 2 This is a structural diagram of the target detection device for drone aerial images provided in an embodiment of this application.

[0024] Figure 3 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0025] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0026] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0027] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0028] This application provides a method, apparatus, device, and medium for detecting targets in drone aerial photography, which will be described in detail in the following embodiments.

[0029] This application first provides a target detection method for drone aerial images. Please refer to... Figure 1 , Figure 1 A flowchart illustrating the target detection method for drone aerial images provided in this application embodiment.

[0030] like Figure 1 As shown, the target detection method for drone aerial images includes the following steps S101 to S104.

[0031] S101, Perform feature extraction processing on the target aerial image to obtain the visual features of the target aerial image.

[0032] Target aerial images refer to aerial views of objects such as buildings, farmland, and vehicles taken by drones. In this embodiment, feature extraction is performed on the target aerial images to detect the visual features of multiple target objects contained in the images, so as to facilitate subsequent target detection for these objects.

[0033] Furthermore, the visual features refer to information ranging from low-level texture to high-level language information, including the edge and texture features of the target object in the target aerial image, as well as the category information of the target object in the target aerial image.

[0034] Specifically, feature extraction of the target aerial image can be achieved using a backbone network composed of ResNet-50+FPN, so as to obtain the visual features by extracting features from the target aerial image through this network.

[0035] In one optional embodiment of this application, in order to further improve the feature performance of the extracted visual features, the visual features can be input into the SE (Squeeze-and-Excitation) attention enhancement module to obtain the enhanced visual features. That is, the SE attention enhancement module dynamically adjusts the weights of the feature channels to enhance important features and suppress redundant information.

[0036] S102, based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph, determine the target node that matches the visual features.

[0037] In this embodiment, a knowledge graph can be understood as a structured prior knowledge base. The knowledge graph employs a schema layer construction method to define entity objects, attributes, and the relationships between entity objects. Entity objects represent various types of objects and scene elements, such as drones, vehicles, buildings, people, roads, vegetation, water bodies, farmland, photovoltaic panels, power transmission lines, and tower cranes. Attributes define visual attributes including size, color, and shape. Relationships define spatial relationships (e.g., entity object A is east of entity object B), distances (e.g., entity object A is close to entity object B), heights (e.g., entity object A is at low altitude), semantic relationships (e.g., a vehicle is parked on a road), and relationships between entity object types (e.g., cars and trucks are similar types).

[0038] In this embodiment of the application, the entity object features refer to the features of the content corresponding to the objects and attributes defined in the knowledge graph above.

[0039] Furthermore, in practical applications, the spatial relationships between the entities can be calculated using the UAV's GPS, IMU, and image geometric information. After the knowledge graph is constructed, it can be stored using the Neo4j graph database, and the relevant features of the graph convolutional network can be obtained through the graph convolutional network (GCN).

[0040] Furthermore, determining the target node matching the visual features based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph includes: calculating the correlation between the visual features of each spatial region in the aerial image and the entity object features of each node in the knowledge graph to determine the correlation between the visual features of each spatial region and each node in the knowledge graph; and determining the target node matching each spatial region based on the correlation.

[0041] That is, the correlation between the visual features of the target aerial image and the entity objects of each node in the knowledge graph is calculated. The correlation is used to establish the connection between each spatial region in the target aerial image and the entity objects of each node in the knowledge graph, so as to combine the knowledge graph as a prior knowledge base to perform target detection on the target aerial image.

[0042] In this embodiment of the application, the relevance is calculated using the following formula (1): (1); in, Indicates the relevance; The relevance is calculated using weights; This represents the transformation matrix that maps the visual features to the latent space; The transformation matrix represents the mapping of the features of the entity object to the causal space; Represents the visual features of the i-th spatial region; The entity object feature representing the j-th node of the spatial graph; This indicates the bias in the correlation calculation.

[0043] In another optional embodiment of this application, for ease of calculation, the correlation obtained above needs to be normalized by softmax. Specifically, the normalization of the correlation can be achieved by the following formula (2): (2); in, This indicates the correlation after normalization; The correlation between the visual features of the i-th spatial region and the entity object features of the n-th node is represented; N represents the number of nodes in the spatial image.

[0044] S103, the visual features of the target aerial image and the node features of the target node are fused to obtain the fused features of the target aerial image; wherein, the node features of each node in the knowledge graph include: the association relationship features between the entity objects corresponding to each node and the entity object features of the entity objects.

[0045] The purpose of S103 is to combine the visual features of the target aerial image with the features of the knowledge graph through the fusion of multimodal features, thereby enhancing the visual features of the target aerial image. At the same time, through feature fusion, the correlation between the visual features of the target aerial image and the knowledge graph is enhanced, so as to facilitate subsequent target detection of the target aerial image by combining the knowledge graph and the fused features.

[0046] In this embodiment, the fusion of the target node features and the visual features is achieved based on a mapping function. The mapping function can be based on a single-layer neural network or a trained feature encoder. The purpose of the mapping function is to map the visual features and the node features of the knowledge graph to the same latent space.

[0047] In practical applications, the matching of the visual features of the target aerial image and the node features is also based on the mapping function. That is, after mapping the visual features and the node features, the correlation between the visual features and the node features is calculated by the above formulas (1) and (2) to complete the matching of visual features with entity nodes of the knowledge graph.

[0048] Specifically, feature fusion of the visual features of the target aerial image and the node features of the target node can be achieved through the following formula (3): (3); in, This represents the fusion feature corresponding to the i-th image region of the target aerial image; This represents the visual features corresponding to the i-th image region of the target aerial image; The similarity between the fusion feature corresponding to the i-th image region and the entity object feature of the j-th target node in the knowledge graph is represented; N represents the number of target nodes. This represents the node characteristics of the j-th target node; It represents the linear transformation matrix that transforms the dimension of the target node's node feature space to the visual feature space.

[0049] S104, Target detection is performed on the target aerial image based on the fusion features of the target aerial image and the knowledge graph.

[0050] The purpose of S104 is to utilize the relationships between entity objects in the knowledge graph to constrain the detection of visual features of each image region in the target aerial image, thereby improving the accuracy and robustness of target detection in the target aerial image.

[0051] Specifically, S104 includes: inputting the fusion features of the target aerial image as a new node of the knowledge graph into a pre-trained target detection model, so as to perform target detection on the target aerial image by combining the fusion features and the association relationship features between the entity objects corresponding to each node in the knowledge graph, and obtaining the target detection result of the target aerial image.

[0052] Specifically, the target detection model is a Relational Graph Convolutional Network (R-GCN), which performs target detection on target images based on a knowledge graph by aggregating neighborhood information of nodes with different relation types.

[0053] In practical applications, by combining fusion features and knowledge graphs to perform target detection on aerial images, the target detection process goes beyond simply recognizing the content in the image. It also incorporates prior knowledge from the knowledge graph to determine the relationships between the identified objects, thereby improving the accuracy of target detection.

[0054] In practical applications, the output of target detection for the target aerial image is usually the probability of detection results for different image regions in the target aerial image.

[0055] In practical applications, in order to further improve the accuracy of target detection through the model and provide training samples for model training, the recognition results of the model can also be verified through a Bayesian network.

[0056] Specifically, in this embodiment of the application, a Bayesian network is used to define the conditional probability of target detection results for different entity objects, and the conditional probability is constructed based on the entity objects in the knowledge graph.

[0057] For example, regarding vehicles and pedestrians, based on the relationships constructed in the knowledge graph, we can define P(pedestrian | zebra crossing, intersection) = 0.9, meaning that if a zebra crossing and an intersection are detected in the target aerial image, the probability that a pedestrian exists in the target aerial image is 0.9. Similarly, P(vehicle | parking lot) = 0.8, meaning that if a parking lot is detected in the target aerial image, the probability that a vehicle exists in the target aerial image is 0.8.

[0058] Furthermore, after obtaining the target detection result of the aerial image of the target through the above-described S104, the method further includes: The target detection results of the aerial image are input into a pre-constructed Bayesian network. The Bayesian network is used to optimize the target detection results by combining the conditional probabilities of the target detection results of different entities, thereby obtaining optimized target detection results.

[0059] For example, in one possible implementation, if the aerial image of the target contains a lawn, and the Bayesian network defines a high probability of a paper bag appearing on the lawn, it means that the target detection model may have missed detections. As another example, if the aerial image of the target contains a lawn, and the probability of a certain image region being a vehicle is 0.6, and the probability of it being a paper bag is 0.65, if the Bayesian network sets the probability of a paper bag appearing on the lawn to be much higher than the probability of a vehicle appearing, it can correct and output that the image region is a paper bag, and the probability of that image region being a paper bag.

[0060] Understandably, in the object detection process, the final output can be the detection results of various image regions in the aerial image based on the detection head (such as the YOLO detection head or the RPN of Faster R-CNN) and the probability of each detection result being true. In practical applications, a certain image region may have multiple detection results, along with the corresponding probabilities of each detection result being true.

[0061] In practical applications, these probabilities can be adjusted by combining the spatial relationships between different entity objects in the knowledge graph. For example, for an image region classified as "vehicle," if the detection result is based on entity object nodes related to "water area" in the knowledge graph, the final output probability can be adjusted based on the pre-set relevance (or weight) between vehicles and water area, thereby reducing the probability. Conversely, for an image region assigned as "vehicle," if the detection result is based on entity object nodes related to "road" in the knowledge graph, the final output probability can be adjusted based on the pre-set relevance (or weight) between vehicles and roads, thereby increasing the probability.

[0062] Furthermore, after optimizing and adjusting the target detection results using the Bayesian network described above, the target detection model can be trained using the optimized probabilities to improve the confidence of the target detection model's output. Specifically, based on the output of the target detection model and the target detection results optimized by the Bayesian network, a loss function for the target detection model is constructed, and the target detection model is trained based on the loss function.

[0063] In summary, the target detection method for UAV aerial images provided in this application fuses the visual features of the target aerial image with the node features of the knowledge graph to obtain fused features. These fused features enhance the visual features of the target aerial image through the node features of the knowledge graph. Combined with the fused image of the target aerial image and the knowledge graph, the method performs target detection on the target aerial image. By combining feature fusion, this method integrates target detection of UAV aerial images with the knowledge graph. During the target detection process, the association relationships between different entity objects in the knowledge graph serve as constraints, which helps to improve the accuracy and generalization ability of target detection.

[0064] This application also provides a target detection device for drone aerial images; please refer to [reference needed]. Figure 2 , Figure 2 This is a structural diagram of the target detection device for drone aerial images provided in an embodiment of this application.

[0065] like Figure 2 As shown, the target detection device for drone aerial images includes: The feature extraction unit 201 is used to perform feature extraction processing on the target aerial image to obtain the visual features of the target aerial image.

[0066] The feature matching unit 202 is used to determine the target node that matches the visual features based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph.

[0067] The feature fusion unit 203 is used to fuse the visual features of the target aerial image with the node features of the target node to obtain the fused features of the target aerial image; wherein, the node features of each node in the knowledge graph include: the association relationship features between the entity objects corresponding to each node and the entity object features of the entity objects.

[0068] The target detection unit 204 is used to perform target detection on the target aerial image based on the fusion features of the target aerial image and the knowledge graph.

[0069] In one optional embodiment of this application, determining the target node matching the visual features based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph includes: calculating the correlation between the visual features of each spatial region in the target aerial image and the entity object features of each node in the knowledge graph to determine the correlation between the visual features of each spatial region and each node in the knowledge graph; and determining the target node matching the visual features of the spatial region based on the correlation.

[0070] In one optional embodiment of this application, the relevance is determined by the following formula: ; in, Indicates the relevance; The relevance is calculated using weights; This represents the transformation matrix that maps the visual features to the latent space; The transformation matrix represents the mapping of the features of the entity object to the causal space; Represents the visual features of the i-th spatial region; The entity object features representing the k-th node of the spatial map; This indicates the bias in the correlation calculation.

[0071] In one optional embodiment of this application, the step of fusing the visual features of the target aerial image with the node features of the target node to obtain the fused features of the target aerial image includes: determining at least one target node that matches the visual features of the target aerial image; Based on the correlation between each target node and the visual feature, the visual features of the target aerial image and the node features of the target node are fused to obtain the fused features of the target aerial image.

[0072] In one optional embodiment of this application, the fusion features of the target aerial image are obtained by the following formula: ; in, This represents the fusion feature corresponding to the i-th image region of the target aerial image; This represents the visual features corresponding to the i-th image region of the target aerial image; The similarity between the fusion feature corresponding to the i-th image region and the entity object feature of the j-th target node in the knowledge graph is represented; N represents the number of target nodes. This represents the node characteristics of the j-th target node; It represents the linear transformation matrix that transforms the dimension of the target node's node feature space to the visual feature space.

[0073] In one optional embodiment of this application, the step of performing target detection on the target aerial image based on the fusion features of the target aerial image and the knowledge graph includes: inputting the fusion features of the target aerial image as a new node of the knowledge graph into a pre-trained target detection model, so as to perform target detection on the target aerial image by combining the fusion features and the association relationship features between the entity objects corresponding to each node in the knowledge graph through the target detection model, and obtaining the target detection result of the target aerial image.

[0074] In one optional embodiment of this application, the apparatus is further configured to: input the target detection result of the target aerial image into a pre-constructed Bayesian network, so as to optimize the target detection result by combining the conditional probability of the target detection results of different entity objects through the Bayesian network, and obtain an optimized target detection result.

[0075] The apparatus embodiments provided in this example belong to the same application concept as the method embodiments of this application. They can execute the target detection method for UAV aerial images provided in any of the above embodiments of this application, and have the corresponding functional modules and beneficial effects for executing the target detection method for UAV aerial images. Technical details not described in detail in this example can be found in the specific processing content of the target detection method for UAV aerial images provided in the above embodiments of this application, and will not be repeated here.

[0076] This application also provides an electronic device, please refer to... Figure 3 , Figure 3 This is a schematic diagram of an electronic device structure provided in an embodiment of this application.

[0077] like Figure 3 As shown, the electronic device includes: Processor 210; memory 200 for storing executable instructions of processor 210; processor 210 is used to execute the target detection method of UAV aerial images disclosed in any of the above embodiments by running the instructions in memory 200.

[0078] The processor 210, memory 200, communication interface 220, input device 230 and output device 240 are interconnected via a bus.

[0079] A bus can include a pathway for transmitting information between various components of a computer system.

[0080] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0081] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0082] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0083] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, touch screen, etc.

[0084] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0085] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0086] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement each step of any of the target detection methods for UAV aerial images provided in the above embodiments of this application.

[0087] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the target detection methods for drone aerial images according to various embodiments of this application.

[0088] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0089] Furthermore, embodiments of this application may also be storage media storing computer programs, which are executed by a processor in the target detection methods for drone aerial images according to various embodiments of this application.

[0090] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0091] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0092] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0093] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0094] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0095] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0096] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0097] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0099] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection method for drone aerial images, characterized in that, include: Feature extraction processing is performed on the target aerial image to obtain the visual features of the target aerial image; Based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph, the target node that matches the visual features is determined. The visual features of the target aerial image are fused with the node features of the target node to obtain the fused features of the target aerial image; wherein, the node features of each node in the knowledge graph include: the relationship features between the entity objects corresponding to each node and the entity object features of the entity objects; Target detection is performed on the target aerial image based on the fusion features of the target aerial image and the knowledge graph.

2. The method according to claim 1, characterized in that, The step of determining the target node matching the visual features based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph includes: The correlation between the visual features of each spatial region in the target aerial image and the entity object features of each node in the knowledge graph is calculated to determine the correlation between the visual features of each spatial region and each node in the knowledge graph. Based on the relevance, target nodes that match the visual features of the spatial region are determined.

3. The method according to claim 2, characterized in that, The relevance is determined by the following formula: ； in, Indicates the relevance; The relevance is calculated using weights; This represents the transformation matrix that maps the visual features to the latent space; The transformation matrix represents the mapping of the entity object's features to the causal space; Represents the visual features of the i-th spatial region; The entity object feature representing the k-th node of the spatial graph; This indicates the bias in the correlation calculation.

4. The method according to claim 1, characterized in that, The step of fusing the visual features of the target aerial image with the node features of the target node to obtain the fused features of the target aerial image includes: Identify at least one target node that matches the visual features of the target aerial image; Based on the correlation between each target node and the visual feature, the visual features of the target aerial image and the node features of the target node are fused to obtain the fused features of the target aerial image.

5. The method according to claim 4, characterized in that, The fusion features of the target aerial image are obtained using the following formula: ； in, This represents the fusion feature corresponding to the i-th image region of the target aerial image; This represents the visual features corresponding to the i-th image region of the target aerial image; The similarity between the fusion feature corresponding to the i-th image region and the entity object feature of the j-th target node in the knowledge graph is represented; N represents the number of target nodes. This represents the node characteristics of the j-th target node; It represents the linear transformation matrix that transforms the dimension of the target node's node feature space to the visual feature space.

6. The method according to claim 1, characterized in that, The target detection process based on the fusion features of the target aerial image and the knowledge graph includes: The fused features of the target aerial image are used as new nodes of the knowledge graph and input into a pre-trained target detection model. The target detection model combines the fused features with the association features between the entity objects corresponding to each node in the knowledge graph to perform target detection on the target aerial image and obtain the target detection result of the target aerial image.

7. The method according to claim 1, characterized in that, The method further includes: The target detection results of the aerial image are input into a pre-constructed Bayesian network. The Bayesian network is used to optimize the target detection results by combining the conditional probabilities of the target detection results of different entities, thereby obtaining optimized target detection results.

8. A target detection device for drone aerial images, characterized in that, include: The feature extraction unit is used to perform feature extraction processing on the target aerial image to obtain the visual features of the target aerial image; The feature matching unit is used to determine the target node that matches the visual features based on the visual features of the target aerial image and the entity object features of each node in the knowledge graph. The feature fusion unit is used to fuse the visual features of the target aerial image with the node features of the target node to obtain the fused features of the target aerial image; wherein, the node features of each node in the knowledge graph include: the association relationship features between the entity objects corresponding to each node and the entity object features of the entity objects; The target detection unit is used to perform target detection on the target aerial image based on the fusion features of the target aerial image and the knowledge graph.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the target detection method for UAV aerial images according to any one of claims 1 to 7 by running instructions in the memory.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed, implement the target detection method for UAV aerial images according to any one of claims 1 to 7.

Images