Low-altitude aircraft panoramic detection method and system based on multi-modal monitoring

Abstract

The invention provides a low-altitude aircraft panoramic detection method and system based on multi-modal monitoring, and the method comprises the steps: determining a plurality of key objects in a multi-modal fusion image, and obtaining an image knowledge base used for indicating the object relation between a plurality of detection objects, detecting objects matched with the key objects are obtained from the multiple detecting objects and serve as matched detecting objects, and detecting object two-tuples including the matched detecting objects serving as source detecting objects and the matched detecting objects serving as termination detecting objects are obtained through exploration one by one in the multiple matched detecting objects; traversing the image knowledge base, obtaining the spatial relationship, generating an association link comprising the source detection object, the spatial relationship and the termination detection object, performing image scene understanding on the multi-modal fusion image according to the association link, and obtaining an investigation analysis result of the multi-modal fusion image. The method and the device can improve the precision of the investigation analysis result.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for panoramic detection of low-altitude aircraft based on multi-modal monitoring. Background Art

[0002] With the continuous development of aviation technology and the increasing application of low-altitude aircraft (such as small civilian drones, low-altitude general aviation aircraft, etc.), the effective detection of low-altitude aircraft has become crucial. In many fields such as aviation management and safety monitoring, accurately grasping the relevant information of low-altitude aircraft, including their location, status, and relationship with the surrounding environment, is the basis for ensuring airspace safety and achieving effective management.

[0003] Traditional aircraft detection technology often has many limitations. For example, if only a single-mode sensor is used for detection, although a visible light image sensor can obtain color images of the aircraft's appearance and surrounding environment and provide rich color and texture information, its detection effect will be greatly reduced in some special circumstances, such as low light conditions or when the aircraft is similar in color to the background. Although infrared thermal imaging detectors can capture the thermal radiation information of objects and are sensitive to heat sources, which helps to detect aircraft at night or in complex weather conditions, the images they provide are mainly based on temperature differences and lack color and detailed texture information. It is difficult to fully and accurately understand the scene information when used alone.

[0004] In the process of detecting and analyzing low-altitude aircraft, how to accurately understand the scene information in the detection image is also a huge challenge. Traditional analysis methods often lack in-depth mining of the relationship between multiple objects in the image. Summary of the invention

[0005] In view of this, the present application provides a method and system for panoramic detection of low-altitude aircraft based on multi-modal monitoring. The technical solution of the present application is implemented as follows: On the one hand, the present application provides a method for panoramic detection of low-altitude aircraft based on multimodal monitoring, comprising: determining a number of key objects in a multimodal fusion image, wherein the multimodal fusion image is obtained by fusing images detected by an infrared thermal imaging detector and a high-definition color visible light image sensor of the low-altitude aircraft; obtaining an image knowledge base for indicating object relationships between a number of detection objects, and obtaining detection objects that match the several key objects respectively from the several detection objects as matching detection objects; among the several matching detection objects, exploring and obtaining detection object tuples including matching detection objects as source detection objects and matching detection objects as termination detection objects one by one; the source detection object is different from the termination detection object; walking the image knowledge base to obtain a spatial relationship for establishing a connection between the source detection object and the termination detection object, and generating an association link including the source detection object, the spatial relationship and the termination detection object; performing image scene understanding on the multimodal fusion image based on the association link to obtain a detection and analysis result of the multimodal fusion image.

[0006] In a second aspect, the present application provides a computer system, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above-described method when executing the program.

[0007] Beneficial effects of the present application: The present application can determine several key objects used to indicate the multimodal fusion image in the multimodal fusion image, and based on the image knowledge base used to indicate the object relationship between several detection objects, obtain the detection objects that match the several key objects respectively as matching detection objects, so that the separated multimodal fusion image can be integrated with the image knowledge base, and among the several matching detection objects, the detection object tuples including the matching detection objects as the source detection objects and the matching detection objects as the termination detection objects are explored one by one, wherein the source detection objects are different from the termination detection objects, and based on walking the image knowledge base, the spatial relationship used to establish the source detection objects and the termination detection objects can be obtained, and the association link including the source detection objects, the spatial relationship and the termination detection objects is generated, and then, based on the association link, the image scene understanding can be performed on the multimodal fusion image to obtain the detection and analysis results of the multimodal fusion image, because the association link includes not only the matching detection objects but also the spatial relationship, so the present application can improve the accuracy of the detection and analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A schematic diagram of the implementation flow of a method for panoramic detection of low-altitude aircraft based on multimodal monitoring provided in an embodiment of the present application.

[0009] Figure 2A hardware entity diagram of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The embodiment of the present application provides a method for panoramic detection of low-altitude aircraft based on multimodal monitoring, which can be executed by a processor of a computer system. The computer system can refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, a desktop computer, a mobile device, etc.

[0011] Figure 1 A schematic diagram of the implementation process of a low-altitude aircraft panoramic detection method based on multi-modal monitoring provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: Step S100: determining a number of key objects in a multimodal fusion image, wherein the multimodal fusion image is obtained by fusing images detected by an infrared thermal imaging detector of a low-altitude aircraft and a high-definition color visible light image sensor.

[0012] In an embodiment of the present application, a multimodal fusion image is formed by fusing images detected by an infrared thermal imaging detector of a low-altitude aircraft and a high-definition color visible light image sensor. For example, an infrared thermal imaging detector can capture the thermal radiation information of an object to form a thermal imaging image. In this image, objects of different temperatures present different grayscales or colors (for example, a high-temperature object may appear white or bright tones, and a low-temperature object may appear black or dark tones). The high-definition color visible light image sensor can capture the visible light reflection information of an object to form a color image familiar to daily life, containing rich color and texture information. By fusing these two images, a multimodal fusion image is obtained. This fused image combines the advantages of two different modal images and can provide more comprehensive and richer information.

[0013] When the computer system processes the multimodal fusion image, it needs to identify the key objects. Key objects refer to objects that are of great significance in the multimodal fusion image and have a key impact on subsequent detection and analysis. The method of determining key objects can be based on a variety of technical means.

[0014] One technical approach is feature analysis based on images. The computer system can extract features from multimodal fusion images, such as edge features, texture features, shape features, etc. Take edge features as an example. Edges are usually the dividing line between an object and the background or between different objects. By detecting edges, the approximate outline of the object can be determined. Assuming that there is an object that looks like a building in a multimodal fusion image, the computer system can determine the edge outline of the building through edge detection algorithms (such as the Canny edge detection algorithm). This is a feature extraction process.

[0015] For texture features, different objects often have different textures. For example, vegetation may have irregular, delicate textures, while man-made metal structures may have relatively regular textures. Computer systems can analyze the texture features in images through texture analysis algorithms (such as the gray-level co-occurrence matrix method) to distinguish different objects and determine possible key objects. Shape features are also an important basis for determining key objects. If there are some objects with specific shapes in the image, such as round oil tanks or rectangular buildings, the computer system can use shape recognition algorithms (such as Hough transform for detecting geometric shapes such as circles or rectangles) to identify these objects with specific shapes as candidates for key objects.

[0016] In addition to image feature analysis, the computer system can also combine prior knowledge to determine key objects. For example, in the detection scenario of low-altitude aircraft, certain types of objects are relatively common and important, such as other aircraft, important facilities on the ground (such as airport runways, communication base stations, etc.), and special terrain (such as mountains, rivers, etc.). The computer system can pre-store relevant information of these common and important objects (such as shape, texture, thermal imaging features, etc.), and then search for matching objects in the multimodal fusion image and determine them as key objects.

[0017] The process of determining key objects by the computer system is a process that comprehensively considers multiple factors. It needs to make full use of the rich information in the multimodal fusion image and the pre-acquired knowledge to accurately find the objects that are critical to the panoramic detection and lay the foundation for the subsequent detection work. This step plays a vital role in the entire low-altitude aircraft panoramic detection method. It directly affects the matching with the image knowledge base in the subsequent steps, the construction of the detection object binary group and the accuracy of the final detection and analysis results.

[0018] Step S200: obtaining an image knowledge base indicating an object relationship between a plurality of detection objects, and obtaining detection objects respectively matched with a plurality of key objects from the plurality of detection objects as matching detection objects.

[0019] In the embodiment of the present application, the image knowledge base (for example, a visual knowledge graph) is a knowledge collection that stores the relationships between various detection objects. It may contain attribute information of different detection objects, spatial relationships between objects (such as adjacent, contained, etc.), semantic relationships (such as category relationships, a detection object belongs to a specific category, and there may be superior-subordinate or associated relationships between different categories), etc. For example, in the low-altitude aircraft detection scenario, the image knowledge base may store the relationship between the aircraft and the airport runway (spatially, the aircraft takes off and lands near the runway or on the runway; semantically, the aircraft relies on the runway for take-off and landing operations), and may also store the relationship between the communication base station and the surrounding environment (such as buildings, vegetation, etc.) (spatially, the base station may be located on the top of the building, surrounded by vegetation; semantically, the base station provides communication services for an area within a certain range, and its coverage range is related to the surrounding terrain, building distribution, etc.) and other relationship information.

[0020] When the computer system executes step S200, it obtains relevant information from the image knowledge base. As for the concept of detection objects, in this scenario, detection objects refer to various entities that may appear within the detection range of low-altitude aircraft, including aircraft, various ground facilities (such as the airport runway and communication base stations mentioned above), natural geographical elements (such as mountains, rivers, etc.) and other objects that may affect flight or be affected by flight.

[0021] After acquiring an image knowledge base indicating object relationships between a plurality of detected objects, the computer system needs to find a matching detected object for the previously determined key object among the plurality of detected objects. The matching process here is based on a variety of factors and technical means.

[0022] One technical means is matching based on feature similarity. The computer system can extract feature vectors for each detected object and key object. For example, for a key object that is a suspected aircraft, the computer system can extract its shape feature vector in the multimodal fusion image (such as a vector composed of parameters reflecting the shape, such as the length, width, and height ratio), thermal imaging feature vector (such as the temperature distribution characteristics of different parts converted into vector form), and color and texture feature vectors in the visible light image (such as the histogram characteristics of color, the roughness of the texture, etc., converted into vectors). For the detected object, the same feature vector extraction is performed. Then, the computer system measures the similarity between them by calculating the distance between the feature vectors (such as the Euclidean distance formula). The smaller the distance, the more similar the two are, and the more likely they are to be matched detection objects.

[0023] Another technical approach is matching based on semantic relationships. If semantic information is stored in the image knowledge base, the computer system can use this information for matching. For example, if the key object is initially judged to be a vehicle-type object, the computer system will look for objects in the detection objects that are also marked as vehicle types and have similar functions or usage scenarios as matching detection objects. For example, aircraft and other flying vehicles (such as helicopters, hot air balloons, etc.) have a certain semantic relationship. If the key object is an aircraft, then the helicopter in the detection object may be considered a candidate for a matching detection object because of this semantic relationship.

[0024] In addition, matching can also be performed based on spatial relationships. Suppose in a multimodal fusion image, the key object is a suspected aircraft object close to a mountain range. The computer system searches the image knowledge base for detected objects that have a spatial relationship with the mountain range (such as objects close to the mountain range), and these detected objects have similar characteristics to the aircraft (such as flight trajectory, speed range, etc., if this information can be analyzed from the image or obtained from the knowledge base), then these detected objects may be determined as objects that match the key object.

[0025] Step S300: among a plurality of matching detection objects, one by one, explore and obtain detection object tuples including a matching detection object as a source detection object and a matching detection object as a termination detection object; the source detection object is different from the termination detection object.

[0026] In the embodiment of the present application, the source detection object is a matching detection object that serves as a starting point or a relationship initiator, and the termination detection object is a matching detection object that has a potential relationship with the source detection object and is at the end position in the relationship chain. For example, in a low-altitude aircraft detection scenario, if one matching detection object is an aircraft and the other matching detection object is an airport runway, then the aircraft can be regarded as the source detection object and the airport runway can be regarded as the termination detection object, because there is a landing, take-off and other association relationship between the aircraft and the runway, and there is a relationship direction from the aircraft to the runway.

[0027] When the computer system performs this step, a feasible technical means is to explore the relationship matrix. Assume that the computer system establishes a relationship matrix for each matching detection object, and the rows and columns of the matrix represent different matching detection objects. In this matrix, each element represents the weight of the relationship that may exist between two matching detection objects. For example, if there is a high possibility of association between matching detection object A (such as an aircraft) and matching detection object B (such as an airport runway) (such as based on the feature similarity, semantic relationship or spatial relationship determined in the previous step), then the corresponding element value in the relationship matrix will be higher.

[0028] The computer system searches for possible detection object tuples by traversing the relationship matrix. Starting from the first row and first column of the relationship matrix, the computer system considers the two matching detection objects corresponding to each element as a possible detection object tuple. If the element value meets a certain threshold condition (this threshold can be set according to the actual situation, such as determining a value indicating a stronger relationship based on historical data or experience), the two matching detection objects are determined as a detection object tuple, one of which is the source detection object and the other is the termination detection object. For example, in the relationship matrix, if the element value corresponding to the aircraft and the airport runway is higher than the set threshold, then (aircraft, airport runway) can be determined as a detection object tuple.

[0029] Another technical approach is based on a graph search algorithm. All matching detection objects can be considered as nodes in a graph, and the relationships between them can be considered as edges in the graph. The computer system starts with a matching detection object (source node, i.e., source detection object) and uses a depth-first search or breadth-first search algorithm to traverse other nodes (matching detection objects) in the graph. When a node (termination node, i.e., termination detection object) that is different from the source node and satisfies certain relationship conditions is found, a detection object binary can be determined. For example, with an aircraft as the source node, a depth-first search is performed in the graph. When the node of the airport runway is found, (aircraft, airport runway) can be determined as a detection object binary.

[0030] In the actual low-altitude aircraft detection scenario, there are still many situations that need to be considered. For example, when multiple aircraft exist in the multimodal fusion image at the same time, the computer system needs to explore each aircraft as a source detection object. Assume that there are matching detection objects such as aircraft A, aircraft B, airport runway C, and communication base station D. For aircraft A, the computer system may find that there is a landing relationship with airport runway C during the exploration process, thereby determining (aircraft A, airport runway C) as a detection object binary; at the same time, aircraft A may have some signal interaction or spatial position relationship with communication base station D (such as aircraft A flying within the signal coverage range of communication base station D), then (aircraft A, communication base station D) can also be determined as a detection object binary. For aircraft B, the same exploration process is also required, and the detection object binary between airport runway C or other matching detection objects may be determined.

[0031] In addition, when determining the detection object binary, the computer system also needs to consider the accuracy and completeness of the data. Since the multimodal fusion image may contain noise, missing or inaccurate information, some misjudgments may occur when based on the relationship matrix or graph search algorithm. For example, due to the influence of image noise, the features of a matching detection object may be misidentified, so that the relationship weight in the relationship matrix is ​​miscalculated, or it is incorrectly connected to other nodes in the graph search. To avoid this situation, the computer system can adopt a data verification and error correction mechanism. For example, for the detection object binary determined based on the relationship matrix, it can be verified again based on the original information in the multimodal fusion image to check whether there is really a reasonable relationship between the two matching detection objects. If there is any doubt, the element value in the relationship matrix can be re-evaluated or the graph search process can be re-performed.

[0032] Step S400: Walk the image knowledge base to obtain a spatial relationship for establishing a connection between a source detection object and a termination detection object, and generate an association link including the source detection object, the spatial relationship and the termination detection object.

[0033] In the embodiment of the present application, when the computer system performs a wandering operation, one possible technical means is a rule-based search. For example, the relationships in the knowledge base may be marked with different type labels, such as spatial relationship labels (including adjacent, contained, far away, etc.), functional relationship labels (such as aircraft relying on runways for takeoff and landing, communication base stations providing communication services for a certain area, etc.). The computer system starts from the source detection object and searches for objects that have a direct association relationship with the source detection object and the corresponding relationship type according to predefined rules.

[0034] Take the aircraft (source detection object) and the airport runway (termination detection object) as examples. In the image knowledge base, the aircraft's related information may have relationship records with multiple objects. The computer system starts searching from the aircraft's record, and when it finds a relationship with the airport runway, it identifies the spatial relationship corresponding to this relationship. Assuming that there is a "close" spatial relationship between the aircraft and the airport runway, this is the spatial relationship that establishes the connection between the two found during the knowledge base wandering process.

[0035] In this process, the computer system may use an index structure to improve search efficiency. For example, an index table is created for each detected object, which records other objects that have a direct relationship with the object and the corresponding relationship type. In this way, when searching from the source detected object, related objects and relationships can be quickly located.

[0036] In addition to rule-based search, graph traversal algorithms can also be used. The image knowledge base is considered as a graph structure, where the detected objects are nodes in the graph and the relationships between them are edges in the graph. The computer system can start from the node corresponding to the source detected object and use a depth-first search or breadth-first search algorithm to traverse the graph.

[0037] For example, taking the communication base station as the source detection object, in the knowledge base graph structure, through the depth-first search algorithm, the computer system sequentially visits the nodes connected to it along the edge (other detection objects). When the termination detection object (such as an aircraft within its signal coverage) is found, the corresponding relationship of the edge from the communication base station to the aircraft is recorded. These relationships are the spatial relationships that establish the connection between the two.

[0038] The determination of spatial relationships is crucial to building an association link. An association link is an ordered combination of source detection objects, spatial relationships, and terminal detection objects. It intuitively represents the connection path between the source detection object and the terminal detection object. Continuing with the example of an aircraft and an airport runway, if the spatial relationship of "approach" is determined, then the association link is (aircraft, approach, airport runway).

[0039] In more complex scenarios, there may be multiple intermediate detection objects and multiple spatial relationships. For example, there is an aircraft (source detection object), a mountain range, and an airport runway (termination detection object). When the computer system walks through the knowledge base, it may find that the aircraft has a spatial relationship with the mountain range that is "located on the side of the mountain range", and the mountain range has a spatial relationship with the airport runway that is "there is a runway near the mountain range". Then the association link may be (aircraft, located on the side of the mountain range, mountain range, there is a runway near the mountain range, airport runway).

[0040] When building an association link, the computer system needs to ensure that the spatial relationship in the link is accurate and reasonable. Since the image knowledge base may contain a large amount of information, there may be ambiguity or uncertainty in some relationships. In order to improve accuracy, the computer system can use a probabilistic model. For example, for each relationship between detected objects, a probability value is assigned to indicate the possibility of the existence of such a relationship. When building an association link, the relationship path with a higher probability value is preferred.

[0041] At the same time, in order to deal with the possible information inconsistency in the knowledge base, the computer system can use data fusion technology. For example, if the contradictory spatial relationship between the same pair of detection objects is obtained from different data sources or different relationship records, the computer system can make a comprehensive judgment based on factors such as the reliability of the data and the authority of the source, and select the most reasonable spatial relationship to construct the association link.

[0042] When the computer system executes step S400, it effectively walks through the image knowledge base, accurately determines the spatial relationship, and reasonably constructs the association link, which provides a key basis for the subsequent image scene understanding of the multimodal fusion image based on the association link and obtaining the investigation and analysis results. If the spatial relationship is determined incorrectly or the association link is not reasonably constructed in this step, it will directly affect the accuracy and effectiveness of the final investigation and analysis results.

[0043] Step S500: performing image scene understanding on the multimodal fusion image based on the associated links to obtain the detection and analysis results of the multimodal fusion image.

[0044] In the embodiment of the present application, image scene understanding is the process by which a computer system uses various information and algorithms to analyze the semantic information, spatial layout, and the relationship between objects contained in an image. For a multimodal fusion image, it contains information from an infrared thermal imaging detector and a high-definition color visible light image sensor, and the fusion of this information provides rich material for scene understanding.

[0045] Taking the low-altitude aircraft detection scene as an example, the associated links are such as (aircraft, approach, airport runway). When the computer system understands the scene in the multimodal fusion image based on such associated links, it first focuses on the performance of each element in the link in the image. For the aircraft, the source detection object, the computer system locates the specific position of the aircraft in the multimodal fusion image. Because it is a multimodal image, the computer system can comprehensively use thermal imaging information (for example, the aircraft engine part may appear as a high temperature area in the thermal image) and visible light image information (such as the appearance shape and color identification of the aircraft) to accurately identify the aircraft.

[0046] Similarly, for the airport runway, the terminal detection object, the computer system uses information in the multimodal image to identify it. The runway has a specific shape (long, flat rectangular shape) and color (such as gray concrete color) in the visible light image, and may appear differently in the thermal imaging image due to the temperature difference with the surrounding environment (for example, it may be cooler than the surrounding grass at night). The computer system determines the location of the runway in the image by analyzing these features.

[0047] As for the spatial relationship of "closeness" in the association link, the computer system measures the actual spatial relationship between the aircraft and the runway in the image according to a predefined metric of the degree of closeness. This metric can be based on the calculation of the pixel distance. If this distance is less than a certain pre-set threshold (for example, according to the actual definition of closeness, when the distance is less than 500 pixels, it is considered close), it means that the spatial relationship between the aircraft and the runway in the image meets the "closeness" relationship in the association link.

[0048] Computer systems can use a model-based approach when understanding image scenes. For example, a semantic model of a low-altitude aircraft scene is established, which defines the roles, relationships, and possible states of different detection objects in the scene. The computer system matches the information in the associated link with this semantic model to understand the entire scene. Taking the aircraft approaching the runway as an example, the semantic model may specify information such as the speed range and altitude range of the aircraft when approaching the runway. The computer system can determine whether the current scene meets the definition in the semantic model by analyzing the dynamic information in the multimodal fusion image (if the image contains continuous frames, the moving speed of the aircraft can be analyzed, etc.) or combining it with other sensor data (such as the altitude data of the aircraft).

[0049] In addition to model-based methods, instance-based reasoning can also be used. The computer system stores a large number of instances of low-altitude aircraft scenes, each of which contains different detection objects, associated links, and corresponding image features. When performing scene understanding on the current multimodal fusion image, the computer system searches for instances in the instance library that are similar to the current associated links and image features. For example, if the current associated link is (aircraft, approach, airport runway), and the features of the aircraft and runway in the image are similar to an instance in the instance library, then the computer system can refer to the scene understanding results in this instance (such as the purpose of the aircraft may be to land, take off, or perform specific operations near the runway, etc.) to infer the scene of the current image.

[0050] Based on the above analysis and understanding of the associated links in the multimodal fusion image, the computer system finally obtains the reconnaissance and analysis results of the multimodal fusion image. This reconnaissance and analysis result includes the understanding of the status of each detected object in the image, the relationship between them, and the meaning of the entire scene. For example, in the scene of an aircraft approaching the runway, the reconnaissance and analysis result may be information such as "the aircraft is approaching the runway and may be preparing to land. The current status of the aircraft is normal and the runway is available."

[0051] In this process, in order to improve the accuracy of the reconnaissance and analysis results, the noise and uncertainty factors in the multimodal fusion image are taken into account. For example, due to weather reasons or sensor errors, there may be some blur or inaccuracy in the multimodal fusion image. Filtering technology can be used to reduce the impact of noise on image feature extraction and analysis. For example, for the noise in the thermal imaging image, the median filtering algorithm can be used to replace the value of each pixel with the median of its neighborhood pixel values, thereby smoothing the image and improving the accuracy of feature extraction. In addition, the computer system can also use data fusion algorithms to combine the advantages of different modal images and reduce uncertainty. For example, in the process of identifying an aircraft, if the outline of the aircraft in the thermal imaging image is not clear due to thermal noise, but the shape and color identification of the aircraft in the visible light image are relatively clear, the computer system can fuse the information of these two modalities to improve the accuracy of aircraft identification, thereby improving the accuracy of the entire reconnaissance and analysis results.

[0052] As an implementation method, the key objects include the key object P a , a≥1, and a is not greater than the total number of the key objects; in step S200, a plurality of detection objects that are matched by the key objects are obtained from the plurality of detection objects as matching detection objects, including: Step S210: Determine the key object P a The object co-occurrence frequency between each of the several detection objects respectively; Step S220: obtaining s object co-occurrence frequencies from among the plurality of object co-occurrence frequencies; s≥1, and s is less than the total number of the plurality of object co-occurrence frequencies; the s object co-occurrence frequencies are greater than the other object co-occurrence frequencies; the other object co-occurrence frequencies include the object co-occurrence frequencies other than the s object co-occurrence frequencies from among the plurality of object co-occurrence frequencies; Step S230: Among the plurality of detected objects, the detected object corresponding to the s object co-occurrence frequency is determined as the key object P a The pending detection object; Step S240: Determine the key object P a The commonality measurement value between the detection object to be determined and the detection object to be determined are obtained, and the maximum commonality measurement value among the commonality measurement values ​​is obtained; Step S250: Determine the pending detection object containing the maximum commonality metric value as the key object P a The matching detection object.

[0053] First, in step S210, the computer system determines the key object P aThe object co-occurrence frequency between each of the several detected objects. The object co-occurrence frequency is an indicator to measure how often two objects appear at the same time in a specific scene or data set. In the low-altitude aircraft detection scenario, for example, the key object P a For an aircraft, the detected objects may include airport runways, communication base stations, mountains, etc. When the computer system determines the co-occurrence frequency of objects, it can use statistical analysis techniques. Assume that the computer system has a database containing multiple historical detection scene images. In these images, the number of times the aircraft and the airport runway appear at the same time, the number of times the aircraft and the communication base station appear at the same time, and the number of times the aircraft and the mountain appear at the same time are counted. If the aircraft and the airport runway appear at the same time 300 times in a total of 1,000 historical images, then the co-occurrence frequency of the aircraft and the airport runway is 300 / 1,000 = 0.3. In this way, the computer system can obtain the key object P a Object co-occurrence frequency between (aircraft) and each detected object.

[0054] Next is step S220, where the computer system obtains s object co-occurrence frequencies from a number of object co-occurrence frequencies. Here, s satisfies s≥1, and s is less than the total number of object co-occurrence frequencies. These s object co-occurrence frequencies are greater than other object co-occurrence frequencies, and other object co-occurrence frequencies refer to object co-occurrence frequencies other than the s object co-occurrence frequencies among a number of object co-occurrence frequencies. Continuing with the above example, assume that the computer system calculates that the object co-occurrence frequency between the aircraft and the airport runway is 0.3, that with the communication base station is 0.1, and that with the mountain range is 0.05. If s = 1 is set, then since 0.3 is the maximum object co-occurrence frequency, the computer system will select the object co-occurrence frequency of 0.3 (i.e., the co-occurrence frequency between the aircraft and the airport runway). The significance of this selection method is that a higher co-occurrence frequency often implies a closer connection between the two objects, and they are more likely to be matching detection objects.

[0055] Then, in step S230, the computer system determines the detected object corresponding to the co-occurrence frequency of s objects as the key object P among the plurality of detected objects. a According to the previous example, since the co-occurrence frequency of the aircraft and the airport runway is 0.3, the airport runway is determined to be the aircraft (key object P a This step is based on the assumption that the previously determined detection objects with a high co-occurrence frequency are likely to match the key objects, and these potential detection objects are temporarily marked for further analysis and confirmation.

[0056] The following step S240 requires the computer system to determine the key object P aThe commonality metric value between the key object and the pending detection object is calculated, and the maximum commonality metric value among the commonality metric values ​​is obtained. The commonality metric value is a value used to measure the degree of similarity or common characteristics between the key object and the pending detection object. The computer system can use the feature vector method to calculate the commonality metric value. For example, for the aircraft (key object P a ) and airport runways (to be determined), the computer system extracts feature vectors for aircraft, which may include shape features of the aircraft (such as fuselage length, wingspan, etc.), thermal imaging features (temperature distribution of engine parts, etc.), and flight attitude features (such as tilt angle, flight altitude, etc.); the feature vectors extracted for airport runways may include the length, width, direction of the runway, and material features of the runway surface (through thermal imaging and visible light image analysis), etc. Then, a distance measurement formula can be used to calculate the commonality measurement value, such as the cosine similarity formula: ,in are the feature vectors of the aircraft and the airport runway respectively. The cosine similarity value obtained by calculation is the commonality measurement value between them. Assume that the computer system also calculates the commonality measurement value between the aircraft and other pending detection objects (such as communication base stations, mountains, etc.), and then finds the largest commonality measurement value.

[0057] Finally, in step S250, the computer system determines the pending detection object containing the maximum commonality metric value as the key object P. a Based on the previous example, if the commonality metric between the aircraft and the airport runway is the largest among all pending detection objects, then the airport runway is finally determined to be the aircraft (key object P a ) of the matching detection object. This determination process takes into account the co-occurrence frequency and commonality measurement of the objects, and ensures the accuracy of the matching detection objects through multiple rounds of screening and comparison. This accuracy has a crucial impact on subsequent steps, such as building detection object tuples, generating associated links, and the final detection and analysis results. Because if the matching detection object is determined incorrectly, the various relationships and analysis results established based on the incorrect match will deviate from the actual situation, resulting in a significant reduction in the accuracy of the entire low-altitude aircraft panoramic detection.

[0058] In the actual low-altitude aircraft detection scene, this method of determining the matching detection object needs to deal with various complex situations. For example, when there are multiple key objects, each key object needs to determine the matching detection object according to the above steps S210-S250. For another example, since the multimodal fusion image may have noise, partial information missing or inaccurate, it may be affected in the process of calculating the object co-occurrence frequency, commonality measurement value, etc. For the calculation of the object co-occurrence frequency, if a certain detection object in the image is misidentified due to noise interference or partial information missing, resulting in inaccurate identification, then the calculated co-occurrence frequency may be inaccurate. Similarly, when calculating the commonality measurement value, if the extraction of the feature vector is affected by the image quality, it will also lead to the deviation of the commonality measurement value. In order to deal with these problems, the computer system can use data preprocessing technology, such as denoising the multimodal fusion image (wavelet denoising and other methods can be used), reasonably filling in the missing information (for example, filling based on the characteristics and relationships of the surrounding objects), etc., to improve the accuracy of the calculation results of each step.

[0059] In addition, the relationship between different types of key objects and detection objects is diverse. Taking aircraft and communication base stations as an example, in addition to the spatial relationship between them (such as the aircraft flying within the signal coverage of the communication base station), there may also be an electromagnetic interaction relationship. This relationship may require special consideration when determining the co-occurrence frequency and commonality metric of objects. For example, when calculating the commonality metric, in addition to considering conventional features such as shape and thermal imaging, it may also be necessary to consider features such as the electromagnetic signal strength between the aircraft and the communication base station. This requires the computer system to fully consider various relationships and features when constructing feature vectors in order to accurately calculate the commonality metric and then determine the correct matching detection object.

[0060] At the same time, as the detection scenes continue to change and the complexity of the detection tasks increases, the computer system may need to continuously adjust the method of calculating the co-occurrence frequency and commonality measurement value of objects. For example, under different weather conditions, the feature performance of multimodal fusion images will be different. In clear weather, visible light images can clearly show the shape and color of objects, and thermal imaging images can accurately reflect the temperature distribution of objects; but in bad weather (such as rain, fog, snow, etc.), the image may become blurred and the feature extraction of objects will become difficult. At this time, the computer system may need to adjust the feature extraction algorithm according to the weather conditions, or make weighted adjustments to the calculated object co-occurrence frequency and commonality measurement value to adapt to different detection environments and ensure the accuracy of matching the detection object.

[0061] As an implementation method, step S210 determines the key object P a The object co-occurrence frequency between each of the several detected objects, including: Step S211: According to the moving step of the sliding control, a Walk around and get the key object P a The corresponding key local features; the number of key local features is equal to the moving step of the sliding control; Step S212: Obtain local features of a plurality of detected objects respectively corresponding to the detected objects; the plurality of detected objects include the detected object M b , b ≥ 1, and b is not greater than the total number of detected objects; the local features of several objects include the detected object M b The corresponding local features of the object; Step S213: Key local features and detection object M b The corresponding local features of the objects are analyzed for commonality, and the key object P is obtained. a With the detected object M b The co-occurrence frequency of objects between .

[0062] First, in step S211, the computer system moves the key object P according to the moving stride of the sliding control (such as a window). a Walk around and get the key object P a The corresponding key local features. The sliding control here is a tool used to traverse the image or data according to certain rules, and the moving step determines the distance or range of each slide. In the low-altitude aircraft detection scenario, for example, the key object P a For an aircraft, the computer system can regard the representation of the aircraft in the multimodal fusion image as a data set. Assume that the sliding control is a rectangular window with a fixed number of pixels (such as 5 pixels each time). The computer system starts from a certain starting position of the aircraft image and moves the sliding window according to this moving step. Each time, the partial features of the aircraft covered by the window are regarded as key local features. For example, the window covers the wing part of the aircraft for the first time. The shape, texture and thermal imaging features of this part constitute a key local feature; as the window moves, it may cover different parts of the aircraft such as the fuselage and engine, thereby obtaining multiple key local features. This method can decompose a complex object such as an aircraft into multiple local features for analysis, which helps to study its relationship with other detected objects in more detail.

[0063] Next is step S212, where the computer system obtains local object features corresponding to several detected objects. In the example, several detected objects include, for example, airport runways, communication base stations, mountains, etc. For airport runways, the computer system can also obtain local object features in a similar manner. For example, the runway is regarded as a long strip of object, and a sliding window (which may have the same or different parameters as the sliding window used for the aircraft, such as size, moving stride, etc.) is used to move on the runway image. The features of the runway portion covered by the window each time, such as the texture features of the runway surface (line texture in the visible light image and temperature difference texture in the thermal imaging image), the shape features of the runway edge, etc., constitute the local object features of the runway. For a communication base station, its local object features may include the shape features of the base station tower (such as height, cross-sectional shape, etc.), the shape and color features of the equipment on the top of the base station, etc.; for a mountain range, the local object features may be the contour shape of the mountain range, the vegetation coverage features of different altitudes (through visible light image analysis) or the temperature change features (through thermal imaging image analysis), etc.

[0064] Then, in step S213, the computer system detects the key local features and the detected object M. b The corresponding local features of the objects are analyzed for commonality, and the key object P is obtained. a With the detected object M b This step is to find the common features or similarities between the key objects and the detected objects by comparing the key local features of the key objects and the object local features of the detected objects, and then derive the co-occurrence frequency. a ) and the airport runway (detection object M b ) as an example, the computer system compares the key local features of the aircraft with the object local features of the runway. For example, the shape features of the aircraft wing may be related to the shape features of the runway, such as the extension direction of the wing and the direction of the runway may be parallel (this can be regarded as a common feature). For another example, the thermal imaging features of the aircraft engine part may be compared with the temperature features of the runway surface due to sunlight or other factors (such as the temperature difference between the high-temperature part of the engine and the relatively low-temperature part of the runway). The computer system determines the object co-occurrence frequency between them by quantitatively analyzing the various relationships between these key local features and object local features.

[0065] In actual operation, the computer system can use a variety of technical means to achieve this object commonality analysis. One method is an algorithm based on feature point matching. For example, for the key local features of the aircraft and the object local features of the runway, the computer system can extract their feature points (such as corner points, edge points, etc.) respectively, and then determine the similarity between them by comparing the position, direction and descriptor information of the feature points. Assuming that a corner point of the aircraft wing and a corner point of the edge of the runway have a certain similarity in position and direction, this can be used as a common feature for statistics. Another method is an algorithm based on template matching. The computer system can regard a key local feature of the aircraft as a template, and then search for matching parts in the object local features of the runway. For example, the thermal imaging template of the aircraft engine (with a specific temperature distribution shape) can look for similar temperature distribution patterns in the thermal imaging features of the runway. If a matching part is found, the value of the object co-occurrence frequency can be increased.

[0066] However, there are many problems that need to be solved in this process. Since multimodal fusion images may have problems such as noise, resolution changes, and object occlusion, the extraction and analysis of key local features and object local features will be affected. For example, noise may lead to misdetection of feature points or inaccurate template matching. In low-resolution images, some key local features and object local features may not be extracted clearly, thus affecting the accuracy of commonality analysis. In the case of object occlusion, if part of the aircraft is blocked by other objects, the corresponding key local features cannot be fully obtained, and the commonality analysis with other detected objects will also be disturbed.

[0067] To overcome these problems, the computer system can adopt a series of optimization measures. For image noise, filtering technology can be used for preprocessing. For example, a Gaussian filtering algorithm is used to reduce the impact of noise on feature extraction by weighted averaging each pixel in the image and its neighboring pixels. For the problem of resolution change, image interpolation technology can be used to improve the quality of low-resolution images so as to better extract features. When dealing with object occlusion problems, the computer system can use multiple frames of images or other sensor information for supplementary analysis. For example, if the aircraft is partially obscured in a certain frame of the image, the computer system can refer to the complete features of the aircraft in the previous and next frames to extract key local features and conduct commonality analysis.

[0068] In addition, the characteristics of different types of key objects and detection objects vary greatly, which also increases the complexity of object commonality analysis. Taking aircraft and communication base stations as examples, aircraft have dynamic characteristics (such as changes in flight attitude, changes in engine status, etc.), while communication base stations are relatively static. When performing object commonality analysis, the computer system needs to take into account this difference between dynamic and static. For the dynamic characteristics of aircraft, the computer system can use the method of time series analysis to integrate and analyze the key local characteristics at different times, and then compare them with the static characteristics of the communication base station. For example, the height change of an aircraft during flight may be related to the signal coverage range of the communication base station (such as the different base station signal strengths received by the aircraft at different altitudes). The computer system needs to use a complex mathematical model to describe this relationship in order to accurately perform commonality analysis.

[0069] As an implementation method, the total number of key local features is t, and the t key local features include key local features Q c , t and c are both greater than 0, and c is not greater than t; Step S213, the key local features and the detection object M b The corresponding local features of the objects are analyzed for commonality, and the key object P is obtained. a With the detected object M b The co-occurrence frequency of objects between, including: Step S2131: The key local feature Q c With the detected object M b The corresponding local features of the object are matched; Step S2132: If the detected object M b The corresponding object local features have the key local features Q c The same object local features will be combined with the key local features Q c The total number of local features of the same object is determined as the key local feature Q c The corresponding object co-occurrence frequency; Step S2133: The sum of the object co-occurrence frequencies corresponding to the t key local features is determined as the key object P a With the detected object M b The co-occurrence frequency of objects between .

[0070] First, in step S2131, the computer system converts the key local feature Q c With the detected object M b In the low-altitude aircraft detection scenario, the key object P a For the aircraft, the key local feature Q cIt may be the characteristics of a specific part of the aircraft wing (such as the wing tip), including its shape (such as a specific curve shape), color (color value in visible light image) and temperature distribution characteristics in thermal imaging. b When it is an airport runway, the corresponding local features of the object are the features of a certain part of the runway, such as the shape of the runway edge (the straight line shape and its boundary features with the surrounding area), the texture of the runway surface (the texture presented under visible light and the temperature texture caused by material and environmental factors under thermal imaging), etc. The computer system performs this matching operation through a specific matching algorithm. A feasible technical means is to use a matching algorithm based on feature descriptors, such as the SIFT (Scale Invariant Feature Transform) algorithm. This algorithm can be used for key local features Q c The local features of the object and the runway are used to generate feature descriptors, which are mathematical descriptions of the area around the feature points and can reflect the essential information of the features to a certain extent. The computer system then compares these feature descriptors to determine whether the two match.

[0071] Then proceed to step S2132. If the detected object M b The corresponding object local features have the key local features Q c The same object local features will be combined with the key local features Q c The total number of local features of the same object is determined as the key local feature Q c The corresponding object co-occurrence frequency. For example, suppose the shape feature of the aircraft wing tip appears multiple times in the local features of the runway object (because the runway may have multiple parts similar to the shape of the wing tip, which may be due to visual angles, image deformation, etc.). After matching, the computer system counts the number of times this same shape feature appears, assuming it is 5 times, then these 5 times are the key local features Q c (wing tip part) The corresponding object co-occurrence frequency. It should be noted here that the determination of the local features of the same object needs to be based on the previously set matching criteria. For example, in terms of shape features, a certain shape similarity threshold may need to be met. This threshold can be set based on actual conditions and experience.

[0072] Finally, in step S2133, the computer system determines the sum of the object co-occurrence frequencies corresponding to the t key local features as the key object P a With the detected object M bContinuing with the example of the aircraft and the airport runway, assuming that the aircraft has five key local features (such as the wing tip, a section of the fuselage, the engine part, etc.), the object co-occurrence frequencies between them and the object local features corresponding to the runway are calculated as 3, 2, 1, 0, and 1 respectively (these values ​​are the object co-occurrence frequencies corresponding to each key local feature obtained in step S2132). Then add these values ​​together, that is, 3 + 2 + 1 + 0 + 1 = 7, and this 7 is the key object P a (Aircraft) and detected object M b (airport runways).

[0073] However, there are many complex situations and challenges in this process. First, due to the characteristics of multimodal fusion images, the fusion of information between different modalities may bring some uncertainty.

[0074] For example, it may be difficult to match a local feature of an aircraft in a visible light image with a local feature of a runway in a thermal image. Because visible light images mainly reflect information such as color and shape, while thermal images mainly reflect temperature distribution information, the features of the two are essentially different. When matching, the computer system needs to consider how to effectively fuse and compare these two features of different natures. One possible solution is to transform the thermal imaging features so that they can be compared with the visible light features in the same feature space. For example, based on the relationship between the temperature value in the thermal imaging feature and factors such as the object material and environment, the temperature value can be mapped to a feature similar to a color value or shape description, and then a matching operation can be performed.

[0075] Secondly, the noise and interference factors in the image will affect the calculation of matching and co-occurrence frequency. Noise may lead to the misrecognition of features, so that originally dissimilar features are mistakenly judged to be similar, or vice versa. For example, due to the noise of the image sensor, the edge of the aircraft wing may appear some irregular fluctuations in the image, which may lead to erroneous results when matching with the runway edge features. In order to deal with this situation, the computer system can pre-process the image before matching, such as using a filtering algorithm to reduce the impact of noise. For interference factors in the image, such as shadows (in visible light images) or thermal interference in thermal imaging (such as interference from nearby heat sources), special processing is also required. For example, for the features of the shadow part, corrections can be made by analyzing the features of the surrounding normal light area and the geometric shape of the object; for thermal interference, its influence on the target features can be eliminated according to the distribution law of the heat source and the principle of heat conduction.

[0076] In addition, different matching algorithms vary in accuracy and efficiency, and may have different applicability for different types of features. For example, the SIFT algorithm works well when processing features with obvious texture and shape changes, but may not perform well for some smoother features with less obvious textures. The computer system may need to select an appropriate matching algorithm based on the specific feature type and requirements, or use a combination of multiple algorithms to improve the accuracy of matching. For example, for local features such as the engine part of an aircraft with relatively complex textures and temperature characteristics, the SIFT algorithm and a matching algorithm based on thermal imaging features (such as matching based on features such as the direction of the temperature gradient) can be used simultaneously, and then the results of the two can be combined to determine the matching status and object co-occurrence frequency with the local features of the runway object.

[0077] As an implementation method, step S240 determines the key object P a Commonality metrics with the pending detection object, including: Step S241: Retrieve the feature matching network to generate key object P a An implicit representation of a first feature of , and an implicit representation of a second feature of a to-be-determined detection object; Step S242: Determine the commonality measure between the first feature implicit representation and the second feature implicit representation as the key object P a A measure of commonality with the pending detection object.

[0078] First, in step S241, the computer system retrieves the feature matching network and generates the key object P a The first feature implicit representation of the object to be detected and the second feature implicit representation of the object to be detected. The feature matching network is a pre-trained neural network model that can learn and abstract the input object features and convert them into an implicit feature representation. In the low-altitude aircraft detection scenario, the key object P a For aircraft, aircraft have many complex features, such as shape features (shape of fuselage, wings, etc.), thermal imaging features (temperature distribution of engine parts, etc.) and flight attitude features (tilt angle, height, etc.). When these feature data of the aircraft are input into the feature matching network, the network will perform a series of complex nonlinear transformations on these features to obtain the first feature implicit representation of the aircraft.

[0079] For example, suppose the feature matching network is a multi-layer convolutional neural network (CNN). The shape features of the aircraft can be represented as image data. CNN gradually extracts high-level features from the image through operations such as convolutional layers and pooling layers. For thermal imaging features and flight attitude features, they can be digitized and used as network input together with shape feature data. After the network's forward propagation process, a vector is finally output. This vector is the aircraft (key object P a ). This vector contains the representation of the aircraft in the abstract feature space learned by the feature matching network, which embodies various feature information of the aircraft in a compressed and abstract form.

[0080] Similarly, for the undetermined detection object, such as the airport runway, the runway also has its own characteristics, such as length, width, direction, surface material (obtained through the analysis of visible light images and thermal imaging images), etc. After the feature data of these runways are input into the feature matching network, the network will generate the second feature implicit representation of the runway according to the same principle. This is also a vector, which represents the abstract representation of the runway in the feature space of the feature matching network.

[0081] Next, in step S242, the computer system determines the commonality measure between the first feature implicit representation and the second feature implicit representation as the key object P a The commonality measure between the object to be detected. Determining the commonality measure is a key operation to measure the similarity between two objects in the feature abstract space. One technical approach is to calculate the distance measure between two feature implicit representation vectors. For example, the cosine similarity formula can be used to calculate it. This value ranges from -1 to 1. The closer the value is to 1, the higher the similarity between the two objects in the feature abstract space; the closer the value is to -1, the greater the difference between the two; when the value is 0, it means that there is no obvious similarity or correlation between the two.

[0082] As an implementation method, step S241, before calling the feature matching network, also includes a training process of the feature matching network, which is specifically as follows: Step S240a: Acquire a plurality of active training object binary groups; the plurality of active training object binary groups include active training object binary group T d , d ≥ 1, and d is not greater than the total number of binary pairs of active training objects; Step S240b: Generate active training object binary group T through basic feature matching network d The first inference feature implicit representation of the training key object in and the positive training object bigram T d The second inference feature of the first detected object in is implicitly represented; the first detected object belongs to the image knowledge base; Step S240c: Determine the first inference feature implicit representation and the second inference feature implicit representation as an active training object binary group T d The inference feature of implicitly represents bigrams; Step S240d: Based on the inference feature implicit representation tuples corresponding to the plurality of active training object tuples, the parameters in the basic feature matching network are optimized to obtain a feature matching network.

[0083] First, in step S240a, the computer system obtains several positive training object tuples (which can be understood as positive samples). In the low-altitude aircraft detection scenario, positive training object tuples refer to those object pairs that are known to have a specific relationship and should be determined to be matched. For example, (aircraft, airport runway) is a typical positive training object tuple, because there is a close relationship between aircraft and airport runways, such as aircraft need to take off and land on the runway. For another example, (aircraft, communication base station) can also be used as a positive training object tuple in certain circumstances, because the aircraft has signal interaction and other connections with the communication base station during the flight. The computer system constructs a set of positive training object tuples by collecting a large number of such object pairs with known matching relationships. These tuples will be used to train the feature matching network so that the network can learn how to identify feature patterns between objects with matching relationships.

[0084] Next, in step S240b, the computer system generates a positive training object binary group T through the basic feature matching network. d The first inference feature implicit representation of the training key object in and the positive training object bigram T d The second inference feature implicit representation of the first detected object in the basic feature matching network. The basic feature matching network is an initial network model that has not been fully trained. It has a certain network structure, such as a neural network structure containing multiple convolutional layers, fully connected layers, etc. Take the active training object binary (aircraft, airport runway) as an example. For the aircraft (as the key training object), the computer system inputs the relevant feature data of the aircraft (such as shape, thermal imaging, flight attitude, etc.) into the basic feature matching network. The network performs convolution operations on the input features through its internal convolutional layer to extract local features, and then compresses the features through the pooling layer, and then integrates and maps these features through the fully connected layer, and finally outputs a vector. This vector is the implicit representation of the first inference feature of the aircraft. Similarly, for the airport runway (as the first detection object), the relevant feature data of the runway (such as length, width, direction, surface material, etc.) are input into the basic feature matching network. After the same network processing process, the second inference feature implicit representation of the runway is obtained.

[0085] Then, in step S240c, the computer system determines the first inference feature implicit representation and the second inference feature implicit representation as a positive training object binary group T d Continuing with the above example (aircraft, airport runway), the first inference feature implicit representation of the aircraft and the second inference feature implicit representation of the runway are combined to form an inference feature implicit representation tuple specific to this active training object tuple. This tuple fully represents the feature abstract representation of this pair of objects with known matching relationships under the basic feature matching network. The purpose of this operation is to subsequently optimize the parameters in the basic feature matching network based on these inference feature implicit representation tuples.

[0086] Finally, step S240d is to optimize the parameters in the basic feature matching network based on the implicit representation tuples of the inference features corresponding to the active training object tuples, and obtain the feature matching network. This step is the core of the entire training process, which aims to adjust the parameters of the network (such as weights and bias parameters in the neural network) so that the network can better represent and identify objects with matching relationships.

[0087] In this process, the computer system can use a variety of technical means to achieve parameter optimization. One method is an optimization algorithm based on a loss function, such as a gradient descent algorithm. Suppose a loss function L is defined to measure the difference between the inference feature implicit representation tuple output by the network and the ideal matching relationship. For each active training object tuple, a loss value is calculated based on its inference feature implicit representation tuple. For example, if the inference feature implicit representation tuple of the aircraft and runway output by the network is significantly different from the actual matching relationship that should be present, then the loss value will be large.

[0088] Taking mean square error (MSE) as the loss function as an example, let the active training object binary group T d The corresponding inference feature implicitly represents the bigram as , where x d To train implicit representations of inference features of key objects, is the implicit representation of the inference feature of the first detected object. Assuming that there is an ideal matching relationship, the corresponding feature representation is , then the mean square error loss function can be expressed as: , where n is the total number of active training object bigrams.

[0089] The computer system calculates the gradient of the loss function to the network parameters, and then updates the parameters according to the gradient descent algorithm. By continuously iterating this process, the loss is calculated and the parameters are updated using the inference features corresponding to multiple active training object pairs to implicitly represent the pairs. The basic feature matching network gradually converges, the parameters are continuously optimized, and finally a feature matching network that can accurately represent the feature relationship between matching objects is obtained.

[0090] In this training process, the selection and construction of active training object pairs have an important impact on the network training effect. If the number of active training object pairs is too small or lacks representativeness, the network may not be able to learn sufficiently comprehensive feature patterns. For example, if only a few types of aircraft and specific runways are collected for active training object pairs, the network may not be able to accurately match features when faced with other types of aircraft or runways in different environments. Therefore, it is necessary to collect as many active training object pairs of different types and scenarios as possible to ensure that the network has a wide range of applicability.

[0091] Secondly, the initial structure and parameter settings of the basic feature matching network will also affect the training effect. If the network structure is too simple, it may not be able to handle complex object features. For example, if the convolution layer of the network has only a few convolution kernels, then the rich shapes, textures and other features of the aircraft and runway may not be fully extracted. At the same time, if the initial parameter settings are unreasonable, such as the initialization value of the weight is too large or too small, the network may fall into a local optimal solution during the training process and fail to find the global optimal parameter value. In order to solve this problem, in terms of network structure design, it is necessary to reasonably determine the number of network layers, the number of convolution kernels, the pooling method and other parameters according to the complexity and diversity of object features. In terms of initial parameter setting, some common initialization methods can be used, such as Xavier initialization, He initialization, etc. These methods can reasonably set the initial weights according to the number of input and output neurons of the network layer, which helps the network converge better.

[0092] In addition, the way different types of features are handled during training also needs to be carefully considered. For example, the thermal imaging features and shape features of an aircraft have different properties and data distributions. When inputting these different types of features into the network, some preprocessing operations, such as normalization, may be required. For thermal imaging features, their value range may be very different from the value range of shape features. If normalization is not performed, the effects of some features may be over-amplified or reduced during network training. The minimum-maximum normalization method can be used to map feature values ​​to the interval [0, 1].

[0093] In the actual low-altitude aircraft panoramic detection scenario, the problem of data imbalance also needs to be considered. When collecting active training object pairs, there may be some types of matching relationship data that are easier to obtain, while others are relatively less. For example, the matching relationship data between aircraft and airport runways may be easier to collect, while the matching relationship data between aircraft and certain special facilities (such as small navigation facilities in remote areas) is less. This data imbalance may cause the network to tend to learn matching relationship patterns with more data during training, while insufficiently learning matching relationship patterns with less data. To solve this problem, data enhancement techniques can be used to increase the number of active training object pairs with less data. For example, for the matching relationship data between aircraft and special facilities, more training data can be generated by rotating, translating, scaling and other transformation operations on the existing data. At the same time, some techniques such as generative adversarial networks (GANs) or variational autoencoders (VAEs) can also be used to generate new data that is similar to the original data but has certain differences, so as to improve the network's learning ability for various matching relationships.

[0094] As an implementation mode, step S240d, based on the inference feature implicit representation tuples corresponding to the plurality of active training object tuples, optimizes the parameter variables in the basic feature matching network to obtain the feature matching network, including: Step S240d1: determining a first feature commonality measure between the first inference feature implicit representation and the second inference feature implicit representation; Step S240d2: Obtain an active training object binary group T from among several active training object binary groups d a second detected object in the active training object binary outside the image knowledge base; the second detected object belongs to the image knowledge base; Step S240d3: determining the inference feature implicit representation corresponding to the second detection object as the third inference feature implicit representation among the inference feature implicit representation tuples corresponding to the plurality of active training object tuples; Step S240d4: determining a second feature commonality measure between the first inference feature implicit representation and the third inference feature implicit representation; Step S240d5: Based on the first feature commonality metric value and the second feature commonality metric value, the parameter variables in the basic feature matching network are optimized to obtain a feature matching network.

[0095] First, in step S240d1, the computer system determines the first feature commonality metric between the first inference feature implicit representation and the second inference feature implicit representation. In the low-altitude aircraft detection scenario, taking the active training object binary (aircraft, airport runway) as an example, after the computer system obtains the first inference feature implicit representation of the aircraft and the second inference feature implicit representation of the airport runway through the basic feature matching network, the computer system can use a certain measurement method to calculate the commonality metric between them. For example, the cosine similarity formula is used to calculate the first feature commonality metric.

[0096] Next, in step S240d2, the computer system obtains a positive training object binary group T from among the plurality of positive training object binary groups. d Continuing with the example of aircraft, if the currently concerned active training object binary T d For example, if the image is (aircraft A, airport runway A), the computer system will obtain other second detection objects, namely, airport runway B, communication base station C, etc., from other active training object binary groups (such as (aircraft B, airport runway B), (aircraft C, communication base station C), etc.) in addition to the second detection object (aircraft runway A) in the current binary group. These second detection objects belong to the image knowledge base. They are objects that have a certain relationship with the corresponding training key objects determined in the process of constructing the active training object binary group, and the feature information of these objects is used to train the basic feature matching network.

[0097] Then comes step S240d3, in which the computer system determines the implicit representation of the inference feature corresponding to the second detection object as the third implicit representation of the inference feature among the implicit representation of the inference feature corresponding to the several active training object tuples. For example, for the active training object tuple (aircraft B, airport runway B), the computer system previously obtained the second implicit representation of the inference feature of the airport runway B through the basic feature matching network, and now determines this vector as the third implicit representation of the inference feature. Similarly, for other active training object tuples containing the second detection object obtained in step S240d2, such operation is also performed to obtain the corresponding third implicit representation of the inference feature. These third implicit representations of the inference feature will be used for subsequent comparison with the first implicit representation of the inference feature to further optimize the parameters of the basic feature matching network.

[0098] Next is step S240d4, where the computer system determines a second feature commonality measure between the first inference feature implicit representation and the third inference feature implicit representation. In the previous example, for aircraft A and airport runway B, a measurement method similar to that in step S240d1 (such as the cosine similarity formula) is also used to calculate the second feature commonality measure.

[0099] Finally, step S240d5 is to optimize the parameters in the basic feature matching network based on the first feature commonality metric value and the second feature commonality metric value to obtain a feature matching network. This step is the core part of the entire optimization process, and is intended to adjust the parameters of the network according to the relationship between different commonality metrics, so that the network can more accurately represent the matching relationship between different objects.

[0100] In this process, the computer system can use a gradient-based optimization algorithm to optimize the parameters. For example, suppose a loss function L is defined, and this loss function is constructed based on the commonality metric value of the first feature and the commonality metric value of the second feature. One possible construction method is: , where f is a function defined according to a specific optimization objective. For example, we can define , this function indicates that we hope that the difference between the first feature commonality metric (the similarity between the aircraft and its corresponding runway) and the second feature commonality metric (the similarity between the aircraft and other runways) is as small as possible, so that the network can better distinguish between correct matches and incorrect matches.

[0101] The computer system calculates the gradient of the loss function L for the parameters (such as weight w and bias b) in the basic feature matching network, and then updates the parameters according to the gradient descent algorithm. By continuously using different active training object pairs to perform the above calculations and parameter update operations, the parameters of the basic feature matching network are gradually optimized, and finally a feature matching network that can more accurately represent the matching relationship between objects is obtained.

[0102] When calculating the first feature commonality measure value and the second feature commonality measure value, the adopted measurement method can reflect the similarity between objects to a certain extent, but may have limitations.

[0103] For example, the cosine similarity metric may not be able to fully capture the complex relationship between object features in some cases. If there is a nonlinear relationship between certain features of the aircraft (such as flight attitude and engine status) and the features of the runway (such as surface condition and surrounding environment), using only cosine similarity may not accurately measure the true relationship between them. To solve this problem, other more complex metrics can be considered, such as kernel function-based metrics. For example, the Gaussian kernel function , where x and y are two eigenvectors, is a hyperparameter. This kernel function can map features to a high-dimensional space, which may more accurately reflect the complex relationships between objects.

[0104] Secondly, when obtaining the active training object binary, there may be incomplete data or noise interference. For example, the collected active training object binary of the aircraft and the runway may cause some feature data to be inaccurate due to measurement errors or environmental factors. This inaccurate data will affect the accuracy of the implicit representation of the inference feature, and then affect the calculation results of the first feature commonality measure and the second feature commonality measure. In order to reduce this influence, data cleaning and denoising techniques can be used in the data preprocessing stage. For data cleaning, obviously erroneous data can be removed by checking the rationality and consistency of the data. For example, if the altitude data of the aircraft is obviously beyond the normal range, it can be regarded as abnormal data and corrected or deleted. For denoising, some filtering techniques, such as Kalman filtering, can be used to filter dynamic data (such as the flight trajectory data of the aircraft) to reduce the impact of noise on the data.

[0105] In addition, when optimizing parameters based on the gradient descent algorithm, the choice of learning rate is very critical. If the learning rate is too large, the parameters may skip the optimal value during the update process, making the network unable to converge to the optimal state; if the learning rate is too small, the update speed of the parameters is too slow, which will lead to a long training time and may fall into a local optimal solution. To solve this problem, adaptive learning rate methods can be used, such as Adagrad, Adadelta or Adam optimization algorithms. These algorithms can automatically adjust the learning rate according to the gradient history information of the parameters, so that the parameters can converge to the optimal value at an appropriate speed.

[0106] At the same time, the parameter optimization process of the network may be affected by the network structure itself. If the structure of the basic feature matching network is too complex (for example, too many layers or too many neurons), it may cause gradient vanishing or gradient explosion problems. The gradient vanishing problem refers to the fact that during the back propagation process, the gradient gradually approaches zero as the number of network layers increases, making it impossible to effectively update the parameters; the gradient explosion problem refers to the fact that the gradient becomes too large during the back propagation process, resulting in a large update amplitude of the parameters and the network cannot converge stably. In order to solve these problems, some technical means can be used, such as using a residual network structure (ResNet) to alleviate the gradient vanishing problem, by adding short-circuit connections in the network, so that the gradient can be propagated more directly to the shallow network; for the gradient explosion problem, the gradient clipping technology can be used to limit the norm of the gradient to a reasonable range, for example, the norm of the gradient vector \vec{g} is limited to no more than a set value C, that is, \vert\vert\vec{g}\vert\vert \leq C.

[0107] As an implementation manner, step S240d5, based on the first feature commonality metric value and the second feature commonality metric value, optimizes the parameter variables in the basic feature matching network to obtain the feature matching network, including: Step S240d51: Determine the pair of active training objects T to optimize the basic feature matching network d A negative weight parameter of the focusing degree of the corresponding negative training object binary; the negative training object binary is constructed by training the key object and the second detection object; Step S240d52: determining a first quotient between the first feature commonality metric value and the negative weight parameter, and determining a second quotient between the second feature commonality metric value and the negative weight parameter; Step S240d53: performing a power operation on the first quotient to obtain a first operation value, and performing a power operation on the second quotient to obtain a second operation value; Step S240d54: adding the second operation values ​​to obtain a combined operation value; Step S240d55: Based on the first operation value and the combined operation value, the parameter variables in the basic feature matching network are optimized to obtain the feature matching network.

[0108] First, in step S240d51, the computer system determines the pair of active training objects T to be optimized for the basic feature matching network. d The negative weight parameter of the focusing degree of the corresponding negative training object binary (which can be understood as a negative sample). In the low-altitude aircraft detection scenario, the positive training object binary T d Taking (aircraft, airport runway) as an example, the negative training object binary is constructed by training the key object (aircraft) and other objects that are different from the second detected object (airport runway) in the current positive training object binary. For example, (aircraft, mountain range) can be used as a negative training object binary because there is no matching relationship between aircraft and mountain range like there is with runways. The negative weight parameter is determined to adjust the degree of attention the network pays to the negative training object binary during the training process. This parameter can be set according to different strategies. One possible way is based on the statistical information of the data. Assuming that in the training data set, the ratio of the number of negative training object binary groups to the number of positive training object binary groups is r, the negative weight parameter can be determined based on this ratio. ,For example If the number of negative training object bigrams is relatively large, The value of will be relatively small, so that the network will not be overly affected by the negative training object binary during the training process; on the contrary, if the number of negative training object binary is small, The value of will be relatively large to increase the network's attention to negative training object binary groups.

[0109] Next, in step S240d52, the computer system determines a first quotient between the first feature commonality metric value and the negative weight parameter, and determines a second quotient between the second feature commonality metric value and the negative weight parameter. Continuing with the above example, let the first feature commonality metric value (the commonality metric value between the aircraft and the airport runway) be , the negative weight parameter is , then the first quotient is ; Let the second feature commonality measure (the commonality measure between the aircraft and other objects, such as mountains) be , then the second quotient is These quotients are calculated to comprehensively consider the relationship between the positive training object binary and the negative training object binary in the subsequent steps and adjust this relationship according to the negative weight parameter.

[0110] Then, in step S240d53, the computer system performs a power operation on the first quotient to obtain a first operation value, and performs a power operation on the second quotient to obtain a second operation value. For example, assuming that the exponent of the power operation is p (p can be determined based on experience or experiment, such as p = 2), then the first operation value is , the second operation value is The purpose of the power operation is to further adjust the values ​​of the first and second quotients so that certain relationships are more prominent or suppressed in subsequent calculations. Through the power operation, the influence of different commonality measures in the overall calculation can be changed. For example, if p > 1, the larger quotient value will become larger after the power operation, and the smaller quotient value will become smaller, thereby enhancing the difference between them; if 0 < p < 1, it will have the opposite effect and reduce the difference between them.

[0111] Next is step S240d54, where the computer system adds the second operation values ​​to obtain a combined operation value. That is, the second operation values ​​corresponding to all previously obtained negative training object tuples are added together. Assuming that in addition to the negative training object tuple (aircraft, mountain), there are other negative training object tuples such as (aircraft, river), and after calculating their corresponding second operation values ​​respectively, these values ​​are added to obtain a combined operation value. This combined operation value integrates the adjusted relationship information between the aircraft and all negative training objects, and it will be used together with the first operation value (the adjusted relationship information between the aircraft and the positive training objects) in the subsequent steps to optimize the parameters of the basic feature matching network.

[0112] Finally, step S240d55 is performed to optimize the parameters in the basic feature matching network based on the first operation value and the combined operation value to obtain the feature matching network. The computer system can use a gradient-based optimization algorithm to optimize the parameters. First, a loss function L is constructed based on the relationship between the first operation value and the combined operation value. For example, it can be defined as , where g is a function defined according to the optimization objective, such as , this function indicates that we hope that the first operation value (the relationship between the aircraft and the positive training object) is as large as possible relative to the merged operation value (the relationship between the aircraft and the negative training object), so that the network can better distinguish the feature differences between the positive training object and the negative training object.

[0113] The computer system calculates the gradient of the loss function L for the parameters (such as weight w and bias b) in the basic feature matching network, and then updates the parameters according to the gradient descent algorithm. By continuously using different positive training object pairs and corresponding negative training object pairs to perform the above calculations and parameter update operations, the parameters of the basic feature matching network are gradually optimized, and finally a feature matching network that can more accurately represent the matching relationship between objects is obtained.

[0114] Although the determination of negative weight parameters can be based on statistical information of the data, this method may have limitations. For example, determining negative weight parameters based solely on the ratio of the number of negative training object binary groups to positive training object binary groups may not fully reflect the actual impact of different negative training object binary groups on network training. Different types of negative training object binary groups may have different importance. For example, the two negative training object binary groups (aircraft, mountains) and (aircraft, large buildings) may have different impacts on network training due to differences in the spatial relationship between mountains and large buildings and aircraft and the degree of interference with aircraft detection. In order to determine the negative weight parameters more accurately, other factors can be considered, such as the distance measurement between the negative training object binary group and the positive training object binary group in the feature space. The distance between the objects in the negative training object binary group and the objects in the positive training object binary group in the feature implicit representation space output by the feature matching network can be calculated, and the negative weight parameters can be adjusted according to this distance. For example, if the objects in a negative training object binary are closer to the objects in the positive training object binary in the feature implicit representation space, it means that the negative training object binary has a greater interference in the network's distinction between positive and negative relationships, and a larger negative weight parameter should be given.

[0115] Secondly, the choice of exponent p in the power operation has an important impact on the result. If p is not chosen properly, the first and second operation values ​​may be unreasonably enlarged or reduced during the calculation process. For example, if p is too large, some small differences in commonality metrics may become too large after the power operation, thereby excessively affecting the training of the network; if p is too small, the difference between positive and negative training objects may not be effectively highlighted. In order to select a suitable p value, a cross-validation method can be used. Divide the training data set into multiple subsets, use different p values ​​on different subsets for network training, and then evaluate the performance of the network on the validation data set, and select the p value that gives the best network performance as the final exponent.

[0116] When calculating the merged operation value, since the second operation values ​​corresponding to multiple negative training object binary groups are added, there may be an imbalance problem between different negative training object binary groups. For example, some types of negative training object binary groups may appear more frequently in the training data set, while others may appear less frequently. This imbalance may cause the influence of some negative training object binary groups to be overly amplified or reduced during the addition process. In order to solve this problem, the second operation value corresponding to each negative training object binary group may be weighted, and the weight may be determined based on the frequency of occurrence of the negative training object binary group in the training data set or its importance. For example, a smaller weight is given to a negative training object binary group with a higher frequency of occurrence, and a larger weight is given to a negative training object binary group with a lower frequency of occurrence but more representative or disruptive.

[0117] In addition, when constructing the loss function L, although some example functions are given, different function forms may have different effects on the network training effect. Choosing a suitable loss function requires considering the specific task requirements and data characteristics. For example, if you want to emphasize the difference between positive training objects and negative training objects, you can choose some loss functions with strong discrimination, such as the contrastive loss function. When calculating the loss, the contrastive loss function not only considers the difference between positive training objects and negative training objects, but also considers the similarity between objects of the same type. However, the calculation of the contrastive loss function may be relatively complex and requires more computing resources. Therefore, it is necessary to balance the computational complexity and training effect according to the actual situation.

[0118] As an implementation mode, step S240d55, based on the first operation value and the combined operation value, optimizes the parameter variables in the basic feature matching network to obtain the feature matching network, including: Step S240d551: determining a third quotient between the first operation value and the combined operation value, performing a logarithmic operation on the third quotient to obtain a logarithmic operation value; Step S240d552: Multiply the logarithmic operation value by -1 to obtain the active training object binary group T d corresponding losses; Step S240d553: Add the losses corresponding to a number of active training object binary groups to obtain the network loss corresponding to the basic feature matching network; Step S240d554: Based on the network loss, the parameters in the basic feature matching network are optimized to obtain the feature matching network.

[0119] First, in step S240d551, the computer system determines the third quotient between the first operation value and the combined operation value, performs logarithmic operation on the third quotient, and obtains a logarithmic operation value. In the low-altitude aircraft detection scenario, continue to explain with the example mentioned above, assuming that the first operation value is (in is the first characteristic commonality measure, is a negative weight parameter, and p is the exponent of the power operation), assuming that the combined operation value obtained after step S240d54 is (in is a second feature commonality measure associated with different pairs of negative training objects). The computer system calculates the third quotient as , and then perform logarithmic operation on z to obtain the logarithmic value .

[0120] The role of the logarithmic operation here is to convert the quotient into a form that is more convenient for subsequent processing and analysis. The logarithmic function has a unique mathematical property that can compress a large range of values ​​into a relatively small, more manageable range and can highlight the proportional relationship between values. For example, in the case of large differences in values, the logarithmic operation can make these differences more consistent with the needs of the optimization algorithm to a certain extent, which helps to adjust the network parameters more accurately in the future.

[0121] Next, in step S240d552, the computer system multiplies the logarithmic operation value by -1 to obtain the active training object binary group T d The corresponding loss. That is, the calculated loss value is .

[0122] The calculation of this loss value is based on the results of the previous steps and plays a key role in the entire network optimization process. The loss value is a measure of the difference between the network output and the expected output. In this scenario, the loss value obtained by the above calculation reflects the extent to which the relationship between the positive training object binary and the negative training object binary under the current network parameters does not meet expectations. If the loss value is large, it means that the network performs poorly in distinguishing between positive and negative training objects, and the network parameters need to be adjusted; if the loss value is small, it means that the network performs well in this regard, but further optimization may still be needed to achieve better performance.

[0123] Then, in step S240d553, the computer system adds the losses corresponding to the several active training object binary groups to obtain the network loss corresponding to the basic feature matching network. Assume that there are multiple active training object binary groups, which are , each active training object binary has its corresponding loss value calculated according to the previous steps The computer system adds these loss values ​​to get the network loss .

[0124] This network loss integrates the information of all positive training object pairs, and it comprehensively reflects the network's processing effect on the relationship between all positive training objects and negative training objects under the current network parameters. By calculating the network loss, the performance of the network can be evaluated as a whole, and a unified goal can be provided for subsequent parameter optimization.

[0125] Finally, step S240d554 is performed to optimize the parameters in the basic feature matching network based on the network loss to obtain the feature matching network. The computer system uses a gradient-based optimization algorithm to optimize the parameters. For example, the gradient descent algorithm is used. Its basic principle is to calculate the gradient of the network loss to the network parameters (such as weight w and bias b, etc.), and then update the parameters in the opposite direction of the gradient to reduce the network loss. Let the network loss function be L. For a certain parameter in the network, the computer system first calculates the gradient. This gradient represents the rate at which the network loss changes with the change of the parameter.

[0126] As an implementation mode, the spatial relationship includes a first spatial relationship; step S400, walking the image knowledge base to obtain a spatial relationship for establishing a connection between a source detection object and a termination detection object, and generating an association link including the source detection object, the spatial relationship and the termination detection object, including: Step S410: acquiring a first set of detected objects having a connection relationship with a source detected object in an image knowledge base; Step S420: if there is a termination detection object in the first detection object set, then obtaining a first spatial relationship indicating a connection relationship between the source detection object and the termination detection object in the image knowledge base; Step S430: constructing the source detection object, the first spatial relationship and the termination detection object in sequence to obtain a first association link; Step S440: clearing the terminated detection objects in the first detection object set to obtain a second detection object set; Step S450: Walk the second detection object set to construct an associated link including the first associated link.

[0127] First, in step S410, the computer system obtains a first set of detection objects that have a connection relationship with the source detection object in the image knowledge base. In the low-altitude aircraft detection scenario, it is assumed that the source detection object is an aircraft, and the image knowledge base contains the relationship information between various detection objects. In this scenario, the aircraft has a connection relationship with multiple detection objects, such as an airport runway, a communication base station, an air traffic control tower, etc. The computer system gathers all the detection objects that have a connection relationship with the aircraft by querying the connection relationship records related to the aircraft in the image knowledge base to form a first set of detection objects. The acquisition of this set may involve traversal and query operations on the relationship data structure in the knowledge base. For example, if the image knowledge base uses a graph structure to represent the detection objects and their relationships, the computer system can start from the node corresponding to the aircraft and find all the nodes connected to it along the edges representing the connection relationship. The detection objects represented by these nodes constitute the first set of detection objects.

[0128] Next is step S420. If there is a termination detection object in the first detection object set, a first spatial relationship indicating the connection relationship between the source detection object and the termination detection object is obtained in the image knowledge base. Continuing with the above example, assuming that the termination detection object is an airport runway, when the computer system finds that the airport runway is in the first detection object set, it will search the image knowledge base for information describing the connection relationship between the aircraft and the airport runway. This relationship is the first spatial relationship. For example, this spatial relationship may be "above and close", indicating that the aircraft is above the airport runway and is in a spatial position close to the runway. This spatial relationship is obtained through a predefined relationship description in the knowledge base or through a specific spatial relationship analysis algorithm. For example, the knowledge base may store information such as the distance range and relative height range between the aircraft and the runway to determine this spatial relationship, or it may be obtained by analyzing the relationship between the flight trajectory of the aircraft (if there is relevant information in the knowledge base) and the layout of the runway.

[0129] Then, in step S430, the computer system constructs the source detection object, the first spatial relationship, and the termination detection object in order to obtain a first association link. According to the previous example, a first association link such as (aircraft, above and close to, airport runway) can be constructed. This association link intuitively represents the connection between the source detection object and the termination detection object through a specific spatial relationship, which is the basis for the subsequent construction of more complex association links.

[0130] Next is step S440, where the computer system clears the terminated detection objects in the first detection object set to obtain a second detection object set. In the previous example, the first detection object set includes airport runways, communication base stations, air traffic control towers, etc. After the airport runways (terminated detection objects) are cleared, the remaining communication base stations, air traffic control towers, etc. constitute the second detection object set. The purpose of this step is to avoid repeated consideration of terminated detection objects that have already constructed links in the subsequent link construction process, thereby gradually constructing more comprehensive and complex associated links.

[0131] Finally, step S450 is performed to walk the second set of detected objects to construct an associated link including the first associated link. This step is a recursive link construction process, which aims to mine other relationships related to the source detected object and the terminal detected object from the remaining detected objects to construct a more complete associated link.

[0132] In this process, the computer system can use a depth-first search or breadth-first search algorithm to walk the second detection object set. Taking depth-first search as an example, the computer system starts from a detection object (such as a communication base station) in the second detection object set and repeats operations similar to steps S410-S440. That is, first obtain a set of detection objects that have a connection relationship with the communication base station (assuming that this set includes other aircraft, ground buildings, etc.), and then check whether there is a termination detection object (airport runway) in this set. If so, obtain the spatial relationship between the communication base station and the airport runway (assuming that it is "within the signal coverage range"), construct a link such as (communication base station, signal coverage range, airport runway), and remove the airport runway from this set to obtain a new set, and continue to walk the new set. If the termination detection object is not found in this process, continue to start a new search process from other detection objects in this set until all possible links are constructed.

[0133] In practical applications, the relationship information in the image knowledge base may be incomplete or uncertain. For example, for some emerging low-altitude aircraft or special detection scenarios, there may not be enough detailed relationship information in the knowledge base. In this case, the computer system may not be able to accurately obtain the spatial relationship between certain source detection objects and terminal detection objects. In order to solve this problem, data mining and knowledge reasoning techniques can be used. Data mining technology can discover new relationship patterns from a large amount of historical detection data. For example, by analyzing a large amount of flight data of aircraft in a special area, new spatial relationships between certain new aircraft and specific facilities can be discovered. Knowledge reasoning technology can perform logical reasoning based on existing relationship information. For example, based on the signal interaction relationship between the aircraft and the communication base station and the relationship between the communication base station and the surrounding terrain, the indirect relationship between the aircraft and the surrounding terrain can be derived.

[0134] Secondly, when building associative links, you may encounter the problem of ambiguous spatial relationships. For example, the spatial relationship of "close" may be understood differently in different scenarios or between different objects. For large aircraft and small airport runways, "close" may mean within a few kilometers; while for small aircraft and large airport runways, this range may be different. In order to deal with this ambiguity, the computer system can use context-aware technology. The computer system dynamically determines the specific meaning of the spatial relationship based on the current detection scene (such as the type of aircraft, the size of the runway, the surrounding environment, etc.). For example, a proportional relationship can be established based on the size of the aircraft and the length of the runway, and the specific distance range of "close" can be determined based on this proportional relationship.

[0135] In addition, when walking the second set of detected objects, the problem of too large a search space may be faced. With the increase in the number of detected objects and the complexity of the relationships, searching all possible associated links may consume a lot of computing resources and time. In order to improve efficiency, a heuristic search algorithm can be used. The heuristic search algorithm uses some heuristic information to guide the search direction and prioritizes those detected objects that are more likely to build valid associated links. For example, the search priority can be determined based on the importance of the detected object (such as facilities related to aircraft flight safety are more important) or the historical association frequency with the source detected object and the termination detected object, thereby reducing unnecessary search operations and improving the efficiency of link construction.

[0136] In addition, in the process of building associative links, the transitivity of the relationship also needs to be considered. For example, if the aircraft has a signal connection relationship with the communication base station, and the communication base station has a data transmission relationship with the ground control center, then there is an indirect relationship between the aircraft and the ground control center. The computer system needs to be able to recognize and process the transitivity of this relationship in order to build a more comprehensive and accurate associative link. This may require a deeper analysis and reasoning of the relationships in the knowledge base, such as by constructing the transitive closure of the relationship graph to represent all possible indirect relationships.

[0137] As an implementation mode, step S450, walking the second detection object set to construct an associated link including the first associated link, includes: Step S451: if in the image knowledge base, the second detection object set does not have a detection object having a connection relationship, then the first association link is determined as an association link; Step S452: If the second detection object set has detection objects with a connection relationship in the image knowledge base, the detection objects with a connection relationship in the second detection object set are determined as the third detection object set; the third detection object set includes the detection objects U e , e ≥ 1, and e is not greater than the total number of detection objects in the third detection object set; Step S453: Obtain the detected object U in the image knowledge base e A fourth detection object set including a connection relationship; the fourth detection object set does not include the source detection object; Step S454: Walk the fourth detection object set to construct an associated link including the first associated link.

[0138] First, in step S451, if the second detection object set does not have a detection object with a connection relationship in the image knowledge base, the first association link is determined as the association link. In the low-altitude aircraft detection scenario, for example, the first association link is (aircraft, above and close to, airport runway). Assume that after the previous cleaning operation, only some isolated detection objects are left in the second detection object set, and these objects have no connection relationship with any other objects in the image knowledge base. In this case, it means that from the perspective of current information and relationships, it is impossible to construct more links related to the source detection object (aircraft) and the terminal detection object (airport runway), then the first association link (aircraft, above and close to, airport runway) is determined as the final association link. This is a simplest case judgment based on existing relationships, which avoids invalid searches and calculations when there are no more valid relationships to be mined.

[0139] Next is step S452. If the second detection object set has detection objects with connection relationships in the image knowledge base, the detection objects with connection relationships in the second detection object set are determined as the third detection object set. Continuing with the aircraft detection scene as an example, assuming that the second detection object set includes a communication base station and an air traffic control tower, and in the image knowledge base, there is a connection relationship (e.g., a communication connection relationship) between the communication base station and the air traffic control tower, then the computer system will determine the communication base station and the air traffic control tower as the third detection object set. The purpose of this step is to extract the detection objects with connection relationships separately so that more in-depth relationship mining and link building operations can be performed on these objects later.

[0140] Then, in step S453, in the image knowledge base, obtain the image data corresponding to the detected object U. e A fourth set of detection objects including connection relationships. Here, the detection objects U e is any detection object in the third detection object set, such as U e The computer system will search the image knowledge base for other detection objects that are connected to the communication base station. Assuming that these objects include nearby buildings, other communication equipment, etc., these objects constitute the fourth detection object set. This process is similar to the operation of obtaining the detection object set that is connected to the source detection object in the previous step, and both determine the relevant detection object set by querying the relationship information in the image knowledge base.

[0141] Finally, step S454 is performed to walk the fourth detection object set and construct an associated link including the first associated link. This step is a recursive exploration process, which aims to further explore the relationship between the source detection object (aircraft) and the terminal detection object (airport runway) from the fourth detection object set, so as to construct a more complex associated link.

[0142] In this process, the computer system can use a graph search algorithm to realize the wandering of the fourth detection object set. For example, using a depth-first search algorithm, starting from a detection object (such as a nearby building) in the fourth detection object set, repeat operations similar to the previous steps. The computer system first checks whether this detection object has a direct connection relationship with the termination detection object (airport runway). If so, it obtains the spatial relationship between them (assuming that there is a spatial relationship of "located around the runway" between the building and the airport runway), and then constructs a new link, such as (building, located around the runway, airport runway). If there is no direct connection relationship, continue to check whether this detection object has a connection relationship with other detection objects in the fourth detection object set, and then repeat the above process, continuously construct new links and clear related detection objects from the set until all possible links are constructed or a certain stop condition is reached (such as the search depth reaches a preset value or no new connection relationship can be found).

[0143] When obtaining the fourth set of detection objects, the problem of too large a search range may be encountered. As the number of detection objects in the knowledge base increases, there may be a large number of detection objects that have a connection relationship with a certain detection object (such as a communication base station), which will cause the fourth set of detection objects to be too large, thereby increasing the computational complexity and time cost of subsequent walks and link construction. In order to deal with this problem, the computer system can use pruning technology. For example, preliminary screening is performed based on the attributes of the detection objects and the correlation with the source detection objects and the termination detection objects, and only those detection objects that are more relevant to the current task are retained. For example, if the focus is on the relationship between the aircraft and the airport runway, for those detection objects that are too far away from the aircraft and the runway or have little functional relevance (such as ordinary buildings in the distance that are not related to flight), they can be excluded when constructing the fourth set of detection objects, thereby narrowing the search range.

[0144] In addition, when walking and constructing links for the fourth set of detection objects, link conflicts or repeated construction may occur. For example, due to the multi-path nature of the relationships in the knowledge base, the same link may be constructed through different paths. In order to avoid this situation, the computer system can establish a link cache mechanism. In the process of link construction, each time a new link is constructed, first check whether the link already exists in the cache. If it does, it will not be constructed repeatedly. At the same time, for links that may conflict (such as two links that contradict each other in describing the relationship between the same pair of detection objects), a choice can be made by comparing the reliability of the links or based on more contextual information. For example, if one link is built based on more accurate sensor data and the other link is built based on more speculative information, the former is preferred.

[0145] As an implementation mode, the spatial relationship further includes a second spatial relationship and a third spatial relationship; Step S454, walking the fourth detection object set, constructing an associated link including the first associated link, including: Step S4541: If the fourth detection object set contains a termination detection object, then in the image knowledge base, obtain a source detection object and a detection object U e and a second spatial relationship for indicating the connection relationship between the detection object U e and a third spatial relationship for terminating the connection relationship between the detected objects; Step S4542: The source detection object, the second spatial relationship, the detection object U e , the third spatial relationship and the termination detection object, and construct a second association link according to the sequence; Step S4543: clearing the terminated detection objects in the fourth detection object set to obtain a fifth detection object set; Step S4544: Walk the fifth detection object set to construct an association link including the first association link and the second association link.

[0146] First, in step S4541, if there is a termination detection object in the fourth detection object set, then in the image knowledge base, a source detection object and a detection object U are obtained. e and a second spatial relationship for indicating the connection relationship between the detection object U e In the low-altitude aircraft detection scenario, assuming that the source detection object is an aircraft, the first association link is (aircraft, above and close to, airport runway), and the fourth detection object set includes nearby buildings, other communication equipment, etc. If there is an airport runway (termination detection object) in this set, assuming that the detection object U e is a nearby building. The computer system will search the image knowledge base for the connection between the aircraft and the building. For example, if the aircraft is found to be within the visible range of the building, this is the second spatial relationship; at the same time, the connection between the building and the airport runway is searched. For example, if the building is located at a specific position of the airport runway (such as on the side of the runway extension line), this is the third spatial relationship. The process of the computer system obtaining these spatial relationships may involve querying the predefined spatial relationship descriptions in the knowledge base, or determining them through spatial analysis algorithms based on the data in the knowledge base. For example, for the visible range relationship, the position, altitude, heading of the aircraft and the coordinates, height and shape of the building can be used to determine whether the aircraft is within the visible range of the building through geometric calculations; for the orientation relationship, the relative azimuth can be calculated based on the coordinate information of the airport runway and the building.

[0147] Next, in step S4542, the computer system stores the source detection object, the second spatial relationship, the detection object U e , the third spatial relationship and the termination detection object, and the second association link is constructed in sequence. According to the previous example, a second association link such as (aircraft, within the visible range, building, located on the side of the runway extension line, airport runway) can be constructed. This second association link further enriches the association relationship between the source detection object (aircraft) and the termination detection object (airport runway). By introducing the intermediate detection object U e (buildings) and their spatial relationship with aircraft and runways, providing a more comprehensive description of the object relationships in the entire detection scene.

[0148] Then, step S4543 is performed, in which the computer system clears the terminated detection objects in the fourth detection object set to obtain the fifth detection object set. For example, in the previous example, the fourth detection object set includes nearby buildings, other communication equipment, etc., among which the airport runway is the terminated detection object. After the airport runway is cleared, the remaining nearby buildings, other communication equipment, etc. constitute the fifth detection object set. The purpose of this step is to avoid repeated consideration of the terminated detection objects that have already constructed the link in the subsequent link construction process, so that the link construction can develop in an orderly manner in a more complex and comprehensive direction.

[0149] Finally, step S4544 is performed to walk the fifth detection object set and construct an associated link including the first associated link and the second associated link. The computer system can adopt a method similar to the method of walking the detection object set in the previous step, such as a depth-first search or a breadth-first search algorithm. Taking the depth-first search as an example, starting from a detection object (such as other communication equipment) in the fifth detection object set, operations similar to steps S4541-S4543 are repeated. That is, first check whether there is a connection relationship between this detection object and the remaining other detection objects. If so, check whether there is an association with the source detection object (aircraft) or the termination detection object (airport runway). If there is an association relationship, obtain the corresponding spatial relationship and construct a new link, then clear the relevant detection objects, and continue to walk the new detection object set until all possible links are constructed or a certain stop condition is reached (such as the search depth reaches a preset value or no new connection relationship can be found).

[0150] When constructing the second association link (step S4542), spatial relationship conflicts may occur. For example, due to data from different sources or different analysis methods, contradictory spatial relationships between the same pair of detected objects may be obtained. For example, according to one analysis method, the building is found to be on one side of the airport runway, but according to another analysis method, the opposite result may be obtained. In order to solve this problem, the computer system can adopt relationship fusion and conflict resolution strategies. Weights can be assigned to data or analysis methods from different sources, and the spatial relationship can be comprehensively judged based on the weights. For example, if the data from a certain source is more reliable (such as data from a high-precision sensor), a higher weight is given. At the same time, contextual information can be used to assist in judgment, such as adjusting the spatial relationship by considering factors such as the flight direction of the aircraft and the layout of the surrounding environment.

[0151] When clearing the terminated detection object in step S4543, it is necessary to accurately identify the terminated detection object. In a complex detection scenario, there may be multiple similar detection objects, which may easily lead to misjudgment. For example, in the case of multiple airport runways or facilities with runway-like shapes, objects that should not be cleared may be mistakenly cleared. In order to avoid this situation, the computer system can use object feature recognition and matching technology. The terminated detection object is accurately identified by analyzing multiple features of the detection object (such as shape, size, relationship with other specific objects, etc.). For example, for an airport runway, it can be accurately judged whether it is a true terminated detection object based on its standard shape and size range, as well as its specific relationship with other airport facilities (such as a control tower).

[0152] When walking and building links for the fifth set of detected objects (step S4544), the problem of excessive consumption of computing resources may be faced. As the number of detected objects increases and the complexity of link construction increases, searching for all possible associated links may take up a lot of computing resources and time. In order to improve efficiency, a heuristic search strategy can be adopted. For example, the search priority is determined based on the possibility of association between the detected object and the source detected object and the terminal detected object. If a detected object has a more direct association with an aircraft or an airport runway (such as historical data showing that it often interacts with the aircraft or the runway), it is searched and linked to the object first. In addition, distributed computing technology can be used to distribute search tasks to multiple computing units (such as multiple processors or computing nodes) and perform them simultaneously to improve computing efficiency.

[0153] An embodiment of the present application provides a computer system, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0154] Figure 2A hardware entity diagram of a computer system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

[0155] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A low-altitude aircraft panoramic detection method based on multi-modal monitoring, characterized in that: include: Determining a number of key objects in a multimodal fusion image, wherein the multimodal fusion image is obtained by fusing images detected by an infrared thermal imaging detector of a low-altitude aircraft and a high-definition color visible light image sensor; Acquire an image knowledge base for indicating object relationships between a plurality of detection objects, and acquire detection objects respectively matched by the plurality of key objects from the plurality of detection objects as matching detection objects; Among a plurality of matching detection objects, exploring one by one to obtain detection object tuples including a matching detection object as a source detection object and a matching detection object as a termination detection object; the source detection object is different from the termination detection object; Walking the image knowledge base to obtain a spatial relationship for establishing a connection between the source detection object and the termination detection object, and generating an association link including the source detection object, the spatial relationship and the termination detection object; Image scene understanding is performed on the multimodal fusion image based on the associated links to obtain a detection and analysis result of the multimodal fusion image.

2. The method according to claim 1, characterized in that The plurality of key objects include key object P a , a≥1, and a is not greater than the total number of the key objects; The obtaining, from the plurality of detection objects, the detection objects respectively matched by the plurality of key objects as matching detection objects, includes: Determine the key object P a an object co-occurrence frequency with each of the plurality of detected objects; Among the plurality of object co-occurrence frequencies, s object co-occurrence frequencies are obtained; s≥1, and s is less than the total number of the plurality of object co-occurrence frequencies; the s object co-occurrence frequencies are greater than other object co-occurrence frequencies; the other object co-occurrence frequencies include object co-occurrence frequencies among the plurality of object co-occurrence frequencies other than the s object co-occurrence frequencies; Among the plurality of detected objects, the detected object corresponding to the co-occurrence frequency of the s objects is determined as the key object P a The pending detection object; Determine the key object P a and the commonality metric value between the to-be-determined detection object, and obtaining the maximum commonality metric value among the commonality metric values; The pending detection object containing the maximum commonality metric value is determined as the key object P a The matching detection object.

3. The method according to claim 2, characterized in that The determination of the key object P a The object co-occurrence frequency between each of the plurality of detected objects respectively includes: According to the moving step of the sliding control, at the key object P a Walk through the key object P a corresponding key local features; the number of features of the key local features is equal to the moving stride of the sliding control; Obtaining local features of objects corresponding to the plurality of detected objects; the plurality of detected objects include the detected object M b , b≥1, and b is not greater than the total number of the plurality of detected objects; the plurality of object local features include the detected object M b The corresponding local features of the object; The key local features and the detection object M b The corresponding object local features are analyzed for object commonality to obtain the key object P a With the detection object M b The co-occurrence frequency of objects between .

4. The method according to claim 3, characterized in that: The total number of the key local features is t, and the t key local features include the key local feature Q c , t and c are both greater than 0, and c is not greater than t; the key local features and the detection object M b The corresponding object local features are analyzed for object commonality to obtain the key object P a With the detection object M b The co-occurrence frequency of objects between, including: The key local feature Q c With the detection object M b The corresponding local features of the object are matched; If the detected object M b The corresponding object local features have the same key local features Q c The same object local features will be the same as the key local features Q c The total number of local features of the same object is determined as the key local feature Q c The corresponding object co-occurrence frequency; The sum of the object co-occurrence frequencies corresponding to the t key local features is determined as the key object P a With the detection object M b The frequency of object co-occurrence between The determination of the key object P a The commonality measurement value between the target object to be detected includes: Call the feature matching network to generate the key object P a A first feature implicit representation of the object to be detected, and a second feature implicit representation of the object to be detected; The commonality measure between the first feature implicit representation and the second feature implicit representation is determined as the key object P a a commonality measurement value with the pending detection object; Before calling the feature matching network, the method further includes: Acquire a number of active training object binary groups; the number of active training object binary groups include active training object binary groups T d , d≥1, and d is not greater than the total number of the plurality of active training object binary groups; Generate the active training object binary group T through the basic feature matching network d The first inference feature implicit representation of the training key object in T, and the positive training object bigram T d The second inference feature of the first detected object in the image knowledge base is implicitly represented; the first detected object belongs to the image knowledge base; The first inference feature implicit representation and the second inference feature implicit representation are determined as the active training object binary group T d The inference feature of implicitly represents bigrams; Based on the inference feature implicit representation tuples respectively corresponding to the plurality of active training object tuples, the parameter variables in the basic feature matching network are optimized to obtain the feature matching network.

5. The method according to claim 4, characterized in that The step of implicitly representing binary groups based on the inference features corresponding to the binary groups of the plurality of active training objects respectively, optimizing the parameter variables in the basic feature matching network to obtain the feature matching network comprises: determining a first feature commonality measure between the first inferred feature implicit representation and the second inferred feature implicit representation; Among the plurality of active training object binary groups, obtain the active training object binary group T d a second detected object in the active training object binary outside the image knowledge base; the second detected object belongs to the image knowledge base; In the inference feature implicit representation tuples corresponding to the plurality of active training object tuples, the inference feature implicit representation corresponding to the second detection object is determined as a third inference feature implicit representation; determining a second feature commonality measure between the first inferred feature implicit representation and the third inferred feature implicit representation; Based on the first feature commonality metric value and the second feature commonality metric value, the parameter variables in the basic feature matching network are optimized to obtain the feature matching network.

6. The method according to claim 5, characterized in that The optimizing the parameter variables in the basic feature matching network based on the first feature commonality metric value and the second feature commonality metric value to obtain the feature matching network includes: Determine the method for optimizing the basic feature matching network for the active training object binary group T d a negative weight parameter of the focusing degree of the corresponding negative training object binary group; the negative training object binary group is constructed by the training key object and the second detection object; determining a first quotient between the first feature commonality measure and the negative weight parameter, and determining a second quotient between the second feature commonality measure and the negative weight parameter; Performing a power operation on the first quotient to obtain a first operation value, and performing a power operation on the second quotient to obtain a second operation value; Adding the second operation values ​​to obtain a combined operation value; Based on the first operation value and the combined operation value, the parameter variables in the basic feature matching network are optimized to obtain the feature matching network.

7. The method according to claim 6, characterized in that The optimizing the parameter variables in the basic feature matching network based on the first operation value and the combined operation value to obtain the feature matching network includes: determining a third quotient between the first operation value and the combined operation value, and performing a logarithmic operation on the third quotient to obtain a logarithmic operation value; Multiply the logarithmic operation value by -1 to obtain the active training object binary group T d corresponding losses; Adding the losses corresponding to the plurality of active training object binary groups respectively to obtain a network loss corresponding to the basic feature matching network; Based on the network loss, the parameters in the basic feature matching network are optimized to obtain the feature matching network.

8. The method according to claim 1, characterized in that: The spatial relationship includes a first spatial relationship; the walking the image knowledge base to obtain a spatial relationship for establishing a connection between the source detection object and the termination detection object, and generating an association link including the source detection object, the spatial relationship and the termination detection object, comprises: In the image knowledge base, obtaining a first set of detection objects having a connection relationship with the source detection object; If the first detection object set includes the termination detection object, acquiring the first spatial relationship indicating the connection relationship between the source detection object and the termination detection object in the image knowledge base; Constructing the source detection object, the first spatial relationship and the termination detection object in sequence to obtain a first association link; clearing the terminated detection objects in the first detection object set to obtain a second detection object set; If in the image knowledge base, the second detection object set does not have a detection object having a connection relationship, determining the first association link as an association link; If, in the image knowledge base, the second detection object set has detection objects with a connection relationship, the detection objects with a connection relationship in the second detection object set are determined as a third detection object set; the third detection object set includes detection objects U e , e≥1, and e is not greater than the total number of detection objects in the third detection object set; In the image knowledge base, obtain e A fourth detection object set including a connection relationship; the fourth detection object set does not include the source detection object; The fourth detection object set is traversed to construct an association link including the first association link.

9. The method according to claim 8, characterized in that The spatial relationship also includes a second spatial relationship and a third spatial relationship; and walking the fourth detection object set to construct an association link including the first association link includes: If the fourth detection object set includes the termination detection object, then in the image knowledge base, a sequence of information indicating the source detection object and the detection object U is obtained. e The second spatial relationship of the connection relationship between the two, and the detection object U e and the third spatial relationship of the connection relationship between the termination detection object; The source detection object, the second spatial relationship, the detection object U e , the third spatial relationship and the terminated detection object, and constructing a second association link in sequence; clearing the terminated detection objects in the fourth detection object set to obtain a fifth detection object set; The fifth detection object set is walked to construct an association link including the first association link and the second association link.

10. A computer system comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.

Images