Space-based and air-based multi-target detection and correlation method, device and equipment

Abstract

The application provides a space-based and air-based multi-target detection and correlation method, device and equipment, and relates to the technical field of remote sensing information processing. The method comprises the following steps: acquiring a space-based target set detected by space-based detection and an air-based remote sensing image; inputting the space-based target set and the air-based remote sensing image into a pre-trained target search network, extracting a space-based target correlation feature set corresponding to multiple space-based targets and air-based data basic features through a shared backbone network in the target search network; performing target detection on the air-based data basic features based on a target detection head in the target search network, determining an air-based target set in the air-based remote sensing image, and screening the space-based target correlation feature set from the air-based data basic features based on multiple air-based targets included in the air-based target set; and correlating the multiple space-based targets and the multiple air-based targets based on the similarity between the space-based target correlation feature set and the air-based target correlation feature set, so that the multi-target correlation efficiency can be improved.

Description

Technical Field

[0001] This application relates to the technical fields of remote sensing information processing, and in particular to a method, apparatus and equipment for multi-target detection and association based on space-based and air-based systems. Background Technology

[0002] In collaborative observation scenarios involving space-based (e.g., low-orbit satellites, high-orbit satellites) and air-based (e.g., drones, manned aircraft) systems, it is often necessary to detect and correlate multiple targets in the air-based system within the scenario, providing key technical support for fields such as national defense security monitoring, disaster emergency response, and marine target monitoring.

[0003] Currently, when performing multi-target detection and association in the airborne region, a separate process of "detect first, then associate" is usually adopted. That is, targets are first extracted from the airborne remote sensing image through target detection algorithms, and then associated with the targets detected by the airborne region through independent association algorithms, such as similarity matching algorithms.

[0004] However, the above method has the problem of repeated extraction of detection features and associated features, resulting in low efficiency of multi-target association. Summary of the Invention

[0005] This application provides a method, apparatus, and device for multi-target detection and association based on space-based and air-based systems, which solves the problem of low association efficiency caused by repeated extraction of detection features and association features in existing multi-target detection and association methods based on space-based and air-based systems, thereby improving the multi-target association efficiency.

[0006] This application provides a multi-target detection and association method based on space-based and air-based systems, characterized by comprising:

[0007] Acquire a set of space-based targets detected by space-based methods and space-based remote sensing images acquired by air-based methods; wherein the set of space-based targets includes multiple space-based targets;

[0008] Both the space-based target set and the airborne remote sensing image are input into a pre-trained target search network. The shared backbone network of the target search network extracts space-based target association feature sets corresponding to multiple space-based targets, and extracts basic airborne data features from the airborne remote sensing image. Based on the target detection head in the target search network, target detection is performed on the basic airborne data features to determine the airborne target set in the airborne remote sensing image. Based on the multiple airborne targets included in the airborne target set, a set of airborne target association features is selected from the basic airborne data features. The target search network is obtained by training an initial search network based on multiple spaceborne triplet samples.

[0009] Based on the similarity between the space-based target association feature set and the air-based target association feature set, the multiple space-based targets are associated with the multiple air-based targets.

[0010] According to the multi-target detection and association method based on space-based and airborne targets provided in this application, the step of performing target detection on the basic features of the airborne data based on the target detection head in the target search network to determine the set of airborne targets in the airborne remote sensing image includes:

[0011] Based on the target detection head, target detection is performed on the basic features of the space-based data to obtain multiple prediction candidate boxes in the space-based remote sensing image, as well as the confidence level corresponding to each prediction candidate box;

[0012] Based on the confidence level corresponding to each predicted candidate box, the set of airborne targets in the airborne remote sensing image is determined.

[0013] According to the multi-target detection and association method based on space-based and air-based targets provided in this application, the method for associating multiple space-based targets with multiple air-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set includes:

[0014] Based on the similarity between the space-based target association feature set and the air-based target association feature set, a corresponding association strategy cost augmentation matrix is ​​constructed; wherein, the association strategy cost augmentation matrix is ​​used to characterize the probability of association between space-based targets and air-based targets;

[0015] Construct an augmented association plan matrix corresponding to the plurality of space-based targets and the plurality of airborne targets; wherein, the augmented association plan matrix is ​​used to perform spatiotemporal consistency constraints on the plurality of space-based targets and the plurality of airborne targets;

[0016] Based on the association plan augmented matrix and the association strategy cost augmented matrix, the multiple space-based targets and the multiple air-based targets are associated.

[0017] According to the multi-target detection and association method based on space-based and airborne targets provided in this application, the step of constructing a corresponding association strategy cost augmentation matrix based on the similarity between the space-based target association feature set and the airborne target association feature set includes:

[0018] Based on the similarity between pairs of targets in the space-based target association feature set and the air-based target association feature set, a corresponding target similarity matrix is ​​constructed;

[0019] Based on the target similarity matrix, the corresponding association strategy cost augmentation matrix is ​​constructed.

[0020] According to the multi-target detection and association method based on space-based and air-based systems provided in this application, the association plan augmented matrix is ​​used to perform spatiotemporal consistency constraints on the multiple space-based targets and the multiple air-based targets, including:

[0021] based on Spatiotemporal consistency constraints are applied to the plurality of space-based targets and the plurality of airborne targets;

[0022] Wherein, N represents the number of the plurality of space-based targets, and M represents the number of the plurality of air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the augmented matrix of the correlation plan.

[0023] According to the multi-target detection and association method based on space-based and air-based systems provided in this application, the association of multiple space-based targets with multiple air-based targets based on the association plan augmented matrix and the association strategy cost augmented matrix includes:

[0024] based on Associating the plurality of space-based targets with the plurality of air-based targets to satisfy the following conditions: The correlation results between the multiple space-based targets and the multiple air-based targets are obtained;

[0025] in, This represents the augmented matrix of the association plan, where N represents the number of the multiple space-based targets, and M represents the number of the multiple air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the correlation plan augmentation matrix. This represents the cost augmentation matrix of the association strategy. This indicates the probability that a space-based target in the i-th row of the association strategy cost augmentation matrix is ​​associated with a space-based target in the j-th column.

[0026] According to the multi-target detection and association method based on space-based and air-based systems provided in this application, each space-based triplet sample includes an air-based remote sensing image sample, a set of positive space-based target samples corresponding to the air-based remote sensing image sample, and a set of negative space-based target samples. The target search network is trained on an initial search network based on the multiple space-based triplet samples, including:

[0027] For each sky-based triplet sample, perform the following operations:

[0028] The initial search network is used to determine the similarity between a space-based target sample and a first-level space-based target sample in the space-based remote sensing image samples, as well as the similarity between the space-based target sample and the second-level space-based target sample; wherein, the first-level space-based target sample is the space-based target sample in the positive space-based target sample set that corresponds to the space-based target sample, and the second-level space-based target sample is the space-based target sample in the negative space-based target sample set that corresponds to the space-based target sample;

[0029] Based on the similarity between the airborne target sample and the first spaceborne target sample, and the similarity between the airborne target sample and the second spaceborne target sample, a target association loss function is constructed;

[0030] Based on the predicted target category to which the spatial target sample belongs and the predicted candidate box corresponding to the spatial target sample, a target detection loss function is constructed;

[0031] Based on the target association loss function and the target detection loss function, construct the multi-task loss function corresponding to the sky-based triplet sample;

[0032] Based on the multi-task loss function corresponding to each sky-based triplet sample, the model parameters of the initial search network are updated to obtain the target search network.

[0033] According to the multi-target detection and association method based on space-based and air-based systems provided in this application, the step of constructing a target detection loss function based on the predicted target category to which the air-based target sample belongs and the predicted candidate box corresponding to the air-based target sample includes:

[0034] Based on the predicted target category to which the spatial target sample belongs and the target category label to which the spatial target sample belongs, a category loss function is constructed;

[0035] Based on the predicted candidate boxes and the labels of the candidate boxes corresponding to the spatial target samples, a regression loss function is constructed.

[0036] The target detection loss function is constructed based on the category loss function and the regression loss function.

[0037] This application also provides a space-based and air-based multi-target detection and correlation device, including:

[0038] The acquisition unit is used to acquire a set of space-based targets detected by space-based systems and space-based remote sensing images acquired by air-based systems; wherein, the set of space-based targets includes multiple space-based targets;

[0039] The processing unit is configured to input both the space-based target set and the space-based remote sensing image into a pre-trained target search network; extract space-based target association feature sets corresponding to multiple space-based targets through a shared backbone network in the target search network; and extract space-based data basic features from the space-based remote sensing image; and perform target detection on the space-based data basic features based on the target detection head in the target search network to determine the space-based target set in the space-based remote sensing image; and filter the space-based target association feature set from the space-based data basic features based on multiple space-based targets included in the space-based target set; wherein, the target search network is obtained by training an initial search network based on multiple space-based triplet samples;

[0040] The association unit is used to associate the plurality of space-based targets with the plurality of space-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set.

[0041] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the space-based and air-based multi-target detection and correlation method as described above.

[0042] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the space-based and air-based multi-target detection and correlation method as described above.

[0043] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the space-based and air-based multi-target detection and correlation method as described above.

[0044] The space-based and airborne multi-target detection and association method, apparatus, and equipment provided in this application, when performing space-based multi-target detection and association, acquire a set of space-based targets for detection and airborne remote sensing images; wherein the set of space-based targets includes multiple space-based targets; and employs a "shared backbone feature + The "multi-task head" target search network architecture inputs both the space-based target set and the airborne remote sensing image into a pre-trained target search network. The shared backbone network within the target search network extracts space-based target association feature sets corresponding to multiple space-based targets, and also extracts basic airborne data features from the airborne remote sensing image. Based on the target detection head in the target search network, target detection is performed on the basic airborne data features to determine the airborne target set in the airborne remote sensing image. Based on the multiple airborne targets included in the airborne target set, an airborne target association feature set is selected from the basic airborne data features. Finally, based on the similarity between the space-based target association feature set and the airborne target association feature set, multiple space-based targets are associated with multiple airborne targets. This ensures that the basic airborne data features in the airborne remote sensing image are extracted only once. These basic airborne data features can simultaneously support the airborne target detection and association tasks, reducing the repeated extraction of basic airborne data features. This addresses the low association efficiency caused by repeated extraction of detection and association features in existing technologies, thereby effectively improving the efficiency of multi-target association. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating a multi-target detection and association method based on space-based and air-based systems, provided as an embodiment of this application.

[0047] Figure 2 This is a schematic diagram of the structure of a target search network provided in an embodiment of this application.

[0048] Figure 3 This is a schematic diagram illustrating a process for determining the set of airborne targets in an airborne remote sensing image by performing target detection on the basic features of airborne data based on a target detection head in a target search network, as provided in an embodiment of this application.

[0049] Figure 4 This is a schematic diagram illustrating a process for associating multiple space-based targets with multiple air-based targets, provided as an embodiment of this application.

[0050] Figure 5 This is a schematic diagram illustrating the process of training an initial search network based on multiple sky-based triplet samples, as provided in an embodiment of this application.

[0051] Figure 6 This is a schematic diagram of a multi-target detection and correlation device based on space-based and air-based systems, provided as an embodiment of this application.

[0052] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0055] Space-based multi-target detection and association is the core of achieving collaborative observation. Its core is to detect targets from space-based remote sensing images and associate them with target slices detected from space-based images, thereby locating the same target.

[0056] Space-based multi-target detection and association can be applied to fields such as national defense and security monitoring, disaster emergency response, and marine target monitoring. Currently, the typical approach to space-based multi-target detection and association is a separate process of "detect first, then associate," where targets are first extracted from space-based remote sensing images using target detection algorithms, and then associated with the space-based detected targets using independent association algorithms, such as similarity matching algorithms.

[0057] However, the above method has the problem of repeated extraction of detection features and associated features, resulting in low efficiency of multi-target association.

[0058] To improve the efficiency of multi-target detection and association in space-based multi-target detection and association, this application provides a space-based and air-based multi-target detection and association method. It employs a target search network architecture of "shared backbone features + multiple task heads," ensuring that the basic features of air-based data in air-based remote sensing images are extracted only once. These basic features can simultaneously support both air-based target detection and association tasks, reducing redundant extraction of basic features. This addresses the shortcomings of existing space-based and air-based multi-target detection and association methods, which suffer from low association efficiency due to repeated extraction of detection and association features, thus effectively improving the efficiency of multi-target association.

[0059] The following detailed embodiments illustrate the space-based and air-based multi-target detection and association method provided in this application. It is understood that these specific embodiments can be combined with each other, and similar concepts or processes may not be repeated in some embodiments.

[0060] It is understood that the execution entity of the space-based and air-based multi-target detection and correlation method provided in this application can be a computer, a server, or a specially set space-based and air-based multi-target detection and correlation device, or a space-based and air-based multi-target detection and correlation device set in the electronic device. The space-based and air-based multi-target detection and correlation device can be implemented by software, hardware, or a combination of both, and can be set according to actual needs.

[0061] Figure 1 This application provides a flowchart illustrating a space-based and air-based multi-target detection and association method as an embodiment of this application. For example, please refer to... Figure 1 As shown, this space-based and air-based multi-target detection and association method may include:

[0062] S101. Obtain a set of space-based targets based on space-based detection, and obtain space-based remote sensing images based on space-based acquisition; wherein, the set of space-based targets includes multiple space-based targets.

[0063] For example, the set of space-based targets can be denoted as... N represents the number of space-based targets included in the space-based target set. Represents the th among N space-based targets. One space-based target Indicates the first A target slice of a space-based target. For example, a space-based remote sensing image can be denoted as... .

[0064] After acquiring the space-based target set detected by space-based detection and the space-based remote sensing images acquired by air-based acquisition, in order to efficiently perform multi-target association, in this embodiment of the application, a target search network architecture of "shared backbone features + multi-task head" can be adopted. This ensures that the basic features of air-based data in the air-based remote sensing images are extracted only once. These basic features of air-based data can simultaneously support the air-based target detection task and the association task, reducing the repeated extraction of basic features of air-based data. That is, the following S102 is executed:

[0065] S102. Input both the space-based target set and the airborne remote sensing image into a pre-trained target search network. Extract the space-based target association feature set corresponding to multiple space-based targets through the shared backbone network in the target search network, and extract the basic features of airborne data in the airborne remote sensing image. Then, perform target detection on the basic features of airborne data based on the target detection head in the target search network to determine the airborne target set in the airborne remote sensing image. Based on the multiple airborne targets included in the airborne target set, select the airborne target association feature set from the basic features of airborne data.

[0066] The target search network is trained on the initial search network using multiple sky-based triplet sample data. The specific training process is detailed below. Figure 5 The example shown.

[0067] For example, it can be combined Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a target search network provided in an embodiment of this application. The target search network may include a shared backbone network and a target detection head, that is, it adopts a target search network architecture of "shared backbone features + multi-task head". The shared backbone network can be referred to as... The target detection head can be denoted as .

[0068] Set of space-based targets and airborne remote sensing images All inputs are sent to the target search network, and then processed through the shared backbone network within the target search network. Extract the set of space-based target association features corresponding to multiple space-based targets. This set of space-based target association features can be represented as follows: , among which, the Target slice of a space-based target Corresponding space-based target association features ; and through a shared backbone network Extracting airborne remote sensing images The basic characteristics of the spaceborne basic data in the data can be denoted as: , .

[0069] For example, a shared backbone network It can be used as a backbone network based on joint learning in the spatial and frequency domains. The specific settings can be configured according to actual needs.

[0070] Considering that in this embodiment of the application, through a shared backbone network Extracted basic features of empty-base data It can simultaneously support space-based target detection and association tasks, after extracting basic features from space-based data. Then, the basic features of the airborne data can be... Input to the target detection head in the target search network Based on target detection head Basic characteristics of airborne data Perform target detection to determine the space-based remote sensing image. The set of empty base targets in the data can be denoted as: And based on the airborne target set The multiple space-based targets included in the data represent the fundamental characteristics of space-based data. Pruning is performed to extract basic features from the empty-base data. The set of features associated with empty base targets can be denoted as: .

[0071] Where M represents the space-based remote sensing image The number of airborne targets included, the first Target slice of an airborne target Corresponding spatial target association features .

[0072] For example, the set of airborne targets ,in, Represents the first of M airborne targets. One air-based target, Indicates the first Target slices of an empty base target Indicates the predicted first The predicted candidate boxes for the nth empty-base target are used to describe the nth empty-base target. Individual airborne targets center point coordinates ,width and height ,Right now , Indicates the actual first The candidate bounding box labels for each empty base target; Indicates the predicted first The target category to which each airborne target belongs. This represents the confidence level corresponding to the predicted candidate box.

[0073] As can be seen, in the embodiments of this application, when performing airborne multi-target detection and association, the target search network architecture of "shared backbone features + multi-task head" is adopted, that is, the integrated modeling of airborne target detection task and association (subsequent similarity determination) task is realized, which reduces the repeated extraction of basic features of airborne data, not only improving the efficiency of multi-target association, but also reducing the computational overhead, adapting to airborne edge computing power, and realizing the improvement of joint optimization performance of detection task and association task.

[0074] S103. Based on the similarity between the space-based target association feature set and the air-based target association feature set, associate multiple space-based targets with multiple air-based targets.

[0075] As can be seen from the embodiments of this application, when performing space-based multi-target detection and association, the system acquires a set of space-based targets detected by space-based systems and space-based remote sensing images acquired by air-based systems; wherein, the set of space-based targets includes multiple space-based targets; and a "shared backbone feature + The "multi-task head" target search network architecture inputs both the space-based target set and the airborne remote sensing image into a pre-trained target search network. The shared backbone network within the target search network extracts space-based target association feature sets corresponding to multiple space-based targets, and also extracts basic airborne data features from the airborne remote sensing image. Based on the target detection head in the target search network, target detection is performed on the basic airborne data features to determine the airborne target set in the airborne remote sensing image. Based on the multiple airborne targets included in the airborne target set, an airborne target association feature set is selected from the basic airborne data features. Finally, based on the similarity between the space-based target association feature set and the airborne target association feature set, multiple space-based targets are associated with multiple airborne targets. This ensures that the basic airborne data features in the airborne remote sensing image are extracted only once. These basic airborne data features can simultaneously support the airborne target detection and association tasks, reducing the repeated extraction of basic airborne data features. This addresses the low association efficiency caused by repeated extraction of detection and association features in existing technologies, thereby effectively improving the efficiency of multi-target association.

[0076] Based on the above Figure 1 In the illustrated embodiment, for example, in S102 above, the specific implementation of determining the set of airborne targets in the airborne remote sensing image by performing target detection on the basic features of airborne data based on the target detection head in the target search network can be found below. Figure 3 The example shown.

[0077] Figure 3 This application provides a flowchart illustrating a process for determining the set of airborne targets in an airborne remote sensing image by performing target detection on basic features of airborne data based on a target detection head in a target search network. For example, please refer to [link to relevant documentation]. Figure 3As shown, the method may include:

[0078] S301. Target detection is performed on the basic features of the space-based data based on the target detection head to obtain multiple prediction candidate boxes in the space-based remote sensing image, as well as the confidence level corresponding to each prediction candidate box.

[0079] For example, in the embodiments of this application, when performing target detection on the basic features of airborne data based on the target detection head, a classic target detection head can be used. The Non-Maximum Suppression (NMS) algorithm is used for target detection based on the fundamental features of spatial data. For example, see Equation 1 below:

[0080] Formula 1

[0081] in, Indicates the predicted first Predicted candidate boxes for each empty base target This represents the confidence level corresponding to the predicted candidate box.

[0082] S302. Based on the confidence level corresponding to each predicted candidate box, determine the set of airborne targets in the airborne remote sensing image.

[0083] For example, when predicting the set of space-based targets in a space-based remote sensing image based on the confidence level corresponding to each prediction candidate box, the targets selected by the multiple prediction candidate boxes with the highest confidence levels can be determined as space-based targets; or the targets selected by the multiple prediction candidate boxes with confidence levels greater than a preset confidence threshold can be determined as space-based targets, etc., where the set of space-based targets constitutes the set of space-based targets in the space-based remote sensing image.

[0084] As can be seen from the embodiments of this application, after extracting the basic features of space-based data from the space-based remote sensing image through the shared backbone network, target detection can be further performed on the basic features of the space-based data based on the target detection head to determine the set of space-based targets in the space-based remote sensing image, thus completing the space-based target detection task. Furthermore, a set of space-based target association features can be filtered from the basic features of the space-based data based on the target detection head. This set can be used to associate multiple space-based targets with multiple space-based targets, thus supporting the association task and reducing the repeated extraction of basic features of the space-based data. This solves the problem of low association efficiency caused by repeated extraction of detection and association features in existing technologies, thereby effectively improving the multi-target association efficiency.

[0085] Based on any of the above embodiments, for example, in S103 above, the specific implementation of associating multiple space-based targets with multiple space-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set can be found below. Figure 4The example shown.

[0086] Figure 4 This application provides a schematic diagram of a process for associating multiple space-based targets with multiple airborne targets, as illustrated in the embodiments of this application. For example, please refer to [link to relevant documentation]. Figure 4 As shown, the method may include:

[0087] S401. Based on the similarity between the set of features associated with space-based targets and the set of features associated with airborne targets, construct the corresponding association strategy cost augmentation matrix; wherein, the association strategy cost augmentation matrix is ​​used to characterize the possibility of association between space-based targets and airborne targets.

[0088] For example, in the embodiments of this application, when constructing the corresponding association strategy cost augmentation matrix based on the similarity between the space-based target association feature set and the air-based target association feature set, the corresponding target similarity matrix can be constructed first based on the similarity between each pair of targets in the space-based target association feature set and the air-based target association feature set; and then the corresponding association strategy cost augmentation matrix can be constructed based on the target similarity matrix.

[0089] Based on the space-based target set The set of airborne targets is For example, we can calculate the similarity between any two targets in the space-based target association feature set and the air-based target association feature set, such as cosine similarity, to obtain the corresponding target similarity matrix. See Formula 2 below:

[0090] Formula 2

[0091] in, Indicates the first Correlation characteristics of individual space-based targets With the Association characteristics of individual airborne targets Similarity between them Indicates the first The correlation characteristics of space-based targets Indicates the first The correlation characteristics of each empty base target.

[0092] For example, in the embodiments of this application, when constructing the corresponding association strategy cost augmentation matrix based on the target similarity matrix, the association strategy cost matrix can be constructed first based on the target similarity matrix, which can be denoted as: ,in, , Indicates the first Correlation characteristics of individual space-based targets With the Association characteristics of individual airborne targets Determine the associated policy costs between them and establish the augmented matrix of associated policy costs. Among them, the augmented first term in the associated strategy cost augmentation matrix Line and number Listed as a hyperparameter for the cost of birth and death This can be used to prevent optimization problems from degenerating.

[0093] Furthermore, in this embodiment, to effectively resolve the "one-to-many" and "many-to-one" association conflicts that exist in multi-target association, thereby improving association accuracy, spatiotemporal consistency constraints should be followed when performing space-based multi-target detection and association, and the birth and death of targets should be considered. Therefore, each space-based target can only be associated with one or no space-based target, and correspondingly, each space-based target can only be associated with one or no space-based target. To comply with spatiotemporal consistency constraints, in this embodiment, an association plan augmentation matrix is ​​constructed, i.e., the following S402 is executed, to apply spatiotemporal consistency constraints to multiple space-based targets and multiple space-based targets, thereby effectively improving association accuracy.

[0094] S402. Construct the associated planning augmented matrix corresponding to multiple space-based targets and multiple air-based targets; wherein, the associated planning augmented matrix is ​​used to constrain the spatiotemporal consistency of multiple space-based targets and multiple air-based targets.

[0095] For example, when constructing the correlation plan augmentation matrix corresponding to multiple space-based targets and multiple air-based targets, the correlation plan matrix can be constructed first. ,in, Indicates the first Target slice of a space-based target With the Target slice of an airborne target Related, Indicates the first Target slice of a space-based target With the Target slice of an airborne target Unrelated. Considering the potential creation and demise of targets, an augmented matrix of related plans can be constructed. The augmented first part of the associated plan augmentation matrix Line and number The column is used to indicate that a space-based target is not associated with any air-based target, and / or that an air-based target is not associated with any space-based target, thereby constraining the spatiotemporal consistency of multiple space-based targets and multiple air-based targets.

[0096] For example, in an embodiment of this application, the association plan augmentation matrix is ​​used to perform spatiotemporal consistency constraints on multiple space-based targets and multiple air-based targets, including:

[0097] based on Spatiotemporal consistency constraints are applied to multiple space-based targets and multiple air-based targets.

[0098] Where N represents the number of space-based targets and M represents the number of air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the correlation plan augmentation matrix.

[0099] It should be noted that in the embodiments of this application, there is no specific order between S401 and S402. S401 can be executed first, followed by S402; or S402 can be executed first, followed by S401; or S401 and S402 can be executed simultaneously. The specific order can be set according to actual needs. Here, the embodiments of this application are only used as an example of executing S401 first, followed by S402, but this does not mean that the embodiments of this application are limited to this.

[0100] S403. Based on the association plan augmentation matrix and the association strategy cost augmentation matrix, associate multiple space-based targets with multiple air-based targets.

[0101] For example, in an embodiment of this application, association is performed between multiple space-based targets and multiple air-based targets based on the association plan augmentation matrix and the association strategy cost augmentation matrix, including:

[0102] based on Associate multiple space-based targets with multiple air-based targets to achieve the desired result. This yields correlation results between multiple space-based targets and multiple air-based targets;

[0103] in, This represents the augmented matrix of the associated plans, where N represents the number of multiple space-based targets and M represents the number of multiple air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the correlation plan augmentation matrix. This represents the augmented matrix of associated policy costs. This represents the probability that the space-based target in the i-th row of the association strategy cost augmentation matrix is ​​associated with the air-based target in the j-th column.

[0104] Based on the above description, it is not difficult to see that, in the embodiments of this application, by imposing spatiotemporal consistency constraints on multiple space-based targets and multiple air-based targets, the multi-target association problem can be transformed into a "bipartite graph matching problem with newly generated / dead nodes," making it possible to use the Hungarian matching algorithm to solve for the globally optimal discrete solution. This strictly constrains that a space-based target can only be associated with one or no air-based target, and a air-based target can also only be associated with one or no space-based target, and the association results of space-based targets are obtained accordingly. ,in, For the number of sky-based associated target logs.

[0105] As can be seen from the embodiments of this application, when associating multiple space-based targets with multiple space-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set, an association plan augmentation matrix is ​​constructed to constrain the spatiotemporal consistency of multiple space-based targets and multiple air-based targets. The creation and extinction of targets are also considered, so that a space-based target can only be associated with one or no air-based target, and a space-based target can only be associated with one or no space-based target. This can effectively solve the "one-to-many" and "many-to-one" association conflicts that exist in multi-target association, thereby improving the association accuracy.

[0106] The following section describes in detail how the target search network is trained on the initial search network based on multiple sky-based triplet samples. Each sky-based triplet sample includes a space-based remote sensing image sample, a set of positive space-based target samples corresponding to the space-based remote sensing image sample, and a set of negative space-based target samples.

[0107] In this embodiment of the application, in order to train the target search network end-to-end, a corresponding intra-batch spatiotemporal consistency sample sampling strategy and a multi-task loss function are designed, with the number of multiple sky-based triplet samples as... For example, a training batch contains A sky-based triplet sample can be denoted as .in, express The first of the sky-based triplet samples A sample of sky-based triplets. Indicates the first Space-based remote sensing image samples from a set of space-based triplet samples. Indicates the first The set of positive space-based target samples in a set of space-based triplet samples , Indicates the first Set of negative samples of space-based targets in a set of space-based triplet samples .

[0108] Airborne remote sensing image samples The set of tags can be denoted as ,in, Represents airborne remote sensing image samples The number of empty-base target samples included. express The first of the empty-base target samples One empty base target sample, Indicates the first Target slices of empty base target samples Indicates the first Candidate bounding box labels for empty basis target samples. Indicates the first Identification of each empty base target sample. Indicates the first The target category label of each hollow-based target sample. Considering that no two targets in the same hollow-based remote sensing image sample will have the same identifier, therefore... The elements in the array are mutually exclusive. Assume... The complete set of identity identifiers. Let represent the set of identifiers for spaceborne target samples in a spaceborne remote sensing image sample. This represents the set of identity identifiers that do not appear in the sample of the airborne remote sensing image.

[0109] For example, in an embodiment of this application, space-based remote sensing image samples are constructed. The corresponding positive sample set of space-based targets and the negative sample set of space-based targets At that time, considering the possible birth and death of the target, random sampling and the first Target slice of a hollow target sample Space-based target slices with the same identity as Corresponding positive samples of space-based targets Based on the probability of newborns dying choose The space-based target slices corresponding to the random identity identifiers in the data are used as Corresponding space-based target negative samples , and / or, with probability choose The space-based target slices corresponding to the random identity identifiers in the data are used as Corresponding space-based target negative samples .

[0110] Figure 5 This application provides a schematic diagram of a target search network training process based on multiple sky-based triplet samples for an initial search network. For example, see [link to relevant documentation]. Figure 5 As shown, the training process may include:

[0111] For each sky-based triplet sample, with the first Taking a single sky-based triplet sample as an example, S501-S504 are executed for each sample:

[0112] S501. Through the initial search network, determine the similarity between the space-based target sample and the first space-based target sample in the space-based remote sensing image sample, as well as the similarity between the space-based target sample and the second space-based target sample.

[0113] Among them, the first space-based target sample is the space-based target sample corresponding to the air-based target sample in the positive space-based target sample set, and the second space-based target sample is the space-based target sample corresponding to the air-based target sample in the negative space-based target sample set.

[0114] It should be noted that, in the embodiments of this application, the specific implementation of determining the similarity between the space-based target sample and the first space-based target sample in the space-based remote sensing image sample through the initial search network, as well as the specific implementation of determining the similarity between the space-based target association feature set and the space-based target association feature set, is similar to the above-described specific implementation. Please refer to the relevant descriptions above. Here, the embodiments of this application will not be repeated.

[0115] S502. Based on the similarity between the airborne target sample and the first airborne target sample in the airborne remote sensing image samples, and the similarity between the airborne target sample and the second airborne target sample, construct a target association loss function.

[0116] Among them, the first space-based target sample is the space-based target sample corresponding to the air-based target sample in the positive space-based target sample set, and the second space-based target sample is the space-based target sample corresponding to the air-based target sample in the negative space-based target sample set.

[0117] For example, construct the target association loss function. See Formula 3 below:

[0118] Formula 3

[0119] in, m This represents the triplet interval hyperparameter. .

[0120] S503. Based on the predicted target category to which the base target sample belongs and the predicted candidate box corresponding to the base target sample, construct the target detection loss function.

[0121] For example, in the embodiments of this application, when constructing the target detection loss function based on the predicted target category to which the empty target sample belongs and the predicted candidate box corresponding to the empty target sample, a category loss function can be constructed based on the predicted target category to which the empty target sample belongs and the target category label to which the empty target sample belongs; and a regression loss function can be constructed based on the predicted candidate box corresponding to the empty target sample and the candidate box label corresponding to the empty target sample; and then a target detection loss function can be constructed based on the category loss function and the regression loss function.

[0122] For example, a category loss function is constructed based on the predicted target category to which the empty-base target sample belongs and the target category label to which the empty-base target sample belongs. For the following example, see Formula 4:

[0123] Formula 4

[0124] in, Indicates the total number of target categories. express The first of the target categories Target categories, Indicates the first Target slice of a hollow target sample The one-hot encoding form of the predicted target category, Indicates the first Target slice of a hollow target sample The one-hot encoded form of the target category label.

[0125] For example, a regression loss function is constructed based on the predicted candidate boxes corresponding to the empty-base target samples and the labels of the candidate boxes corresponding to the empty-base target samples. In this case, a loss based on the calculated Intersection over Union (IOU) ratio can be used, as shown in Formula 5 below:

[0126] Formula 5

[0127] in, Indicates the first Predicted candidate boxes for each empty base target sample Indicates the first Candidate box labels for empty base target samples.

[0128] For example, an object detection loss function is constructed based on the category loss function and the regression loss function. For the following example, see Formula 6:

[0129] Formula 6

[0130] in, Represents the category loss function The corresponding weights Represents the regression loss function The corresponding weights.

[0131] S504. Based on the target association loss function and the target detection loss function, construct the multi-task loss function corresponding to the sky-based triplet samples.

[0132] For example, based on the target association loss function and the target detection loss function, a multi-task loss function corresponding to the sky-based triplet samples is constructed. For the following example, see Formula 7:

[0133] Formula 7

[0134] in, Represents the target association loss function The corresponding weights.

[0135] Combining S501-S503 above, we can obtain the multi-task loss function corresponding to each sky-based triplet sample. This allows us to update the model parameters of the initial search network using the multi-task loss function corresponding to each sky-based triplet sample, i.e., execute S504 as follows:

[0136] S505. Based on the multi-task loss function corresponding to each sky-based triplet sample, the model parameters of the initial search network are updated to obtain the target search network.

[0137] For example, based on the multi-task loss function corresponding to each sky-based triplet sample, the model parameters of the initial search network are updated. This can be achieved by determining the corresponding average loss function based on the multi-task loss function for each sky-based triplet sample, and then updating the model parameters of the initial search network based on the average loss function to obtain the target search network. In this way, by combining multi-task integrated modeling and joint optimization of detection and association tasks during the training of the initial search network, the system-level accuracy of the trained target search network in both detection and association tasks can be effectively improved.

[0138] As can be seen, in this embodiment of the application, by pre-training the target search network, the basic features of the airborne data in the airborne remote sensing image are extracted only once by the target search network. These basic features of the airborne data can simultaneously support the airborne target detection task and the association task, reducing the repeated extraction of basic features of the airborne data. This can solve the defect of low association efficiency caused by repeated extraction of detection features and association features in the prior art, thereby effectively improving the multi-target association efficiency.

[0139] The following describes the space-based and air-based multi-target detection and correlation device provided in this application. The space-based and air-based multi-target detection and correlation device described below can be referred to in correspondence with the space-based and air-based multi-target detection and correlation method described above.

[0140] Figure 6 A schematic diagram of a space-based and air-based multi-target detection and correlation device provided in this application embodiment is shown below. For example, please refer to... Figure 6 As shown, the space-based and air-based multi-target detection and correlation device 60 may include:

[0141] The acquisition unit 601 is used to acquire a set of space-based targets detected by space-based systems and space-based remote sensing images acquired by air-based systems; wherein, the set of space-based targets includes multiple space-based targets;

[0142] Processing unit 602 is configured to input both the space-based target set and the space-based remote sensing image into a pre-trained target search network; extract space-based target association feature sets corresponding to multiple space-based targets through a shared backbone network in the target search network; and extract space-based data basic features from the space-based remote sensing image; and perform target detection on the space-based data basic features based on the target detection head in the target search network to determine the space-based target set in the space-based remote sensing image; and filter the space-based target association feature set from the space-based data basic features based on multiple space-based targets included in the space-based target set; wherein, the target search network is obtained by training an initial search network based on multiple space-based triplet samples;

[0143] The association unit 603 is used to associate the plurality of space-based targets with the plurality of space-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set.

[0144] For example, in an embodiment of this application, the processing unit 602 is used to perform target detection on the basic features of the airborne data based on the target detection head in the target search network, and to determine the set of airborne targets in the airborne remote sensing image, including:

[0145] Based on the target detection head, target detection is performed on the basic features of the space-based data to obtain multiple prediction candidate boxes in the space-based remote sensing image, as well as the confidence level corresponding to each prediction candidate box;

[0146] Based on the confidence level corresponding to each predicted candidate box, the set of airborne targets in the airborne remote sensing image is determined.

[0147] For example, in an embodiment of this application, the association unit 603 is used to associate the plurality of space-based targets with the plurality of space-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set, including:

[0148] Based on the similarity between the space-based target association feature set and the air-based target association feature set, a corresponding association strategy cost augmentation matrix is ​​constructed; wherein, the association strategy cost augmentation matrix is ​​used to characterize the probability of association between space-based targets and air-based targets;

[0149] Construct an augmented association plan matrix corresponding to the plurality of space-based targets and the plurality of airborne targets; wherein, the augmented association plan matrix is ​​used to perform spatiotemporal consistency constraints on the plurality of space-based targets and the plurality of airborne targets;

[0150] Based on the association plan augmented matrix and the association strategy cost augmented matrix, the multiple space-based targets and the multiple air-based targets are associated.

[0151] For example, in an embodiment of this application, the association unit 603 is used to construct a corresponding association strategy cost augmentation matrix based on the similarity between the space-based target association feature set and the air-based target association feature set, including:

[0152] Based on the similarity between pairs of targets in the space-based target association feature set and the air-based target association feature set, a corresponding target similarity matrix is ​​constructed;

[0153] Based on the target similarity matrix, the corresponding association strategy cost augmentation matrix is ​​constructed.

[0154] For example, in an embodiment of this application, the association plan augmentation matrix is ​​used to perform spatiotemporal consistency constraints on the plurality of space-based targets and the plurality of air-based targets, including:

[0155] based on Spatiotemporal consistency constraints are applied to the plurality of space-based targets and the plurality of airborne targets;

[0156] Wherein, N represents the number of the plurality of space-based targets, and M represents the number of the plurality of air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the augmented matrix of the correlation plan.

[0157] For example, in an embodiment of this application, the association unit 603 is used to associate the plurality of space-based targets with the plurality of air-based targets based on the association plan augmentation matrix and the association strategy cost augmentation matrix, including:

[0158] based on Associating the plurality of space-based targets with the plurality of air-based targets to satisfy the following conditions: The correlation results between the multiple space-based targets and the multiple air-based targets are obtained;

[0159] in, This represents the augmented matrix of the association plan, where N represents the number of the multiple space-based targets, and M represents the number of the multiple air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the correlation plan augmentation matrix. This represents the cost augmentation matrix of the association strategy. This indicates the probability that a space-based target in the i-th row of the association strategy cost augmentation matrix is ​​associated with a space-based target in the j-th column.

[0160] For example, in this embodiment of the application, each space-based triplet sample includes a space-based remote sensing image sample, a set of positive space-based target samples corresponding to the space-based remote sensing image sample, and a set of negative space-based target samples. The target search network is trained on an initial search network based on the multiple space-based triplet samples, including:

[0161] For each sky-based triplet sample, perform the following operations:

[0162] The initial search network is used to determine the similarity between a space-based target sample and a first-level space-based target sample in the space-based remote sensing image samples, as well as the similarity between the space-based target sample and the second-level space-based target sample; wherein, the first-level space-based target sample is the space-based target sample in the positive space-based target sample set that corresponds to the space-based target sample, and the second-level space-based target sample is the space-based target sample in the negative space-based target sample set that corresponds to the space-based target sample;

[0163] Based on the similarity between the airborne target sample and the first spaceborne target sample, and the similarity between the airborne target sample and the second spaceborne target sample, a target association loss function is constructed;

[0164] Based on the predicted target category to which the spatial target sample belongs and the predicted candidate box corresponding to the spatial target sample, a target detection loss function is constructed;

[0165] Based on the target association loss function and the target detection loss function, construct the multi-task loss function corresponding to the sky-based triplet sample;

[0166] Based on the multi-task loss function corresponding to each sky-based triplet sample, the model parameters of the initial search network are updated to obtain the target search network.

[0167] For example, in an embodiment of this application, the step of constructing a target detection loss function based on the predicted target category to which the spatial target sample belongs and the predicted candidate box corresponding to the spatial target sample includes:

[0168] Based on the predicted target category to which the spatial target sample belongs and the target category label to which the spatial target sample belongs, a category loss function is constructed;

[0169] Based on the predicted candidate boxes and the labels of the candidate boxes corresponding to the spatial target samples, a regression loss function is constructed.

[0170] The target detection loss function is constructed based on the category loss function and the regression loss function.

[0171] The space-based and air-based multi-target detection and association device 60 provided in this application embodiment can execute the technical solution of the space-based and air-based multi-target detection and association method in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the above-mentioned space-based and air-based multi-target detection and association method. Please refer to the implementation principle and beneficial effects of the above-mentioned space-based and air-based multi-target detection and association method. It will not be repeated here.

[0172] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a multi-target detection and association method based on space-based and air-based systems. The method includes: acquiring a set of space-based targets detected by space-based systems and air-based remote sensing images acquired by air-based systems; wherein the set of space-based targets includes multiple space-based targets; inputting both the set of space-based targets and the air-based remote sensing images into a pre-trained target search network; extracting a set of space-based target association features corresponding to multiple space-based targets through a shared backbone network in the target search network; and extracting basic air-based data features from the air-based remote sensing images. The system performs target detection on the basic features of the space-based data based on the target detection head in the target search network to determine the set of space-based targets in the space-based remote sensing image. Based on the multiple space-based targets included in the set, a set of space-based target association features is selected from the basic features of the space-based data. The target search network is trained on an initial search network based on multiple space-based triplet samples. Based on the similarity between the set of space-based target association features and the set of space-based target association features, the multiple space-based targets are associated with the multiple space-based targets.

[0173] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the space-based and airborne multi-target detection and association method provided by the above methods. This method includes: acquiring a set of space-based targets detected by space-based methods and airborne remote sensing images acquired by airborne methods; wherein the set of space-based targets includes multiple space-based targets; inputting both the set of space-based targets and the airborne remote sensing images into a pre-trained target search network, and extracting the corresponding targets of the multiple space-based targets through a shared backbone network in the target search network. The system involves: establishing a base target association feature set; extracting basic features of space-based data from the space-based remote sensing image; performing target detection on the basic features of the space-based data based on the target detection head in the target search network; determining the set of space-based targets in the space-based remote sensing image; and selecting a set of space-based target association features from the basic features of the space-based data based on multiple space-based targets included in the set of space-based targets. The target search network is trained on an initial search network based on multiple space-based triplet samples. The multiple space-based targets are associated with the multiple space-based targets based on the similarity between the set of space-based target association features and the set of space-based target association features.

[0175] Furthermore, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the multi-target detection and association methods based on space-based and airborne methods provided by the above-described methods. The method includes: acquiring a set of space-based targets detected by space-based methods and airborne remote sensing images acquired by airborne methods; wherein the set of space-based targets includes multiple space-based targets; inputting both the set of space-based targets and the airborne remote sensing images into a pre-trained target search network; extracting a set of space-based target association features corresponding to multiple space-based targets through a shared backbone network in the target search network; and extracting the... The system identifies basic features of space-based data in space-based remote sensing images; targets are detected based on the target detection head in the target search network using these basic features to determine a set of space-based targets in the space-based remote sensing images; and a set of associated features of space-based targets is selected from the basic features of the space-based data based on multiple space-based targets included in the set of space-based targets. The target search network is trained on an initial search network using multiple space-based triplet samples. Based on the similarity between the set of associated features of space-based targets and the set of associated features of space-based targets, the multiple space-based targets are associated with the multiple space-based targets.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-target detection and correlation method based on space-based and air-based systems, characterized in that, include: Acquire a set of space-based targets detected by space-based methods and space-based remote sensing images acquired by air-based methods; wherein the set of space-based targets includes multiple space-based targets; Both the space-based target set and the airborne remote sensing image are input into a pre-trained target search network. The shared backbone network of the target search network extracts space-based target association feature sets corresponding to multiple space-based targets, and extracts basic airborne data features from the airborne remote sensing image. Based on the target detection head in the target search network, target detection is performed on the basic airborne data features to determine the spaceborne target set in the airborne remote sensing image. Based on the multiple spaceborne targets included in the spaceborne target set, a set of spaceborne target association features is selected from the basic airborne data features. The target search network is trained on an initial search network using multiple spaceborne triplet samples. Each spaceborne triplet sample includes an airborne remote sensing image sample, a set of positive spaceborne target samples corresponding to the spaceborne remote sensing image sample, and a set of negative spaceborne target samples. Based on the similarity between the space-based target association feature set and the air-based target association feature set, the multiple space-based targets are associated with the multiple air-based targets.

2. The method according to claim 1, characterized in that, The step of performing target detection on the basic features of the space-based data based on the target detection head in the target search network to determine the set of space-based targets in the space-based remote sensing image includes: Based on the target detection head, target detection is performed on the basic features of the space-based data to obtain multiple prediction candidate boxes in the space-based remote sensing image, as well as the confidence level corresponding to each prediction candidate box; Based on the confidence level corresponding to each predicted candidate box, the set of airborne targets in the airborne remote sensing image is determined.

3. The method according to claim 1, characterized in that, The step of associating the multiple space-based targets with the multiple space-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set includes: Based on the similarity between the space-based target association feature set and the air-based target association feature set, a corresponding association strategy cost augmentation matrix is ​​constructed; wherein, the association strategy cost augmentation matrix is ​​used to characterize the probability of association between space-based targets and air-based targets; Construct an augmented association plan matrix corresponding to the plurality of space-based targets and the plurality of airborne targets; wherein, the augmented association plan matrix is ​​used to perform spatiotemporal consistency constraints on the plurality of space-based targets and the plurality of airborne targets; Based on the association plan augmented matrix and the association strategy cost augmented matrix, the multiple space-based targets and the multiple air-based targets are associated.

4. The method according to claim 3, characterized in that, The construction of a corresponding association strategy cost augmentation matrix based on the similarity between the space-based target association feature set and the air-based target association feature set includes: Based on the similarity between pairs of targets in the space-based target association feature set and the air-based target association feature set, a corresponding target similarity matrix is ​​constructed; Based on the target similarity matrix, the corresponding association strategy cost augmentation matrix is ​​constructed.

5. The method according to claim 3, characterized in that, The association plan augmentation matrix is ​​used to impose spatiotemporal consistency constraints on the plurality of space-based targets and the plurality of air-based targets, including: based on Spatiotemporal consistency constraints are applied to the plurality of space-based targets and the plurality of airborne targets; Wherein, N represents the number of the plurality of space-based targets, and M represents the number of the plurality of air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the augmented matrix of the correlation plan.

6. The method according to claim 3, characterized in that, The association of the plurality of space-based targets with the plurality of air-based targets based on the association plan augmented matrix and the association strategy cost augmented matrix includes: based on Associating the plurality of space-based targets with the plurality of air-based targets to satisfy the following conditions: The correlation results between the multiple space-based targets and the multiple air-based targets are obtained; in, The matrix represents the augmented matrix of the association plan, where N represents the number of the multiple space-based targets and M represents the number of the multiple air-based targets. This represents the correlation degree between the space-based target in the i-th row and the air-based target in the j-th column of the correlation plan augmentation matrix. This represents the cost augmentation matrix of the association strategy. This indicates the probability that a space-based target in the i-th row of the association strategy cost augmentation matrix is ​​associated with a space-based target in the j-th column.

7. The method according to any one of claims 1-6, characterized in that, The target search network is trained on the initial search network based on the multiple sky-based triplet samples, including: For each sky-based triplet sample, perform the following operations: The initial search network is used to determine the similarity between a space-based target sample and a first-level space-based target sample in the space-based remote sensing image samples, as well as the similarity between the space-based target sample and the second-level space-based target sample; wherein, the first-level space-based target sample is the space-based target sample in the positive space-based target sample set that corresponds to the space-based target sample, and the second-level space-based target sample is the space-based target sample in the negative space-based target sample set that corresponds to the space-based target sample; Based on the similarity between the airborne target sample and the first spaceborne target sample, and the similarity between the airborne target sample and the second spaceborne target sample, a target association loss function is constructed; Based on the predicted target category to which the spatial target sample belongs and the predicted candidate box corresponding to the spatial target sample, a target detection loss function is constructed; Based on the target association loss function and the target detection loss function, construct the multi-task loss function corresponding to the sky-based triplet sample; Based on the multi-task loss function corresponding to each sky-based triplet sample, the model parameters of the initial search network are updated to obtain the target search network.

8. The method according to claim 7, characterized in that, The step of constructing a target detection loss function based on the predicted target category to which the spatial target sample belongs and the predicted candidate box corresponding to the spatial target sample includes: Based on the predicted target category to which the spatial target sample belongs and the target category label to which the spatial target sample belongs, a category loss function is constructed; Based on the predicted candidate boxes and the labels of the candidate boxes corresponding to the empty-base target samples, a regression loss function is constructed. The target detection loss function is constructed based on the category loss function and the regression loss function.

9. A multi-target detection and correlation device based on space-based and air-based systems, characterized in that, include: The acquisition unit is used to acquire a set of space-based targets detected by space-based systems and space-based remote sensing images acquired by air-based systems; wherein, the set of space-based targets includes multiple space-based targets; The processing unit is configured to input both the space-based target set and the airborne remote sensing image into a pre-trained target search network; extract space-based target association feature sets corresponding to multiple space-based targets through a shared backbone network in the target search network; and extract airborne data basic features from the airborne remote sensing image. It then performs target detection on the airborne data basic features based on the target detection head in the target search network to determine the airborne target set in the airborne remote sensing image. Finally, it filters the airborne target association feature set from the airborne data basic features based on the multiple airborne targets included in the airborne target set. The target search network is trained on an initial search network using multiple spaceborne triplet samples, each of which includes an airborne remote sensing image sample, a set of positive spaceborne target samples corresponding to the airborne remote sensing image sample, and a set of negative spaceborne target samples. The association unit is used to associate the plurality of space-based targets with the plurality of space-based targets based on the similarity between the space-based target association feature set and the air-based target association feature set.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multi-target detection and association method based on space-based and air-based systems as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-target detection and association method based on space-based and air-based systems as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-target detection and association method based on space-based and air-based systems as described in any one of claims 1 to 8.

Images