Weak registration multi-modal remote sensing image target association method and device based on group structure

Abstract

The invention provides a weak registration multi-mode remote sensing image target association method and device based on a group structure, and relates to the technical field of remote sensing image processing and computer vision. The method comprises the following steps: acquiring a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image comprises a plurality of first targets, and the second modal remote sensing image comprises a plurality of second targets; clustering the plurality of first targets based on the respective positions of the plurality of first targets to obtain a first group structure corresponding to the first modal remote sensing image; clustering the plurality of second targets based on the respective positions of the plurality of second targets to obtain a second group structure corresponding to the second modal remote sensing image; and associating the plurality of first targets and the plurality of second targets based on the first group structure and the second group structure. By adopting the technical scheme provided by the invention, the accuracy of target association can be effectively improved under the condition that a weak registration phenomenon exists.

Description

Technical Field

[0001] This application relates to the fields of remote sensing image processing and computer vision technology, and in particular to a method and apparatus for target association in weakly registered multimodal remote sensing images based on group structure. Background Technology

[0002] In target detection scenarios, taking ship detection as an example, information fusion is usually performed using multimodal remote sensing images. By associating ships in the multimodal remote sensing images, the ship can be detected through the association results.

[0003] When associating ships in multimodal remote sensing images, taking optical red-green-blue (RGB) images and synthetic aperture radar (SAR) images as an example, RGB images provide rich details, color, and texture information about ground features; while SAR images have all-weather, day-and-night imaging capabilities, can penetrate clouds and fog, and compensate for the limitations of optical sensors in adverse weather conditions. Therefore, fusing the complementary advantages of RGB and SAR images can effectively improve the accuracy of ship association.

[0004] However, considering that the acquisition operations of RGB remote sensing images and SAR remote sensing images are not strictly synchronized, spatial misalignment is a common phenomenon between RGB and SAR remote sensing images. To address this spatial misalignment, image registration algorithms are typically used for preprocessing. However, given the limited accuracy of existing registration algorithms and their inability to completely eliminate inherent spatial positional deviations caused by target displacement or different imaging modes (i.e., weak registration), the accuracy of target association is low. Summary of the Invention

[0005] This application provides a target association method and apparatus based on weak registration of multimodal remote sensing images with a group structure, which solves the problem of low accuracy of target association caused by weak registration of different modal remote sensing images in the prior art. It can effectively improve the accuracy of target association even in the presence of weak registration.

[0006] This application provides a weakly registered multimodal remote sensing image target association method based on group structure, including: Acquire a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes a plurality of first targets, and the second modal remote sensing image includes a plurality of second targets; Based on the respective positions of the plurality of first targets, the plurality of first targets are clustered to obtain a first group structure corresponding to the first modality remote sensing image; and based on the respective positions of the plurality of second targets, the plurality of second targets are clustered to obtain a second group structure corresponding to the second modality remote sensing image; Based on the first group structure and the second group structure, the plurality of first targets and the plurality of second targets are associated.

[0007] According to the method for weakly registered multimodal remote sensing image target association based on cluster structure provided in this application, the method includes: clustering the plurality of first targets based on their respective positions to obtain a first cluster structure corresponding to the first modality remote sensing image, including: Based on the positions of the plurality of first targets, the first targets whose spatial distance is less than a first distance threshold are clustered into the same target cluster to obtain a plurality of first target clusters; wherein, the first targets in different first target clusters do not overlap, and the first group structure includes the plurality of first target clusters; The step of clustering the multiple second targets based on their respective locations to obtain a second group structure corresponding to the second modality remote sensing image includes: Based on the respective positions of the plurality of second targets, the second targets whose spatial distance is less than a second distance threshold are clustered into the same target cluster to obtain a plurality of second target clusters; wherein, the second targets in different second target clusters do not overlap, and the second group structure includes the plurality of second target clusters.

[0008] According to the weak registration multimodal remote sensing image target association method based on group structure provided in this application, the association of the plurality of first targets and the plurality of second targets based on the first group structure and the second group structure includes: The plurality of first target clusters are traversed, and for the currently traversed first target cluster, the distance between the centroid of the currently traversed first target cluster and the centroid of each of the plurality of remaining second target clusters is determined; wherein, the remaining second target clusters are the target clusters among the plurality of second target clusters that have not been successfully matched with the traversed first target clusters; If, among multiple distances, there exists a target distance less than a third distance threshold, the remaining second target cluster corresponding to the target distance and the currently traversed first target cluster are determined as a successfully matched target cluster pair; Associativity between multiple first targets and multiple second targets in multiple target clusters.

[0009] According to the weak registration multimodal remote sensing image target association method based on group structure provided in this application, the association of multiple first targets and multiple second targets in multiple target cluster pairs includes: For each target cluster pair, based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs, an inter-cluster cost matrix corresponding to the target cluster pair is constructed; The inter-cluster cost matrix is ​​solved using the Hungarian matching algorithm to obtain the association results between the first target and the second target in the target cluster pair.

[0010] According to the weak registration multimodal remote sensing image target association method based on group structure provided in this application, the step of constructing the inter-cluster cost matrix corresponding to the target cluster pair based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs includes: For any first target and second target in the target cluster pair, if the category to which the first target belongs is the same as the category to which the second target belongs, the distance between the first target and the second target is determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix; If the category to which the first target belongs is different from the category to which the second target belongs, the maximum value is determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix.

[0011] According to the group structure-based weak registration multimodal remote sensing image target association method provided in this application, the method further includes: If there is an unassociated first target in the first modal remote sensing image and an unassociated second target in the second modal remote sensing image, determine the distance between any two unassociated first targets and unassociated second targets; Associat the unassociated first target and the unassociated second target based on the distance between any two unassociated first targets and unassociated second targets; The unassociated first target includes at least one of the following: The first target that was not successfully clustered in the first group structure; The first target in the first target cluster that did not successfully match the second target cluster; The first target in the target cluster pair that was not successfully associated with the second target; The unrelated second objective includes at least one of the following: The second group structure includes second targets that were not successfully clustered; The second target in the second target cluster that failed to match the first target cluster; The second target in the target cluster pair that was not successfully associated with the first target.

[0012] According to the weak registration multimodal remote sensing image target association method based on group structure provided in this application, the method of associating the unassociated first target and the unassociated second target based on the distance between any two unassociated first targets and unassociated second targets includes: Based on the distance between any two unrelated first targets and unrelated second targets, construct the corresponding distance matrix; If there are unrelated first targets and unrelated second targets that do not meet the preset conditions, the element values ​​corresponding to the unrelated first targets and unrelated second targets in the distance matrix are masked to the maximum value to obtain the masked target distance matrix; wherein, the preset conditions include: the unrelated first targets and unrelated second targets belong to the same category, the absolute value of the difference between the aspect ratios of the unrelated first targets and unrelated second targets is less than a preset threshold, and the distance between the unrelated first targets and unrelated second targets is less than a fourth distance threshold; Based on the target distance matrix, the unassociated first target and the unassociated second target are associated.

[0013] According to the group-structure-based weak registration multimodal remote sensing image target association method provided in this application, the step of associating the unassociated first target and the unassociated second target based on the target distance matrix includes: The target distance matrix is ​​solved using a greedy algorithm to determine the target pair corresponding to the target element value with the smallest distance in the target distance matrix, and to determine that the first target and the second target in the target pair are associated. In the target distance matrix, all other element values ​​in the row where the first target is located in the target pair are masked to maximum values, and all other element values ​​in the column where the second target is located in the target pair are masked to maximum values; The masked target distance matrix is ​​determined as the new target distance matrix, and the target distance matrix is ​​solved based on a greedy algorithm until all element values ​​in the masked target distance matrix are maxima. The maxima are used to characterize that the first target and the second target in the corresponding target pair are not related.

[0014] This application also provides a weakly registered multimodal remote sensing image target association device based on group structure, comprising: An acquisition unit is used to acquire a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes a plurality of first targets, and the second modal remote sensing image includes a plurality of second targets; A clustering unit is used to cluster the plurality of first targets based on their respective locations to obtain a first group structure corresponding to the first modality remote sensing image; and to cluster the plurality of second targets based on their respective locations to obtain a second group structure corresponding to the second modality remote sensing image; The association unit is used to associate the plurality of first targets and the plurality of second targets based on the first group structure and the second group structure.

[0015] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the weak registration multimodal remote sensing image target association method based on group structure as described in any of the preceding claims.

[0016] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the group-structure-based weak registration multimodal remote sensing image target association method as described in any of the preceding claims.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the group-structure-based weak registration multimodal remote sensing image target association method as described in any of the preceding claims.

[0018] The method and apparatus for target association in multimodal remote sensing images based on weak registration using a cluster structure provided in this application acquire a first modal remote sensing image and a second modal remote sensing image. The first modal remote sensing image includes multiple first targets, and the second modal remote sensing image includes multiple second targets. The multiple first targets are clustered based on their respective locations to obtain a first cluster structure corresponding to the first modal remote sensing image. Similarly, the multiple second targets are clustered based on their respective locations to obtain a second cluster structure corresponding to the second modal remote sensing image. Finally, the multiple first targets and multiple second targets are associated based on the first and second cluster structures. This method utilizes the group structure formed by the spatial locations of targets as a more stable and robust matching feature for subsequent target matching and association, rather than relying on unstable individual target locations. This effectively improves the accuracy of target association even in the presence of weak registration. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a weak registration multimodal remote sensing image target association method based on group structure, provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram illustrating a process for associating multiple first targets and multiple second targets based on a first group structure and a second group structure, as provided in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram illustrating a process for associating an unassociated first target in a first modality remote sensing image and an unassociated second target in a second modality remote sensing image, as provided in an embodiment of this application.

[0023] Figure 4 This is a schematic diagram illustrating the visualization of a fusion label result provided in an embodiment of this application.

[0024] Figure 5 This is a schematic diagram of a weakly registered multimodal remote sensing image target association device based on a group structure, provided in an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0028] In multimodal remote sensing image target association scenarios, considering that the acquisition operations of remote sensing images of different modalities are not strictly synchronized, spatial misalignment is a common phenomenon between remote sensing images of different modalities.

[0029] To address this spatial misalignment, existing technologies typically employ image registration algorithms for preprocessing. However, given the limited accuracy of existing registration algorithms and their inability to completely eliminate inherent spatial position deviations caused by target displacement or different imaging modes (i.e., weak registration), the accuracy of target association is low.

[0030] To effectively improve the accuracy of target association in the presence of weak registration, and considering that the macroscopic attributes of a group, such as the number of targets, centroid, and shape, are insensitive to small positional shifts, this application provides a weak registration multimodal remote sensing image target association method based on group structure. This method involves acquiring a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes multiple first targets, and the second modal remote sensing image includes multiple second targets; clustering the multiple first targets based on their respective positions to obtain a first group structure corresponding to the first modal remote sensing image; clustering the multiple second targets based on their respective positions to obtain a second group structure corresponding to the second modal remote sensing image; and then associating the multiple first targets and multiple second targets based on the first and second group structures. This approach utilizes the collective structure formed by the spatial positions of targets as a more stable and robust matching feature for subsequent target matching and association, rather than relying on the unstable positions of individual targets. This effectively improves the accuracy of target association even in the presence of weak registration issues. It is particularly effective in complex scenarios with densely packed targets, significantly enhancing the accuracy and robustness of target association.

[0031] The following detailed embodiments illustrate the weak registration multimodal remote sensing image target association method based on group structure provided in this application. It is understood that these specific embodiments can be combined with each other, and similar concepts or processes may not be repeated in some embodiments.

[0032] It is understood that the execution subject of the group-structure-based weak registration multimodal remote sensing image target association method provided in this application can be a computer, a server, or a specially set group-structure-based weak registration multimodal remote sensing image target association device, or a group-structure-based weak registration multimodal remote sensing image target association device set in the electronic device. The group-structure-based weak registration multimodal remote sensing image target association device can be implemented by software, hardware, or a combination of both, and can be set according to actual needs.

[0033] Figure 1 A flowchart illustrating a weak registration multimodal remote sensing image target association method based on group structure provided in this application embodiment is shown below. For example, please refer to... Figure 1 As shown, this group-structure-based weak registration multimodal remote sensing image target association method may include: S101. Acquire a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes multiple first targets and the second modal remote sensing image includes multiple second targets.

[0034] For example, the first modal remote sensing image can be a SAR remote sensing image, and the second modal remote sensing image can be an RGB remote sensing image. The specific settings can be made according to actual needs. Here, the embodiments of this application are only used as examples of SAR remote sensing images and RGB remote sensing images for illustration, but it does not mean that the embodiments of this application are limited to this.

[0035] Taking the first modality remote sensing image as a SAR remote sensing image and the second modality remote sensing image as an RGB remote sensing image as an example, the SAR remote sensing image corresponds to a target label set. The target tag set This includes the category to which each primary target belongs, the coordinates of its center point, its width, and its height. For example, this target label set... The target can be determined through the target annotation file corresponding to the SAR remote sensing image. In this target annotation file, each first target is defined as a structure containing category, center point coordinates, width, and height. ;in, Indicates the category to which the first target belongs. This indicates the coordinates of the center point of the location of the first target. Indicates the width of the first target. Indicates the altitude of the first target.

[0036] Similarly, an RGB remote sensing image also corresponds to a target label set. The target tag set This includes the category to which each second target belongs, the coordinates of its center point, width, and height. For example, this target label set... The target can be determined through the target annotation file corresponding to the RGB remote sensing image. In the target annotation file, each second target is defined as a structure containing category, center point coordinates, width and height.

[0037] Considering that a group, as a whole, has macroscopic attributes such as the number of targets, centroid, and shape that are not sensitive to small positional shifts, it can effectively solve the weak registration problem existing in the prior art. Therefore, in the embodiments of this application, when associating remote sensing image targets of different modalities, a first group structure corresponding to the first modal remote sensing image is obtained by clustering multiple first targets; and a second group structure corresponding to the second modal remote sensing image is obtained by clustering multiple second targets, i.e., executing the following S102. In this way, the group structure formed by the spatial position of the targets can be used as a more stable and robust matching feature for subsequent target matching and association, instead of relying on the unstable individual target positions. This can effectively improve the accuracy of target association in the presence of weak registration.

[0038] S102. Cluster the multiple first targets based on their respective locations to obtain the first group structure corresponding to the first modality remote sensing image; and cluster the multiple second targets based on their respective locations to obtain the second group structure corresponding to the second modality remote sensing image.

[0039] For example, the first group structure may include not only the first target clusters obtained by clustering multiple first targets, but also scattered first targets that have not been successfully clustered. The specific configuration can be set according to actual needs.

[0040] For example, in addition to the second target clusters obtained by clustering multiple second targets, the second group structure may also include scattered second targets that have not been successfully clustered, which can be set according to actual needs.

[0041] S103. Based on the first group structure and the second group structure, associate multiple first targets and multiple second targets.

[0042] As can be seen from the embodiments of this application, when performing multimodal remote sensing image target association, a first modal remote sensing image and a second modal remote sensing image are acquired. The first modal remote sensing image includes multiple first targets, and the second modal remote sensing image includes multiple second targets. The multiple first targets are clustered based on their respective positions to obtain a first group structure corresponding to the first modal remote sensing image. Similarly, the multiple second targets are clustered based on their respective positions to obtain a second group structure corresponding to the second modal remote sensing image. Then, based on the first and second group structures, the multiple first targets and multiple second targets are associated. This utilizes the group structure formed by the spatial positions of targets as a more stable and robust matching feature for subsequent target matching and association, rather than relying on unstable individual target positions. This effectively improves the accuracy of target association even in the presence of weak registration phenomena.

[0043] Based on the above Figure 1 In the illustrated embodiment, for example, in S102 above, when clustering multiple first targets based on their respective positions to obtain a first group structure corresponding to the first modality remote sensing image, the first targets whose spatial distance is less than a first distance threshold among the multiple first targets can be clustered into the same target cluster based on their respective positions to obtain multiple first target clusters; wherein, the first targets in different first target clusters do not overlap, and the first group structure includes multiple first target clusters.

[0044] The value of the first distance threshold can be set according to actual needs, and this application embodiment does not impose further restrictions.

[0045] The first targets in different first target clusters do not overlap. This can be understood as meaning that for the same first target, it can only be clustered into one first target cluster, that is, it cannot be clustered into different first target clusters at the same time.

[0046] For example, in an embodiment of this application, when clustering multiple first targets based on their respective locations, a target label set corresponding to the first modality remote sensing image can be extracted. The point set corresponding to the first modality remote sensing image, formed by the coordinates of the center points of all the first targets. and for point sets The density-based spatial clustering algorithm DBSCAN is executed. Based on the positions of multiple first targets, points with a spatial distance less than a first distance threshold are connected to form density-connected regions. This clustering is used to obtain multiple first target clusters, which can be denoted as the SAR target cluster set. , where the target cluster set Each element in the array is a first target cluster, for example Represents the first in a plurality of first target clusters The first target cluster.

[0047] Based on the individual locations of multiple first targets, the first targets whose spatial distance is less than a first distance threshold are clustered into the same target cluster. After obtaining multiple first target clusters, the first group structure can be obtained. For example, in addition to the first target clusters obtained by clustering multiple first targets, the first group structure may also include scattered first targets that were not successfully clustered. The specific settings can be configured according to actual needs.

[0048] For example, in S102 above, when clustering multiple second targets based on their respective positions to obtain the second group structure corresponding to the second modality remote sensing image, the second targets whose spatial distance is less than a second distance threshold can be clustered into the same target cluster based on their respective positions to obtain multiple second target clusters; wherein, the second targets in different second target clusters do not overlap, and the second group structure includes multiple second target clusters.

[0049] The value of the second distance threshold can be set according to actual needs, and this application embodiment does not impose further restrictions.

[0050] The second objectives in different second objective clusters do not overlap. This can be understood as meaning that for the same second objective, it can only be clustered into one second objective cluster, that is, it cannot be clustered into different second objective clusters at the same time.

[0051] For example, in the embodiments of this application, when clustering multiple second targets based on their respective locations, the target label set corresponding to the second modality remote sensing image can be extracted. The point set corresponding to the second modality remote sensing image, formed by the coordinates of the center points of all the second targets. and for point sets The density-based spatial clustering algorithm DBSCAN is executed. Based on the positions of multiple second targets, points with a spatial distance less than a second distance threshold are connected to form density-connected regions. This allows for the clustering of multiple second targets, resulting in multiple second target clusters. The set of these multiple second target clusters can be denoted as the RGB target cluster set. , where the target cluster set Each element in the array is a second target cluster, for example Represents the first in multiple second target clusters A second target cluster.

[0052] Based on the individual locations of multiple second targets, second targets whose spatial distance is less than a second distance threshold are clustered into the same target cluster. After obtaining multiple second target clusters, a second group structure can be obtained. For example, in addition to the second target clusters obtained by clustering multiple second targets, the second group structure may also include scattered second targets that were not successfully clustered. The specific settings can be configured according to actual needs.

[0053] Based on the above embodiments, for example, in S103 above, when associating multiple first targets and multiple second targets based on the first group structure and the second group structure, the first group structure and the second group structure can be matched, that is, matched between cluster sets of different modalities, i.e., matching at the group structure level. For specific implementation details, please refer to the following... Figure 2 The example shown.

[0054] Figure 2 This application provides a schematic flowchart illustrating the association of multiple first targets and multiple second targets based on a first group structure and a second group structure. For example, please refer to [link to relevant documentation]. Figure 2 As shown, the method may include: S201. Traverse multiple first target clusters. For the first target cluster currently traversed, determine the distance between the centroid of the currently traversed first target cluster and the centroid of each of the multiple remaining second target clusters. The remaining second target clusters are the target clusters among the multiple second target clusters that have not been successfully matched with the traversed first target clusters.

[0055] It should be noted that during the target cluster matching process, the remaining second target clusters are continuously updated. In each target cluster matching operation, if a second target cluster successfully matches a first target cluster, that successfully matched second target cluster needs to be removed from the remaining second target clusters to update the remaining second target clusters after this matching operation. It can be understood that when the currently executed target cluster matching operation is the first target cluster matching operation, the remaining second target clusters are the entirety of the second target clusters.

[0056] For example, the centroid of a first target cluster can be defined as the arithmetic mean of the coordinates of the center points of all first targets within the first target cluster. Similarly, the centroid of a second target cluster can be defined as the arithmetic mean of the coordinates of the center points of all second targets within the second target cluster.

[0057] The first target cluster encountered in the current traversal is taken as the first target cluster. For example, the Euclidean distance between the centroids of the target clusters can be used as the matching criterion to calculate the first target cluster. The Euclidean distances between the centroids of the first target cluster and the centroids of the remaining second target clusters are calculated. Assuming the calculation of the first target cluster... The centroid and the remaining second target cluster The Euclidean distance between the centroids of the two points is denoted as . .

[0058] Combining with S201 above, by calculating the distances between the centroid of the currently traversed first target cluster and the centroids of the remaining second target clusters, multiple distances can be obtained. It is then determined whether any of these distances contains a target distance less than a third distance threshold. If no target distance less than the third distance threshold exists, it can be understood that the currently traversed first target cluster and the remaining second target clusters have all failed to match. In this case, the traversal continues to the next first target cluster. If a target distance less than the third distance threshold exists, the following S202 is executed: S202. If there is a target distance less than the third distance threshold among multiple distances, the remaining second target cluster corresponding to the target distance and the currently traversed first target cluster are determined as a successfully matched target cluster pair.

[0059] The value of the third distance threshold can be set according to actual needs, and this application embodiment does not impose further restrictions.

[0060] Assuming the first target cluster encountered during the current traversal The centroid and the remaining second target cluster If the Euclidean distance between the centroids is minimized and less than the third distance threshold, then the first target cluster currently being traversed can be determined. With the second target cluster For example, a successfully matched target cluster pair can be denoted as: The successfully matched target cluster pair can be added to the candidate matching cluster pair set. middle.

[0061] Combining the above S201-S202, by traversing multiple first target clusters and matching the currently traversed first target cluster with the remaining second target clusters, multiple target cluster pairs can be obtained.

[0062] For example, if the first target cluster currently being traversed does not match any of the remaining second target clusters, the first target in the currently traversed first target cluster can be recorded as a scattered first target that has not successfully matched with a second target cluster; or, if the remaining second target cluster does not match any of the first target clusters, the second target in the remaining second target cluster can be recorded as a scattered second target that has not successfully matched with a first target cluster. The specific settings can be configured according to actual needs.

[0063] S203. Associatively link multiple first targets and multiple second targets in multiple target clusters.

[0064] For example, in the embodiments of this application, multiple first targets and multiple second targets in multiple target cluster pairs are associated, that is, when targets within a target cluster are accurately associated, for each target cluster pair, an inter-cluster cost matrix corresponding to the target cluster pair can be constructed based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs; and the inter-cluster cost matrix is ​​solved based on the Hungarian matching algorithm to obtain the association result between the first target and the second target in the target cluster pair.

[0065] For example, in the embodiments of this application, when constructing the inter-cluster cost matrix corresponding to the target cluster pair based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs, for any first target and second target in the target cluster pair, if the category to which the first target belongs is the same as the category to which the second target belongs, the distance between the first target and the second target can be determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix; if the category to which the first target belongs is different from the category to which the second target belongs, the maximum value can be determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix, thus obtaining the inter-cluster cost matrix corresponding to the target cluster pair.

[0066] With target cluster pair For example, based on the first target cluster The category to which the first target belongs and the cluster of the second target. Construct the target cluster pair according to the category to which the second target belongs. For example, when dealing with the inter-cluster cost matrix, the target cluster pair The corresponding inter-cluster cost matrix can be denoted as: If the first target cluster The Middle The category to which the first objective belongs and the cluster of the second objective The Middle If the two second objectives belong to the same category, then the inter-cluster cost matrix... The Middle Line number Column elements The element value is the first The first goal and the first The Euclidean distance between the second targets; conversely, if the first target cluster... The Middle The category to which the first objective belongs and the cluster of the second objective The Middle If the two second objectives belong to the same category, then the inter-cluster cost matrix... The Middle Line number Column elements The element value is a maximum value, used to represent the first... The first goal and the first The second objective is not related.

[0067] For example, the inter-cluster cost matrix is ​​based on the Hungarian matching algorithm. The solution is performed to find the one-to-one allocation scheme corresponding to the inter-cluster cost matrix, with the objective of minimizing the total cost of the distance between different target pairs.

[0068] As can be seen, in the embodiments of this application, when associating multiple first targets and multiple second targets based on the first group structure and the second group structure, the multiple first target clusters can be traversed firstly. For the currently traversed first target cluster, the distance between the centroid of the currently traversed first target cluster and the centroid of each of the multiple remaining second target clusters can be determined. If there is a target distance less than a third distance threshold among the multiple distances, the remaining second target clusters corresponding to the target distances and the currently traversed first target clusters are determined as successfully matched target cluster pairs. Then, the multiple first targets and multiple second targets in the multiple target cluster pairs are associated. In this way, matching and association are performed using the group structure information of the targets, instead of relying on the unstable individual target positions. Moreover, the macroscopic attributes of the group as a whole, such as the number of targets, centroid, and shape, are not sensitive to small positional shifts. This can effectively solve the weak registration problem existing in the prior art, and effectively improve the accuracy and robustness of target association in the presence of weak registration phenomena.

[0069] Based on the above embodiments, it is easy to see that in the multi-target matching process, the association between the first target cluster in the first group structure and the second target cluster in the second group structure is performed on a target cluster basis. However, in the above matching process, there may be unassociated first targets. Taking an unassociated first target as an example, this unassociated first target can be a first target in the first group structure that has not been successfully clustered, a first target in a first target cluster that has not been successfully matched with the second target cluster, or a first target in a target cluster pair that has not been successfully associated with the second target, etc., which can be set according to actual needs.

[0070] The following section will describe in detail the association process between unassociated first targets in the first modality remote sensing image and unassociated second targets in the second modality remote sensing image. For details, please refer to the following... Figure 3 The example shown.

[0071] Figure 3This application provides a schematic flowchart for associating an unassociated first target in a first modality remote sensing image and an unassociated second target in a second modality remote sensing image. For example, please refer to [link to relevant documentation]. Figure 3 As shown, the method may include: S301. If there is an unassociated first target in the first modal remote sensing image and an unassociated second target in the second modal remote sensing image, determine the distance between any two unassociated first targets and unassociated second targets.

[0072] The unassociated first target includes at least one of the following: The first target that was not successfully clustered in the first group structure; The first target in the first target cluster that failed to match the second target cluster; The first target in the target cluster that was not successfully associated with the second target; Unrelated second objectives include at least one of the following: The second objective that was not successfully clustered in the second group structure; The second target in the second target cluster that failed to match the first target cluster; The second target in the target cluster that was not successfully associated with the first target.

[0073] Taking the unassociated first target as an example, the unclustered first target included in the first group structure refers to the scattered first targets that were not successfully clustered in S102; the first target in the first target cluster that was not successfully matched with the second target cluster refers to the scattered first targets in the first target cluster that were not successfully matched with the second target cluster in S202; the first target in the target cluster pair that was not successfully associated with the second target refers to the first target that was not successfully associated with the second target solved based on the Hungarian matching algorithm in S203.

[0074] For example, when determining the distance between any two unrelated first targets and unrelated second targets, the distance between any two unrelated first targets and unrelated second targets can be calculated based on the center coordinates of the targets.

[0075] S302. Associate the unassociated first target and the unassociated second target based on the distance between any two unassociated first targets and unassociated second targets.

[0076] For example, in this embodiment of the application, when associating unassociated first targets and unassociated second targets based on the distance between any two unassociated first targets and unassociated second targets, a corresponding distance matrix, such as the Intersection over Union (IOU) matrix, can be constructed first based on the distance between any two unassociated first targets and unassociated second targets. In order to improve the accuracy of matching, after constructing the distance matrix, it can be determined whether the unassociated first targets and unassociated second targets meet the preset conditions in order to mask the distance matrix.

[0077] If there are unassociated first targets and unassociated second targets that do not meet the preset conditions, the element values ​​corresponding to the unassociated first targets and unassociated second targets in the distance matrix are masked to the maximum value to obtain the masked target distance matrix. The preset conditions include: the unassociated first targets and unassociated second targets belong to the same category, the absolute value of the difference between the aspect ratios of the unassociated first targets and unassociated second targets is less than a preset threshold, and the distance between the unassociated first targets and unassociated second targets is less than a fourth distance threshold. Based on the target distance matrix, the unassociated first targets and unassociated second targets are associated.

[0078] The values ​​of the preset threshold and the fourth distance threshold can be set according to actual needs. In this embodiment, no further restrictions are imposed.

[0079] Based on the masked target distance matrix, a greedy algorithm can be used for matching. This involves iteratively searching for and selecting the target pair with the smallest distance in the target distance matrix, and determining the association between the first and second targets in that target pair. Specifically, this includes: when associating an unassociated first target with an unassociated second target based on the target distance matrix, firstly, solving the target distance matrix using a greedy algorithm to determine the target pair corresponding to the smallest distance element value, and then determining the association between the first and second targets in that target pair; masking all other element values ​​in the row containing the first target in the target distance matrix as maxima, and masking all other element values ​​in the column containing the second target in the target pair as maxima; then determining the masked target distance matrix as a new target distance matrix, and solving the target distance matrix using a greedy algorithm until all element values ​​in the masked target distance matrix are maxima. These maxima represent that the first and second targets in the corresponding target pair are not associated.

[0080] For example, when using a greedy algorithm for matching, and iteratively searching for and selecting the target pair with the smallest distance in the target distance matrix, in addition to processing the target distance matrix by using a mask with a maximum value, you can also delete the rows and columns corresponding to the target pair with the smallest distance from the target distance matrix until the row or column is empty.

[0081] As can be seen, in this embodiment, after associating the first target cluster in the first group structure and the second target cluster in the second group structure with the target cluster as the unit, for the discrete first and second targets that have not been successfully associated, the distance between any two unassociated first targets and unassociated second targets can be determined first; and based on the distance between any two unassociated first targets and unassociated second targets, the unassociated first targets and unassociated second targets can be associated. In this way, the hierarchical matching strategy of "group first, individual later" can first determine the correspondence of most targets at a high level using strong constraints (group structure), and then process the remaining discrete targets, which effectively reduces the ambiguity in the matching process and avoids the possible misassignment in the traditional global matching method.

[0082] It is not difficult to see that, in the embodiments of this application, the weak registration multimodal remote sensing image target association method based on group structure provided in this application can be used to associate targets in different modalities. The results obtained can include: targets associated through group structure, targets associated with discrete targets, and targets unique to different modalities.

[0083] For example, after associating targets in different modalities as described above, the bounding boxes of the targets can also be fused. Taking the fusion of the bounding boxes of a first target in a SAR remote sensing image as an example, the initially predicted bounding box of the first target in the SAR remote sensing image can be fused based on the associated and labeled bounding box of the first target. A new bounding box is generated by taking the circumscribed rectangle to label the first target. In addition, the location of the first target can be updated based on the new bounding box, and the fused new bounding box is added to the final fused label set. middle.

[0084] By generating a new bounding box using the circumscribed rectangle method for object labeling, better labels can be generated for multi-source object detection tasks. This method can be applied to object detection model training, inference, and validation scenarios. When applied to object detection model training, it can guide a more accurate model training process; when applied to object detection inference, it can optimize object detection performance; and when applied to object detection validation, it can optimize evaluation metrics.

[0085] For example, see Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the result visualization of fused labels provided in an embodiment of this application. It can be seen that, for baseline methods (such as the IoU fusion algorithm), since only the relationship between individual target labels can be utilized, a large number of redundant labels are generated, that is, many targets are not correctly associated; while the weak registration multimodal remote sensing image target association method based on group structure provided in this application can achieve effective target association.

[0086] Furthermore, to illustrate the effectiveness of the group-structure-based weak registration multimodal remote sensing image target association method provided in this application for detection tasks, please refer to Table 1, which presents quantitative experimental results on a dataset of 1092 SAR-RGB fusion detection images.

[0087] Table 1

[0088] Where P represents accuracy, R represents recall, and mAP represents mean precision.

[0089] As can be seen from Table 1, when using SAR single-source label, RGB single-source label, and the fusion label proposed in this application as ground values, the weak registration multimodal remote sensing image target association method based on group structure provided in this application can effectively enhance the performance of multi-source target detection algorithms, thereby verifying the effectiveness of the weak registration multimodal remote sensing image target association method based on group structure provided in this application.

[0090] The following describes the target association device for weakly registered multimodal remote sensing images based on group structure provided in this application. The target association device for weakly registered multimodal remote sensing images based on group structure described below can be referred to in correspondence with the target association method for weakly registered multimodal remote sensing images based on group structure described above.

[0091] Figure 5 A schematic diagram of a weakly registered multimodal remote sensing image target association device based on group structure is provided in this application embodiment. For example, please refer to [link to relevant documentation]. Figure 5 As shown, the weak registration multimodal remote sensing image target association device 50 based on group structure may include: The acquisition unit 501 is used to acquire a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes a plurality of first targets and the second modal remote sensing image includes a plurality of second targets; Clustering unit 502 is used to cluster the plurality of first targets based on their respective positions to obtain a first group structure corresponding to the first modality remote sensing image; and to cluster the plurality of second targets based on their respective positions to obtain a second group structure corresponding to the second modality remote sensing image; The association unit 503 is used to associate the plurality of first targets and the plurality of second targets based on the first group structure and the second group structure.

[0092] For example, in an embodiment of this application, the clustering unit 502 is used to cluster the plurality of first targets based on their respective locations to obtain a first group structure corresponding to the first modality remote sensing image, including: Based on the positions of the plurality of first targets, the first targets whose spatial distance is less than a first distance threshold are clustered into the same target cluster to obtain a plurality of first target clusters; wherein, the first targets in different first target clusters do not overlap, and the first group structure includes the plurality of first target clusters; The clustering unit 502 is used to cluster the plurality of second targets based on their respective positions to obtain a second group structure corresponding to the second modality remote sensing image, including: Based on the respective positions of the plurality of second targets, the second targets whose spatial distance is less than a second distance threshold are clustered into the same target cluster to obtain a plurality of second target clusters; wherein, the second targets in different second target clusters do not overlap, and the second group structure includes the plurality of second target clusters.

[0093] For example, in an embodiment of this application, the association unit 503 is used to associate the plurality of first targets and the plurality of second targets based on the first group structure and the second group structure, including: The plurality of first target clusters are traversed, and for the currently traversed first target cluster, the distance between the centroid of the currently traversed first target cluster and the centroid of each of the plurality of remaining second target clusters is determined; wherein, the remaining second target clusters are the target clusters among the plurality of second target clusters that have not been successfully matched with the traversed first target clusters; If, among multiple distances, there exists a target distance less than a third distance threshold, the remaining second target cluster corresponding to the target distance and the currently traversed first target cluster are determined as a successfully matched target cluster pair; Associativity between multiple first targets and multiple second targets in multiple target clusters.

[0094] For example, in an embodiment of this application, the association unit 503 is used to associate multiple first targets and multiple second targets in multiple target cluster pairs, including: For each target cluster pair, based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs, an inter-cluster cost matrix corresponding to the target cluster pair is constructed; The inter-cluster cost matrix is ​​solved using the Hungarian matching algorithm to obtain the association results between the first target and the second target in the target cluster pair.

[0095] For example, in an embodiment of this application, the association unit 503 is configured to construct an inter-cluster cost matrix corresponding to the target cluster pair based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs, including: For any first target and second target in the target cluster pair, if the category to which the first target belongs is the same as the category to which the second target belongs, the distance between the first target and the second target is determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix; If the category to which the first target belongs is different from the category to which the second target belongs, the maximum value is determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix.

[0096] For example, in an embodiment of this application, the weak registration multimodal remote sensing image target association device 50 based on group structure further includes: The determining unit is configured to determine the distance between any two unassociated first targets and unassociated second targets when there are unassociated first targets in the first modal remote sensing image and unassociated second targets in the second modal remote sensing image. The association unit 503 is further configured to associate the unassociated first target with the unassociated second target based on the distance between any two unassociated first targets and unassociated second targets; The unassociated first target includes at least one of the following: The first target that was not successfully clustered in the first group structure; The first target in the first target cluster that did not successfully match the second target cluster; The first target in the target cluster pair that was not successfully associated with the second target; The unrelated second objective includes at least one of the following: The second group structure includes second targets that were not successfully clustered; The second target in the second target cluster that failed to match the first target cluster; The second target in the target cluster pair that was not successfully associated with the first target.

[0097] For example, in an embodiment of this application, the association unit 503 is used to associate the unassociated first target and the unassociated second target based on the distance between any two unassociated first targets and unassociated second targets, including: Based on the distance between any two unrelated first targets and unrelated second targets, construct the corresponding distance matrix; If there are unrelated first targets and unrelated second targets that do not meet the preset conditions, the element values ​​corresponding to the unrelated first targets and unrelated second targets in the distance matrix are masked to the maximum value to obtain the masked target distance matrix; wherein, the preset conditions include: the unrelated first targets and unrelated second targets belong to the same category, the absolute value of the difference between the aspect ratios of the unrelated first targets and unrelated second targets is less than a preset threshold, and the distance between the unrelated first targets and unrelated second targets is less than a fourth distance threshold; Based on the target distance matrix, the unassociated first target and the unassociated second target are associated.

[0098] For example, in an embodiment of this application, the association unit 503 is used to associate the unassociated first target with the unassociated second target based on the target distance matrix, including: The target distance matrix is ​​solved using a greedy algorithm to determine the target pair corresponding to the target element value with the smallest distance in the target distance matrix, and to determine that the first target and the second target in the target pair are associated. In the target distance matrix, all other element values ​​in the row where the first target is located in the target pair are masked to maximum values, and all other element values ​​in the column where the second target is located in the target pair are masked to maximum values; The masked target distance matrix is ​​determined as the new target distance matrix, and the target distance matrix is ​​solved based on a greedy algorithm until all element values ​​in the masked target distance matrix are maxima. The maxima are used to characterize that the first target and the second target in the corresponding target pair are not related.

[0099] The weak registration multimodal remote sensing image target association device 50 based on group structure provided in this application embodiment can execute the technical solution of the weak registration multimodal remote sensing image target association method based on group structure in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the weak registration multimodal remote sensing image target association method based on group structure described above. Please refer to the implementation principle and beneficial effects of the weak registration multimodal remote sensing image target association method based on group structure described above. It will not be repeated here.

[0100] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a weak registration multimodal remote sensing image target association method based on cluster structure. The method includes: acquiring a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes multiple first targets, and the second modal remote sensing image includes multiple second targets; clustering the multiple first targets based on their respective positions to obtain a first cluster structure corresponding to the first modal remote sensing image; and clustering the multiple second targets based on their respective positions to obtain a second cluster structure corresponding to the second modal remote sensing image; and associating the multiple first targets and the multiple second targets based on the first cluster structure and the second cluster structure.

[0101] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the weak registration multimodal remote sensing image target association method based on cluster structure provided by the above methods. The method includes: acquiring a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes a plurality of first targets, and the second modal remote sensing image includes a plurality of second targets; clustering the plurality of first targets based on their respective positions to obtain a first cluster structure corresponding to the first modal remote sensing image; and clustering the plurality of second targets based on their respective positions to obtain a second cluster structure corresponding to the second modal remote sensing image; and associating the plurality of first targets and the plurality of second targets based on the first cluster structure and the second cluster structure.

[0103] In another aspect, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for weakly registered multimodal remote sensing image target association based on cluster structure provided by the methods described above. The method includes: acquiring a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes a plurality of first targets, and the second modal remote sensing image includes a plurality of second targets; clustering the plurality of first targets based on their respective positions to obtain a first cluster structure corresponding to the first modal remote sensing image; clustering the plurality of second targets based on their respective positions to obtain a second cluster structure corresponding to the second modal remote sensing image; and associating the plurality of first targets and the plurality of second targets based on the first cluster structure and the second cluster structure.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A target association method for weakly registered multimodal remote sensing images based on group structure, characterized in that, include: Acquire a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes a plurality of first targets, and the second modal remote sensing image includes a plurality of second targets; Based on the respective positions of the plurality of first targets, the plurality of first targets are clustered to obtain a first group structure corresponding to the first modality remote sensing image; and based on the respective positions of the plurality of second targets, the plurality of second targets are clustered to obtain a second group structure corresponding to the second modality remote sensing image; Based on the first group structure and the second group structure, the plurality of first targets and the plurality of second targets are associated.

2. The method according to claim 1, characterized in that, include: The step of clustering the plurality of first targets based on their respective locations to obtain a first group structure corresponding to the first modality remote sensing image includes: Based on the positions of the plurality of first targets, the first targets whose spatial distance is less than a first distance threshold are clustered into the same target cluster to obtain a plurality of first target clusters; wherein, the first targets in different first target clusters do not overlap, and the first group structure includes the plurality of first target clusters; The step of clustering the multiple second targets based on their respective locations to obtain a second group structure corresponding to the second modality remote sensing image includes: Based on the respective positions of the plurality of second targets, the second targets whose spatial distance is less than a second distance threshold are clustered into the same target cluster to obtain a plurality of second target clusters; wherein, the second targets in different second target clusters do not overlap, and the second group structure includes the plurality of second target clusters.

3. The method according to claim 2, characterized in that, The association of the plurality of first targets and the plurality of second targets based on the first group structure and the second group structure includes: The plurality of first target clusters are traversed, and for the currently traversed first target cluster, the distance between the centroid of the currently traversed first target cluster and the centroid of each of the plurality of remaining second target clusters is determined; wherein, the remaining second target clusters are the target clusters among the plurality of second target clusters that have not been successfully matched with the traversed first target clusters; If, among multiple distances, there exists a target distance less than a third distance threshold, the remaining second target cluster corresponding to the target distance and the currently traversed first target cluster are determined as a successfully matched target cluster pair; Associativity between multiple first targets and multiple second targets in multiple target clusters.

4. The method according to claim 3, characterized in that, The association of multiple first targets and multiple second targets in multiple target clusters includes: For each target cluster pair, based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs, an inter-cluster cost matrix corresponding to the target cluster pair is constructed; The inter-cluster cost matrix is ​​solved using the Hungarian matching algorithm to obtain the association results between the first target and the second target in the target cluster pair.

5. The method according to claim 4, characterized in that, constructing the inter-cluster cost matrix corresponding to the target cluster pair based on the category to which the first target in the target cluster pair belongs and the category to which the second target in the target cluster pair belongs includes: For any first target and second target in the target cluster pair, if the category to which the first target belongs is the same as the category to which the second target belongs, the distance between the first target and the second target is determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix; If the category to which the first target belongs is different from the category to which the second target belongs, the maximum value is determined as the element value corresponding to the first target and the second target in the inter-cluster cost matrix.

6. The method according to any one of claims 3-5, characterized in that, The method further includes: If there is an unassociated first target in the first modal remote sensing image and an unassociated second target in the second modal remote sensing image, determine the distance between any two unassociated first targets and unassociated second targets; Associat the unassociated first target and the unassociated second target based on the distance between any two unassociated first targets and unassociated second targets; The unassociated first target includes at least one of the following: The first target that was not successfully clustered in the first group structure; The first target in the first target cluster that did not successfully match the second target cluster; The first target in the target cluster pair that was not successfully associated with the second target; The unrelated second objective includes at least one of the following: The second group structure includes second targets that were not successfully clustered; The second target in the second target cluster that failed to match the first target cluster; The second target in the target cluster pair that was not successfully associated with the first target.

7. The method according to claim 6, characterized in that, Associating the unassociated first target and the unassociated second target based on the distance between any two unassociated first targets and unassociated second targets includes: Based on the distance between any two unrelated first targets and unrelated second targets, construct the corresponding distance matrix; If there are unrelated first targets and unrelated second targets that do not meet the preset conditions, the element values ​​corresponding to the unrelated first targets and unrelated second targets in the distance matrix are masked to the maximum value to obtain the masked target distance matrix; wherein, the preset conditions include: the unrelated first targets and unrelated second targets belong to the same category, the absolute value of the difference between the aspect ratios of the unrelated first targets and unrelated second targets is less than a preset threshold, and the distance between the unrelated first targets and unrelated second targets is less than a fourth distance threshold; Based on the target distance matrix, the unassociated first target and the unassociated second target are associated.

8. The method according to claim 7, characterized in that, The step of associating the unassociated first target and the unassociated second target based on the target distance matrix includes: The target distance matrix is ​​solved using a greedy algorithm to determine the target pair corresponding to the target element value with the smallest distance in the target distance matrix, and to determine that the first target and the second target in the target pair are associated. In the target distance matrix, all other element values ​​in the row where the first target is located in the target pair are masked to maximum values, and all other element values ​​in the column where the second target is located in the target pair are masked to maximum values; The masked target distance matrix is ​​determined as the new target distance matrix, and the target distance matrix is ​​solved based on a greedy algorithm until all element values ​​in the masked target distance matrix are maxima. The maxima are used to characterize that the first target and the second target in the corresponding target pair are not related.

9. A target association device for weakly registered multimodal remote sensing images based on group structure, characterized in that, include: An acquisition unit is used to acquire a first modal remote sensing image and a second modal remote sensing image; wherein the first modal remote sensing image includes a plurality of first targets, and the second modal remote sensing image includes a plurality of second targets; A clustering unit is used to cluster the plurality of first targets based on their respective locations to obtain a first group structure corresponding to the first modality remote sensing image; and to cluster the plurality of second targets based on their respective locations to obtain a second group structure corresponding to the second modality remote sensing image; The association unit is used to associate the plurality of first targets and the plurality of second targets based on the first group structure and the second group structure.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the weak registration multimodal remote sensing image target association method based on group structure as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the weak registration multimodal remote sensing image target association method based on group structure as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the weak registration multimodal remote sensing image target association method based on group structure as described in any one of claims 1 to 8.

Images