Remote sensing satellite image ship target association method and system

Abstract

The invention provides a remote sensing satellite image ship target association method and system, and relates to the technical field of satellite remote sensing image processing, and the method comprises the steps: S1, constructing a ship multi-source satellite remote sensing image training sample set; s2, based on a deep convolutional neural network, learning coupling and correlation of multi-source satellite remote sensing images, mapping different types of ship target satellite remote sensing images to a common depth feature space, and learning in the depth feature space to obtain a metric function; s3, inputting a ship target multi-source satellite remote sensing image test set into the depth measurement learning network obtained by training, and carrying out feature similarity measurement calculation; s4, designing an association cost matrix ACM; and S5, through a binary discrete particle swarm optimization algorithm DPSO, acceleration association cost matrix ACM optimal scheme solving is carried out, and multi-source remote sensing image ship multi-target association is carried out. According to the method, fuzziness caused by single feature association can be reduced, and the accuracy of ship multi-target association is improved.

Description

Technical Field

[0001] This invention relates to the field of satellite remote sensing image processing technology, specifically to a method for associating ship targets in remote sensing satellite images based on multi-dimensional feature fusion and matching, and more particularly to a method and system for associating ship targets in remote sensing satellite images. Background Technology

[0002] The technology for associating ship targets with remote sensing satellite imagery faces multidimensional and complex challenges, primarily in the fusion of spatiotemporal features and heterogeneous features. From the perspective of data acquisition characteristics, remote sensing satellites, constrained by orbital periods and payload modes, exhibit significant non-uniform sampling characteristics: First, optical satellites have long revisit periods, resulting in significant temporal discontinuities between adjacent passing images, leading to fragmented ship trajectories. Traditional association methods based on kinematic modeling (such as Kalman filtering and trajectory extrapolation) fail under sparse observation conditions, especially when ships perform nonlinear maneuvers such as speed changes and turns, where the extrapolation error of motion features grows exponentially. Second, the spatiotemporal and resolution differences of heterogeneous sensors during multi-satellite collaborative observation exacerbate the uncertainty of motion feature extraction. For example, the imaging time difference between SAR satellites and optical satellites can be more than 30 minutes, during which time a ship may complete a displacement of 5-10 nautical miles, causing inaccurate cross-modal motion feature matching. Furthermore, when ship targets operate in formation, multiple similar targets of interest exist in the same area, and using only target image features as the association quantity can easily lead to erroneous associations.

[0003] Existing related technologies include: Patent CN118536549A, which discloses a data association model construction method, association method, device, equipment, and medium. This method constructs a target data association model including two sub-models, obtaining efficient and accurate association results between remote sensing images and AIS data, thus reducing processing costs. However, its application scenarios are narrow, and it is not suitable for multi-source remote sensing image association using optical SAR.

[0004] The invention patent with publication number CN119313879A discloses a ship target detection method based on an attention-enhanced feature pyramid network. It fully utilizes the ability of the attention mechanism to more comprehensively capture global semantic information associations, enabling the model to extract data features more comprehensively in remote sensing image ship target detection tasks. However, this method involves complex training set construction and stringent requirements for sample preprocessing.

[0005] The invention patent with publication number CN118941965A discloses a method and system for ship wake detection based on multi-dimensional feature collaborative fusion. It utilizes multi-dimensional feature collaborative fusion in DML to solve the problem of inconsistent feature layers in the feature pyramid of existing single-stage target detection technologies, achieving better feature fusion results. However, this method still has high requirements for the training dataset and incurs high training costs.

[0006] The invention patent with publication number CN118228195A discloses a method for analyzing maritime ship targets based on the fusion of image and electronic detection. This method merges electronic detection identification data and ship target identification data that satisfy spatiotemporal and ID constraints into correlated data, and then uses a weighted average and linear equation to predict the ship's trajectory, effectively improving the positioning accuracy of maritime ship targets. However, this kinematic modeling-based prediction method fails under sparse observation conditions, especially when the ship performs nonlinear maneuvers such as speed changes and turns, where the extrapolation error of motion characteristics increases exponentially, significantly reducing accuracy.

[0007] Patent CN116188519A discloses a method and system for estimating the motion state of ship targets based on video satellites. This method associates targets based on ship size and position parameters to obtain an estimated ship heading, ultimately estimating the ship's motion state. However, this method is not suitable for scenarios involving ship formation operations. When multiple similar targets of interest exist in the same area, relying solely on target image features as the association parameter can easily lead to incorrect associations. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a method and system for associating ship targets with remote sensing satellite images.

[0009] According to the present invention, a method and system for associating ship targets in remote sensing satellite images are provided, the scheme of which is as follows:

[0010] Firstly, a method for associating ship targets with remote sensing satellite images is provided, the method comprising:

[0011] Step S1: Construct a training sample set of multi-source satellite remote sensing images of ships;

[0012] Step S2: Based on deep convolutional neural networks, learn the coupling and correlation of multi-source satellite remote sensing images, map different types of ship target satellite remote sensing images to a common deep feature space, and learn the metric function in this deep feature space;

[0013] Step S3: Input the test set of multi-source satellite remote sensing images of ship targets into the trained deep metric learning network to calculate feature similarity.

[0014] Step S4: Design the Association Cost Matrix (ACM);

[0015] Step S5: Accelerate the solution of the optimal solution of the association cost matrix ACM by using the binary discrete particle swarm optimization algorithm DPSO, and perform multi-target association of ships in multi-source remote sensing images.

[0016] Preferably, in step S1, the sample set is constructed online, with positive sample pairs consisting of different images of the same type of ship and negative sample pairs consisting of two images of different types of ships. Furthermore, each training batch generates positive sample pairs to negative sample pairs in a 1:1 ratio. The implementation steps include:

[0017] Step S1.1: Count the number of samples N for each category. c c = 1, 2, ..., C, and reclassified according to the number of samples from largest to smallest: N1 > N2 > ... > N C Each category of sample is randomly assigned a unique number, with the number values ​​being 1, 2, 3, ..., Nc.

[0018] Step S1.2: Determine the number of training batches M, let the category included in a single training batch be C, let the number of positive sample pairs in a single training batch be L, L>C, and the number of negative sample pairs be L, then the total number of training samples is 2×L×M.

[0019] Step S1.3: Construct positive sample pairs, repeatedly drawing one category L times from C categories. After each category c is drawn, select its corresponding N... c N samples were randomly selected from the samples. c / 2 samples, and calculate the distance between their features learned by the network, and select the pair of samples with the largest distance as the positive sample pair;

[0020] Step S1.4: Construct negative sample pairs, calculate the feature centers of C classes respectively, and calculate the distances between them and arrange them in ascending order. First, select the two classes c and d with the smallest feature center distance, and then select them from their N... c One sample and N d For each sample, one sample is randomly selected to form a negative sample pair. The above process is repeated in ascending order of feature center distance until L pairs of negative samples are generated.

[0021] Preferably, in step S2, the convolutional neural network model used is VGGNet with weight sharing, and the learning ability of the network is improved by repeatedly stacking 3×3 convolutional kernels and 2×2 max pooling layers.

[0022] The distance between feature vectors is defined using Euclidean distance:

[0023] Suppose the mapping functions of the first convolutional neural network and the second convolutional neural network are f, respectively. θ (x) and θ and These represent the weight matrices and bias vectors of the two convolutional neural networks, respectively.

[0024] Using a weight-sharing, deep convolutional network with a parameter set of θ, the similarity between image samples x and y is represented as:

[0025]

[0026] d i,j This represents the distance between the features obtained after image samples x and y are mapped by a Siamese convolutional neural network. At this point, deep metric learning changes from learning the metric matrix or linear transformation matrix using traditional methods to learning the weights and bias coefficients of a deep convolutional neural network.

[0027] Preferably, in step S2, the loss function of deep metric learning includes the loss of all positive and negative sample pairs in the training dataset, where positive sample pairs are used to reduce intra-class distance and negative sample pairs are used to increase inter-class distance.

[0028] The loss functions for deep distance metric learning mainly include the contrastive loss function and the triplet loss function. The contrastive loss function minimizes the distance between positive sample pairs of the same class and penalizes the distance between negative sample pairs that are less than a certain threshold. It is defined as follows:

[0029]

[0030] In the formula, l i,j ∈{0,1} represents a sample pair (x i ,y j Do they belong to the same category? i,j Indicates sample x i and y j The distance between features obtained after mapping through the Siamese convolutional neural network, α is the threshold used to penalize the distance between negative sample pairs, h(α-d i,j ) represents the Hinge loss function, which has the form h(x) = max(0,x);

[0031] The contrastive loss function only considers the metric loss and not the classification loss. The total loss of the designed deep metric learning includes the positive sample pair constraint L. P Negative sample pairs constrain L N and central clustering constraint L O Losses in three aspects:

[0032] L = L P -L N +L O

[0033] In the formula, L P The constraint is to minimize the distance between positive sample pairs, L N The constraint is to maximize the distance between negative sample pairs, and to maximize the distance between positive sample pairs. P and negative samples to constraint L NUsing the contrastive loss function L C This indicates that the central clustering constraint L O The classification loss is represented by Softmax, and is defined as follows:

[0034]

[0035] In the formula, C represents the number of categories, C kx and C ky It is the central feature vector of the k-th class;

[0036] The overall loss function is defined as:

[0037]

[0038] Preferably, in step S4, the target association of multi-source satellite remote sensing images involves matching the target information in two different time periods, t-ΔT and t, where ΔT is the revisit period of the remote sensing satellite, to identify the same target, determine newly appearing targets, and determine whether any targets have disappeared. The solution is viewed as a combinatorial optimization problem, that is, given M targets in the satellite remote sensing image at the previous time t-ΔT and N measurements in the satellite remote sensing image at the current time t, the N measurements are assigned to the M known targets.

[0039] Assume that there are M targets in the satellite remote sensing image at time t-ΔT. And in the satellite remote sensing image at time t, there are N targets. An improved association cost matrix for multi-target association of multi-source satellite remote sensing images is proposed by adopting the design concept and optimization method of association cost matrix;

[0040] Step S4.1: Define the association probability matrix P, whose elements p mn Indicate target and The feature association probability, i.e., the matching cost and the matching probability:

[0041]

[0042] In the formula, and Corresponding to the target and raw data and Mapping in the deep feature space;

[0043] Step S4.2: Define the association indicator matrix S = [s mn ] M×N , representing the target association scheme, whose element s mn Indicate target and Whether associated:

[0044]

[0045] It can be seen that in the association indication matrix S, each row and each column can have at most one value of 1.

[0046] Step S4.3: Expand the association indication matrix S = [s mn (M+N)×(M+N) and the association probability matrix P = [p mn (M+N)×(M+N) , where m = 0, 1,..., M + N, n = 0, 1,..., M + N, and define Analyze the following cases:

[0047] (1) When m ≤ M and n ≤ N, s mn = 1 indicates that the target 1 in T and the target 2 in T are associated with each other.

[0048] (2) When m ≤ M and N < n ≤ M + N, s mn = 1 indicates that the target 1 in T is not associated with any target in T 2 , and p mn = exp{-D};

[0049] (3) When M < m ≤ M + N and n ≤ N, s mn = 1 indicates that the target 2 in T is not associated with any target in T 1 , and p mn = exp{-D};

[0050] (4) When M < m ≤ M + N and N < n ≤ M + N, s mn = 1 is meaningless, and define p mn = 1;

[0051] Step S4.4: Define the association cost between the target and as c mn = p mn · s mn . Then, for each possible association scheme S, construct an association cost matrix C:

[0052] C = [c mn × s mn (M+N)×(M+N) ​​​

[0053] Its overall association cost function is expressed as:

[0054]

[0055] The purpose of association is to solve the optimal association scheme S among all feasible association schemes * , that is, to obtain a subset of T 1 corresponding to a subset of T 2 such that the overall association cost is minimized:

[0056]

[0057] The optimization problem of the association cost matrix is a combinatorial optimization problem with NP complexity. The constraint condition of the association indicator matrix S is that each row and each column can only have one value of 1. When M < m ≤ M + N and N < n ≤ M + N, s mn = 0; Considering the above constraint conditions, the objective function is transformed into:

[0058]

[0059] In the formula, when and only when each row or column in the association indicator matrix S has only one element as 1 and the remaining elements are 0, the first term or the second term is 0, and when M < m ≤ M + N, N < n ≤ M + N, the third term is meaningless and the association cost is 0; Any meaningful association scheme requires the first 3 terms to be 0, and the remaining 4th term represents the actual association cost; When any one of the first 3 terms is not equal to 0, the association cost increases, and large positive values are selected for A, B, and C.

[0060] Preferably, in step S5, in the multi-source satellite remote sensing image, when the number of association targets is relatively large compared to the set threshold, it is necessary to comprehensively consider the calculation amount and the local optimal solution problem; Through the discrete binary particle swarm optimization algorithm DPSO, iterative search for the global optimal solution, design an improved DPSO algorithm to solve the optimal association scheme, by associating the indicator matrix S i and its objective function E i corresponding to the state and fitness of particle i, using binary coding to represent the multi-objective association state, each dimension of the particle position is limited to 0 or 1, and the particle velocity represents the possibility of changing the association state;

[0061] The algorithm first initializes the particle swarm state, randomly assigns an association scheme to each particle, but the position of one particle is initialized by calculating the association scheme through the global nearest neighbor algorithm, and at the same time calculates the fitness of each particle to judge its pros and cons;

[0062] Calculate the individual optimal fitness value and the global optimal fitness value for each particle, introduce distance constraints to iteratively update the position and velocity of particles with non-global optimal fitness values, until the optimal solution of the correlation indicator matrix is ​​finally obtained;

[0063] In updating the particle position state, a matrix constraint is introduced, which ensures that each row and each column of the association indicator matrix has only one element that is 1.

[0064] Secondly, a remote sensing satellite image ship target association system is provided, the system comprising:

[0065] Module M1: Construct a training sample set of multi-source satellite remote sensing images of ships;

[0066] Module M2: Based on deep convolutional neural networks, it learns the coupling and correlation of multi-source satellite remote sensing images, maps different types of ship target satellite remote sensing images to a common deep feature space, and learns a metric function in this deep feature space;

[0067] Module M3: Inputs the test set of multi-source satellite remote sensing images of ship targets into the trained deep metric learning network to perform feature similarity measurement calculation;

[0068] Module M4: Design the Association Cost Matrix (ACM);

[0069] Module M5: Accelerates the solution of the optimal solution of the association cost matrix ACM through the binary discrete particle swarm optimization algorithm DPSO, and performs multi-target association of ships in multi-source remote sensing images.

[0070] Preferably, in module M1, the sample set is constructed online, with positive sample pairs consisting of different images of the same type of ship and negative sample pairs consisting of two images of different types of ships. Furthermore, positive sample pairs are generated in a 1:1 ratio to negative sample pairs in a single training batch. This is implemented by:

[0071] Module M1.1: Count the number of samples N in each category c c = 1, 2, ..., C, and reclassified according to the number of samples from largest to smallest: N1 > N2 > ... > N C Each category of sample is randomly assigned a unique number, with the number values ​​being 1, 2, 3, ..., Nc.

[0072] Module M1.2: Determine the number of training batches M, let the category contained in a single training batch be C, let the number of positive sample pairs in a single training batch be L, L>C, and the number of negative sample pairs be L, then the total number of training samples is 2×L×M.

[0073] Module M1.3: Construct positive sample pairs by repeatedly drawing one class L times from C classes. After each class c is drawn, select its corresponding N... c N samples were randomly selected from the samples. c / 2 samples, and calculate the distance between their features learned by the network, and select the pair of samples with the largest distance as the positive sample pair;

[0074] Module M1.4: Construct negative sample pairs, calculate the feature centers of C classes, and calculate the distances between them, arranging them in ascending order. First, select the two classes c and d with the smallest feature center distance, and then select them from their N... c One sample and N d For each sample, one sample is randomly selected to form a negative sample pair. The above process is repeated in ascending order of feature center distance until L pairs of negative samples are generated.

[0075] Preferably, in module M2, the convolutional neural network model used is VGGNet with weight sharing, which improves the network's learning ability by repeatedly stacking 3×3 convolutional kernels and 2×2 max pooling layers.

[0076] The distance between feature vectors is defined using Euclidean distance:

[0077] Suppose the mapping functions of the first convolutional neural network and the second convolutional neural network are f, respectively. θ (x) and θ and These represent the weight matrices and bias vectors of the two convolutional neural networks, respectively.

[0078] Using a weight-sharing, deep convolutional network with a parameter set of θ, the similarity between image samples x and y is represented as:

[0079]

[0080] d i,j This represents the distance between the features obtained after image samples x and y are mapped by a Siamese convolutional neural network. At this point, deep metric learning changes from learning the metric matrix or linear transformation matrix using traditional methods to learning the weights and bias coefficients of a deep convolutional neural network.

[0081] The loss function of deep metric learning includes the loss of all positive and negative sample pairs in the training dataset. Positive sample pairs are used to reduce intra-class distance, and negative sample pairs are used to increase inter-class distance.

[0082] The loss functions for deep distance metric learning mainly include the contrastive loss function and the triplet loss function. The contrastive loss function minimizes the distance between positive sample pairs of the same class and penalizes the distance between negative sample pairs that are less than a certain threshold. It is defined as follows:

[0083]

[0084] In the formula, l i,j ∈{0,1} represents a sample pair (x i ,y j Do they belong to the same category? i,j Indicates sample x i and y j The distance between features obtained after mapping through the Siamese convolutional neural network, α is the threshold used to penalize the distance between negative sample pairs, h(α-d i,j ) represents the Hinge loss function, which has the form h(x) = max(0,x);

[0085] The contrastive loss function only considers the metric loss and not the classification loss. The total loss of the designed deep metric learning includes the positive sample pair constraint L. P Negative sample pairs constrain L N and central clustering constraint L O Losses in three aspects:

[0086] L = L P -L N +L O

[0087] In the formula, L P The constraint is to minimize the distance between positive sample pairs, L N The constraint is to maximize the distance between negative sample pairs, and to maximize the distance between positive sample pairs. P and negative samples to constraint L N Using the contrastive loss function L C This indicates that the central clustering constraint L O The classification loss is represented by Softmax, and is defined as follows:

[0088]

[0089] In the formula, C represents the number of categories, C kx and C ky It is the central feature vector of the k-th class;

[0090] The overall loss function is defined as:

[0091]

[0092] Preferably, in module M4, the multi-source satellite remote sensing image target association involves matching the target information in two different time periods, t-ΔT and t, where ΔT is the revisit period of the remote sensing satellite, to identify the same target, newly appearing targets, and whether any targets have disappeared. The solution is viewed as a combinatorial optimization problem, i.e., given M targets in the satellite remote sensing image at the previous time t-ΔT and N measurements in the satellite remote sensing image at the current time t, the N measurements are assigned to the M known targets.

[0093] Assume that there are M targets in the satellite remote sensing image at time t-ΔT. And in the satellite remote sensing image at time t, there are N targets. An improved association cost matrix for multi-target association of multi-source satellite remote sensing images is proposed by adopting the design concept and optimization method of association cost matrix;

[0094] Module M4.1: Defines the association probability matrix P, whose elements p mn Indicate target and The feature association probability, i.e., the matching cost and the matching probability:

[0095]

[0096] In the formula, and Corresponding to the target and raw data and Mapping in the deep feature space;

[0097] Module M4.2: Define the association indicator matrix S = [s mn ] M×N , representing the target association scheme, whose element s mn Indicate target and Is it related?

[0098]

[0099] It is evident that in the correlation indicator matrix S, each row and each column can have at most one value of 1;

[0100] Module M4.3: Extended Association Indicator Matrix S = [s mn ] (M+N)×(M+N) And the correlation probability matrix P = [p mn ] (M+N)×(M+N) Let m = 0, 1, ..., M+N, and n = 0, 1, ..., M+N, then define... Analyze the following situations:

[0101] When m ≤ M and n ≤ N, s mn = 1 indicates the target in T 1 and the target in T are correlated with each other. 2

[0102] (2) When m ≤ M and N < n ≤ M + N, s mn = 1 indicates the target in T 1 is not correlated with any target in T and p 2 = exp{-D}; mn

[0103] (3) When M < m ≤ M + N and n ≤ N, s mn = 1 indicates the target in T 2 is not correlated with any target in T and p 1 = exp{-D}; mn

[0104] (4) When M < m ≤ M + N and N < n ≤ M + N, s mn = 1 is meaningless, and p mn is defined as 1;

[0105] Module M4.4: Define the correlation cost between the target and as c mn = p mn · s mn , then for each possible correlation scheme S, construct a correlation cost matrix C:

[0106] C = [c mn × s mn (M+N)×(M+N)

[0107] The overall correlation cost function is expressed as:

[0108]

[0109] The purpose of correlation is to find the optimal correlation scheme S among all feasible correlation schemes * , that is, to find a subset of T 1 corresponding to a subset of T 2 such that the overall correlation cost is minimized:

[0110]

[0111] ​​​​​The associated cost matrix optimization problem is a combinatorial optimization problem with NP complexity. The constraint condition of the associated indication matrix S is that each row and each column can only have one value of 1. When M < m ≤ M + N and N < n ≤ M + N, s mn = 0; Considering the above constraint conditions, the objective function is transformed into:

[0112]

[0113] In the formula, when and only when each row or column in the associated indication matrix S has only one element of 1 and the remaining elements are 0, the first term or the second term is 0, and when M < m ≤ M + N, N < n ≤ M + N, the third term is meaningless and the associated cost is 0; Any meaningful association scheme requires the first 3 terms to be 0, and the remaining fourth term represents the actual associated cost; When any one of the first 3 terms is not equal to 0, it causes an increase in the associated cost, and large positive values are selected for A, B, and C;

[0114] In the module M5, in multi-source satellite remote sensing images, when the number of associated targets is relatively large compared to the set threshold, it is necessary to comprehensively consider the computational complexity and the problem of local optimal solutions; Through the discrete binary particle swarm optimization algorithm DPSO, iterative search for the global optimal solution is carried out, and an improved DPSO algorithm is designed to solve the optimal association scheme. By associating the indication matrix S i and its objective function E i corresponding to the state and fitness of particle i, the multi-objective association state is represented in binary coding. Each dimension of the particle position is restricted to 0 or 1, and the particle velocity represents the possibility of changing the association state;

[0115] The algorithm first initializes the particle swarm state, randomly assigns an association scheme to each particle, but the position of one particle is initialized by calculating the association scheme through the global nearest neighbor algorithm, and at the same time calculates the fitness of each particle to judge its quality;

[0116] Calculate the individual optimal fitness value and the global optimal fitness value of each particle, introduce the distance constraint condition to iteratively update the positions and velocities of the particles with non-global optimal fitness values until the optimal solution of the associated indication matrix is finally obtained;

[0117] Among them, when updating the particle position state, a matrix constraint condition is introduced, that is, ensuring that each row and each column in the associated indication matrix has only one element of 1.

[0118] Compared with the prior art, the present invention has the following beneficial effects:

[0119] 1. This invention first utilizes a constructed training set of multi-source satellite remote sensing images of ships to learn the coupling and correlation of multi-source satellite remote sensing images based on a deep convolutional neural network, mapping different types of ship target satellite remote sensing images to a common deep feature space, in which a metric function is learned; secondly, a test set of multi-source satellite remote sensing images of ship targets is input into the trained deep metric learning network to perform feature similarity measurement calculation, and an association cost matrix (ACM) is designed. Then, the optimal solution of the ACM is accelerated by the binary discrete particle swarm optimization algorithm (DPSO), realizing the association of multiple targets of ships in multi-source remote sensing images;

[0120] 2. This invention integrates ship target information acquired from satellite remote sensing images and designs a multi-dimensional feature fusion matching ship target association algorithm, which can reduce the ambiguity caused by using a single feature for association and improve the accuracy of ship multi-target association.

[0121] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0122] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0123] Figure 1 This is an overall flowchart of the present invention;

[0124] Figure 2 This is a schematic diagram of binary encoding for multi-target associated states. Detailed Implementation

[0125] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0126] This invention provides a method for associating ship targets in remote sensing satellite images, referring to... Figure 1 As shown, the method includes:

[0127] Step S1: Construct a training sample set of multi-source satellite remote sensing images of ships.

[0128] In step S1, the depth metric learning sample set is constructed online. Positive sample pairs consist of different images of the same type of ship, and negative sample pairs consist of two images of different types of ships. Furthermore, positive and negative sample pairs are generated in a 1:1 ratio within a single training batch. The implementation steps include:

[0129] Step S1.1: Count the number of samples N for each category. c c = 1, 2, ..., C, and reclassified according to the number of samples from largest to smallest: N1 > N2 > ... > N C Each category of sample is randomly assigned a unique number, with the number values ​​being 1, 2, 3, ..., Nc.

[0130] Step S1.2: Determine the number of training batches M. In order to ensure that the categories are balanced, let the category included in a single training batch be C, the number of positive sample pairs in a single training batch be L, L>C, and the number of negative sample pairs be L. Then the total number of training samples is 2×L×M.

[0131] Step S1.3: Construct positive sample pairs, repeatedly drawing one category L times from C categories. After each category c is drawn, select its corresponding N... c N samples were randomly selected from the samples. c / 2 samples, and calculate the distance between their features learned by the network, and select the pair of samples with the largest distance as the positive sample pair;

[0132] Step S1.4: Construct negative sample pairs, calculate the feature centers of C classes respectively, and calculate the distances between them and arrange them in ascending order. First, select the two classes c and d with the smallest feature center distance, and then select them from their N... c One sample and N d For each sample, one sample is randomly selected to form a negative sample pair. The above process is repeated in ascending order of feature center distance until L pairs of negative samples are generated.

[0133] It should be noted that this method does not guarantee that the positive sample pair with the largest distance will be extracted, but it can guarantee that positive sample pairs with a large distance will be extracted. Similarly, the same applies to negative sample pairs. Because the same type of target can be further subdivided into more subtypes, and their differences can be significant, negative sample pairs cannot be selected from datasets of the same target type.

[0134] Step S2: Based on deep convolutional neural networks, learn the coupling and correlation of multi-source satellite remote sensing images, map different types of ship target satellite remote sensing images to a common deep feature space, and learn the metric function in this deep feature space.

[0135] In step S2, a weight-sharing VGGNet convolutional neural network model is used (in this embodiment, VGGNet is a classic deep convolutional network model with wide applications in practice. For example, in tasks such as image classification, object detection, and image segmentation, VGGNet can serve as a powerful feature extractor, providing rich feature information for subsequent tasks. In addition, VGGNet can be combined with other deep learning models to form a more powerful hybrid model to cope with more complex tasks). The learning ability of the network is improved by repeatedly stacking 3×3 convolutional kernels and 2×2 max pooling layers.

[0136] The distance between feature vectors is defined using Euclidean distance:

[0137] Suppose the mapping functions of the first convolutional neural network and the second convolutional neural network are f, respectively. θ (x) and θ and These represent the weight matrices and bias vectors, respectively, of the two convolutional neural networks.

[0138] Specifically, the required amount of sample data for network training differs between the weight-sharing and weight-non-sharing strategies. The weight-non-sharing convolutional neural network requires significantly more sample data. Since the dataset designed for this method has a relatively small number of samples, a weight-sharing (network parameter set θ) deep convolutional network is adopted. The similarity between image samples x and y is represented as:

[0139]

[0140] d i,j This represents the distance between the features obtained after image samples x and y are mapped by a Siamese convolutional neural network. At this point, deep metric learning changes from learning the metric matrix or linear transformation matrix using traditional methods to learning the weights and bias coefficients of a deep convolutional neural network.

[0141] The loss function in deep metric learning includes the loss of all positive and negative sample pairs in the training dataset. Positive sample pairs are used to reduce intra-class distance, and negative sample pairs are used to increase inter-class distance.

[0142] The main loss functions for deep distance metric learning are the contrastive loss function and the triplet loss function. The basic idea of ​​the contrastive loss function is to minimize the distance between samples of the same class (distance between positive sample pairs) and penalize the distance between samples of different classes (distance between negative sample pairs) that is less than a certain threshold. Its definition is:

[0143]

[0144] In the formula, li,j ∈{0,1} represents a sample pair (x i ,y j Do they belong to the same category? i,j Indicates sample x i and y j The distance between features obtained after mapping by the Siamese convolutional neural network, α is the threshold used to penalize the distance between negative sample pairs, h(α-d i,j ) represents the Hinge loss function, which has the form h(x) = max(0,x);

[0145] The contrastive loss function only considers the metric loss and not the classification loss. The total loss of the designed deep metric learning includes the positive sample pair constraint L. P Negative sample pairs constrain L N and central clustering constraint L O Losses in three aspects:

[0146] L = L P -L N +L O

[0147] In the formula, L P The constraint is to minimize the distance between similar samples (positive sample pairs), L N The constraint maximizes the distance between outlier samples (negative sample pairs), while the constraint L is applied to positive sample pairs. P and negative samples to constraint L N Using the contrastive loss function L C This indicates that the central clustering constraint L O The classification loss is represented by Softmax, and is defined as follows:

[0148]

[0149] In the formula, C represents the number of categories, C kx and C ky It is the central feature vector of the k-th class;

[0150] The overall loss function is defined as:

[0151]

[0152] Step S3: Input the test set of multi-source satellite remote sensing images of the ship target into the trained deep metric learning network to calculate the feature similarity measurement.

[0153] Step S4: Design the Association Cost Matrix (ACM).

[0154] In step S4, multi-source satellite remote sensing image target association involves matching target information from two satellite remote sensing images at two different times (t-ΔT time and t time, where ΔT is the revisit period of the remote sensing satellite) one-to-one to determine which targets are the same targets, which are newly appearing targets, and whether any targets have disappeared. The solution can be regarded as a combinatorial optimization problem, that is, given M targets in the satellite remote sensing image at the previous time t-ΔT and N measurements in the satellite remote sensing image at the current time t, how to assign the N measurements to the M known targets.

[0155] Assume that there are M targets in the satellite remote sensing image at time t-ΔT. And in the satellite remote sensing image at time t, there are N targets. An improved association cost matrix for multi-target association of multi-source satellite remote sensing images is proposed by adopting the design concept and optimization method of association cost matrix.

[0156] Step S4.1: Define the association probability matrix P, whose elements p mn Indicate target and Feature association probability (matching cost, matching probability):

[0157]

[0158] In the formula, and Corresponding to the target and raw data and Mapping in deep feature space.

[0159] Step S4.2: Define the association indicator matrix S = [s mn ] M×N , represents a possible target association scheme, whose element s mn Indicate target and Is it related?

[0160]

[0161] It is evident that in the correlation indicator matrix S, each row and each column can have at most one value of 1;

[0162] Step S4.3: Expand the association indicator matrix S = [s mn ] (M+N)×(M+N) And the correlation probability matrix P = [p mn ] (M+N)×(M+N) Let m = 0, 1, ..., M+N, and n = 0, 1, ..., M+N, then define... Analyze the following situations:

[0163] (1) When m ≤ M and n ≤ N, s mn = 1 indicates the target in T 1 and the target in T are correlated with each other. 2 中的目标 相互关联,

[0164] (2) When m ≤ M and N < n ≤ M + N, s mn = 1 indicates the target in T 1 中的目标 is not correlated with any target in T, p 2 中的任何目标相互关联,p mn = exp{-D};

[0165] (3) When M < m ≤ M + N and n ≤ N, s mn = 1 indicates the target in T 2 中的目标 is not correlated with any target in T, p 1 中的任何目标相互关联,p mn = exp{-D};

[0166] (4) When M < m ≤ M + N and N < n ≤ M + N, s[[ID=*43]] mn = 1 is meaningless, and p is defined as mn = 1;

[0167] Step S4.4: Define the correlation cost c between the targets and as c mn = p mn · s mn , then for each possible correlation scheme S, construct a correlation cost matrix C:<*

[0168] C = [c mn × s mn (M+N)×(M+N)

[0169] The overall correlation cost function is expressed as:

[0170]

[0171] The purpose of correlation is to find the optimal correlation scheme S among all feasible correlation schemes * , that is, to find a subset of T 1 and a subset of T 2 corresponding to each other, so that the overall correlation cost is minimized:

[0172] [[ID=*85]]

[0173] Note: There are two items marked with '*' in the translation which seem to be incorrect in the original text. In item , the expression '[[ID=*43]] mn ' might be a mislabeled ID. In item , '[[ID=* ' also seems incorrect. Please check the original text for accuracy.The problem of optimizing the association cost matrix is a combinatorial optimization problem with NP complexity. The constraint condition of the association indicator matrix S is that each row and each column can only have one value of 1. When M < m ≤ M + N and N < n ≤ M + N, s mn = 0; Considering the above constraint conditions, the objective function is transformed into:

[0174]

[0175] In the formula, when and only when each row (column) in the association indicator matrix S has only one element equal to 1 and the rest of the elements are 0, the first term (the second term) is 0, and when M < m ≤ M + N, N < n ≤ M + N, the third term is meaningless and the association cost is 0; Any meaningful association scheme requires the first 3 terms to be 0, and the remaining fourth term represents the actual association cost; When any one of the first 3 terms is not equal to 0, it will cause a rapid increase in the association cost. A, B, and C are selected as relatively large positive values.

[0176] Step S5: Through the binary discrete particle swarm optimization algorithm (DPSO, Discrete Partial Swarm Optimization), accelerate the solution of the optimal scheme of the association cost matrix ACM, and perform multi-source remote sensing image ship multi-target association.

[0177] In this step, in multi-source satellite remote sensing images, when the number of associated targets is relatively large compared to the set threshold, it is necessary to comprehensively consider issues such as computational complexity and local optimal solutions; The discrete binary particle swarm optimization algorithm (DPSO) can iteratively search for the global optimal solution with a relatively high probability, has a fast convergence speed and high computational efficiency. Design an improved DPSO algorithm to quickly solve the optimal association scheme. By associating the indicator matrix S i and its objective function E i corresponding to the state and fitness of particle i, use binary coding to represent the multi-target association state. Each dimension of the particle position is restricted to 0 or 1, and the particle velocity represents the possibility of changing the association state.

[0178] The algorithm first initializes the particle swarm state, randomly assigns an association scheme to each particle, but the position of one particle is initialized by calculating the association scheme through the global nearest neighbor algorithm, and at the same time calculates the fitness of each particle to judge its quality.

[0179] Calculate the individual optimal fitness and global optimal fitness of each particle, introduce distance constraints to iteratively update the position and velocity of particles with non-global optimal fitness, until the optimal solution of the association indicator matrix is ​​finally obtained. It is particularly important to note that, unlike the commonly used DPSO algorithm, a matrix constraint is introduced when updating the particle position state, that is, to ensure that each row and each column of the association indicator matrix has only one element with a value of 1.

[0180] Furthermore, for the M targets at time t-ΔT and N targets at time t Its binary encoding method is as follows Figure 2 As shown, each segment of the code contains exactly one 1.

[0181] This invention also provides a remote sensing satellite imagery ship target association system. This system can be implemented by executing the steps of the remote sensing satellite imagery ship target association method. That is, those skilled in the art can understand the remote sensing satellite imagery ship target association method as a preferred embodiment of the remote sensing satellite imagery ship target association system. Specifically, the system includes:

[0182] Module M1: Construct a training sample set of multi-source satellite remote sensing images of ships.

[0183] In module M1, the depth metric learning sample set is constructed online. Positive sample pairs consist of different images of the same type of ship, and negative sample pairs consist of two images of different types of ships. Furthermore, positive and negative sample pairs are generated in a 1:1 ratio within a single training batch. This is implemented as follows:

[0184] Module M1.1: Count the number of samples N in each category c c = 1, 2, ..., C, and reclassified according to the number of samples from largest to smallest: N1 > N2 > ... > N C Each category of sample is randomly assigned a unique number, with the number values ​​being 1, 2, 3, ..., Nc.

[0185] Module M1.2: Determine the number of training batches M. In order to ensure that the categories are balanced, let the category included in a single training batch be C, the number of positive sample pairs in a single training batch be L, L>C, and the number of negative sample pairs be L. Then the total number of training samples is 2×L×M.

[0186] Module M1.3: Construct positive sample pairs by repeatedly drawing one class L times from C classes. After each class c is drawn, select its corresponding N... c N samples were randomly selected from the samples. c / 2 samples, and calculate the distance between their features learned by the network, and select the pair of samples with the largest distance as the positive sample pair;

[0187] Module M1.4: Construct negative sample pairs, calculate the feature centers of C classes, and calculate the distances between them, arranging them in ascending order. First, select the two classes c and d with the smallest feature center distance, and then select them from their N... c One sample and N d For each sample, one sample is randomly selected to form a negative sample pair. The above process is repeated in ascending order of feature center distance until L pairs of negative samples are generated.

[0188] It should be noted that this method does not guarantee that the positive sample pair with the largest distance will be extracted, but it can guarantee that positive sample pairs with a large distance will be extracted. Similarly, the same applies to negative sample pairs. Because the same type of target can be further subdivided into more subtypes, and their differences can be significant, negative sample pairs cannot be selected from datasets of the same target type.

[0189] Module M2: Based on deep convolutional neural networks, it learns the coupling and correlation of multi-source satellite remote sensing images, maps different types of ship target satellite remote sensing images to a common deep feature space, and learns a metric function in this deep feature space.

[0190] In module M2, the convolutional neural network model used is VGGNet with weight sharing. The learning ability of the network is improved by repeatedly stacking 3×3 convolutional kernels and 2×2 max pooling layers.

[0191] The distance between feature vectors is defined using Euclidean distance:

[0192] Suppose the mapping functions of the first convolutional neural network and the second convolutional neural network are f, respectively. θ (x) and θ and These represent the weight matrices and bias vectors, respectively, of the two convolutional neural networks.

[0193] Specifically, the required amount of sample data for network training differs between the weight-sharing and weight-non-sharing strategies. The weight-non-sharing convolutional neural network requires significantly more sample data. Since the dataset designed for this method has a relatively small number of samples, a weight-sharing (network parameter set θ) deep convolutional network is adopted. The similarity between image samples x and y is represented as:

[0194]

[0195] d i,jThis represents the distance between the features obtained after image samples x and y are mapped by a Siamese convolutional neural network. At this point, deep metric learning changes from learning the metric matrix or linear transformation matrix using traditional methods to learning the weights and bias coefficients of a deep convolutional neural network.

[0196] The loss function in deep metric learning includes the loss of all positive and negative sample pairs in the training dataset. Positive sample pairs are used to reduce intra-class distance, and negative sample pairs are used to increase inter-class distance.

[0197] The main loss functions for deep distance metric learning are the contrastive loss function and the triplet loss function. The basic idea of ​​the contrastive loss function is to minimize the distance between samples of the same class (distance between positive sample pairs) and penalize the distance between samples of different classes (distance between negative sample pairs) that is less than a certain threshold. Its definition is:

[0198]

[0199] In the formula, l i,j ∈{0,1} represents a sample pair (x i ,y j Do they belong to the same category? i,j Indicates sample x i and y j The distance between features obtained after mapping by the Siamese convolutional neural network, α is the threshold used to penalize the distance between negative sample pairs, h(α-d i,j ) represents the Hinge loss function, which has the form h(x) = max(0,x);

[0200] The contrastive loss function only considers the metric loss and not the classification loss. The total loss of the designed deep metric learning includes the positive sample pair constraint L. P Negative sample pairs constrain L N and central clustering constraint L O Losses in three aspects:

[0201] L = L P -L N +L O

[0202] In the formula, L P The constraint is to minimize the distance between similar samples (positive sample pairs), L N The constraint maximizes the distance between outlier samples (negative sample pairs), while the constraint L is applied to positive sample pairs. P and negative samples to constraint L N Using the contrastive loss function L C This indicates that the central clustering constraint L O The classification loss is represented by Softmax, and is defined as follows:

[0203]

[0204] In the formula, C represents the number of categories, C kx and C ky It is the central feature vector of the k-th class;

[0205] The overall loss function is defined as:

[0206]

[0207] Module M3: Inputs the test set of multi-source satellite remote sensing images of ship targets into the trained deep metric learning network to perform feature similarity measurement calculation.

[0208] Module M4: Design the Association Cost Matrix (ACM).

[0209] In module M4, the multi-source satellite remote sensing image target association is to match the target information in two satellite remote sensing images at two different times (t-ΔT time and t time, where ΔT is the revisit period of the remote sensing satellite) one by one, to determine which targets are the same targets, which are newly appearing targets, and whether any targets have disappeared. The solution can be regarded as a combinatorial optimization problem, that is, given M targets in the satellite remote sensing image at the previous time t-ΔT and N measurements in the satellite remote sensing image at the current time t, how to assign the N measurements to the M known targets.

[0210] Assume that there are M targets in the satellite remote sensing image at time t-ΔT. And in the satellite remote sensing image at time t, there are N targets. An improved association cost matrix for multi-target association of multi-source satellite remote sensing images is proposed by adopting the design concept and optimization method of association cost matrix.

[0211] Module M4.1: Defines the association probability matrix P, whose elements p mn Indicate target and Feature association probability (matching cost, matching probability):

[0212]

[0213] In the formula, and Corresponding to the target and raw data and Mapping in deep feature space.

[0214] Module M4.2: Define the association indicator matrix S = [s mn ]M×N , representing a possible target association scheme, whose element s mn represents the target and whether it is associated:

[0215]

[0216] It can be seen that in each row and each column of the association indication matrix S, there can be at most one value of 1;

[0217] Module M4.3: Expand the association indication matrix S = [s mn (M+N)×(M+N) and the association probability matrix P = [p mn (M+N)×(M+N) , m = 0, 1,..., M + N, n = 0, 1,..., M + N, define Analyze the following situations:

[0218] (1) When m ≤ M and n ≤ N, s mn = 1 indicates that the target in T 1 and the target in T and T 2 are associated with each other,

[0219] (2) When m ≤ M and N < n ≤ M + N, s mn = 1 indicates that the target in T 1 is not associated with any target in T and T 2 , p mn = exp{-D};[[ID=I54]]

[0220] (3) When M < m ≤ M + N and n ≤ N, s[[ID=5I]] mn = 1 indicates that the target in T 2 is not associated with any target in T and T 1 , p mn = exp{-D};

[0221] (4) When M < m ≤ M + N and N < n ≤ M + N, s mn = 1 is meaningless, define p mn = 1;

[0222] Module M4.4: Define the association cost of the target and as c mn = p mn · s mn , then for each possible association scheme S, construct an association cost matrix C: ​​​

[0223] C = [c mn × s mn (M+N)×(M+N)

[0224] Its overall association cost function is expressed as:

[0225]

[0226] The purpose of association is to solve the optimal association scheme S among all feasible association schemes * , that is, to obtain a subset of T 1 corresponding to a subset of T 2 such that the overall association cost is minimized:

[0227]

[0228] The optimization problem of the association cost matrix is a combinatorial optimization problem with NP complexity. The constraint condition of the association indicator matrix S is that each row and each column can only have one value of 1. When M < m ≤ M + N and N < n ≤ M + N, s mn = 0; Considering the above constraint conditions, the objective function is transformed into:

[0229]

[0230] In the formula, when and only when each row (column) of the association indicator matrix S has only one element of 1 and the remaining elements are 0, the first term (second term) is 0, and when M < m ≤ M + N, N < n ≤ M + N, the third term is meaningless and the association cost is 0; Any meaningful association scheme requires the first 3 terms to be 0, and the remaining 4th term represents the actual association cost; When any one of the first 3 terms is not equal to 0, it will cause a rapid increase in the association cost. A, B, and C are selected as relatively large positive values.

[0231] Module M5: Through the binary discrete particle swarm optimization algorithm (DPSO, Discrete Partial Swarm Optimization), accelerate the solution of the optimal scheme of the association cost matrix ACM, and perform multi-source remote sensing image ship multi-target association.

[0232] In this module, in multi-source satellite remote sensing images, when the number of associated targets is relatively large compared to the set threshold, it is necessary to comprehensively consider issues such as computational complexity and local optimal solutions; The discrete binary particle swarm optimization algorithm (DPSO) can iteratively search for the global optimal solution with a high probability, has a fast convergence speed and high computational efficiency. Design an improved DPSO algorithm to quickly solve the optimal association scheme by associating the association indicator matrix S i and its objective function E i ​Corresponding to the state and fitness of particle i, the multi-objective association state is represented by binary encoding. Each dimension of the particle position is restricted to 0 or 1, and the particle velocity represents the probability of the association state changing.

[0233] The algorithm first initializes the particle swarm state, randomly assigning each particle an association scheme. However, the position of one particle is initialized by calculating the association scheme using the global nearest neighbor algorithm. At the same time, the fitness of each particle is calculated to determine its quality.

[0234] Calculate the individual optimal fitness and global optimal fitness of each particle, introduce distance constraints to iteratively update the position and velocity of particles with non-global optimal fitness, until the optimal solution of the association indicator matrix is ​​finally obtained. It is particularly important to note that, unlike the commonly used DPSO algorithm, a matrix constraint is introduced when updating the particle position state, that is, to ensure that each row and each column of the association indicator matrix has only one element with a value of 1.

[0235] Furthermore, for the M targets at time t-ΔT and N targets at time t Its binary encoding method is as follows Figure 2 As shown, each segment of the code contains exactly one 1.

[0236] This invention provides a method and system for associating ship targets using remote sensing satellite images. It constructs a training set of multi-source satellite remote sensing images of ships, building the sample set online. Positive sample pairs consist of different images of the same type of ship, while negative sample pairs consist of two images of different types of ships. Positive and negative sample pairs are generated in a 1:1 ratio within a single training batch. Based on Euclidean distance-based image semantic layer similarity measurement, the traditional method of learning the metric matrix or linear transformation matrix is ​​transformed into learning the weights and bias coefficients of a deep convolutional neural network. The method includes loss function definitions for metric loss and classification loss. An improved association cost matrix for multi-target association of multi-source satellite remote sensing images is proposed, employing the association cost matrix design concept. The optimal solution for ACM is accelerated using the binary discrete particle swarm optimization (DPSO) algorithm, achieving multi-target association of ships in multi-source remote sensing images. This method can reduce the ambiguity caused by using single-feature association and improve the accuracy of multi-target ship association.

[0237] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0238] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for associating ship targets in remote sensing satellite images, characterized in that, include: Step S1: Construct a training sample set of multi-source satellite remote sensing images of ships; Step S2: Based on deep convolutional neural networks, learn the coupling and correlation of multi-source satellite remote sensing images, map different types of ship target satellite remote sensing images to a common deep feature space, and learn the metric function in this deep feature space; Step S3: Input the test set of multi-source satellite remote sensing images of ship targets into the trained deep metric learning network to calculate feature similarity. Step S4: Design the Association Cost Matrix (ACM); Step S5: Accelerate the solution of the optimal solution of the association cost matrix ACM by using the binary discrete particle swarm optimization algorithm DPSO, and perform multi-target association of ships in multi-source remote sensing images.

2. The method for associating ship targets in remote sensing satellite images according to claim 1, characterized in that, In step S1, the sample set is constructed online. Positive sample pairs consist of different images of the same type of ship, and negative sample pairs consist of two images of different types of ships. Furthermore, each training batch generates positive sample pairs to negative sample pairs in a 1:1 ratio. The implementation steps include: Step S1.1: Count the number of samples N for each category. c c = 1, 2, ..., C, and reclassified according to the number of samples from largest to smallest: N1 > N2 > ... > N C Each category of sample is randomly assigned a unique number, with the number values ​​being 1, 2, 3, ..., Nc. Step S1.2: Determine the number of training batches M, let the category included in a single training batch be C, let the number of positive sample pairs in a single training batch be L, L>C, and the number of negative sample pairs be L, then the total number of training samples is 2×L×M. Step S1.3: Construct positive sample pairs, repeatedly drawing one category L times from C categories. After each category c is drawn, select its corresponding N... c N samples were randomly selected from the samples. c / 2 samples, and calculate the distance between their features learned by the network, and select the pair of samples with the largest distance as the positive sample pair; Step S1.4: Construct negative sample pairs, calculate the feature centers of C classes respectively, and calculate the distances between them and arrange them in ascending order. First, select the two classes c and d with the smallest feature center distance, and then select them from their N... c One sample and N d For each sample, one sample is randomly selected to form a negative sample pair. The above process is repeated in ascending order of feature center distance until L pairs of negative samples are generated.

3. The method for associating ship targets in remote sensing satellite images according to claim 1, characterized in that, In step S2, the convolutional neural network model used is VGGNet with weight sharing. The learning ability of the network is improved by repeatedly stacking 3×3 convolutional kernels and 2×2 max pooling layers. The distance between feature vectors is defined using Euclidean distance: Suppose the mapping functions of the first convolutional neural network and the second convolutional neural network are f, respectively. θ (x) and θ and These represent the weight matrices and bias vectors of the two convolutional neural networks, respectively. Using a weight-sharing, deep convolutional network with a parameter set of θ, the similarity between image samples x and y is represented as: d i,j This represents the distance between the features obtained after image samples x and y are mapped by a Siamese convolutional neural network. At this point, deep metric learning changes from learning the metric matrix or linear transformation matrix using traditional methods to learning the weights and bias coefficients of a deep convolutional neural network.

4. The method for associating ship targets in remote sensing satellite images according to claim 1, characterized in that, In step S2, the loss function of deep metric learning includes the loss of all positive and negative sample pairs in the training dataset. Positive sample pairs are used to reduce intra-class distance, and negative sample pairs are used to increase inter-class distance. The loss functions for deep distance metric learning mainly include the contrastive loss function and the triplet loss function. The contrastive loss function minimizes the distance between positive sample pairs of the same class and penalizes the distance between negative sample pairs that are less than a certain threshold. It is defined as follows: In the formula, l i,j ∈{0,1} represents a sample pair (x i ,y j Do they belong to the same category? i,j Indicates sample x i and y j The distance between features obtained after mapping through the Siamese convolutional neural network, α is the threshold used to penalize the distance between negative sample pairs, h(α-d i,j ) represents the Hinge loss function, which has the form h(x) = max(0,x); The contrastive loss function only considers the metric loss and not the classification loss. The total loss of the designed deep metric learning includes the positive sample pair constraint L. P Negative sample pairs constrain L N and central clustering constraint L O Losses in three aspects: L=L P -L N +L O In the formula, L P The constraint is to minimize the distance between positive sample pairs, L N The constraint is to maximize the distance between negative sample pairs, and to maximize the distance between positive sample pairs. P and negative samples to constraint L N Using the contrastive loss function L C This indicates that the central clustering constraint L O The classification loss is represented by Softmax, and is defined as follows: In the formula, C represents the number of categories, C kx and C ky It is the central feature vector of the k-th class; The overall loss function is defined as:

5. The method for associating ship targets in remote sensing satellite images according to claim 1, characterized in that, In step S4, the multi-source satellite remote sensing image target association involves matching the target information in two different time periods, t-ΔT and t, where ΔT is the revisit period of the remote sensing satellite, to identify the same target, newly appearing targets, and whether any targets have disappeared. The solution is viewed as a combinatorial optimization problem, that is, given M targets in the satellite remote sensing image at the previous time t-ΔT and N measurements in the satellite remote sensing image at the current time t, the N measurements are assigned to the M known targets. Assume that there are M targets in the satellite remote sensing image at time t-ΔT. And in the satellite remote sensing image at time t, there are N targets. An improved association cost matrix for multi-target association of multi-source satellite remote sensing images is proposed by adopting the design concept and optimization method of association cost matrix; Step S4.1: Define the association probability matrix P, whose elements p mn Indicate target and The feature association probability, i.e., the matching cost and the matching probability: In the formula, and Corresponding to the target and raw data and Mapping in the deep feature space; Step S4.2: Define the association indicator matrix S = [s mn ] M×N , representing the target association scheme, whose element s mn Indicate target and Is it related? It is evident that in the correlation indicator matrix S, each row and each column can have at most one value of 1; Step S4.3: Expand the association indicator matrix S = [s mn ] (M+N)×(M+N) And the correlation probability matrix P = [p mn ] (M+N)×(M+N) Let m = 0, 1, ..., M+N, and n = 0, 1, ..., M+N, then define... Analyze the following situations: (1) When m≤M and n≤N, s mn =1 indicates T 1 The target and T 2 The target Interrelated When \(m\leq M\) and \(N < n\leq M + N\), \(s\) mn \(= 1\) indicates that the target 1 in \(T\) is not associated with any target in \(T\), \(p\) 2 \(= \exp\{-D\}\);​​ When \(3\leq M\lt m\leq M + N\) and \(n\leq N\), \(s mn = 1 indicates the target in \(T 2 is not associated with any target in \(T and \(p 1 = exp\{-D\}; mn ​ When \(4\leq m\leq M + N\) and \(N\lt n\leq M + N\), \(s\lt0000049\gt = 1\) is meaningless, and define \(p\lt0000050\gt = 1\); mn is meaningless, define p mn = 1; Step S4.4: Define the target and The association cost is c mn =p mn ·s mn For each possible association scheme S, an association cost matrix C is constructed: C=[c mn ×s mn ] (M+N)×(M+N) Its overall association cost function is expressed as: The purpose of association is to find the optimal association scheme S among all feasible association schemes. * That is, to find T 1 A subset and T 2 A subset corresponds to minimizes the overall association cost: The associated cost matrix optimization problem is a combinatorial optimization problem with NP complexity. The constraint condition of the associated indicator matrix S is that each row and each column can only have one value of 1. When M < m ≤ M + N and N < n ≤ M + N, s mn = 0; Considering the above constraint conditions, the objective function is transformed into: In the formula, when and only when there is only one element equal to 1 in each row or column of the association indication matrix S and the remaining elements are 0, the first term or the second term is 0, and when M < m ≤ M + N, N < n ≤ M + N, the meaningless association cost of the third term is 0; any meaningful association scheme requires the first 3 terms to be 0, and the remaining fourth term represents the actual association cost; when any one of the first 3 terms is not equal to 0, it causes the association cost to increase, and large positive values are selected for A, B, and C.

6. The method for associating ship targets in remote sensing satellite images according to claim 1, characterized in that, In step S5, when the number of associated targets in a multi-source satellite remote sensing image is relatively large compared to a set threshold, it is necessary to comprehensively consider the computational load and local optima. An improved DPSO algorithm is designed to solve for the optimal association scheme by iteratively searching for a global optimal solution using the Discrete Binary Particle Swarm Optimization (DPSO) algorithm, through the association indicator matrix S. i and its objective function E i Corresponding to the state and fitness of particle i, the multi-objective association state is represented by binary encoding. Each dimension of the particle position is restricted to 0 or 1, and the particle velocity represents the probability of the association state changing. The algorithm first initializes the particle swarm state, randomly assigns an association scheme to each particle, but the position of one particle is initialized by calculating the association scheme through the global nearest neighbor algorithm, and at the same time calculates the fitness of each particle to judge its quality. Calculate the individual optimal fitness value and the global optimal fitness value of each particle, introduce the distance constraint condition to iteratively update the positions and velocities of the non-global optimal fitness value particles until the optimal solution of the association indication matrix is finally obtained. Among them, when updating the particle position state, a matrix constraint condition is introduced, that is, it is ensured that there is only one element equal to 1 in each row and each column of the association indication matrix.

7. A remote sensing satellite image ship target association system, characterized in that, It includes: Module M1: Construct a training sample set of multi-source satellite remote sensing images of ships. Module M2: Based on the deep convolutional neural network, learn the coupling and correlation of multi-source satellite remote sensing images, map satellite remote sensing images of different types of ship targets to a common deep feature space, and learn a metric function in this deep feature space. Module M3: Input the test set of multi-source satellite remote sensing images of ship targets into the trained deep metric learning network to calculate the feature similarity. Module M4: Design an association cost matrix ACM. Module M5: Through the binary discrete particle swarm optimization algorithm DPSO, accelerate the solution of the optimal scheme of the association cost matrix ACM, and perform multi-source remote sensing image ship multi-target association.

8. The remote sensing satellite image ship target association system according to claim 7, characterized in that, In the module M1, the sample set is constructed in an online manner. The positive sample pairs are composed of different images of the same type of ship, and the negative sample pairs are composed of two images of different types of ships, and are generated in a ratio of 1:1 of positive sample pairs to negative sample pairs in a single training batch. Its implementation includes: Module M1.1: Count the number of samples N in each category c c = 1, 2, ..., C, and reclassified according to the number of samples from largest to smallest: N1 > N2 > ... > N C Each category of sample is randomly assigned a unique number, with the number values ​​being 1, 2, 3, ..., Nc. Module M1.2: Determine the number of training batches M, let the classes included in a single training batch be C, let the number of positive sample pairs in a single training batch be L, L > C, and the number of negative sample pairs be L, then the total number of training samples is 2 × L × M. Module M1.3: Construct positive sample pairs by repeatedly drawing one class L times from C classes. After each class c is drawn, select its corresponding N... c N samples were randomly selected from the samples. c / 2 samples, and calculate the distance between their features learned by the network, and select the pair of samples with the largest distance as the positive sample pair; Module M1.4: Construct negative sample pairs, calculate the feature centers of C classes, and calculate the distances between them, arranging them in ascending order. First, select the two classes c and d with the smallest feature center distance, and then select them from their N... c One sample and N d For each sample, one sample is randomly selected to form a negative sample pair. The above process is repeated in ascending order of feature center distance until L pairs of negative samples are generated.

9. The remote sensing satellite image ship target association system according to claim 7, characterized in that, In the module M2, the convolutional neural network model used is VGGNet with weight sharing, and the network learning ability is improved by repeatedly stacking 3×3 convolutional kernels and 2×2 max-pooling layers. Define the distance between feature vectors by the Euclidean distance: Suppose the mapping functions of the first convolutional neural network and the second convolutional neural network are f, respectively. θ (x) and θ and These represent the weight matrices and bias vectors of the two convolutional neural networks, respectively. Adopt weight sharing, and for a deep convolutional network with a network parameter set of θ, the similarity between image samples x and y is expressed as: d i,j This represents the distance between the features obtained after image samples x and y are mapped by a Siamese convolutional neural network. At this point, deep metric learning changes from learning the metric matrix or linear transformation matrix using traditional methods to learning the weights and bias coefficients of a deep convolutional neural network. The loss function of deep metric learning includes the losses of all positive and negative sample pairs in the training dataset. Positive sample pairs are used to reduce the intra-class distance, and negative sample pairs are used to increase the inter-class distance. The loss functions for deep distance metric learning mainly include the contrastive loss function and the triplet loss function. The contrastive loss function minimizes the distance of positive sample pairs among similar samples and penalizes the distance of negative sample pairs less than a certain threshold, and its definition is as follows: In the formula, l i,j ∈{0,1} represents a sample pair (x i ,y j Do they belong to the same category? i,j Indicates sample x i and y j The distance between features obtained after mapping through the Siamese convolutional neural network, α is the threshold used to penalize the distance between negative sample pairs, h(α-d i,j ) represents the Hinge loss function, which has the form h(x) = max(0,x); The contrastive loss function only considers the metric loss and not the classification loss. The total loss of the designed deep metric learning includes the positive sample pair constraint L. P Negative sample pairs constrain L N and central clustering constraint L O Losses in three aspects: L=L P -L N +L O In the formula, L P The constraint is to minimize the distance between positive sample pairs, L N The constraint is to maximize the distance between negative sample pairs, and to maximize the distance between positive sample pairs. P and negative samples to constraint L N Using the contrastive loss function L C This indicates that the central clustering constraint L O The classification loss is represented by Softmax, and is defined as follows: In the formula, C represents the number of categories, C kx and C ky It is the central feature vector of the k-th class; The total loss function is defined as:

10. The remote sensing satellite image ship target association system according to claim 7, characterized in that, In the module M4, the target association of multi-source satellite remote sensing images corresponds the target information in two satellite remote sensing images at different times, i.e., at time t - ΔT and at time t, where ΔT is the revisit period of the remote sensing satellite, determines the same target, determines newly emerged targets and whether there are disappearing targets, and regards the solution as a combinatorial optimization problem, that is, given M targets in the satellite remote sensing image at the previous time t - ΔT and N measurements in the satellite remote sensing image at the current time t, assigns the N measurements to the M known targets; Assume that there are M targets in the satellite remote sensing image at time t-ΔT. And in the satellite remote sensing image at time t, there are N targets. An improved association cost matrix for multi-target association of multi-source satellite remote sensing images is proposed by adopting the design concept and optimization method of association cost matrix; Module M4.1: Defines the association probability matrix P, whose elements p mn Indicate target and The feature association probability, i.e., the matching cost and the matching probability: In the formula, and Corresponding to the target and raw data and Mapping in the deep feature space; Module M4.2: Define the association indicator matrix S = [s mn ] M×N , representing the target association scheme, whose element s mn Indicate target and Is it related? It can be seen that in the association indicator matrix S, at most one value in each row and each column can be 1; Module M4.3: Extended Association Indicator Matrix S = [s mn ] (M+N)×(M+N) And the correlation probability matrix P = [p mn ] (M+N)×(M+N) Let m = 0, 1, ..., M+N, and n = 0, 1, ..., M+N, then define... Analyze the following situations: (1) When m≤M and n≤N, s mn =1 indicates T 1 The target and T 2 The target Interrelated When \(m\leq M\) and \(N < n\leq M + N\), \(s\) mn \(= 1\) indicates that the 1 target in is not associated with any 2 targets in \(T\), \(p\) mn \(=\exp\{-D\}\); (3) When \(M < m\leq M + N\) and \(n\leq N\), \(s mn = 1\) indicates the target in \(T 2 is not associated with any target in \(T , and \(p 1 = \exp\{-D\}\);​​ When \(4\leq m\leq M + N\) and \(N\lt n\leq M + N\), \(s mn being equal to 1 is meaningless, and define \(p mn = 1; Module M4.4: Define the target and The association cost is c mn =p mn ·s mn For each possible association scheme S, an association cost matrix C is constructed: C=[c mn ×s mn ] (M+N)×(M+N) Its overall association cost function is expressed as: The purpose of association is to find the optimal association scheme S among all feasible association schemes. * That is, to find T 1 A subset and T 2 A subset corresponds to minimizes the overall association cost: The associated cost matrix optimization problem is a combinatorial optimization problem with NP complexity. The constraint condition of the associated indicator matrix S is that each row and each column can only have one value of 1. When M < m ≤ M + N and N < n ≤ M + N, s mn = 0; Considering the above constraint conditions, the objective function is transformed into: In the formula, when and only when there is only one element in each row or column of the association indicator matrix S that is 1 and the rest are 0, the first or second term is 0, and when M < m ≤ M + N, N < n ≤ M + N, the third term is meaningless and the association cost is 0; any meaningful association scheme requires the first 3 terms to be 0, and the remaining fourth term represents the actual association cost; when any one of the first 3 terms is not equal to 0, it causes an increase in the association cost, and large positive values are selected for A, B, and C; In module M5, when the number of associated targets in multi-source satellite remote sensing images is relatively large compared to a set threshold, it is necessary to comprehensively consider the computational load and local optima. An improved DPSO algorithm is designed to find the optimal association scheme by iteratively searching for a global optimal solution using the Discrete Binary Particle Swarm Optimization (DPSO) algorithm, through the association indicator matrix S. i and its objective function E i Corresponding to the state and fitness of particle i, the multi-objective association state is represented by binary encoding. Each dimension of the particle position is restricted to 0 or 1, and the particle velocity represents the probability of the association state changing. The algorithm first initializes the particle swarm state, randomly assigns an association scheme to each particle, but the position of one particle is initialized by calculating the association scheme through the global nearest neighbor algorithm, and at the same time calculates the fitness of each particle to judge its quality; Calculate the individual optimal fitness value and the global optimal fitness value of each particle, introduce the distance constraint condition to iteratively update the positions and velocities of the particles with non-global optimal fitness values until the optimal solution of the association indicator matrix is finally obtained; Among them, when updating the particle position state, a matrix constraint condition is introduced, that is, ensuring that each row and each column in the association indicator matrix has only one element that is 1.

Images