Marine target continuous tracking method based on low-orbit optical satellite constellation data fusion

Abstract

The invention discloses a marine target continuous tracking method based on low-orbit optical satellite constellation data fusion. The method comprises the following steps: acquiring an optical satellite remote sensing image sequence shot by a plurality of low-orbit optical satellites; performing target detection on the optical satellite remote sensing image by using the inclined frame target detection network model to obtain a detection result corresponding to each detection target in the image; inputting each detection target slice into a convolutional neural network model, obtaining depth appearance features of the detection target, and performing data association by using multi-feature matching according to the target trajectory and the target state of the current known target and the detection state and the depth appearance features of the detection target, determining a target trajectory associated with the detection result, an unassociated target trajectory and the detection result; data association is carried out by using oblique frame intersection-to-union ratio matching according to the unassociated target tracks and detection results; and performing target trajectory updating and target state prediction according to the association result. According to the invention, continuous tracking identification of the marine target can be rapidly and accurately realized.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing information processing, and in particular to a method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion. Background Art

[0002] Low-orbit optical satellites are an important means of ocean surveillance. At present, the information processing research on low-orbit optical satellites in ocean surveillance mainly focuses on the detection and recognition of marine targets based on low-orbit optical satellite remote sensing images. At present, the detection and recognition of marine targets based on low-orbit optical satellite remote sensing images are mainly based on the interpretation of remote sensing images observed by a single low-orbit optical satellite, without considering the fusion application of massive observation data of large-scale low-orbit optical satellite constellations. Since a single low-orbit satellite can only perform overhead push-scan imaging or video imaging of marine targets in a short period of time, the detection and recognition of marine targets based on remote sensing images observed by a single low-orbit optical satellite can only achieve short-term monitoring of marine targets, and cannot achieve long-term continuous monitoring and tracking of marine targets, which is difficult to meet the actual monitoring task requirements. Summary of the invention

[0003] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion, which can realize detection and identification, continuous tracking and trajectory detection of maritime targets.

[0004] The technical solution of the present invention is as follows:

[0005] A method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion is provided, comprising:

[0006] Using multiple low-orbit optical satellites in the low-orbit optical satellite constellation to push-broom image the preset mission area, and obtaining optical satellite remote sensing image sequences taken by multiple low-orbit optical satellites;

[0007] Using a pre-trained oblique frame target detection network model, target detection is performed on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image are obtained. The detection result includes the oblique frame position and geographic coordinate position corresponding to the detected target;

[0008] Input each detected target slice into a pre-trained convolutional neural network model, obtain the deep appearance features corresponding to the detected target output by the convolutional neural network model, and use multi-feature matching to perform data association between the target trajectory and the detection result according to the target trajectory and target state of the currently known target, as well as the detection state and deep appearance features of each detected target, to determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory;

[0009] For the target track not associated with the detection result and the detection result not associated with the target track, the target track and the detection result are data associated by using the oblique frame intersection and union ratio matching to determine the target track associated with the detection result, the target track not associated with the detection result, and the detection result not associated with the target track;

[0010] The target trajectory is updated and the target state is predicted based on the association results.

[0011] In some optional implementations, a pre-trained oblique frame target detection network model is used to perform target detection on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image are obtained, including:

[0012] Input each optical satellite remote sensing image in the optical satellite remote sensing image sequence into a pre-trained oblique frame target detection network model, and obtain the type and oblique frame position corresponding to the detected target in each optical satellite remote sensing image output by the oblique frame target detection network model;

[0013] The discovery time corresponding to the detection target is determined according to the shooting time of the optical satellite remote sensing image, and the geographic coordinate position corresponding to the detection target is calculated and obtained according to the image metadata and the oblique frame position corresponding to the detection target in the optical satellite remote sensing image.

[0014] In some optional implementations, the oblique frame object detection network model is trained by the following steps:

[0015] Acquire a first training data set, the first training data comprising: sample images containing marine targets, types of marine targets in the sample images, and oblique frame position information;

[0016] The sample image of the first training data in the first training data set is taken as input, the type and oblique frame position information of the marine target corresponding to the input sample image is taken as output, and the oblique frame target detection network model is trained.

[0017] In some optional implementations, the longitude and latitude coordinates of the target, the speed of the target in the longitude and latitude directions, and the length and width of the target are used as the target state of the known target, which is expressed as:

[0018]

[0019] X represents the state vector of the target, and λ represent the longitude and latitude coordinates of the target, respectively, including the longitude and latitude coordinates of the four corner points of the target oblique frame. and They represent the speed of the target in the longitude and latitude directions respectively, and l and w represent the length and width of the target respectively;

[0020] The longitude and latitude coordinates of the detection target, as well as the length and width of the detection target are used as the detection state of the detection target, which is expressed as:

[0021]

[0022] Z represents the detection state vector of the detected target, and Respectively represent the longitude and latitude coordinates of the detected target, including the longitude and latitude coordinates of the four corner points of the oblique frame of the detected target. and Represent the length and width of the detected target respectively.

[0023] In some optional implementations, according to the target trajectory and target state of the currently known target, and the detection state and deep appearance features of each detected target, multi-feature matching is used to perform data association on the target trajectory and the detection result, including:

[0024] Target motion measurement using target state and detection state;

[0025] Use the appearance features of the target to measure the target attributes;

[0026] Using the target motion measurement results and target attribute measurement results to perform joint motion and attribute measurement;

[0027] The joint measurement results of motion and attributes corresponding to the known target and the detected target are used as the allocation cost, and the allocation cost corresponding to the i-th known target and the j-th detected target is used as the matrix element of the i-th row and j-th column to form a cost matrix C, and determine whether the target motion measurement result and the target attribute measurement result corresponding to the i-th known target and the j-th detected target meet the preset first distance threshold value and the preset attribute threshold value respectively. If so, keep the corresponding matrix element in the cost matrix unchanged, if not, set the corresponding matrix element in the cost matrix to infinity to update the cost matrix;

[0028] The allocation algorithm is used to solve the minimum allocation cost of the cost matrix, obtain the optimal data association result, and obtain the corresponding relationship between the target trajectory and the detection result.

[0029] In some optional embodiments, the target motion metric is defined as:

[0030]

[0031] Among them, d (1) (i, j) represents the target motion metric between the i-th known target and the j-th detected target, X i and S i denote the target state and covariance matrix of the i-th known target, Z j represents the detection state of the jth detection target, and T represents the transposed matrix.

[0032] In some optional implementations, the target attribute metric is defined as:

[0033]

[0034] Among them, d (2) (i,j) represents the target attribute measurement of the i-th known target and the j-th detected target, r j represents the deep appearance feature corresponding to the jth detected target, R i represents the appearance feature set of the i-th known target, L k represents the number of appearance feature vectors, Represents the kth appearance feature vector of the ith known target.

[0035] In some optional implementations, the joint motion and attribute metric is defined as:

[0036] c i,j =qd (1) (i,j)+(1-q)d (2) (i,j);

[0037] Among them, c i,j represents the joint motion and attribute metric of the target motion metric of the i-th known target and the j-th detected target, and q represents the weighting coefficient.

[0038] In some optional implementations, for a target track that is not associated with a detection result and a detection result that is not associated with the target track, data association is performed on the target track and the detection result using oblique frame intersection and union matching, including:

[0039] For target tracks that are not associated with detection results and detection results that are not associated with target tracks, the intersection-and-union ratio of the target oblique frame corresponding to each target track and the detection target oblique frame corresponding to each detection result is calculated, and the oblique frame intersection-and-union ratio calculation result is used to perform data association between the target track and the detection result.

[0040] In some optional implementations, updating the target trajectory and predicting the target state according to the association result include:

[0041] If the current target trajectory of a known target has a detection result associated with it, the detection result is used to update the target trajectory and predict the target state of the corresponding known target;

[0042] If there is no detection result associated with the current target track of a known target, further determine whether the current target track is a confirmed track, if not, delete the current target track, if yes, further determine whether the number of consecutive unassociated times of the current target track reaches a first preset number, if yes, delete the current target track;

[0043] If a detection result is not associated with any target track, the detection result is regarded as an unconfirmed track;

[0044] Wherein, when the number of times that the unconfirmed trajectory is continuously associated with the detection result reaches a second preset number, it is adjusted to a confirmed trajectory.

[0045] The main advantages of the technical solution of the present invention are as follows:

[0046] The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion of the present invention obtains an image sequence of the mission area through a low-orbit optical satellite constellation, and based on the target detection results of the image sequence, uses multi-feature matching and oblique frame intersection and union ratio matching to perform data association, which can quickly and accurately realize continuous tracking and identification of maritime targets in the mission area, and significantly improve the maritime situation awareness capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flow chart of a method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to an embodiment of the present invention;

[0049] Figure 2It is a schematic diagram of the trajectory association process in a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of target detection results of an optical satellite remote sensing image provided in Example 1 of the present invention;

[0051] Figure 4 A schematic diagram of a target tracking result provided in Example 1 of the present invention;

[0052] Figure 5 A schematic diagram of a slice of a moving target extracted from five optical satellite remote sensing images provided in Example 1 of the present invention;

[0053] Figure 6 A schematic diagram of slices of another moving target extracted from five optical satellite remote sensing images provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] The technical solution provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.

[0056] See also Figure 1 An embodiment of the present invention provides a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion, the method comprising the following steps S1-S5:

[0057] Step S1, using multiple low-orbit optical satellites in a low-orbit optical satellite constellation to perform push-scan imaging on a preset mission area, and obtaining an optical satellite remote sensing image sequence taken by the multiple low-orbit optical satellites.

[0058] Specifically, according to the operating parameters of the low-orbit optical satellite constellation, the visibility time of different low-orbit optical satellites in the low-orbit optical satellite constellation to the mission area is determined, and then according to the visibility times corresponding to multiple low-orbit optical satellites in the low-orbit optical satellite constellation, the mission area is push-scanned using the low-orbit optical satellite during the visibility time corresponding to each low-orbit optical satellite, so as to obtain optical satellite remote sensing images corresponding to the mission area taken by different low-orbit optical satellites at different times, and then obtain an optical satellite remote sensing image sequence composed of optical satellite remote sensing images taken by multiple low-orbit optical satellites.

[0059] Among them, the mission area is set specifically according to the actual maritime target monitoring mission requirements.

[0060] Step S2, using a pre-trained oblique frame target detection network model to perform target detection on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and obtaining the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image, wherein the detection result includes the oblique frame position and geographic coordinate position corresponding to the detected target.

[0061] Due to the complex distribution of targets at sea, especially when the mission area includes a port, there will often be dense and adjacent targets at sea. In order to improve the target detection accuracy, in one embodiment of the present invention, an oblique frame-based target detection algorithm, i.e., an oblique frame target detection network model, is used to perform target detection on optical satellite remote sensing images.

[0062] Furthermore, in one embodiment of the present invention, a pre-trained oblique frame target detection network model is used to perform target detection on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image are obtained, which specifically includes the following steps:

[0063] Step S21, inputting each optical satellite remote sensing image in the optical satellite remote sensing image sequence into a pre-trained oblique frame target detection network model, and obtaining the type and oblique frame position corresponding to the detection target in each optical satellite remote sensing image output by the oblique frame target detection network model;

[0064] Step S22, determining the discovery time corresponding to the detection target according to the shooting time of the optical satellite remote sensing image, and calculating and obtaining the geographic coordinate position corresponding to the detection target according to the image metadata and the oblique frame position corresponding to the detection target in the optical satellite remote sensing image.

[0065] Specifically, the image coordinates of the oblique frame corresponding to the target in the optical satellite remote sensing image are expressed as:

[0066] x ima =(x1,y1,x2,y2,x3,y3,x4,y4) T ;

[0067] Among them, (x n ,y n ) represents the image coordinates of the nth corner point of the oblique frame corresponding to the target in the optical satellite remote sensing image, n = 1, 2, 3, 4, x n is the coordinate in the x-axis direction of the image coordinate system, y n is the coordinate in the y-axis direction of the image coordinate system.

[0068] According to the conversion relationship from the image coordinates to the object coordinates of the optical satellite remote sensing image, the geographic coordinate position of the oblique frame corresponding to the target can be obtained, which is specifically expressed as:

[0069]

[0070] in, Indicates the geographic coordinates of the nth corner point of the oblique box corresponding to the target, that is, (x n ,y n ) corresponding to the geographical coordinates, n = 1, 2, 3, 4, is longitude, λ n is the latitude.

[0071] Furthermore, in an embodiment of the present invention, the geographical coordinate position corresponding to the target is represented by the geographical coordinate position of the oblique frame corresponding to the target.

[0072] In order to improve the accuracy of target detection, an embodiment of the present invention further provides a training method for an oblique frame target detection network model, and the training method includes the following steps:

[0073] Step S211, obtaining a first training data set, the first training data including: sample images containing marine targets, types of marine targets in the sample images, and oblique frame position information;

[0074] Step S212, taking the sample image of the first training data in the first training data set as input, taking the type and oblique frame position information of the marine target corresponding to the input sample image as output, and training to obtain an oblique frame target detection network model.

[0075] In one embodiment of the present invention, the oblique frame target detection network model adopts the existing YOLOv7 oblique frame target detection network. The sample image is selected from the optical satellite remote sensing image data acquired by the existing low-orbit optical satellite in the push-broom mode, and the type and oblique frame position information of each marine target in the sample image can be determined by manual annotation.

[0076] Furthermore, in one embodiment of the present invention, in order to improve the detection accuracy of the oblique-frame target detection network model when performing target detection on optical satellite remote sensing images, when using the oblique-frame target detection network model to perform target detection on optical satellite remote sensing images, the optical satellite remote sensing images are first divided into blocks, and then the oblique-frame target detection model network is used to perform target detection on each optical satellite remote sensing image block. After obtaining the detection result corresponding to each optical satellite remote sensing image block, the detection results of each optical satellite remote sensing image block are merged through the maximum suppression algorithm to obtain the detection result of the entire optical satellite remote sensing image.

[0077] Furthermore, in one embodiment of the present invention, in order to further improve the detection accuracy of the oblique frame target detection network model when performing target detection on the optical satellite remote sensing image, the image is preprocessed before using the oblique frame target detection network model to perform target detection on the optical satellite remote sensing image. The preprocessing includes geometric correction processing to eliminate geometric distortion of the image.

[0078] Step S3, input each detection target slice obtained by detection into the pre-trained convolutional neural network model respectively, obtain the deep appearance features corresponding to the detection target output by the convolutional neural network model, and use multi-feature matching to perform data association between the target trajectory and the detection result according to the target trajectory and target state of the current known target, as well as the detection state and deep appearance features of each detection target, to determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory.

[0079] Since optical satellite remote sensing images are easily affected by weather factors such as lighting and clouds, maritime targets may not be detected in subsequent images, resulting in target loss. In order to achieve robust tracking of maritime targets, in one embodiment of the present invention, multi-feature matching is performed in combination with the state features and appearance features of the target to perform data association between known target trajectories and detection results.

[0080] In one embodiment of the present invention, based on the detection of the optical satellite remote sensing image in step S2, each detection target detected by the oblique frame target detection network model is extracted from the optical satellite remote sensing image to obtain a detection target slice, and each detection target slice is respectively input into a pre-trained convolutional neural network model to obtain a depth appearance feature corresponding to the detection target output by the convolutional neural network model.

[0081] Furthermore, in one embodiment of the present invention, the convolutional neural network model is pre-trained in the following manner:

[0082] Step S311, obtaining a second training data set, the second training data including: a target sample image, and a target appearance feature corresponding to the target sample image;

[0083] Step S312: taking the target sample image of the second training data in the second training data set as input, taking the target appearance feature corresponding to the input target sample image as output, and training to obtain a convolutional neural network model.

[0084] In one embodiment of the present invention, the convolutional neural network model adopts an existing convolutional neural network (CNN), which is composed of a convolutional layer, a pooling layer, a residual layer, a fully connected layer, and a normalization layer. The target sample image is extracted from the optical satellite remote sensing image data acquired by the existing low-orbit optical satellite in the push-broom mode, and the target appearance features corresponding to the target sample image can be determined by manual annotation.

[0085] In one embodiment of the present invention, in order to further improve the detection accuracy of the convolutional neural network when detecting the detection target slice, the detection target slice is size normalized before being input into the convolutional neural network.

[0086] Furthermore, considering the influence of factors such as the optical satellite platform type, orbital altitude, and observation angle, the resolution, width, and other information of different satellite remote sensing images are usually different. Therefore, in the continuous tracking scenario of marine targets based on low-orbit optical satellites, it is difficult to uniformly describe the pixel position and size of the target as a state, as in traditional video tracking. In one embodiment of the present invention, in combination with the characteristics of satellite applications, the longitude and latitude coordinates of the target, the speed of the target in the longitude and latitude directions, and the length and width of the target are used as the state of the target, and the state of the target is specifically expressed as:

[0087]

[0088] Where X represents the state vector of the target, and λ represent the longitude and latitude coordinates of the target, respectively, including the longitude and latitude coordinates of the four corner points of the target oblique frame. and They represent the speed of the target in the longitude and latitude directions respectively, l and w represent the length and width of the target respectively. The length and width of the target can be estimated by the length and width of the target oblique frame and can be considered unchanged.

[0089] Further, in one embodiment of the present invention, the detection state corresponding to the detection target is determined according to the detection result corresponding to the detection target, and the longitude coordinates and latitude coordinates of the detection target, and the length and width of the detection target are used as the detection state. The detection state is specifically expressed as:

[0090]

[0091] Among them, Z represents the detection state vector of the detection target, and Respectively represent the longitude and latitude coordinates of the detection target, including the longitude and latitude coordinates of the four corner points of the detection target oblique frame, and They represent the length and width of the detection target respectively. The length and width of the detection target can be estimated by the length and width of the detection target oblique frame.

[0092] Further, in one embodiment of the present invention, according to the target trajectory and target state of the currently known target, and the detection state and deep appearance features of each detected target, multi-feature matching is used to perform data association on the target trajectory and the detection result, specifically including the following steps S321-S325:

[0093] Step S321, using the target state and the detection state to measure the target motion.

[0094] In one embodiment of the present invention, taking the i-th known target and the j-th detected target as an example, the distance metric between the motion information, that is, the target motion metric is defined as:

[0095]

[0096] Among them, d (1) (i,j) represents the target motion metric of the i-th known target and the j-th detected target, X i and S i denote the target state and covariance matrix of the i-th known target, Z j represents the detection state of the jth detection target, and T represents the transposed matrix.

[0097] It should be noted that, in one embodiment of the present invention, the target state covariance matrix S i It is obtained by recursion through the target state filter.

[0098] Step S322, using the appearance features of the target to measure the target attributes.

[0099] In one embodiment of the present invention, taking the i-th known target and the j-th detected target as an example, the distance metric between the attribute information, that is, the target attribute metric is defined as:

[0100]

[0101] Among them, d (2) (i,j) represents the target attribute measurement of the i-th known target and the j-th detected target, r j represents the deep appearance feature corresponding to the jth detected target, R i represents the appearance feature set of the i-th known target, L k represents the number of appearance feature vectors, represents the kth appearance feature vector of the ith known target, and T represents the transposed matrix.

[0102] It should be noted that, in one embodiment of the present invention, when a detection target is associated with a target, the depth appearance features corresponding to the detection target are added to the appearance feature set of the target, that is, the appearance feature set of the known target is composed of the depth appearance features of the detection target associated with it.

[0103] Step S323: perform joint motion and attribute measurement using the target motion measurement result and the target attribute measurement result.

[0104] In one embodiment of the present invention, taking the i-th known target and the j-th detected target as an example, the joint measurement of motion and attribute is defined as:

[0105] c i,j =qd (1) (i,j)+(1-q)d (2) (i,j)

[0106] Among them, c i,j represents the joint measurement of the motion and attributes of the i-th known target and the j-th detected target, and q represents the weighting coefficient. The weighting coefficient q is set according to the actual situation.

[0107] Step S324, taking the joint measurement results of motion and attributes corresponding to the known target and the detected target as the allocation cost, and taking the allocation cost corresponding to the ith known target and the jth detected target as the matrix element of the ith row and jth column to form the cost matrix C, and determining whether the target motion measurement results and target attribute measurement results corresponding to the ith known target and the jth detected target respectively meet the preset first distance threshold value and the preset attribute threshold value, if so, keeping the corresponding matrix elements in the cost matrix unchanged, if not, setting the corresponding matrix elements in the cost matrix to infinity to update the cost matrix.

[0108] In an embodiment of the present invention, the first distance threshold and the attribute threshold are specifically set according to actual conditions.

[0109] Step S325 , using the allocation algorithm to solve the minimum allocation cost of the cost matrix, obtain the optimal data association result, and obtain the corresponding relationship between the target trajectory and the detection result.

[0110] In one embodiment of the present invention, the allocation algorithm adopts the Kuhn-Munkres algorithm. The allocation algorithm is used to solve the minimum allocation cost of the cost matrix, obtain the optimal data association result, determine the corresponding relationship between the target trajectory and the detection result, and then determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory.

[0111] Step S4, for the target trajectory not associated with the detection result and the detection result not associated with the target trajectory, the target trajectory and the detection result are data associated by using the oblique frame intersection and union ratio matching to determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory.

[0112] refer to Figure 2 In one embodiment of the present invention, for the target track not associated with the detection result and the detection result not associated with the target track determined in the above step S3, the intersection over union (IOU) of the target oblique frame corresponding to each target track and the detection target oblique frame corresponding to each detection result is calculated, and the oblique frame intersection over union calculation result is used to perform data association between the target track and the detection result, and determine the target track associated with the detection result, the target track not associated with the detection result, and the detection result not associated with the target track.

[0113] Specifically, in one embodiment of the present invention, the intersection-over-union (IOU) of the target oblique frame corresponding to each target trajectory and the detected target oblique frame corresponding to each detection result is calculated, and the data association of the target trajectory and the detection result is performed using the calculation result of the oblique frame intersection-over-union, including the following steps:

[0114] Calculate the intersection and union ratio of the target oblique box corresponding to each target trajectory and the detected target oblique box corresponding to each detection result, use the calculated result of the intersection and union ratio of the oblique boxes corresponding to the known target and the detected target as the allocation cost, and use the allocation cost corresponding to the mth known target and the nth detected target as the matrix element of the mth row and nth column to form the cost matrix D, and determine the intersection and union ratio RIOU of the oblique boxes corresponding to the mth known target and the nth detected target m,n Whether the preset second distance threshold is met, if so, the corresponding matrix element in the cost matrix is ​​kept unchanged, if not, the corresponding matrix element in the cost matrix is ​​set to infinity to update the cost matrix.

[0115] It should be noted that, in an embodiment of the present invention, the second distance threshold is specifically set according to actual conditions.

[0116] In one embodiment of the present invention, the allocation algorithm is also used to solve the minimum allocation cost of the cost matrix D, obtain the optimal data association result, and obtain the corresponding relationship between the target trajectory and the detection result.

[0117] In one embodiment of the present invention, the allocation algorithm adopts the Kuhn-Munkres algorithm. The allocation algorithm is used to solve the minimum allocation cost of the cost matrix to obtain the optimal data association result, and then determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory.

[0118] Step S5: updating the target trajectory and predicting the target state according to the association result.

[0119] In one embodiment of the present invention, target trajectory update and target state prediction are performed according to the association result obtained in step S3 and the association result obtained in step S4.

[0120] refer to Figure 2 Specifically, updating the target trajectory and predicting the target state according to the association result further includes the following steps:

[0121] If the current target trajectory of a known target has a detection result associated with it, the detection result is used to update the target trajectory and predict the target state of the corresponding known target;

[0122] If there is no detection result associated with the current target track of a known target, further determine whether the current target track is a confirmed track, if not, delete the current target track, if yes, further determine whether the number of consecutive unassociated times of the current target track reaches a first preset number, if yes, delete the current target track;

[0123] If a detection result is not associated with any target track, the detection result is regarded as an unconfirmed track;

[0124] Wherein, when the number of times that the unconfirmed trajectory is continuously associated with the detection result reaches a second preset number, it is adjusted to a confirmed trajectory.

[0125] In one embodiment of the present invention, the first preset number K and the second preset number M can be specifically set according to actual conditions.

[0126] Furthermore, in one embodiment of the present invention, when using the detection results to update the target trajectory and predict the target state of the corresponding known target, the detection result is used as the next-moment trajectory point of the target trajectory associated with it to update the target trajectory, and the extended Kalman filter algorithm is used to predict the target state at the next moment based on the current target trajectory, the current target state, the detection result and the detection state.

[0127] A method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion provided by an embodiment of the present invention obtains an image sequence of a mission area through a low-orbit optical satellite constellation, and performs data association based on the target detection results of the image sequence by using multi-feature matching and oblique frame intersection and union ratio matching. It can quickly and accurately realize continuous tracking and identification of maritime targets in the mission area, and significantly improve the maritime situation awareness capability.

[0128] The beneficial effects of the method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion provided by an embodiment of the present invention are described below with reference to specific examples:

[0129] In Example 1, a port was selected as the mission area for the experiment. Five low-orbit optical satellites were used to continuously image the mission area within one hour. The resolution of all satellite remote sensing images was about 1 meter. The YOLOv7 oblique frame target detection network was used to detect targets on the acquired optical satellite remote sensing images. The target detection results of a complete optical satellite remote sensing image are shown in the attached figure. Figure 3 As shown. Figure 3 From the target detection results corresponding to the two local areas enlarged in the figure, it can be seen that each detection result is represented as an oblique frame. There are many static and moving marine targets in the mission area, the target sizes vary greatly, and some targets are closely arranged, so the scene is relatively complex. In this example 1, the YOLOv7 oblique frame target detection algorithm is used for target detection. The target detection accuracy and recall rate are both above 80%, which can meet the application requirements of subsequent tracking methods.

[0130] In Example 1, data association is performed based on the target detection results of the optical satellite remote sensing image, and the marine target tracking results obtained are shown in the attached figure. Figure 4 As shown, the target detection result is represented as a point, the trajectory obtained by target tracking is represented as a line, and "o" and "+" represent the starting point and the end point of the target trajectory, respectively. It can be seen that through target tracking, the target information from multiple low-orbit optical satellites can be effectively integrated to form a more comprehensive maritime target situation map. In this example 1, by adopting the data association method provided by an embodiment of the present invention, the target tracking recall rate has been significantly improved, close to the target detection recall rate, indicating that the target can be successfully tracked once it is found, verifying the effectiveness of the method provided by an embodiment of the present invention for continuous tracking of maritime targets.

[0131] Attached Figure 5 Schematic diagram of a slice of a moving target extracted from five optical satellite remote sensing images. Figure 6 This is a schematic diagram of a slice of another moving target extracted from five optical satellite remote sensing images. It can be seen that in different optical satellite remote sensing images, there are some differences in the appearance of the target. These targets can be effectively identified by matching features such as motion and appearance. By continuously tracking the marine target, not only the dynamic information of the target can be obtained, but also more attribute information can be obtained from the target slice, which supports further target subclassification and data fusion, verifying the application value of the method provided in one embodiment of the present invention in marine target tracking.

[0132] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, "front", "back", "left", "right", "upper" and "lower" in this article are all referenced to the placement state shown in the accompanying drawings.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion, characterized in that: include: Using multiple low-orbit optical satellites in the low-orbit optical satellite constellation to push-broom image the preset mission area, and obtaining optical satellite remote sensing image sequences taken by multiple low-orbit optical satellites; Using a pre-trained oblique frame target detection network model, target detection is performed on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image are obtained. The detection result includes the oblique frame position and geographic coordinate position corresponding to the detected target; Input each detected target slice into a pre-trained convolutional neural network model, obtain the deep appearance features corresponding to the detected target output by the convolutional neural network model, and use multi-feature matching to perform data association between the target trajectory and the detection result according to the target trajectory and target state of the currently known target, as well as the detection state and deep appearance features of each detected target, to determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory; For the target track not associated with the detection result and the detection result not associated with the target track, the target track and the detection result are data associated by using the oblique frame intersection and union ratio matching to determine the target track associated with the detection result, the target track not associated with the detection result, and the detection result not associated with the target track; The target trajectory is updated and the target state is predicted based on the association results.

2. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 1 is characterized in that: The pre-trained oblique frame target detection network model is used to detect targets in each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and the discovery time and detection results corresponding to each detected target in the optical satellite remote sensing image are obtained, including: Input each optical satellite remote sensing image in the optical satellite remote sensing image sequence into a pre-trained oblique frame target detection network model, and obtain the type and oblique frame position corresponding to the detected target in each optical satellite remote sensing image output by the oblique frame target detection network model; The discovery time corresponding to the detection target is determined according to the shooting time of the optical satellite remote sensing image, and the geographic coordinate position corresponding to the detection target is calculated and obtained according to the image metadata and the oblique frame position corresponding to the detection target in the optical satellite remote sensing image.

3. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 1 is characterized in that: The oblique frame object detection network model is trained through the following steps: Acquire a first training data set, the first training data comprising: sample images containing marine targets, types of marine targets in the sample images, and oblique frame position information; The sample image of the first training data in the first training data set is taken as input, the type and oblique frame position information of the marine target corresponding to the input sample image is taken as output, and the oblique frame target detection network model is trained.

4. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 1 is characterized in that: The target state of the known target is represented by the longitude and latitude coordinates of the target, the speed of the target in the longitude and latitude directions, and the length and width of the target: X represents the state vector of the target, and λ represent the longitude and latitude coordinates of the target, respectively, including the longitude and latitude coordinates of the four corner points of the target oblique frame. and They represent the speed of the target in the longitude and latitude directions respectively, and l and w represent the length and width of the target respectively; The longitude and latitude coordinates of the detection target, as well as the length and width of the detection target are used as the detection state of the detection target, expressed as: Z represents the detection state vector of the detected target, and Respectively represent the longitude and latitude coordinates of the detection target, including the longitude and latitude coordinates of the four corner points of the detection target oblique frame, and Represent the length and width of the detected target respectively.

5. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 4 is characterized in that: According to the target trajectory and target state of the currently known target, as well as the detection state and deep appearance features of each detected target, multi-feature matching is used to perform data association between the target trajectory and the detection results, including: Target motion measurement using target state and detection state; Use the appearance features of the target to measure the target attributes; Use the target motion measurement results and target attribute measurement results to perform joint motion and attribute measurement; The joint measurement results of motion and attributes corresponding to the known target and the detected target are used as the allocation cost, and the allocation cost corresponding to the i-th known target and the j-th detected target is used as the matrix element of the i-th row and j-th column to form a cost matrix C, and determine whether the target motion measurement result and the target attribute measurement result corresponding to the i-th known target and the j-th detected target meet the preset first distance threshold value and the preset attribute threshold value respectively. If so, keep the corresponding matrix element in the cost matrix unchanged, if not, set the corresponding matrix element in the cost matrix to infinity to update the cost matrix; The allocation algorithm is used to solve the minimum allocation cost of the cost matrix, obtain the optimal data association result, and obtain the corresponding relationship between the target trajectory and the detection result.

6. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 5 is characterized in that: The target motion metric is defined as: Among them, d (1) (i, j) represents the target motion metric between the i-th known target and the j-th detected target, X i and S i denote the target state and covariance matrix of the i-th known target, Z j represents the detection state of the jth detection target, and T represents the transposed matrix.

7. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 6 is characterized in that: The target attribute measure is defined as: Among them, d (2) (i,j) represents the target attribute measurement of the i-th known target and the j-th detected target, r j represents the deep appearance feature corresponding to the jth detected target, R i represents the appearance feature set of the i-th known target, L k represents the number of appearance feature vectors, Represents the kth appearance feature vector of the ith known target.

8. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 7 is characterized in that: The joint motion and attribute metric is defined as: c i,j =qd (1) (i,j)+(1-q)d (2) (i,j); Among them, c i,j It represents the joint measurement of motion and attributes of the i-th known target and the j-th detected target, and q represents the weighting coefficient.

9. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 8, characterized in that: For target tracks that are not associated with detection results and detection results that are not associated with target tracks, data association is performed between target tracks and detection results using oblique frame intersection and union matching, including: For target tracks that are not associated with detection results and detection results that are not associated with target tracks, the intersection-and-union ratio of the target oblique frame corresponding to each target track and the detection target oblique frame corresponding to each detection result is calculated, and the oblique frame intersection-and-union ratio calculation result is used to perform data association between the target track and the detection result.

10. The method for continuous tracking of marine targets based on low-orbit optical satellite constellation data fusion according to claim 9, characterized in that: According to the association results, the target trajectory is updated and the target state is predicted, including: If the current target trajectory of a known target has a detection result associated with it, the detection result is used to update the target trajectory and predict the target state of the corresponding known target; If there is no detection result associated with the current target track of a known target, further determine whether the current target track is a confirmed track, if not, delete the current target track, if yes, further determine whether the number of consecutive unassociated times of the current target track reaches a first preset number, if yes, delete the current target track; If a detection result is not associated with any target track, the detection result is regarded as an unconfirmed track; Wherein, when the number of times that the unconfirmed trajectory is continuously associated with the detection result reaches a second preset number, it is adjusted to a confirmed trajectory.

Images