Continuous tracking method for maritime targets based on data fusion of low-orbit optical satellite constellation

Abstract

The present invention discloses a method for continuous tracking of maritime targets based on data fusion of a low-orbit optical satellite constellation. The method comprises the following steps: obtaining a sequence of optical satellite remote sensing images taken by multiple low-orbit optical satellites; performing target detection on the optical satellite remote sensing images using a slanted-frame target detection network model to obtain detection results corresponding to each detected target in the image; inputting each detected target slice into a convolutional neural network model to obtain the depth appearance features of the detected target; performing data association using multi-feature matching based on the target trajectory and target state of the currently known target, the detection state and depth appearance features of the detected target, and determining the target trajectory associated with the detection result, the unassociated target trajectory, and the detection result; performing data association using slanted-frame intersection-over-union matching for the unassociated target trajectory and detection result; and updating the target trajectory and predicting the target state based on the association results. The present invention can quickly and accurately achieve continuous tracking and identification of maritime targets.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing information processing, and in particular to a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion. Background Art

[0002] Low-orbit optical satellites are an important means of ocean surveillance. Current research on information processing for low-orbit optical satellites in ocean surveillance focuses primarily on maritime target detection and recognition based on remote sensing imagery from low-orbit optical satellites. Currently, maritime target detection and recognition based on remote sensing imagery from low-orbit optical satellites is primarily based on the interpretation of remote sensing images observed by a single low-orbit optical satellite, without considering the integration and application of massive observation data from large-scale low-orbit optical satellite constellations. Because a single low-orbit satellite can only perform push-scan imaging or video imaging of maritime targets over a short period of time, maritime target detection and recognition based on remote sensing imagery from a single low-orbit optical satellite can only achieve short-term monitoring of maritime targets, but cannot achieve long-term continuous monitoring and tracking of maritime targets, making it difficult to meet the needs of actual monitoring missions. Summary of the Invention

[0003] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion, which can realize detection and identification, continuous tracking and trajectory detection of maritime targets.

[0004] The technical solutions of the present invention are as follows:

[0005] A method for continuous tracking of maritime targets based on data fusion of a low-orbit optical satellite constellation is provided, comprising:

[0006] Utilize multiple low-orbit optical satellites in a low-orbit optical satellite constellation to carry out push-broom imaging of a preset mission area, and obtain a sequence of optical satellite remote sensing images taken by multiple low-orbit optical satellites;

[0007] Utilize the pre-trained oblique frame target detection network model to perform target detection on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and obtain the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image. The detection result includes the oblique frame position and geographic coordinate position corresponding to the detected target.

[0008] Each detected target slice is input into a pre-trained convolutional neural network model to obtain the deep appearance features corresponding to the detected target output by the convolutional neural network model. Based on the target trajectory and target state of the currently known target, as well as the detection state and deep appearance features of each detected target, multi-feature matching is used to perform data association between the target trajectory and the detection result to determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory;

[0009] For target tracks that are not associated with detection results and detection results that are not associated with target tracks, data association is performed on the target tracks and detection results using oblique frame intersection-union matching to determine target tracks associated with detection results, target tracks that are not associated with detection results, and detection results that are not associated with target tracks;

[0010] The target trajectory is updated and the target state is predicted based on the association results.

[0011] In some optional embodiments, a pre-trained oblique-frame target detection network model is used to perform target detection on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and a discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image are obtained, including:

[0012] Input each optical satellite remote sensing image in the optical satellite remote sensing image sequence into a pre-trained oblique frame target detection network model, and obtain the type and oblique frame position corresponding to the detected target in each optical satellite remote sensing image output by the oblique frame target detection network model;

[0013] The discovery time of the detection target is determined according to the shooting time of the optical satellite remote sensing image, and the geographic coordinate position of the detection target is calculated according to the image metadata and the oblique frame position of the detection target in the optical satellite remote sensing image.

[0014] In some optional implementations, the oblique frame object detection network model is trained by the following steps:

[0015] Acquire a first training data set, the first training data including: sample images containing marine targets, types of marine targets in the sample images, and oblique frame position information;

[0016] The sample image of the first training data in the first training data set is used as input, the type and oblique frame position information of the marine target corresponding to the input sample image is used as output, and an oblique frame target detection network model is trained.

[0017] In some optional implementations, the longitude and latitude coordinates of the target, the speed of the target in the longitude and latitude directions, and the length and width of the target are used as the target state of the known target, which is expressed as:

[0018]

[0019] X represents the state vector of the target, and λ represent the longitude and latitude coordinates of the target, respectively, including the longitude and latitude coordinates of the four corner points of the target oblique frame. and They represent the speed of the target in the longitude and latitude directions respectively, l and w represent the length and width of the target respectively;

[0020] The detection state of the detection target is expressed as the longitude and latitude coordinates of the detection target, as well as the length and width of the detection target, as follows:

[0021]

[0022] Z represents the detection state vector of the detected target, and Respectively represent the longitude and latitude coordinates of the detection target, including the longitude and latitude coordinates of the four corner points of the detection target oblique frame, and Represent the length and width of the detection target respectively.

[0023] In some optional embodiments, based on the target trajectory and target state of the currently known target, and the detection state and depth appearance features of each detected target, multi-feature matching is used to perform data association between the target trajectory and the detection result, including:

[0024] Target motion measurement using target state and detection state;

[0025] Utilize the appearance features of the target to measure the target attributes;

[0026] Utilize the target motion measurement results and target attribute measurement results to perform joint motion and attribute measurement;

[0027] The joint measurement results of motion and attributes corresponding to the known target and the detected target are used as the allocation cost, and the allocation cost corresponding to the i-th known target and the j-th detected target is used as the matrix element of the i-th row and j-th column to form a cost matrix C, and determine whether the target motion measurement result and the target attribute measurement result corresponding to the i-th known target and the j-th detected target meet the preset first distance threshold value and the preset attribute threshold value respectively. If so, keep the corresponding matrix element in the cost matrix unchanged; if not, set the corresponding matrix element in the cost matrix to infinity to update the cost matrix;

[0028] The allocation algorithm is used to solve the minimum allocation cost of the cost matrix, obtain the optimal data association result, and obtain the correspondence between the target trajectory and the detection result.

[0029] In some optional embodiments, the target motion metric is defined as:

[0030]

[0031] Among them, d (1) (i, j) represents the target motion measurement between the i-th known target and the j-th detected target, X i and S i denote the target state and covariance matrix of the i-th known target, Z j represents the detection state of the jth detection target, and T represents the transposed matrix.

[0032] In some optional implementations, the target attribute metric is defined as:

[0033]

[0034] Among them, d (2) (i, j) represents the target attribute measurement of the i-th known target and the j-th detected target, r j Represents the deep appearance feature corresponding to the j-th detection target, R i represents the appearance feature set of the i-th known target, L k represents the number of appearance feature vectors, represents the kth appearance feature vector of the i-th known target.

[0035] In some optional implementations, the joint motion and attribute metric is defined as:

[0036] c i,j =qd (1) (i,j)+(1-q)d (2) (i,j);

[0037] Among them, c i,j It represents the joint motion and attribute metric of the target motion metric of the i-th known target and the j-th detected target, and q represents the weighting coefficient.

[0038] In some optional implementations, for target trajectories that are not associated with detection results and detection results that are not associated with target trajectories, data association is performed on the target trajectories and the detection results using oblique frame intersection-over-union matching, including:

[0039] For target tracks that are not associated with detection results and detection results that are not associated with target tracks, the intersection-and-union ratio of the target slant box corresponding to each target track and the detection target slant box corresponding to each detection result is calculated, and the result of the slant box intersection-and-union ratio calculation is used to perform data association between the target track and the detection results.

[0040] In some optional implementations, updating the target trajectory and predicting the target state based on the association result include:

[0041] If there is a detection result associated with the current target trajectory of a known target, the detection result is used to update the target trajectory and predict the target state of the corresponding known target;

[0042] If there is no detection result associated with the current target track of a known target, further determine whether the current target track is a confirmed track. If not, delete the current target track. If so, further determine whether the number of consecutive unassociated times of the current target track reaches a first preset number. If so, delete the current target track.

[0043] If a detection result is not associated with any target track, the detection result is regarded as an unconfirmed track;

[0044] Wherein, when the number of times an unconfirmed trajectory is continuously associated with a detection result reaches a second preset number, it is adjusted to a confirmed trajectory.

[0045] The main advantages of the technical solution of the present invention are as follows:

[0046] The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion of the present invention obtains an image sequence of the mission area through a low-orbit optical satellite constellation, and uses multi-feature matching and oblique frame intersection and union matching based on the target detection results of the image sequence to perform data association. It can quickly and accurately realize continuous tracking and identification of maritime targets in the mission area, significantly improving the maritime situation awareness capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a flow chart of a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to an embodiment of the present invention;

[0049] Figure 2Schematic diagram of the trajectory association process in a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of target detection results of an optical satellite remote sensing image provided in Example 1 of the present invention;

[0051] Figure 4 A schematic diagram of target tracking results provided in Example 1 of the present invention;

[0052] Figure 5 A schematic diagram of a slice of a moving target extracted from five optical satellite remote sensing images provided in Example 1 of the present invention;

[0053] Figure 6 A schematic diagram of a slice of another moving target extracted from five optical satellite remote sensing images provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] The technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] See also Figure 1 An embodiment of the present invention provides a method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion, the method comprising the following steps S1-S5:

[0057] Step S1: Use multiple low-orbit optical satellites in a low-orbit optical satellite constellation to perform push-scan imaging on a preset mission area to obtain a sequence of optical satellite remote sensing images taken by the multiple low-orbit optical satellites.

[0058] Specifically, according to the operating parameters of the low-orbit optical satellite constellation, the visibility time of different low-orbit optical satellites in the low-orbit optical satellite constellation to the mission area is determined, and then according to the visibility time corresponding to multiple low-orbit optical satellites in the low-orbit optical satellite constellation, the mission area is pushed and scanned by using the low-orbit optical satellite during the visibility time corresponding to each low-orbit optical satellite, so as to obtain optical satellite remote sensing images corresponding to the mission area taken by different low-orbit optical satellites at different times, and then obtain an optical satellite remote sensing image sequence composed of optical satellite remote sensing images taken by multiple low-orbit optical satellites.

[0059] Among them, the mission area is set specifically according to the actual maritime target monitoring mission requirements.

[0060] Step S2: Detect targets in each optical satellite remote sensing image in the optical satellite remote sensing image sequence using a pre-trained oblique frame target detection network model to obtain the discovery time and detection results corresponding to each detected target in the optical satellite remote sensing image. The detection results include the oblique frame position and geographic coordinate position corresponding to the detected target.

[0061] Due to the complex distribution of maritime targets, especially when the mission area includes a port, there is often a situation where maritime targets are densely packed and adjacent to each other. In order to improve the accuracy of target detection, in one embodiment of the present invention, a slant-frame-based target detection algorithm, i.e., a slant-frame target detection network model, is used to perform target detection on optical satellite remote sensing images.

[0062] Furthermore, in one embodiment of the present invention, a pre-trained oblique-frame target detection network model is used to perform target detection on each optical satellite remote sensing image in an optical satellite remote sensing image sequence, and the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image are obtained, specifically comprising the following steps:

[0063] Step S21, inputting each optical satellite remote sensing image in the optical satellite remote sensing image sequence into a pre-trained oblique frame target detection network model, and obtaining the type and oblique frame position corresponding to the detected target in each optical satellite remote sensing image output by the oblique frame target detection network model;

[0064] Step S22, determining the discovery time corresponding to the detection target based on the shooting time of the optical satellite remote sensing image, and calculating and obtaining the geographic coordinate position corresponding to the detection target based on the image metadata and the oblique frame position corresponding to the detection target in the optical satellite remote sensing image.

[0065] Specifically, the image coordinates of the oblique frame corresponding to the target in the optical satellite remote sensing image are expressed as:

[0066] x ima =(x1,y1,x2,y2,x3,y3,x4,y4) T ;

[0067] Among them, (x n ,y n ) represents the image coordinates of the nth corner point of the oblique frame corresponding to the target in the optical satellite remote sensing image, n = 1, 2, 3, 4, x n is the coordinate in the x-axis direction of the image coordinate system, y n is the coordinate in the y-axis direction of the image coordinate system.

[0068] According to the conversion relationship between the image space coordinates and the object space coordinates of the optical satellite remote sensing image, the geographic coordinate position of the oblique frame corresponding to the target can be obtained, which is specifically expressed as:

[0069]

[0070] in, Indicates the geographic coordinates of the nth corner point of the oblique box corresponding to the target, that is, (x n ,y n ) corresponding geographic coordinates, n = 1, 2, 3, 4, is longitude, λ n is the latitude.

[0071] Furthermore, in one embodiment of the present invention, the geographical coordinate position corresponding to the target is represented by the geographical coordinate position of the oblique frame corresponding to the target.

[0072] In order to improve the accuracy of target detection, an embodiment of the present invention further provides a training method for an oblique frame target detection network model, which includes the following steps:

[0073] Step S211: obtaining a first training data set, where the first training data includes: sample images containing marine targets, and the type and oblique frame position information of each marine target in the sample images;

[0074] Step S212: taking the sample image of the first training data in the first training data set as input, taking the type and oblique frame position information of the marine target corresponding to the input sample image as output, and training to obtain an oblique frame target detection network model.

[0075] In one embodiment of the present invention, the oblique frame target detection network model uses the existing YOLOv7 oblique frame target detection network. Sample images are selected from optical satellite remote sensing image data acquired by existing low-orbit optical satellites in push-broom mode. The type and oblique frame position information of each maritime target in the sample image can be determined through manual annotation.

[0076] Furthermore, in one embodiment of the present invention, in order to improve the detection accuracy of the oblique frame target detection network model when performing target detection on the optical satellite remote sensing image, when using the oblique frame target detection network model to perform target detection on the optical satellite remote sensing image, the optical satellite remote sensing image is first divided into blocks, and then the oblique frame target detection model network is used to perform target detection on each optical satellite remote sensing image block. After obtaining the detection result corresponding to each optical satellite remote sensing image block, the detection results of each optical satellite remote sensing image block are merged through the maximum suppression algorithm to obtain the detection result of the entire optical satellite remote sensing image.

[0077] Furthermore, in one embodiment of the present invention, to further improve the detection accuracy of an oblique-frame target detection network model when performing target detection on optical satellite remote sensing images, the images are preprocessed before being used to detect targets on the optical satellite remote sensing images. The preprocessing includes geometric correction to eliminate geometric distortion of the images.

[0078] In step S3, each detection target slice obtained by detection is input into a pre-trained convolutional neural network model respectively to obtain the depth appearance features corresponding to the detection target output by the convolutional neural network model. According to the target trajectory and target state of the currently known target, as well as the detection state and depth appearance features of each detection target, multi-feature matching is used to perform data association between the target trajectory and the detection result to determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory.

[0079] Since optical satellite remote sensing images are easily affected by weather factors such as lighting and clouds, maritime targets may not be detected in subsequent images, resulting in target loss. To achieve robust tracking of maritime targets, in one embodiment of the present invention, multi-feature matching is performed in combination with the state features and appearance features of the target to perform data association between known target trajectories and detection results.

[0080] In one embodiment of the present invention, based on the detection of the optical satellite remote sensing image in step S2, each detection target detected by the oblique frame target detection network model is extracted from the optical satellite remote sensing image to obtain a detection target slice. Each detection target slice is input into a pre-trained convolutional neural network model to obtain the depth appearance features corresponding to the detection target output by the convolutional neural network model.

[0081] Furthermore, in one embodiment of the present invention, the convolutional neural network model is pre-trained in the following manner:

[0082] Step S311, obtaining a second training data set, the second training data including: a target sample image, and target appearance features corresponding to the target sample image;

[0083] Step S312: taking the target sample image of the second training data in the second training data set as input, taking the target appearance feature corresponding to the input target sample image as output, and training to obtain a convolutional neural network model.

[0084] In one embodiment of the present invention, the convolutional neural network model utilizes an existing convolutional neural network (CNN), consisting of a convolutional layer, a pooling layer, a residual layer, a fully connected layer, and a normalization layer. The target sample images are extracted from optical satellite remote sensing image data acquired by existing low-orbit optical satellites in push-broom mode. The target appearance features corresponding to the target sample images can be determined through manual annotation.

[0085] In one embodiment of the present invention, in order to further improve the detection accuracy of the convolutional neural network when detecting the detection target slice, the detection target slice is size normalized before being input into the convolutional neural network.

[0086] Furthermore, considering the influence of factors such as the optical satellite platform type, orbital altitude, and observation angle, the resolution, width, and other information of different satellite remote sensing images are usually different. Therefore, in the continuous tracking scenario of marine targets based on low-orbit optical satellites, it is difficult to uniformly describe the pixel position and size of the target as the state as in traditional video tracking. In one embodiment of the present invention, combined with the characteristics of satellite applications, the longitude and latitude coordinates of the target, the speed of the target in the longitude and latitude directions, and the length and width of the target are used as the state of the target. The state of the target is specifically expressed as follows:

[0087]

[0088] Where X represents the state vector of the target, and λ represent the longitude and latitude coordinates of the target, respectively, including the longitude and latitude coordinates of the four corner points of the target oblique frame. and They represent the speed of the target in the longitude and latitude directions respectively, l and w represent the length and width of the target respectively. The length and width of the target can be estimated by the length and width of the target oblique frame and can be considered unchanged.

[0089] Furthermore, in one embodiment of the present invention, the detection state corresponding to the detection target is determined based on the detection result corresponding to the detection target, with the longitude and latitude coordinates of the detection target, and the length and width of the detection target as the detection state. The detection state is specifically expressed as:

[0090]

[0091] Among them, Z represents the detection state vector of the detection target, and Respectively represent the longitude and latitude coordinates of the detection target, including the longitude and latitude coordinates of the four corner points of the detection target oblique frame, and Represent the length and width of the detection target respectively. The length and width of the detection target can be estimated by the length and width of the detection target oblique frame.

[0092] Furthermore, in one embodiment of the present invention, based on the target trajectory and target state of the currently known target, and the detection state and depth appearance features of each detected target, multi-feature matching is used to perform data association between the target trajectory and the detection result, specifically including the following steps S321-S325:

[0093] Step S321: Measure target motion using the target state and the detection state.

[0094] In one embodiment of the present invention, taking the i-th known target and the j-th detected target as an example, the distance metric between the motion information, that is, the target motion metric, is defined as:

[0095]

[0096] Among them, d (1) (i, j) represents the target motion metric of the i-th known target and the j-th detected target, X i and S i denote the target state and covariance matrix of the i-th known target, Z j represents the detection state of the jth detection target, and T represents the transposed matrix.

[0097] It should be noted that, in one embodiment of the present invention, the target state covariance matrix S i Obtained through recursion of the target state filter.

[0098] Step S322: Measure the target attributes using the target's appearance features.

[0099] In one embodiment of the present invention, taking the i-th known target and the j-th detected target as an example, the distance metric between the attribute information, that is, the target attribute metric, is defined as:

[0100]

[0101] Among them, d (2) (i, j) represents the target attribute measurement of the i-th known target and the j-th detected target, r j Represents the deep appearance feature corresponding to the j-th detection target, R i represents the appearance feature set of the i-th known target, L k represents the number of appearance feature vectors, represents the kth appearance feature vector of the i-th known target, and T represents the transposed matrix.

[0102] It should be noted that, in one embodiment of the present invention, when a detection target is associated with a target, the depth appearance features corresponding to the detection target are added to the appearance feature set of the target, that is, the appearance feature set of the known target is composed of the depth appearance features of the detection target associated with it.

[0103] Step S323: perform joint motion and attribute measurement using the target motion measurement result and the target attribute measurement result.

[0104] In one embodiment of the present invention, taking the i-th known target and the j-th detected target as an example, the joint motion and attribute metric is defined as:

[0105] c i,j =qd (1) (i,j)+(1-q)d (2) (i,j)

[0106] Among them, c i,j represents the joint measurement of the motion and attributes of the jth detected target of the i-th known target, and q represents the weighting coefficient. The weighting coefficient q is set according to the actual situation.

[0107] In step S324, the joint motion and attribute measurement results corresponding to the known target and the detected target are used as the allocation cost, and the allocation cost corresponding to the i-th known target and the j-th detected target is used as the matrix element of the i-th row and j-th column to form the cost matrix C, and it is determined whether the target motion measurement result and the target attribute measurement result corresponding to the i-th known target and the j-th detected target meet the preset first distance threshold value and the preset attribute threshold value respectively. If so, the corresponding matrix elements in the cost matrix are kept unchanged; if not, the corresponding matrix elements in the cost matrix are set to infinity to update the cost matrix.

[0108] In one embodiment of the present invention, the first distance threshold and the attribute threshold are specifically set according to actual conditions.

[0109] In step S325 , the minimum allocation cost of the cost matrix is ​​solved using the allocation algorithm to obtain the optimal data association result and the corresponding relationship between the target trajectory and the detection result.

[0110] In one embodiment of the present invention, the Kuhn-Munkres algorithm is used as the allocation algorithm. This algorithm solves the cost matrix for the minimum allocation cost, obtains the optimal data association result, determines the correspondence between target trajectories and detection results, and further determines target trajectories associated with detection results, target trajectories not associated with detection results, and detection results not associated with target trajectories.

[0111] Step S4: For target trajectories not associated with detection results and detection results not associated with target trajectories, data association is performed between the target trajectories and detection results using oblique frame intersection-over-union matching to determine target trajectories associated with detection results, target trajectories not associated with detection results, and detection results not associated with target trajectories.

[0112] refer to Figure 2 In one embodiment of the present invention, for the target tracks not associated with the detection results and the detection results not associated with the target tracks determined in step S3 above, the intersection over union (IOU) of the target oblique box corresponding to each target track and the detection target oblique box corresponding to each detection result is calculated, and the IOU calculation result of the oblique box IOU is used to perform data association between the target track and the detection results to determine the target tracks associated with the detection results, the target tracks not associated with the detection results, and the detection results not associated with the target tracks.

[0113] Specifically, in one embodiment of the present invention, the intersection-over-union (IOU) of the target oblique frame corresponding to each target trajectory and the detected target oblique frame corresponding to each detection result is calculated, and the data association of the target trajectory and the detection result is performed using the IOU calculation result of the oblique frame. The following steps are included:

[0114] Calculate the intersection and union ratio of the target oblique box corresponding to each target trajectory and the detection target oblique box corresponding to each detection result, use the intersection and union ratio calculation result of the oblique box corresponding to the known target and the detection target as the allocation cost, and use the allocation cost corresponding to the mth known target and the nth detection target as the matrix element of the mth row and nth column to form the cost matrix D, and determine the intersection and union ratio of the oblique box corresponding to the mth known target and the nth detection target RIOU m,n Whether the preset second distance threshold is met, if so, the corresponding matrix element in the cost matrix is ​​kept unchanged, if not, the corresponding matrix element in the cost matrix is ​​set to infinity to update the cost matrix.

[0115] It should be noted that, in an embodiment of the present invention, the second distance threshold is specifically set according to actual conditions.

[0116] In one embodiment of the present invention, the allocation algorithm is also used to solve the minimum allocation cost of the cost matrix D, obtain the optimal data association result, and acquire the corresponding relationship between the target trajectory and the detection result.

[0117] In one embodiment of the present invention, the Kuhn-Munkres algorithm is used as an allocation algorithm. This algorithm solves the cost matrix for the minimum allocation cost, yielding an optimal data association result. This algorithm then determines target trajectories associated with detection results, target trajectories not associated with detection results, and detection results not associated with target trajectories.

[0118] Step S5: update the target trajectory and predict the target state according to the association result.

[0119] In one embodiment of the present invention, target trajectory update and target state prediction are performed based on the association result obtained in step S3 and the association result obtained in step S4.

[0120] refer to Figure 2 Specifically, target trajectory updating and target state prediction are performed according to the association results, further comprising the following steps:

[0121] If there is a detection result associated with the current target trajectory of a known target, the detection result is used to update the target trajectory and predict the target state of the corresponding known target;

[0122] If there is no detection result associated with the current target track of a known target, further determine whether the current target track is a confirmed track. If not, delete the current target track. If so, further determine whether the number of consecutive unassociated times of the current target track reaches a first preset number. If so, delete the current target track.

[0123] If a detection result is not associated with any target track, the detection result is regarded as an unconfirmed track;

[0124] Wherein, when the number of times an unconfirmed trajectory is continuously associated with a detection result reaches a second preset number, it is adjusted to a confirmed trajectory.

[0125] In one embodiment of the present invention, the first preset number K and the second preset number M can be specifically set according to actual conditions.

[0126] Furthermore, in one embodiment of the present invention, when using the detection results to update the target trajectory and predict the target state of the corresponding known target, the detection result is used as the next-moment trajectory point of the target trajectory associated with it to update the target trajectory, and the extended Kalman filter algorithm is used to predict the target state at the next moment based on the current target trajectory, the current target state, the detection result and the detection state.

[0127] A method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion provided by one embodiment of the present invention obtains an image sequence of the mission area through a low-orbit optical satellite constellation. Based on the target detection results of the image sequence, multi-feature matching and oblique frame intersection-to-union matching are used for data association. This method can quickly and accurately achieve continuous tracking and identification of maritime targets in the mission area, significantly improving maritime situation awareness capabilities.

[0128] The following describes the beneficial effects of the method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion provided by an embodiment of the present invention with reference to specific examples:

[0129] In Example 1, a port was selected as the mission area for the experiment. Five low-orbit optical satellites were used to continuously image the mission area within one hour. The resolution of all satellite remote sensing images was about 1 meter. The YOLOv7 oblique frame target detection network was used to detect targets on the acquired optical satellite remote sensing images. The target detection results of one complete optical satellite remote sensing image are shown in the attached figure. Figure 3 As shown. Figure 3 The target detection results for the two zoomed-in local areas in the figure show that each detection result is represented by a slanted box. This mission area contains many stationary and moving maritime targets, with significant variations in size and some closely spaced, resulting in a complex scene. In Example 1, the YOLOv7 slanted box target detection algorithm was used for target detection. Both target detection accuracy and recall rates exceeded 80%, meeting the application requirements of subsequent tracking methods.

[0130] In Example 1, data association is performed based on the target detection results of optical satellite remote sensing images, and the marine target tracking results obtained are shown in the attached figure. Figure 4 As shown, the target detection results are represented as points, the target tracking trajectory is represented as a line, and "o" and "+" represent the starting point and end point of the target trajectory, respectively. It can be seen that through target tracking, target information from multiple low-orbit optical satellites can be effectively integrated to form a more comprehensive maritime target situation map. In Example 1, by adopting the data association method provided above by an embodiment of the present invention, the target tracking recall rate has been significantly improved, approaching the target detection recall rate, indicating that the target can be successfully tracked once it is discovered, verifying the effectiveness of the method provided by an embodiment of the present invention for continuous tracking of maritime targets.

[0131] Attachment Figure 5 Schematic diagram of a slice of a moving target extracted from 5 optical satellite remote sensing images. Figure 6 The figure shows a slice diagram of another moving target extracted from five optical satellite remote sensing images. It can be seen that there are some differences in the appearance of the targets in different optical satellite remote sensing images. These targets can be effectively identified by matching features such as motion and appearance. By continuously tracking the maritime target, not only can the dynamic information of the target be obtained, but more attribute information can also be obtained from the target slices, supporting further target classification and data fusion. This verifies the application value of the method provided in one embodiment of the present invention for maritime target tracking.

[0132] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, "front", "back", "left", "right", "upper" and "lower" in this document are all referenced to the placement states shown in the accompanying drawings.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion, characterized in that: include: Utilize multiple low-orbit optical satellites in a low-orbit optical satellite constellation to carry out push-broom imaging of a preset mission area, and obtain a sequence of optical satellite remote sensing images taken by multiple low-orbit optical satellites; Utilize the pre-trained oblique frame target detection network model to perform target detection on each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and obtain the discovery time and detection result corresponding to each detected target in the optical satellite remote sensing image. The detection result includes the oblique frame position and geographic coordinate position corresponding to the detected target. Each detected target slice is input into a pre-trained convolutional neural network model to obtain the deep appearance features corresponding to the detected target output by the convolutional neural network model. Based on the target trajectory and target state of the currently known target, as well as the detection state and deep appearance features of each detected target, multi-feature matching is used to perform data association between the target trajectory and the detection result to determine the target trajectory associated with the detection result, the target trajectory not associated with the detection result, and the detection result not associated with the target trajectory; For target tracks not associated with detection results and detection results not associated with target tracks, data association is performed on the target tracks and detection results using oblique frame intersection-union matching to determine target tracks associated with detection results, target tracks not associated with detection results, and detection results not associated with target tracks; The target trajectory is updated and the target state is predicted based on the association results.

2. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 1 is characterized in that: The pre-trained oblique frame target detection network model is used to detect targets in each optical satellite remote sensing image in the optical satellite remote sensing image sequence, and the discovery time and detection results corresponding to each detected target in the optical satellite remote sensing image are obtained, including: Input each optical satellite remote sensing image in the optical satellite remote sensing image sequence into a pre-trained oblique frame target detection network model, and obtain the type and oblique frame position corresponding to the detected target in each optical satellite remote sensing image output by the oblique frame target detection network model; The discovery time of the detection target is determined according to the shooting time of the optical satellite remote sensing image, and the geographic coordinate position of the detection target is calculated according to the image metadata and the oblique frame position of the detection target in the optical satellite remote sensing image.

3. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 1 is characterized in that: The oblique frame object detection network model is trained through the following steps: Acquire a first training data set, the first training data including: sample images containing marine targets, types of marine targets in the sample images, and oblique frame position information; The sample image of the first training data in the first training data set is used as input, the type and oblique frame position information of the marine target corresponding to the input sample image is used as output, and an oblique frame target detection network model is trained.

4. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 1, characterized in that: The target state is expressed as follows: the target's longitude and latitude coordinates, the target's speed in the longitude and latitude directions, and the target's length and width are known. X represents the state vector of the target, and λ represent the longitude and latitude coordinates of the target, respectively, including the longitude and latitude coordinates of the four corner points of the target oblique frame. and They represent the speed of the target in the longitude and latitude directions respectively, l and w represent the length and width of the target respectively; The detection state of the detection target is expressed as the longitude and latitude coordinates of the detection target, as well as the length and width of the detection target, as follows: Z represents the detection state vector of the detected target, and Respectively represent the longitude and latitude coordinates of the detection target, including the longitude and latitude coordinates of the four corner points of the detection target oblique frame, and Represent the length and width of the detection target respectively.

5. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 4 is characterized in that: Based on the target trajectory and target state of the currently known target, as well as the detection state and deep appearance features of each detected target, multi-feature matching is used to perform data association between the target trajectory and detection results, including: Target motion measurement using target state and detection state; Utilize the appearance features of the target to measure the target attributes; Utilize the target motion measurement results and target attribute measurement results to perform joint motion and attribute measurement; The joint measurement results of motion and attributes corresponding to the known target and the detected target are used as the allocation cost, and the allocation cost corresponding to the i-th known target and the j-th detected target is used as the matrix element of the i-th row and j-th column to form a cost matrix C, and determine whether the target motion measurement result and the target attribute measurement result corresponding to the i-th known target and the j-th detected target meet the preset first distance threshold value and the preset attribute threshold value respectively. If so, keep the corresponding matrix element in the cost matrix unchanged; if not, set the corresponding matrix element in the cost matrix to infinity to update the cost matrix; The allocation algorithm is used to solve the minimum allocation cost of the cost matrix, obtain the optimal data association result, and obtain the correspondence between the target trajectory and the detection result.

6. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 5, characterized in that: The target motion metric is defined as: Among them, d (1) (i, j) represents the target motion measurement between the i-th known target and the j-th detected target, X i and S i denote the target state and covariance matrix of the i-th known target, Z j represents the detection state of the jth detection target, and T represents the transposed matrix.

7. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 6, characterized in that: The target attribute metric is defined as: Among them, d (2) (i, j) represents the target attribute measurement of the i-th known target and the j-th detected target, r j Represents the deep appearance feature corresponding to the j-th detection target, R i represents the appearance feature set of the i-th known target, L k represents the number of appearance feature vectors, represents the kth appearance feature vector of the i-th known target.

8. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 7, characterized in that: The joint motion and attribute metric is defined as: c i,j =qd (1) (i,j)+(1-q)d (2) (i,j); Among them, c i,j It represents the joint measurement of motion and attributes of the i-th known target and the j-th detected target, and q represents the weighting coefficient.

9. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 8, characterized in that: For target trajectories that are not associated with detection results and detection results that are not associated with target trajectories, data association is performed between the target trajectories and detection results using oblique frame intersection and union matching, including: For target tracks that are not associated with detection results and detection results that are not associated with target tracks, the intersection-and-union ratio of the target slant box corresponding to each target track and the detection target slant box corresponding to each detection result is calculated, and the result of the slant box intersection-and-union ratio calculation is used to perform data association between the target track and the detection results.

10. The method for continuous tracking of maritime targets based on low-orbit optical satellite constellation data fusion according to claim 9, characterized in that: Target trajectory update and target state prediction are performed based on the association results, including: If there is a detection result associated with the current target trajectory of a known target, the detection result is used to update the target trajectory and predict the target state of the corresponding known target; If there is no detection result associated with the current target track of a known target, further determine whether the current target track is a confirmed track. If not, delete the current target track. If so, further determine whether the number of consecutive unassociated times of the current target track reaches a first preset number. If so, delete the current target track. If a detection result is not associated with any target track, the detection result is regarded as an unconfirmed track; Wherein, when the number of times an unconfirmed trajectory is continuously associated with a detection result reaches a second preset number, it is adjusted to a confirmed trajectory.

Images