Optical satellite video multi-target tracking method and device based on target relation interaction
Through the multi-objective tracking method of optical satellite video based on target relationship interaction, the image segmentation and multi-grained space-time comparison learning module are used to solve the problem of low tracking accuracy caused by weak target features and similar individual features in optical satellite videos, and the multi-efficient multi-dimensional Target tracking accuracy is improved.
Patent Information
- Application Number
- CN202410984718.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-12-03
Smart Images

Figure CN119068168A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an optical satellite video multi-target tracking method and device based on target relationship interaction. Background Art
[0002] As a new type of space-based earth observation method that has emerged in recent years, HROSV (High-Resolution Optical Satellite Video) can achieve continuous monitoring of moving targets in a specific area over a period of time through staring imaging. Compared with high-resolution optical remote sensing images, it has a higher temporal resolution and a wider observation field of view than drone ground videos and ground surveillance videos. Multi-target tracking based on HROSV can extract the movement trajectories of a large number of ground targets over a period of time, and plays an important role in traffic perception, military reconnaissance, commercial analysis and other fields, so it has attracted the attention of many researchers.
[0003] Multi-target tracking usually includes two subtasks: detection of targets of interest and inter-frame association. Compared with multi-target tracking based on other video data, multi-target tracking based on HRSOV has the following two significant challenges: (1) Detection is difficult due to weak target features. Satellite videos are collected from far away. Moving targets (such as cars, ships, and airplanes) have low spatial resolution, small size, sparse texture features, low contrast with the background, and motion blur, making it difficult to form a stable and effective appearance feature expression for the targets; (2) Similarity of individual features leads to association errors. The basis for inter-frame association is that the feature expression of the same individual target is consistent and unique in time sequence. The long-distance bird's-eye view used in satellite videos makes the texture features and perspective features of the target missing, resulting in low individual target distinction in local areas and confusion.
[0004] In the prior art, multi-target tracking methods can be divided into two mainstream paradigms according to the coupling form of the two subtasks of target detection and inter-frame association: TBD (Tracking-By-Detection, detection and association serial processing) and JDT (Joint Detection and Tracking, detection and association joint learning). TBD first detects the target in each frame and then associates it. This two-stage processing method can achieve flexible combination of local optimal modules, but the performance of TBD is heavily dependent on high-precision detection results. In this regard, JDT adopts a one-stage multi-task learning approach to simultaneously achieve detection and association.
[0005] In terms of weak target detection, JDT currently mainly adopts two strategies: multi-scale feature extraction and historical information compensation, and searches for the target area of information compensation on a global scale, which inevitably leads to problems of error compensation and complex calculations, and ultimately limits the improvement of target detection performance. In addition, in terms of target association between JDT frames, appearance feature matching and motion feature matching are the two mainstream solutions. For satellite video targets with scarce texture features, using the feature vector of the detection area as a target association clue is inefficient and unreliable, and cannot effectively distinguish individual differences. In this case, the target motion feature can have a certain degree of individual distinguishability, but it is easy to fail in local scenes with dense targets. In addition, the learning competition between detection tasks and association tasks is a common problem faced by JDT methods.
[0006] Therefore, the multi-target tracking method in the related technology is prone to error compensation and increases computational complexity, resulting in limited improvement in target detection performance. Especially for satellite video targets with scarce texture features, the feature vectors of the detection area cannot be used for target association clues, and individual differences cannot be effectively distinguished, which reduces the accuracy of satellite video multi-target tracking and needs to be solved urgently. Summary of the invention
[0007] The present application provides an optical satellite video multi-target tracking method and device based on target relationship interaction, so as to solve the problem that the multi-target tracking method in the related art is prone to error compensation and increases the computational complexity, resulting in limited improvement in target detection performance, especially for satellite video targets with scarce texture features, individual differences cannot be effectively distinguished, thereby reducing the accuracy of satellite video multi-target tracking.
[0008] The first aspect of the present application provides an optical satellite video multi-target tracking method based on target relationship interaction, comprising the following steps: obtaining a current frame satellite video image and a previous frame satellite video image of the target satellite video, and performing target detection on the current frame satellite video image to determine the target detection result of the current frame satellite video image; performing image segmentation processing on the current frame satellite video image and the previous frame satellite video image to obtain a plurality of target slice images that meet preset size conditions; inputting the plurality of target slice images and the target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking result based on target relationship interaction.
[0009] Optionally, in one embodiment of the present application, the multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, including: obtaining a target node set based on a feature vector set of the target center position of the previous frame satellite video image and a neighborhood feature vector set corresponding to the current frame satellite video image; making the target node sets adjacent to each other and non-adjacent to each other within the target node sets to obtain an edge set; and using the edge set to construct an undirected bipartite graph that meets preset conditions.
[0010] Optionally, in one embodiment of the present application, the multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, including: determining a template node based on the feature vector set, and determining a search node using the neighborhood feature vector set; determining a target node to be enhanced using the attention score of the template node and the attention score of the search node; guiding the target node to be enhanced to perform feature enhancement to obtain an enhanced target node; based on the enhanced target node, normalizing the attention value of each edge to obtain a normalized processing result; using the normalized processing result to guide the node features of the previous moment to aggregate to the node features of the current moment, and updating the aggregated result to obtain the features of the enhanced target area of the current frame satellite video image.
[0011] Optionally, in one embodiment of the present application, the multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, including: based on the characteristics of the enhanced target area, the target true value target features of the current frame satellite video image and the previous frame satellite video image, respectively calculating the spatiotemporal contrast loss of individual granularity and the spatiotemporal contrast loss of category granularity; and determining the multi-granularity spatiotemporal contrast loss using the spatiotemporal contrast loss of the individual granularity and the spatiotemporal contrast loss of the category granularity.
[0012] Optionally, in one embodiment of the present application, the multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, including: based on the multi-granularity spatiotemporal contrast loss, all target positions of the previous frame satellite video image and the current frame satellite video image, respectively obtaining the topological relationship between each target and the adjacent target in the frame of the previous frame and the current frame, and obtaining the target detection results of two adjacent frames; using the target detection results of the two adjacent frames to obtain the appearance features and motion offsets of the previous frame satellite video image and the current frame satellite video image; based on the topological relationship, the appearance features and the motion offset, evaluating the similarity of target features between frames to obtain a first similarity of the topological relationship, a second similarity of the appearance features and a third similarity of the motion offset; using the first similarity, the second similarity and the third similarity to obtain the similarity of target joint features, and obtaining the optical satellite video multi-target tracking results based on target relationship interaction according to the similarity of the target joint features.
[0013] Optionally, in one embodiment of the present application, the formula for the similarity of the target joint features is:
[0014]
[0015] Among them, Sim(F t,j ,F t-1,i ) is the similarity of appearance features, is the similarity of motion offset, is the predicted value of the envelope of the tth frame, B t,j is the true value, IoU is the intersection over union ratio, Sim(T t,j ,T t-1,i ) is the target V t,j With target V t-1,i The topological similarity of .
[0016] A second aspect of the present application provides an optical satellite video multi-target tracking device based on target relationship interaction, including: an acquisition module, used to acquire a current frame satellite video image and a previous frame satellite video image of a target satellite video, and perform target detection on the current frame satellite video image to determine the target detection result of the current frame satellite video image; a determination module, used to perform image segmentation processing on the current frame satellite video image and the previous frame satellite video image to obtain a plurality of target slice images that meet preset size conditions; a tracking module, used to input the plurality of target slice images and the target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking result based on target relationship interaction.
[0017] Optionally, in one embodiment of the present application, the tracking module includes: a first acquisition unit, used to obtain a target node set based on a feature vector set of a target center position of the previous frame of satellite video image and a neighborhood feature vector set corresponding to the current frame of satellite video image; a first determination unit, used to make the target node sets adjacent to each other and non-adjacent to each other within the target node sets to obtain an edge set; and a construction unit, used to use the edge set to construct an undirected bipartite graph that meets preset conditions.
[0018] Optionally, in one embodiment of the present application, the tracking module includes: a second determination unit, used to determine the template node according to the feature vector set, and determine the search node using the neighborhood feature vector set; a third determination unit, used to determine the target node to be enhanced using the attention score of the template node and the attention score of the search node; a fourth determination unit, used to guide the target node to be enhanced to perform feature enhancement to obtain an enhanced target node; a fifth determination unit, used to normalize the attention value of each edge based on the enhanced target node to obtain a normalized processing result; a processing unit, used to use the normalized processing result to guide the node features of the previous moment to aggregate to the node features of the current moment, and update the aggregated result to obtain the features of the enhanced target area of the current frame satellite video image.
[0019] Optionally, in one embodiment of the present application, the tracking module includes: a calculation unit, used to calculate the spatiotemporal contrast loss of individual granularity and the spatiotemporal contrast loss of category granularity based on the characteristics of the enhanced target area, the target true value target characteristics of the current frame satellite video image and the previous frame satellite video image; a sixth determination unit, used to determine the multi-granularity spatiotemporal contrast loss using the spatiotemporal contrast loss of the individual granularity and the spatiotemporal contrast loss of the category granularity.
[0020] Optionally, in one embodiment of the present application, the tracking module includes: a second acquisition unit, which is used to obtain the topological relationship between each target and the adjacent target in the previous frame and the current frame based on the multi-granularity spatiotemporal contrast loss, all target positions of the previous frame satellite video image and the current frame satellite video image, and obtain the target detection results of two adjacent frames; a third acquisition unit, which is used to obtain the appearance features and motion offsets of the previous frame satellite video image and the current frame satellite video image using the target detection results of the two adjacent frames; a seventh determination unit, which is used to evaluate the similarity of target features between frames based on the topological relationship, the appearance features and the motion offset, so as to obtain a first similarity of the topological relationship, a second similarity of the appearance features and a third similarity of the motion offset; a fourth acquisition unit, which is used to obtain the similarity of target joint features using the first similarity, the second similarity and the third similarity, and obtain the optical satellite video multi-target tracking result based on target relationship interaction according to the similarity of the target joint features.
[0021] Optionally, in one embodiment of the present application, the formula for the similarity of the target joint features is:
[0022]
[0023] Among them, Sim(F t,j ,F t-1,i ) is the similarity of appearance features, is the similarity of motion offset, is the predicted value of the envelope of the tth frame, B t,j is the true value, IoU is the intersection over union ratio, Sim(T t,j ,T t-1,i ) is the target V t,j With target V t-1,i The topological similarity of .
[0024] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the optical satellite video multi-target tracking method based on target relationship interaction as described in the above embodiment.
[0025] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned optical satellite video multi-target tracking method based on target relationship interaction.
[0026] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned optical satellite video multi-target tracking method based on target relationship interaction.
[0027] The embodiment of the present application can perform image segmentation processing on the current frame and the previous frame of satellite video images to obtain multiple target slice images that meet certain size conditions and the target detection results obtained by performing target detection on the current frame of satellite video images, and input them into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction, so as to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, effectively reduce the computational complexity of multi-target tracking, and improve the accuracy of satellite video multi-target tracking. Thus, the problem that the multi-target tracking method in the related art is prone to error compensation and increases computational complexity, resulting in limited improvement in target detection performance, especially for satellite video targets with scarce texture features, individual differences cannot be effectively distinguished, and the accuracy of satellite video multi-target tracking is reduced is solved.
[0028] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0030] Figure 1 A flowchart of an optical satellite video multi-target tracking method based on target relationship interaction provided according to an embodiment of the present application;
[0031] Figure 2 A schematic diagram of an optical satellite video multi-target tracking network model based on target relationship interaction according to a specific embodiment of the present application;
[0032] Figure 3 A schematic diagram of the structure of an optical satellite video multi-target tracking device based on target relationship interaction provided according to an embodiment of the present application;
[0033] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0034] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0035] The following describes the optical satellite video multi-target tracking method and device based on target relationship interaction of the embodiment of the present application with reference to the accompanying drawings. In view of the fact that the multi-target tracking method in the related art mentioned in the above background technology is easy to cause error compensation and increase the computational complexity, resulting in limited improvement in target detection performance, especially for satellite video targets with insufficient texture features, individual differences cannot be effectively distinguished, and the accuracy of satellite video multi-target tracking is reduced, the present application provides an optical satellite video multi-target tracking method based on target relationship interaction, in which the target detection results obtained by performing image segmentation processing on the current frame and the previous frame of satellite video images and performing target detection on the current frame of satellite video images can be input into the pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, effectively reducing the computational complexity of multi-target tracking and improving the accuracy of satellite video multi-target tracking. Thus, the problem that the multi-target tracking method in the related art is easy to cause error compensation and increase the computational complexity, resulting in limited improvement in target detection performance, especially for satellite video targets with insufficient texture features, individual differences cannot be effectively distinguished, and the accuracy of satellite video multi-target tracking is reduced is solved.
[0036] Specifically, Figure 1 A flowchart of an optical satellite video multi-target tracking method based on target relationship interaction provided in an embodiment of the present application.
[0037] like Figure 1 As shown, the optical satellite video multi-target tracking method based on target relationship interaction includes the following steps:
[0038] In step S101, a current frame satellite video image and a previous frame satellite video image of a target satellite video are obtained, and target detection is performed on the current frame satellite video image to determine a target detection result of the current frame satellite video image.
[0039] It can be understood that the embodiments of the present application can obtain the current frame satellite video image and the previous frame satellite video image of the target satellite video, that is, obtain two adjacent frames of satellite video images, and perform target detection on the current frame satellite video image to determine the target detection result of the current frame satellite video image. For example, the embodiments of the present application can perform target detection on the current frame video image through a deep learning model, and determine the category and position of the target, etc., thereby effectively improving the feasibility of multi-target tracking of optical satellite videos.
[0040] In step S102, the current frame satellite video image and the previous frame satellite video image are subjected to image segmentation processing to obtain a plurality of target slice images that meet a preset size condition.
[0041] It can be understood that the embodiment of the present application can perform image segmentation processing on the current frame satellite video image and the previous frame satellite video image to obtain multiple target slice images that meet the preset size conditions. For example, the embodiment of the present application can segment the previous frame satellite video image and the previous frame satellite video image into several slice images of size h. I × I Pixel slice images effectively improve the accuracy of multi-target tracking in high-resolution optical satellite videos.
[0042] In step S103, the multiple target slice images and target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction.
[0043] It can be understood that the embodiments of the present application can input the multiple target slice images and target detection results in the following steps into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain high-resolution optical satellite video multi-target tracking results based on target relationship interaction, thereby achieving higher-precision satellite video multi-target tracking.
[0044] For example, Figure 2 As shown, the embodiment of the present application can perform tracking processing according to the spatiotemporal relationship sparse attention module, multi-granularity spatiotemporal contrast learning module, motion clue extraction module, target detection module, joint feature matching module and target inter-frame association module in the pre-trained optical satellite video multi-target tracking model based on target relationship interaction to obtain satellite video multi-target tracking results, wherein the spatiotemporal relationship sparse attention module is composed of an inter-frame graph construction module guided by historical trajectories and a target node feature enhancement module based on graph attention aggregation, which effectively reduces the computational complexity of multi-target tracking and improves the accuracy of optical satellite video multi-target tracking.
[0045] Optionally, in one embodiment of the present application, multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain optical satellite video multi-target tracking results based on target relationship interaction, including: obtaining a target node set based on a feature vector set of the target center position of the previous frame satellite video image and a neighborhood feature vector set corresponding to the current frame satellite video image; making the target node sets adjacent to each other and the target node sets non-adjacent to each other to obtain an edge set; and using the edge set to construct an undirected bipartite graph that meets preset conditions.
[0046] In the actual implementation process, Figure 2As shown, the embodiment of the present application can use the historical trajectory of the spatiotemporal relationship sparse attention module to guide the inter-frame graph construction module to construct an undirected bipartite graph for describing the relationship between target frames. Specifically, the feature vector set V of the target center position of the previous frame of satellite video image can be t-1 The neighborhood feature vector set corresponding to the current frame satellite video image As two sets of node sets, in addition, the two sets of node sets are adjacent to each other, but the two sets of node sets are not adjacent to each other, so as to obtain the edge set E, where E and Respectively expressed as:
[0047]
[0048] Where E is the edge set, is an undirected bipartite graph, V t-1 is the feature vector set, is the neighborhood feature vector set, V j and V i Represents search node and template node respectively.
[0049] Optionally, in one embodiment of the present application, multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain optical satellite video multi-target tracking results based on target relationship interaction, including: determining a template node based on a feature vector set, and determining a search node using a neighborhood feature vector set; determining a target node to be enhanced using an attention score of the template node and an attention score of the search node; guiding the target node to be enhanced to perform feature enhancement to obtain an enhanced target node; based on the enhanced target node, normalizing the attention value of each edge to obtain a normalized processing result; using the normalized processing result to guide the node features of the previous moment to aggregate to the node features of the current moment, and updating the aggregated result to obtain the features of the enhanced target area of the current frame satellite video image.
[0050] In some embodiments, Figure 2 As shown, the target node feature enhancement module based on graph attention aggregation of the spatiotemporal relationship sparse attention module in the embodiment of the present application obtains After that, V t-1 and The target nodes to be enhanced are respectively regarded as template nodes and search nodes, and the target nodes to be enhanced are determined by the attention scores of the template nodes and the search nodes. Since the higher the attention score, the higher the similarity between the search node and the template node, the more likely they are the same target. Therefore, based on the above analysis, the embodiment of the present application can introduce graph attention aggregation to guide the enhancement of target node features. For each edge (V j , V i )∈E, whose attention value is e j,i It is expressed as:
[0051] e j,i =A(V j , V i )=(W j V j )(W i V i ) T (3)
[0052] Among them, A is the graph attention module, V j and V i are search nodes and template nodes respectively, W j and W i They are the linear mapping matrices corresponding to the search node and template node respectively.
[0053] Next, for e j,i Normalize to get e_norm j,i , to represent V j With V i The degree of similarity is:
[0054]
[0055] Among them, softmax(e j,i ) is the activation function.
[0056] Secondly, e_norm j,i Used to guide the node features at time t-1 Node characteristics at time t Perform aggregation and update, namely:
[0057]
[0058] Among them, l is the number of the current computational layer after the node feature propagation; L is the number of graph attention layers, which can be set to 2 here; W h is the learnable weight coefficient.
[0059] In addition, for each edge (V i , V j)∈E, and its feature update process can also refer to formulas (3)-(5). After multiple graph attention layers are calculated, the features of the target area of the current frame satellite video image can be effectively enhanced.
[0060] It can be seen from this that the embodiment of the present application can construct a spatiotemporal sparse graph of the known position of the target in the previous frame and the potential position in the current frame through the spatiotemporal relationship sparse attention module, accurately capture the cross-frame semantics related to the target area, and thus effectively enhance the expression of weak target features.
[0061] Optionally, in one embodiment of the present application, multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain optical satellite video multi-target tracking results based on target relationship interaction, including: based on the characteristics of the enhanced target area, the target true value target features of the current frame satellite video image and the previous frame satellite video image, respectively calculating the spatiotemporal contrast loss of individual granularity and the spatiotemporal contrast loss of category granularity; using the spatiotemporal contrast loss of individual granularity and the spatiotemporal contrast loss of category granularity to determine the multi-granularity spatiotemporal contrast loss.
[0062] As a possible way to achieve this, Figure 2 As shown, the multi-granularity spatiotemporal contrast learning module in the embodiment of the present application can calculate the spatiotemporal contrast loss of the individual granularity. Specifically, for a true value target feature V in the current frame satellite video image and the previous frame satellite video image, that is, in two adjacent frames of satellite video images, the true value target feature with the same ID is regarded as a positive sample. True target features with different IDs are considered negative samples Then the temporal and spatial contrast loss of individual particle size can be calculated according to the following formula (6):
[0063]
[0064] in, is the similarity of negative sample pairs of true target features with different IDs from a true target feature, is the similarity of the positive sample pairs of the true target feature with the same ID as a true target feature, L ind Always greater than or equal to 0.
[0065] Among them, when The smaller and The larger the value, that is, the more dissimilar the features of the negative sample pairs are and the more similar the features of the positive sample pairs are, the greater the temporal and spatial contrast loss L of the individual granularity ind The smaller it is, the better the association performance will be.
[0066] Secondly, the embodiment of the present application can also calculate the temporal and spatial contrast loss of category granularity. Specifically, similar to the temporal and spatial contrast loss of individual granularity, for a true value target feature V in two adjacent frames, the true value target feature belonging to the same category is regarded as a positive sample The true target features belonging to different categories are considered negative samples Then the spatiotemporal contrast loss L of the category granularity can be calculated according to the following formula (7): cat ,Right now:
[0067]
[0068] in, is the similarity of negative sample pairs of true target features that belong to different categories from a true target feature, is the similarity of the positive sample pairs of the true target feature that belongs to the same category as a true target feature. cat The smaller it is, the higher the classification accuracy of the target.
[0069] Again, the embodiment of the present application can perform weighted addition of the spatiotemporal contrast loss of the individual granularity and the spatiotemporal contrast loss of the category granularity to obtain the multi-granularity spatiotemporal contrast loss L MGSTCL ,Right now:
[0070] L MGSTCL =0.5*L ind +0.5*L cat (8)
[0071] Among them, placing the multi-granularity spatiotemporal contrast loss after the calculation step of the spatiotemporal relationship sparse attention module can alleviate the conflict of multi-task related feature extraction and improve the multi-category and multi-target tracking accuracy of optical satellite videos.
[0072] It should be noted that the multi-granularity spatiotemporal contrast learning module in the embodiment of the present application only uses the L calculated above during training. MGSTCL Used for back propagation of network model parameters.
[0073] It can be seen from this that the embodiments of the present application can use a multi-granularity spatiotemporal contrast learning module to perform distinguishability training on target features at individual granularity and category granularity, so that the differences between individuals in the same category are reduced and the differences between individuals in the same category are smaller than the differences between categories, thereby maintaining a multi-task optimization balance and improving the accuracy of multi-target tracking.
[0074] Optionally, in one embodiment of the present application, multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, including: based on multi-granular spatiotemporal contrast loss, all target positions of the previous frame satellite video image and the current frame satellite video image, respectively obtaining the topological relationship between each target and the adjacent target in the previous frame and the current frame, and obtaining the target detection results of two adjacent frames; using the target detection results of two adjacent frames to obtain the appearance features and motion offsets of the previous frame satellite video image and the current frame satellite video image; based on the topological relationship, appearance features and motion offset, evaluating the similarity of target features between frames to obtain a first similarity of the topological relationship, a second similarity of the appearance features and a third similarity of the motion offset; using the first similarity, the second similarity and the third similarity to obtain the similarity of the target joint features, and obtaining the optical satellite video multi-target tracking results based on the target relationship interaction according to the similarity of the target joint features.
[0075] In some embodiments, Figure 2 As shown, after obtaining the target detection result of the current frame, the joint feature matching module in the embodiment of the present application knows all the target positions of the previous frame satellite video image and the current frame satellite video image, and can describe the topological relationship between each target in the frame and the adjacent targets respectively. For the jth target V t,j , and the topological relationship of the M targets closest to it can be described as follows:
[0076]
[0077] Among them, T t,j,m The target V t,j With adjacent target V t,m The relative position relationship of x j ,y j 、w j and h j The target V t,j The horizontal coordinate, vertical coordinate, width and height of x m ,y m 、w m and h m are the neighboring targets V t,m The horizontal coordinate, vertical coordinate, width and height of ; w0 and h0 are the width and height of the tth frame image respectively.
[0078] The target V can be obtained from the above formula (9): t,j The topological relationship description T t,j , i.e., a vector with a dimension of M×8. Similarly, the embodiment of the present application can also obtain the target V t-1,iThe topological relationship description T t-1,i .
[0079] Secondly, the embodiment of the present application can use the cosine distance to evaluate the target V t,j With target V t-1,i The topological relationship similarity Sim(T t,j , T t-1,i ),Right now:
[0080]
[0081] Among them, Sim(T t,j , T t-1,i ) is the topological similarity; when Sim(T t,j , T t-1,i ) approaches 1, indicating that the target V t,j With target V t-1,i The more similar the neighbor relationship structures are, the higher the probability that they belong to the same target, and vice versa.
[0082] Again, the embodiment of the present application can obtain the DLA-34 appearance feature F of the corresponding position after obtaining the target detection results of two adjacent frames. t-1 and F t , and the corresponding position motion offset Therefore, the similarity of target features between frames can be evaluated from three dimensions: appearance features, motion offset, and topological relationship, to improve the accuracy of association between target frames.
[0083] Among them, the cosine distance can be used to evaluate the similarity of appearance features and the similarity of motion offset. Use the envelope prediction value of the tth frame and the truth value B t,j The intersection over union (IoU) is used to evaluate the similarity of the topological relationship, and the similarity of the topological relationship is obtained using the Sim(T t,j , T t-1,i ). Similarity of target joint features Sim(V t,j , V t-1,i ) is calculated by formula (11), namely:
[0084]
[0085] Among them, Sim(F t,j , F t-1,i ) is the similarity of appearance features, is the similarity of motion offset, Sim(T t,j , T t-1,i ) is the target V t,j With target V t-1,i The topological similarity of is the predicted value of the envelope of the tth frame, B t,j is the true value, IoU is the intersection over union ratio; when Sim(V t,j , V t-1,i ) approaches 1, indicating that the target V t,j With target V t-1,i The more similar the individual characteristics are, the higher the probability that they belong to the same target, and vice versa.
[0086] It can be seen from this that the embodiments of the present application can measure the similarity of the topological subgraphs of each target in adjacent frames through a joint feature matching module, and construct a joint feature description of the target individual of appearance, motion and topology, so as to achieve robust tracking of multiple targets in scenarios with dense target distribution.
[0087] According to the optical satellite video multi-target tracking method based on target relationship interaction proposed in the embodiment of the present application, the target detection results obtained by performing target detection on the current frame satellite video image and the previous frame satellite video image by image segmentation to obtain multiple target slice images that meet certain size conditions and the current frame satellite video image can be input into the pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction, effectively reducing the computational complexity of multi-target tracking and improving the accuracy of high-resolution optical satellite video multi-target tracking. Thus, the problem that the multi-target tracking method in the related art is prone to error compensation and increases computational complexity, resulting in limited improvement in target detection performance, especially for satellite video targets with scarce texture features, individual differences cannot be effectively distinguished, and the accuracy of satellite video multi-target tracking is reduced is solved.
[0088] Next, an optical satellite video multi-target tracking device based on target relationship interaction proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.
[0089] Figure 3 It is a block diagram of an optical satellite video multi-target tracking device based on target relationship interaction according to an embodiment of the present application.
[0090] like Figure 3 As shown, the optical satellite video multi-target tracking device 10 based on target relationship interaction includes: an acquisition module 100 , a determination module 200 and a tracking module 300 .
[0091] Specifically, the acquisition module 100 is used to acquire the current frame satellite video image and the previous frame satellite video image of the target satellite video, and perform target detection on the current frame satellite video image to determine the target detection result of the current frame satellite video image.
[0092] The determination module 200 is used to perform image segmentation processing on the current frame satellite video image and the previous frame satellite video image to obtain a plurality of target slice images that meet preset size conditions.
[0093] The tracking module 300 is used to input multiple target slice images and target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction.
[0094] Optionally, in one embodiment of the present application, the tracking module 300 includes: a first acquiring unit, a first determining unit and a constructing unit.
[0095] The first acquisition unit is used to acquire the target node set according to the feature vector set of the target center position of the previous frame of satellite video image and the neighborhood feature vector set corresponding to the current frame of satellite video image.
[0096] The first determining unit is used to make the target node sets adjacent to each other and the target node sets non-adjacent to each other, so as to obtain an edge set.
[0097] A construction unit is used to construct an undirected bipartite graph that meets preset conditions using an edge set.
[0098] Optionally, in one embodiment of the present application, the tracking module 300 includes: a second determination unit, a third determination unit, a fourth determination unit, a fifth determination unit and a processing unit.
[0099] The second determination unit is used to determine the template node according to the feature vector set, and determine the search node using the neighborhood feature vector set.
[0100] The third determination unit is used to determine the target node to be enhanced by using the attention score of the template node and the attention score of the search node.
[0101] The fourth determining unit is used to guide the target node to be enhanced to perform feature enhancement to obtain the enhanced target node.
[0102] The fifth determination unit is used to normalize the attention value of each edge based on the enhanced target node to obtain a normalized processing result.
[0103] The processing unit is used to guide the node features of the previous moment to aggregate to the node features of the current moment using the normalization processing result, and update the aggregated result to obtain the features of the enhanced target area of the current frame satellite video image.
[0104] Optionally, in one embodiment of the present application, the tracking module 300 includes: a calculation unit and a sixth determination unit.
[0105] Among them, the calculation unit is used to calculate the spatiotemporal contrast loss of individual granularity and the spatiotemporal contrast loss of category granularity based on the characteristics of the enhanced target area, the target true value target characteristics of the current frame satellite video image and the previous frame satellite video image.
[0106] The sixth determination unit is used to determine the multi-granularity spatiotemporal contrast loss by using the spatiotemporal contrast loss of the individual granularity and the spatiotemporal contrast loss of the category granularity.
[0107] Optionally, in one embodiment of the present application, the tracking module 300 includes: a second acquiring unit, a third acquiring unit, a seventh determining unit and a fourth acquiring unit.
[0108] Among them, the second acquisition unit is used to obtain the topological relationship between each target and the adjacent targets in the previous frame and the current frame based on the multi-granularity spatiotemporal contrast loss, the previous frame of satellite video image and all target positions of the current frame of satellite video image, so as to obtain the target detection results of two adjacent frames.
[0109] The third acquisition unit is used to acquire the appearance features and motion offset of the previous satellite video image and the current satellite video image by using the target detection results of two adjacent frames.
[0110] The seventh determination unit is used to evaluate the similarity of the target features between frames based on the topological relationship, the appearance feature and the motion offset to obtain a first similarity of the topological relationship, a second similarity of the appearance feature and a third similarity of the motion offset.
[0111] The fourth acquisition unit is used to obtain the similarity of the target joint features by using the first similarity, the second similarity and the third similarity, and obtain the optical satellite video multi-target tracking result based on the target relationship interaction according to the similarity of the target joint features.
[0112] Optionally, in one embodiment of the present application, the formula for the similarity of the target joint features is:
[0113]
[0114] Among them, Sim(F t,j , F t-1,i ) is the similarity of appearance features, is the similarity of motion offset, is the predicted value of the envelope of the tth frame, B t,j is the true value, IoU is the intersection over union ratio, Sim(T t,j , T t-1,i ) is the target V t,j With target V t-1,i The topological similarity of .
[0115] It should be noted that the aforementioned explanation of the embodiment of the optical satellite video multi-target tracking method based on target relationship interaction is also applicable to the optical satellite video multi-target tracking device based on target relationship interaction of this embodiment, and will not be repeated here.
[0116] According to the optical satellite video multi-target tracking device based on target relationship interaction proposed in the embodiment of the present application, the target detection result obtained by performing target detection on the current frame satellite video image and the previous frame satellite video image by image segmentation to obtain multiple target slice images that meet certain size conditions and the current frame satellite video image can be input into the pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking result based on target relationship interaction, effectively reducing the computational complexity of multi-target tracking and improving the accuracy of satellite video multi-target tracking. Thus, the problem that the multi-target tracking method in the related art is prone to cause error compensation and increase computational complexity, resulting in limited improvement in target detection performance, especially for satellite video targets with scarce texture features, individual differences cannot be effectively distinguished, and the accuracy of satellite video multi-target tracking is reduced is solved.
[0117] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0118] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .
[0119] When the processor 402 executes the program, the optical satellite video multi-target tracking method based on target relationship interaction provided in the above embodiment is implemented.
[0120] Furthermore, the electronic device further comprises:
[0121] The communication interface 403 is used for communication between the memory 401 and the processor 402 .
[0122] The memory 401 is used to store computer programs that can be executed on the processor 402 .
[0123] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0124] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0125] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.
[0126] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0127] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the optical satellite video multi-target tracking method based on target relationship interaction as described above is implemented.
[0128] This embodiment also provides a computer program product, including a computer program. When the computer program is executed, it is used to implement the above optical satellite video multi-target tracking method based on target relationship interaction.
[0129] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0130] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0131] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0133] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0134] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0135] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0136] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. An optical satellite video multi-target tracking method based on target relationship interaction, characterized in that: The following steps are involved: Acquire a current frame satellite video image and a previous frame satellite video image of the target satellite video, and perform target detection on the current frame satellite video image to determine a target detection result of the current frame satellite video image; Performing image segmentation processing on the current frame satellite video image and the previous frame satellite video image to obtain a plurality of target slice images that meet a preset size condition; The multiple target slice images and the target detection results are input into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction.
2. The method according to claim 1, characterized in that The step of inputting the plurality of target slice images and the target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction includes: Acquire a target node set according to a feature vector set of a target center position of the previous frame of satellite video image and a neighborhood feature vector set corresponding to the current frame of satellite video image; The target node sets are mutually adjacent, and the target node sets are mutually non-adjacent, so as to obtain an edge set; An undirected bipartite graph satisfying preset conditions is constructed using the edge set.
3. The method according to claim 2, characterized in that The step of inputting the plurality of target slice images and the target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction includes: Determine a template node according to the feature vector set, and determine a search node using the neighborhood feature vector set; Determine a target node to be enhanced using the attention score of the template node and the attention score of the search node; Guiding the target node to be enhanced to perform feature enhancement to obtain an enhanced target node; Based on the enhanced target node, normalizing the attention value of each edge to obtain a normalized processing result; The normalization processing result is used to guide the node features of the previous moment to be aggregated to the node features of the current moment, and the aggregated result is updated to obtain the features of the enhanced target area of the current frame satellite video image.
4. The method according to claim 3, characterized in that: The step of inputting the plurality of target slice images and the target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction includes: Based on the features of the enhanced target area, the target true value target features of the current frame satellite video image and the previous frame satellite video image, respectively calculating the spatiotemporal contrast loss of the individual granularity and the spatiotemporal contrast loss of the category granularity; The multi-granularity spatiotemporal contrast loss is determined by using the spatiotemporal contrast loss of the individual granularity and the spatiotemporal contrast loss of the category granularity.
5. The method according to claim 4, characterized in that The step of inputting the plurality of target slice images and the target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction includes: Based on the multi-granularity spatiotemporal contrast loss, all target positions of the previous frame of satellite video image and the current frame of satellite video image, the topological relationship between each target and the adjacent targets in the previous frame and the current frame is respectively obtained to obtain target detection results of two adjacent frames; Using the target detection results of the two adjacent frames, the appearance features and motion offsets of the previous satellite video image and the current satellite video image are obtained; Based on the topological relationship, the appearance feature and the motion offset, the similarity of the target features between frames is evaluated to obtain a first similarity of the topological relationship, a second similarity of the appearance feature and a third similarity of the motion offset; The first similarity, the second similarity and the third similarity are used to obtain the similarity of target joint features, and the optical satellite video multi-target tracking result based on target relationship interaction is obtained according to the similarity of the target joint features.
6. The method according to claim 5, characterized in that The formula for the similarity of the target joint features is: Among them, Sim(F t,j ,F t-1,i ) is the similarity of appearance features, is the similarity of motion offset, is the predicted value of the envelope of the tth frame, B t,j is the true value, IoU is the intersection over union ratio, Sim(T t,j ,T t-1,i ) is the target V t,j With target V t-1,i The topological similarity of .
7. An optical satellite video multi-target tracking device based on target relationship interaction, characterized in that: include: An acquisition module is used to acquire a current frame satellite video image and a previous frame satellite video image of a target satellite video, and perform target detection on the current frame satellite video image to determine a target detection result of the current frame satellite video image; A determination module, configured to perform image segmentation processing on the current frame satellite video image and the previous frame satellite video image to obtain a plurality of target slice images satisfying a preset size condition; The tracking module is used to input the multiple target slice images and the target detection results into a pre-trained optical satellite video multi-target tracking model based on target relationship interaction, so as to track and obtain the optical satellite video multi-target tracking results based on target relationship interaction.
8. The device according to claim 7, characterized in that The tracking module includes: A first acquisition unit is used to acquire a target node set according to a feature vector set of a target center position of the previous frame of satellite video image and a neighborhood feature vector set corresponding to the current frame of satellite video image; A first determining unit, configured to connect the target node sets to each other and disconnect the target node sets from each other, so as to obtain an edge set; A construction unit is used to construct an undirected bipartite graph that meets preset conditions using the edge set.
9. The device according to claim 8, characterized in that The tracking module includes: A second determining unit, configured to determine a template node according to the feature vector set, and determine a search node using the neighborhood feature vector set; A third determining unit, configured to determine a target node to be enhanced by using the attention score of the template node and the attention score of the search node; A fourth determining unit is used to guide the target node to be enhanced to perform feature enhancement to obtain an enhanced target node; A fifth determining unit, configured to normalize the attention value of each edge based on the enhanced target node to obtain a normalized processing result; The processing unit is used to use the normalization processing result to guide the node features of the previous moment to aggregate to the node features of the current moment, and update the aggregated results to obtain the features of the enhanced target area of the current frame satellite video image.
10. The device according to claim 9, characterized in that The tracking module includes: A calculation unit, used to calculate the temporal and spatial contrast loss of the individual granularity and the temporal and spatial contrast loss of the category granularity based on the characteristics of the enhanced target area, the target true value target characteristics of the current frame satellite video image and the previous frame satellite video image; The sixth determining unit is used to determine the multi-granularity spatiotemporal contrast loss by using the spatiotemporal contrast loss of the individual granularity and the spatiotemporal contrast loss of the category granularity.
11. The device according to claim 10, characterized in that The tracking module includes: A second acquisition unit is used to acquire the topological relationship between each target and the adjacent targets in the previous frame and the current frame respectively based on the multi-granularity spatiotemporal contrast loss, the previous frame satellite video image and all target positions of the current frame satellite video image, so as to obtain target detection results of two adjacent frames; A third acquisition unit is used to acquire the appearance features and motion offset of the previous frame of satellite video image and the current frame of satellite video image by using the target detection results of the two adjacent frames; a seventh determination unit, configured to evaluate the similarity of target features between frames based on the topological relationship, the appearance feature and the motion offset, so as to obtain a first similarity of the topological relationship, a second similarity of the appearance feature and a third similarity of the motion offset; The fourth acquisition unit is used to obtain the similarity of the target joint features by using the first similarity, the second similarity and the third similarity, and obtain the optical satellite video multi-target tracking result based on the target relationship interaction according to the similarity of the target joint features.
12. The device according to claim 11, characterized in that The formula for the similarity of the target joint features is: Among them, Sim(F t,j ,F t-1,i ) is the similarity of appearance features, is the similarity of motion offset, is the predicted value of the envelope of the tth frame, B t,j is the true value, IoU is the intersection over union ratio, Sim(T t,j ,T t-1,i ) is the target V t,j With target V t-1,i The topological similarity of .
13. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the optical satellite video multi-target tracking method based on target relationship interaction as described in any one of claims 1 to 6.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the optical satellite video multi-target tracking method based on target relationship interaction as described in any one of claims 1 to 6.
15. A computer program product comprising a computer program, characterized in that The computer program is executed by a processor to implement the optical satellite video multi-target tracking method based on target relationship interaction as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Satellite video target tracking method and network training method and system
CN116109676A
Multi-target tracking method and system based on satellite video data association and storage medium
CN116704368A
Multi-target tracking method, system and device and readable storage medium
CN118261942A
Cited By
Satellite video multi-target tracking method based on optical flow and state space model
CN121236121A
