Large-view-field camera space target association method based on space-time high-order attribute hypergraph

Abstract

The invention discloses a large-field-of-view camera space target association method based on a space-time high-order attribute hypergraph, and the method comprises the steps: obtaining a large-field-of-view image sequence through a large-field-of-view camera, and extracting a static attribute set corresponding to each frame of large-field-of-view image; constructing attribute hypergraphs based on the first three frames of large-field-of-view images, determining a potential target association matching matrix corresponding to each attribute hypergraph through a space-time attribute association module, and determining a high-order space-time attribute of a third frame of large-field-of-view image; for each frame of large-field-of-view image after the third frame, constructing a reasoning hypergraph based on the high-order space-time attribute of the previous frame of large-field-of-view image, and determining the high-order space-time attribute of the current frame of large-field-of-view image; taking the reasoning hypergraph as input, utilizing a space-time reasoning model to realize prediction of a potential target association matching matrix of the adjacent frames of large-field-of-view images, and utilizing the potential target association matching matrix to realize continuous association and classification of targets; the method can better adapt to feature extraction and association requirements of multiple target groups.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for associating targets in a large-field-of-view camera space based on a spatiotemporal high-order attribute hypergraph. Background Art

[0002] Large field of view cameras have a wider observation range and stronger detection capabilities. They can often observe space targets in different orbits at the same time, thus presenting different positions and morphological changes on the image plane. When using large field of view cameras to observe near-Earth objects, affected by factors such as the position and posture of the observation platform and the observed object, different targets often show different types and degrees of position and morphological changes in the imaging data. For example, large field of view cameras located in LEO (Low Earth Orbit) can often capture space objects distributed from LEO to HEO (High Earth Orbit). HEO objects often show slow linear position movement and morphological changes in the observation data, while LEO objects closer to the observation platform show strong nonlinearity and time-varying properties. Traditional single-type processing strategies are often unable to process these targets with different states at the same time, resulting in poor correlation performance under large field of view conditions.

[0003] Reference 1 introduces temporal dynamic information based on traditional spatial static graphs, and proposes a spatiotemporal graph neural network by combining graph shift operators with time shift operators. This method can simultaneously process the topological structure of time and space, and has good generalization ability in processing diverse spatiotemporal dynamic changes. However, this method requires that the state of the same node in the spatiotemporal graph is known at different times and the time shift operator only operates on the same node, which cannot adapt to the application requirements of multi-type target association; Reference 1: "Hadou S, Kanatsoulis CI, Ribeiro A. Space-time graph neuralnetworks[J]. arXiv preprint arXiv:2110.02880, 2021.". Summary of the invention

[0004] The purpose of the present invention is to provide a large-field-of-view camera space target association method based on spatiotemporal high-order attribute hypergraph to better adapt to the feature extraction and association requirements of multiple target groups.

[0005] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0006] A large-field-of-view camera space target association method based on spatiotemporal high-order attribute hypergraph includes:

[0007] A large-field-of-view image sequence is acquired by using a large-field-of-view camera, and for each frame of the large-field-of-view image in the large-field-of-view image sequence, a static attribute extraction unit is used to extract image plane static attributes of each target, thereby obtaining a static attribute set corresponding to each frame of the large-field-of-view image;

[0008] Combine the first three frames of large field of view images in pairs in order, and construct an attribute hypergraph for each combination based on a static attribute set; use the attribute hypergraphs corresponding to the first three frames of large field of view images to determine the potential target association matching matrix corresponding to each attribute hypergraph through a spatiotemporal attribute association module; determine the high-order spatiotemporal attributes of the third frame of large field of view image based on the static attribute set and potential target association matching matrix of the first three frames of large field of view images;

[0009] For each large field of view image after the third frame, an inference hypergraph is constructed based on the high-order spatiotemporal attributes of the previous large field of view image of the current frame, and the high-order spatiotemporal attributes of the current frame are determined. With the inference hypergraph as input, the spatiotemporal inference model is used to predict the potential target association matching matrix of adjacent large field of view images, and the potential target association matching matrix is ​​used to achieve continuous association and classification of targets.

[0010] Furthermore, the static attributes of the image plane of each target include the target position, morphological features and background information; wherein the morphological features include the maximum grayscale of the target area, the standard deviation matrix and covariance matrix of the target offset Gaussian model; the background information includes the background grayscale mean and standard deviation of the target neighborhood.

[0011] Furthermore, the first and second layers of the static attribute extraction unit are convolutional layers, the third to tenth layers are C3 modules, and the last two layers are convolutional layers; the feature maps output by the fifth, seventh, ninth, and twelfth layers constitute a feature pyramid, which is regressed by a regression head, and the image plane static attributes are output by the regression head, thereby forming a static attribute set corresponding to each frame of the large field of view image; an activation function is set after each layer of the static attribute extraction unit;

[0012] The C3 module includes three convolutional layers connected in sequence. The output features of the first convolutional layer are superimposed with the output features of the third convolutional layer, and then fused with the output features of the first convolutional layer after passing through another convolutional layer on the branch. The fusion result is processed by another convolutional layer to obtain the feature map output by the C3 module.

[0013] Furthermore, the regression head adopts a decoupled design, including a shared convolutional layer and parallel position regression branches, morphological regression branches and confidence branches, the position regression branch, the morphological regression branch and the confidence branch all adopt a Conv(SiLU(BN(Conv(·)))) structure; the position regression branch is used to obtain the target position, the morphological regression branch is used to obtain the morphological features and background information of the target, and the confidence branch is used to predict the confidence of the target existence, and then determine whether it is a real target; after the feature map output by the feature pyramid passes through the shared convolutional layer, the position regression branch, the morphological regression branch and the confidence branch cooperate with each other to finally obtain the static properties of the target image.

[0014] Furthermore, the first three frames of large field of view images are combined in pairs in order, and an attribute hypergraph of each combination is constructed based on a static attribute set, including:

[0015] Construct the attribute hypergraph corresponding to the first frame of large field of view image and the second frame of large field of view image And the attribute hypergraph corresponding to the second frame large field of view image and the third frame large field of view image ; The attribute hypergraph is represented by the following formula:

[0016] ;

[0017] In the above formula, represents an attribute hypergraph constructed with the targets of the i-th frame large field of view image and the i+1-th frame large field of view image as nodes; are the static attribute sets corresponding to the i-th frame large field of view image and the i+1-th frame large field of view image respectively; are the sets of edges connecting different nodes in the i-th frame large field of view image and the i+1-th frame large field of view image, respectively. It represents the edge set connected between the nodes of the i-th frame large field of view image and the i+1-th frame large field of view image; define the threshold and , for two nodes belonging to the same frame of large field of view image, such as the Euclidean distance between the two nodes Less than , then an edge is established between the two nodes; for two nodes belonging to different frames of large field of view images, such as the Euclidean distance between the two nodes Less than , an edge is established between two nodes.

[0018] Furthermore, a potential target association matching matrix corresponding to each attribute hypergraph is determined by a spatiotemporal attribute association module, including:

[0019] The input of the spatiotemporal attribute association module is the attribute hypergraph. The spatiotemporal attribute association module includes a six-layer network structure, in which the first, third and fifth layers use spatial hypergraph convolution, and the second, fourth and sixth layers use temporal hypergraph convolution. An activation function is set after each layer of the network structure, and an association head is set after the sixth layer. The association head uses a dot product attention layer, which can be expressed as:

[0020] ;

[0021] in, is the potential target association matching matrix corresponding to the attribute hypergraph constructed using the i-th frame large field of view image and the i+1-th frame large field of view image, which contains the association relationship between each target in the two frames of large field of view images; and They represent the feature matrices of the targets in the i-th frame large field of view image and the i+1-th frame large field of view image, respectively. is the normalization factor, and the superscript T indicates transposition;

[0022] The structures of spatial hypergraph convolution and temporal hypergraph convolution are as follows:

[0023] ;

[0024] Where X represents the input of spatial hypergraph convolution or temporal hypergraph convolution, Y is the processed output, A is the adjacency matrix, which stores the edge connection and weight information, and W is the trainable parameter. is the activation function.

[0025] Furthermore, based on the static attribute set of the first three frames of large field of view images and the potential target association matching matrix, the high-order spatiotemporal attributes of the third frame of large field of view image are determined, including:

[0026] For each target in the third frame of the large field of view image, difference and second-order difference operations are performed based on the static image properties of the target at different times to obtain the first-order differential and second-order differential of the target; finally, the static image properties, first-order differential and second-order differential of the target are stacked to obtain its high-order spatiotemporal properties. The high-order spatiotemporal properties of all targets constitute the high-order spatiotemporal properties of the third frame of the large field of view image.

[0027] Furthermore, a reasoning hypergraph is constructed based on the high-order spatiotemporal attributes of the previous large field of view image of the current frame, and the high-order spatiotemporal attributes of the current large field of view image are determined, including:

[0028] The structure of the inference hypergraph is the same as that of the attribute hypergraph. The difference between the inference hypergraph and the attribute hypergraph in the construction method is the edge set connecting the nodes of the i-th frame large field of view image and the i+1-th frame large field of view image. The construction process is different; in the reasoning hypergraph middle, The construction method is as follows:

[0029] First, for the current frame large field of view image with i > 2, when constructing the inference hypergraph When the jth node in the i-th frame of the large field of view image is connected to any node in the i+1-th frame of the large field of view image, The nodes are assumed to be associated with each other, that is, they are assumed to be the same potential target. Secondly, the high-order spatiotemporal attributes of the jth node in the i-th frame of the large field of view image are calculated. as its assumed higher-order spatiotemporal properties; finally, define the hyperparameters , determine the assumed high-order spatiotemporal properties of the jth node in the i-th frame of the large field of view image , and the i+1th frame of the large field image Is the Euclidean distance between the assumed higher-order spatiotemporal attributes of the nodes less than the hyperparameter If it is less than, the assumed association relationship is taken as the association relationship between the jth node in the i-th frame large field of view image and the i+1-th frame large field of view image, and the jth node and the nodes are connected by an edge; higher-order spatiotemporal properties will be assumed As the high-order spatiotemporal attributes of the j-th node, the high-order spatiotemporal attributes of the i-th frame large field of view image are obtained.

[0030] Furthermore, taking the reasoning hypergraph as input, the spatiotemporal reasoning model is used to predict the potential target association matching matrix of adjacent frame large field of view images, including:

[0031] The spatiotemporal reasoning model is a graph convolutional neural network. The first, fourth and seventh layers of the spatiotemporal reasoning model network structure are spatial hypergraph convolutions, the second, fifth and eighth layers are temporal hypergraph convolutions, the third, sixth and ninth layers are temporal attention layers, an activation function is set after each layer of the network structure, and an association head and a classification head are set after the ninth layer; wherein the association head adopts a dot product attention layer, and the classification head adopts a softmax function to implement;

[0032] The reasoning hypergraph is input into the spatiotemporal reasoning model, and finally the potential target association matching matrix and the target category are obtained through the combination of the association head and the classification head, thereby realizing the continuous association and classification of the target; the temporal attention layer adopts a graph convolutional long short-term memory network.

[0033] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for associating targets in a large-field-of-view camera space based on a spatiotemporal high-order attribute hypergraph is implemented.

[0034] A computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the method for associating targets in a large-field-of-view camera space based on a spatiotemporal high-order attribute hypergraph is implemented.

[0035] Compared with the prior art, the present invention has the following technical features:

[0036] The present invention aims to solve the problem that the motion patterns and image morphology generated by the different types of targets observed by wide-field-of-view cameras are diverse, which makes it difficult for traditional processing methods to effectively and comprehensively deal with the problem that traditional spatiotemporal graph networks cannot match and associate unknown target groups. The present invention proposes a target association strategy that fully explores deep spatiotemporal semantics, provides a new idea for tracking and associating large numbers of multi-type targets, and proposes a more comprehensive and sufficient high-order attribute representation. At the same time, by constructing a spatiotemporal neighborhood hypergraph structure and fully mining spatiotemporal information through spatial and temporal hypergraph convolution and attention mechanisms, more effective large-field-of-view detection data target association tracking is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the principle of the method of the present invention;

[0038] Figure 2 It is a network structure diagram of the static attribute extraction unit in the present invention;

[0039] Figure 3 The figure is a comparison chart of the experimental effect of an embodiment of the present invention and other algorithms. DETAILED DESCRIPTION

[0040] The present invention provides a method for associating targets in a large-field-of-view camera space based on a spatiotemporal high-order attribute hypergraph. Given continuous detection images as input, a high-order attribute representation of the target is constructed for potential moving targets (light spots) to embed in hypergraph nodes, and a spatiotemporal hypergraph is constructed using spatiotemporal relationships. High-order spatiotemporal semantic information is extracted through a spatiotemporal hypergraph neural network, and nested reasoning of multi-dimensional attributes is performed to associate targets of different types and states. The present invention achieves more precise feature guidance by constructing a high-order attribute representation, and at the same time introduces a spatiotemporal hypergraph structure. On the basis of giving full play to the dynamic information processing capabilities of the spatiotemporal graph neural network, it is more adaptable to the feature extraction and association requirements of multiple target groups at different locations. The specific implementation steps of the present invention are as follows:

[0041] Step 1: Use a large field of view camera to obtain a large field of view image sequence, and use a static attribute extraction unit to extract the image surface static attributes of each target for each frame of the large field of view image in the large field of view image sequence, so as to obtain a static attribute set corresponding to each frame of the large field of view image.

[0042] Given a sequence of k continuous large field of view images ,in Represents the i-th frame of the large field of view image; constructs a static attribute extraction unit , using the static attribute extraction unit to Perform feature extraction to obtain the corresponding static attribute set .

[0043] Wherein, the static attribute extraction unit The first and second layers are convolutional layers, the third to tenth layers are C3 modules, and the last two layers are convolutional layers; the feature maps output by the fifth, seventh, ninth, and twelfth layers constitute the feature pyramid FPN, which is regressed by the regression head and uses the regression head to output the static attributes of the image surface , thus forming a static attribute set corresponding to the i-th frame large field image ; is the static property of the image plane of the jth target in the i-th frame of the large field of view image, and l is the number of targets. Static property extraction unit The activation function SiLU is set after each layer.

[0044] ;

[0045] in, Represents the static properties of the image plane The target position in Represents morphological characteristics, Represents background information, and Represent the centroid coordinates of the jth target, is the maximum grayscale of the target area, and Represent the standard deviation matrix and covariance matrix of the target offset Gaussian model, and Represents the target k n The background grayscale mean and standard deviation of the pixel neighborhood, k n is a hyperparameter. In this solution, the superscript T represents the transposition operation, and the same applies below. In actual use, All attributes of are normalized to the range of (0, 1).

[0046] Large field of view images First, it is cut to a predetermined size (640, 640), and then passes through the static attribute extraction unit The first and second layers are downsampled to (320, 320) size and then enter the C3 module for subsequent processing.

[0047] See attached Figure 2The regression head in this scheme adopts a decoupled design, including a shared convolutional layer and parallel position regression branches, morphological regression branches and confidence branches. The position regression branch, morphological regression branch and confidence branch all adopt the Conv(SiLU(BN(Conv(·)))) structure; the position regression branch is used to obtain the target position, that is, In ; The morphological regression branch is used to obtain the morphological features and background information of the target, that is, In and ; The confidence branch is used to predict the confidence of the target existence, and then determine whether it is a real target; the feature map output by the feature pyramid FPN passes through the shared convolution layer, and is processed by the position regression branch, the morphology regression branch and the confidence branch to finally obtain the static properties of the target image surface ; In the Conv(SiLU(BN(Conv(·)))) structure, Conv represents convolution, BN represents batch normalization, and SiLU represents the Swish activation function.

[0048] The C3 module includes three convolutional layers connected in sequence. The output features of the first convolutional layer are superimposed with the output features of the third convolutional layer, and then fused with the output features of the first convolutional layer after passing through another convolutional layer on the branch. The fusion result is processed by another convolutional layer to obtain the feature map output by the C3 module.

[0049] In one embodiment of the present invention, the network parameters of the first to tenth layers of the static attribute extraction unit are shown in the following table:

[0050] Table 1: Network structure of the static attribute extraction unit.

[0051]

[0052] Step 2, sequentially combine the first three frames of large field of view images in pairs to construct an attribute hypergraph for each combination; using the attribute hypergraph corresponding to the first three frames of large field of view images, determine the potential target association matching matrix corresponding to each attribute hypergraph through the spatiotemporal attribute association module; based on the static attribute set and potential target association matching matrix of the first three frames of large field of view images, determine the high-order spatiotemporal attributes of the third frame of large field of view image.

[0053] Step 2.1, attribute hypergraph of the initial static attribute set.

[0054] Use the static attribute set corresponding to the first three frames of large field of view images , and As the initial static attribute set, they are combined in pairs in chronological order to construct the attribute hypergraph corresponding to the first frame of large field of view image and the second frame of large field of view image. And the attribute hypergraph corresponding to the second frame large field of view image and the third frame large field of view image ; The attribute hypergraph is represented by the following formula:

[0055] ;

[0056] In the above formula, represents an attribute hypergraph constructed with the targets of the i-th frame large field of view image and the i+1-th frame large field of view image as nodes; represents the set of edges connecting different nodes in the i-th frame of the large field of view image, It represents the edge set connected between the nodes of the i-th frame large field of view image and the i+1-th frame large field of view image; define the threshold and , for two nodes belonging to the same frame of large field of view image, such as the Euclidean distance between the two nodes Less than , then an edge is established between the two nodes with a weight of , e is a natural constant; for two nodes belonging to different frames of large field of view images, such as the Euclidean distance between the two nodes Less than , then an edge is established between the two nodes, and the weight is .

[0057] Step 2.2, potential target association matching matrix.

[0058] Attributed Hypergraph and After the construction is completed, the spatiotemporal attribute association module F M Process it to obtain the potential target association matching matrix in the initial stage and , thereby obtaining the state change information of the same target between different frames.

[0059] The potential target association matching matrix can be obtained as follows:

[0060] ;

[0061] in is the potential target association matching matrix corresponding to the attribute hypergraph constructed using the i-th frame large field of view image and the i+1-th frame large field of view image. Its row and column coordinates represent the index of the target in two consecutive frames, and its elements are the similarities under the corresponding indexes. For example, the element at the (m, n) position represents the similarity between the m-th target in the i-frame large field of view image and the n-th target in the i+1-frame large field of view image. The value with the highest similarity under the row where the m-th target is located is selected, and the value is used as the association relationship between the m-th target in the i-frame large field of view image and the i+1-frame large field of view image (that is, which target does the m-th target in the i+1-frame large field of view image correspond to).

[0062] Spatial-temporal attribute association module F M It includes a six-layer network structure, in which the first, third and fifth layers use spatial hypergraph convolution, and the second, fourth and sixth layers use temporal hypergraph convolution; an activation function ReLU is set after each layer of the network structure, and a correlation head is set after the sixth layer; the correlation head uses a dot product attention layer, which is expressed as follows:

[0063] ;

[0064] and They represent the feature matrices of the targets in the i-th frame of large field of view image and the i+1-th frame of large field of view image, i.e., the output of the sixth layer of the network structure; softmax is used for normalization; is the normalization factor, and its value is the node feature dimension.

[0065] Spatial-temporal attribute association module F M In the above, spatial hypergraph convolution is used to extract the spatial structural features contained in the same moment, that is, the same frame, while temporal hypergraph convolution is used to integrate spatiotemporal features and extract deep temporal information; the structures of spatial hypergraph convolution and temporal hypergraph convolution are as follows:

[0066] ;

[0067] Where X represents the input of spatial hypergraph convolution or temporal hypergraph convolution, Y is the processed output, A is the adjacency matrix, which stores the edge connection and weight information, and W is the trainable parameter. is the activation function; the spatial hypergraph convolution and the temporal hypergraph convolution are essentially different only in W. The spatial hypergraph convolution only processes the edges in the same frame of large field of view image, while the temporal hypergraph convolution only processes the edges in different frames of large field of view images.

[0068] In one embodiment of the present invention, a spatiotemporal attribute association module F is provided. M The network structure is shown in Table 2:

[0069] Table 2: Network structure of the spatiotemporal attribute association module.

[0070]

[0071] Step 2.3, high-order spatiotemporal properties of the third frame of large field of view image.

[0072] Using static property collections , and Combined with its corresponding potential target association matching matrix and , determine the initial high-order space-time properties.

[0073] Based on the target association matching matrix and , we can obtain the association relationship of the jth target at different times (i.e. different frames), and according to the static attribute set , and , we can obtain the static properties of the image plane of the jth target at different times , and ; Then, for the jth target in the third frame of the large field of view image, the difference and second-order difference operations are performed based on the static properties of the image plane of the target at different times to obtain the first-order differential of the target With the second-order differential ; Finally, the static properties of the target image plane, the first-order differential and the second-order differential are stacked to obtain the target The high-order spatiotemporal properties of the third frame of large field of view image are :

[0074] ;

[0075] in Represents the high-order spatiotemporal properties of the j-th target obtained after the third frame of large field of view image input.

[0076] For example, for the initial static property collection , Static properties of the image plane , , taking the target position as an example, although the image plane static attributes of each target in each frame of the large field of view image are extracted in step 1, the image plane static attributes extracted for each frame of the large field of view image are independent, and it is not clear which target in the previous frame of the large field of view image corresponds to which target in the next frame of the large field of view image; and using The position of the jth target in the first frame of the large field of view image can be obtained by In the second frame of the large field of view image, , and thus the first-order differential of the target position is obtained by making a difference and differentiation between the two; the first-order differential of other morphological features and background information in the static properties of the image plane is also obtained in this way, thus forming the first-order differential of the static properties of the image plane of the jth target in the third frame of the large field of view image ; The principle of second-order differential is the same and will not be repeated here.

[0077] Step 3: For each large field of view image after the third frame, construct an inference hypergraph based on the high-order spatiotemporal attributes of the previous large field of view image of the current frame, and determine the high-order spatiotemporal attributes of the current frame; use the inference hypergraph as input, use the spatiotemporal inference model to predict the potential target association matching matrix of adjacent large field of view images, and use the potential target association matching matrix to achieve continuous association and classification of targets.

[0078] Step 3.1, construction of reasoning hypergraph.

[0079] For large field images after the third frame, the reasoning hypergraph is constructed starting from the fourth frame of large field image as the current frame; the structure of the reasoning hypergraph is basically the same as the attribute hypergraph of the initial static attribute set in step 2.1, which is expressed as , where i >2.

[0080] ;

[0081] Where: When i > 2, the static attribute extraction unit in step 1 can be used to obtain the static attribute set corresponding to the i-th frame of the large field of view image. , the set of edges connecting different nodes in the i-th frame large field of view image The determination method is the same as step 2.1, that is, to screen the i-th frame of the large field of view image whose Euclidean distance is less than the threshold Node pairs and establish edges.

[0082] The construction method of the attribute hypergraph of the inference hypergraph and the initial static attribute set is basically the same. The main difference lies in the edge set connecting the nodes of the i-th frame large field of view image and the i+1-th frame large field of view image. The construction process is different; in the reasoning hypergraph middle, The construction method is as follows:

[0083] First, when building the inference hypergraph When the jth node (i.e., target) in the i-th frame of the large field of view image is connected to any jth node (i.e., target) in the i+1-th frame of the large field of view image The nodes are assumed to be associated with each other, that is, they are assumed to be the same potential target. Secondly, according to the method in step 2.3, the high-order spatiotemporal attributes of the jth node in the i-th frame of the large field of view image are calculated. as its assumed higher-order spatiotemporal properties; finally, define the hyperparameters , determine the jth node in the i-th frame large field image and assume high-order spatiotemporal attributes , and the i+1th frame of the large field image Is the Euclidean distance between the assumed higher-order spatiotemporal attributes of the nodes (when i>3) or between the higher-order spatiotemporal attributes (when i=3) less than the hyperparameter If it is less than, the assumed association relationship is taken as the association relationship between the jth node in the i-th frame large field image and the i+1-th frame, and the jth node and the nodes are connected by an edge; higher-order spatiotemporal properties will be assumed As the high-order spatiotemporal attributes of the j-th node, the high-order spatiotemporal attributes of the i-th frame large field of view image are obtained.

[0084] Step 3.2, nested reasoning.

[0085] This step is to reason about the hypergraph As input, through the spatiotemporal reasoning model Realize the potential target correlation matching matrix between the i-th frame large field of view image and the i+1-th frame large field of view image prediction; by continuously generating , you can use Continuously associate and classify different targets.

[0086] ;

[0087] in Reasoning Hypergraph The predicted potential target association matching matrix between the i-th frame large field of view image and the i+1-th frame large field of view image.

[0088] By repeating step 3, the corresponding potential target association matching matrix can be continuously generated for each subsequent frame of large field of view image, thereby achieving continuous association of spatial targets.

[0089] The spatiotemporal reasoning model It is a graph convolutional neural network. The first, fourth and seventh layers of its network structure are spatial hypergraph convolutions, the second, fifth and eighth layers are temporal hypergraph convolutions, and the third, sixth and ninth layers are temporal attention layers. An activation function ReLU is set after each layer of the network structure, and an association head and a classification head are set after the ninth layer. Among them, the association head adopts the dot product attention layer mentioned in step 2.2, and the classification head is implemented by the softmax function. Finally, the potential target association matching matrix is ​​obtained by combining the association head and the classification head. and the category of the target (LEO, GEO, non-target, etc.), thereby achieving continuous association and classification of the target.

[0090] The spatial hypergraph convolution and temporal hypergraph convolution are described in step 2.2 above; the temporal attention layer uses a graph convolutional long short-term memory network, which is expressed as follows:

[0091] ;

[0092] in, and They represent the spatial structural features extracted by the spatial hypergraph convolution and the temporal hypergraph convolution before the temporal attention layer and the deep temporal information respectively. Represents the deep temporal information of the prediction; , They represent the forget gate and the input gate respectively.

[0093] In one embodiment of the present invention, the spatiotemporal reasoning model The network structure is shown in Table 3.

[0094] Table 3: Network structure of the spatiotemporal reasoning model.

[0095]

[0096] like Figure 3 As shown, in one embodiment of the present invention, for large field of view detection data, the F1 score of the present invention is improved by 0.04-0.17 and the MOTA is improved by 0.03-0.16 compared with other association strategies, which proves the effectiveness of the present invention.

[0097] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A large-field-of-view camera space target association method based on spatiotemporal high-order attribute hypergraph, characterized in that: include: A large-field-of-view image sequence is acquired by using a large-field-of-view camera, and for each frame of the large-field-of-view image in the large-field-of-view image sequence, a static attribute extraction unit is used to extract image plane static attributes of each target, thereby obtaining a static attribute set corresponding to each frame of the large-field-of-view image; Combine the first three frames of large field of view images in pairs in order, and construct an attribute hypergraph for each combination based on a static attribute set; use the attribute hypergraphs corresponding to the first three frames of large field of view images to determine the potential target association matching matrix corresponding to each attribute hypergraph through a spatiotemporal attribute association module; determine the high-order spatiotemporal attributes of the third frame of large field of view image based on the static attribute set and potential target association matching matrix of the first three frames of large field of view images; For each large field of view image after the third frame, an inference hypergraph is constructed based on the high-order spatiotemporal attributes of the previous large field of view image of the current frame, and the high-order spatiotemporal attributes of the current frame are determined. With the inference hypergraph as input, the spatiotemporal inference model is used to predict the potential target association matching matrix of adjacent large field of view images, and the potential target association matching matrix is ​​used to achieve continuous association and classification of targets.

2. The method for associating targets in large-field-of-view camera space based on spatiotemporal high-order attribute hypergraph according to claim 1, characterized in that: The static properties of each target’s image plane include target position, morphological features, and background information; The morphological features include the maximum grayscale of the target area, the standard deviation matrix and covariance matrix of the target offset Gaussian model; the background information includes the background grayscale mean and standard deviation of the target neighborhood.

3. The method for associating targets in a large field of view camera space based on a spatiotemporal high-order attribute hypergraph according to claim 1, characterized in that: The first and second layers of the static attribute extraction unit are convolutional layers, the third to tenth layers are C3 modules, and the last two layers are convolutional layers; the feature maps output by the fifth, seventh, ninth, and twelfth layers constitute a feature pyramid, which is regressed by a regression head, and the image plane static attributes are output by the regression head, thereby forming a static attribute set corresponding to each frame of a large field of view image; an activation function is set after each layer of the static attribute extraction unit; The C3 module includes three convolutional layers connected in sequence. The output features of the first convolutional layer are superimposed with the output features of the third convolutional layer, and then fused with the output features of the first convolutional layer after passing through another convolutional layer on the branch. The fusion result is processed by another convolutional layer to obtain the feature map output by the C3 module.

4. The method for associating targets in a large field of view camera space based on a spatiotemporal high-order attribute hypergraph according to claim 3, characterized in that: The regression head adopts a decoupled design, including a shared convolutional layer and parallel position regression branches, morphological regression branches and confidence branches, and the position regression branch, morphological regression branch and confidence branch all adopt a Conv(SiLU(BN(Conv(·)))) structure; The position regression branch is used to obtain the target position, the morphological regression branch is used to obtain the morphological features and background information of the target, and the confidence branch is used to predict the confidence of the target existence, and then determine whether it is a real target; After the feature map output by the feature pyramid passes through the shared convolution layer, it is processed by the position regression branch, the morphology regression branch and the confidence branch to finally obtain the static properties of the target image surface.

5. The method for associating targets in large-field-of-view camera space based on spatiotemporal high-order attribute hypergraph according to claim 1, characterized in that: Combine the first three frames of large field of view images in pairs in order, and construct the attribute hypergraph of each combination based on the static attribute set, including: Construct the attribute hypergraph corresponding to the first frame of large field of view image and the second frame of large field of view image And the attribute hypergraph corresponding to the second frame large field of view image and the third frame large field of view image ; The attribute hypergraph is represented by the following formula: ； In the above formula, represents an attribute hypergraph constructed with the targets of the i-th frame large field of view image and the i+1-th frame large field of view image as nodes; are the static attribute sets corresponding to the i-th frame large field of view image and the i+1-th frame large field of view image respectively; are the sets of edges connecting different nodes in the i-th frame large field of view image and the i+1-th frame large field of view image, respectively. It represents the edge set connected between the nodes of the i-th frame large field of view image and the i+1-th frame large field of view image; define the threshold and , for two nodes belonging to the same frame of large field of view image, such as the Euclidean distance between the two nodes Less than , then an edge is established between the two nodes; for two nodes belonging to different frames of large field of view images, such as the Euclidean distance between the two nodes Less than , an edge is established between two nodes.

6. The method for associating targets in large-field-of-view camera space based on spatiotemporal high-order attribute hypergraph according to claim 1, characterized in that: The potential target association matching matrix corresponding to each attribute hypergraph is determined by the spatiotemporal attribute association module, including: The input of the spatiotemporal attribute association module is the attribute hypergraph. The spatiotemporal attribute association module includes a six-layer network structure, in which the first, third and fifth layers use spatial hypergraph convolution, and the second, fourth and sixth layers use temporal hypergraph convolution. An activation function is set after each layer of the network structure, and an association head is set after the sixth layer. The association head uses a dot product attention layer, which can be expressed as: ； in, is the potential target association matching matrix corresponding to the attribute hypergraph constructed using the i-th frame large field of view image and the i+1-th frame large field of view image, which contains the association relationship between each target in the two frames of large field of view images; and They represent the feature matrices of the targets in the i-th frame large field of view image and the i+1-th frame large field of view image, respectively. is the normalization factor, and the superscript T indicates transposition; The structures of spatial hypergraph convolution and temporal hypergraph convolution are as follows: ； Where X represents the input of spatial hypergraph convolution or temporal hypergraph convolution, Y is the processed output, A is the adjacency matrix, which stores the edge connection and weight information, and W is the trainable parameter. is the activation function.

7. The method for associating targets in large-field-of-view camera space based on spatiotemporal high-order attribute hypergraph according to claim 1, characterized in that: Based on the static attribute set of the first three frames of large field of view images and the potential target association matching matrix, the high-order spatiotemporal attributes of the third frame of large field of view image are determined, including: For each target in the third frame of the large field of view image, difference and second-order difference operations are performed based on the static image properties of the target at different times to obtain the first-order differential and second-order differential of the target; finally, the static image properties, first-order differential and second-order differential of the target are stacked to obtain its high-order spatiotemporal properties. The high-order spatiotemporal properties of all targets constitute the high-order spatiotemporal properties of the third frame of the large field of view image.

8. The method for associating targets in a large field of view camera space based on a spatiotemporal high-order attribute hypergraph according to claim 5, characterized in that: The reasoning hypergraph is constructed based on the high-order spatiotemporal attributes of the previous large field of view image of the current frame, and the high-order spatiotemporal attributes of the current large field of view image are determined, including: The structure of the inference hypergraph is the same as that of the attribute hypergraph. The difference between the inference hypergraph and the attribute hypergraph in the construction method is the edge set connecting the nodes of the i-th frame large field of view image and the i+1-th frame large field of view image. The construction process is different; in the reasoning hypergraph middle, The construction method is as follows: First, for the current frame large field of view image with i > 2, when constructing the inference hypergraph When the jth node in the i-th frame of the large field of view image is connected to any node in the i+1-th frame of the large field of view image, The nodes are assumed to be associated with each other, that is, they are assumed to be the same potential target. Secondly, the high-order spatiotemporal attributes of the jth node in the i-th frame of the large field of view image are calculated. as its assumed higher-order spatiotemporal properties; finally, define the hyperparameters , determine the assumed high-order spatiotemporal properties of the jth node in the i-th frame of the large field of view image , and the i+1th frame of the large field image Is the Euclidean distance between the assumed higher-order spatiotemporal attributes of the nodes less than the hyperparameter If it is less than, the assumed association relationship is taken as the association relationship between the jth node in the i-th frame large field of view image and the i+1-th frame large field of view image, and the jth node and the nodes are connected by an edge; higher-order spatiotemporal properties will be assumed As the high-order spatiotemporal attributes of the j-th node, the high-order spatiotemporal attributes of the i-th frame large field of view image are obtained.

9. The method for associating targets in large-field-of-view camera space based on spatiotemporal high-order attribute hypergraph according to claim 1, characterized in that: Taking the inference hypergraph as input, the spatiotemporal inference model is used to predict the potential target association matching matrix of adjacent frame large field of view images, including: The spatiotemporal reasoning model is a graph convolutional neural network. The first, fourth and seventh layers of the spatiotemporal reasoning model network structure are spatial hypergraph convolutions, the second, fifth and eighth layers are temporal hypergraph convolutions, the third, sixth and ninth layers are temporal attention layers, an activation function is set after each layer of the network structure, and an association head and a classification head are set after the ninth layer; wherein the association head adopts a dot product attention layer, and the classification head adopts a softmax function to implement; The reasoning hypergraph is input into the spatiotemporal reasoning model, and finally the potential target association matching matrix and the target category are obtained through the combination of the association head and the classification head, thereby realizing the continuous association and classification of the target; the temporal attention layer adopts a graph convolutional long short-term memory network.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that: When the processor executes the computer program, the method for associating targets in a large-field-of-view camera space based on a spatiotemporal high-order attribute hypergraph according to any one of claims 1 to 9 is implemented.

Images