Space debris intelligent detection method and device based on spatiotemporal feature fusion

Abstract

The present application relates to a method and device for intelligent detection of space debris based on spatiotemporal feature fusion. The method comprises: obtaining a training sample consisting of multiple frames of images containing space debris target labels and / or star target labels; inputting the training sample into a multi-frame detection network; an information fusion module is used to fuse and extract multiple frames of images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features; and reversely trains the multi-frame detection network to perform space debris target detection on the multi-frame images to be detected based on the trained multi-frame detection network. The use of this method can solve the problem of insufficient information utilization in the current single-frame image target detection algorithm.

Description

Technical Field

[0001] The present application relates to the technical field of spatial intelligent target detection based on deep learning, and in particular to a method and device for intelligent detection of spatial debris based on spatiotemporal feature fusion. Background Art

[0002] For the detection of weak space debris against the background of complex starry sky images, this type of target detection method based on single-frame images still has some shortcomings. First, when the telescope exposure time is short, the imaging size of space debris in the detected image is dimmer and similar to the shape of stars. It is difficult for the network to identify debris by extracting morphological feature information, which increases the probability of missed detection and false alarm. Second, the detection method based on single-frame images has limitations. It can only extract spatial features and cannot obtain temporal features. This limits the performance of the algorithm and makes it difficult to obtain high-precision detection results. Summary of the invention

[0003] Based on this, it is necessary to provide a method and device for intelligent detection of spatial debris based on spatiotemporal feature fusion, which can solve the current problem of insufficient information in single-frame image target detection, in response to the above technical problems.

[0004] A spatial debris intelligent detection method based on spatiotemporal feature fusion, the method comprising:

[0005] Acquire a training sample consisting of multiple frames of images containing space debris target labels and / or star target labels;

[0006] The training sample is input into a multi-frame detection network; the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract the multi-frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features;

[0007] The multi-frame detection network is reversely trained according to the space debris target labels and / or star target labels of the training samples and the prediction results, so as to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network.

[0008] In one of the embodiments, it also includes: downsampling multiple frames of images in the training sample respectively as a bidirectional recurrent network structure composed of ConvGRU units, associating and fusing the multiple frame image information along the forward time and reverse time directions to obtain preliminary spatiotemporal semantic information.

[0009] In one embodiment, the backbone network includes multiple spatiotemporal feature extraction modules and convolutional layers; the spatiotemporal feature extraction modules include spatial feature extraction units and temporal feature extraction units; multiple spatiotemporal feature extraction modules are connected in sequence, the convolutional layers serve as the first and last layers of the backbone network, the spatiotemporal feature extraction modules serve as the middle layer of the backbone network, each layer of the backbone network is connected in sequence, and the outputs from the second layer to the last layer are connected to the neck module.

[0010] In one of the embodiments, the spatial feature extraction unit is an inverted residual structure of a lightweight network MobileNetV2, and an activation function of the inverted residual structure is replaced by a LeakyRelu function from a RELU6 function.

[0011] In one of the embodiments, the temporal feature extraction unit linearly maps the output of the spatial feature extraction unit to three different subspaces; the subspaces include: a query matrix, a key matrix and a value matrix; the query matrix, the key matrix and the value matrix are flattened along the channel dimension, and a block calculation method is adopted, that is, each matrix is ​​first decomposed into multiple parts from the channel dimension, and the query matrix and the key matrix of each part are used to obtain the corresponding relationship matrix; the output features of each part are calculated using the relationship matrix and the value matrix, the output features are spliced, and finally the spatiotemporal semantic information of the current layer of the backbone network is output.

[0012] In one of the embodiments, the neck module includes an information fusion unit and a distribution unit; the information fusion unit is used to fuse multi-scale spatiotemporal semantic information to obtain global features; the distribution unit is used to distribute the global features to each branch of the backbone network, fuse the spatiotemporal semantic information of the scales with its branches, and output the adjustment information of the current branch.

[0013] In one of the embodiments, the detection head includes three detection heads of different sizes, which are respectively responsible for detecting space debris of a first size, a second size, and a third size.

[0014] A space debris intelligent detection device based on spatiotemporal feature fusion, the device comprising:

[0015] A training sample acquisition module is used to acquire training samples consisting of multiple frames of images containing space debris target labels and / or star target labels;

[0016] A prediction module is used to input the training sample into a multi-frame detection network; the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract the multi-frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features;

[0017] A training module is used to reversely train the multi-frame detection network according to the space debris target labels and / or star target labels of the training samples and the prediction results, so as to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network.

[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0019] Acquire a training sample consisting of multiple frames of images containing space debris target labels and / or star target labels;

[0020] The training sample is input into a multi-frame detection network; the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract the multi-frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features;

[0021] The multi-frame detection network is reversely trained according to the space debris target labels and / or star target labels of the training samples and the prediction results, so as to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0023] Acquire a training sample consisting of multiple frames of images containing space debris target labels and / or star target labels;

[0024] The training sample is input into a multi-frame detection network; the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract the multi-frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features;

[0025] The multi-frame detection network is reversely trained according to the space debris target labels and / or star target labels of the training samples and the prediction results, so as to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network.

[0026] The above-mentioned spatial debris intelligent detection method and device based on spatiotemporal feature fusion associates multi-frame image information through an information fusion module, captures dependencies in sequence images, performs preliminary extraction of temporal features, further obtains deep temporal information, and constructs a backbone network for deep spatiotemporal semantic extraction to obtain multi-scale spatiotemporal semantic information. The network helps the network learn the temporal information between sequence images by establishing the changing relationship of spatial information at different moments in continuous image features, and finally adjusts and fuses multi-scale spatiotemporal semantic information through the neck module, and finally detects and outputs through the detection head. The present invention fully extracts the spatiotemporal information in multi-frame images, alleviates the information loss in the feature fusion process, and improves the detection accuracy of spatial target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 1 is a flow chart of a method for intelligently detecting spatial debris based on spatiotemporal feature fusion in one embodiment;

[0028] Figure 2 It is an overall structure diagram of a multi-frame detection network in one embodiment;

[0029] Figure 3 It is a schematic diagram of the overall process of a detection method in one embodiment;

[0030] Figure 4 Schematic diagram of a multi-frame image information fusion module in one embodiment;

[0031] Figure 5 Schematic diagram of the overall structure of a frame information fusion module in one embodiment;

[0032] Figure 6 Schematic diagram of the structure of a spatiotemporal feature extraction module in one embodiment;

[0033] Figure 7Schematic diagram of the structure of the inverted residual structure and the spatial feature extraction block in one embodiment; (a) is a schematic diagram of the inverted residual structure after the activation function is changed, and (b) is a schematic diagram of the spatial feature extraction unit;

[0034] Figure 8 A schematic diagram of a temporal feature extraction unit in one embodiment;

[0035] Fig. 9 This is a schematic diagram of the structure of a multi-scale feature fusion module in one embodiment;

[0036] Fig.10 A schematic diagram of a scale alignment structure in an embodiment;

[0037] Fig.11 A schematic diagram of information fusion and distribution in an embodiment;

[0038] Fig.12 A schematic diagram of an information injection structure in an embodiment;

[0039] Fig.13 A schematic diagram of the structure of a detection head in an embodiment;

[0040] Fig.14 Graphs showing the change of various types of losses in one embodiment; (a) is a total loss change curve, (b) is a position loss curve, (c) is a W and H loss change curve, and (d) is a confidence loss change curve;

[0041] Fig.15 1 is a curve diagram of detection rate and accuracy change in one embodiment; (a) is a curve of detection rate change, (b) is a curve of accuracy change;

[0042] Fig.16 It is a structural block diagram of a space debris intelligent detection device based on spatiotemporal feature fusion in one embodiment;

[0043] Fig.17 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] First, in ground-based optical detection images, both space debris and stars follow the imaging model of the optoelectronic system. Due to the non-ideality of optical sensor imaging, point light sources will have a diffusion effect, and the imaging effect is manifested as a nearly circular light spot centered on the ideal imaging point, following a two-dimensional Gaussian function distribution. In the actual detection process, the imaging morphology of space debris and stars will change due to the influence of the telescope observation mode. At present, there are two main observation modes for optical telescopes: star tracking mode and target tracking mode.

[0046] In star tracking mode, the telescope overcomes the rotation of the earth and rotates in the opposite direction at a speed of 15'' / s to ensure the stable position of the stars in the field of view. At this time, the telescope is stationary relative to the stars in the field of view, and the stars are in the form of points. However, due to the relative motion between the telescope and the space debris, the stripes are in the form of strips. The length of the strips depends mainly on the orbital altitude and the exposure time. Generally speaking, the space debris in low orbit has a faster relative motion speed and presents longer stripes. In contrast, the detected space debris is mainly in the geosynchronous orbit, and the relative motion speed of the space target is slower. In this observation mode, it usually appears as short strips. At the same time, increasing the exposure time will also lengthen the target.

[0047] In the target tracking mode, the telescope tracks the space target of interest by rotating. In the detection image, the target of interest is usually in the center as a point, while the stars appear as strips due to relative motion. It should be noted that other space debris in the image that is inconsistent with the direction of motion of the target of interest will usually appear as strips with inconsistent deflection angles with the stars.

[0048] In summary, no matter what working mode they are in, there will always be morphological differences between massive stars and a few space debris. Based on this characteristic and the powerful feature extraction and generalization capabilities of deep learning, this application proposes a multi-frame detection network to extract space debris that is inconsistent with the morphology of massive stars.

[0049] At present, the working modes of ground-based telescopes are mainly star tracking mode and target tracking mode. In both working modes, space debris and stars are in relative motion. The position information of each object in different frames of images is highly correlated, and there is rich temporal information in the sequence detection images. Based on this, a space debris intelligent detection method based on spatiotemporal feature fusion is provided. Figure 1 As shown, the following steps are included:

[0050] Step 102: Acquire a training sample consisting of multiple frames of images containing space debris target labels and / or star target labels.

[0051] It is worth noting that a multi-frame image is composed of multiple single-frame images with a time sequence, and each single-frame image may include only one or more space debris targets and be marked in the single-frame image, or may include star targets, or may include both space debris targets and star targets. The number of training samples is sufficient for the training of the multi-frame detection network, and the extraction process of training samples is not described here.

[0052] Step 104: input the training samples into the multi-frame detection network.

[0053] like Figure 2 As shown, the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract multiple frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs prediction results based on the fusion features.

[0054] It is worth noting that in Figure 2 Only the case where the backbone network is 5 layers is shown, but 6 layers or 7 layers can also be selected, which will not be repeated here.

[0055] like Figure 3 As shown in the figure, it is a schematic diagram of the overall process of the detection method of the present invention. The detection process mainly includes three stages. The first stage mainly preprocesses the image. Since the original image resolution is large, direct input will cause the algorithm parameters and the amount of calculation to increase sharply. Therefore, a 1024×1024 window is used to adjust the input resolution, and at the same time, the input area of ​​each frame image is guaranteed to be consistent. In addition, since space debris mainly exhibits two states during imaging, namely, static points and moving strips relative to the image, the two relative states have uneven data distribution, which easily leads to the network training falling into an overfitting or underfitting state. Therefore, before the network training, without changing the trajectory of the space debris, some data are augmented by applying motion blur and adding noise. The second stage is the training and prediction of the network. In the training part, the network uses the input data and the corresponding true value label for training to obtain the optimized and updated weight parameters. In the prediction part, the network uses the optimized parameters to output preliminary prediction results. The third stage mainly decodes and non-maximum suppresses the initial prediction value of the network. First, the relative position and size information in the prediction value is converted into absolute position and size information in the image, and then the output result is non-maximum suppressed to screen out the optimal prediction result.

[0056] Step 106, reverse training the multi-frame detection network according to the space debris target labels and / or star target labels of the training samples and the prediction results, so as to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network.

[0057] In the above-mentioned intelligent detection method of spatial debris based on spatiotemporal feature fusion, the information of multiple frames of images is associated through the information fusion module, the dependency relationship in the sequence image is captured, the temporal features are initially extracted, the deep temporal information is further obtained, and the backbone network is constructed for deep spatiotemporal semantic extraction to obtain multi-scale spatiotemporal semantic information. The network helps the network learn the temporal information between sequence images by establishing the change relationship of spatial information at different moments in continuous image features, and finally the multi-scale spatiotemporal semantic information is adjusted and fused through the neck module, and finally the detection output is performed through the detection head. The present invention fully extracts the spatiotemporal information in multiple frames of images, alleviates the information loss in the feature fusion process, and improves the detection accuracy of spatial target detection.

[0058] In one of the embodiments, multiple frames of images in the training sample are downsampled separately and used as a bidirectional recurrent network structure composed of ConvGRU units to associate and fuse the multiple frame image information along the forward time and reverse time directions to obtain preliminary spatiotemporal semantic information.

[0059] Specifically, in the sequence of starry sky images taken by ground-based telescopes, space debris and stars have rich motion information and maintain temporal and spatial consistency. In order to effectively associate multi-frame information and help the network extract temporal features, the present invention introduces the ConvGRU network, and constructs a multi-frame image information fusion (Multi-frame information fusion, MIF) module on its basis. ConvGRU is a convolutional gated recurrent network, which can be regarded as a variant of the long short-term memory network. Compared with the original recurrent neural network, ConvGRU can use the gating mechanism to overcome the gradient vanishing problem and effectively utilize the temporal information in the sequence image. Compared with the long short-term memory network, the ConvGRU structure is simpler and the training and fitting speed is faster. It can prevent overfitting during network training to a certain extent, and is suitable for the multi-frame detection task of the present invention. Its specific structure is as follows Figure 4 shown.

[0060] As can be seen from the figure, ConvGRU merges the candidate unit state with the hidden state, that is, only the candidate hidden state and hidden state , and only two gates are set at the same time, namely the update gate With reset gate The calculation of the update gate and the reset gate can be expressed as follows:

[0061] ;

[0062] ;

[0063] in the formula is the convolution operation, and The update gate and reset gate are input from the current time t. The hidden state output at the previous moment t-1 It is obtained by convolution and Sigmoid function, and the values ​​of the two are between 0 and 1. After obtaining the update gate and reset gate, the reset gate controls the hidden state output of the previous moment t-1 The candidate hidden state flowing into the current time t as follows

[0064] ;

[0065] in the formula and They are convolution and element-wise point multiplication operations, and They are the convolution kernel and bias in the convolution operation respectively.

[0066] The smaller the value of the reset gate is, the less information flows into the image at the previous moment, the more information is forgotten, and the network's ability to capture short-term dependencies in sequence images is enhanced. The update gate also controls the hidden state output of the previous moment t-1. , and the candidate hidden state at the current time t Transmit to the hidden state at the current moment The size is as follows:

[0067] ;

[0068] The larger the value of the update gate, the more information flows into the image at the previous moment, which helps the network capture long-term dependencies in the sequence of images.

[0069] The overall structure of the frame information fusion module proposed in the present invention is as follows: Figure 5As shown in the figure, at the front end of the module, in order to reduce training parameters and computational complexity, ease the training burden brought by large-size images, and prevent the loss of weak target information, each frame of the image will pass through an independent downsampling block with a downsampling multiple of 2. The sampling block consists of a convolution block containing 1×1 and 3×3 convolutional layers, and a pooling layer using the Focus structure. Focus performs slicing operations on alternate rows and columns to downsample the input without losing information. The module inputs the convolution feature map after downsampling of each frame into a bidirectional recurrent network structure composed of several ConvGRUs, associates and fuses multi-frame image information along the forward and reverse time directions, and extracts spatiotemporal features between images.

[0070] Through the multi-frame information module, the network reduces the size of input features, reduces the number of network parameters and computational complexity, completes the fusion of multi-frame image information and the preliminary extraction of spatiotemporal features, and provides feature input for the subsequent backbone network to extract deep spatiotemporal semantic information.

[0071] In one embodiment, the backbone network includes multiple spatiotemporal feature extraction modules and convolutional layers; the spatiotemporal feature extraction modules include spatial feature extraction units and temporal feature extraction units; multiple spatiotemporal feature extraction modules are connected in sequence, the convolutional layers serve as the first and last layers of the backbone network, the spatiotemporal feature extraction modules serve as the middle layers of the backbone network, each layer of the backbone network is connected in sequence, and the outputs from the second layer to the last layer are connected to the neck module.

[0072] Specifically, after the information fusion part, the network obtains feature inputs containing fine-grained spatial information and temporal information. However, the extracted feature receptive field is small and greatly affected by noise, making it difficult for the network to identify the target and other interfering objects in the background. Generally speaking, after multiple layers of convolution, the deep network is less affected by noise, the extracted feature receptive field is large, and contains rich contextual information, that is, the relationship and connection between pixels and other surrounding pixels. This information is helpful to enhance the network's ability to identify targets and other interfering objects. Based on the above ideas, in the intermediate feature extraction part, a backbone network is constructed for the extraction of deep spatiotemporal features, which is mainly composed of conventional convolution, downsampling, and spatiotemporal feature extraction structures. The spatiotemporal feature extraction structure includes two modules: spatial feature extraction (Spatial Feature Extraction, SFE) and temporal feature extraction (Temporal Feature Extraction, TFE). It is worth noting that as the number of network layers continues to deepen and the receptive field expands, it is easy to cause the loss of weak target detail information. Therefore, considering the size of conventional space debris targets and the appropriate receptive field, this chapter divides the backbone network structure into 5 layers. Each layer uses a standard 3×3 convolution block to expand the dimension of the feature channel, and the maximum downsampling method is used for pooling between layers. Finally, at the output of the backbone network, the deep features extracted from the second to fifth layers are used as the input of the subsequent network neck. Figure 6 This is a brief structural diagram of the spatiotemporal feature extraction module. Represents a splicing operation, Indicates the number of spatial feature modules.

[0073] In one embodiment, the spatial feature extraction unit is an inverted residual structure of a lightweight network MobileNetV2, and the activation function of the inverted residual structure is replaced by a LeakyRelu function from a RELU6 function.

[0074] Specifically, for spatial feature extraction, many high-performance feature extractor structures have been proposed, such as ResNet, Xception, etc. These structures have played an important role in different visual tasks. However, these structures have a large number of parameters and a high floating-point operation cost, which is not conducive to the real-time performance of network detection. Therefore, the present invention refers to the inverted residual structure of MobileNetV2, a lightweight network with better performance and fewer parameters, and builds a spatial feature extraction module on its basis to help the network learn spatial information. The original inverted residual structure uses the RELU6 function as the activation function. It was originally designed to enable the network to work well on low-precision mobile devices and reduce precision loss, but it is not suitable for the task of the present invention. It is easy to cause the "death" of neurons, that is, the gradient is always 0. Therefore, the present invention uses the LeakyRelu function as the activation function. The schematic diagram of the inverted residual structure and the specific structure of the spatial feature extraction module are shown as follows. Figure 7 (a) and 7(b).

[0075] In one of the embodiments, the temporal feature extraction unit linearly maps the output of the spatial feature extraction unit to three different subspaces; the subspaces include: a query matrix, a key matrix, and a value matrix; the query matrix, the key matrix, and the value matrix are flattened along the channel dimension, and a block calculation method is adopted, that is, each matrix is ​​first decomposed into multiple parts from the channel dimension, and the query matrix and the key matrix of each part are used to obtain the corresponding relationship matrix; the output features of each part are calculated using the relationship matrix and the value matrix, the output features are spliced, and finally the spatiotemporal semantic information of the current layer of the backbone network is output.

[0076] Specifically, in the deep feature map, each feature element contains the network's learning of a local area information in the input image. In the sequence detection image, the target and the background are dynamic, and the content of the area at the same position in the star map will change at different times, so the feature information contained in the feature elements at the corresponding position will also change. Based on these priors, the present invention combines the non-local attention mechanism to construct a temporal feature extraction module, which helps the network learn the temporal information between sequence images by establishing the spatial information relationship at different times in the continuous image features. The specific structure of this module is as follows Figure 8 shown.

[0077] For simplicity, assume that the output features of the spatial feature extraction block of layer N are , represents the dimension of the spliced ​​image features of T frames, H and W represent the height and width of the image features respectively. In the temporal feature extraction module, the features are first extracted on the spliced ​​channel dimension. Transform to get , and then use three 3D 1×1×1 convolutions ( , , )Will Projected into three different subspaces, namely the query matrix, the key matrix and the value matrix (Q, K, V), the mapping process is as follows.

[0078]

[0079] Similar to the self-attention mechanism, since the module inputs a sequence of images, the query matrix represents the features of all regions in the sequence of images that the attention focuses on, the key matrix represents the information about the features at the positions that the attention mechanism focuses on, and the value matrix represents the information contained in the features at all positions in the multi-frame images. After the projection, the query matrix, key matrix, and value matrix are flattened along the channel dimension, and the block calculation method is adopted, that is, each matrix is ​​first decomposed from the channel dimension into parts, and then use the query matrix and key matrix of each part to get the corresponding relationship matrix , the calculation formula is as follows.

[0080] ;

[0081] in , represents the number of channels of each head after multi-frame channel decomposition, represents the normalization operation, Indicates The relationship matrix corresponding to each part.

[0082] The relationship matrix can be used to model the correlation of spatial information between the feature maps of each channel at different times. Finally, the relationship matrix and the value matrix are used to calculate the output features of each part, and then these features are spliced, fused and adjusted by convolution to obtain the final output features of the temporal feature extraction module in the Nth layer. The calculation process is shown in the formula below.

[0083] ;

[0084] ;

[0085] In the above formula is a 1×1 convolution, For splicing operation, The output features corresponding to each part.

[0086] In one of the embodiments, the neck module includes an information fusion unit and a distribution unit; the information fusion unit is used to fuse multi-scale spatiotemporal semantic information to obtain global features; the distribution unit is used to distribute the global features to each branch of the backbone network, fuse the spatiotemporal semantic information of the scales with its branches, and output the adjustment information of the current branch.

[0087] Specifically, the network neck is mainly responsible for fusing the features of different scales output by the backbone network, enhancing the network's detection capability for targets of different scales, and providing feature input for the final detection head. At present, for a variety of multi-scale feature fusions, the more representative ones are feature pyramid networks, path aggregation networks, etc. These feature fusion methods similar to pyramid structures show good performance in many detection networks, but there are still some defects. For example, when the output feature information of the second and fourth layers of the backbone network needs to be fused across layers, the above method can only be used to fuse and adjust with the adjacent layers first, and then further pass to the next layer. This means that the interaction of information between layers can only exchange the information selected by the middle layer, while the information discarded by the middle layer cannot be exchanged with each other, and the information cannot be transmitted losslessly, which affects the overall effectiveness of information fusion. In order to alleviate the above problems, the present invention will construct a multi-scale feature fusion module based on a collection and distribution mechanism with a simpler structure, that is, first collect and fuse all the original multi-scale features output to obtain global features, and then the global features distribute the local feature information required by each layer, and finally the original output features of each layer are fused with the distributed feature information to obtain the final output. This method makes the information exchange between layers unimpeded, and each layer output feature can obtain the required feature information, which is conducive to improving the network's multi-scale feature fusion capability and improving detection performance. At the same time, this feature fusion structure is more concise and will not significantly increase the parameters and computing costs. Fig. 9 This is a schematic diagram of the structure of the multi-scale feature fusion module in the network of this chapter.

[0088] In the scale alignment process, the collection and retention of shallow detail information and high-level semantic information are comprehensively considered, while minimizing the number of parameters and the complexity of subsequent calculations. The present invention uses bilinear interpolation and maximum pooling to align the size of the original output features of each layer in the backbone network with the size of the third layer, and then splices them to obtain the global features. The number of channels is the same as the output features of each backbone network ( , , , )The sum of the number of channels, Fig.10 Schematic diagram of scale alignment structure.

[0089] After scale alignment, the module uses multiple layers of 3×3 residual convolution blocks and one layer of 1×1 convolution to fuse the global features. The number of feature channels output by this process is not fixed and can be adjusted according to the needs of the network size. In the experiments in this chapter, the number of feature channels after fusion is 1 / 2 of the initial global features. The fused features are distributed through the convolution blocks of the decoupled structure, that is, the adjustment information required for the original features of different scales is output through three branches. , as well as The number of channels of each adjustment information is equal to the corresponding original output feature, and is finally fused with the original output feature through the information injection structure. Fig.11 The structural diagram of the two processes of information fusion and distribution is shown.

[0090] The information injection structure of the multi-scale feature fusion module is designed to fuse the adjustment information with each original output feature. Since the global features are not pooled and upsampled during the distribution process, there is a difference in the scale of some adjustment information and the original output features of the corresponding layer. In order to more effectively inject the adjustment information containing global features into the output features of different levels, this chapter adopts the spatial attention method for fusion. The specific structure is as follows Fig.12 As shown in the figure, the adjustment information is passed through a 1×1 convolutional layer and a Sigmoid function to obtain an attention weight matrix, which is aligned with the original features through average pooling or bilinear interpolation and adjusted by dot multiplication. The adjusted original features are added and fused with the adjustment information to obtain the final output features.

[0091] In one embodiment, the detection head includes three detection heads of different sizes, which are respectively responsible for detecting space debris of a first size, a second size, and a third size.

[0092] The first size is responsible for the detection of large streak-like space debris, the second size is responsible for the detection of medium-sized space debris, and the third size is responsible for the detection of point-like small-sized space debris.

[0093] Specifically, the detection head is mainly responsible for the final prediction output of the network. Its input is the output features of the second, third and fourth layers of the backbone network after multi-scale information adjustment, and the output is information including target confidence, coordinates, size and category. The network in the present invention adopts a coupled detection head with a simple structure and a small number of parameters. For point-shaped and strip-shaped space debris targets of different sizes, this network has three different sizes of detection heads (H2, H3, H4) for prediction output, which are responsible for the detection of large stripes, medium-sized and point-shaped small-sized space debris respectively. Its specific structure is as follows Fig.13 shown.

[0094] The experiment of the present invention uses the sequence detection images taken by the ground-based optical telescope after background suppression for training and testing. Due to the large resolution of the original image and the limited computing power and storage space of the experimental hardware, the sequence images are segmented using a window of size 1024×1024, and the segmented continuous 3-frame images are used as the input of the network. The experimental data includes 13 sequence images and some synthetic sequence images blurred by adding Gaussian noise, a total of 2104 images. It can be seen from the above that in the detection image, the target and the star are in a state of relative motion, which is specifically manifested in the sequence image as the target relative image is stationary and the star relative image is moving, the star relative image is stationary and the target relative image is moving, and the star and the target relative image are both moving. Due to the limited amount of data, there are differences in the number of images between different sequence data. Random selection will lead to an imbalance in the proportion of samples in different states in the training set or validation set. The trained network has a strong bias, that is, it only performs well under the type with a large number of types. Therefore, this chapter screens the training samples to keep the data in different states balanced.

[0095] The hardware platform and environment configuration used in the training phase of the experiment are shown in Table 1. The experiments in this chapter are based on the Windows operating system and use the PyTorch deep learning development framework. The hardware CPU and GPU used in the experiment are Intel i9-10980XE and RTX3090 respectively. In terms of network training parameter settings, the number of training iterations is 200 and the batch size is 4. The network uses Adam as a parameter optimizer, and its initial learning rate and minimum learning rate are set to 1× and 1× , using the cosine annealing strategy.

[0096] Table 1 Experimental environment parameter configuration

[0097]

[0098] The detection target of the present invention is space debris, which belongs to single-category target detection. In order to verify the performance of the proposed space debris intelligent detection network based on spatiotemporal feature fusion, some conventional evaluation indicators such as detection rate, false alarm rate and The detection performance of the algorithm is verified by scores, and the detection rate used in this chapter is calculated as follows:

[0099] ;

[0100] In the formula and They represent the number of correctly detected targets and the number of all targets in the sample, respectively.

[0101] The calculation methods of false alarm rate and F1 score are as follows:

[0102] ;

[0103] In the above formula and They represent the total number of targets and the number of targets that were incorrectly identified.

[0104] ;

[0105] In the above formula represents the correct prediction target, represents the wrong prediction target, Indicates an undetected target.

[0106] In the detection task, the network only detects space debris, which is a single-category target detection. Therefore, the training loss does not include the classification loss, but only the target positioning loss. With confidence loss The composition is as follows.

[0107] ;

[0108] In the formula The number of anchor boxes corresponding to the true value predicted by the network, and is a weight parameter used to balance different types of losses.

[0109] Positioning loss The loss is shifted by the center of all objects Length and width loss Composition, where a single target The localization loss is shown below.

[0110] ;

[0111] In order to increase the loss ratio of small-scale point-shaped space debris targets in the image, Set it as a dynamic enlargement loss weight, which is related to the size of the target. The smaller the target, the larger its value. The calculation method is as follows. For the center offset loss and length and width loss, the experiments use the Smooth_L1 loss function for calculation.

[0112] ;

[0113] In the above formula and A single target for participation in loss calculation The length and width, and is the length and width of the input image, is a constant and is set to 2 during the experiments in this chapter.

[0114] Since the prediction part of the network is composed of three detection heads, the total confidence loss is the sum of the confidence losses of the three detection heads. In the experiments in this chapter, the network's attention to tiny point-like spatial fragments is further improved, and the confidence loss ratio of each detection head is different. The experiment sets the loss ratio of each detection head to ƒ=[0.4, 1, 4], corresponding to the first layer of large-scale detection heads, the second layer of medium-scale detection heads, and the third layer of small-scale detection heads. Since the confidence loss is a binary classification loss, the experiment uses the cross entropy loss function for calculation. At the same time, due to the scarcity of targets and the imbalance of positive and negative samples, the reference weights are used. Balance the positive and negative sample losses. In this experiment The value is set to 0.1, and the formula is as follows. The confidence loss calculation method for the positive sample target is as follows.

[0115] ;

[0116] ;

[0117] In the above formula and They represent the confidence loss of positive samples and the confidence loss of negative samples respectively.

[0118] During the network training process, the changes in various losses and evaluation index values ​​reflect the rationality of the network structure design to a certain extent. A good and reasonable structural layout can enable the network to converge stably during the training process and prevent overfitting. In order to verify the rationality of the proposed multi-frame detection network structure, the experiment displays and analyzes the curves of various losses of the network on the training set and the detection rate and accuracy on the validation set. At the same time, in order to test the performance of the proposed multi-frame detection algorithm, the experiment selects the multi-frame target detection algorithm LSTM-SSD, the single-frame weak target detection algorithm Yolo-s, and the detection method based on morphological difference learning (MFD-Net) as comparisons. During the experiment, in order to eliminate other interference factors, all detection networks do not perform parameter pre-training, and the network training and testing are uniformly performed on the measured sequence star map data.

[0119] Fig.14 The graph shows how various types of losses change with the number of iterations during the experiment, including confidence, coordinates (x, y), and target size (w, h). It can be clearly seen from the figure that among all types of losses, the confidence loss value calculated using the cross entropy function is the largest and accounts for the highest proportion. As the number of iterations increases, the various types of losses generally show a trend of decreasing and then gradually converging, without large-scale fluctuations, indicating that the network training process on the training set is relatively stable and the loss can converge well.

[0120] In terms of detection rate and accuracy, Fig.15 It can be seen that the detection rate and accuracy of the network for the validation set will fluctuate during the training process, but generally they tend to increase gradually with the number of iterations and eventually stabilize without overfitting or underfitting.

[0121] In summary, the experimental results show that the network has good stability during the iterative optimization process, the network structure is relatively reasonable, and can adapt to the task of space debris detection in the sequence detection images in this chapter.

[0122] In terms of comparative experiments, the detection results of different algorithms in the test set are shown in Table 2 below. It can be seen from the table that in the single-frame detection algorithm, SSDebris-Net achieves better performance than Yolo-s, with a detection rate of 86.26% and a false alarm rate of 9.91%. However, after the image input size is fixed to 1024×1024, the pixel ratio of the target in the image is further reduced, and there are interfering objects with similar shapes to the target in the image. Compared with the single-frame image test results, the performance of MFD-Net has declined. Compared with the single-frame detection algorithm, the multi-frame detection algorithm proposed in the present invention combines temporal information and spatial information, and has a stronger ability to identify space debris and interfering stars. The optimal detection rate and false alarm rate are obtained on the test data, which are 98.45% and 2.51% respectively. Compared with MFD-Net, the detection rate is increased by 12.19%, the false alarm rate is reduced by 7.4%, and the detection performance is greatly improved. The experimental results effectively prove that the fusion of temporal information and spatial information is conducive to improving the network's detection performance for weak space debris.

[0123] Table 2 Detection results of different algorithms on test data

[0124]

[0125] In the multi-frame detection network of the present invention, the main purpose of the multi-frame image information fusion module and the temporal feature extraction module based on the bidirectional ConvGRU is to help the network learn the temporal information in the sequence image and improve the detection performance of the network. In order to verify the effectiveness of these two components and their contribution to network detection, this section conducts an ablation experiment on the two components. The experiment regards the network without the multi-frame image information fusion and temporal feature extraction modules as the basic network, and compares it with the improved network after adding the multi-frame information fusion and temporal feature extraction modules, excluding other interference factors, and each network is not pre-trained, and the input is 3 consecutive frames of original images. In order to comprehensively compare the performance of each network in terms of detection rate and detection accuracy, this ablation experiment uses The score and false alarm rate are used as quantitative evaluation indicators to measure the performance of each network. The specific experimental results are shown in Table 3.

[0126] Table 3 Ablation experiment test results

[0127]

[0128] It can be clearly seen from the data in the table that after adding the structure for extracting temporal information to the basic network, the detection performance of the network on the test data set is greatly improved. After adding the multi-frame image information fusion module and the temporal feature extraction module separately, the detection F1 score increased by 3.27% and 5.91% respectively, and the false alarm rate decreased by 9.91% and 8.03%. After adding the combination of the two modules, the detection performance of the overall network is further improved, and the optimal F1 score is obtained, while the false alarm rate is reduced to 2.51%. The above quantitative ablation experiment results show that the two components for temporal information extraction constructed in this chapter can improve the network's performance in detecting space debris in the test data and enhance the network's ability to identify space debris and other interfering objects.

[0129] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0130] In one embodiment, Fig.16 As shown, a spatial debris intelligent detection device based on spatiotemporal feature fusion is provided, comprising: a training sample acquisition module 1602, a prediction module 1604 and a training module 1606, wherein:

[0131] The training sample acquisition module 1602 is used to acquire a training sample consisting of multiple frames of images containing space debris target labels and / or star target labels;

[0132] The prediction module 1604 is used to input the training sample into a multi-frame detection network; the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract the multi-frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features;

[0133] The training module 1606 is used to reversely train the multi-frame detection network according to the space debris target labels and / or star target labels of the training samples and the prediction results, so as to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network.

[0134] In one embodiment, the prediction module 1604 is also used to downsample multiple frames of images in the training sample as a bidirectional recurrent network structure composed of ConvGRU units, associate and fuse multiple frame image information along the forward time and reverse time directions, and obtain preliminary spatiotemporal semantic information.

[0135] In one embodiment, the backbone network includes multiple spatiotemporal feature extraction modules and convolutional layers; the spatiotemporal feature extraction modules include spatial feature extraction units and temporal feature extraction units; multiple spatiotemporal feature extraction modules are connected in sequence, the convolutional layers serve as the first and last layers of the backbone network, the spatiotemporal feature extraction modules serve as the middle layer of the backbone network, each layer of the backbone network is connected in sequence, and the outputs from the second layer to the last layer are connected to the neck module.

[0136] In one of the embodiments, the spatial feature extraction unit is an inverted residual structure of a lightweight network MobileNetV2, and an activation function of the inverted residual structure is replaced by a LeakyRelu function from a RELU6 function.

[0137] In one of the embodiments, the temporal feature extraction unit linearly maps the output of the spatial feature extraction unit to three different subspaces; the subspaces include: a query matrix, a key matrix and a value matrix; the query matrix, the key matrix and the value matrix are flattened along the channel dimension, and a block calculation method is adopted, that is, each matrix is ​​first decomposed into multiple parts from the channel dimension, and the query matrix and the key matrix of each part are used to obtain the corresponding relationship matrix; the output features of each part are calculated using the relationship matrix and the value matrix, the output features are spliced, and finally the spatiotemporal semantic information of the current layer of the backbone network is output.

[0138] In one of the embodiments, the neck module includes an information fusion unit and a distribution unit; the information fusion unit is used to fuse multi-scale spatiotemporal semantic information to obtain global features; the distribution unit is used to distribute the global features to each branch of the backbone network, fuse the spatiotemporal semantic information of the scales with its branches, and output the adjustment information of the current branch.

[0139] In one of the embodiments, the detection head includes three detection heads of different sizes, which are respectively responsible for detecting space debris of a first size, a second size, and a third size.

[0140] For the specific definition of the space debris intelligent detection device based on spatiotemporal feature fusion, please refer to the definition of the space debris intelligent detection method based on spatiotemporal feature fusion mentioned above, which will not be repeated here. Each module in the above-mentioned space debris intelligent detection device based on spatiotemporal feature fusion can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0141] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.17 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multiple frames of image data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a spatial debris intelligent detection method based on spatiotemporal feature fusion is implemented.

[0142] Those skilled in the art will understand that Fig.17 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0145] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A spatial debris intelligent detection method based on spatiotemporal feature fusion, characterized in that: The method comprises: Acquire a training sample consisting of multiple frames of images containing space debris target labels and / or star target labels; The training sample is input into a multi-frame detection network; the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract the multi-frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features; According to the space debris target labels and / or star target labels of the training samples and the prediction results, reverse training is performed on the multi-frame detection network to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network; The multiple frames of images in the training sample are fused and extracted to obtain preliminary spatiotemporal semantic information, including: After downsampling multiple frames in the training sample, they are used as a bidirectional recurrent network structure composed of ConvGRU units to associate and fuse the multi-frame image information along the forward and reverse time directions to obtain preliminary spatiotemporal semantic information. The backbone network includes multiple spatiotemporal feature extraction modules and convolutional layers; the spatiotemporal feature extraction module includes a spatial feature extraction unit and a temporal feature extraction unit; Multiple spatiotemporal feature extraction modules are connected in sequence, the convolution layer serves as the first and last layer of the backbone network, the spatiotemporal feature extraction module serves as the middle layer of the backbone network, each layer of the backbone network is connected in sequence, and the output from the second layer to the last layer is connected to the neck module.

2. The method according to claim 1, characterized in that The spatial feature extraction unit is an inverted residual structure of a lightweight network MobileNetV2, and the activation function of the inverted residual structure is replaced by a LeakyRelu function from a RELU6 function.

3. The method according to claim 1, characterized in that The temporal feature extraction unit linearly maps the output of the spatial feature extraction unit to three different subspaces; the subspaces include: a query matrix, a key matrix, and a value matrix; The query matrix, key matrix and value matrix are flattened along the channel dimension, and a block calculation method is adopted, that is, each matrix is ​​first decomposed into multiple parts from the channel dimension, and the corresponding relationship matrix is ​​obtained using the query matrix and key matrix of each part; The output features of each part are calculated using the relationship matrix and the value matrix, and the output features are concatenated to finally output the spatiotemporal semantic information of the current layer of the backbone network.

4. The method according to any one of claims 1 to 3, characterized in that: The neck module includes an information fusion unit and a distribution unit; The information fusion unit is used to fuse multi-scale spatiotemporal semantic information to obtain global features; The distribution unit is used to distribute the global feature to each branch of the backbone network, fuse the spatiotemporal semantic information of the scale with its branches, and output the adjustment information of the current branch.

5. The method according to any one of claims 1 to 3, characterized in that: The detection head includes three detection heads of different sizes, which are respectively responsible for detecting space debris of a first size, a second size and a third size.

6. A space debris intelligent detection device based on spatiotemporal feature fusion, used to implement the space debris intelligent detection method based on spatiotemporal feature fusion according to any one of claims 1 to 5, characterized in that: The device comprises: A training sample acquisition module is used to acquire training samples consisting of multiple frames of images containing space debris target labels and / or star target labels; A prediction module is used to input the training sample into a multi-frame detection network; the multi-frame detection network includes: an information fusion module, a backbone network, a neck module and a detection head; the information fusion module is used to fuse and extract the multi-frame images in the training sample to obtain preliminary spatiotemporal semantic information; the backbone network performs deep spatiotemporal semantic extraction based on the preliminary spatiotemporal semantic information to obtain multi-scale spatiotemporal semantic information; the neck module adjusts and fuses the multi-scale spatiotemporal semantic information to obtain fusion features; the detection head outputs a prediction result based on the fusion features; A training module is used to reversely train the multi-frame detection network according to the space debris target labels and / or star target labels of the training samples and the prediction results, so as to perform space debris target detection on the multi-frame images to be detected according to the trained multi-frame detection network.

Images