Remote sensing video weak and small target detection method and system based on three-branch space-time modeling

Abstract

The invention relates to the technical field of remote sensing video target detection, and particularly discloses a remote sensing video weak and small target detection method and system based on three-branch space-time modeling. According to the method, a continuous multi-frame image sequence serves as input, and firstly, feature information of a space branch, a time branch and a space-time branch is extracted through a three-branch space-time encoder; the features output by the three branches are fused in an encoder to form a time-space enhanced multi-frame joint feature; further inputting the fused features into a feature extraction network, extracting high-level semantic information of the target region, and retaining a time sequence context; a result after feature extraction is input into a space-time decoder for time sequence dimension aggregation to generate a unified perception feature map of the current frame; and finally, the feature map input detection head outputs detection frame positions and confidence degree results of all moving targets in the current frame. According to the invention, the response capability to a tiny target can be effectively enhanced, and the detection precision is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing video target detection, and particularly relates to a remote sensing video weak small target detection method and system based on three-branch spatiotemporal modeling. BACKGROUND

[0002] With the gradual opening of high-resolution remote sensing videos and the development of remote sensing technology, automatically detecting and identifying moving targets on the ground from continuous frame sequences has become an important research direction of intelligent interpretation of remote sensing videos, and is widely used in urban traffic management, marine monitoring and border patrol scenes. However, due to its unique imaging method and overhead perspective, remote sensing videos face many technical challenges in multi-target detection tasks: the targets in remote sensing videos are often small in size and densely distributed, which brings great difficulties to target perception and detection; secondly, due to complex background and low contrast, single images are difficult to provide sufficient motion cues, resulting in limited recognition accuracy of detection models based on static images; existing methods mostly rely on image-level feature modeling or global attention mechanisms, lack explicit modeling of fine-grained motion information and differential modeling capability of local motion regions, and are difficult to adapt to complex remote sensing scenes where weak and small targets coexist with significant interference.

[0003] In recent years, some methods have tried to enhance the model's perception of target dynamic characteristics through temporal modeling, such as introducing temporal feature fusion, optical flow guided feature propagation, etc. However, most of these methods fail to achieve fine modeling in the spatial and temporal dimensions, and cannot capture weak motion information and multi-scale spatial structures simultaneously, resulting in significant deficiencies in the detection accuracy of weak and small targets. Therefore, there is an urgent need for a remote sensing video target detection method that has efficient spatiotemporal modeling capability, can perceive weak target motion changes, and improve detection robustness, to break through the bottleneck of existing technology in fine-grained motion modeling and weak target perception. SUMMARY

[0004] The present application aims to solve the technical problems of low detection accuracy of weak and small targets in remote sensing videos, weak temporal modeling capability, and severe background interference, and proposes a detection method and system based on three-branch spatiotemporal modeling. In view of the lack of perception and modeling of fine-grained dynamic features in existing methods when dealing with weak motion targets, the present application designs a three-branch coding structure with spatial, temporal and joint modeling and a multi-layer spatiotemporal decoding mechanism, which can model target features from three angles of spatial structure, temporal change and their coupling relationship, and improve the detection accuracy and robustness of low-speed moving and appearance blurred targets.

[0005] The purpose of the present application can be achieved by the following technical solutions: A remote sensing video weak small target detection method based on three-branch spatiotemporal modeling, comprising the following steps: Step one: Obtain a remote sensing video image sequence with length of and perform uniform preprocessing operations including cropping, scaling and normalization, etc., to arrange it into a feature input tensor , where is the batch size, , , are the channel number, image height and width respectively.

[0006] Step two: input the above image sequence into a three-branch spatio-temporal coding module to extract time domain features, spatial domain features and spatio-temporal joint features respectively, and fuse them into a time series enhanced feature sequence .

[0007] Step three: input the time series enhanced feature sequence into a static feature extraction network (such as ResNet or DLA) to extract static image features and extract a unified feature sequence after spatio-temporal fusion ; Step four: input the unified feature sequence into a spatio-temporal decoding module, use multiple stacked 3D deconvolution and three-dimensional pooling strategy to restore the motion structure features of each frame, and output the final spatio-temporal detection feature map corresponding to the current frame ; Step five: input the fused features into the detection head, combine Top-k screening and non-maximum suppression (NMS) and other post-processing operations, and output the weak and small moving target detection result of the current frame, including target category, position and confidence, etc.

[0008] Optionally, in the step two, the three-branch spatio-temporal encoder includes the following three feature extraction paths: S2.1: the time branch extracts the inter-frame dynamic changes through three layers of 3D convolution structure, represented as: ; S2.2: the spatial branch extracts the intra-image texture structure through three layers of 3D convolution, represented as: ; S2.3: the spatio-temporal branch models the joint spatio-temporal features through three layers of 3D convolution: ; S2.4: the three paths are output in channel dimension, and are fused into a unified encoding sequence using convolution: ; S2.5: the fused features pass through a The three-dimensional convolution is used for channel compression to obtain a final encoding output: ; Optionally, in the step four, the decoder structure comprises the following process: S4.1: three-layer 3D de-convolution is used to gradually restore the spatial resolution: ; S4.2: the final feature of the current frame is obtained by maximum pooling in the time direction: ; In addition, the present application also provides a remote sensing video weak small target detection system based on three-branch spatiotemporal modeling, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to execute the remote sensing video weak small target detection method based on three-branch spatiotemporal modeling.

[0009] In addition, the present application also provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being used for programming or configuring the microprocessor to execute the remote sensing video weak small target detection method based on three-branch spatiotemporal modeling.

[0010] The present application has the following beneficial effects: 1. Enhancing the perception ability of small moving targets: by introducing a three-branch structure to model the time dimension, the space dimension and their joint relationship respectively in the encoding stage, the weak displacement and fine-grained appearance change of the target in the remote sensing video are effectively captured, and the detection ability of low-speed moving or almost static targets is significantly improved.

[0011] 2. Simple and efficient structure, easy to deploy: the proposed three-branch spatiotemporal encoder is constructed by using lightweight three-dimensional convolution, which balances the model expression and computational efficiency, and is suitable for deployment and operation in satellite image analysis platforms with limited computing power.

[0012] 3. Improving the temporal consistency of target detection: using the spatiotemporal decoder to dynamically aggregate and analyze the encoded features helps to strengthen the correlation between different frames, improves the continuity and stability of the detection results, and reduces the missed detection and jitter phenomenon.

[0013] 4. Good adaptability and universality: the method does not rely on specific category or scene priori, and can adapt to targets of different sizes and different motion patterns, and has good performance in typical remote sensing applications such as vehicles, ships and aircrafts, and has wide engineering application potential. BRIEF DESCRIPTION OF DRAWINGS

[0014] The present application will be further described below with reference to the accompanying drawings.

[0015] Figure 1is a flowchart of a remote sensing video weak and small target detection method based on three-branch spatio-temporal modeling according to the present application; Figure 2 is a structural schematic diagram of a three-branch spatio-temporal encoder in an embodiment of the present application; Figure 3 is a structural schematic diagram of a spatio-temporal decoder in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0017] Please refer to Figures 1-3 The present application is a remote sensing video weak and small target detection method based on three-branch spatio-temporal modeling, which comprises the following steps: Step 1: Image sequence input and preprocessing. This step is used to construct a remote sensing video image sequence tensor conforming to the network input requirements as the input basis of the subsequent spatio-temporal encoder. The process includes image sequence acquisition, image size unification, normalization processing and other key operations. Specifically, first, a frame sequence with a fixed length of is continuously intercepted from the remote sensing video, denoted as: , where is the number of frames. In order to ensure that all frames have consistent spatial resolution in the spatio-temporal modeling process, the original image needs to be uniformly scaled to a fixed size, i.e. size redirection operation: , so that ; where H, W are the image height and width; Next, each frame of image is normalized to map its value range from [0, 255] to the interval [0, 1] to adapt to the input requirements of the deep learning model: ; Through the above operations, the original remote sensing video sequence will be standardized to an image sequence with consistent size and data format. The original image sequence is uniformly formatted, resolution cropped and pixel normalized to adapt to the input specifications of the deep learning model, ensuring the stability and consistency of subsequent model inference.

[0018] Step Two: Three-Branch Spatiotemporal Coding Module Modeling Feature Sequence: This step extracts multi-dimensional features from the input image temporal tensor to capture subtle motion information of targets in remote sensing videos and their spatial texture distribution. By designing three paths—temporal branch, spatial branch, and spatiotemporal joint branch—and ultimately fusing them into a unified coding representation, a temporally enhanced feature sequence with stronger perception capabilities for small-scale, weakly moving targets is obtained. The specific process is as follows: The sequence obtained in step one Simultaneously input to the three-branch module, such as Figure 2 As shown, the temporal branch extracts inter-frame dynamic changes, focusing on extracting short-term dynamic patterns between video frames. It employs a three-layer stacked 3D convolution with a kernel size of 3×1×1, meaning convolution is performed only in the temporal dimension, preserving the spatial structure. The output of this branch is: , ; in Indicates the first The temporal convolution operation is performed on each layer, followed by BatchNorm and ReLU activation functions after each convolution.

[0019] The spatial branch extracts images focusing on intra-frame static feature modeling, capturing the contour and texture information of the target. This branch uses a three-layer 1×3×3 convolutional structure, modeling only in the spatial dimension. The output of this branch is: , ; in, Indicates the first Spatial convolution operation of layers.

[0020] The role of spatiotemporal branch modeling is to jointly feature and simultaneously perceive the coupling pattern of temporal and spatial changes. A 3×3×3 stacked convolutional structure is introduced to model the joint structural changes across frames. The output is: , ; Then, the three-branch feature channels are concatenated and fused. The feature tensors obtained from the three branches are concatenated along the channel dimension to obtain the joint feature representation before fusion. ; Finally, a 1×1×1 three-dimensional convolution kernel is applied to perform channel fusion, forming a temporal enhanced feature sequence: ; Step three: spatial detail feature extraction of the current frame, by sequentially performing static feature extraction on the time sequence enhanced feature sequence obtained in step two, more accurate positioning and discrimination of small weak targets are realized. The input video sequence is sequentially input into the static image feature extraction network, such as DLA-34, ResNet-18, etc. lightweight detection backbone network, and the spatial semantic unified feature sequence is extracted: , specifically: ; Among them represents a static feature extraction network, such as DLA-34, ResNet-18 network, etc. represents the i-th frame feature obtained by taking the i-th slice in the time dimension.

[0021] Step four: spatio-temporal decoding and fusion detection prediction. The spatio-temporal decoding module is used to reconstruct and aggregate the time sequence enhanced feature sequence, the inter-frame motion structure details are recovered through stacked 3D deconvolution operation, and the target features of the current frame are focused to generate the final detection feature map. First, the unified feature sequence is input into the stacked 3D deconvolution module to gradually recover its time sequence structure and spatial boundary information to construct motion enhanced features, as shown in Figure 3 , the specific steps are as follows: sequentially passing through three deconvolution modules: ; ; ; Among them is a module composed of three-dimensional deconvolution, normalization and activation function. In order to obtain the motion perception detection features of the current frame, the is executed along the time dimension three-dimensional maximum pooling operation, focusing on the spatio-temporal context of the current time: ; This feature combines the static appearance information of the current frame and the dynamic changes between multiple frames, effectively improving the detection ability of low-speed and small targets.

[0022] Step five: target prediction and detection output. This step uses the final fusion feature map obtained in step four to detect small weak targets in the current frame. The detection module regresses the class heat map, size and offset information of the target through multiple convolutional decoding feature maps, and combines post-processing operations to output the complete detection result of the current frame. Among them, the multi-branch detection head structure inputs the current frame fusion feature map obtained in step four into three convolutional prediction branches after compressing the time dimension, respectively generating the following prediction results: Class heat map: used to represent whether there is a target of a certain class at each spatial position, specifically: ; wherein is the number of target classes, is the probability map of each target class of the current frame.

[0023] Width-height size map: used to estimate the width and height of the target, specifically: ; represents the value of the target at the current position.

[0024] Center offset map: used to fine-tune the position of the detection box to compensate for the deviation of the center point caused by down-sampling.

[0025] ; After obtaining the class heat map , a Top-K operation is performed to obtain the top K target candidate points with the highest response value: ; Each candidate point combines the size prediction and offset of its corresponding position to decode the candidate box: ; wherein, and represent the offset values of xi and yi, respectively; Then, non-maximum suppression is performed to remove redundant or highly overlapping candidate boxes and only keep the most confident box: ; The is output as the final detection result.

[0026] In addition, the embodiment also provides a remote sensing video weak and small target detection system based on three-branch spatio-temporal modeling, which comprises a microprocessor and a memory connected with each other, and the microprocessor is programmed or configured to execute the optical flow driven remote sensing video multi-target tracking method. The embodiment also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is used for programming or configuring the microprocessor to execute the remote sensing video weak and small target detection multi-target tracking method based on three-branch spatio-temporal modeling.

[0027] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, various software modules in accordance with embodiments of the present application are stored in a memory such as a computer memory or disk storage for use by, or in connection with, the software on the computer system. The software can provide for programs to be transferred to another computer readable medium (e.g., a removable medium either in the form of computer disks or other hardware) for implementation of the various techniques taught herein. In a hardware embodiment, one or more application specific integrated circuits (ASICs) can be used. Broadly, the term "computer readable media" can be taken to include both storage devices such as conventional computer-readable storage media and communication media such as electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, etc.). The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks

[0028] While the present application has been described with reference to one or more particular embodiments, those skilled in the art will recognize that other changes can be made thereto without departing from the scope of the present application. It is intended to cover all such changes and modifications that fall within the scope of the application.

Claims

1. A method for detecting a weak and small target in a remote sensing video based on a three-branch spatio-temporal modeling, characterized in that, The method comprises the following steps: Step one: obtain a remote sensing video image sequence, format, crop and normalize the image sequence to obtain an image input sequence with a size of , wherein is a preset frame number, is an image height, is an image width, t is a frame number corresponding to a current time, and is a current frame;​ Step two: input the image input sequence into the three-branch spatio-temporal encoder, respectively pass through the time branch, the space branch and the space-time branch to model the features, extract the time domain feature, the space domain feature and the space-time joint feature information of the image input sequence, and splice and fuse the output results of the three branches into the time sequence enhanced sequence where B is the batch size and C is the number of channels. Step three: sequentially inputting the time-enhanced feature sequence into the static feature extraction network to extract static image features, and extracting a unified feature sequence after spatio-temporal fusion , U t denotes the unified feature corresponding to the t-th frame; Step four: input the unified feature sequence into the space-time decoding module, restore the motion structure features of each frame by using multiple stacked 3D deconvolution and three-dimensional pooling strategy, and output the final space-time detection feature map corresponding to the current frame ;​ Step five: input the fusion feature into the detection head, combine Top-k screening and non-maximum suppression for post-processing operation, and output the weak and small moving target detection result of the current frame, wherein the weak and small moving target detection result comprises target category, position and confidence.

2. The method according to claim 1, wherein, The three-branch spatio-temporal encoder in step two comprises: S2.1: time branch module: used for modeling the time dynamic information of the image sequence, a three-dimensional convolution operation with a shape of 3x1x1 is adopted to extract the inter-frame motion pattern along the time dimension, and the calculation process is as follows: ； wherein, denotes the feature results extracted by the temporal branch module, denotes the layer temporal convolution operation, I denotes the image input sequence; S2.2: spatial branch module: used for extracting the spatial structure information of each frame, a three-dimensional convolution with a shape of 1x3x3 is adopted to model the space of each frame, and the calculation formula is as follows: ； wherein, represents the feature results extracted by the spatial branch module, represents the first layer of spatial convolution operation; S2.3: spatio-temporal branch module: used for jointly modeling the spatial and temporal variation law, a plurality of three-dimensional convolutions with a shape of 3x3x3 are adopted to capture complex spatio-temporal interaction features, and the calculation is as follows: ； wherein, denotes the feature results extracted by the spatio-temporal branch module, denotes the first layer spatio-temporal convolution operation; S2.4: features output by the above three branches Concatenate in the channel dimension to get the fused feature: ； Wherein, Concat() represents a feature stacking operation; S2.5: the fusion feature is compressed in channel through a 1x1x1 three-dimensional convolution to obtain the final encoding output: ； wherein, denotes a 1 x 1 x 1 three-dimensional convolution operation.

3. The method according to claim 2, wherein, The spatio-temporal decoding module in step four comprises: S4.1: input is a sequence of unified features as encoder output recovering the temporal dimension and the spatial dimension jointly by a stacked three-dimensional deconvolution operation, specifically: ； ； ； wherein, represents the first layer three-dimensional transpose convolution operation, used to gradually enhance the motion features and semantic information; S4.2: Up-sampled features A pooling operation along the time dimension aggregates the multi-frame information into a motion-aware feature representation for the current frame: ； wherein, is a three-dimensional max pooling operation, compressed along the time dimension, retaining significant motion response features; As the input of the final detection module, it is used to generate the class heat map, size prediction map and center offset map of the target, so as to realize high-precision moving target detection of the current frame.

4. A remote sensing video weak and small target detection system based on three-branch space-time modeling, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to perform the remote sensing video weak and small target detection method based on three-branch spatio-temporal modeling according to any one of claims 1-3.

5. A computer-readable storage medium having stored therein a computer program, characterized in that, The computer program is used for being programmed or configured by the microprocessor to perform the remote sensing video weak and small target detection method based on three-branch spatio-temporal modeling according to any one of claims 1-3.

Images