Video Small Moving Target Detection Methods and Related Equipment
By performing channel expansion on the first convolutional layer of the YOLO network and stacking and stitching multiple frames of images, motion information between video frames is captured, solving the problem of insufficient representation of small moving target features in the single-frame YOLO detection method, and achieving efficient target detection with low computational cost.
Patent Information
- Application Number
- CN202611584721.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-10
AI Technical Summary
Existing single-frame YOLO target detection methods cannot effectively utilize the temporal motion information between video frames, resulting in a high rate of missed detections and false detections for small moving targets.
By expanding the channels of the first convolutional layer of the pre-trained YOLO network and stacking K consecutive frames of images along the channel dimension, the multi-frame stacked input tensor is used for initialization and weakly supervised training to capture pixel-level motion differences between frames and fuse temporal motion information.
Without increasing computational load, it significantly reduces the false negative and false positive rates for small moving targets, meets real-time detection requirements, and is compatible with all mainstream YOLO variants.
Smart Images

Figure CN123071918A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, and in particular to a method and related equipment for detecting small moving targets in videos. Background Technology
[0002] Target detection technology is widely used in security monitoring, drone collision avoidance, and autonomous driving. Among existing technologies, the YOLO series of algorithms uses a single frame of RGB image as input and completes target detection through a backbone network, feature fusion network, and detection head. It has the advantages of fast detection speed and convenient deployment, and is currently the mainstream real-time target detection algorithm.
[0003] However, for small moving targets with small pixel areas and displacement between consecutive frames in a video, single-frame YOLO relies only on the static spatial features of a single frame for detection and cannot utilize the temporal motion information between video frames. This results in insufficient feature representation of small moving targets and a high false negative rate. Summary of the Invention
[0004] This invention provides a method and related equipment for detecting small moving targets in video, in order to solve the technical problem that existing single-frame YOLO target detection methods cannot utilize the temporal motion information between video frames, resulting in a high rate of missed detection and false detection of small moving targets.
[0005] In a first aspect, embodiments of this application provide a method for detecting small moving targets in a video, including: Stack and stitch K consecutive frames of images from a historical video stream along the channel dimension to obtain a multi-frame stacked input tensor. Based on the number of channels in the multi-frame stacked input tensor, the number of input channels in the first convolutional layer of the pre-trained YOLO network is expanded while keeping the structure of the remaining layers unchanged, resulting in a network model with expanded channels. The original pre-trained weights of the first convolutional layer of the pre-trained YOLO network are copied K times along the channel dimension to obtain expanded weights. The expanded weights are then averaged and loaded into the first convolutional layer of the network model with expanded channels to obtain the initialized object detection network. Using the target annotation information of the last frame in the multi-frame stacked input tensor as a supervision signal, the initial target detection network is trained to obtain a trained target detection network. K consecutive frames of images in the video stream to be detected are stacked and stitched along the channel dimension to obtain the stacked input tensor of the multi-frame to be detected. The stacked input tensor of the multi-frames to be detected is input into the trained target detection network, and moving target detection is performed on the last frame in the stacked input tensor of the multi-frames to be detected to obtain the target detection result of the last frame.
[0006] Secondly, embodiments of this application provide a video small moving target detection device, comprising: The data stacking module is used to stack and stitch K consecutive frames of images in a historical video stream along the channel dimension to obtain a multi-frame stacked input tensor. The network initialization module is used to expand the number of input channels of the first convolutional layer of the pre-trained YOLO network according to the number of channels of the multi-frame stacked input tensor, while keeping the structure of the remaining network layers of the pre-trained YOLO network unchanged, to obtain a network model with expanded channels; to copy the original pre-trained weights of the first convolutional layer of the pre-trained YOLO network K times in the channel dimension to obtain expanded weights; to perform mean processing on the expanded weights, and to load the mean-processed expanded weights into the first convolutional layer of the network model with expanded channels, to obtain the initialized object detection network; The weakly supervised training module is used to train the initial target detection network by using the target annotation information of the last frame in the multi-frame stacked input tensor as a supervision signal, so as to obtain the trained target detection network. The inference input module is used to stack and stitch K consecutive frames of images in the video stream to be detected along the channel dimension to obtain a stacked input tensor of multiple frames to be detected. The target detection module is used to input the stacked input tensor of the multi-frames to be detected into the trained target detection network, perform moving target detection on the last frame in the stacked input tensor of the multi-frames to be detected, and obtain the target detection result of the last frame.
[0007] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described video small moving target detection method.
[0008] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described video small moving target detection method.
[0009] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, enables the implementation of the steps of the above-described video small moving target detection method.
[0010] In one of the solutions provided by the aforementioned video small moving target detection methods, devices, equipment, media, and program products, K consecutive frames of images are stacked along the channel dimension as input to a YOLO network. This allows the first convolutional layer to capture pixel-level motion differences between adjacent frames while perceiving the spatial features of a single frame. Thus, without introducing additional temporal modules, the temporal motion information between video frames is integrated into the feature extraction process, effectively compensating for the deficiency of single-frame input in representing small moving target features and reducing the false negative and false positive rates. Furthermore, this invention only expands the channels of the first convolutional layer of the pre-trained YOLO network, keeping the rest of the network structure unchanged. The increase in computation is minimal, and the inference speed is essentially unaffected, meeting the requirements of real-time video detection. It is also compatible with various mainstream YOLO variants, exhibiting strong versatility. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a schematic diagram of a video small moving target detection system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a video small-moving target detection method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S10; Figure 4 yes Figure 2 A schematic diagram of the implementation process of step S20; Figure 5 yes Figure 2 A schematic diagram of the implementation process of step S30; Figure 6 This is a schematic diagram of a video small moving target detection device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation
[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0014] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0015] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0016] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0017] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0018] To illustrate the technical solution of the present invention, specific embodiments are described below.
[0019] To address the problems mentioned above in the background art, this application provides a method, apparatus, device, medium, and program product for detecting small moving targets in videos. The video small moving target detection method provided by this invention can be applied to applications such as... Figure 1 The video small moving target detection system shown includes a video acquisition device, a data preprocessing module, a target detection server, and a display terminal.
[0020] In one embodiment, such as Figure 2 As shown, a method for detecting small moving targets in videos is provided, which is then applied to... Figure 1 Taking a video small moving target detection system as an example, the following steps are included: S10: Stack and stitch K consecutive frames of images in the historical video stream along the channel dimension to obtain a multi-frame stacked input tensor; S20: Based on the number of channels in the multi-frame stacked input tensor, expand the number of input channels in the first convolutional layer of the pre-trained YOLO network while keeping the structure of the remaining network layers unchanged to obtain a network model with expanded channels; copy the original pre-trained weights of the first convolutional layer of the pre-trained YOLO network K times in the channel dimension to obtain expanded weights; perform mean averaging on the expanded weights and load the averaged expanded weights into the first convolutional layer of the network model with expanded channels to obtain the initialized object detection network; S30: Using the target annotation information of the last frame in the multi-frame stacked input tensor as a supervision signal, train the initial target detection network to obtain the trained target detection network. S40: Stack and stitch K consecutive frames of images in the video stream to be detected along the channel dimension to obtain the stacked input tensor of the multi-frame to be detected; S50: Input the stacked input tensor of the multi-frames to be detected into the trained target detection network, perform moving target detection on the last frame in the stacked input tensor of the multi-frames to be detected, and obtain the target detection result of the last frame.
[0021] In this embodiment, the overall process of the video small moving target detection method is divided into two parts: a training phase and an inference phase. The training phase sequentially completes the preparation of multi-frame data, the adaptation and initialization of the network structure, and weakly supervised training. The inference phase processes multi-frame data from the video stream to be detected in the same way and inputs it into the trained network to complete the detection output of the small moving target. The two phases maintain consistency in data processing methods, together forming a complete detection scheme.
[0022] During the training phase, small moving targets in videos typically have small pixel areas and exhibit displacement between consecutive frames, making it difficult to effectively distinguish them using only the static spatial features of a single frame. Therefore, this embodiment introduces a multi-frame channel stacking mechanism during the data preparation phase, concatenating K consecutive frames from the historical video stream along the channel dimension into a single input tensor. Specifically, K consecutive frames are extracted from the historical video stream in chronological order, with the latest frame as the last frame. The K frames are then arranged and concatenated sequentially along the channel dimension to obtain a multi-frame stacked input tensor. It should be noted that the concatenation order of the K frames along the channel dimension maintains the same chronological order, i.e., earlier frames correspond to earlier channel positions, and the last frame corresponds to the last channel position. This arrangement allows the first convolutional layer of the subsequent network to capture the temporal changes in pixels between frames while simultaneously perceiving the spatial features of each frame.
[0023] Based on the above explanation, after constructing the multi-frame stacked input tensor, the pre-trained YOLO network needs to be adapted to accept the multi-frame stacked input tensor as input. The first convolutional layer of the pre-trained YOLO network originally used the number of channels of a single frame image as the input channel count, which does not match the channel count of the multi-frame stacked input tensor. Therefore, the input channel count of the first convolutional layer needs to be expanded. Furthermore, the dimensions of the expanded first convolutional layer weight parameters are no longer consistent with the original pre-trained weights. If the expanded weights are directly and randomly initialized, it will cause a significant shift in the distribution of activation values in each layer of the network during the initial training phase, affecting training stability. Therefore, this embodiment performs a special initialization process on the pre-trained weights of the first convolutional layer, obtaining an initialized object detection network while keeping the structure and weights of the remaining layers of the network completely unchanged.
[0024] After network initialization, this embodiment trains the initial object detection network using a weakly supervised approach. Specifically, a multi-frame stacked input tensor is input into the initial object detection network. The network outputs the predicted detection result for the last frame. The target annotation information of the last frame is used as the sole supervision signal to calculate the loss, and the network parameters are iteratively updated through backpropagation until the network parameters converge, resulting in a trained object detection network. It should be noted that this embodiment only provides annotation supervision for the last frame; no additional annotation is required for the first K-1 frames. This design allows the network to automatically learn to utilize multi-frame temporal information to assist in target detection in the last frame during training without increasing annotation costs.
[0025] During the inference phase, K consecutive frames in the video stream to be detected are processed using the same channel stacking method as in the training phase to obtain a stacked input tensor of the multi-frames to be detected. This tensor is then input into the trained object detection network. The network performs moving object detection on the last frame of the stacked input tensor and outputs the object detection result for that last frame. For example, in a security monitoring scenario, K consecutive frames in a real-time video stream are stacked and input into the network. The network outputs the bounding boxes and category information of small moving targets such as drones and birds in the current frame, completing real-time detection.
[0026] In one embodiment, such as Figure 3 As shown, step S10 specifically includes the following steps: S11: Extract K consecutive RGB images from the historical video stream in chronological order, with the labeled frame as the last frame, to obtain a K-frame image sequence; S12: Sequentially stitch together each frame of the K-frame image sequence along the channel dimension to obtain a multi-frame stacked input tensor with 3K channels and spatial dimensions consistent with a single frame image.
[0027] In this embodiment, each frame in the historical video stream is an RGB three-channel image, containing the spatial appearance information of the target at that moment. Processing a single frame only yields the static spatial features of the target at that moment, but cannot perceive the target's displacement changes in the time dimension. Therefore, in the data preparation stage, this embodiment uses the labeled frame as the last frame and traces back K consecutive frames, organizing the K frames into an ordered sequence as the data source for subsequent channel stacking.
[0028] Specifically, K RGB images are extracted sequentially from the historical video stream, with the earliest frame at the beginning of the sequence and the labeled frame at the end, resulting in a K-frame image sequence. It should be noted that the frame intervals between frames in the K-frame image sequence are maintained as in the original video. The displacement of the target between adjacent frames is directly related to the original video frame rate; a higher frame rate results in smaller target displacement between adjacent frames, and a lower frame rate results in larger displacement. In practical applications, the value of K can be selected based on the matching between the target's motion speed and the video frame rate.
[0029] Based on this, the images in the K-frame image sequence are sequentially stitched along the channel dimension. Since each image is an RGB three-channel image, the total number of channels after stitching the K-frame images along the channel dimension is 3K, while the spatial size of each image remains unchanged during the stitching process, resulting in a multi-frame stacked input tensor with 3K channels and the same spatial size as a single frame image. In conjunction with the above explanation, the channel stitching order is consistent with the chronological order; that is, the three channels of the earliest frame are located in the first three channel positions of the multi-frame stacked input tensor, and the three channels of the last frame are located in the last three channel positions.
[0030] For example, with K equal to 3, three RGB images corresponding to times t-2, t-1, and t are extracted from the historical video stream and labeled as the first frame, second frame, and last frame, respectively. These three frames are then concatenated sequentially along the channel dimension in chronological order to obtain a multi-frame stacked input tensor with 9 channels and the same spatial dimensions as a single frame image. Specifically, the first frame corresponds to channels 1 to 3, the second frame to channels 4 to 6, and the last frame to channels 7 to 9, with the channel order perfectly corresponding to the chronological order of the frames.
[0031] In one embodiment, such as Figure 4 As shown, step S20 specifically includes the following steps: S21: Based on the number of channels of the multi-frame stacked input tensor, expand the number of input channels of the first convolutional layer of the pre-trained YOLO network, while keeping the structure of the remaining network layers of the pre-trained YOLO network unchanged, to obtain the network model after channel expansion. S22: Obtain the original pre-trained weight tensor of the first layer convolution of the pre-trained YOLO network. The dimensions of the original pre-trained weight tensor correspond to the number of output channels, the number of input channels, the height of the convolution kernel, and the width of the convolution kernel in sequence. S23: The original weight tensor is repeatedly concatenated K times in the input channel dimension to obtain an extended weight tensor with the number of input channels being K times that of the original weight tensor, and the remaining dimensions being consistent with the original weight tensor; S24: The extended weights are averaged, and the averaged extended weights are loaded into the first convolutional layer of the network model after channel expansion to obtain the initialized target detection network.
[0032] In this embodiment, the pre-trained YOLO network is trained under a single-frame image input scenario, and the number of input channels of its first convolutional layer matches the number of channels of a single-frame RGB image. After introducing multi-frame channel stacking, the number of channels of the input tensor expands from the number of channels in a single frame to the sum of the number of channels in K frames, which is no longer consistent with the number of input channels of the original first convolutional layer. Therefore, the number of input channels of the first convolutional layer needs to be expanded accordingly. It should be noted that this embodiment only modifies the number of input channels of the first convolutional layer. The remaining layers of the network, including the backbone network, feature fusion network, and all layers of the detection head, remain completely consistent with the original pre-trained YOLO network without any changes, resulting in the network model with expanded channels. This design allows the pre-trained weights to be directly reused in most network layers, avoiding the large computational overhead of training from scratch.
[0033] Based on the above explanation, the number of input channels in the first convolutional layer of the expanded network model matches the number of channels in the multi-frame stacked input tensor. However, the weight parameters of the first convolutional layer have not yet been initialized. If the expanded weight parameters are randomly initialized directly, the activation value distribution of each layer in the early stages of training will deviate significantly from the pre-training state, leading to gradient instability and a significantly slower convergence speed. Therefore, this embodiment copies the original pre-training weights to construct the initial weights required for the expanded first convolutional layer.
[0034] Specifically, the dimensions of the original pre-trained weight tensor correspond to the number of output channels, the number of input channels, the kernel height, and the kernel width, respectively. This weight tensor is then repeatedly concatenated K times along the input channel dimension, resulting in expanded weights with the number of input channels increased to K times the original weight's input channels, while the other dimensions remain unchanged. If these expanded weights, after being copied K times, are directly loaded into the network model with expanded channels, the response amplitude of the first convolutional layer to the stacked input tensor across multiple frames will be K times that of the original single-frame case. This leads to an overall larger activation value in subsequent layers, which also negatively impacts training stability.
[0035] To eliminate the aforementioned amplitude bias, this embodiment performs averaging on the expanded weights. This involves dividing each weight element in the expanded weight tensor by K, scaling the weight value at each channel position to one-Kth of the original weight value. After averaging, the response amplitude of the multi-frame stacked input tensor after the first convolution layer remains consistent with the original single-frame case, allowing the activation value distribution of each layer of the network to continue from the pre-training state. Then, the weight values in the averaged expanded weight tensor are assigned one by one to the weight parameter positions of the first convolution layer of the expanded network model according to the correspondence between the output channel dimension and the input channel dimension. This ensures that the weight parameters of the first convolution layer completely correspond to the number of expanded input channels, resulting in the initialized object detection network.
[0036] For example, when K is 3, the original pre-trained weight tensor is copied 3 times along the input channel dimension, expanding the number of input channels to 3 times the original. Then, each element in the expanded weights is divided by 3 to achieve mean distribution, and finally loaded into the first convolutional layer of the network model after channel expansion, resulting in the initialized object detection network. After the above processing, the activation value distribution of the initialized object detection network at the beginning of training is basically consistent with that of the original pre-trained YOLO network, which is beneficial to the stable progress of subsequent training.
[0037] In this embodiment, the original pre-trained weight tensor is the first-layer convolutional weight obtained by the pre-trained YOLO network through large-scale data training in a single-frame image input scenario. It contains the network's ability to extract low-level features from the single-frame image. This weight tensor has a four-dimensional structure, with the four dimensions corresponding to the number of output channels, the number of input channels, the kernel height, and the kernel width, respectively. The number of input channels is consistent with the number of channels in a single-frame RGB image. It should be noted that before performing the weight copying operation, this four-dimensional weight tensor must be fully obtained to ensure that the subsequent copying operation is performed on the correct tensor structure, avoiding incorrect copying direction due to misunderstanding of the dimension order.
[0038] Based on the above explanation, after obtaining the original weight tensor, it is repeatedly concatenated K times along the input channel dimension. Specifically, the original weight tensor is concatenated K times along the input channel dimension, with the complete original weight tensor appended to the end of the input channel dimension of the concatenated result each time. After concatenation, the number of input channels becomes K times the number of input channels of the original weight tensor, while the values of the number of output channels, kernel height, and kernel width remain completely consistent with the original weight tensor, resulting in an expanded weight tensor.
[0039] Building upon this, the expanded weight tensor is constructed by sequentially arranging K identical original weight tensors along the input channel dimension. Each tensor corresponds to a channel position in one frame of the stacked input tensor. For example, when K is 3, the original weight tensors are concatenated three times along the input channel dimension to obtain an expanded weight tensor with three times the number of input channels as the original weight tensor. The first weight corresponds to the channel position in the first frame of the stacked input tensor, the second weight corresponds to the channel position in the second frame, and the third weight corresponds to the channel position in the last frame. The values of the three weights are identical during the initialization phase and will be loaded into the expanded network model after averaging.
[0040] In one embodiment, the step of averaging the expanded weights and loading the averaged expanded weights into the first convolutional layer of the channel-expanded network model to obtain the initialized object detection network includes: Divide each weight element in the extended weight tensor by K to obtain the mean-valued extended weight tensor. The weight values in the mean-normalized expanded weight tensor are assigned one by one to the weight parameter positions of the first convolutional layer of the expanded network model according to the correspondence between the output channel dimension and the input channel dimension, so that the weight parameters of the first convolutional layer completely correspond to the number of expanded input channels, thus obtaining the initialized object detection network.
[0041] In this embodiment, the expanded weight tensor is obtained by repeatedly concatenating the original weight tensor K times along the input channel dimension, where the value of each weight element is exactly the same as the value at the corresponding position in the original weight tensor. If this expanded weight tensor is directly loaded into the first convolutional layer of the network model after channel expansion, the output response amplitude of the multi-frame stacked input tensor after the first convolutional layer will be K times that of the original single-frame case. This is because the expanded first convolutional layer performs convolution operations on the channels of K frames of images simultaneously, and the response values of each frame are superimposed and accumulated at the output, resulting in an overall larger activation value. This amplitude deviation will be propagated to subsequent layers, causing the activation value distribution of each layer of the network to deviate from the pre-training state, which has an adverse effect on the stability in the early stages of training.
[0042] To eliminate the aforementioned amplitude bias, this embodiment divides each weight element in the expanded weight tensor by K, scaling the value of each weight element to one-Kth of its original value, resulting in a mean-scaled expanded weight tensor. After mean-scale processing, the output response amplitude of the multi-frame stacked input tensor after the first convolution layer remains consistent with that of the original single-frame case, allowing the activation value distribution of subsequent layers in the network to continue from the pre-training state. It should be noted that the mean-scale operation applies to all weight elements of the expanded weight tensor, regardless of whether the element is located in the output or input channel, ensuring that the weight values at each channel position are scaled uniformly.
[0043] Based on the above explanation, after the mean-squared processing is completed, the mean-squared expanded weight tensor is loaded into the first convolutional layer of the channel-expanded network model. Specifically, according to the correspondence between the output channel dimension and the input channel dimension, the weight value at each position in the mean-squared expanded weight tensor is assigned one by one to the corresponding weight parameter position of the first convolutional layer of the channel-expanded network model. The assignment process is performed sequentially from the first output channel to the last output channel according to the output channel dimension, and within each output channel, the assignment is then performed sequentially from the first input channel to the last input channel according to the input channel dimension. This ensures that each weight element in the expanded weight tensor is accurately written into the corresponding weight parameter position of the first convolutional layer, so that the weight parameters of the first convolutional layer completely correspond to the number of expanded input channels, thus obtaining the initialized object detection network.
[0044] For example, when K is 3, each element in the expanded weight tensor is divided by 3 to achieve mean normalization. Then, the weights are assigned one by one to the weight parameters of the first convolutional layer of the expanded network model according to the correspondence between the output channel dimension and the input channel dimension. After the assignment, the weight values corresponding to the channel positions of the first, second, and last frames in the first convolutional layer are all the same, and the value of each weight element is one-third of the value of the corresponding position of the original pre-trained weight. The weight parameters of the remaining layers of the network are consistent with the original pre-trained YOLO network, resulting in the initialized object detection network.
[0045] In one embodiment, such as Figure 5 As shown, step S30 specifically includes the following steps: S31: Input the multi-frame stacked input tensor into the first convolutional layer of the initial target detection network, and perform spatiotemporal joint filtering on the pixel differences between the last frame and the previous K-1 frames in the multi-frame stacked input tensor to obtain a feature map that integrates inter-frame motion information. S32: Based on the feature map, detect the moving target in the last frame to obtain the prediction detection result for the last frame; S33: Based on the predicted detection result and the target annotation information of the last frame, calculate the loss between the two to obtain the loss value; S34: Update the network parameters of the initial target detection network by backpropagation according to the loss value, and re-input the multi-frame stacked input tensor and the target annotation information of the corresponding last frame into the initial target detection network after the network parameters are updated for iterative training until the network parameters converge, and obtain the trained target detection network.
[0046] In this embodiment, the entire training process sequentially goes through four stages: temporal feature extraction, target detection and prediction, loss calculation, and parameter iterative update. Each stage is sequentially connected and jointly drives the network parameters to be updated in the convergence direction.
[0047] In the temporal feature extraction stage, the multi-frame stacked input tensor is input into the first convolutional layer of the object detection network. Since the multi-frame stacked input tensor sequentially arranges the image data of the first K-1 frames and the last frame along the channel dimension, the convolutional kernel of the first layer naturally perceives the pixel values of the first K-1 frames and the last frame at corresponding spatial locations when performing weighted calculations on the pixel values at each channel position. For static background regions, the pixel values of the first K-1 frames and the last frame at the same spatial location are similar, and the pixel differences at corresponding channel positions are small, resulting in a relatively low response amplitude of the first convolutional layer for this region. For regions with moving targets, the target has shifted between frames, and there are significant differences in the pixel values of the first K-1 frames and the last frame at corresponding spatial locations, leading to a relatively high response amplitude of the first convolutional layer for this region. Through this spatiotemporal joint filtering mechanism, the first convolutional layer implicitly captures the pixel-level motion differences between the last frame and the first K-1 frames while extracting the spatial features of each frame, resulting in a feature map that incorporates inter-frame motion information. It should be noted that the extraction of the above-mentioned temporal motion information is automatically learned by the first layer convolutional kernel during the training process, without the need to introduce an additional optical flow estimation module or temporal modeling module, resulting in extremely low computational overhead.
[0048] Based on the above explanation, the feature map, which incorporates inter-frame motion information, is processed through the network backbone, feature fusion, and detection head layers, outputting the predicted detection result for the last frame. The predicted detection result includes the predicted bounding boxes of each candidate target in the last frame and their corresponding class confidence scores. Then, the predicted detection result is compared with the target annotation information of the last frame, and the difference between the two is calculated to obtain the loss value. Based on this, backpropagation is performed on all network parameters of the initialized target detection network according to the loss value, the gradient of each parameter is calculated, and the network parameters are updated according to the gradient direction.
[0049] After completing one parameter update, the next set of multi-frame stacked input tensors and the target annotation information of the corresponding last frame are extracted from the training data. The iterative process of forward inference, loss calculation, and backpropagation is repeated until the update magnitude of the network parameters is lower than the preset convergence threshold, resulting in a trained target detection network. For example, in the training data of a security monitoring scenario, for a video clip of a bird flying across the screen at a certain moment, a multi-frame stacked input tensor containing K frames before and after that moment is input into the network. The network perceives the displacement difference of the bird between frames through the first convolutional layer, calculates the loss and updates the parameters by combining the bounding box of the last frame. After multiple iterations, the network gradually learns to accurately locate small moving targets in the last frame using inter-frame motion cues.
[0050] In one embodiment, the step of inputting the multi-frame stacked input tensor into the first convolutional layer of the initialization target detection network, and performing spatiotemporal joint filtering on the pixel differences between the last frame and the previous K-1 frames in the multi-frame stacked input tensor to obtain a feature map that incorporates inter-frame motion information includes: Based on the number of channels in the multi-frame stacked input tensor, the input channels of the first convolutional layer are divided into K channel groups. Each channel group corresponds to the input channel of one frame in the multi-frame stacked input tensor, where the last frame corresponds to the Kth channel group, and the first K-1 frames correspond to the first to the K-1th channel groups respectively. Convolutional operations are performed on the input channels of each channel group using convolutional kernels corresponding to each channel group. Each convolutional kernel perceives the pixel spatial distribution of the corresponding frame to obtain K groups of intermediate feature maps. The number of channels in each group of intermediate feature maps is equal to the number of output channels of the first convolutional layer. The weights of the convolutional kernels corresponding to each channel group are independent of each other and are updated separately during training. The K groups of intermediate feature maps are summed element by element according to their corresponding channels. The difference between the pixel differences between the last frame and the previous K-1 frames is reflected by the difference between the convolutional kernel weights corresponding to each channel group, thus obtaining a feature map that integrates inter-frame motion information. The number of channels in the feature map is equal to the number of output channels of the first convolutional layer.
[0051] In this embodiment, the convolution kernel in the standard convolution operation operates simultaneously on all input channels of the input tensor. The convolution results of each channel are summed at the output to obtain a single output feature value. In scenarios with stacked input tensors across multiple frames, the weight allocation of the first convolution layer for each frame channel is automatically learned and determined by the network during training. To facilitate the explanation of the role of each frame's information in the first convolution layer, this embodiment divides the input channels of the first convolution layer by the number of frames K, so that each channel group corresponds to one frame of image. The convolution kernel corresponding to each channel group is specifically responsible for the convolution operation of that frame's channel. The operation results of each frame are then fused according to the corresponding output channel, thereby structurally supporting the independent perception of information in each frame. It should be noted that the above-mentioned frame-by-frame division and element-by-element summation are mathematically equivalent to performing a standard convolution on all input channels uniformly. The number of output channels of the first convolution layer remains unchanged, therefore the structure of the remaining layers of the network does not require any modification.
[0052] Specifically, based on the number of channels in the multi-frame stacked input tensor, the input channels of the first convolutional layer of the initialization object detection network are evenly divided into K channel groups. The number of input channels in each channel group is equal to the number of channels in a single frame image. The division is performed according to the channel arrangement order of each frame in the multi-frame stacked input tensor. The channels of the first K-1 frames correspond to the first to the K-1th channel groups, and the channels of the last frame correspond to the Kth channel group, resulting in K channel groups. Each channel group corresponds to a set of convolutional kernel weights, whose values are taken as the corresponding part of the expanded weights after the aforementioned mean-based processing during the initialization phase. Therefore, the convolutional kernel weights of each channel group are the same during initialization; however, the convolutional kernel weights of each channel group are independent parameters and are updated separately during training. Therefore, as training progresses, differences gradually emerge between the convolutional kernel weights of each channel group.
[0053] Based on this, convolutional kernels corresponding to each channel group are used to perform convolution operations on the input channels within each channel group. For each channel group corresponding to the first K-1 frames, its convolutional kernel extracts the pixel spatial distribution features of that frame on the corresponding frame's channels; for the Kth channel group corresponding to the last frame, its convolutional kernel extracts the pixel spatial distribution features of the last frame on the last frame's channels. The convolutional operation of each channel group yields a set of intermediate feature maps, and the number of channels in each set of intermediate feature maps is equal to the number of output channels of the first convolutional layer.
[0054] Subsequently, the K groups of intermediate feature maps are summed element-wise according to their corresponding channels to obtain a feature map that incorporates inter-frame motion information. The number of channels in this feature map is equal to the number of output channels of the first convolutional layer. It should be noted that during the initialization phase, since the convolutional kernel weights of each channel group are the same, the summation result is equivalent to performing a single-frame convolution on the average image of the K frames. As training progresses, the convolutional kernel weights of the channel group corresponding to the last frame differ from those of the channel groups corresponding to the previous K-1 frames. The contribution of each frame during summation is no longer equal. At this point, the summation result reflects the pixel differences between the last frame and the previous K-1 frames, thus achieving spatiotemporal joint filtering of the pixel differences between the last frame and the previous K-1 frames: spatially, the convolutional kernels of each channel group filter the spatial distribution of pixels in the corresponding frame; temporally, the weighted summation of the results from each channel group filters the inter-frame pixel differences. The acquisition of the aforementioned temporal motion information is entirely automatically learned by the first convolutional kernel during training, without the need for an additional optical flow estimation module or temporal modeling module.
[0055] For example, when K is 3, the input channels of the multi-frame stacked input tensor are divided into three channel groups: the first channel group corresponds to the channels of the first frame, the second channel group corresponds to the channels of the second frame, and the third channel group corresponds to the channels of the last frame. The convolutional kernels corresponding to the three channel groups perform convolution operations on the channels of their respective frames, resulting in three sets of intermediate feature maps. The number of channels in each set of intermediate feature maps is equal to the number of output channels of the first convolutional layer. The three sets of intermediate feature maps are then summed element-wise according to their corresponding channels to obtain a fused feature map that simultaneously contains spatial features of the three frames and inter-frame motion difference information. The number of channels in this fused feature map is still equal to the number of output channels of the first convolutional layer, which is then used for further processing by subsequent network layers.
[0056] In this embodiment, the step of calculating the loss value based on the difference between the predicted detection result and the target annotation information of the last frame includes: Based on the predicted bounding box in the predicted detection result and the labeled bounding box in the target annotation information of the last frame, the ratio of the intersection region to the union region between the two is calculated to obtain the intersection-union ratio. Based on the center point coordinates of the predicted bounding box and the labeled bounding box, the ratio of the center point distance between the two to the diagonal distance of the minimum bounding box is calculated to obtain the center point distance penalty term; Based on the aspect ratio of the predicted bounding box and the labeled bounding box, the consistency of the aspect ratio between the two is calculated to obtain the aspect ratio penalty term; The loss value is obtained by weighted summation of the intersection-union ratio, the center point distance penalty term, and the aspect ratio penalty term.
[0057] In this embodiment, the pixel area of small moving targets is typically small, and the positional, dimensional, and shape deviations between the predicted and labeled bounding boxes are highly sensitive to the final detection accuracy. If only the intersection-union ratio (IU) of the predicted and labeled bounding boxes is used as the loss criterion, the IU will be zero when there is no overlap, resulting in gradient vanishing and the network being unable to obtain effective parameter update signals from the sample. Therefore, this embodiment employs a loss calculation method that simultaneously considers three dimensions: overlap, center point distance, and aspect ratio consistency, to comprehensively measure the differences between the predicted and labeled bounding boxes.
[0058] Specifically, the overlap between the predicted bounding box and the labeled bounding box is calculated. The areas of the intersection and union regions of the two bounding boxes on the image plane are extracted, and the intersection-union ratio is obtained by dividing the intersection area by the union area. The closer the intersection-union ratio is to 1, the higher the degree of overlap between the predicted and labeled bounding boxes, and the closer their positions and sizes are; the closer the intersection-union ratio is to 0, the lower the degree of overlap between the two.
[0059] Based on this, relying solely on the intersection-union ratio (IUR) cannot fully reflect the spatial deviation between two bounding boxes, especially when they do not overlap, as the IUR is always zero, failing to distinguish between different degrees of positional deviation. Therefore, this embodiment introduces a center point distance penalty term. The coordinates of the center points of both the predicted and labeled bounding boxes are extracted, the Euclidean distance between the two center points is calculated, and this distance is normalized using the diagonal length of the minimum bounding box of the two bounding boxes to obtain the center point distance penalty term. It should be noted that using the diagonal length of the minimum bounding box as the normalization benchmark ensures that the numerical range of the center point distance penalty term remains relatively consistent with the image scale, avoiding excessive fluctuations in the penalty term value due to differences in image resolution.
[0060] Based on the above explanation, the center point distance and overlap jointly describe the positional differences between the predicted and labeled bounding boxes, but neither constrains the shape deviation of the bounding boxes. For small moving targets, the aspect ratio of the target often exhibits certain regularity, and the consistency of the aspect ratio between the predicted and labeled bounding boxes directly affects the degree to which the detection box fits the target contour. Therefore, this embodiment further introduces an aspect ratio penalty term, extracting the width and height of the predicted and labeled bounding boxes respectively, calculating the degree of difference between their aspect ratios, and obtaining the aspect ratio penalty term.
[0061] Then, the intersection-union ratio (IUGR), center point distance penalty, and aspect ratio penalty are weighted and summed according to their respective weight coefficients to obtain the final loss value. For example, in a certain training sample, the predicted bounding box and the labeled bounding box have a certain center point offset and a slight difference in aspect ratio. The IUGR reflects the degree of overlap between the two, the center point distance penalty applies an additional penalty to the center point offset, and the aspect ratio penalty imposes a constraint on the shape difference. The loss value obtained by weighting and summing the three factors comprehensively reflects the overall difference between the predicted bounding box and the labeled bounding box in terms of position, size, and shape.
[0062] In this embodiment, the step of backpropagating to update the network parameters of the initialized target detection network based on the loss value, and then re-inputting the multi-frame stacked input tensor and the target annotation information of the corresponding last frame into the initialized target detection network after the network parameters have been updated for iterative training until the network parameters converge to obtain the trained target detection network, includes: Backpropagation is performed on the network parameters of each layer of the initialized target detection network based on the loss value to obtain the gradient corresponding to each layer of network parameters; Based on the gradients corresponding to the network parameters of each layer, the first-order moment estimates and second-order moment estimates of each layer are updated by moving average, resulting in updated first-order moment estimates and second-order moment estimates. The updated first-order moment estimate is divided by the square root of the updated second-order moment estimate to obtain the adaptive learning rate corresponding to the network parameters of each layer. The network parameters of each layer are updated according to the adaptive learning rate, and the updated network parameters are subject to weight decay to obtain the initialized target detection network with updated network parameters. The initial target detection network after re-inputting the multi-frame stacked input tensor and the target annotation information of the corresponding last frame into the network parameters is iteratively trained until the network parameters converge, thus obtaining the trained target detection network.
[0063] In this embodiment, the network parameter update process employs an adaptive learning rate optimization method, independently adjusting the update step size for different network parameters to address situations where the gradient magnitudes of parameters in different layers differ significantly. In small moving target detection scenarios, the parameters of the first convolutional layer face a new feature learning task after introducing multi-frame stacked input, and their gradient distribution differs somewhat from the gradient distribution of parameters in deeper layers of the network. A fixed learning rate is insufficient to simultaneously meet the update needs of parameters in each layer, while the adaptive learning rate mechanism can dynamically adjust the update step size based on the historical gradient information of parameters in each layer, which is beneficial for the stable progress of the training process.
[0064] Specifically, after obtaining the loss value of the current iteration, the partial derivatives of the loss value with respect to the network parameters of each layer are calculated layer by layer from the network output layer to the input layer to obtain the gradients corresponding to the network parameters of each layer. The gradient calculation is propagated forward layer by layer according to the chain rule. The gradient of each layer is obtained by multiplying the local derivative of the output of that layer with respect to the gradient of the next layer with the gradient of the next layer, until the gradients of all network layer parameters are calculated.
[0065] After obtaining the gradients corresponding to the network parameters of each layer, two statistics are maintained for each network parameter: a first-order moment estimate and a second-order moment estimate. The first-order moment estimate is an exponentially weighted moving average of historical gradients, reflecting the overall trend of the gradient's direction and magnitude; the second-order moment estimate is an exponentially weighted moving average of the squares of historical gradients, reflecting the degree of fluctuation in the gradient magnitude. In each iteration, the first-order moment estimate and the second-order moment estimate are updated by moving averages based on the current gradient. During the update, the historical statistics and the current gradient information are weighted and fused using a preset decay coefficient to obtain the updated first-order moment estimate and second-order moment estimate. It should be noted that in the early stages of training, due to insufficient accumulation of historical statistics, the first-order moment estimate and the second-order moment estimate have an initialization bias towards zero values. In actual use, it is necessary to correct the bias for both to eliminate the impact of the initially small statistics on parameter updates.
[0066] Based on the above explanation, after updating the first-order and second-order moment estimates, the adaptive learning rate for each network layer's parameters is obtained by dividing the first-order moment estimate by the square root of the second-order moment estimate. For parameters with large gradient magnitude fluctuations, the second-order moment estimate is larger, resulting in a smaller adaptive learning rate and suppressed update step size, thus avoiding excessive parameter updates. For parameters with relatively stable gradient magnitudes, the second-order moment estimate is smaller, resulting in a larger adaptive learning rate and relatively larger update step size, which is beneficial for rapid parameter convergence.
[0067] Based on this, according to the adaptive learning rate corresponding to the network parameters of each layer, the network parameters of each layer are updated in the opposite direction of the gradient. At the same time, a weight decay is applied to the updated network parameters, that is, after the parameter update, each parameter value is multiplied by a decay coefficient slightly less than 1 to suppress excessive growth of parameter values, thus obtaining the network parameters for the current iteration. Then, the next set of multi-frame stacked input tensors and the target annotation information of the corresponding last frame are extracted from the training data, and the above iterative process of forward inference, loss calculation, gradient backpropagation, and parameter update is repeated until the update magnitude of each layer's network parameters is lower than the preset convergence threshold, thus obtaining the trained object detection network.
[0068] For example, in a certain iteration, the parameters of the first convolutional layer generate a large gradient due to the multi-frame temporal feature learning task, and its second moment estimate increases accordingly. The adaptive learning rate is automatically narrowed to prevent the parameters of this layer from deviating from the stable range due to excessive update step size. Meanwhile, the gradient magnitude of the parameters of the deeper layers of the network is relatively stable, and the adaptive learning rate is maintained at a reasonable level to ensure the continuous and effective update of the parameters of the deeper layers. The synergistic effect of the two keeps the overall training process stable.
[0069] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0070] In one embodiment, a video small moving target detection device is provided, which corresponds one-to-one with the video small moving target detection methods described in the above embodiments. For example... Figure 6 As shown, the video small moving target detection device includes: The data stacking module 601 is used to stack and stitch K consecutive frames of images in a historical video stream along the channel dimension to obtain a multi-frame stacked input tensor. The network initialization module 602 is used to expand the channels of the first convolutional layer of the pre-trained YOLO network according to the number of channels of the multi-frame stacked input tensor, and to initialize the pre-trained weights of the first convolutional layer to obtain an initialized object detection network. The weakly supervised training module 603 is used to train the initial target detection network by using the target annotation information of the last frame in the multi-frame stacked input tensor as a supervision signal, so as to obtain a trained target detection network. The inference input module 604 is used to stack and stitch K consecutive frames of images in the video stream to be detected along the channel dimension to obtain a stacked input tensor of multiple frames to be detected. The target detection module 605 is used to input the stacked input tensor of the multi-frames to be detected into the trained target detection network, perform moving target detection on the last frame in the stacked input tensor of the multi-frames to be detected, and obtain the target detection result of the last frame.
[0071] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0073] This application also provides a computer device, such as... Figure 7 As shown, the computer device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments, or when the processor executes the computer program, it implements the functions of each module / unit in the above device embodiments.
[0074] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.
[0075] Those skilled in the art will understand that Figure 7 The computer device described is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.
[0076] The aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0077] The memory can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the computer device.
[0078] This application also provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.
[0079] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.
[0080] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0081] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0083] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0085] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for detecting small moving targets in video, characterized in that, The video small moving target detection method includes: Stack and stitch K consecutive frames of images from a historical video stream along the channel dimension to obtain a multi-frame stacked input tensor. Based on the number of channels in the multi-frame stacked input tensor, the number of input channels in the first convolutional layer of the pre-trained YOLO network is expanded while keeping the structure of the remaining layers unchanged, resulting in a network model with expanded channels. The original pre-trained weights of the first convolutional layer of the pre-trained YOLO network are copied K times along the channel dimension to obtain expanded weights. The expanded weights are then averaged and loaded into the first convolutional layer of the network model with expanded channels to obtain the initialized object detection network. Using the target annotation information of the last frame in the multi-frame stacked input tensor as a supervision signal, the initial target detection network is trained to obtain a trained target detection network. K consecutive frames of images in the video stream to be detected are stacked and stitched along the channel dimension to obtain the stacked input tensor of the multi-frame to be detected. The stacked input tensor of the multi-frames to be detected is input into the trained target detection network, and moving target detection is performed on the last frame in the stacked input tensor of the multi-frames to be detected to obtain the target detection result of the last frame.
2. The video small moving target detection method according to claim 1, characterized in that, The process of stacking and stitching K consecutive frames of images from a historical video stream along the channel dimension to obtain a multi-frame stacked input tensor includes: Extract K consecutive RGB images from the historical video stream in chronological order, with the labeled frame as the last frame, to obtain a K-frame image sequence; The images in the K-frame image sequence are sequentially stitched along the channel dimension to obtain a multi-frame stacked input tensor with 3K channels and the same spatial size as a single frame image.
3. The video small moving target detection method according to claim 1, characterized in that, The original pre-trained weights of the first convolutional layer of the pre-trained YOLO network are copied K times along the channel dimension to obtain expanded weights, including: Obtain the original pre-trained weight tensor of the first convolutional layer of the pre-trained YOLO network. The dimensions of the original pre-trained weight tensor correspond to the number of output channels, the number of input channels, the height of the convolutional kernel, and the width of the convolutional kernel in sequence. The original weight tensor is repeatedly concatenated K times along the input channel dimension to obtain an extended weight tensor with the number of input channels being K times that of the original weight tensor, while the remaining dimensions remain consistent with the original weight tensor.
4. The video small moving target detection method according to claim 3, characterized in that, The step of averaging the expanded weights and loading the averaged expanded weights into the first convolutional layer of the channel-expanded network model to obtain the initialized object detection network includes: Divide each weight element in the extended weight tensor by K to obtain the mean-valued extended weight tensor. The weight values in the mean-normalized expanded weight tensor are assigned one by one to the weight parameter positions of the first convolutional layer of the expanded network model according to the correspondence between the output channel dimension and the input channel dimension, so that the weight parameters of the first convolutional layer completely correspond to the number of expanded input channels, thus obtaining the initialized object detection network.
5. The video small moving target detection method according to claim 1, characterized in that, The step of using the target annotation information of the last frame in the multi-frame stacked input tensor as a supervision signal to train the initial target detection network to obtain a trained target detection network includes: The multi-frame stacked input tensor is input into the first convolutional layer of the initial target detection network. The first convolutional layer performs spatiotemporal joint filtering on the pixel differences between the last frame and the previous K-1 frames in the multi-frame stacked input tensor to obtain a feature map that incorporates inter-frame motion information. Based on the feature map, moving targets in the last frame are detected to obtain the predicted detection results for the last frame; Based on the predicted detection results and the target annotation information of the last frame, the difference between the two is calculated to obtain the loss value. The network parameters of the initial target detection network are updated by backpropagation based on the loss value. The multi-frame stacked input tensor and the target annotation information of the corresponding last frame are re-input into the updated initial target detection network for iterative training until the network parameters converge, thus obtaining the trained target detection network.
6. The video small moving target detection method according to claim 5, characterized in that, The step of inputting the multi-frame stacked input tensor into the first convolutional layer of the initialized target detection network, and performing spatiotemporal joint filtering on the pixel differences between the last frame and the previous K-1 frames in the multi-frame stacked input tensor to obtain a feature map that incorporates inter-frame motion information includes: Based on the number of channels in the multi-frame stacked input tensor, the input channels of the first convolutional layer are divided into K channel groups. Each channel group corresponds to the input channel of one frame in the multi-frame stacked input tensor, where the last frame corresponds to the Kth channel group, and the first K-1 frames correspond to the first to the K-1th channel groups respectively. Convolutional operations are performed on the input channels of each channel group using convolutional kernels corresponding to each channel group. Each convolutional kernel perceives the pixel spatial distribution of the corresponding frame to obtain K groups of intermediate feature maps. The number of channels in each group of intermediate feature maps is equal to the number of output channels of the first convolutional layer. The weights of the convolutional kernels corresponding to each channel group are independent of each other and are updated separately during training. The K groups of intermediate feature maps are summed element by element according to their corresponding channels. The difference between the pixel differences between the last frame and the previous K-1 frames is reflected by the difference between the convolutional kernel weights corresponding to each channel group, thus obtaining a feature map that integrates inter-frame motion information. The number of channels in the feature map is equal to the number of output channels of the first convolutional layer.
7. The video small moving target detection method according to claim 6, characterized in that, The step of calculating the loss value based on the difference between the predicted detection result and the target annotation information of the last frame includes: Based on the predicted bounding box in the predicted detection result and the labeled bounding box in the target annotation information of the last frame, the ratio of the intersection region to the union region between the two is calculated to obtain the intersection-union ratio. Based on the center point coordinates of the predicted bounding box and the labeled bounding box, the ratio of the center point distance between the two to the diagonal distance of the minimum bounding box is calculated to obtain the center point distance penalty term; Based on the aspect ratio of the predicted bounding box and the labeled bounding box, the consistency of the aspect ratio between the two is calculated to obtain the aspect ratio penalty term; The loss value is obtained by weighted summation of the intersection-union ratio, the center point distance penalty term, and the aspect ratio penalty term.
8. The video small moving target detection method according to claim 6, characterized in that, The process of backpropagating to update the network parameters of the initialized object detection network based on the loss value, and then re-inputting the multi-frame stacked input tensor and the target annotation information of the corresponding last frame into the updated initialized object detection network for iterative training until the network parameters converge, thereby obtaining a trained object detection network, includes: Backpropagation is performed on the network parameters of each layer of the initialized target detection network based on the loss value to obtain the gradient corresponding to each layer of network parameters; Based on the gradients corresponding to the network parameters of each layer, the first-order moment estimates and second-order moment estimates of each layer are updated by moving average, resulting in updated first-order moment estimates and second-order moment estimates. The updated first-order moment estimate is divided by the square root of the updated second-order moment estimate to obtain the adaptive learning rate corresponding to the network parameters of each layer. The network parameters of each layer are updated according to the adaptive learning rate, and the updated network parameters are subject to weight decay to obtain the initialized target detection network with updated network parameters. The initial target detection network after re-inputting the multi-frame stacked input tensor and the target annotation information of the corresponding last frame into the network parameters is iteratively trained until the network parameters converge, thus obtaining the trained target detection network.
9. A video small moving target detection device, characterized in that, include: The data stacking module is used to stack and stitch K consecutive frames of images in a historical video stream along the channel dimension to obtain a multi-frame stacked input tensor. The network initialization module is used to expand the number of input channels of the first convolutional layer of the pre-trained YOLO network according to the number of channels of the multi-frame stacked input tensor, while keeping the structure of the remaining network layers of the pre-trained YOLO network unchanged, to obtain a network model with expanded channels; to copy the original pre-trained weights of the first convolutional layer of the pre-trained YOLO network K times in the channel dimension to obtain expanded weights; to perform mean processing on the expanded weights, and to load the mean-processed expanded weights into the first convolutional layer of the network model with expanded channels, to obtain the initialized object detection network; The weakly supervised training module is used to train the initial target detection network by using the target annotation information of the last frame in the multi-frame stacked input tensor as a supervision signal, so as to obtain the trained target detection network. The inference input module is used to stack and stitch K consecutive frames of images in the video stream to be detected along the channel dimension to obtain a stacked input tensor of multiple frames to be detected. The target detection module is used to input the stacked input tensor of the multi-frames to be detected into the trained target detection network, perform moving target detection on the last frame in the stacked input tensor of the multi-frames to be detected, and obtain the target detection result of the last frame.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video small moving target detection method as described in any one of claims 1 to 8.
11. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the video small moving target detection method as described in any one of claims 1 to 8.
12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to implement the steps of the video small moving target detection method as described in any one of claims 1 to 8.
