Target detection method and device based on multiple images and electronic equipment

Abstract

The application relates to the technical field of computer vision, and provides a target detection method and device based on multiple frames of images and electronic equipment, the target detection method based on multiple frames of images comprising the following steps: inputting multiple frames of to-be-detected images into a target detection model, and acquiring feature maps of the frames of images output by a fine-grained detail feature layer in the target detection model; flattening the feature maps into feature segment sequences; performing spatial alignment on the feature segment sequences of the frames of images, so that spatially aligned feature segment sequences are obtained; stacking the spatially aligned feature segment sequences in time sequence, so that a time sequence feature sequence is formed; performing weighted fusion on the time sequence feature sequence, so that a fused feature map is obtained; continuously inputting the fused feature map into a feature layer of the target detection model, which has a granularity larger than that of the fine-grained detail feature layer, so that comprehensive features are obtained; inputting the comprehensive features into a detection head of the target detection model, performing target regression and classification, and outputting a target detection result. The method improves the accuracy of target detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a target detection method, apparatus, and electronic device based on multi-frame images. Background Technology

[0002] Object detection technology, as one of the core tasks in the field of computer vision, has widely adopted algorithmic frameworks based on deep convolutional neural networks (CNNs), with models such as the YOLO series and Faster R-CNN being the most typical. The YOLO series, due to its combination of efficiency and real-time performance, has been widely used in various practical scenarios such as autonomous driving and security monitoring.

[0003] Taking the mainstream YOLOv8 model as an example, its network structure is mainly divided into three parts: Backbone, Neck, and Detection Head. The Backbone typically uses the CSPDarknet53 structure, which extracts features through multiple stages, outputting multi-scale feature maps such as P3, P4, and P5 at different levels, corresponding to details and semantic information at different resolutions. The Neck part utilizes feature fusion structures such as PANet to integrate multi-scale features across levels, enhancing the expressive power of the features. Finally, the fused features are fed into the Detection Head to complete the target's location regression and category classification.

[0004] To optimize target detection performance in harsh environments (such as sudden changes in lighting, occlusion, noise interference, etc.), traditional target detection models, such as YOLOv8, rely solely on single-frame image features, making them susceptible to interference from harsh environments. This leads to frequent issues such as missed detection of small targets and false detection of targets, making it difficult to meet the detection needs in complex scenarios. Summary of the Invention

[0005] This invention provides a target detection method, device, and electronic device based on multi-frame images to solve the problems of missed detection and false detection of small targets in the traditional single-frame target detection model in the prior art under harsh environments, thereby reducing the risk of accidents in scenarios such as autonomous driving.

[0006] This invention provides a target detection method based on multi-frame images, comprising: acquiring multiple frames of images to be detected; inputting the multiple frames of images to be detected into a pre-constructed target detection model to obtain feature maps of each frame of images output by the fine-grained detail feature layer in the target detection model; flattening the feature maps of each frame of images into a sequence of feature segments of each frame of images; spatially aligning the sequence of feature segments of each frame of images to obtain a spatially aligned sequence of feature segments; stacking the spatially aligned sequence of feature segments in chronological order to form a temporal feature sequence; performing weighted fusion on the temporal feature sequence to obtain a fused feature map; further inputting the fused feature map into a feature layer in the target detection model with a granularity greater than that of the fine-grained detail feature layer to obtain a comprehensive feature; inputting the comprehensive feature into the detection head of the target detection model for target regression and classification, and outputting the target detection result.

[0007] According to the target detection method based on multi-frame images provided by the present invention, the step of spatially aligning the feature segment sequences of each frame image to obtain a spatially aligned feature segment sequence includes: using the feature segment sequence of the current frame in the feature segment sequence of each frame image as a query, using the feature segment sequences of other frames in the feature segment sequence of each frame image as keys or values, generating an attention matrix based on the query, the attention matrix including a weight vector corresponding to each query segment and a spatial offset parameter, the spatial offset parameter being used to adjust the sampling position; and performing weighted fusion of the feature segment sequences corresponding to the values ​​according to the attention matrix to obtain a spatially aligned feature segment sequence.

[0008] According to the target detection method based on multi-frame images provided by the present invention, the step of weighted fusion of the temporal feature sequence to obtain a fused feature map includes: using the feature segment sequence of the current frame as a query, weighted fusion of the temporal feature sequence using a deformable attention mechanism to obtain a fused feature map.

[0009] According to the target detection method based on multi-frame images provided by the present invention, the target detection model includes a single-detection YOLOv8 version 8 model, the fine-grained detail feature layer includes a P3 layer, and the method of inputting the fused feature map into the fine-grained detail feature layer of the target detection model and the neck network to obtain comprehensive features includes: replacing the output of the original P3 layer with the fused feature map and inputting it into the feature pyramid network of the target detection model; through the horizontal fusion of the P3 layer and the P4 layer, the semantic transfer between the P4 layer and the P5 layer, and the multi-scale feature integration through the neck network to obtain comprehensive features.

[0010] According to the target detection method based on multi-frame images provided by the present invention, the feature pyramid network of the target detection model that replaces the original P3 layer with the fused feature map is used. Through horizontal fusion of P3 and P4 layers and semantic transfer between P4 and P5 layers, multi-scale feature integration is completed through the neck network to obtain comprehensive features. This includes: calculating the cosine similarity between the fused features of the current frame in the fused feature map and the corresponding position feature segments of the preceding preset number of frames to obtain a similarity sequence; if the mean of the similarity sequence is less than a first preset threshold T, it is determined that there is an inter-frame mismatch in the position feature, triggering adaptive enhancement; the enhancement magnitude of the adaptive enhancement is... Calculate using the following formula: ;in, For adjustment coefficients, The mean of the similarity sequence is used; the adaptive enhancement performs channel-by-channel enhancement on the feature segments at corresponding positions in the fused feature map, wherein the edge feature channels are enhanced according to... Magnified by 10 times, semantic feature channels are arranged according to The amplification is multiplied, and the enhanced feature value range is constrained by L2 regularization to avoid numerical overflow affecting subsequent fusion.

[0011] According to the target detection method based on multi-frame images provided by the present invention, the step of weighted fusion of the temporal feature sequence to obtain a fused feature map includes: setting dynamic weight coefficients for different feature channels when weighted fusion of the temporal feature sequence; for edge feature channels, calculating the edge response value of each frame feature segment using the Sobel operator, and normalizing the edge response value using the Sigmoid function as the weight coefficient of the channel, wherein the higher the edge response value, the larger the corresponding weight coefficient; for motion feature channels, estimating the pixel displacement of corresponding feature segments in adjacent frames using optical flow, and mapping the displacement to a suppression coefficient using an inverse exponential function, wherein the suppression coefficient is inversely proportional to the weight coefficient of the motion feature channel, and the larger the displacement, the smaller the weight coefficient; for brightness feature channels, calculating the deviation between the average gray value of the current frame image and a preset normal illumination gray value threshold, and automatically increasing the weight coefficient of the brightness feature channel to a preset multiple when the deviation exceeds a second preset threshold.

[0012] According to the target detection method based on multi-frame images provided by the present invention, after weighted fusion of the temporal feature sequence to obtain a fused feature map, the method further includes: extracting a dynamic confidence mask from the fused feature map, wherein the dynamic confidence mask is generated in the following manner: calculating the variance value of each feature segment in the fused feature map and the corresponding position feature segment in the preceding preset number of frames, mapping the variance value to a confidence value in the 0-1 interval using a Gaussian function, forming a mask matrix with the same shape as the fused feature map; marking the regions in the mask matrix with confidence values ​​lower than a third preset threshold as low-confidence regions; performing residual supplementation on the feature segments in the low-confidence regions through the edge feature channel output by the fine-grained detail feature layer, and simultaneously performing contextual constraints on the feature segments in the low-confidence regions through the semantic feature channel output by the feature layer with a granularity greater than that of the fine-grained detail feature layer, to obtain a corrected fused feature map; and performing feature normalization processing on the corrected fused feature map to keep the numerical distribution of the corrected fused feature map within a preset range, to obtain a normalized fused feature map.

[0013] This invention also provides a target detection device based on multi-frame images, characterized by comprising: a first acquisition module configured to acquire multiple frames of images to be detected; a second acquisition module configured to input the multiple frames of images to be detected into a pre-constructed target detection model to acquire feature maps of each frame of images output by the fine-grained detail feature layer in the target detection model; a flattening module configured to flatten the feature maps of each frame of images into a sequence of feature segments of each frame of images; a spatial alignment module configured to spatially align the sequence of feature segments of each frame of images to obtain a spatially aligned sequence of feature segments; a stacking module configured to stack the spatially aligned sequence of feature segments in chronological order to form a temporal feature sequence; a fusion module configured to perform weighted fusion on the temporal feature sequence to obtain a fused feature map; an input module configured to further input the fused feature map into a feature layer in the target detection model with a granularity greater than that of the fine-grained detail feature layer to obtain a comprehensive feature; and an output module configured to input the comprehensive feature into the detection head of the target detection model for target regression and classification, and output the target detection result.

[0014] The present invention provides a target detection method, device and electronic device based on multi-frame images, which proposes a novel target detection solution. It can effectively integrate the effective information of multi-frame images by processing the spatial alignment and weighted fusion of features in harsh environments, thereby improving the robustness and reliability of features and avoiding problems such as missed detection of small targets and false detection of targets. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the target detection method based on multi-frame images provided by the present invention.

[0017] Figure 2 This is a schematic diagram of the YOLOv8 target detection process enhanced by multi-frame image feature fusion provided by the present invention.

[0018] Figure 3 This is a schematic diagram of the target detection method based on multi-frame images provided by the present invention.

[0019] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising,” “including,” or “including,” and similar terms mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or “connected,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0022] The terminology involved in this invention will be briefly explained below.

[0023] The following is combined with Figures 1-4 This invention describes a target detection method, apparatus, and electronic device based on multi-frame images.

[0024] Figure 1 This is a flowchart illustrating the target detection method based on multi-frame images provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0025] Step 101: Obtain multiple frames of images to be detected.

[0026] In this embodiment, the multi-frame image to be detected refers to a sequence of images (such as video frames) containing the target that are continuously acquired or acquired at preset intervals in the time dimension, and can be acquired by sensors such as vehicle-mounted cameras.

[0027] Step 102: Input multiple frames of images to be detected into the pre-constructed target detection model, and obtain the feature maps of each frame of the image output by the fine-grained detail feature layer in the target detection model.

[0028] In this embodiment, the shape of the feature map of each frame image can be H represents the feature map height (pixel dimension), W represents the feature map width (pixel dimension), and C represents the number of feature channels (each channel corresponds to a feature type, such as edge or texture). The fine-grained detail feature layer is the feature layer in the model responsible for extracting local detail information (such as the P3 layer in YOLOv8), and its output feature map retains more low-level details such as object edges and textures. Implementation: Pre-trained models such as YOLOv8 and Faster R-CNN are used, and features are extracted through the mid-to-low layers of the backbone network (such as CSPDarknet53).

[0029] Step 103: Flatten the feature maps of each frame into a sequence of feature segments for each frame.

[0030] In this embodiment, the length of the feature segment sequence of each frame image can be Each feature fragment has a dimension of C. A feature fragment sequence is a one-dimensional sequence of two-dimensional feature maps expanded according to spatial location. Each fragment corresponds to a complete channel feature at a spatial location in the feature map. Feature map pixels can be read and concatenated row by row in priority order, or processed in column-priority order, ensuring a one-to-one correspondence between fragments and their original spatial locations in the feature map.

[0031] Step 104: Spatially align the feature segment sequences of each frame image to obtain spatially aligned feature segment sequences.

[0032] In this embodiment, spatial alignment is achieved by adjusting the spatial positions of feature segments from different frames, enabling corresponding segments in a multi-frame sequence to describe the same physical region (such as the same part of a target) in the image, thus resolving positional shifts caused by target motion or changes in viewpoint. Implementation methods include: utilizing a deformable attention mechanism, using the current frame segment as the query and other frame segments as the key, to generate an attention matrix (containing spatial offset parameters) to adjust the sampling position; or calculating inter-frame motion offsets based on algorithms such as optical flow estimation to correct segment positions.

[0033] Step 105: Stack the spatially aligned feature fragment sequences in chronological order to form a temporal feature sequence.

[0034] In this embodiment, the temporal feature sequence is a three-dimensional sequence formed by stacking and sorting feature segments from multiple aligned frames according to their acquisition time, integrating the temporal variation information of the target. Implementation method: The segments are arranged in ascending order by frame timestamps, and missing frames are padded with linear interpolation to ensure the temporal continuity of the sequence.

[0035] Step 106: Perform weighted fusion on the time-series feature sequences to obtain the fused feature map.

[0036] In this embodiment, the shape of the fused feature map can be Weighted fusion is a process of dynamically assigning weights based on the reliability of features in each frame, integrating effective information from multiple frames, and reducing interference from noisy or blurred frames. Specifically, a deformable attention mechanism can be used to weight the temporal sequence, focusing on features in clear frames; or dynamic weights can be assigned according to channel type (e.g., edge channel weights are positively correlated with edge response values), and the fused sequence can be restored to its original form using a reshape operation. Two-dimensional feature map.

[0037] Step 107: Input the fused feature map into the feature layer of the target detection model with a granularity greater than that of the fine-grained detail feature layer to obtain the comprehensive feature.

[0038] In this embodiment, the feature layer with a granularity larger than the fine-grained detail feature layer refers to the higher-level feature layer in the model with a larger receptive field and richer semantic information (such as the P4 and P5 layers in YOLOv8). The comprehensive feature is a multi-scale, multi-semantic fusion feature obtained after the fused features are processed by a higher-level network. Implementation method: The fused feature map is input into the feature pyramid network. Details are supplemented through lateral fusion of P3 and P4, and semantic transfer between P4 and P5 enhances the contextual information. After integration by the neck network (such as PANet), the comprehensive feature is output.

[0039] Step 108: Input the comprehensive features into the detection head of the target detection model, perform target regression and classification, and output the target detection results.

[0040] In this embodiment, the detection head is the component in the model responsible for predicting the target location and category. Target regression outputs bounding box coordinates (such as x, y center coordinates and width and height), and classification outputs the target category probability. Specifically, comprehensive features can be processed through convolutional layers, CIoU loss is used to optimize bounding box regression, and cross-entropy loss is used to optimize category classification. The output includes detection box coordinates, category labels, and confidence scores. Redundant boxes are removed using non-maximum suppression (NMS) to obtain the final detection result.

[0041] The target detection method based on multi-frame images provided by this invention achieves a hierarchical improvement in target detection performance by constructing an end-to-end process of "multi-frame feature extraction - spatial alignment - temporal fusion - multi-scale integration": First, it overcomes the information limitations of single-frame detection by expanding isolated details of a single frame into temporally correlated features including motion trajectories and occlusion complementarity through multi-frame image acquisition and feature extraction, thus solving the feature loss problem caused by environmental interference such as poor lighting and dynamic blur; Second, it eliminates spatial misalignment of multi-frame features caused by target motion or viewpoint changes through spatial alignment, ensuring that each feature is integrated within the frame. The feature fragments at corresponding positions in the frames accurately map to the same physical region, providing a reliable foundation for subsequent fusion. On this basis, the fused feature map is input into a multi-scale feature layer, combining fine-grained details with cross-level integration of high-level semantics. This allows the comprehensive features to contain both accurate local details and global semantic constraints, optimizing the ability to describe complex targets. Finally, the end-to-end closed-loop process outputs more accurate target regression and classification results from the detection head. Under complex conditions such as harsh environments and dynamic scenes, it can effectively reduce problems such as missed detection of small targets and false detection of occluded targets, significantly improving detection accuracy and engineering practicality.

[0042] In some optional implementations, spatial alignment is performed on the feature segment sequences of each frame image to obtain a spatially aligned feature segment sequence. This includes: using the feature segment sequence (Token sequence) of the current frame in the feature segment sequence of each frame image as a query, and using the feature segment sequences of other frames in the feature segment sequence of each frame image as keys or values; generating an attention matrix based on the query, the attention matrix including the weight vectors corresponding to each query segment and spatial offset parameters, the spatial offset parameters being used to adjust the sampling positions; and weighted fusion of the feature segment sequences corresponding to the values ​​according to the attention matrix to obtain the spatially aligned feature segment sequence. In the deformable attention mechanism, each query, through a learnable mapping, not only generates corresponding attention weights but also predicts a set of sparse spatial sampling offsets. These offsets indicate the sampling positions of the query Token in the keys and values ​​of other frames, thereby ensuring that cross-frame information can be accurately matched. Subsequently, relevant features are extracted based on the predicted offset position keys and values, and different importances are assigned to different sampling points according to attention weights. Finally, the weighted sampling features are summed to obtain a feature representation aligned with the current frame query, thus achieving accurate spatial alignment of cross-frame features.

[0043] In this implementation, query, key, and value are the core elements of the attention mechanism. The query is used to generate weights, the key is used to calculate the matching degree, and the value is the actual fused feature. The attention matrix is ​​a matrix containing a weight vector (representing feature importance) and a spatial offset parameter (used to correct positional deviations). The spatial offset parameter is a quantized value that adjusts the sampling position of feature segments, compensating for inter-frame offsets caused by target motion. This implementation dynamically generates the offset parameter through the attention mechanism, achieving fine-grained spatial alignment and solving the feature misalignment problem caused by target motion or viewpoint changes. Alternatively, convolutional layers or other network structures can be used to generate the attention matrix, leveraging the local receptive field of convolution to enhance local feature associations; or the spatial offset parameter can be directly calculated based on optical flow estimation results, simplifying the alignment logic.

[0044] In some optional implementations, the temporal feature sequences are weighted and fused to obtain a fused feature map, including: using the feature segment sequence of the current frame as a query, the temporal feature sequences are weighted and fused using a deformable attention mechanism to obtain a fused feature map.

[0045] In this implementation, the token sequence of the current frame is used as the query. The input is a time-stacked sequence S, serving as both key and value. A deformable attention mechanism is employed, directly generating an attention matrix through a fully connected layer in the temporal dimension via queries. This matrix is ​​then adjusted using deformable sampling to weightedly fuse tokens from multiple frames. Each current frame's query token generates attention weights through a learnable mapping, and sparse temporal sampling offsets are predicted to indicate the positions of interest within the stacked tokens. Subsequently, features are extracted from the stacked key and value tokens based on the predicted offset positions, and then weighted and summed according to the attention weights to achieve dynamic fusion of tokens from multiple frames. ;

[0046] in, This refers to the fused spatiotemporal features, the output of which already contains information about spatial alignment and temporal fusion. Rearranging can restore it to its original shape. Aligned feature map.

[0047] Deformable attention mechanisms are attention mechanisms that can dynamically adjust sampling positions, flexibly focusing on key regions by learning offsets rather than using a fixed sampling grid. This implementation can adapt to scenarios such as target deformation and partial occlusion, enhancing the ability to capture features of dynamic targets and improving the targeting of fused features. In addition, self-attention mechanisms can be used to capture long-distance dependencies between frames, or cross-attention mechanisms can be used to strengthen the feature interaction between the current frame and the previous frame; alternatively, basic weights can be assigned by combining frame quality scores (such as sharpness), and then fine-tuned using attention mechanisms.

[0048] In some optional implementations, the object detection model includes a single-detection YOLOv8 version 8 model, a fine-grained detail feature layer including the P3 layer, and a feature layer and neck network after the fine-grained detail feature layer in the object detection model to obtain comprehensive features. This includes replacing the output of the original P3 layer with the fused feature map as input to the feature pyramid network of the object detection model, and completing multi-scale feature integration through the neck network via the horizontal fusion of the P3 and P4 layers and the semantic transfer between the P4 and P5 layers to obtain comprehensive features.

[0049] In this implementation, the YOLOv8 model is a high-efficiency, real-time object detection model, comprising a backbone network, a neck network, and a detection head. The P3 layer is a high-resolution, fine-grained feature layer in YOLOv8, emphasizing detailed information. The feature pyramid network enhances multi-scale feature representation through cross-layer fusion. Lateral fusion connects high- and low-level features laterally, supplementing details and semantics. Semantic propagation is the process of transferring high-level semantic features to lower layers. This implementation adds a feature fusion module to the YOLOv8 backbone, after the P3 layer feature output and before the P4 layer input, replacing the single-frame P3 layer output with multi-frame fused features. This preserves details while injecting temporal information, improving the multi-scale feature integration effect. Furthermore, the fused feature map can be weighted and fused with the original P3 layer output (rather than directly replacing it) to retain the original single-frame feature information; or PANet can be used to replace the feature pyramid network, strengthening semantic propagation through a bottom-up enhancement path.

[0050] As an example, see Figure 2 , Figure 2 A schematic diagram of the process for YOLOv8 object detection enhanced by multi-frame image feature fusion is shown. Figure 2 The algorithm takes N consecutive frames of images as input and improves image quality through size normalization and data augmentation (such as deblurring and illumination compensation). Then, YOLOv8's Backbone extracts three layers of features: P3 (fine-grained, focusing on details), P4 (medium-grained, balancing details and semantics), and P5 (coarse-grained, focusing on global semantics). The P3 features are incorporated into the core innovative "feature fusion module," which includes spatial alignment and temporal fusion. Spatial alignment flattens the P3 features into a token sequence (each token corresponding to the spatial position of the feature map). Using the current frame token as the query and historical frame tokens as the key / value, deformable attention learns the inter-frame spatial offset, correcting the positional deviation of historical frame tokens and resolving feature misalignment caused by target motion / viewpoint changes. Temporal fusion stacks the spatially aligned multi-frame tokens along the time dimension and dynamically weights them using temporal attention (such as enhancing clear frames and suppressing blurred frames) to generate fused features (preserving complementary fine-grained information from multiple frames). Finally, multi-scale fusion and detection are performed. The fused P3 features, along with the original P4 and P5 features, are input into the Neck layer (PANet) for multi-scale feature integration (lateral fusion to supplement details and semantic transmission to enhance context). The detection head then performs target bounding box regression and category classification. Finally, NMS (Non-Maximum Suppression) filters out duplicate detection boxes, and the final result is output. The entire process follows a progressive logic of "spatial alignment to eliminate inter-frame bias → temporal fusion to aggregate the advantages of multiple frames → multi-scale integration to enhance semantics," specifically enhancing fine-grained feature representation and improving target detection accuracy in complex scenes (such as dynamic blur and occlusion).

[0051] In some optional implementations, the feature pyramid network of the original P3 layer of the target detection model is replaced by a fused feature map. Through lateral fusion between P3 and P4 layers and semantic transfer between P4 and P5 layers, multi-scale feature integration is completed via the neck network to obtain comprehensive features. This includes: calculating the cosine similarity between the fused features of the current frame and the corresponding positional feature segments of a preset number of previous frames in the fused feature map, obtaining a similarity sequence; if the mean of the similarity sequence is less than a first preset threshold T, it is determined that there is an inter-frame mismatch in the positional feature, triggering adaptive enhancement; the enhancement magnitude of the adaptive enhancement... Calculate using the following formula: ;in, For adjustment coefficients, The mean of the similarity sequence is used; adaptive enhancement performs channel-by-channel enhancement on the corresponding feature segments in the fused feature map, where the edge feature channels are enhanced according to... Magnified by 10 times, semantic feature channels are arranged according to After amplification, the feature value range is constrained by L2 regularization to avoid numerical overflow affecting subsequent fusion.

[0052] In this implementation, cosine similarity is used to measure the similarity of feature vectors (values ​​closer to 1 indicate greater similarity); the similarity sequence is the set of similarities between the current fused feature and the corresponding features in the previous frame; adaptive enhancement is a dynamic feature strengthening operation targeting mismatched regions between frames; L2 regularization is a method to prevent numerical overflow by constraining the sum of squared features. This implementation locates abnormal features through inter-frame consistency verification, differentially enhances edges and semantic channels, and improves feature reliability. Furthermore, Euclidean distance can be used instead of cosine similarity to measure feature differences; the enhancement magnitude can be achieved using a non-linear function to make the changes smoother; and regularization can also be replaced with L1 regularization.

[0053] In some optional implementations, the temporal feature sequences are weighted and fused to obtain a fused feature map. This includes: when weighting and fusing the temporal feature sequences, setting dynamic weight coefficients for different feature channels; for edge feature channels, calculating the edge response values ​​of each frame feature segment using the Sobel operator, and normalizing the edge response values ​​using the Sigmoid function as the weight coefficient for that channel, where a higher edge response value corresponds to a larger weight coefficient; for motion feature channels, estimating the pixel displacement of corresponding feature segments in adjacent frames using optical flow, mapping the displacement to a suppression coefficient using an inverse exponential function, where the suppression coefficient is inversely proportional to the weight coefficient of the motion feature channel, with a larger displacement resulting in a smaller weight coefficient; for brightness feature channels, calculating the deviation between the average grayscale value of the current frame image and a preset normal illumination grayscale threshold, and automatically increasing the weight coefficient of the brightness feature channel to a preset multiple when the deviation exceeds a second preset threshold.

[0054] In this implementation, the dynamic weight coefficients are weight values ​​that are dynamically adjusted according to the characteristics of the feature channels; the Sobel operator is a gradient operator for extracting edge features; the Sigmoid function is a normalization function that maps values ​​to the 0-1 interval; optical flow estimation is a technique for calculating inter-frame pixel motion displacement; and the inverse exponential function is a function that decreases as the input increases (e.g., exp(-kx)). This implementation allocates weights differently according to channel characteristics, strengthening effective features (such as sharp edges) and suppressing noise features (such as motion blur), thus improving the targeting of fusion. Furthermore, edge detection can use the Canny operator instead of the Sobel operator to improve edge continuity; motion information can be calculated using target tracking speed instead of optical flow estimation; and brightness evaluation can use the brightness histogram entropy value instead of the average gray value, which is more robust.

[0055] In some optional implementations, after weighted fusion of temporal feature sequences to obtain a fused feature map, the method further includes: extracting a dynamic confidence mask from the fused feature map. The dynamic confidence mask is generated as follows: calculating the variance of each feature segment in the fused feature map with the corresponding feature segments in the preceding preset number of frames; mapping the variance to a confidence value in the 0-1 interval using a Gaussian function to form a mask matrix with the same shape as the fused feature map; marking regions in the mask matrix with confidence values ​​lower than a third preset threshold as low-confidence regions; performing residual supplementation on the feature segments in the low-confidence regions using the edge feature channels output by the fine-grained detail feature layer; and simultaneously applying contextual constraints to the feature segments in the low-confidence regions using the semantic feature channels output by the feature layer with a granularity greater than that of the fine-grained detail feature layer, to obtain a corrected fused feature map; and performing feature normalization processing on the corrected fused feature map to keep the numerical distribution of the corrected fused feature map within a preset range, to obtain a normalized fused feature map.

[0056] In this implementation, the dynamic confidence mask is a matrix representing the reliability of feature fragments (higher values ​​indicate greater reliability); the variance value is an indicator of feature volatility (larger variance indicates lower reliability); the Gaussian function is a smoothing function that maps variance to confidence; residual supplementation is the operation of adding fine-grained edge features to low-confidence regions; and contextual constraints are the process of normalizing low-confidence region features using high-level semantic features. This implementation locates low-quality features through the confidence mask and achieves accurate correction by combining detailed supplementation and semantic constraints, thereby improving feature robustness. Furthermore, the variance value can be replaced with the standard deviation or mean absolute deviation to measure the degree of dispersion; the Gaussian function can be replaced with the Sigmoid function to map confidence; and normalization can be replaced with Z-Score normalization (mean of 0, standard deviation of 1) instead of Min-Max normalization.

[0057] The target detection device based on multi-frame images provided by the present invention is described below. The target detection device based on multi-frame images described below and the target detection method based on multi-frame images described above can be referred to and correspond to each other.

[0058] Figure 3 This is a schematic diagram of the structure of the target detection device based on multi-frame images provided in the embodiments of this application, as shown below. Figure 3 As shown, the module specifically includes: a first acquisition module 301, configured to acquire multiple frames of images to be detected; a second acquisition module 302, configured to input the multiple frames of images to be detected into a pre-constructed target detection model, and acquire feature maps of each frame of images output by the fine-grained detail feature layer in the target detection model; a flattening module 303, configured to flatten the feature maps of each frame of images into a sequence of feature segments of each frame of images; a spatial alignment module 304, configured to perform spatial alignment on the sequence of feature segments of each frame of images, and obtain a spatially aligned sequence of feature segments; a stacking module 305, configured to stack the spatially aligned sequence of feature segments in chronological order to form a temporal feature sequence; a fusion module 306, configured to perform weighted fusion on the temporal feature sequence to obtain a fused feature map; an input module 307, configured to continue inputting the fused feature map into a feature layer in the target detection model with a granularity greater than that of the fine-grained detail feature layer to obtain comprehensive features; and an output module 308, configured to input the comprehensive features into the detection head of the target detection model, perform target regression and classification, and output the target detection result.

[0059] The target detection device based on multi-frame images provided by this invention can effectively reduce problems such as missed detection of small targets and false detection of occluded targets under complex conditions such as harsh environments and dynamic scenes, and significantly improve detection accuracy and engineering practicality.

[0060] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can call logical instructions in the memory 430 to execute a target detection method based on multi-frame images. The method includes: inputting multiple frames of images to be detected into a pre-constructed target detection model to obtain feature maps of each frame of images output by the fine-grained detail feature layer in the target detection model; flattening the feature maps of each frame of images into a sequence of feature segments of each frame of images; spatially aligning the feature segment sequences of each frame of images to obtain a spatially aligned feature segment sequence; stacking the spatially aligned feature segment sequences in chronological order to form a temporal feature sequence; performing weighted fusion on the temporal feature sequence to obtain a fused feature map; further inputting the fused feature map into a feature layer in the target detection model with a granularity greater than the fine-grained detail feature layer to obtain a comprehensive feature; inputting the comprehensive feature into the detection head of the target detection model to perform target regression and classification, and outputting the target detection result.

[0061] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target detection method based on multi-frame images provided by the above methods. The method includes: inputting multiple frames of images to be detected into a pre-constructed target detection model to obtain feature maps of each frame of images output by the fine-grained detail feature layer in the target detection model; flattening the feature maps of each frame of images into a sequence of feature segments of each frame of images; spatially aligning the feature segment sequences of each frame of images to obtain a spatially aligned feature segment sequence; stacking the spatially aligned feature segment sequences in chronological order to form a temporal feature sequence; performing weighted fusion on the temporal feature sequence to obtain a fused feature map; further inputting the fused feature map into a feature layer in the target detection model with a granularity greater than that of the fine-grained detail feature layer to obtain a comprehensive feature; inputting the comprehensive feature into the detection head of the target detection model to perform target regression and classification, and outputting the target detection result.

[0063] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the target detection method based on multi-frame images provided by the above methods. The method includes: inputting multiple frames of images to be detected into a pre-constructed target detection model to obtain feature maps of each frame of images output by the fine-grained detail feature layer in the target detection model; flattening the feature maps of each frame of images into a sequence of feature segments of each frame of images; spatially aligning the feature segment sequences of each frame of images to obtain a spatially aligned feature segment sequence; stacking the spatially aligned feature segment sequences in chronological order to form a temporal feature sequence; performing weighted fusion on the temporal feature sequence to obtain a fused feature map; further inputting the fused feature map into a feature layer in the target detection model with a granularity greater than that of the fine-grained detail feature layer to obtain a comprehensive feature; inputting the comprehensive feature into the detection head of the target detection model to perform target regression and classification, and outputting the target detection result.

[0064] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0065] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method based on multi-frame images, characterized in that, include: Acquire multiple frames of images to be detected; The multi-frame images to be detected are input into a pre-constructed target detection model to obtain feature maps of each frame of the images output by the fine-grained detail feature layer in the target detection model. Flatten the feature maps of each frame image into a sequence of feature segments for each frame image; Spatially align the feature segment sequences of each frame image to obtain spatially aligned feature segment sequences; The spatially aligned feature fragment sequence is stacked in chronological order to form a temporal feature sequence. The time-series feature sequences are weighted and fused to obtain a fused feature map; The fused feature map is then input into a feature layer in the target detection model with a granularity larger than that of the fine-grained detail feature layer to obtain a comprehensive feature. The comprehensive features are input into the detection head of the target detection model to perform target regression and classification, and the target detection results are output.

2. The method according to claim 1, characterized in that, The step of spatially aligning the feature segment sequences of each frame image to obtain spatially aligned feature segment sequences includes: The feature segment sequence of the current frame in the feature segment sequence of each frame image is used as the query, and the feature segment sequences of other frames in the feature segment sequence of each frame image except the current frame are used as keys or values. An attention matrix is ​​generated based on the query. The attention matrix includes the weight vector corresponding to each query segment and a spatial offset parameter. The spatial offset parameter is used to adjust the sampling position. The feature fragment sequences corresponding to the values ​​are weighted and fused according to the attention matrix to obtain spatially aligned feature fragment sequences.

3. The method according to claim 1, characterized in that, The weighted fusion of the time-series feature sequences to obtain a fused feature map includes: Using the feature segment sequence of the current frame as a query, the temporal feature sequence is weighted and fused using a deformable attention mechanism to obtain a fused feature map.

4. The method according to claim 1, characterized in that, The target detection model includes a single-detection YOLOv8 version 8 model, the fine-grained detail feature layer includes a P3 layer, and the feature layer after the fine-grained detail feature layer in the target detection model, and the neck network to obtain the comprehensive features, including: The fused feature map replaces the original P3 layer's input / output feature pyramid network of the target detection model. Through the horizontal fusion of P3 and P4 layers and the semantic transfer between P4 and P5 layers, multi-scale feature integration is completed via the neck network to obtain comprehensive features.

5. The method according to claim 4, characterized in that, The feature pyramid network that replaces the original P3 layer's output with the fused feature map in the target detection model, through lateral fusion of P3 and P4 layers and semantic transfer between P4 and P5 layers, completes multi-scale feature integration via the neck network to obtain comprehensive features, including: Calculate the cosine similarity between the fused features of the current frame in the fused feature map and the corresponding position feature segments of the preceding preset number of frames to obtain a similarity sequence; If the mean of the similarity sequence is less than a first preset threshold T, then it is determined that there is an inter-frame mismatch in the location feature, triggering adaptive enhancement: The enhancement magnitude of the adaptive enhancement Calculate using the following formula: ； in, For adjustment coefficients, The mean of the similarity sequences; The adaptive enhancement performs channel-by-channel enhancement on the feature segments at corresponding positions in the fused feature map, wherein the edge feature channels are enhanced according to... Magnified by 10 times, semantic feature channels are arranged according to The amplification is multiplied, and the enhanced feature value range is constrained by L2 regularization to avoid numerical overflow affecting subsequent fusion.

6. The method according to claim 1, characterized in that, The weighted fusion of the time-series feature sequences to obtain a fused feature map includes: When performing weighted fusion on the time-series feature sequences, dynamic weight coefficients are set for different feature channels; For the edge feature channel, the edge response value of each frame feature segment is calculated by the Sobel operator. The edge response value is then normalized by the Sigmoid function and used as the weight coefficient of the channel. The higher the edge response value, the larger the corresponding weight coefficient. For the motion feature channel, the pixel displacement of the corresponding feature segment in adjacent frames is estimated by optical flow. The displacement is then mapped to a suppression coefficient by an inverse exponential function. The suppression coefficient is inversely proportional to the weight coefficient of the motion feature channel; the larger the displacement, the smaller the weight coefficient. For the brightness feature channel, the deviation between the average gray value of the current frame image and the preset normal illumination gray value threshold is calculated. When the deviation exceeds the second preset threshold, the weight coefficient of the brightness feature channel is automatically increased to the preset multiple of the benchmark value.

7. The method according to claim 1, characterized in that, After weighting and fusing the temporal feature sequences to obtain the fused feature map, the method further includes: A dynamic confidence mask is extracted from the fused feature map. The dynamic confidence mask is generated in the following way: the variance value of each feature segment in the fused feature map and the corresponding feature segment in the preceding preset number of frames is calculated, and the variance value is mapped to a confidence value in the 0-1 interval through a Gaussian function to form a mask matrix with the same shape as the fused feature map. Regions in the mask matrix with confidence values ​​lower than a third preset threshold are marked as low-confidence regions; The feature fragments in the low-confidence region are supplemented with residuals by the edge feature channel output by the fine-grained detail feature layer, and the feature fragments in the low-confidence region are constrained by the semantic feature channel output by the feature layer with a granularity larger than that of the fine-grained detail feature layer, so as to obtain the corrected fused feature map. The corrected fused feature map is then normalized to ensure that the numerical distribution of the corrected fused feature map remains within a preset range, thus obtaining the normalized fused feature map.

8. A target detection device based on multi-frame images, characterized in that, include: The first acquisition module is configured to acquire multiple frames of images to be detected; The second acquisition module is configured to input the multi-frame images to be detected into a pre-constructed target detection model, and acquire the feature maps of each frame of the images output by the fine-grained detail feature layer in the target detection model. The flattening module is configured to flatten the feature maps of each frame image into a sequence of feature segments of each frame image; The spatial alignment module is configured to spatially align the feature segment sequences of each frame image to obtain a spatially aligned feature segment sequence. The stacking module is configured to stack the spatially aligned feature fragment sequence in chronological order to form a temporal feature sequence. The fusion module is configured to perform weighted fusion on the temporal feature sequence to obtain a fused feature map; The input module is configured to further input the fused feature map into a feature layer in the target detection model with a granularity larger than the fine-grained detail feature layer to obtain a comprehensive feature. The output module is configured to input the comprehensive features into the detection head of the target detection model, perform target regression and classification, and output the target detection results.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the target detection method based on multi-frame images as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the target detection method based on multi-frame images as described in any one of claims 1 to 7.

Images