Video target detection method and system based on coupled frame difference yolov8 model

Abstract

The invention relates to the technical field of computer vision and industrial safety monitoring, and discloses a video target detection method and system based on a coupling frame difference yolov8 model, and the method comprises the steps: 1, collecting a video stream in real time, extracting an RGB image of a current frame, and converting the RGB image into a gray-scale map; meanwhile, a historical frame grey-scale map cache queue with a fixed length is constructed and maintained and is used for storing grey-scale information of recent frames; step 2, calculating a frame difference between a current frame grey-scale map and a historical frame grey-scale map at an interval of 10 frames in a buffer queue, introducing a space-time coupling causal attention mechanism, and generating a self-adaptive attention weight single-channel frame difference image; 3, taking the single-channel frame difference image as a motion feature channel, and splicing the single-channel frame difference image with the RGB three-channel image of the current frame in the channel dimension to construct a four-channel fusion data tensor; and step 4, training a yolov8 detection model by using the four-channel fusion data, so that the yolov8 detection model can cooperatively utilize static appearance and dynamic motion clues, and a detection result is output.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial safety monitoring technology, specifically to a video target detection method and system based on the coupled frame difference yolov8 model. Background Technology

[0002] In the safety supervision system of petrochemical production, existing technologies face a dual challenge: On the one hand, traditional manual inspections are easily constrained by human factors, have blind spots in monitoring and are slow to respond, while fixed-point monitoring systems are difficult to support global risk assessment due to the data silo effect; on the other hand, although mainstream gun-ball linkage systems rely on wide-angle guns to achieve panoramic monitoring, they generally face the problem of detecting ultra-small targets in long-distance scenarios, with the target pixel size often less than 0.1% of the image (<32×32), resulting in a sharp drop in recognition accuracy.

[0003] To address the aforementioned issues, current improvement schemes attempt to incorporate frame difference information into the YOLOv8 model, but this exposes fundamental technical bottlenecks: First, the classic frame difference algorithm is affected by non-causal noise such as equipment vibration, sudden changes in lighting, and dust interference in industrial environments, resulting in a motion feature signal-to-noise ratio of less than 15dB, which severely weakens the reliability of moving target extraction; Second, the fusion strategy of stitching RGB and frame difference map channels forces the network to process spatiotemporally heterogeneous static appearance features and dynamic motion features synchronously at the bottom layer, leading to gradient conflicts and feature degradation.

[0004] Experiments show that the average detection accuracy of this method on industrial-grade NPU edge computing devices is only 68.7%, which is 42% less efficient than the hierarchical processing mechanism of biological vision systems, and falls far short of the ≥95% accuracy requirement for detecting personnel intrusion in dangerous areas. A deeper problem lies in the fact that traditional fusion architectures lack adaptive suppression mechanisms for non-causal noise, have insufficient contextual information about small target motion, suffer significant interference from early heterogeneous features, and lack explicit correlation constraints between multimodal features, resulting in low information utilization and weakened feature complementarity.

[0005] Therefore, there is an urgent need to develop a robust multimodal feature fusion architecture to overcome the technical bottleneck of accurate identification of small targets under complex industrial conditions. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a video target detection method and system based on the coupled frame difference YOLOv8 model. It has the advantages of high-precision small target detection, strong anti-interference capability, and efficient hierarchical feature fusion, and solves the problems of perception blind spots and analysis lag in traditional security monitoring technology in small target detection scenarios.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution: a video target detection method based on a coupled frame difference YOLOv8 model, comprising the following steps:

[0010] Step 1: Real-time acquisition of video stream, extraction of the RGB image of the current frame and conversion to grayscale; simultaneously, construction and maintenance of a fixed-length historical frame grayscale image cache queue to store grayscale information of recent frames;

[0011] Step 2: By calculating the frame difference between the current frame grayscale image and the historical frame grayscale images in the cache queue at intervals of 10 frames, a spatiotemporally coupled causal attention mechanism is introduced to generate an adaptive attention weight single-channel frame difference image.

[0012] Step 3: Use the single-channel frame difference image as a motion feature channel and concatenate it with the RGB three-channel image of the current frame in the channel dimension to construct a four-channel fused data tensor.

[0013] Step 4: Train the YOLOv8 detection model using four-channel fused data, enabling it to collaboratively utilize static appearance and dynamic motion cues to output accurate detection results.

[0014] The video target detection system based on the coupled frame difference YOLOv8 model is constructed according to the above-mentioned video target detection method based on the coupled frame difference YOLOv8 model. The system consists of a video frame acquisition module, a frame buffer management module, a lightweight causal attention module, a multimodal coupling module, a dual-branch detection network module, and a deployment and application module.

[0015] The video frame acquisition module has a built-in video stream processing model, which is responsible for acquiring video streams from industrial monitoring equipment in real time, extracting RGB images and generating grayscale images simultaneously.

[0016] The frame buffer management module is responsible for building and maintaining the historical frame grayscale image queue;

[0017] The lightweight causal attention module is responsible for calculating motion saliency weights;

[0018] The multimodal coupling module is responsible for fusing the RGB image and the weighted frame difference image in the channel dimension to generate four-channel input data;

[0019] The dual-branch detection network module constructs a YOLOv8 model to process the fused multimodal data and achieves accurate target detection through hierarchical feature fusion and mutual information constraints.

[0020] The deployment application module is responsible for deploying the trained detection model to the actual production environment to realize online real-time analysis and target detection of industrial video streams.

[0021] Preferably, the video frame acquisition module acquires continuous video streams from monitoring equipment in the industrial scene via a network, and assigns video stream sequence numbers, denoted as the video frame sequence. In the formula, Indicates at time The video frame image is extracted in real time from the video stream using a built-in video stream processing model. Output the corresponding RGB image Simultaneously convert the current frame to grayscale. :

[0022]

[0023] In the formula, , , These are the values ​​of the three color channels for the corresponding pixel in the current frame.

[0024] Preferably, the frame buffer management module constructs a grayscale image buffer queue of historical frames with a fixed length of 10, and stores the processed historical frames... Stored in the cache queue in chronological order, when a new frame... During input, the earliest frame in the queue is removed. and add new frames. To maintain dynamic updates to the cache queue.

[0025] Preferably, the lightweight causal attention module includes a feature splicing and causal convolution unit, a contribution modulation unit, a normalization and aggregation unit, and a motion information weighting unit;

[0026] The feature concatenation and causal convolution units use the current frame's grayscale image. and historical frame grayscale image sequence As input, a shallow convolutional network is used to perform feature mapping on each frame of grayscale image to obtain the feature map of each frame. Then, the feature map of the current frame is concatenated with the feature maps of each historical frame in the channel dimension and input into a causal convolutional layer for processing; for each spatial location and every historical frame Calculate the original causal attention score .

[0027] Preferably, the contribution modulation unit introduces a contribution attenuation factor. Calculate the original causal attention score The calculation formula is as follows:

[0028]

[0029] The spatiotemporal causal contribution is then calculated by normalizing using the Softmax function along the time dimension. The calculation formula is as follows:

[0030]

[0031] In the formula, Integers between 1 and 10;

[0032] Finally, the normalization and aggregation unit calculates the spatiotemporal causal contribution of each historical frame. Aggregate along the time dimension to generate and Same size integrated attention weight map The calculation formula is as follows:

[0033]

[0034]

[0035] In the formula, Each value in the table represents the probability that the change in position is caused by actual motion. It is the current frame and historical frames. Motion consistency mask;

[0036] The motion information weighting unit calculates the base frame difference image. With attention weight map Element-wise multiplication yields a weighted frame difference image. Its calculation formula is:

[0037]

[0038] In the formula, This is for element-wise multiplication.

[0039] Preferably, the multimodal coupling module obtains the single-channel frame difference image from the lightweight causal attention module. As a new channel, it is coupled and stitched with the three-channel RGB image in the channel dimension to generate a brand-new four-channel fused data:

[0040]

[0041] In the formula, the four channels represent red, green, blue, and motion data, respectively. To splice along the channel dimension.

[0042] Preferably, the dual-branch detection network module includes a model initialization unit, a branch construction unit, a shallow fusion unit, a deep fusion unit, a mutual information constraint unit, and a loss optimization unit;

[0043] The model initialization unit constructs a YOLOv8 model based on the native YOLOv8 network architecture and uses four-channel fused data. The pre-labeled training data was input into the YOLOv8 model, and the number of convolution channels in the input layer of the model was adjusted to 4. The other parameters and network structure remained unchanged.

[0044] The branch construction unit constructs a dual-branch hierarchical fusion network, splitting the data into RGB three-channel static input and frame difference single-channel dynamic input, which are input into the static branch and the dynamic branch respectively. In the static branch, the initial downsampling layer of the YOLOv8 backbone network is initialized using ImageNet pre-trained weights, and multi-scale static spatial features are extracted from the RGB three-channel input through the C2f module. The specific features are as follows:

[0045]

[0046] Simultaneously, a lightweight parallel convolutional branch is constructed, consisting of convolutional layers, each followed by BatchNorm and SiLU activation functions, to process the frame difference channel input and extract primary motion saliency features from the frame difference image. These features are:

[0047]

[0048] The shallow fusion unit performs shallow feature fusion after the output of the first C2f module in the backbone network. It takes the motion feature map output by the dynamic motion branch, adds it element-wise, and adjusts the number of channels using convolution to fuse it into the corresponding layer feature map of the static appearance branch.

[0049]

[0050] Before the deep fusion unit performs fusion in the third C2f module of the backbone network, it uses a spatial attention mechanism to weight the motion feature map. Based on the features that fuse static and dynamic information, it further injects motion information to extract more abstract and higher-level joint features.

[0051]

[0052]

[0053]

[0054] In the formula, This is a static branch multi-scale feature map. For static feature extraction backbone network, The output features of the initial, second, and third layers of the dynamic branching are... For batch normalization operations, Output features for the first and second fusion points. To upsample the motion feature map to the same resolution as the static feature map, Adjust the number of channels for a 1×1 convolution. This is a spatial attention map. It is the Sigmoid activation function. , These are average pooling and max pooling, respectively. This refers to the C2f module in the static appearance branch. For adaptive pooling operations, Dynamic features weighted for attention;

[0055] The fused features continue to pass through the remaining layers of the backbone network:

[0056]

[0057] The processed features are then input into the neck module of YOLOv8 for multi-scale feature fusion, and multi-scale features are extracted:

[0058]

[0059] The detection head then outputs the prediction results, which include the target bounding box coordinates, class probability, and confidence score.

[0060]

[0061] In the formula, For the final multi-scale feature map, For the remaining backbone network modules, A multi-scale feature pyramid for the neck output. For the neck network, For the detection head module, This is the output.

[0062] Preferably, the mutual information constraint unit introduces mutual information constraints, constructing the bidirectional prediction task as an InfoNCE-style contrastive learning problem. The goal is to maximize the mutual information of positive sample pairs while minimizing the mutual information of negative sample pairs. Positive sample pairs consist of dynamic and static features from the same image region, describing the features of the same target. Negative sample pairs consist of features from different image regions, describing the features of background regions or other targets. An InfoNCE-style loss function is used for optimization, employing a cosine similarity function. This method measures the consistency of two feature vectors in the projection space. In this way, the network can automatically discover and strengthen the inherent and meaningful statistical dependencies between dynamic and static modalities without additional explicit annotations. The specific relationships are as follows:

[0063]

[0064] Furthermore, a mutual information comparison loss is constructed between tasks:

[0065]

[0066] In the formula, It is a dynamic branching feature Vector representation after projection network It is a static branching feature Vector representation after projection network These are the positive sample feature vectors of the same target in the static branch. These are the positive sample feature vectors of the same target in the dynamic branch. , For a sample containing a positive sample and A negative sample feature vector extracted from other samples within the batch. The cosine similarity function is used. This refers to temperature hyperparameters.

[0067] The total mutual information constraint loss is:

[0068]

[0069] The loss optimization unit supervises model learning based on labeled data information, calculates the total loss function, and learns to accurately identify targets from fused features by optimizing the loss function.

[0070]

[0071] In the formula, For the total loss, For classifying losses, For bounding box regression loss, For target confidence loss, , , and These are the loss weight parameters.

[0072] Preferably, after the dual-branch detection network module completes model training, the deployment application module's built-in real-time inference engine processes the data from the real-time video stream of the industrial site, the repeated video frame acquisition module, the frame buffer management module, the lightweight causal attention module, and the multimodal coupling module. This generates real-time four-channel fused data, which is then input into the trained and optimized YOLOv8 detection model. After forward propagation and non-maximum suppression post-processing, the final prediction result for small target personnel is output, enabling the system to adapt to the complex environment requirements of PTZ cameras and wide-angle monitoring.

[0073] Compared with existing technologies, this invention provides a video target detection method and system based on the coupled frame difference YOLOv8 model, which has the following advantages:

[0074] 1. This invention introduces temporal frame difference calculation and performs channel-level fusion with the current RGB image to construct four-channel input data containing motion information. While preserving the color and texture features of the RGB image, it supplements the inter-frame motion change features, effectively solving the problem of missed detection caused by the low pixel ratio and insufficient features of small targets at a distance under the wide field of view of the bullet camera in industrial scenarios, thereby improving the system's small target detection sensitivity and recognition accuracy.

[0075] 2. This invention, by addressing the actual needs of the camera-ball linkage system in industrial intelligent safety monitoring scenarios, customizes a fusion strategy with a frame difference interval of 10 frames. This strategy can accurately capture the motion trajectory features of targets such as personnel while avoiding noise interference caused by short frame differences, thus adapting to the real-time monitoring needs in complex industrial environments and significantly enhancing the practical value of the intelligent safety monitoring system of this invention.

[0076] 3. This invention proposes a bio-inspired hierarchical perception mechanism. Based on the multi-level information processing pathway of retina → thalamus → cortex in the biological visual system, it constructs a progressive technical system of spatiotemporal perception → information fusion → hierarchical detection. In the backbone network of YOLOv8, static appearance features and dynamic motion features are separated, extracted, and progressively fused, thereby more efficiently utilizing multimodal information and ultimately improving the generalization ability and robustness of the YOLOv8 model. Attached Figure Description

[0077] Figure 1 This is a flowchart of the method of the present invention.

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Please see Figure 1 A video target detection method based on the coupled frame difference YOLOv8 model includes the following steps:

[0080] Step 1: Real-time acquisition of video stream, extraction of the RGB image of the current frame and conversion to grayscale; simultaneously, construction and maintenance of a fixed-length historical frame grayscale image cache queue to store grayscale information of recent frames;

[0081] Step 2: By calculating the frame difference between the current frame grayscale image and the historical frame grayscale images in the cache queue at intervals of 10 frames, a spatiotemporally coupled causal attention mechanism is introduced to generate an adaptive attention weight single-channel frame difference image.

[0082] Step 3: Use the single-channel frame difference image as a motion feature channel and concatenate it with the RGB three-channel image of the current frame in the channel dimension to construct a four-channel fused data tensor.

[0083] Step 4: Train the YOLOv8 detection model using four-channel fusion data, enabling it to collaboratively utilize static appearance and dynamic motion cues to output accurate detection results, thereby improving the detection accuracy and robustness of small targets in complex environments such as wide-angle surveillance.

[0084] The video target detection system based on the coupled frame difference YOLOv8 model is constructed according to the above-mentioned video target detection method based on the coupled frame difference YOLOv8 model. The system consists of a video frame acquisition module, a frame buffer management module, a lightweight causal attention module, a multimodal coupling module, a dual-branch detection network module, and a deployment and application module.

[0085] The video frame acquisition module has a built-in video stream processing model, which is responsible for acquiring video streams from industrial monitoring equipment in real time, extracting RGB images and generating grayscale images simultaneously.

[0086] The frame buffer management module is responsible for building and maintaining the historical frame grayscale image queue to ensure the continuity and integrity of time-series data;

[0087] The lightweight causal attention module is responsible for calculating motion saliency weights, suppressing background interference, and enhancing the real motion region.

[0088] The multimodal coupling module is responsible for fusing the RGB image and the weighted frame difference image in the channel dimension to generate four-channel input data;

[0089] The dual-branch detection network module is built on the native YOLOv8 network framework to construct a YOLOv8 model, which is used to process the fused multimodal data and achieve accurate target detection through hierarchical feature fusion and mutual information constraints.

[0090] The application deployment module is responsible for deploying the trained detection model to the actual production environment, enabling online real-time analysis and target detection of industrial video streams.

[0091] The video frame acquisition module acquires continuous video streams from monitoring equipment in industrial scenarios (such as the bullet cameras in a PTZ system) via a network, and assigns video stream sequence numbers, denoted as the video frame sequence. In the formula, Indicates at time The video frame image is extracted in real time from the video stream using a built-in video stream processing model. Output the corresponding RGB image Simultaneously convert the current frame to grayscale. :

[0092]

[0093] In the formula, , , These are the values ​​of the three color channels for the corresponding pixel in the current frame. This represents the two-dimensional spatial coordinates of the pixel in the image.

[0094] The advantages are: through the above-mentioned real-time synchronous extraction of RGB and grayscale images, a parallel and consistent data source is provided for subsequent static appearance analysis and dynamic motion perception, ultimately laying the foundation for multimodal information fusion.

[0095] The frame buffer management module constructs a grayscale image buffer queue of historical frames with a fixed length of 10, and stores the processed historical frames. Stored in the cache queue in chronological order, when a new frame... During input, the earliest frame in the queue is removed. and add new frames. To maintain dynamic updates to the cache queue.

[0096] The advantages are: by using the fixed-length first-in-first-out queue update mechanism, the system can review historical information with a constant, short-term memory window, providing a stable and continuous data foundation for time-series differential calculation, thereby effectively avoiding unlimited memory growth.

[0097] The lightweight causal attention module includes a feature concatenation and causal convolution unit, a contribution modulation unit, a normalization and aggregation unit, and a motion information weighting unit;

[0098] Feature concatenation and causal convolution units use the current frame's grayscale image and historical frame grayscale image sequence As input, a shallow convolutional network (consisting of two convolutional layers and ReLU) is used to perform feature mapping on each frame's grayscale image, obtaining a feature map for each frame. The current frame's feature map is then concatenated with the feature maps of each historical frame along the channel dimension and input into a causal convolutional layer for processing. For each spatial location... and every historical frame Calculate the original causal attention score .

[0099] The contribution modulation unit introduces a contribution attenuation factor. ( (with an attenuation coefficient less than 1) modulates the original causal attention score to reflect the recency effect, i.e., the closer a historical frame is to the current frame, the higher its potential contribution:

[0100]

[0101] Then, by normalizing using the Softmax function along the time dimension, we ensure that the attention score at the current moment depends only on current and historical information, and calculate the spatiotemporal causal contribution:

[0102]

[0103] In the formula, Integers between 1 and 10;

[0104] Finally, the normalization and aggregation unit calculates the spatiotemporal causal contribution of each historical frame. Aggregate along the time dimension to generate and Same size integrated attention weight map :

[0105]

[0106]

[0107] In the formula, Each value in the table represents the probability that the change in position is caused by actual motion. It is the current frame and historical frames. Motion consistency mask;

[0108] The motion information weighting unit calculates the base frame difference image. With attention weight map Element-wise multiplication yields a weighted frame difference image. :

[0109]

[0110] In the formula, This is for element-wise multiplication.

[0111] The advantages are: by constructing a feature splicing and causal convolution unit, a contribution modulation unit, a normalization and aggregation unit, and a motion information weighting unit through the above-mentioned lightweight causal attention module, a complete causal processing chain from feature splicing, contribution modulation, normalization aggregation to motion information weighting is formed to simulate the recency effect and selective attention mechanism of biological vision, so that the system can adaptively distinguish real moving targets from complex background perturbations, thereby improving the signal-to-noise ratio of motion features.

[0112] The multimodal coupling module obtains the single-channel frame difference image from the lightweight causal attention module. As a new channel, it is coupled and stitched with the three-channel RGB image in the channel dimension to generate a brand-new four-channel fused data:

[0113]

[0114] In the formula, the four channels represent red, green, blue, and motion data, respectively. To splice along the channel dimension.

[0115] The advantage is that by directly splicing static appearance (RGB) and dynamic motion (weighted frame difference) information in the channel dimension, a four-channel fusion tensor is creatively constructed, which enables the subsequent detection model to see the shape and motion state of the target at the lowest level, thus providing sufficient data for the deep fusion of high-level features.

[0116] The dual-branch detection network module includes a model initialization unit, a branch construction unit, a shallow fusion unit, a deep fusion unit, a mutual information constraint unit, and a loss optimization unit;

[0117] The model initialization unit builds a YOLOv8 model based on the native YOLOv8 network architecture and uses four-channel fused data. The pre-labeled training data (the annotation information of the corresponding frame includes the bounding box coordinates and category labels of small target people) is input into the YOLOv8 model, and the number of convolution channels in the input layer of the model is adjusted to 4, while the other parameters and network structure remain unchanged.

[0118] The branch construction unit constructs a dual-branch hierarchical fusion network, splitting the data into RGB three-channel static input and frame difference single-channel dynamic input, which are input to the static branch and the dynamic branch respectively, wherein:

[0119] In the static branch, the initial downsampling layer of the YOLOv8 backbone network (CSPDarknet) is initialized using ImageNet pre-trained weights. Multiple C2f modules extract multi-scale static spatial features, such as edges, textures, and colors, from the RGB three-channel input to obtain robust prior knowledge of static features.

[0120]

[0121] Simultaneously, a lightweight parallel convolutional branch is constructed, consisting of three 3×3 convolutional layers, each followed by a BatchNorm and SiLU activation function to process the frame difference channel input and extract primary motion saliency features from the frame difference image:

[0122]

[0123] The shallow fusion unit performs shallow feature fusion after the output of the first C2f module in the backbone network. It takes the motion feature map output by the dynamic motion branch, adds it element-wise, and adjusts the number of channels using a 1×1 convolution to fuse it into the corresponding layer feature map of the static appearance branch.

[0124]

[0125] Before fusing in the third C2f module of the backbone network, the deep fusion unit uses a spatial attention mechanism to weight the motion feature maps. Based on the features that fuse static and dynamic information, motion information is further injected to extract more abstract and higher-level joint features.

[0126]

[0127]

[0128]

[0129] In the formula, This is a static branch multi-scale feature map. For static feature extraction backbone network, The output features of the initial, second, and third layers of the dynamic branching are... For batch normalization operations, Output features for the first and second fusion points. To upsample the motion feature map to the same resolution as the static feature map, Adjust the number of channels for a 1×1 convolution. This is a spatial attention map. It is the Sigmoid activation function. , These are average pooling and max pooling, respectively. This refers to the C2f module in the static appearance branch. For adaptive pooling operations, Dynamic features weighted for attention;

[0130] The fused features continue to pass through the remaining layers of the backbone network:

[0131]

[0132] Secondly, these features are then fed into the neck module (FPN+PAN) of YOLOv8 for multi-scale feature fusion and extraction of multi-scale features:

[0133]

[0134] The detection head then outputs the prediction results, which include the target bounding box coordinates, class probability, and confidence score.

[0135]

[0136] In the formula, For the final multi-scale feature map, For the remaining backbone network modules, A multi-scale feature pyramid for the neck output. For the neck network, For the detection head module, This is the output.

[0137] The mutual information constraint unit introduces mutual information constraints (to measure and maximize the complementarity between features in two branches), constructing the bidirectional prediction task as an InfoNCE (noise-contrast estimation) style contrastive learning problem. The goal is to maximize the mutual information of positive sample pairs (dynamic and static features from the same image region) while minimizing the mutual information of negative sample pairs (features from different image regions). Positive sample pairs, representing dynamic and static features from the same image region, describe the features of the same target, while negative sample pairs, from different image regions, describe the features of the background region or other targets. An InfoNCE-style loss function is used for optimization. By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the network learns to determine which dynamic and static features should correspond, thereby enhancing their complementarity. This is achieved by using a cosine similarity function. To measure the similarity between two feature vectors in the projection space, the goal is to make the similarity between zdiff and z+rgb much higher than the similarity between zdiff and all other static feature vectors zkrgb (as negative samples) in the batch. In this way, the network can automatically discover and strengthen the inherent and meaningful statistical dependencies between dynamic and static modalities without additional explicit annotations. The specific relationships are as follows:

[0138]

[0139] Furthermore, a mutual information comparison loss is constructed between tasks:

[0140]

[0141] In the formula, It is a dynamic branching feature Vector representation after projection network It is a static branching feature Vector representation after projection network These are the positive sample feature vectors of the same target in the static branch. These are the positive sample feature vectors of the same target in the dynamic branch. , For a sample containing a positive sample and A negative sample feature vector extracted from other samples within the batch. The cosine similarity function is used. This refers to temperature hyperparameters.

[0142] The total mutual information constraint loss is:

[0143]

[0144] The loss optimization unit supervises model learning based on labeled data information, calculates the total loss function, and learns to accurately identify targets from fused features by optimizing the loss function.

[0145]

[0146] In the formula, For the total loss, For classifying losses, For bounding box regression loss, For target confidence loss, , , and These are the loss weight parameters.

[0147] The advantages are: through the modular design of the dual-branch detection network module for model initialization, dual-branch construction, hierarchical fusion (shallow and deep layers), mutual information constraints and loss optimization, the network can efficiently process dynamic and static information with a divide-and-conquer strategy, and strengthen the complementarity between features by maximizing mutual information. Finally, under the supervision of the loss function, it learns a more robust and accurate representation ability of small targets in complex environments from the fused features.

[0148] After the dual-branch detection network module completes model training, the application deployment module uses its built-in real-time inference engine to process the data from the real-time video stream of the industrial site, the repeated video frame acquisition module, the frame buffer management module, the lightweight causal attention module, and the multimodal coupling module. This generates real-time four-channel fused data, which is then input into the trained and optimized YOLOv8 detection model. After forward propagation and non-maximum suppression post-processing, the final prediction result (boundary box, category, confidence score) for small target personnel is output, adapting to the complex environment requirements of PTZ cameras and wide-angle monitoring.

[0149] The advantages are: by seamlessly integrating the trained model with the front-end data processing flow as described above, a complete end-to-end real-time video analysis system is built, which verifies the feasibility of this method from theoretical model to industrial application and can meet the stringent requirements of low latency and high accuracy in actual safety supervision scenarios.

[0150] In summary, this invention constructs a progressive technical system of spatiotemporal perception → information fusion → hierarchical detection, and innovatively couples the causal attention mechanism with a dual-branch YOLOv8 network. This significantly improves the system's detection accuracy, robustness, and real-time performance for small targets in complex industrial environments such as wide-angle monitoring and camera-ball linkage, effectively overcoming the limitations of traditional security monitoring models and existing video analysis technologies.

[0151] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A video target detection method based on a coupled frame difference YOLOv8 model, characterized in that, Includes the following steps: Step 1: Real-time acquisition of video stream, extraction of the RGB image of the current frame and conversion to grayscale; simultaneously, construction and maintenance of a fixed-length historical frame grayscale image cache queue to store grayscale information of recent frames; Step 2: By calculating the frame difference between the current frame grayscale image and the historical frame grayscale images in the cache queue at intervals of 10 frames, a spatiotemporally coupled causal attention mechanism is introduced to generate an adaptive attention weight single-channel frame difference image. Step 3: Use the single-channel frame difference image as a motion feature channel and concatenate it with the RGB three-channel image of the current frame in the channel dimension to construct a four-channel fused data tensor. Step 4: Train the YOLOv8 detection model using four-channel fused data, enabling it to collaboratively utilize static appearance and dynamic motion cues to output accurate detection results.

2. A video target detection system based on a coupled frame difference YOLOv8 model, characterized in that, The system framework for the video target detection method based on the coupled frame difference YOLOv8 model as described in claim 1 is constructed. The system consists of a video frame acquisition module, a frame buffer management module, a lightweight causal attention module, a multimodal coupling module, a dual-branch detection network module, and a deployment and application module. The video frame acquisition module has a built-in video stream processing model, which is responsible for acquiring video streams from industrial monitoring equipment in real time, extracting RGB images and generating grayscale images simultaneously. The frame buffer management module is responsible for building and maintaining the historical frame grayscale image queue; The lightweight causal attention module is responsible for calculating motion saliency weights; The multimodal coupling module is responsible for fusing the RGB image and the weighted frame difference image in the channel dimension to generate four-channel input data; The dual-branch detection network module constructs a YOLOv8 model to process the fused multimodal data and achieves accurate target detection through hierarchical feature fusion and mutual information constraints. The deployment application module is responsible for deploying the trained detection model to the actual production environment to realize online real-time analysis and target detection of industrial video streams.

3. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 2, characterized in that: The video frame acquisition module acquires continuous video streams from monitoring equipment in the industrial setting via a network, and assigns video stream sequence numbers, denoted as the video frame sequence. In the formula, Indicates at time The video frame image is extracted in real time from the video stream using a built-in video stream processing model. Output the corresponding RGB image Simultaneously convert the current frame to grayscale. : ； In the formula, , , These are the values ​​of the three color channels for the corresponding pixel in the current frame.

4. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 2, characterized in that: The frame buffer management module constructs a grayscale image buffer queue of historical frames with a fixed length of 10, storing the processed historical frames. Stored in the cache queue in chronological order, when a new frame... During input, the earliest frame in the queue is removed. and add new frames. To maintain dynamic updates to the cache queue.

5. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 2, characterized in that: The lightweight causal attention module includes a feature splicing and causal convolution unit, a contribution modulation unit, a normalization and aggregation unit, and a motion information weighting unit. The feature concatenation and causal convolution units use the current frame's grayscale image. and historical frame grayscale image sequence As input, a shallow convolutional network is used to perform feature mapping on each frame of grayscale image to obtain the feature map of each frame. Then, the feature map of the current frame is concatenated with the feature maps of each historical frame in the channel dimension and input into a causal convolutional layer for processing; for each spatial location and every historical frame Calculate the original causal attention score .

6. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 5, characterized in that: The contribution modulation unit introduces a contribution attenuation factor. Calculate the original causal attention score The calculation formula is as follows: ； The spatiotemporal causal contribution is then calculated by normalizing using the Softmax function along the time dimension. The calculation formula is as follows: ； In the formula, Integers between 1 and 10; Finally, the normalization and aggregation unit calculates the spatiotemporal causal contribution of each historical frame. Aggregate along the time dimension to generate and Same size integrated attention weight map The calculation formula is as follows: ； ； In the formula, Each value in the table represents the probability that the change in position is caused by actual motion. It is the current frame and historical frames. Motion consistency mask; The motion information weighting unit calculates the base frame difference image. With attention weight map Element-wise multiplication yields a weighted frame difference image. Its calculation formula is: ； In the formula, This is for element-wise multiplication.

7. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 2, characterized in that: The multimodal coupling module will obtain the single-channel frame difference image from the lightweight causal attention module. As a new channel, it is coupled and stitched with the three-channel RGB image in the channel dimension to generate a brand-new four-channel fused data: ； In the formula, the four channels represent red, green, blue, and motion data, respectively. To splice along the channel dimension.

8. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 2, characterized in that: The dual-branch detection network module includes a model initialization unit, a branch construction unit, a shallow fusion unit, a deep fusion unit, a mutual information constraint unit, and a loss optimization unit; The model initialization unit constructs a YOLOv8 model based on the native YOLOv8 network architecture and uses four-channel fused data. The pre-labeled training data was input into the YOLOv8 model, and the number of convolution channels in the input layer of the model was adjusted to 4. The other parameters and network structure remained unchanged. The branch construction unit constructs a dual-branch hierarchical fusion network, splitting the data into RGB three-channel static input and frame difference single-channel dynamic input, which are input into the static branch and the dynamic branch respectively. In the static branch, the initial downsampling layer of the YOLOv8 backbone network is initialized using ImageNet pre-trained weights, and multi-scale static spatial features are extracted from the RGB three-channel input through the C2f module. The specific features are as follows: ； Simultaneously, a lightweight parallel convolutional branch is constructed, consisting of convolutional layers, each followed by BatchNorm and SiLU activation functions, to process the frame difference channel input and extract primary motion saliency features from the frame difference image. These features are: ； The shallow fusion unit performs shallow feature fusion after the output of the first C2f module in the backbone network. It takes the motion feature map output by the dynamic motion branch, adds it element-wise, and adjusts the number of channels using convolution to fuse it into the corresponding layer feature map of the static appearance branch. ； Before the deep fusion unit performs fusion in the third C2f module of the backbone network, it uses a spatial attention mechanism to weight the motion feature map. Based on the features that fuse static and dynamic information, it further injects motion information to extract more abstract and higher-level joint features. ； ； ； In the formula, This is a static branch multi-scale feature map. For static feature extraction backbone network, The output features of the initial, second, and third layers of the dynamic branching are... For batch normalization operations, Output features for the first and second fusion points. To upsample the motion feature map to the same resolution as the static feature map, Adjust the number of channels for a 1×1 convolution. This is a spatial attention map. It is the Sigmoid activation function. , These are average pooling and max pooling, respectively. This refers to the C2f module in the static appearance branch. For adaptive pooling operations, Dynamic features weighted for attention; The fused features continue to pass through the remaining layers of the backbone network: ； The processed features are then input into the neck module of YOLOv8 for multi-scale feature fusion, and multi-scale features are extracted: ； The detection head then outputs the prediction results, which include the target bounding box coordinates, class probability, and confidence score. ； In the formula, For the final multi-scale feature map, For the remaining backbone network modules, A multi-scale feature pyramid for the neck output. For the neck network, For the detection head module, This is the output.

9. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 8, characterized in that: The mutual information constraint unit introduces mutual information constraints, constructing the bidirectional prediction task as an InfoNCE-style contrastive learning problem. The goal is to maximize the mutual information of positive sample pairs while minimizing the mutual information of negative sample pairs. Positive sample pairs consist of dynamic and static features from the same image region, describing the features of the same target. Negative sample pairs come from different image regions, describing the features of background regions or other targets. An InfoNCE-style loss function is used for optimization, employing a cosine similarity function. This method measures the consistency of two feature vectors in the projection space. In this way, the network can automatically discover and strengthen the inherent and meaningful statistical dependencies between dynamic and static modalities without additional explicit annotations. The specific relationships are as follows: ； Furthermore, a mutual information comparison loss is constructed between tasks: ； In the formula, It is a dynamic branching feature Vector representation after projection network It is a static branching feature Vector representation after projection network These are the positive sample feature vectors of the same target in the static branch. These are the positive sample feature vectors of the same target in the dynamic branch. , For a sample containing a positive sample and A negative sample feature vector extracted from other samples within the batch. The cosine similarity function is used. This refers to temperature hyperparameters. The total mutual information constraint loss is: ； The loss optimization unit supervises model learning based on labeled data information, calculates the total loss function, and learns to accurately identify targets from fused features by optimizing the loss function. ； In the formula, For the total loss, For classifying losses, For bounding box regression loss, For target confidence loss, , , and These are the loss weight parameters.

10. The video target detection system based on the coupled frame difference YOLOv8 model according to claim 2, characterized in that: After the dual-branch detection network module completes model training, the deployment application module uses its built-in real-time inference engine to process the data from the real-time video stream from the industrial site, the repeated video frame acquisition module, the frame buffer management module, the lightweight causal attention module, and the multimodal coupling module. This generates real-time four-channel fused data, which is then input into the trained and optimized YOLOv8 detection model. After forward propagation and non-maximum suppression post-processing, the final prediction result for small target personnel is output, enabling the system to adapt to the complex environment requirements of PTZ cameras and wide-angle monitoring.

Images