A method and system for detecting violations based on high-resolution video

Abstract

This invention belongs to the field of computer vision technology and discloses a method and system for detecting violations based on high-resolution video. The method includes: acquiring a high-resolution image and segmenting it into segmented images using a sliding window strategy; inputting the high-resolution image and the segmented images into the global and local branches of a dual-branch cross-scale guided aggregation deep learning model, respectively; in the model, the guided aggregation module uses the global feature map to guide and fuse the local feature map, outputting a first-round detection result for identifying human-related targets; and based on this result, determining the region of interest (ROI) on the original high-resolution image, and performing a second round of refined target detection within the ROI to determine the violation. This invention preserves high-definition details through a dual-branch architecture and combines a two-round detection strategy, significantly improving the detection accuracy of minute violations in videos while ensuring real-time performance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, specifically relating to a method and system for detecting violations based on high-resolution video. Background Technology

[0002] In many specific locations, such as production workshops, gas stations, forest fire prevention zones, and military-related classified areas, certain violations by personnel, such as smoking or unauthorized use of mobile phones, can lead to serious safety accidents such as fires and explosions, or result in information leaks, posing a significant threat to life, property, and information security. The traditional solution is to rely on manual real-time monitoring of video footage. However, this method is not only labor-intensive, but also prone to fatigue among monitoring personnel due to long working hours, leading to missed or incorrect detections.

[0003] With the development of deep learning technology, automated detection methods based on video streams have emerged. Mainstream methods, such as object detection algorithms based on the YOLO (You Only Look Once) network family, while exhibiting excellent real-time performance and robustness, face inherent technical bottlenecks when processing increasingly common high-definition (e.g., 2K or higher resolution) videos. To meet real-time requirements, these models typically need to downsample the input image to a smaller size (e.g., 640x640 pixels). This process causes severe information loss for high-resolution images; the effective pixel features of small targets in the image (e.g., a cigarette or mobile phone in a distant person's hand) may be completely erased, making it difficult for the model to detect and resulting in serious false negatives.

[0004] To address the problem of small target detection, one technical solution proposes pre-dividing high-resolution images into multiple tiles, and then feeding them one by one into the model for detection. However, this "brute-force" tile-division method has two significant drawbacks: first, it drastically increases the computational burden, as a single high-resolution image may be divided into dozens of tiles, leading to a significant increase in inference time and making it difficult to meet the needs of real-time monitoring; second, it cannot effectively handle targets that cross tile boundaries, easily resulting in targets being cut off and unable to be completely identified.

[0005] Therefore, there is an urgent need in this field for a new technical solution that can make full use of the rich visual details provided by high-resolution video without sacrificing real-time performance, thereby significantly improving the detection accuracy and robustness of tiny illegal targets in video. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies by providing a method and system for detecting violations based on high-resolution video, thereby solving the problem of poor performance in detecting small targets due to image downsampling mentioned in the background technology.

[0007] Technical solution: The first aspect of this invention provides a method for detecting violations based on high-resolution video, comprising the following steps:

[0008] S1: Acquire high-resolution images from a high-resolution video stream;

[0009] S2: The high-resolution image is segmented into segmentation maps using a sliding window strategy;

[0010] S3: Input the high-resolution image into the global branch of the deep learning model, and input the segmentation image into the local branch of the deep learning model;

[0011] S4: In the deep learning model, the guided aggregation module uses the global feature map output from the global branch to guide and fuse the local feature map output from the local branch to obtain the first round of detection results.

[0012] S5: Based on the results of the first round of detection, identify and locate the human-related targets (such as heads or hands) that are the subjects of the violations.

[0013] S6: Determine the region of interest (ROI) on the high-resolution image based on the location of the human body associated target.

[0014] S7: Perform a second round of target detection on the region of interest to detect tiny targets corresponding to violations, and determine whether a violation exists based on the results of the second round of detection.

[0015] Accordingly, the present invention also provides a system for implementing the above-described method for detecting violations based on high-resolution video, comprising:

[0016] The image acquisition and segmentation module is used to acquire high-resolution images from high-resolution video streams and segment the high-resolution images into at least one segmentation image using a sliding window strategy.

[0017] The first-round detection module is equipped with a deep learning model with a dual-branch cross-scale guided aggregation architecture. It is used to receive the high-resolution image and the segmentation map, and output the first-round detection result of the human body associated target as the subject of the violation through feature guidance and fusion of global and local branches.

[0018] The second-round detection module is used to determine the region of interest on the high-resolution image based on the location of the human-related target, and to perform refined target detection on the region of interest to identify the tiny target corresponding to the violation, thereby obtaining the second-round detection result.

[0019] The comprehensive judgment module is used to combine the results of the first round of detection and the results of the second round of detection to determine whether there is any violation.

[0020] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

[0021] This invention, through its dual-branch architecture, especially its local branches, directly uses high-resolution segmented images that have not undergone severe downsampling as input. The model's convolutional network can directly extract features from images that retain rich visual details, ensuring that the original feature information of small targets can enter the deep network for analysis from the source. This is the basis for the invention's ability to successfully detect small targets, fundamentally solving the problem of small target information loss caused by downsampling of high-resolution images, thereby significantly reducing the false negative rate and significantly improving detection accuracy.

[0022] This invention employs collaborative reasoning between the entire image (global branch) and the segmented images (local branch). The global branch performs a rapid inference on the downsampled entire image, providing global context; while the local branch processes only a limited number (e.g., 2x2=4) of segmented images. This parallel processing method of "global guidance of local" has a total computational load far less than the sum of independent inference on dozens of slices, thus ensuring high-precision detection while meeting the stringent speed requirements of real-time video stream processing, achieving an effective balance between real-time performance and high accuracy.

[0023] This invention effectively solves the challenge of cross-slice target detection and improves the robustness of the model by guiding the fusion of global and local features and applying consistency constraints. The global branch can capture complete large targets across slices; the guided aggregation module uses the global feature map to guide and fuse the feature maps of the local branches. Even if a target is segmented in a local branch, the model can still restore its integrity at the fused feature level by leveraging the complete target information provided by the global branch, thus achieving accurate detection. Furthermore, the global-local consistency loss introduced during training forces the model to produce consistent predictions for the same target within overlapping regions in both global and local branches, which greatly enhances the model's stability and generalization ability in multi-scale recognition.

[0024] The first round of detection doesn't aim for a one-step solution, but rather leverages the powerful recognition capabilities of the dual-branch network to confidently locate key human body regions (such as the head and hands) that are larger and more feature-rich than tiny targets like cigarettes or mobile phones. The second round of detection then inputs regions of interest (ROIs) directly from the uncompressed, high-resolution original image. These ROIs are not only small, facilitating rapid processing, but more importantly, they preserve the original high-definition details to the greatest extent possible. Finally, a specifically optimized small target detection model (such as a customized YOLO) is used to perform a detailed analysis of this ROI. Through this intelligent two-round decision-making strategy, computational resources are precisely focused, enabling accurate and efficient detection of tiny, illegal targets that are extremely difficult to identify using traditional methods, significantly improving the overall detection system's capabilities. Attached Figure Description

[0025] Figure 1 This is an overall flowchart of the violation detection method of the present invention.

[0026] Figure 2 This is a schematic diagram of the deep learning model based on the dual-branch cross-scale guided aggregation architecture of this invention. Detailed Implementation

[0027] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0028] Example 1: Refer to Figure 1 The present invention provides a method for detecting violations based on high-resolution video, comprising the following steps:

[0029] S1: Generate the segmented dataset and label files required for the training task;

[0030] S2: Build a dual-branch cross-scale guided aggregation network architecture and train a real-time detection model;

[0031] S3: Input the high-definition video stream from the site during the task phase and perform preprocessing;

[0032] S4: Use a real-time detection model with a dual-branch cross-scale guided aggregation architecture for inference to obtain the final detection results;

[0033] S5: Perform a second round of judgment based on the identification results to obtain the final judgment on the person's violation behavior, and push an alarm to the target terminal.

[0034] In step S1, generating the segmented dataset and label file required for the training task includes the following steps:

[0035] S11: During the dataset creation phase, continuous video recordings were conducted using cameras with a resolution of at least 2K in the required application scenarios to ensure image quality and detail visibility. Personnel were organized to simulate violations to create footage. Keyframes were uniformly extracted from the videos, ensuring temporal continuity while covering various lighting and background scenes. The collected data was then used to create the original image dataset. .

[0036] S12: For the high-resolution original images, manually annotate them to obtain label files for the object detection task, and create label files corresponding to the original image dataset. .

[0037] S13: The original image is segmented using a sliding window segmentation strategy, and adjacent slices maintain a certain proportion of overlap. The horizontal and vertical step sizes of the sliding window are calculated as follows:

[0038]

[0039]

[0040] in, and These represent the width and height of the sliding window, respectively. and These represent the horizontal and vertical step sizes, respectively. This indicates the overlap rate. The number of horizontal and vertical windows is calculated as follows:

[0041]

[0042] in, and The width and height of the high-resolution original image are represented separately. and These represent the number of horizontal and vertical windows, respectively. The coordinates of the top-left corner of each window after segmentation are also shown. The expression is as follows:

[0043]

[0044]

[0045] In practical project applications, through extensive experimental verification, the overlap rate... At around 0.3, good coverage can be achieved while preserving good small target feature information for the segmented image during subsequent training. Considering that the size of the segmented image should not be smaller than the model's input size, and balancing real-time performance and computational cost, the application will... and Set all values ​​to 2 and calculate the corresponding sliding window size. and Finally, the images are segmented using a sliding window method and saved to the segmentation dataset. At the same time, the relationship between the original image and the segmented sub-image is preserved through naming conventions.

[0046] S14: After the image is segmented using a sliding window method, the label files corresponding to the original image need to be mapped to each slice image. The labels after slicing need to be cropped and their coordinates transformed: targets that fall completely outside the slice area are discarded, and partially overlapping boxes are cropped into the slice area and their normalized coordinates are updated.

[0047] First, normalize the data of the annotation boxes in the original full-image label. Restore the original image width and height:

[0048]

[0049]

[0050] in, The coordinates () represent the coordinates of the top-left corner of the normalized annotation box. () represents the width and height of the normalized annotation box. () indicates the actual width and height of the entire image. This indicates the coordinates of the top-left corner of the bounding box after being converted to pixel coordinates. The bracket () represents the coordinates of the bottom right corner of the bounding box after being converted to pixel coordinates. Then, the intersection of the bounding box and the slice is calculated:

[0051]

[0052]

[0053] Simultaneously, determine if the target is retained if the following conditions are met:

[0054] and

[0055] Then discard targets with excessively large cropping ratios, when the area of ​​the intersection region... Discard items when the area of ​​the original annotation box is 5% of its original size.

[0056]

[0057] If the area of ​​the intersection region exceeds 5% of the original label frame area, retain the cropped label frame. Then, calculate the normalized coordinates within the slice:

[0058]

[0059]

[0060] Then save the tag file for the corresponding slice. .

[0061] like Figure 2 As shown, step S2, building a dual-branch cross-scale guided aggregation network architecture and training a real-time detection model includes the following steps:

[0062] S21: In the dual-branch cross-scale guided aggregation architecture, both the global and local branches are designed based on YOLOv11. First, a lightweight convolutional structure (C3k2 Blocks) is used to extract mid-to-low-level features; then, multi-scale spatial pyramid pooling (SPPF) is used to enhance contextual information; next, a cross-scale pixel spatial attention module (C2PSA) is used for subsequent cross-branch fusion, and finally, a three-level multi-scale feature map is output.

[0063]

[0064] in, , representing multi-scale semantic features, and Indicates the height and width of the model input dimensions; scale This represents a feature map with high relative spatial resolution (1 / 8 scale of the input image), corresponding to the feature resolution for detecting small-sized targets; scale 4 represents a medium-resolution feature map (scale 1 / 16 of the input image), corresponding to the feature resolution for detecting medium-sized targets; scale 5 represents the lowest resolution feature map (1 / 32 scale of the input image), which corresponds to the feature resolution for detecting large targets.

[0065] Lightweight convolutional structures (C3k2 Blocks) are a new module introduced in the open-source neural network YOLOv11. It's an improvement on the CSP (Cross-Stage Partial) architecture, achieving higher parameter and computational efficiency. While maintaining the Bottleneck architecture, it uses multiple smaller convolutions (e.g., 3×3) instead of a single large convolution to improve speed and generalization ability. C2PSA is another new mechanism in YOLOv11, located after SPPF, used to enhance spatial attention, especially strengthening the model's focus on key regions (small objects, complex backgrounds, etc.).

[0066] S22: In the dual-branch cross-scale guided aggregation architecture, the local branch architecture is consistent with the global branch and shares the shallow weights of the global branch. Shallow weights represent the weights of the shallow segments of the first three convolutional layers of the network module. Shallow features mainly extract basic visual features such as edges, textures, and colors. These basic features have universality in both global and local slices; sharing them improves efficiency and consistency. The shared shallow weights of the global branch are represented as follows:

[0067]

[0068] in, This represents the original graph used to input the global branch. express A partial cropped image. This indicates shared shallow weights. and These represent the independent parameters of the global branch and the local branch in the deeper section, respectively. and This represents the operation of each deep network.

[0069] The local branches will correspond to the original image. The slice images are merged into a single tensor. The output for each slice is represented as follows:

[0070]

[0071] Then merge them into batches:

[0072]

[0073] S23: In the dual-branch cross-scale guided aggregation architecture, the guided aggregation module first performs feature alignment of the output feature maps of the global branch and local branches. A cropping alignment method is used to align the global branch feature map with multiple feature maps of the local branches. For the k-th slice, the cropping start point and size of the original image are obtained through pre-recording. The corresponding feature map cropping region is:

[0074]

[0075]

[0076]

[0077] From global feature map The feature region is obtained by cropping:

[0078]

[0079] The cropped global features and local branch features are then channel-aligned using a unified structure.

[0080]

[0081] in, Indicates by Convolution, and The channel alignment module, composed of functions, outputs a uniform number of channels. . Represents a non-linear activation function. This refers to the BatchNorm function.

[0082] Then, a spatial attention guidance map is generated using global cropped features:

[0083]

[0084] Global cropping feature map generates channel attention guidance map:

[0085]

[0086] in, This represents the sigmoid function. This represents the global flat pooling function. Represents a compression matrix. Indicates the compression ratio. Used to restore dimensions.

[0087] Based on the spatial attention guidance map and the channel attention guidance map, weighted local features are calculated:

[0088]

[0089] Finally, the weighted local features are fused with the global features:

[0090]

[0091] in, This indicates channel concatenation, resulting in 2 channels after concatenation. Then through Convolution compression to .

[0092] The concatenation operation preserves the complete features of both branches in parallel. The concatenation method involves connecting along the channel dimension, which retains all the original semantics of both the "local" and "global" aspects, avoiding information loss. After concatenation, a 1×1 convolution is used to compress and fuse information. This method belongs to intermediate fusion, implementing a feature fusion strategy commonly used in deep learning at an intermediate stage, allowing the model to learn which information is more important and enhancing its expressive power. Simultaneously, the channel dimension is compressed, preventing rapid expansion of parameters and computation.

[0093] S24: The loss function of the dual-branch cross-scale guided aggregation architecture consists of a detection loss function and a global-local consistency loss function. The detection loss is calculated based on the output of the global branch and the final output of the fusion module. The loss function references the default settings of YOLOv11.

[0094]

[0095]

[0096] in, Indicates IOU loss. Represents classification loss, This represents the confidence loss. Global detection loss is used... Dataset corresponding labels, branch detection loss used Dataset and corresponding labels.

[0097] Global-local consistency loss function It is expressed as follows:

[0098]

[0099] in Represents the set of small targets with overlapping predictions. This represents the difference in coordinates between the two branches. and The category confidence score of the corresponding box. It is a loss function used in classification tasks.

[0100] Finally, the overall loss function is expressed as:

[0101]

[0102] in, Indicates the weights of the fusion feature loss. This represents the consistency loss weight.

[0103] S25: After completing the construction of the dual-branch cross-scale guided aggregation architecture deep network, through , The dataset is used to train an object detection model.

[0104] In step S3, the input and preprocessing of the on-site high-definition video stream during the task phase includes the following steps:

[0105] S31: Access the high-definition video stream, decode the video stream according to the source stream size, and divide the decoded high-definition images into segments according to the sliding window segmentation strategy in step S13. The segmented image.

[0106] In step S4, the real-time detection model using a two-branch cross-scale guided aggregation architecture is used for inference to obtain the final detection result, including the following steps:

[0107] S41: Downsample the high-resolution full image and the four segmented images to the input size of the model, merge these images and input them into a dual-branch cross-scale guided aggregation deep network. The final model outputs the real-time detection results.

[0108] In step S5, a second round of judgment is performed on the identification results to obtain the final judgment on the person's violation behavior, and an alarm is pushed to the target terminal; this includes the following steps:

[0109] S51: Determine if there is a human-related target in step S41, such as a head or hand. If the target exists, extract the coordinate information of the detection box. Then, based on the coordinate information, expand the detection box outward by a certain number of pixels to obtain the expanded target detection box. Extract a local image from the high-resolution original image based on the expanded target detection box and use it as the region of interest (ROI).

[0110] S52: For ROIs, a standard YOLOv11 model is used to detect small targets, such as masks, gloves, mobile phones, and cigarettes. In the first stage of full-image detection, global and local branches are used to first extract the ROIs where small targets might exist, and then extract local images from the high-resolution image, preserving the high-resolution features of the local images completely. Extracting the image before performing image preprocessing and downsampling greatly avoids feature loss from the high-resolution image. S53: Based on the detection results obtained in step S52, it is determined whether there is any violation by personnel, and the final alarm result is pushed to the relevant platform.

[0111] This invention designs a dual-branch, cross-scale guided aggregation architecture deep network and its corresponding dataset processing strategy for real-time detection of small object violations in high-definition video streams. First, high-definition images are acquired as training data. A sliding window strategy is used to segment each image into multiple slices, and the coordinates of the detection boxes in the original whole image annotation file are remapped and assigned to the corresponding slice annotation files according to the segmentation positions. The network architecture includes a global branch and a local branch. When inputting to the model, the whole image and its slices are uniformly scaled (downsampled) to a preset input size and then merged into a batch tensor input: the scaled whole image is input to the global branch to extract global feature maps, and the multiple slices are input to the local branch to extract local feature maps. Subsequently, the global feature maps are cropped to match the spatial size of the local feature maps and channel aligned. Based on the aligned global feature maps, spatial attention guidance maps and channel attention guidance maps are generated; these guidance maps are used to weight the local feature maps. Finally, the weighted local features are fused with the cropped and aligned global features to form the final object detection result. During training, the loss function consists of three parts: whole-frame detection loss, fusion feature loss, and global-local consistency loss. In practical applications, after decoding a high-definition video stream: a sliding window is used to segment the single-frame whole image to generate slice images; the whole image and slice images are preprocessed and then input into the network for the first round of inference; the bounding boxes associated with the human body (such as the head and hands) in the detection results are expanded, and corresponding local image patches are extracted from the high-definition original image as regions of interest (ROIs); the ROIs are input into a conventional YOLO model (such as YOLOv11) to detect small targets such as mobile phones and cigarettes; the results of the two rounds of detection are combined to determine whether there are violations such as smoking or using mobile phones, and an alarm is pushed to the target platform.

[0112] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for detecting violations based on high-resolution video, characterized in that, Includes the following steps: S1: Acquire high-resolution images from a high-resolution video stream; S2: The high-resolution image is segmented into segmentation maps using a sliding window strategy, including: acquiring a high-resolution image with original bounding boxes; setting the width, height, and overlap rate of the sliding window, and segmenting the high-resolution image into several segmentation maps with overlapping regions using the sliding window strategy; for each segmentation map, mapping the original bounding boxes in the high-resolution image to generate a corresponding annotation file for that segmentation map, wherein the mapping rule is: when the area of ​​the intersection region between an original bounding box and the current segmentation map is greater than a preset threshold of the total area of ​​the original bounding boxes, the intersection region is retained as a valid bounding box; the coordinates of the valid bounding boxes are transformed from the coordinate system of the high-resolution image to the coordinate system of the current segmentation map to generate a corresponding annotation file for that segmentation map; S3: Input the high-resolution image into the global branch of the deep learning model, and input the segmentation image into the local branch of the deep learning model; S4: In the deep learning model, the guided aggregation module uses the global feature map output from the global branch to guide and fuse the local feature map output from the local branch, including: cropping the global feature map according to the corresponding position of the segmentation map in the high-resolution image to obtain cropped global features; performing channel alignment processing on the cropped global features and the local features; generating a spatial attention guidance map and a channel attention guidance map based on the aligned global features; weighting the aligned local features with the spatial attention guidance map and the channel attention guidance map; and fusing the weighted local features with the cropped and aligned global features to generate fused features for outputting the first round of detection results. S5: Based on the results of the first round of detection, identify and locate the human-related targets that are the subjects of the violations; S6: Determine the region of interest on the high-resolution image based on the location of the human-related target; S7: Perform a second round of target detection on the region of interest to detect tiny targets corresponding to violations, and determine whether a violation exists based on the results of the second round of detection.

2. The violation detection method based on high-resolution video according to claim 1, characterized in that, In the sliding window strategy, the overlap rate between adjacent segmented graphs is set to be between 0.2 and 0.

4.

3. The violation detection method based on high-resolution video according to claim 1, characterized in that, The global and local branches in the deep learning model share the weights of the initial convolutional module used to extract shallow features; and both the global and local branches include a multi-scale spatial pyramid pooling module and a cross-scale pixel spatial attention module connected in sequence after the initial convolutional module.

4. The violation detection method based on high-resolution video according to claim 3, characterized in that, The deep learning model is trained using an overall loss function, which includes: global branch detection loss, fusion branch inspection loss, and global-local consistency loss to constrain the consistency of prediction results of the global branch and local branch for the same target in overlapping regions.

5. The violation detection method based on high-resolution video according to claim 1, characterized in that, The human body-related targets include the head or hand; the micro-targets include cigarettes or mobile phones.

6. A violation detection system based on high-resolution video, characterized in that, include: The image acquisition and segmentation module is used to acquire high-resolution images from high-resolution video streams and segment the high-resolution images into at least one segmentation image using a sliding window strategy. The first-round detection module is equipped with a deep learning model with a dual-branch cross-scale guided aggregation architecture. It is used to receive the high-resolution image and the segmentation map, and output the first-round detection result of the human body associated target as the subject of the violation through feature guidance and fusion of global and local branches. The second-round detection module is used to determine the region of interest on the high-resolution image based on the location of the human-related target, and to perform refined target detection on the region of interest to identify the tiny target corresponding to the violation, thereby obtaining the second-round detection result; The comprehensive judgment module is used to combine the results of the first round of inspections and the results of the second round of testing to determine whether there is any violation.

7. The violation detection system based on high-resolution video according to claim 6, characterized in that, The global and local branches of the deep learning model with the dual-branch cross-scale guided aggregation architecture each include: an initial convolutional module for extracting mid-to-low-level features, and a multi-scale spatial pyramid pooling (SPPF) module and a cross-scale pixel spatial attention (C2PSA) module connected sequentially after the initial convolutional module; and the global and local branches share the weights of the initial convolutional module.

8. The violation detection system based on high-resolution video according to claim 6, characterized in that, The steps of guiding and fusion implemented by the guidance aggregation module include: cropping the global features according to the position of the segmentation map, aligning the cropped global features and local features through channels, generating spatial and channel attention guidance maps based on the aligned global features, weighting the aligned local features using the guidance maps, and finally fusing the weighted local features with the cropped and aligned global features. The guided aggregation module in the first round of detection module achieves guidance and fusion through the following steps: cropping the global features according to the position of the segmentation map, aligning the cropped global features and local features through channels, generating spatial and channel attention guidance maps based on the aligned global features, weighting the aligned local features using the guidance maps, and finally fusing the weighted local features with the cropped and aligned global features.

Images