Method for extracting moving segments from surveillance videos based on multi-scale adaptive sampling lines

Abstract

The application discloses a kind of based on multi-scale self-adapting sampling line's monitoring video motion segment extraction method, it is related to video processing method technical field, to solve the problem of existing technology irregular area sampling not fit, multi-scale target information capture imbalance.It mainly contains the following steps:First, by multi-scale self-adapting threshold Canny edge detection to obtain effective activity area contour;Second, construct region shape self-adapting initial sampling line and divide sub effective area;Then estimate the scale of moving target and dynamically adjust sampling density, get refined sampling line;Then establish the weighted cumulative space-time flow model of fusion target scale and region importance;Finally, realize motion segment self-adapting division and pruning in combination with statistical threshold.The method gives consideration to scene adaptability and computational efficiency, significantly improves the accuracy of motion segment extraction.

Description

Technical Field

[0001] This invention relates to the field of video processing methods, specifically to a method for extracting motion segments from surveillance videos based on multi-scale adaptive sampling lines. Background Technology

[0002] With the increasing demand for security and the rapid development of hardware technology, surveillance cameras have been deployed in public places, transportation hubs, and smart buildings, generating massive amounts of video data. This video data is characterized by its long duration, high content redundancy, and sparse effective information. How to automatically and accurately extract short segments containing motion of interest from these lengthy video streams—i.e., motion segment extraction—has become a key preprocessing task in the field of intelligent video analytics. Efficient motion segment extraction can not only significantly reduce storage space and computing resource consumption but also provide high-quality input for subsequent high-level analysis tasks such as behavior recognition and anomaly detection, thus possessing significant theoretical research value and practical application value.

[0003] In the field of motion segment extraction, existing pipeline methods rely heavily on fixed-shape sampling. These fixed-shape pipelines struggle to adapt to the boundary features of irregular monitoring areas, resulting in poor practical performance. For example, in an L-shaped corridor, it's difficult to fit the corridor corners, leading to insufficient or redundant sampling in some key areas. At occluded entrances, obstructions may be included in the sampling range, introducing invalid information. Furthermore, for multi-scale targets, existing methods lack effective sampling density adjustment mechanisms. Large, nearby target areas often suffer from significant computational redundancy, while small, distant target areas experience information loss due to insufficient sampling points, severely impacting the accuracy and efficiency of motion segment extraction. However, this problem remains unresolved in the era of deep learning. Most network structures have fixed spatiotemporal receptive fields; this scale uniformity makes it difficult for the technology to maintain optimal performance across all types of motion events, becoming a key factor limiting the generalization ability of existing methods.

[0004] In summary, the core bottleneck of current motion segment extraction technology in surveillance videos lies in how to design a robust framework that can both adapt to multi-scale motion changes and maintain computational efficiency. Summary of the Invention

[0005] To address the problems existing in current motion segment extraction techniques for surveillance videos, this invention proposes a motion segment extraction method for surveillance videos based on multi-scale adaptive sampling lines, aiming to solve the problems of non-fitting sampling in irregular areas and unbalanced capture of multi-scale target information.

[0006] This invention provides a method for extracting motion segments from surveillance videos based on multi-scale adaptive sampling lines, the method comprising the following steps:

[0007] S1: Input the surveillance video to be processed, and use the multi-scale adaptive threshold Canny edge detection operator to perform edge detection on the surveillance video frames to obtain the effective activity area contour of the surveillance scene.

[0008] S2: Based on the outline of the effective active area, construct an initial sampling line that adapts to the shape of the area. At the same time, divide the effective active area into boundary sub-effective areas and internal sub-effective areas according to the distribution of the initial sampling line. The boundary sub-effective areas correspond to the edges of the effective active area and are used to accurately sense the entry and exit events of moving targets. The internal sub-effective areas correspond to the remaining areas of the effective active area except for the edges and are used to continuously track the activity state of moving targets within the area.

[0009] S3: Scale estimation of moving targets in surveillance video is performed by calculating the area of ​​connected components of the moving target in consecutive video frames and applying a preset small target area threshold. Large target area threshold Moving targets are classified into small targets, standard targets, and large targets. Based on the target scale determination results of the current frame and the previous and next frames, the sampling point density of the initial sampling line in the area where the current target is located is dynamically and continuously adjusted to obtain a refined sampling line, so that the sampling line density can adapt to the scale changes of moving targets in the scene.

[0010] S4: Based on the refined sampling line, establish a weighted cumulative spatiotemporal flow model and calculate the weighted cumulative spatiotemporal flow of each sub-effective region;

[0011] S5: Based on the weighted cumulative spatiotemporal flow, and combined with the preset minimum duration threshold... With preset noise threshold It performs adaptive segmentation and pruning operations on motion segments and outputs a list of high-quality motion segments.

[0012] Preferably, the multi-scale adaptive threshold Canny edge detection operator is used to perform edge detection on the monitoring video frame. Specifically, multi-scale Gaussian filtering is used to adapt the edge feature extraction of targets at different scales, and local grayscale statistics are used to dynamically adjust the dual thresholds to adapt to changes in illumination. This effectively removes static background interference and dynamic illumination noise, while completely preserving the activity range boundary of the moving target, and finally accurately obtains the effective activity area contour of the moving target in the monitoring scene.

[0013] Preferably, the shape of the region includes a straight-line region, a corner region, and an arc-shaped region, wherein:

[0014] Linear regions use line segment sampling lines to ensure that the overall sampling line closely matches the contour of the effective active area;

[0015] The corner area includes L-shaped corridor corners and T-shaped passage intersections. For the corner area, a polyline splicing sampling line is used, and the bend angle of the polyline splicing sampling line is consistent with the actual angle of the corner area.

[0016] The arc-shaped region includes an arc-shaped entrance and a circular monitoring area. For this region, an arc-fitting sampling line is used. The least squares method is employed to fit the arc contour, ensuring the fitting error meets the region's shape conformation accuracy requirements. This guarantees that the positioning deviation between the sampling line and the arc contour does not affect target tracking. The arc-fitting formula is:

[0017] ,

[0018] in This means finding the minimum value of variables O and R. This indicates that for N contour points Summation, Let O be the coordinates of the i-th contour point, O be the center of the fitted graph, and R be the radius of the fitted graph. This represents the Euclidean distance between two points.

[0019] Preferably, the scale estimation of moving targets in the surveillance video is achieved by calculating the area of ​​the connected components of the moving target in consecutive video frames, using the area of ​​the bounding rectangle of the moving target as a quantitative indicator of the area of ​​the connected components. The scale estimation formula is as follows: ,in Let be the connected component of the moving target in frame t. This represents calculating the area of ​​the bounding rectangle of this connected region and using it as... The quantization value used for target scale category determination, and the target scale category determination criteria are:

[0020] ,

[0021] in, For small target area threshold, This is the threshold for the large target area.

[0022] Preferably, the step of dynamically and continuously adjusting the sampling point density of the initial sampling lines in the region where the current target is located based on the target scale determination results of the current frame and the frames before and after it specifically involves: for each frame in the video sequence, identifying the sub-effective region where the moving target is located, and updating the sampling line density corresponding to that region in real time according to the real-time scale classification result of the target using the sampling density adjustment formula; wherein the sampling density adjustment formula is:

[0023] ,

[0024] in The initial sampling point density is the base number of sampling points per unit area set for the standard-scale target, ensuring complete and non-redundant sampling coverage of the standard target. Let be the area of ​​the circumscribed rectangle of the connected region. For small target area threshold, For large target area threshold, The density enhancement factor is set based on the integrity requirements of small target detection. The density reduction factor is set based on the requirement to avoid redundancy in large target sampling;

[0025] The specific adjustment rules are as follows: for small target areas, increase the sampling point density according to the requirement of small target detection integrity; for large target areas, reduce the sampling point density according to the requirement of sampling redundancy avoidance.

[0026] Preferably, the calculation process of the weighted cumulative spatiotemporal flow includes the following steps:

[0027] (1) Determination of basic inflow and outflow rates

[0028] For each video frame, the position of the moving target in each sub-effective region is tracked by inter-frame difference and intersection calculation of refined sampling lines, and the basic inflow and outflow rates are calculated. Defined as:

[0029] ,

[0030] in, This is the vector representing the positional change of the moving target's center in adjacent frames. This is the normal vector pointing from the sampling line to the outside of the region, i.e., "entering the region". Assigned value "Exit the area" Assigned value In other cases, the value is assigned to 0;

[0031] (2) Weighting coefficient setting

[0032] Target scale weight: This weight is set based on the contribution of the target scale to the detection results. The weight of a small target is greater than that of a standard target, which in turn is greater than that of a large target. The formula is set as follows:

[0033] ,

[0034] Where TSize represents the target scale category. Indicates the weight of small objectives, Indicates the standard target weight, The weights of the large objective are all coefficients greater than 0, and satisfy the following conditions: ;

[0035] Region importance weight: Based on the importance of region location to motion detection, the weight of the boundary sub-region is greater than that of the interior sub-region. The settings are as follows:

[0036] ,

[0037] Where k=1 corresponds to the boundary sub-effective region, and k=2 corresponds to the internal sub-effective region. Represents the effective region weight of the boundary sub-region. The weights of the internal sub-regions are all coefficients greater than 0, and satisfy the following conditions: ;

[0038] (3) Weighted flow calculation

[0039] Weighted traffic of a single frame and a single region ;

[0040] (4) Cumulative spatiotemporal flow calculation

[0041] The weighted cumulative spatiotemporal flow is obtained by accumulating the weighted cumulative flow of each sub-effective region frame by frame, and the initial cumulative flow is set. The formula for calculating subsequent frames is: ;

[0042] in, The weighted cumulative spatiotemporal flow of the k-th sub-effective region in the t-th frame. The weighted cumulative spatiotemporal flow of the k-th sub-effective region in the (t-1)-th frame. This represents the weighted traffic of the k-th sub-valid region in the t-th frame.

[0043] Preferably, the preset minimum duration threshold With preset noise threshold The settings are as follows:

[0044] The preset minimum duration threshold Based on the statistical analysis of the duration of typical noise segments in the monitoring scenario, the final value was determined to be 0.3 seconds; the preset noise threshold... The weighted cumulative spatiotemporal flow mean, determined based on invalid motion segments, was ultimately set at 0.3, and can be expressed as follows:

[0045] ,

[0046] in Indicates the expected value. The duration of the noise segment. The weighted cumulative spatiotemporal flow mean of invalid motion segments;

[0047] The specific steps for performing adaptive segmentation and pruning operations on motion segments are as follows:

[0048] Motion segment division: Based on the weighted cumulative spatiotemporal flow curves of each sub-effective region, if That is, there exists a sub-effective region k such that the weighted cumulative spatiotemporal flow of this sub-effective region in frame t is If the value is greater than 0, the current frame is determined to be a moving frame; otherwise, it is a still frame. Once consecutive moving frames are identified, these consecutive moving frames are sequentially connected to form a sequence, which constitutes a motion segment.

[0049] Noise segment pruning: For the segmented motion, if its duration is... Or the weighted cumulative spatiotemporal flow average of all valid sub-regions within this segment. If it is, it is identified as a noise segment and removed.

[0050] Compared with the prior art, the present invention has at least the following beneficial effects:

[0051] 1. This invention provides an effective active area coverage scheme using multi-scale adaptive threshold Canny edge detection and region shape adaptive sampling lines: It adapts to target edge features at different scales through multi-scale Gaussian filtering, and dynamically adjusts dual thresholds based on local grayscale statistics to resist illumination interference; for straight, corner, and arc-shaped monitoring areas, it employs line segments, polylines (with corners matching the actual angle of the area), and least-squares fitted arc sampling lines, respectively. Compared to existing fixed-shape sampling methods that suffer from insufficient or redundant sampling in irregular areas, this significantly improves the accuracy and fit of effective active area coverage, laying a precise foundation for motion information capture.

[0052] 2. This invention proposes a dynamic sampling density adjustment mechanism for moving target scale estimation: scale estimation is achieved by calculating the area of ​​the bounding rectangle of the connected component of the moving target in consecutive frames, with a preset small target area threshold. Large target area threshold The targets are categorized into three types: small, standard, and large. The sampling density is then dynamically adjusted according to the mapping relationship between target scale and sampling density. This addresses the problems of information loss for small targets and computational redundancy for large targets caused by the lack of multi-scale adaptation in existing technologies. It achieves balanced capture of multi-scale target information, taking into account both the details of distant small targets and the computational efficiency of nearby large targets.

[0053] 3. This invention designs a weighted cumulative spatiotemporal flow model that integrates target scale and regional importance: It determines the basic inflow and outflow flow through the intersection of inter-frame difference and refined sampling lines; based on the relationship between the target center position change vector and the sampling line normal vector direction, it sets differentiated weighting coefficients, and then... Frame-by-frame cumulative weighted flow. Compared to existing single flow calculations which are susceptible to noise interference, this improves the accuracy of motion information quantification and provides a reliable basis for motion segmentation.

[0054] 4. This invention constructs an adaptive motion segment pruning mechanism based on statistical thresholds: a preset minimum duration threshold is calculated based on the duration of noise segments in the monitored scene. A noise threshold is set based on the weighted cumulative spatiotemporal flow mean of invalid motion segments. Remove duration Or weighted cumulative spatiotemporal flow mean The project aims to improve the quality of output by compressing massive amounts of video while ensuring high-quality output, addressing issues such as noise residue and accidental deletion of valid segments caused by existing brute-force segmentation or fixed threshold methods. Attached Figure Description

[0055] The advantages and innovations of the present invention will become clearer and more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 This is a flowchart of the method for extracting motion segments from surveillance video based on multi-scale adaptive sampling lines in an embodiment of the present invention;

[0057] Figure 2 This is a diagram illustrating the effective activity area extraction effect in a monitoring scene according to an embodiment of the present invention.

[0058] Figure 3 This is a diagram showing the region shape adaptive four-layer sampling line system and the sub-effective region division results in an embodiment of the present invention;

[0059] Figure 4 This is a comparison diagram of the density of the initial boundary sampling line and the refined boundary sampling line in an embodiment of the present invention;

[0060] Figure 5 This is a comparison diagram of the density of the internal initial sampling line and the internal refined sampling line in an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram illustrating the relationship between T and weighted cumulative spatiotemporal flow before noise pruning of motion segments in an embodiment of the present invention;

[0062] Figure 7 This is a schematic diagram illustrating the relationship between T and weighted cumulative spatiotemporal flow after noise pruning of motion segments in an embodiment of the present invention.

[0063] Figure 8 This is a dynamic change diagram of the weighted cumulative spatiotemporal flow when a moving target crosses a sampling line in an embodiment of the present invention. Detailed Implementation

[0064] To enable those skilled in the art to better understand the present invention, the following description is provided in conjunction with specific embodiments.

[0065] The flowchart of the motion segment extraction method for monitoring video based on multi-scale adaptive sampling lines provided by this invention is as follows: Figure 1 As shown, it includes:

[0066] S1: Input the surveillance video to be processed and obtain the outline of the effective active area of ​​the surveillance scene;

[0067] S2: Based on the effective active area contour, construct an adaptive shape initial sampling line, and then divide it into boundary sub-effective areas and internal sub-effective areas according to the sampling line;

[0068] S3: Estimate the scale of the moving target, and dynamically adjust the sampling point density based on the mapping relationship between scale and sampling density to obtain a refined sampling line;

[0069] S4: Based on the refined sampling line, establish a weighted cumulative spatiotemporal flow model and calculate the weighted cumulative spatiotemporal flow of each sub-effective region;

[0070] S5: Based on weighted cumulative spatiotemporal flow, duration and noise threshold, adaptively segment and prune motion segments to output a list of high-quality motion segments.

[0071] Example 1:

[0072] This invention provides a preferred embodiment for executing S1, which involves inputting a surveillance video to be processed, and using a multi-scale adaptive threshold Canny edge detection operator to perform edge detection on the active region of the surveillance video to obtain the effective active region contour of the surveillance scene. The specific steps are as follows:

[0073] S11: Video Preprocessing and Background Modeling: The input video is preprocessed, including grayscale conversion and noise suppression; the MOG2 background subtraction algorithm is used to construct a background model, with the history frame number set to history=500 and the variance threshold varThreshold=16, effectively separating moving targets from static backgrounds;

[0074] S12: Cumulative Activity Analysis: A frame sampling strategy is used to process the complete video sequence. The motion foreground mask for each frame is obtained through background subtraction. Motion activity is accumulated across consecutive video frames to construct a cumulative activity map and calculate the cumulative activity intensity. The calculation formula is:

[0075] ,

[0076] in Let N be the binary motion mask for the t-th frame, and N be the total number of frames processed.

[0077] S13: Stable Region Extraction: Based on the cumulative activity graph, an adaptive thresholding method is used to extract stable activity regions. The threshold calculation formula is as follows:

[0078] ,

[0079] in The cumulative activity chart shows the mean, and 0.3 is the scaling factor.

[0080] S14: Morphological Optimization: Morphological operations are performed on the active mask, using 31×31 elliptical structuring elements to perform closing and opening operations, connecting discrete regions and removing noise. The formula is expressed as: ,

[0081] in, For the optimized connection mask, `open` performs a morphological opening operation, and `close` performs a morphological closing operation. This is the initial activity mask;

[0082] S15: Optimal Bounding Shape Calculation: For the extracted maximum connected region, calculate the compactness of three bounding shapes (convex hull, minimum bounding rectangle, minimum bounding circle), and select the shape that best fits the original contour. The compactness is equal to the area of ​​the bounding shape divided by the area of ​​the original contour. The bounding shape with a compactness closest to 1 is selected as the final effective active region contour. Through processing the input video, the convex hull is ultimately selected as the effective active region contour. Figure 2 This is a diagram showing the extraction effect of the processing results in this embodiment.

[0083] This embodiment significantly improves the accuracy and scene adaptability of active area extraction through complete video analysis and multi-shape adaptation, laying the foundation for subsequent sampling line construction.

[0084] Example 2:

[0085] This invention provides a preferred embodiment for executing S2, which involves constructing a four-layer initial sampling line with adaptive region shape based on the effective active region contour, and dividing the effective active region into boundary sub-effective regions and internal sub-effective regions. The specific steps are as follows:

[0086] S21: Shape recognition of the monitoring area

[0087] Based on the effective active region contour extracted by S1, the region shape is identified through geometric feature quantization, providing a basis for subsequent sampling line adaptation:

[0088] During feature extraction, the rectangularity is first calculated using the formula Rect = Ac / Arect, where Ac is the contour area and Arect is the area of ​​the minimum bounding rectangle of the contour. Next, the Harris corner detection algorithm is used with a 3×3 window, a Sobel aperture of 5, and a response threshold of 0.04 to filter strong corner points on the contour. Finally, least-squares circle fitting is performed on the contour, with the circle fitting error calculated using the formula... ,in This means finding the minimum value of variables O and R. This indicates that for N contour points Summation, Let O be the coordinates of the i-th contour point, O be the center of the fitted figure, and R be the radius of the fitted figure. The fitting error meets the requirements for the accuracy of the region shape fitting, ensuring that the positioning deviation between the sampling line and the arc contour does not affect the target tracking.

[0089] Regarding shape classification rules, linear regions must satisfy Rect > 0.9, the number of strong corner points ≤ 2, and >5 pixels, typical scenario is a straight corridor; corner areas need to satisfy 0.5≤Rect≤0.9, and the number of strong corner points is 2 (L-shaped) or 3 (T-shaped) and >5 pixels, typical scenarios include L-shaped corridor corners and T-shaped passage intersections; curved areas must satisfy Rect < 0.5 and the number of strong corner points ≤ 1. ≤2 pixels.

[0090] S22: Sampling Line System Design

[0091] Based on the region shape identified by S21, a multi-level sampling system containing four sampling lines is constructed to ensure that the sampling lines closely match the contour of the effective active region.

[0092] For the main boundary sampling lines, line segment sampling lines are used in straight-line regions. The contour is fitted to a straight line using the RANSAC algorithm, and the intersection of the straight line and the contour boundary is taken as the endpoint of the line segment. For corner regions, polyline splicing sampling lines are used. The strong corner point is taken as the inflection point, and the contours on both sides of the corner are fitted to line segments. The inflection angle is calculated by vector dot product. For arc-shaped regions, arc fitting sampling lines are used. Based on the least squares circle fitting results (O, R) of S21, an arc that fits the contour is generated.

[0093] The secondary boundary sampling line is generated by interpolating the boundary sampling line and the internal sampling line, with an interpolation ratio of 0.15.

[0094] The main internal sampling lines are generated by contour scaling. The scaling ratio for straight lines and corner lines is inner_scale=0.7; for arcs, the scaling center is O of the fitted circle, and the scaling ratio is inner_scale=0.65.

[0095] The secondary internal sampling line is generated by offsetting the internal sampling line towards the centroid, and the offset distance is the square root of the area of ​​the region multiplied by secondary_scale=0.03;

[0096] In this system, boundary sampling lines and internal sampling lines each have their own functional focus: boundary sampling lines are mainly used to detect the entry and exit of moving targets, defining the boundaries of motion segments with high confidence; internal sampling lines are used to continuously sense the activity status of targets within the area, ensuring the continuity of motion segments. Their functions complement each other, together forming a complete motion sensing network.

[0097] S23: Contour Scaling Algorithm

[0098] The scaling method based on the centroid of the contour is adopted, and its calculation formula is as follows: ,in, C represents the scaled coordinates of the contour points, and C represents the coordinates of the contour centroid. Here are the original contour point coordinates, where `scale` is the scaling factor, and the contour centroid coordinates are calculated using image moments, using the following formula:

[0099] ,

[0100] in, Let be the component of the first moment in the x-direction. The zero matrix represents the total area of ​​the contour. The component of the first moment in the y-direction is given; for the arc region, additional verification is performed to ensure that the radial deviation between the scaled contour and the original fitted circle is ≤1 pixel, thus ensuring scaling accuracy.

[0101] S24: Regional Division Strategy

[0102] Based on four sampling lines, the effective activity area is divided into a boundary sub-effective area and an internal sub-effective area, with the division method adapting to the shape of the area. Specifically, the boundary sub-effective area is the annular region between the boundary sampling line and the internal sampling line; the internal sub-effective area is the central region inside the internal sampling line.

[0103] The partitioning method uses mask difference operations:

[0104] ,

[0105] in, For the boundary sub-effective region, For internal sub-valid regions, For the boundary sampling line mask, For internal sampling line mask, For effective activity masking; additional verification of the arc-shaped region shows that the two regions do not overlap after division.

[0106] This design means that as long as valid motion is detected in any region, whether at the boundary or within the frame, the current frame is counted as a motion segment. This design allows the clarity of the boundary signal and the continuity of the internal signal to complement each other, jointly ensuring the accuracy and completeness of the output motion segment list.

[0107] S25: System Parameter Optimization

[0108] Based on the dynamic adjustment of inner_scale according to contour compactness, and combined with the region shape adaptation parameters, the formula for calculating contour compactness is as follows:

[0109] ,

[0110] Where Ac is the contour area and Ah is the convex hull area;

[0111] Figure 3 This diagram illustrates a region shape adaptive four-layer sampling line system and its sub-effective region division. The diagram shows the distribution of the main boundary sampling line, secondary boundary sampling line, main internal sampling line, and secondary internal sampling line. The main and secondary boundary sampling lines, along with the main and secondary internal sampling lines, can be used to determine whether a target enters or exits the monitoring area. Furthermore, the coordination between the main boundary sampling line and the main internal sampling line allows the monitoring area to be divided into boundary sub-effective regions and internal sub-effective regions.

[0112] This embodiment achieves refined coverage of complex monitoring scenarios through a four-layer sampling line system and intelligent area division, solving the problem of non-fitting sampling in irregular areas.

[0113] Example 3:

[0114] This invention provides a preferred embodiment that performs step S3, estimating the scale of moving targets in the surveillance video and dynamically adjusting the sampling point density based on the target scale to obtain a refined sampling line. The specific steps are as follows:

[0115] S31: Moving Target Detection and Connected Component Extraction: A frame difference method combined with morphological operations is used to extract connected components of moving targets. Specifically, consecutive video frames are grayscaled, and the absolute difference between adjacent frames is calculated to obtain a frame difference image. Subsequently, adaptive threshold binarization is applied to the frame difference image to generate a binary motion mask. Morphological opening operations (3×3 rectangular structuring elements) and closing operations (5×5 rectangular structuring elements) are performed on the binary mask to eliminate noise and fill holes. Finally, connected component analysis is used to extract the connected components of the moving targets.

[0116] S32: Scale estimation: Calculate the area of ​​the bounding rectangle of the connected components of a moving target in consecutive video frames. , as a scale quantification indicator. The scale estimation formula is: ,

[0117] in, Let be the connected component of the moving target in frame t. Classify the target scale according to a preset threshold:

[0118] ,

[0119] Among them, the small target area threshold =500 Large target area threshold =5000 These thresholds are based on statistical settings for typical surveillance scenarios. For example, at 1080p resolution, the target area of ​​pedestrians is mostly distributed between 500 and 5000. between.

[0120] S33: Dynamic Sampling Density Adjustment: Based on the mapping relationship between the target scale and the sampling density, the sampling point density of the initial sampling line is dynamically adjusted. The sampling density adjustment formula is as follows:

[0121] ,

[0122] Among them, the initial sampling density Points / pixels are set for standard-scale targets to ensure complete and non-redundant sampling coverage. Let be the area of ​​the circumscribed rectangle of the connected region. The density enhancement factor is set based on the integrity requirements of small target detection (example value is 1.3, the value range can be adjusted according to the scenario). The density reduction factor is set to avoid redundancy in large target sampling (example value is 0.7, the value range can be adjusted according to the scenario).

[0123] Specific adjustment rules:

[0124] Small target areas: Increase the sampling point density according to the completeness requirements of small target detection (in the example, increase it to 1.3 times the initial density) to enhance the ability to capture details;

[0125] For large target areas: reduce the sampling point density to avoid sampling redundancy (in the example, reduce it to 0.7 times the initial density) to reduce computational redundancy;

[0126] Standard target area: Keep the initial density unchanged;

[0127] Density adjustment is achieved by modifying the sampling line point spacing. For example, if the initial point spacing is 2 pixels, the point spacing in small target areas is shortened to 1.4 pixels after adjustment, and the point spacing in large target areas is increased to 2.6 pixels.

[0128] The comparison results between the initial sampling line and the refined sampling line are as follows: Figure 4 , Figure 5As shown, by estimating the target scale and dynamically adjusting the sampling point density, the refined sampling line has a high sampling point density at small targets and important areas, and a low sampling point density at large targets and unimportant areas.

[0129] This embodiment effectively balances the capture of details of small targets and the computational efficiency of large targets through scale-adaptive density adjustment, thus solving the problem of unbalanced capture of multi-scale target information.

[0130] Example 4:

[0131] This invention provides a preferred embodiment for executing S4, which establishes a weighted cumulative spatiotemporal flow model based on refined sampling lines, and calculates the weighted cumulative spatiotemporal flow of each sub-effective region. The specific steps are as follows:

[0132] S41: Basic Inflow / Outflow Flow Determination Preprocessing

[0133] To eliminate background interference and accurately locate the cross-regional behavior of moving targets, the following preprocessing operations must be performed first:

[0134] Motion change mask extraction: The motion difference between adjacent frames (frame t and frame t-1) is captured using the inter-frame difference method. The difference formula is as follows: ,in This represents the pixel grayscale value at coordinates (x, y) in frame t. Set the difference threshold. (Adapted for 8-bit grayscale surveillance video), binarize the difference results: This yields a binary mask containing only the regions of motion change. ;

[0135] Sampling line mask constraint: Based on the "refined sampling lines" generated in Example 3, construct a binary mask for the sampling line coverage area. :

[0136] ,

[0137] The valid motion region is filtered out by logical AND operation, excluding irrelevant background outside the sampling line: , where AL represents the fine sampling line region;

[0138] Calculation of the center of motion of the target: For each connected component (single moving target) in the graph, the centroid method is used to calculate the target center coordinates in frame t. :

[0139] ,

[0140] Where Conn represents the connected component range of a single moving target, and the difference in center coordinates between adjacent frames is the position change vector: ;

[0141] Sampling line normal vector determination: For a linear sampling line, let the line segment direction vector be... Then the normal vector (Pointing to the outside of the region); For corner-shaped sampling lines, calculate the normal vector for each straight line segment on both sides of the corner according to the straight line rule; For arc-shaped sampling lines, let the tangent vector of the arc at the sampling point (x,y) be... Then the normal vector (Pointing to the outer side of the arc).

[0142] S42: Basic Inbound / Outbound Flow Determination and Execution

[0143] Determine the basic inflow and outflow rates based on the directional relationship between the position change vector and the normal vector. (k=1 corresponds to the boundary sub-effective region, k=2 corresponds to the internal sub-effective region), the specific process is as follows:

[0144] First, calculate the vector dot product, which is expressed as follows: ,in , Normal vector The x and y components. Then, based on the dot product result, ... Assignment:

[0145] ,

[0146] in, This is the vector representing the positional change of the moving target's center in adjacent frames. This is the normal vector pointing from the sampling line to the outside of the region, i.e., "entering the region". Assigned value "Exit the area" Assigned value In other cases, the value is assigned to 0.

[0147] S43: Weighting Coefficient Matching and Assignment

[0148] To accurately quantify motion information, two types of weighting coefficients are set to ensure precise correlation with target scale and regional importance:

[0149] Target Scale Weight The values ​​are assigned based on the scale determination results of Example 3, following the principle of "small target weight > standard target weight > large target weight" (example values:). ):

[0150] ,

[0151] Where TSize represents the target scale category. Indicates the weight of small objectives, Indicates the standard target weight, The weights of the large objective are all coefficients greater than 0, and satisfy the following conditions: ;

[0152] Regional importance weight The assignment of values ​​based on the sub-effective region division results of Example 2 follows the principle of "boundary sub-effective region weight > internal sub-effective region weight" (example values: ):

[0153] ,

[0154] Where k=1 corresponds to the boundary sub-effective region, and k=2 corresponds to the internal sub-effective region. Represents the effective region weight of the boundary sub-region. The weights of the internal sub-regions are all coefficients greater than 0, and satisfy the following conditions: .

[0155] S44: Weighted Flow and Cumulative Flow Calculation

[0156] Single-frame, single-region weighted traffic: calculated as "base traffic × scale weight × region weight", the formula is as follows: Example: A small target enters the effective region of the boundary sub-region. The value assigned is ,but: ;

[0157] Weighted cumulative spatiotemporal flow: Initialize the cumulative flow of the first frame Starting from frame 2, calculate according to the frame-by-frame accumulation formula: Example: Same valid sub-region in frame 2 The cumulative traffic in frame 2 is: ;

[0158] This embodiment improves the accuracy of motion information quantification by integrating a weighted model that combines target scale and regional importance, providing a reliable basis for motion segmentation.

[0159] Example 5:

[0160] This invention provides a preferred embodiment that executes step S5, performing adaptive segmentation and pruning operations on motion segments based on weighted cumulative spatiotemporal flow and a preset threshold, and outputting a high-quality motion segment list. The specific steps are as follows:

[0161] S51: Threshold setting based on statistical experiments

[0162] To set a preset minimum duration threshold and preset noise threshold To provide a scientific and reliable basis for value selection, we conducted a systematic statistical experiment, the specific process of which is as follows:

[0163] Data Acquisition and Preprocessing: Over 50 hours of real surveillance video data were collected from 8 different typical monitoring scenarios. This data covers different lighting conditions, different weather conditions, and common environmental noises such as swaying leaves and changes in light and shadow.

[0164] Sample labeling and definition:

[0165] Typical noise segments: defined as short frame sequences caused by unwanted target motion that are misjudged as valid motion by the basic detection algorithm;

[0166] Invalid motion segments: defined as motion segments in which a real moving target is detected, but the movement is brief, does not enter the core monitoring area, or does not involve any behavior of monitoring value;

[0167] By combining automated algorithm initial screening with manual fine-tuning, a total of 356 typical noise segments and 187 invalid motion segments were extracted from the above video data;

[0168] Statistical analysis process:

[0169] Calculate the duration of all noise segments To obtain the expected value of its duration. =0.26 seconds;

[0170] Calculate the mean weighted cumulative spatiotemporal flow of each sub-effective region during the duration of all invalid motion segments. To obtain its expected value =0.28;

[0171] Threshold determination and basis:

[0172] Based on the above statistical results, in order to achieve the best balance between effectively filtering noise and retaining valuable motion segments, in a preferred embodiment of the present invention, a preset minimum duration threshold is used. Set to 0.3 seconds, and set the preset noise threshold. The value is set to 0.3. This value provides an appropriate safety margin based on the statistical expectation, ensuring robust performance in most scenarios.

[0173] S52: Motion Segmentation

[0174] Weighted cumulative spatiotemporal flow curves based on each sub-effective region Determine moving frames and still frames:

[0175] like If the frame is moving, it is considered a moving frame; otherwise, it is considered a still frame. Continuous moving frames constitute a motion segment. For example, from frame... arrive Continuous motion frame sequence This is a candidate motion segment.

[0176] S53: Noise Clip Pruning

[0177] For the segmented motion fragments, apply the following pruning rules: if the fragment duration is long... If the weighted cumulative spatiotemporal flow mean of all sub-valid regions within the segment is... If it is a noise segment, then it is determined to be a noise segment; among which, the weighted cumulative spatiotemporal flow mean The calculation formula is:

[0178] ,

[0179] Where T represents the duration of the motion segment. This is the starting frame index for the motion segment. This is the index of the last frame of the motion segment. This represents the weighted traffic of the k-th sub-valid region in the t-th frame;

[0180] Pruning operations directly remove noisy segments and retain valid motion segments. For example... Figure 6 As shown, there are noisy segments in frames 408 to 416 along the boundary fine-sampling line. After pruning, as shown... Figure 7 As shown, this ensures that the output is a high-quality motion clip.

[0181] S54: Output a list of high-quality motion clips

[0182] The pruned motion segments are output as a list in chronological order, with each segment containing a start frame, an end frame, and weighted cumulative spatiotemporal flow information. This embodiment effectively removes noisy segments and improves the accuracy and reliability of motion segment extraction through an adaptive pruning mechanism based on statistical thresholds.

[0183] Experiment and Results Analysis:

[0184] The operating environment for this experiment was configured as follows: Intel(R) Xeon(R) Gold 6226R processor (2.90GHz), CentOS 7 operating system, and NVIDIA Tesla V100S-PCIE-32GB professional computing graphics card. The video data used in this experiment consisted of self-collected surveillance video data and publicly available online datasets. A total of 10 video clips covering classic scenes (playground, street, campus road, shopping mall) were tested. Figure 8The video shows a portion of the experiment. It can be seen that when the target crosses the sampling line and enters the sub-effective region, the weighted cumulative spatiotemporal flow increases, and when the target crosses the sampling line and exits the sub-effective region, the weighted cumulative spatiotemporal flow decreases.

[0185] Table 1 records the comparison results between the algorithm of this invention and the current mainstream algorithms. The evaluation indicators are F1 and running time. It can be seen from the observation that the method of this invention has the best average F1 and average running time results in the experimental video, reaching 83.3% and 52.3s respectively. Compared with other methods, the method can accurately and quickly extract motion segments.

[0186] Table 1 Comparison of Experimental Results

[0187]

[0188] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The various preferred features described above can be used in any combination without conflict.

Claims

1. A method for extracting motion segments from surveillance video based on multi-scale adaptive sampling lines, characterized in that, Includes the following steps: S1: Input the surveillance video to be processed, and use the multi-scale adaptive threshold Canny edge detection operator to perform edge detection on the surveillance video frames to obtain the effective activity area contour of the surveillance scene. S2: Based on the outline of the effective active area, construct an initial sampling line that adapts to the shape of the area. At the same time, divide the effective active area into a boundary sub-effective area and an internal sub-effective area according to the distribution of the initial sampling line. The boundary sub-effective area is the annular area between the boundary sampling line and the internal sampling line, which is used to accurately sense the entry and exit events of moving targets. The internal sub-effective area is the central area inside the internal sampling line, which is used to continuously track the activity state of moving targets inside the area. The region shapes include straight-line regions, corner regions, and curved regions, wherein: Linear regions use line segment sampling lines to ensure that the overall sampling line closely matches the contour of the effective active area; The corner area includes L-shaped corridor corners and T-shaped passage intersections. For the corner area, a polyline splicing sampling line is used, and the bend angle of the polyline splicing sampling line is consistent with the actual angle of the corner area. The arc-shaped region includes an arc-shaped entrance and a circular monitoring area. For this region, an arc-fitting sampling line is used. The least squares method is employed to fit the arc contour, ensuring the fitting error meets the region's shape conformation accuracy requirements. This guarantees that the positioning deviation between the sampling line and the arc contour does not affect target tracking. The arc-fitting formula is: ， in This means finding the minimum value of variables O and R. This indicates that for N contour points P i Summation, P i Let O be the coordinates of the i-th contour point, O be the center of the fitted graph, and R be the radius of the fitted graph. This represents the Euclidean distance between two points; S3: Scale estimation of moving targets in surveillance video is performed by calculating the area of ​​connected components of the moving target in consecutive video frames and applying a preset small target area threshold S. min And large target area threshold S max Moving targets are classified into Small, Standard, and Large targets. Based on the target scale determination results of the current frame and the previous and next frames, the sampling point density of the initial sampling line in the current target area is dynamically and continuously adjusted to obtain a refined sampling line, so that the sampling line density can adapt to the scale changes of moving targets in the scene. S4: Based on the refined sampling line, establish a weighted cumulative spatiotemporal flow model and calculate the weighted cumulative spatiotemporal flow of each sub-effective region; The calculation process of the weighted cumulative spatiotemporal flow includes the following steps: (1) Determination of basic inflow and outflow rates For each video frame, the position of the moving target in each sub-effective region is tracked by inter-frame difference and intersection calculation of refined sampling lines, and the basic inflow and outflow rates are calculated. Defined as: , in, This is the vector representing the positional change of the moving target's center in adjacent frames. The sampling line is the normal vector pointing outwards from the region; when the target enters the region, Set the value to +1 and exit the region. The value is assigned as -1, and the value is assigned as 0 in other cases; (2) Weighting coefficient setting Target scale weight: This weight is set based on the contribution of the target scale to the detection results. The weight of a small target is greater than that of a standard target, which in turn is greater than that of a large target. The formula is set as follows: , Where TSize is the target scale category, W s Represents the weight of the small objective, W m Represents the standard objective weight, W l The weights of the large objective are all coefficients greater than 0, and satisfy the following conditions: ; Region importance weight: Based on the importance of region location to motion detection, the weight of the boundary sub-region is greater than that of the interior sub-region. Region importance weight W region The settings are as follows: , Where k=1 corresponds to the boundary sub-effective region, k=2 corresponds to the internal sub-effective region, W b W represents the effective region weight of the boundary sub-region. i The weights of the internal sub-regions are all coefficients greater than 0, and satisfy the following conditions: ; (3) Weighted flow calculation Weighted traffic of a single frame and a single region ; (4) Cumulative spatiotemporal flow calculation The weighted cumulative spatiotemporal flow is obtained by accumulating the weighted cumulative flow of each sub-effective region frame by frame, and the initial cumulative flow is set. The formula for calculating subsequent frames is: , in, The weighted cumulative spatiotemporal flow of the k-th sub-effective region in the t-th frame. The weighted cumulative spatiotemporal flow of the k-th sub-effective region in the (t-1)-th frame. The weighted traffic of the k-th sub-valid region in the t-th frame; S5: Based on the weighted cumulative spatiotemporal flow, and combined with the preset minimum duration threshold T min With preset noise threshold F noise It performs adaptive segmentation and pruning operations on motion segments and outputs a list of high-quality motion segments.

2. The method for extracting motion segments from surveillance video based on multi-scale adaptive sampling lines as described in claim 1, characterized in that, The method employs a multi-scale adaptive threshold Canny edge detection operator to perform edge detection on the monitoring video frames. Specifically, it uses multi-scale Gaussian filtering to adapt to the edge feature extraction of targets at different scales, and dynamically adjusts the dual thresholds with local grayscale statistics to adapt to changes in illumination. This effectively removes static background interference and dynamic illumination noise, while completely preserving the activity range boundary of the moving target, and finally accurately obtains the effective activity area contour of the moving target in the monitoring scene.

3. The method for extracting motion segments from surveillance video based on multi-scale adaptive sampling lines as described in claim 1, characterized in that, The scale estimation of moving targets in surveillance video is achieved by calculating the area of ​​the connected components of the moving target in consecutive video frames. The area of ​​the bounding rectangle of the moving target is used as a quantitative indicator of the area of ​​the connected components. The scale estimation formula is as follows: ,in Let be the connected component of the moving target in frame t. This represents calculating the area of ​​the bounding rectangle of this connected region and using it as... The quantization value used for target scale category determination, and the target scale category determination criteria are: , Among them, S min S is the threshold for the area of ​​small targets. max This is the threshold for the large target area.

4. The method for extracting motion segments from surveillance video based on multi-scale adaptive sampling lines as described in claim 1, characterized in that, The method of dynamically and continuously adjusting the sampling point density of the initial sampling lines in the current target area based on the target scale determination results of the current frame and the frames before and after it is specifically as follows: For each frame in the video sequence, the sub-effective region where the moving target is located is identified, and the sampling density adjustment formula is applied to update the sampling line density corresponding to the region in real time according to the real-time scale classification result of the target. The sampling density adjustment formula is as follows: , Where D init The initial sampling point density is the base number of sampling points per unit area set for the standard-scale target, ensuring complete and non-redundant sampling coverage of the standard target. S t Let S be the area of ​​the circumscribed rectangle of the connected region. min S is the threshold for the area of ​​small targets. max α is the area threshold for large targets, β is the density enhancement coefficient set based on the requirement of small target detection integrity, and β is the density reduction coefficient set based on the requirement of large target sampling redundancy avoidance. The specific adjustment rules are as follows: for small target areas, increase the sampling point density according to the requirement of small target detection integrity; for large target areas, reduce the sampling point density according to the requirement of sampling redundancy avoidance.

5. The method for extracting motion segments from surveillance video based on multi-scale adaptive sampling lines as described in claim 1, characterized in that, The preset minimum duration threshold T min With preset noise threshold F noise The settings are as follows: The preset minimum duration threshold T min Based on the statistical analysis of the duration of typical noise segments in the monitoring scenario, the final value was determined to be 0.3 seconds. The preset noise threshold F noise The weighted cumulative spatiotemporal flow mean, determined based on invalid motion segments, was ultimately set at 0.3, and can be expressed as follows: , in T represents the expected value. noise The duration of the noise segment. The weighted cumulative spatiotemporal flow mean of invalid motion segments; The specific steps for performing adaptive segmentation and pruning operations on motion segments are as follows: Motion segment division: Based on the weighted cumulative spatiotemporal flow curves of each sub-effective region, if That is, there exists a sub-effective region k such that the weighted cumulative spatiotemporal flow of this sub-effective region in frame t is If the value is greater than 0, the current frame is determined to be a moving frame; otherwise, it is a still frame. Once consecutive moving frames are identified, these consecutive moving frames are sequentially connected to form a sequence, which constitutes a motion segment. Noise segment pruning: For the segmented motion, if its duration is... Or the weighted cumulative spatiotemporal flow average of all valid sub-regions within this segment. If it is, it is identified as a noise segment and removed.

Images