Multi-target interactive behavior-preserving surveillance video condensation method based on semantic enhancement and spatiotemporal constraints

Abstract

The present invention discloses a method for concentrating surveillance videos based on semantic enhancement and spatiotemporal constraints to preserve multi-target interactive behaviors, which belongs to the field of video concentration technology and mainly comprises the following steps: obtaining the video background through a Gaussian mixture model MOG2, combined with random frame sampling and median fusion strategy; utilizing the DeepSORT algorithm and a deep learning model based on the CNN+GRU architecture to perform multi-target tracking and behavior recognition on the video, extracting target trajectories, spatial positions, movement directions, movement speeds, and behavior label features; judging the interactive behaviors between targets based on spatiotemporal features and semantic information, and dividing the targets with interactions into the same target tube group; integrating collision constraints and time domain constraints, assigning new time labels to the target tube group through a greedy strategy and a spatial hashing algorithm, and optimizing the temporal logic with the help of a temporal loss function; fusing the tube group with the background to generate a condensed video. The video concentration method of the present invention can accurately preserve the interactive behaviors between targets.

Description

Technical Field

[0001] The present invention relates to the technical field of video concentration, and in particular to a method for concentrating surveillance videos with multi-target interactive behavior preservation based on semantic enhancement and spatiotemporal constraints. Background Art

[0002] With the widespread use of surveillance equipment and the realization of 24 / 7 continuous operation, the storage and efficient retrieval of massive amounts of video data have become urgent challenges. Video compression technology, which reorders target trajectories in time and space, shortens video length while retaining key information, has become a research hotspot in this field. However, existing technologies still have the following core pain points in practical applications:

[0003] When a video contains a large number of objects, traditional methods struggle to accurately extract target tubes, often resulting in frequent object collisions in the condensed video, significantly reducing the viewing experience. For example, the crowd detection method proposed by Li et al. attempts to separate dense and sparse scenes, but it fragments the target tubes into multiple segments, resulting in lost trajectories and an inability to fully preserve target motion information. Furthermore, some methods reduce collisions by expanding the video background or resizing the objects, but these methods only work for scenes with fixed backgrounds or small objects and have poor adaptability to complex dynamic scenes.

[0004] Existing technologies often use a single target tube as a processing unit, ignoring the interactive relevance between targets. This leads to a logical discontinuity in the interactive behaviors in the condensed video. For example, He et al. used an image-based approach to avoid collisions, but failed to consider the semantic interactions between targets. Nie et al.'s background expansion strategy can only handle a small number of targets and cannot preserve complex interactions such as multi-person interactions. Although Li et al. proposed combining spatial distance and a fixed threshold to determine interactive behaviors, and Namitha et al. further employed dynamic threshold optimization, the threshold still suffers from insufficient adaptability, making it difficult to accurately preserve semantic interactions such as "waving" or "pointing at someone."

[0005] Video compression requires a balance between shortening video duration and minimizing collisions, but existing methods often compromise one over the other. For example, while methods like particle swarm optimization and event reordering can reduce collision rates, their compression rates are low, making them inadequate for rapid previews of large-scale data. Some methods, in pursuit of high compression rates, over-compress the timeline, resulting in significant object overlap and a poor user experience.

[0006] In summary, existing video condensation technologies face significant technical gaps in the coordinated application of semantic information and spatiotemporal constraints. For instance, in the process of determining target interactive behaviors, there's a lack of a semantically-based spatiotemporal constraint fusion mechanism, making it difficult to accurately identify interactions occurring at a certain distance, such as "waving." Furthermore, in the area of ​​spatiotemporal collision optimization, a systematic solution that balances behavioral semantics with spatiotemporal logic has yet to be developed, leading to frequent problems such as "spatial-temporal semantic inconsistencies" in condensed videos. Developing a framework for collaboratively determining semantic information and spatiotemporal constraints to accurately identify and preserve interactive behaviors has become a key support point for overcoming existing technical bottlenecks. Summary of the Invention

[0007] In order to solve the related problems of existing video concentration methods, the present invention proposes a multi-target interactive behavior preservation surveillance video concentration method based on the combination of semantic enhancement and spatiotemporal constraints, aiming to solve the problems of existing video concentration technology over-relying on spatiotemporal information, ignoring semantic information for interactive behavior judgment, and poor concentration effect in multi-target scenarios.

[0008] The present invention provides a multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints, the method comprising the following steps:

[0009] S1: Input the video to be processed and obtain the video background image;

[0010] S2: Perform multi-target tracking and target behavior recognition on the input video, obtain the target trajectories of all targets, and record the spatial position, movement direction, movement speed, and behavior label features of the corresponding targets;

[0011] S3: Determine whether there is interaction between targets based on the spatiotemporal features and semantic information of the targets. The spatiotemporal features include the spatial distance, movement direction, and movement speed of the targets. The semantic information is the matching degree of the target behavior label. Based on the spatiotemporal similarity formula According to the dynamic adjustment formula of semantic weight, targets whose ratio of the number of interactive frames to the number of shared frames exceeds the preset ratio ρ are divided into the same target group. The dynamic adjustment formula of semantic weight is as follows:

[0012] ,

[0013] Among them, O i and O j , is the target tube, dist_sim(·) is the spatial distance similarity, dir_sim(·) is the moving direction similarity, speed_sim(·) is the moving speed similarity, α, β, γ are the spatiotemporal weight coefficients, α+β+γ=1, W S is the semantic information weight, b i , b jis the target behavior label, match_score(·) is the behavior matching score, B P A set of preset interactive behaviors, including "waving", "pointing at the other person", "nodding / shaking head", "OK gesture", "bowing", "saluting", and "heart gesture".

[0014] S4: Based on The collision function combines spatial hash grid acceleration detection with a greedy strategy to assign new time labels to the target tube group;

[0015] Among them, Collision(·) represents the collision value of the target tube group, Overlap(·) represents the spatial overlap of the target tube group in the picture, and the spatial overlap is obtained by calculating the overlap rate of the detection frame. TimeOverlap(·) represents the temporal overlap of the adjusted time interval. and are the i-th and j-th target tube groups respectively, S i and S j Target tube group space area ， T i ′ and T j ′ is the time interval reallocated to the target tube group, which is used to calculate the time overlap of the adjusted time interval;

[0016] S5: Fusion the arranged target tube group with the background to generate a condensed video.

[0017] Preferably, the video background image is obtained by using a Gaussian mixture model MOG2 and combining random frame sampling and median fusion strategy to perform background extraction and update.

[0018] Preferably, the multi-target tracking and target behavior recognition are performed on the input video, the DeepSORT algorithm is used for multi-target tracking, and the deep learning model based on the CNN+GRU architecture is used to extract features and classify behaviors of the target area to obtain the behavior label of the target.

[0019] Preferably, the temporal and spatial characteristics and semantic information of the integrated targets are used to determine whether there is interaction between targets, specifically including:

[0020] Determine whether there is a shared frame between targets based on the extracted target trajectory features. If not, it is determined that there is no interaction behavior;

[0021] For target pipes with shared frames i and O j , based on the spatial distance, moving direction, and moving speed spatiotemporal characteristics between targets, a preliminary interactive judgment is made. The formula is as follows:

[0022] ,

[0023] Among them, the target tube is used to describe the spatiotemporal trajectory characteristics of the target in the video sequence. It integrates the position, appearance and behavior information of the target in consecutive frames to form a "tubular" representation throughout the time dimension. The target tube group is a collection of multiple target tubes. Shared frames refer to video frames in which two targets exist at the same time, and interactive frames refer to video frames in which there is interaction between targets. s1 represents spatiotemporal similarity, dist_sim(·) is spatial distance similarity, dir_sim(·) is movement direction similarity, speed_sim(·) is movement speed similarity, α, β, γ are spatiotemporal weight coefficients, α+β+γ=1;

[0024] If the spatiotemporal similarity s1 in a certain frame exceeds the spatiotemporal similarity threshold T st , then the number of interactive frames C ij Add 1; otherwise, check the behavior label of the target in the frame. If the behavior label belongs to the preset interactive behavior set B p , then the number of interactive frames is C ij +=W S1 ×1 calculation, at this time W S1 ≥0.7, if the behavior label does not belong to B p , then the number of interactive frames is C ij +=W S2 ×1 calculation, at this time W S2 ≤0.3, W S1 and W S2 The behavior labels belong to B P and behavior tags do not belong to B P The weight of semantic information in the situation;

[0025] When the ratio of the number of interactive frames to the number of shared frames between targets exceeds the preset ratio ρ, it is determined that there is interaction between targets and they are divided into the same target group. The determination conditions are as follows:

[0026] ,

[0027] Among them, SFrames(·) represents the target tube O i and O j The number of shared frames.

[0028] Preferably, when judging the interaction between targets, the temporal and spatial characteristics and semantic information of the targets are comprehensively considered to determine whether there is interaction between the targets, wherein:

[0029] The spatiotemporal feature judgment conditions include quantitative calculation based on spatiotemporal similarity and preset threshold judgment, as follows:

[0030] dist_sim(·) is the spatial distance similarity, which is calculated based on the ratio of the distance between the center points of the two target detection frames to the average height. When the ratio is less than 1.17, dist_sim(·) is normalized according to the rule. The smaller the ratio, the closer the similarity is to 1. dir_sim(·) is the motion direction similarity, which is calculated based on the angle between the moving directions of the two target frames. When the angle is less than 60 degrees, dir_sim(·) is converted into a similarity value through the cosine function. The larger the cosine value of the angle, the higher the similarity. speed_sim(·) is the speed similarity, which is calculated based on the normalized difference between the moving speeds of the two targets. The smaller the speed difference, the closer the similarity is to 1.

[0031] Semantic information weight W S The arrangement includes: when the target's behavior belongs to the preset interactive behavior set {"waving", "pointing at the other party", "nodding / shaking head", "OK gesture", "bowing", "saluting", "making a heart"}, the weight of semantic information is increased to more than 70%, and the weight of spatiotemporal constraints is correspondingly reduced to less than 30%; if the target does not perform the preset behavior, the weight of semantic information is reduced to less than 30%, and the weight of spatiotemporal constraints is increased to more than 70%. The specific dynamic adjustment is based on the following formula:

[0032] ,

[0033] Among them, W S is the semantic information weight, b i , b j is the target behavior label, B p is the preset interactive behavior set, match_score(·) is the behavior matching score;

[0034] The semantic information weight W S Used to adjust the spatiotemporal similarity calculation in the target group division: when the target behavior belongs to the preset interactive behavior set B p When , the spatiotemporal weight coefficients α, β, and γ are as follows: Dynamic adjustment is performed to increase the proportion of semantic constraints in the total similarity calculation. The specific similarity formula is as follows:

[0035] ,

[0036] Where s2 represents semantic similarity, the same behavior label is 1, and different is 0, dist_sim(·) is the spatial distance similarity, dir_sim(·) is the movement direction similarity, speed_sim(·) is the movement speed similarity, and α+β+γ=1.

[0037] Preferably, the The collision function combines spatial hash grid acceleration detection with a greedy strategy to assign new time labels to the target pipe group. Specifically, it includes:

[0038] Sort by the original start time of the target tube group, and assign new time labels to the tube groups in chronological order after sorting, giving priority to preserving the original timing logic;

[0039] Based on the combination of greedy strategy and spatial hash algorithm, a new time position with minimum space-time collision characteristics is searched, and the space-time collision function is defined:

[0040] ,

[0041] Among them, Collision(·) represents the collision value of the target tube group, Overlap(·) represents the spatial overlap of the target tube group in the picture, and the spatial overlap is obtained by calculating the overlap rate of the detection frame. TimeOverlap(·) represents the temporal overlap of the adjusted time interval. and are the i-th and j-th target tube groups respectively, S i and S j is the target tube group space area, T i ′ and T j ′ is the time interval reallocated to the target tube group, which is used to calculate the time overlap of the adjusted time interval.

[0042] Preferably, the combined spatial hash grid accelerated detection specifically includes:

[0043] Divide the video spatiotemporal domain into a regular grid G m×n , for each target tube group g i Create a spatial hash table H i ,The hash table is used to quickly determine the spatial overlap relationship between the target tube groups and detect the spatial collision risk. The logical judgment is as follows:

[0044] If there is at least one grid cell coordinate (x, y) such that the target tube group g i Corresponding hash table H i Record at (x,y) position, with target tube group g j Corresponding hash table H j There is an intersection of the records at the (x, y) position, that is, H i (x,y)∩H j (x,y)≠0, then determine g i With g j There is a risk of space collision;

[0045] Among them, there is at least one grid coordinate (x, y), which represents the regular grid G m×n There exists a coordinate point that meets the conditions, Hi (x,y) and H j (x,y) represents the target tube group g i and g j The records in the small grid (x,y).

[0046] Preferably, allocating a new time tag to the target tube group specifically includes:

[0047] Find the time when the target first appears in each target tube group, use this time as the start time of the corresponding target tube group, and sort them from small to large;

[0048] Prioritize preserving the spatiotemporal integrity of the target tube group, minimize the timeline span according to the original temporal logic, and define the temporal loss function:

[0049] ,

[0050] Where i and j represent the index of the target tube group, Indicates traversing "all two combinations without duplication", t i ′ and t j ' is the adjusted timestamp, I is the indicator function, and the formula is as follows:

[0051] , used to quantify the loss caused by timing reversal.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects:

[0053] 1. The present invention provides a multi-target interactive behavior preservation surveillance video concentration method based on semantic enhancement and spatiotemporal constraints. The background is extracted by combining the Gaussian mixture model MOG2 with random frame sampling and median fusion strategy, and the DeepSORT algorithm and a deep learning model based on the CNN+GRU architecture are used to realize multi-target tracking and behavior feature extraction. Compared with the single background modeling method and simple target tracking algorithm in the existing technology, the robustness of background update and the accuracy of target behavior recognition are significantly improved, providing more accurate basic data for interactive behavior analysis.

[0054] 2. The present invention proposes an interactive behavior judgment mechanism that integrates semantic information and spatiotemporal characteristics. By dynamically adjusting the semantic weight and spatiotemporal constraint weight: when the behavior belongs to a preset set, the semantic weight is ≥70%, and interactive behaviors with clear semantics such as "waving", "saluting", and "pointing at the other party" are prioritized. This effectively solves the problem of missing semantic events caused by the existing technology that only relies on spatiotemporal distance to judge interactive behaviors, and improves the event integrity of the monitored content.

[0055] 3. The present invention designs a spatiotemporal scheduling method based on a greedy strategy and a spatial hash algorithm. By dividing the video frames into regular grids and establishing a spatial hash table, it realizes the rapid detection of target tube group collisions. Combined with the timing loss function, it minimizes the original timing deviation. While effectively compressing the original video, it maintains the spatiotemporal integrity and timing logic of the target behavior. Compared with the brute force collision detection and disordered scheduling methods in the existing technology, it greatly improves the timing rationality and viewing experience of the condensed video.

[0056] 4. The present invention selected 10 representative videos from VISOR, CAVIAR, KTH and self-built datasets for experimental verification. The experimental results show that compared with the most advanced existing methods, the method proposed in the present invention is the best in preserving target interactive behaviors. Compared with other methods, it can accurately preserve the interactive behaviors between targets and improve the viewing experience of condensed videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The advantages and novelties of the present invention will become more apparent upon reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0058] Figure 1 A flowchart of a multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints according to an embodiment of the present invention;

[0059] Figure 2 A flowchart of interaction behavior determination according to an embodiment of the present invention;

[0060] Figure 3 An interactive behavior scenario based on spatiotemporal information weight-dominated judgment according to an embodiment of the present invention;

[0061] Figure 4 An interactive behavior scenario based on semantic information weight-dominated judgment according to an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the completion of spatiotemporal constraints and time tag scheduling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to understand the present invention more clearly, relevant descriptions are given below in conjunction with specific embodiments.

[0064] like Figure 1 FIG. 1 is a flowchart of a multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints provided by the present invention, including:

[0065] S1: Input the video to be processed and obtain the video background image;

[0066] S2: Perform multi-target tracking and target behavior recognition on the input video, obtain the target trajectories of all targets, and record the spatial position, movement direction, movement speed, and behavior label features of the corresponding targets;

[0067] S3: The temporal and spatial characteristics and semantic information of the targets are integrated to determine whether there is interaction between targets. Targets whose ratio of the number of interaction frames to the number of shared frames exceeds a preset ratio ρ are divided into the same target group.

[0068] S4: Based on The collision function combines spatial hash grid acceleration detection with a greedy strategy to assign new time labels to the target tube group;

[0069] S5: Fusion the arranged target tube group with the background to generate a condensed video.

[0070] Example 1:

[0071] The present invention provides a preferred embodiment for performing S1, inputting the video to be processed, and obtaining the video background image. The background is extracted using the Gaussian mixture model MOG2 combined with random frame sampling and median fusion strategy. The specific steps are as follows:

[0072] S11: Randomly sample the input video at 10% of the frame rate to obtain 50-100 frames as the background training set;

[0073] S12: Use the MOG2 model to perform Gaussian distribution modeling on each frame pixel and initialize 3-5 Gaussian components to represent the background;

[0074] S13: Update the background model every 100 frames using the median fusion strategy. The formula is:

[0075] ,

[0076] Among them, B old is the old background model, F sample The frames are randomly sampled to ensure robustness to dynamic backgrounds (such as swaying leaves).

[0077] Example 2:

[0078] The present invention provides a preferred embodiment to execute S2, perform multi-target tracking and target behavior recognition on the input video, obtain the target trajectories of all targets, and record the spatial position, movement direction, movement speed, and behavior label features of the corresponding targets.

[0079] The first step is to build a CNN-GRU model for behavioral category training. The dataset integrates public and self-built data. Among them, the public dataset selects some categories of KTH datasets and Weizmann datasets, supplemented by 7 categories of self-built behavioral action datasets (covering "waving" and "pointing at each other" interactive behaviors, forming a preset set B p After data preprocessing, it is input into the CNN-GRU model training, and the target detection frame and the corresponding behavior label b are output. i ;

[0080] In the second step, DeepSORT algorithm is used for multi-target tracking. DeepSORT matches the target appearance features (256-dimensional features extracted by ResNet50) through cosine distance, combines with Kalman filter to predict the trajectory, and generates the target tube. i , accurately record the spatial position (x, y) and movement direction of each frame , speed v, providing basic data for subsequent interactive behavior analysis.

[0081] Example 3:

[0082] The present invention provides a preferred embodiment for executing S3, comprehensively analyzing the spatiotemporal characteristics and semantic information of the targets to determine whether there is interaction between the targets, and dividing the targets whose ratio of the number of interaction frames to the number of shared frames exceeds a preset ratio ρ into the same target control group.

[0083] like Figure 2 The figure shows the flow chart of interaction behavior judgment in this embodiment. It includes:

[0084] S31: Start. The process starts with the "Start" node. Before this, the statistics of the number of shared frames SFrames and the number of interactive frames C ij Initialization and spatiotemporal similarity threshold T st And the setting of the interactive frame ratio threshold ρ.

[0085] S32: Shared frame filtering: First, determine whether the frame is a shared frame between two target tubes. If not, directly determine that the current frame "does not have interactive behavior". If so, proceed to the next stage of determination.

[0086] S33: Calculate the spatiotemporal similarity and determine whether it meets the spatiotemporal similarity threshold T st . Calculate the spatiotemporal similarity s1:

[0087] ,

[0088] Where, , where d is the distance between the center points of the two target detection boxes, h avg is the average height; when d<1.17h avg When dist_sim≥0.5. , when the angle between the two target motion directions is less than 60°, dir_sim≥0.5; are the motion direction angles of targets i and j respectively. , where v i 、v j is the moving speed of target tube i, j, v max The total spatiotemporal similarity s1 is the weighted sum of the above three items, with weights α=0.4, β=0.3, and γ=0.3. The spatiotemporal similarity threshold T is set according to the experiment. st =0.6, used to determine whether the spatiotemporal similarity meets the conditions. If the conditions are met, the corresponding number of interaction frames C ij Add 1, otherwise, enter the behavior semantic similarity judgment.

[0089] S34: Determine whether the current frame satisfies the interactive frame based on the semantic similarity.

[0090] When the target behavior label When the semantic weight W S Improved by the following formula: , when the target behavior label When the semantic weight W S Improved by the following formula: , where match_score(·) is the behavior matching score (1 for a complete match). At this time, the spatiotemporal weight coefficients α, β, and γ are calculated as Dynamic adjustment, the total similarity S is calculated as follows: , where s2 = 1 (same behavior labels) or 0 (different behavior labels). The number of interaction frames C is ij Add the corresponding proportional weights.

[0091] S35: Final Interaction Behavior Determination. After completing the interaction frame counting logic, determine whether the interaction frame ratio is ≥ ρ (i.e., whether the ratio of the number of interaction frames to the total number of shared frames meets the standard). If so, it is determined that "interaction behavior exists" and the process ends. If not, it is determined that "no interaction behavior exists" and the process ends.

[0092] like Figure 3 As shown in the figure, when two people are walking side by side, the interaction is determined by the spatiotemporal feature dominant logic: the spatiotemporal information weight of the spatial position, moving direction and moving speed of the two targets is high, and the calculated spatiotemporal similarity s1 meets the threshold T st Therefore, the two target tubes are divided into the same target tube group and enter the subsequent concentration process. Figure 4 As shown, the spatial distance and moving direction of the two people do not meet the conditions of time and space constraints, and the semantic similarity weight W is used. S Accumulate interactive frame count Cij When the interaction frame ratio meets the threshold ρ, it is determined that the two targets interact and are also divided into the same target group for processing.

[0093] Example 4:

[0094] The present invention provides a preferred embodiment to perform S4, based on The collision function combines spatial hash grid accelerated detection with a greedy strategy to assign new time labels to the target tube group.

[0095] This embodiment focuses on the spatiotemporal coordination and trajectory rearrangement of target tube groups during surveillance video compression. By rationally allocating time tags and optimizing timing loss, the original timing logic of target interaction behaviors is preserved to the greatest extent possible while compressing the video. The specific implementation is as follows:

[0096] S41: Time label allocation for target tube groups. The time label allocation for target tube groups is centered on ensuring the rationality of video timing and is performed in two steps: first, sort by original timing, and sort by the time when the target tube groups first appear in the original video, giving priority to preserving the original time order to ensure that the basic timing logic of the interactive behavior is not destroyed; second, greedy strategy + spatial hashing optimizes the position. In order to further improve the visual rationality of the compressed video, a greedy strategy combined with a spatial hashing algorithm is used to search for the optimal time position of the tube group in the condensed video. Specifically, the video frame space is divided into a 16×16 grid, and for each tube group g i Construct spatial hash table H i , record the grid coordinates occupied by the tube group; define the collision function at the same time:

[0097] ,

[0098] Among them, Overlap(S i ,S j ) Calculate tube group g i With g j The intersection-over-union (IOU) of the detection box measures the degree of spatial overlap; TimeOverlap (T' i ,T' j ) calculates the overlap rate of the two time intervals to measure the degree of temporal conflict. This function is used to assess the collision risk between pipe groups and assist the greedy strategy in finding the time position with the lowest conflict for each pipe group.

[0099] S42: Timing loss optimization. To strictly constrain the timing relationship of the rearranged tube group, define the timing loss function: ,in, is the indicator function, when t' i >t' j (i.e. rearranged tube group g iStart time later than g j , conflicts with the original timing), , otherwise 0. This function quantifies the deviation between the rearranged tube group timing and the original timing by accumulating the time difference of the timing conflict. In practical applications, this can ensure a high degree of preservation of the original timing and effectively maintain the timing logic of the interactive behavior.

[0100] Figure 5 This is a schematic diagram of an embodiment of the present invention after the spatiotemporal constraints and time tag scheduling are completed. The large rectangle represents the condensed video, the arrows x and y represent the horizontal and vertical coordinates in the video frame, and t represents the time axis of the video. The target tube groups g1 to g6 in the figure represent the target tube groups whose start times have been arranged through collision constraints and time domain constraints. The start times are t1', t2'...t6', respectively.

[0101] Through the above-mentioned spatiotemporal scheduling and trajectory rearrangement strategy, we can not only compress the video length, but also ensure the comprehensibility of interactive behaviors in terms of spatial layout and temporal logic, laying the foundation for the subsequent generation of condensed videos.

[0102] Embodiment 5:

[0103] The present invention provides a preferred embodiment to execute S5, fusing the arranged target tube group with the background to generate a condensed video.

[0104] The visual conflict between the target and the background is resolved through texture-aware dynamic weighted fusion. First, background texture analysis is performed, and the Sobel operator is used to calculate the gradient amplitude of the background area to quantify the texture complexity. Fusion weights are then dynamically determined based on the texture complexity. For complex texture areas such as crowds and traffic, the target transparency is reduced (to a minimum of 0.7), while for smooth areas such as walls and the sky, the transparency is increased (to a maximum of 1.0). The specific formula is as follows:

[0105] ,

[0106] Here, gradient_norm is the normalized gradient magnitude in the range [0,1]. Finally, pixel-level fusion is performed using numpy to perform a weighted overlay operation on the target frame and the background frame to avoid the visual fragmentation problem caused by "hard pasting" and allow the target and background to blend naturally.

[0107] To verify the effectiveness of the above examples, we conducted extensive experiments. We primarily compared the results of interactive behavior. To enhance the convincingness of the experimental results, we selected 10 representative videos from the VISOR, CAVIAR, KTH, and self-built datasets, encompassing a variety of scenarios.

[0108] The more accurate the interactive behavior is, the more accurate the interactive behavior is retained in the condensed video. Our method is compared with the method DT that judges based on the minimum distance and dynamic threshold, and the method RD that judges based on the spatial distance and movement direction between targets. The experimental results are shown in Table 1. As can be seen from the table, our method is more accurate in the experimental video. The average value is the highest, reaching 0.778. Compared with other methods, this method can accurately preserve the interaction behavior between targets.

[0109] Table 1 Compared with the advanced methods Comparison results

[0110]

[0111] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and that persons skilled in the art may make various variations or modifications within the scope of the claims without affecting the essence of the present invention. The various preferred features described above may be used in any combination as long as they do not conflict with each other.

Claims

1. A multi-target interactive behavior preservation surveillance video concentration method based on semantic enhancement and spatiotemporal constraints, characterized by The steps include: S1: Input the video to be processed and obtain the video background image; S2: Perform multi-target tracking and target behavior recognition on the input video, obtain the target trajectories of all targets, and record the spatial position, movement direction, movement speed, and behavior label features of the corresponding targets; S3: Determine whether there is interaction between targets based on the spatiotemporal features and semantic information of the targets. The spatiotemporal features include the spatial distance, movement direction, and movement speed of the targets. The semantic information is the matching degree of the target behavior label. Based on the spatiotemporal similarity formula According to the dynamic adjustment formula of semantic weight, targets whose ratio of the number of interactive frames to the number of shared frames exceeds the preset ratio ρ are divided into the same target group. The dynamic adjustment formula of semantic weight is as follows: ， Among them, O i and O j , is the target tube, dist_sim(·) is the spatial distance similarity, dir_sim(·) is the moving direction similarity, speed_sim(·) is the moving speed similarity, α, β, γ are the spatiotemporal weight coefficients, α+β+γ=1, W S is the semantic information weight, b i , b j is the target behavior label, match_score(·) is the behavior matching score, B P A set of preset interactive behaviors, including "waving", "pointing at the other person", "nodding / shaking head", "ok gesture", "bowing", "saluting", and "heart gesture". S4: Based on The collision function combines spatial hash grid acceleration detection with a greedy strategy to assign new time labels to the target tube group; Among them, Collision(·) represents the collision value of the target tube group, Overlap(·) represents the spatial overlap of the target tube group in the picture, and the spatial overlap is obtained by calculating the overlap rate of the detection frame. TimeOverlap(·) represents the temporal overlap of the adjusted time interval. and are the i-th and j-th target tube groups respectively, S i and S j Target tube group space area ， T i ′ and T j ′ is the time interval reallocated to the target tube group, which is used to calculate the time overlap of the adjusted time interval; S5: Fusion the arranged target tube group with the background to generate a condensed video.

2. The multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints according to claim 1 is characterized in that: The video background image is obtained by using a Gaussian mixture model MOG2 and combining random frame sampling and median fusion strategy to perform background extraction and update.

3. The multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints according to claim 1 is characterized in that: The method performs multi-target tracking and target behavior recognition on the input video, adopts the DeepSORT algorithm to perform multi-target tracking, and uses a deep learning model based on the CNN+GRU architecture to perform feature extraction and behavior classification on the target area to obtain the target behavior label.

4. The multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints according to claim 1 is characterized in that: The temporal and spatial characteristics and semantic information of the comprehensive target are used to determine whether there is interaction between the targets, specifically including: Determine whether there is a shared frame between targets based on the extracted target trajectory features. If not, it is determined that there is no interaction behavior; For the target pipe with shared frames i and O j , based on the spatial distance, moving direction, and moving speed spatiotemporal characteristics between targets, a preliminary interactive judgment is made. The formula is as follows: ， Among them, the target tube is used to describe the spatiotemporal trajectory characteristics of the target in the video sequence. It integrates the position, appearance and behavior information of the target in consecutive frames to form a "tubular" representation throughout the time dimension. The target tube group is a collection of multiple target tubes. Shared frames refer to video frames in which two targets exist at the same time, and interactive frames refer to video frames in which there is interaction between targets. s1 represents spatiotemporal similarity, dist_sim(·) is spatial distance similarity, dir_sim(·) is movement direction similarity, speed_sim(·) is movement speed similarity, α, β, γ are spatiotemporal weight coefficients, α+β+γ=1; If the spatiotemporal similarity s1 in a certain frame exceeds the spatiotemporal similarity threshold T st , then the number of interactive frames C ij Add 1; otherwise, check the behavior label of the target in the frame. If the behavior label belongs to the preset interactive behavior set B p , then the number of interactive frames is C ij +=W S1 ×1 calculation, at this time W S1 ≥0.7, if the behavior label does not belong to B p , then the number of interactive frames is C ij +=W S2 ×1 calculation, at this time W S2 ≤0.3, W S1 and W S2 The behavior labels belong to B P and behavior tags do not belong to B P The weight of semantic information in the situation; When the ratio of the number of interactive frames to the number of shared frames between targets exceeds the preset ratio ρ, it is determined that there is interaction between targets and they are divided into the same target group. The determination conditions are as follows: ， Among them, SFrames(·) represents the target tube O i and O j The number of shared frames.

5. The multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints according to claim 1 is characterized in that: The temporal and spatial characteristics and semantic information of the targets are integrated to determine whether there is interaction between targets, where: The spatiotemporal feature judgment conditions include quantitative calculation based on spatiotemporal similarity and preset threshold judgment, as follows: dist_sim(·) is the spatial distance similarity, which is calculated based on the ratio of the distance between the center points of the two target detection frames to the average height. When the ratio is less than 1.17, dist_sim(·) is normalized according to the rule. The smaller the ratio, the closer the similarity is to 1. dir_sim(·) is the motion direction similarity, which is calculated based on the angle between the moving directions of the two target frames. When the angle is less than 60 degrees, dir_sim(·) is converted into a similarity value through the cosine function. The larger the cosine value of the angle, the higher the similarity. speed_sim(·) is the speed similarity, which is calculated based on the normalized difference between the moving speeds of the two targets. The smaller the speed difference, the closer the similarity is to 1. Semantic information weight W S The arrangement includes: when the target's behavior belongs to the preset interactive behavior set {"wave", "point at the other party", "nod / shake head", "OK gesture", "bow", "salute", "heart"}, the weight of semantic information is increased to more than 70%, and the weight of spatiotemporal constraints is correspondingly reduced to less than 30%; if the target does not perform the preset behavior, the weight of semantic information is reduced to less than 30%, and the weight of spatiotemporal constraints is increased to more than 70%. The specific dynamic adjustment is based on the following formula: ， Among them, W S is the semantic information weight, b i , b j is the target behavior label, B p is the preset interactive behavior set, match_score(·) is the behavior matching score; The semantic information weight W S Used to adjust the spatiotemporal similarity calculation in the target group division: when the target behavior belongs to the preset interactive behavior set B p When the spatiotemporal weight coefficients α, β, and γ are Dynamic adjustment is performed to increase the proportion of semantic constraints in the total similarity calculation. The specific similarity formula is as follows: ， Where s2 represents semantic similarity, the same behavior label is 1, and different is 0, dist_sim(·) is the spatial distance similarity, dir_sim(·) is the movement direction similarity, speed_sim(·) is the movement speed similarity, and α+β+γ=1.

6. The multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints according to claim 1 is characterized in that: The based The collision function combines spatial hash grid acceleration detection with a greedy strategy to assign new time labels to the target pipe group. Specifically, it includes: Sort by the original start time of the target tube group, and assign new time labels to the tube groups in chronological order after sorting, giving priority to preserving the original timing logic; Based on the combination of greedy strategy and spatial hash algorithm, a new time position with minimum space-time collision characteristics is searched, and the space-time collision function is defined: ， Among them, Collision(·) represents the collision value of the target tube group, Overlap(·) represents the spatial overlap of the target tube group in the picture, and the spatial overlap is obtained by calculating the overlap rate of the detection frame. TimeOverlap(·) represents the temporal overlap of the adjusted time interval. and are the i-th and j-th target tube groups respectively, S i and S j is the target tube group space area, T i ′ and T j ′ is the time interval reallocated to the target tube group, which is used to calculate the time overlap of the adjusted time interval.

7. The multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints according to claim 1 is characterized in that: The combined spatial hash grid accelerated detection specifically includes: Divide the video spatiotemporal domain into a regular grid G m×n , for each target tube group g i Create a spatial hash table H i ,The hash table is used to quickly determine the spatial overlap relationship between the target tube groups and detect the spatial collision risk. The logical judgment is as follows: If there is at least one grid cell coordinate (x, y) such that the target tube group g i Corresponding hash table H i Record at (x,y) position, with target tube group g j Corresponding hash table H j There is an intersection of the records at the (x, y) position, that is, H i (x,y)∩H j (x,y)≠0, then determine g i With g j There is a risk of space collision; Among them, there is at least one grid coordinate (x, y), which represents the regular grid G m×n There exists a coordinate point that meets the conditions, H i (x,y) and H j (x,y) represents the target tube group g i and g j The record in the small grid (x,y).

8. The multi-target interactive behavior-preserving surveillance video concentration method based on semantic enhancement and spatiotemporal constraints as claimed in claim 1 is characterized in that: The allocating a new time tag to the target tube group specifically includes: Find the time when the target first appears in each target tube group, use this time as the start time of the corresponding target tube group, and sort them from small to large; Prioritize preserving the spatiotemporal integrity of the target tube group, minimize the timeline span according to the original temporal logic, and define the temporal loss function: ， Where i and j represent the index of the target tube group, Indicates traversing "all two combinations without duplication", t i ′ and t j ' is the adjusted timestamp, I is the indicator function, and the formula is , used to quantify the loss caused by timing reversal.

Images