A real-time video stream analysis method and system

Abstract

This invention discloses a real-time video stream analysis method and system, specifically relating to the field of video surveillance. It addresses the shortcomings of existing technologies that rely on differences in image quality and changes in the number of targets, misjudging irrelevant disturbances as changes in business semantics, leading to deviations in the allocation of analysis resources from the actual importance of events. The method involves periodically extracting frames from the real-time video stream according to a basic analysis frame rate to generate a frame state description sequence. Adjacent frame state descriptions are read sequentially according to the state sequence table, and overlap, displacement, and continuity comparisons are performed to identify state breakpoint intervals. Frames are interpolated within these intervals to generate continuity verification frame groups. A state transition corridor is constructed based on the target occupancy areas of preceding and following frames. The deviation in the transition corridor occupancy and the main contour continuity swing are calculated and input into a continuity discrimination model to generate transition judgment results. Based on the transition judgment results, an event analysis model is selectively invoked to output event judgment results or to block and mark disturbance segments, which are then written into a result sequence table and output.

Description

Technical Field

[0001] This invention relates to the field of video surveillance, and in particular to a real-time video stream analysis method and system for event screening, disturbance filtering and event determination of multiple real-time video streams. Background Technology

[0002] In intelligent analysis scenarios involving parallel access to multiple real-time video streams, systems typically need to continuously perform target detection, event recognition, and alarm determination on video footage from different sources, all within limited decoding, caching, and inference resources. Existing material, "Adaptive Adjustment Method, Apparatus, Computer Equipment, and Medium for Video Analysis," application number 202010667297.2, discloses a scheme for dynamically adjusting the analysis frame rate based on the relationship between analysis capability values ​​and preset thresholds. It further combines the changes in the number of targets of interest, event types, and the amount of differential prediction frame data relative to historical averages to determine the analysis priority and frame rate of the video streams. The aim is to maintain the analysis of as many video streams as possible even when device capabilities are limited. This scheme demonstrates that existing technologies are no longer limited to fixed frame extraction but are beginning to attempt to dynamically control the analysis density based on the degree of image change and target variation.

[0003] However, under the aforementioned technical approach, existing technologies still have a relatively hidden but critical deficiency: they primarily base the adjustment of the analysis frame rate on changes in the number of targets in adjacent analysis frames and changes in the amount of differential prediction frame data relative to the historical average. This easily leads to misinterpreting changes in the encoding layer or surface image as changes in business semantics, causing the allocation of analysis resources to deviate from the actual importance of the event. This problem arises because the increase or decrease in the amount of differential prediction frame data essentially reflects the strength of the difference between the image and the preceding image. This difference may originate from changes in target behavior, or from fluctuations in lighting, shadow swaying, reflection interference, slight lens shake, or changes in compression. Furthermore, an increase or decrease in the number of targets does not necessarily correspond to an increase or decrease in the level of risk. The process manifests as the system tending to increase the analysis density when it detects an increase in image difference and tending to decrease the analysis density when it detects a decrease in difference. However, this adjustment may not follow the actual evolution of events that need to be identified, but rather merely the superficial manifestations of irrelevant disturbances. This can lead to the failure to retain subtle anomalies, slow approach, or weak movements that should be closely monitored, while video fluctuations with no real business significance consume analytical capabilities, ultimately affecting the stability, relevance, and reliability of real-time video stream analysis results.

[0004] To address the aforementioned problems, a technical solution is provided. Summary of the Invention

[0005] To overcome the aforementioned deficiencies in existing technologies, this invention provides a real-time video stream analysis method and system. The method first extracts frames from the real-time video stream at regular intervals according to a basic analysis frame rate, generating a sequence of frame state descriptions. Then, it performs overlap comparison, displacement comparison, and continuity comparison on adjacent frame state descriptions in chronological order to identify state breakpoint intervals. Subsequently, it interpolates frames within the state breakpoint intervals to generate a continuity verification frame group. Based on the target occupancy areas of preceding and following frames, it constructs a state transition corridor, calculates the transition corridor occupancy backfill deviation and the main contour continuity swing, and inputs these into a continuity discrimination model to generate transition judgment results. Finally, based on the transition judgment results, it selectively calls an event analysis model to output event judgment results, or blocks and marks disturbed segments, and writes the results into a result sequence table before outputting them.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A real-time video stream analysis method includes the following steps: S1: The real-time video stream is periodically captured according to the basic analysis frame rate to obtain the basic analysis frame sequence. For each basic analysis frame, target recognition and scene occupancy recognition are performed to generate a frame state description and write it into the state sequence table. S2: Read the state descriptions of adjacent previous frames and subsequent frames in the time order of the state sequence table, and perform overlap comparison, displacement comparison and continuity comparison on the positional relationship between the target's occupied area, the target's centroid and the target's relative to the scene boundary. When the continuity condition is not met, it is truncated as a state breakpoint interval and a state breakpoint record is formed. S3: Interpolate frames within the state breakpoint interval in each state breakpoint record to obtain a set of acceptance verification frames. Generate a state transition corridor based on the target occupancy area. Calculate the backfill deviation of the transition corridor occupancy and the main contour acceptance swing amount and input them into the acceptance discrimination model to generate the transition judgment result. S4: Based on the transition determination result, when there is a continuous evolution process, call the event analysis model to jointly analyze the state breakpoint interval and the state description of the preceding and following frames and output the event determination result. When there is no such process, block and mark it as a disturbance segment and output the disturbance filtering result. Write the result into the result sequence table and output it according to the time position.

[0007] Furthermore, the generation of the basic analysis frame sequence adopts a video decoder that triggers a frame capture mechanism at equal intervals on the time axis to decompress the real-time video stream and extract key image data. The resulting frame images are strictly sorted according to the extraction timestamps, and each frame retains the original timestamp and video source identifier.

[0008] Furthermore, the target recognition adopts the YOLOv5 target detection model, which locates potential target objects through the feature extraction network and generates target contours using the boundary regression branch. At the same time, it calculates the target centroid and the target occupancy region. The scene occupancy recognition adopts the DeepLabv3 semantic segmentation model, which performs pixel-level classification through multi-scale feature fusion and outputs the scene boundary, scene region division results, and the positional relationship of the target relative to the scene boundary.

[0009] Furthermore, the overlap comparison calculates the ratio of the number of intersection pixels to the number of union pixels of the target-occupied area in the state description of the previous frame and the state description of the next frame; the displacement comparison calculates the Euclidean distance between the centroid coordinates of the target in the previous and next frames; and the transition comparison calculates the magnitude of the change in the positional relationship vector of the target relative to the scene boundary in the previous and next frames.

[0010] Furthermore, when the overlapping relationship of the target's occupied area does not meet the continuous extension condition, the displacement relationship of the target's centroid does not meet the continuous migration condition, and the positional relationship of the target relative to the scene boundary does not meet the continuous insertion condition, the original video segment between the two corresponding basic analysis frames is extracted as a state breakpoint interval and associated with the state description of the previous frame and the state description of the next frame to form a state breakpoint record.

[0011] Furthermore, the time-interpolated frame extraction uses a linear time interpolation frame extraction algorithm to extract the intermediate frame sequence for each state breakpoint interval according to a preset interpolation multiple to form a receiving verification frame group. The receiving verification frame group is located between the previous frame state description and the subsequent frame state description and retains the original timestamp and pixel data.

[0012] Furthermore, the state transition corridor is constructed based on the target occupancy regions in the previous frame state description and the subsequent frame state description through mathematical morphological dilation operations combined with connected component analysis. It consists of the target occupancy region in the previous frame, the target occupancy region in the subsequent frame, and the minimum connected region connecting the outer contours of the two.

[0013] Furthermore, for each intermediate video frame in the acceptance verification frame group, the intermediate target occupancy area is extracted, the corridor occupancy ratio is calculated to form a corridor occupancy ratio sequence, and at the same time, a target occupancy progression sequence is constructed and the occupancy deviation value is accumulated to obtain the transition corridor occupancy backfill deviation. For each intermediate video frame, the target orientation bounding rectangle is extracted, the principal axis direction is calculated to generate a pose angle difference sequence, and the principal contour acceptance swing amount is accumulated.

[0014] Furthermore, the event analysis model performs joint analysis on the state breakpoint interval and the state descriptions of the preceding and following frames to output the event determination results. The result sequence table is indexed by time position and outputs a standardized record sequence containing the event determination results and perturbation filtering results.

[0015] A real-time video stream analysis system, comprising: Timed frame capture module: The real-time video stream is captured at timed intervals according to the basic analysis frame rate to obtain the basic analysis frame sequence. For each basic analysis frame, target recognition and scene occupancy recognition are performed to generate a frame state description and write it into the state sequence table. Breakpoint identification module: Reads the state descriptions of adjacent previous frames and subsequent frames in the time order of the state sequence table, performs overlap comparison, displacement comparison and continuity comparison on the positional relationship between the target's occupied area, the target's centroid and the target's position relative to the scene boundary, and extracts the state breakpoint interval when the continuity condition is not met and forms a state breakpoint record. Continuous verification module: Interpolate frames within the state breakpoint interval in each state breakpoint record to obtain a set of acceptance verification frames. Generate a state transition corridor based on the target occupancy area. Calculate the backfill deviation of the transition corridor and the main contour acceptance swing amount, input them into the acceptance discrimination model to generate the transition judgment result. Event Judgment Module: Based on the transition judgment result, when there is a continuous evolution process, the event analysis model is called to jointly analyze the state breakpoint interval and the state description of the preceding and following frames and output the event judgment result. When there is no breakpoint, the event is blocked and marked as a disturbance segment and the disturbance filtering result is output. The result is written into the result sequence table and output according to the time position.

[0016] The technical effects and advantages of the real-time video stream analysis method and system of this invention are as follows: This invention achieves precise quantitative verification of target behavior continuity by introducing frame state description sequences and state transition corridors, effectively avoiding the occupation of analysis resources by irrelevant disturbances such as lighting fluctuations, shadow swings, or camera shake. In resource-constrained scenarios with parallel access to multiple real-time video streams, this method precisely focuses inference capabilities on the evolution range of real events, significantly improving the stability and relevance of target detection and alarm determination. Simultaneously, the selective event analysis model invocation guided by transition determination results reduces the false alarm rate and ensures the system's continuous monitoring of subtle anomalies, slow approach, or weak movements, ultimately improving the credibility of the overall analysis results and real-time response efficiency. Furthermore, the ordered output of the result sequence list provides a standardized interface for the unified management and tracing of subsequent multi-channel video, adapting to flexible deployment needs under different access numbers and computing resource configurations. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a real-time video stream analysis method according to the present invention. Figure 2 Generate a logic diagram for the frame state description of this invention; Figure 3 This is a diagram showing the triple comparison of adjacent frames and the identification of state breakpoints in this invention. Figure 4 This is a diagram showing the interpolated frames and the inherited verification frames within the time interval of the state breakpoint of this invention. Figure 5 This is a diagram illustrating the state transition corridor construction and continuity feature extraction of the present invention. Figure 6 This is a schematic diagram of the structure of a real-time video stream analysis system according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figures 1-5 This invention provides a real-time video stream analysis method, comprising the following steps: S1: The real-time video stream is periodically captured according to the basic analysis frame rate to obtain the basic analysis frame sequence. For each basic analysis frame, target recognition and scene occupancy recognition are performed to generate a frame state description and write it into the state sequence table. S2: Read the state descriptions of adjacent previous frames and subsequent frames in the time order of the state sequence table, and perform overlap comparison, displacement comparison and continuity comparison on the positional relationship between the target's occupied area, the target's centroid and the target's relative to the scene boundary. When the continuity condition is not met, it is truncated as a state breakpoint interval and a state breakpoint record is formed. S3: Interpolate frames within the state breakpoint interval in each state breakpoint record to obtain a set of acceptance verification frames. Generate a state transition corridor based on the target occupancy area. Calculate the backfill deviation of the transition corridor occupancy and the main contour acceptance swing amount and input them into the acceptance discrimination model to generate the transition judgment result. S4: Based on the transition determination result, when there is a continuous evolution process, call the event analysis model to jointly analyze the state breakpoint interval and the state description of the preceding and following frames and output the event determination result. When there is no such process, block and mark it as a disturbance segment and output the disturbance filtering result. Write the result into the result sequence table and output it according to the time position.

[0020] The processing scheme of this invention includes the following steps: First, the real-time video stream is periodically frame-by-frame according to the basic analysis frame rate to obtain a basic analysis frame sequence; target recognition and scene occupancy recognition are performed on each basic analysis frame, outputting the target contour, target centroid, target occupancy area, scene boundary, scene area division result, and the positional relationship of the target relative to the scene boundary, thereby generating a frame state description and writing it into the state sequence table in chronological order. Second, the state descriptions of adjacent previous and subsequent frames are read in chronological order according to the state sequence table, and overlap comparison is performed on the target occupancy area, displacement comparison is performed on the target centroid, and continuity comparison is performed on the positional relationship of the target relative to the scene boundary; when the three continuous conditions are not met simultaneously, the corresponding original video segment is truncated into a state breakpoint interval and a state breakpoint record is formed. Third, frames are interpolated in time for the state breakpoint interval in each state breakpoint record to obtain a continuity verification frame group; a state transition corridor is generated based on the target occupancy area of ​​the preceding and following frames, and the occupancy backfill deviation and main contour continuity swing amount of the transition corridor are calculated respectively, and then the two are input into the continuity discrimination model to generate a transition judgment result. Finally, based on the transition determination results, the event determination results are output for intervals with continuous evolution processes in the event analysis model; intervals without continuous evolution processes are blocked from entering the event analysis model and marked as disturbance segments, and the corresponding results are written into the result sequence table and output.

[0021] In intelligent analysis scenarios with parallel access to multiple real-time video streams, the raw video data is massive and contains numerous irrelevant disturbances such as lighting fluctuations and camera shake. Directly performing event analysis on all frames would severely waste limited decoding and inference resources. Real-time video stream analysis methods must first perform stable timed frame-taking processing on the input video stream and extract target geometric features and scene semantic information. This transforms continuous video into a structured sequence of frame state descriptions, providing a reliable data benchmark for accurately identifying the evolution intervals of real events. This allows the analysis focus to be on changes in business semantics rather than surface-level image fluctuations, effectively ensuring the stability and relevance of subsequent judgments in resource-constrained environments.

[0022] The specific processing logic for step S1 is as follows: S101: Timed frame capture to obtain the basic analysis frame sequence.

[0023] In intelligent analysis scenarios involving parallel access to multiple real-time video streams, due to limited decoding buffers and inference resources, continuous target detection and event determination are required for video images from different sources. To avoid deviations in state description caused by irregular sampling, which could affect the accuracy of subsequent breakpoint interval identification, it is essential to first implement stable timed frame-taking processing on the real-time video streams to establish a time-ordered basic analysis frame sequence. This process ensures that all video streams are captured at the same temporal granularity through fixed-frequency sampling, providing a unified benchmark for subsequent continuity verification based on semantic features. This allows the analysis focus to be on the evolution of real events rather than surface fluctuations caused by illumination or compression, thereby significantly improving the stability and relevance of real-time video stream analysis methods under resource-constrained conditions.

[0024] The real-time video stream is timed to capture frames according to the basic analysis frame rate. The basic analysis frame rate refers to the number of complete image frames extracted from the video stream per unit time. This value is predetermined based on the number of access video channels and available computing resources. Specifically, the video decoder triggers the frame capture mechanism at equal intervals on the time axis to decompress the input stream and extract key image data. The resulting frame images are strictly sorted according to the extraction timestamp to form a basic analysis frame sequence. Each frame retains the original timestamp and video source identifier to support multi-channel parallel processing.

[0025] In one embodiment, for four high-definition video streams deployed in a large parking lot monitoring system, when the basic analysis frame rate is set to 1 frame per second, the real-time video stream analysis method synchronously extracts one frame from each stream every second, accumulating to form a basic analysis frame sequence containing several hours of continuous footage for subsequent identification.

[0026] The basic analysis frame sequence obtained through the above-mentioned timed frame acquisition process provides standardized input for the next step of target recognition and scene occupancy recognition, thereby ensuring the temporal consistency of the entire frame state description generation link and preparing the necessary time reference for accurate positioning of state breakpoint intervals.

[0027] S102: Perform target recognition for each basic analysis frame, outputting the target contour, target centroid, and target occupancy region.

[0028] Following the establishment of the basic analysis frame sequence, in a resource-constrained real-time video analysis environment, accurately extracting the geometric features of targets in video images is crucial for distinguishing between business semantic changes and irrelevant disturbances. Therefore, real-time video stream analysis methods need to apply target recognition processing to each basic analysis frame to output the target contour, target centroid, and target occupied area. These features together constitute the core semantic basis for subsequent state comparison. By using detailed contour description, misjudgments caused by simply relying on pixel differences are avoided, allowing analysis resources to be prioritized for the target evolution process with continuous behavioral significance, thereby improving the credibility of event judgment.

[0029] Each basic analysis frame is input to the YOLOv5 object detection model for object recognition processing. The YOLOv5 object detection model first locates potential target objects in the image through the feature extraction network, and then uses the boundary regression branch to generate the target contour. The target contour refers to the closed curve formed by the ordered set of pixels that constitute the edge of the target object. At the same time, the target centroid and the target occupied area are calculated. The target occupied area refers to the set of all pixels enclosed by the target contour. The target centroid is calculated using the contour integral method to ensure geometric accuracy. The area of ​​the target occupied area is obtained by performing a closed integral on the closed curve of the target contour. That is, the area is equal to half of the product of the differential of the x-coordinate and y-coordinate of each point on the contour minus the sum of the differential products of the y-coordinate and x-coordinate. The x-coordinate of the target centroid is obtained by integrating the product of the square of the x-coordinate and the differential of the y-coordinate on the closed curve of the contour and dividing by six times the area. The y-coordinate is obtained by the corresponding calculation method. The target contour, target centroid and target occupied area are output together.

[0030] In one embodiment, for pedestrian targets in road surveillance video, the YOLOv5 target detection model identifies the human silhouette and calculates the centroid coordinates and the area of ​​the occupied pixel region to record slow movement rather than changes in light and shadow. The target silhouette, centroid, and occupied region output by target recognition provide accurate geometric input data for subsequent scene occupancy recognition and the comprehensive generation of frame state descriptions, thus directly supporting the overlap comparison and displacement comparison operations of adjacent frame state descriptions in subsequent steps.

[0031] S103: Perform scene occupancy recognition on the same basic analysis frame, outputting the scene boundary scene region division results and the positional relationship of the target relative to the scene boundary.

[0032] After the target geometric features are extracted in the basic analysis frame, considering that the information of a single target is insufficient to determine the importance of an event, the real-time video stream analysis method needs to further perform scene occupancy recognition to integrate the environmental semantic context. This is to ensure that the state description fully reflects the interaction between the target and the scene, avoid mistaking disturbances such as camera shake for key events, and thus prioritize the processing of video segments that are truly risk-related with limited resources.

[0033] The same basic analysis frame is input into the DeepLabv3 semantic segmentation model in parallel to perform scene occupancy recognition. The DeepLabv3 semantic segmentation model performs pixel-level classification of the image through multi-scale feature fusion. The output scene boundary refers to the set of dividing lines formed by key structures in the scene, such as the ground, walls, and passage edges. The scene region division result refers to the label map that divides the image into different functional sub-regions, such as warning zone, passage zone, and stopping zone. At the same time, the positional relationship of the target relative to the scene boundary is calculated. The positional relationship of the target relative to the scene boundary is represented by the distance vector from the target centroid to the nearest scene boundary and the azimuth angle. Specifically, the vector projection method is used to determine the relative azimuth, that is, to calculate the projection length of the line vector connecting the target centroid and the boundary point on the boundary normal vector to determine the relative azimuth. The obtained scene boundary scene region division result and the positional relationship of the target relative to the scene boundary are integrated with the target information.

[0034] In one embodiment, in indoor security video, the DeepLabv3 semantic segmentation model divides the door and window boundaries and the ground area, and calculates the distance and direction of a pedestrian target relative to the door boundary to determine whether the pedestrian has entered a sensitive area.

[0035] The scene occupancy recognition results, together with the target recognition output, constitute the complete input set for generating the frame state description, ensuring that the state description has scene perception capabilities and providing a positional relationship basis for subsequent steps of comparison.

[0036] S104: Using the target contour, target centroid, target occupancy area, scene boundary, scene region division results, and the positional relationship of the target relative to the scene boundary in the same basic analysis frame as input, generate a frame state description and write it into the state sequence table.

[0037] After integrating the aforementioned identification results, in order to achieve structured recording of video stream states, real-time video stream analysis methods need to encapsulate multi-dimensional features into frame state descriptions and store them sequentially. This encapsulation process establishes a traceable time series, laying the data foundation for subsequent reading of adjacent frame state descriptions for breakpoint detection. This enables real-time video stream analysis methods to efficiently filter disturbed segments, focus computational power on continuously evolving event intervals, and improve the overall efficiency and reliability of real-time analysis results.

[0038] Taking the target contour, target centroid, target occupancy area, scene boundary, scene region division results, and the target's positional relationship relative to the scene boundary in the same basic analysis frame as input, a frame state description uniquely corresponding to the basic analysis frame is generated through feature encapsulation processing. Feature encapsulation processing refers to the processing unit that encapsulates the multi-dimensional recognition results into a structured record. The frame state description refers to a structured data record containing all the above-mentioned target geometric parameters, scene semantic parameters, and relative position parameters. Then, the frame state descriptions corresponding to each basic analysis frame are written into a state sequence table in chronological order. The state sequence table refers to a database structure indexed by frame timestamps, which supports fast sequential reading and related queries.

[0039] In one embodiment, for a continuously monitored factory workshop video stream, the frame state description encapsulates the centroid of the worker's target profile and its position data relative to the safety line, and stores them sequentially in a list for easy comparison of abnormal movements later.

[0040] The state sequence table generated and stored through the above processing completes the core output of this step. It provides a directly callable data foundation for subsequent steps to read the state descriptions of adjacent frames in chronological order to perform overlap comparison, displacement comparison and continuation comparison, thus seamlessly connecting the subsequent state breakpoint interval interception and verification process.

[0041] The real-time video stream analysis method extracts frames from the real-time video stream at regular intervals according to the basic analysis frame rate, resulting in a sequence of basic analysis frames arranged in chronological order. For each basic analysis frame, target recognition is performed, outputting the target contour, target centroid, and target occupied region. For the same basic analysis frame, scene occupancy recognition is performed, outputting the scene boundary, scene region division results, and the positional relationship of the target relative to the scene boundary. Using all the above features as input, a frame state description uniquely corresponding to each basic analysis frame is generated, and these frame state descriptions are completely written into the state sequence table in chronological order, realizing the complete transformation of the video stream from raw pixel data to structured semantic state records.

[0042] After the frame state descriptions of each basic analysis frame have been completely recorded in the state sequence table in chronological order, in order to avoid misjudging differences in the encoding layer or surface layer as changes in business semantics and thus wasting analysis resources, the real-time video stream analysis method needs to perform target occupancy region overlap comparison, target centroid displacement comparison, and target positional relationship comparison relative to scene boundary comparison on the state descriptions of adjacent frames. This allows for the accurate location of the intervals where the target behavior breaks at the semantic level, concentrating computing power on the truly important continuously evolving video segments. In multi-channel real-time video stream parallel scenarios, this significantly improves the accuracy of event determination and resource utilization efficiency.

[0043] The specific processing logic for step S2 is as follows: S201: Read the state descriptions of two adjacent frames in time sequentially according to the time order in the state sequence table as the previous frame state description and the next frame state description.

[0044] In intelligent analysis scenarios with parallel access to multiple real-time video streams, the state sequence table generated by the aforementioned steps has completely recorded the target outline, target centroid, target occupancy area, scene boundary, scene region division results, and the positional relationship of the target relative to the scene boundary for each basic analysis frame. To avoid excessive consumption of limited inference resources by performing indiscriminate event analysis on all video segments, the real-time video stream analysis method must read the state descriptions of adjacent frames sequentially according to the time order of the state sequence table to identify potential state change points. This process ensures the targeting of subsequent breakpoint detection through precise temporal traversal, thereby enabling analysis resources to focus on the evolution of real events rather than irrelevant screen fluctuations, significantly improving the stability and reliability of real-time video stream analysis results in resource-constrained environments.

[0045] The temporal order in the state sequence table is strictly followed. Two adjacent frames in time are read sequentially and labeled as the previous frame state description and the next frame state description, respectively. The previous frame state description refers to the state description of the earlier frame, and the next frame state description refers to the state description of the frame that follows immediately after it. The reading operation adopts a database sequential query mechanism and extracts only adjacent pairs to minimize computational overhead.

[0046] In one embodiment, for a real-time video stream of factory workshop monitoring, the state descriptions of the 5th and 6th frames recorded in the state sequence table are extracted sequentially. The state description of the previous frame contains the worker target's occupied area and its relative position to the safety line, while the state description of the subsequent frame updates the geometric parameters of the same target for subsequent determination of whether there is a state break.

[0047] After the above reading and processing, the previous frame state description and the next frame state description are obtained for comparison, providing directly callable input data for the next step of overlap comparison, displacement comparison and connection comparison, thus seamlessly connecting the identification process of state breakpoint intervals and preparing the necessary prerequisite records for subsequent connection verification.

[0048] S202: Perform overlap comparison on the target occupancy area in the previous frame state description and the next frame state description, perform displacement comparison on the target centroid, and perform continuity comparison on the positional relationship of the target relative to the scene boundary.

[0049] Based on the reading results of the state descriptions of adjacent frames, relying solely on pixel differences in a limited decoding buffer environment can easily lead to misjudgment of disturbances. Therefore, real-time video stream analysis methods need to perform triple semantic comparison of the state descriptions of the previous frame and the state descriptions of the next frame to distinguish between the continuous evolution of the target and irrelevant fluctuations. This process avoids mistaking illumination or jitter as event changes through precise matching of geometric and positional features, thereby ensuring that the analysis density only follows the evolution of real risks and improving the pertinence and overall credibility of event judgment.

[0050] The target occupancy regions in the previous frame state description and the next frame state description are extracted for overlap comparison. Overlap comparison refers to the overlap relationship value obtained by dividing the number of intersection pixels of the two target occupancy regions by the number of union pixels. The target centroid is extracted for displacement comparison. Displacement comparison refers to the position migration relationship value obtained by calculating the Euclidean distance between the target centroid coordinates of the next frame and the target centroid coordinates of the previous frame. The position relationship of the target relative to the scene boundary is extracted for continuity comparison. Continuity comparison refers to the position continuity relationship value obtained by calculating the distance vector and azimuth angle change of the target centroid from the next frame to the nearest scene boundary. The continuous extension condition is defined as the overlap relationship value being greater than the continuous extension ratio predetermined based on the continuous behavior characteristics of the target. The continuous migration condition is defined as the displacement relationship value being less than the continuous migration distance value predetermined based on the basic analysis frame rate and the maximum movement speed of the scene. The continuous in-position condition is defined as the position continuity relationship value being less than the continuous in-position change value predetermined based on the continuity of the scene boundary. The overlap relationship is preferably represented by the Intersection over Union (IoU) of the target's occupied areas in consecutive frames; when the IoU is greater than or equal to a preset continuous extension threshold T1, the continuous extension condition is satisfied. The displacement relationship is preferably represented by the Euclidean distance D between the centroids of the targets in consecutive frames; when D is less than or equal to a preset continuous migration threshold T2, the continuous migration condition is satisfied, where T2 can be calculated based on the basic analysis frame rate and the maximum reasonable movement speed of the target within the scene. The positional continuity relationship is preferably represented by the weighted sum B of the change in distance and azimuth angle from the target to the nearest scene boundary; when B is less than or equal to a preset continuous insertion threshold T3, the continuous insertion condition is satisfied. T1, T2, and T3 can be obtained offline based on the target category, deployment scene, and historical labeled samples.

[0051] In one embodiment, for a pedestrian target in a road surveillance video, the overlap value of the target's occupied area between the previous frame and the next frame reaches more than 80%, the centroid displacement is less than the maximum stride distance in each frame, and the vector change of the position relative to the gate boundary remains stable, thereby satisfying three conditions to indicate that the target behavior is continuous.

[0052] After the above three-fold comparison process, it was determined whether the positional relationship between the target's occupied area, the target's centroid, and the target's position relative to the scene boundary remained continuous. This provided a basis for judgment in the next step of state breakpoint interval interception, thus directly supporting the filtering mechanism of disturbance segments and preparing associated data for subsequent transition judgment.

[0053] S203: When the overlap relationship of the target's occupied area does not meet the continuous extension condition, the displacement relationship of the target's centroid does not meet the continuous migration condition, and the positional relationship of the target relative to the scene boundary does not meet the continuous insertion condition, the original video segment between the two corresponding basic analysis frames is extracted as the state breakpoint interval.

[0054] After the triple comparison of adjacent frame state descriptions is completed, considering that irrelevant disturbances will occupy valuable event analysis resources, the real-time video stream analysis method needs to accurately extract potentially fragmented video segments based on the comparison results. This process ensures that only semantically discontinuous intervals are further verified through strict logic and condition judgments, thereby concentrating computing power on segments that truly require event recognition. This effectively improves the stability and relevance of the analysis in parallel scenarios of multiple real-time video streams.

[0055] When the overlapping relationship of the target's occupied area does not meet the continuous extension condition, the displacement relationship of the target's centroid does not meet the continuous migration condition, and the positional relationship of the target relative to the scene boundary does not meet the continuous insertion condition, the original video segment between the two corresponding basic analysis frames is truncated as a state breakpoint interval. The state breakpoint interval refers to the continuous original video segment between the end time of the basic analysis frame corresponding to the state description of the previous frame and the start time of the basic analysis frame corresponding to the state description of the next frame. The truncating operation is implemented through a video cutting mechanism aligned with timestamps, and only the complete frame data within this interval is retained for subsequent frame interpolation verification.

[0056] In one embodiment, for a segment in an indoor security video where the camera shakes slightly, if none of the three conditions are met, the original video segment between frames 10 and 12 is extracted as a state breakpoint interval and marked as a potential disturbance rather than an event evolution.

[0057] After the above condition judgment and interception processing, the state breakpoint interval to be verified is formed, which provides a precise video range for the next step of associated storage and the generation of verification frame groups, thereby ensuring the accuracy of perturbation filtering and laying the selective input foundation for the event analysis model call in step S4.

[0058] S204: Associate the state breakpoint interval with the corresponding previous frame state description and subsequent frame state description to form a state breakpoint record.

[0059] After integrating the aforementioned interception results, in order to achieve traceable management of breakpoint intervals and subsequent verification calls, the real-time video stream analysis method needs to associate and store the state breakpoint intervals with the relevant frame state descriptions. This process establishes a complete breakpoint link through structured records, enabling the real-time video stream analysis method to efficiently block irrelevant segments from entering the event analysis model, reserving limited resources for the continuous evolution process. This significantly improves the credibility of the analysis results and the overall processing efficiency in resource-constrained multi-channel video scenarios.

[0060] The state breakpoint interval is associated with the corresponding previous frame state description and the next frame state description and stored to form a state breakpoint record. The state breakpoint record refers to a triple structured entry that contains the video data of the state breakpoint interval, all features of the previous frame state description, and all features of the next frame state description. The storage operation is implemented using an index database mechanism, which supports fast retrieval and associated queries by time position.

[0061] In one embodiment, for the breakpoint interval caused by light and shadow fluctuations in the parking lot monitoring video, the state breakpoint record will completely bind the captured video segment with the target outline, centroid, occupied area and relative boundary position of the preceding and following frames, so as to facilitate subsequent frame interpolation verification of whether it is a disturbance.

[0062] After the above-mentioned associated storage processing, the generation of state breakpoint records is completed. This provides a complete dataset that can be directly called upon for subsequent steps to perform time-interpolation frames and construct transition corridors for each state breakpoint record, thereby seamlessly connecting the calculation process of the confidence coefficient and ultimately supporting the ordered output of the result sequence table.

[0063] The real-time video stream analysis method reads the state descriptions of two adjacent frames in time sequence according to the time order in the state sequence table as the previous frame state description and the next frame state description. It completes the comparison of target occupancy area overlap, target centroid displacement, and target positional relationship with the scene boundary. When the three consecutive conditions are not met at the same time, the original video segment between the two corresponding basic analysis frames is truncated as a state breakpoint interval. This state breakpoint interval is associated with the corresponding previous frame state description and the next frame state description to form a state breakpoint record, realizing the accurate identification and structured storage of potentially semantically broken video segments.

[0064] After the state breakpoint records have been formed, due to the limitations of the basic analysis frame rate, subtle and continuous changes in the target's behavior may be missed. Therefore, the real-time video stream analysis method must perform time-interpolated frame processing for each state breakpoint interval and construct a state transition corridor. By calculating the deviation between the corridor occupancy ratio sequence of the intermediate target occupancy area and the target occupancy progression sequence, as well as the swing amount of the main contour, the semantic continuity between the state descriptions of the preceding and following frames can be accurately quantified. This distinguishes between the evolution of real events and irrelevant disturbance segments, providing a reliable basis for subsequent selective event analysis under the condition of limited inference resources.

[0065] The specific processing logic for step S3 is as follows: S301: Interpolate frames within the time interval of each state breakpoint record to obtain the receiving verification frame group.

[0066] After the state breakpoint records have been associated and stored through the aforementioned steps, in the intelligent analysis scenario where multiple real-time video streams are accessed in parallel, in order to accurately verify whether there is a semantically continuous evolution process between the state descriptions of two adjacent frames and to avoid misjudging continuous events as disturbances due to the sampling blind spot caused by the low frame rate of the basic analysis, the real-time video stream analysis method needs to perform higher-density time-interpolation frame processing on the state breakpoint interval. This process improves the verification accuracy by uniformly inserting intermediate frames in the breakpoint interval, so that the subsequent transition corridor analysis and swing amount calculation can capture subtle changes in target behavior, and significantly improve the pertinence of event judgment and the reliability of results under the condition of limited inference resources.

[0067] The state breakpoint interval in each state breakpoint record is input into the linear time interpolation frame extraction algorithm for processing. The linear time interpolation frame extraction algorithm is a frame generation method that inserts new video frames into the state breakpoint interval at uniform time intervals. The original video segment located between the previous frame state description and the next frame state description is subjected to frame extraction operation according to a preset interpolation multiple. The resulting intermediate frame sequence is marked as the successor verification frame group. The successor verification frame group refers to the ordered set of all interpolated video frames from the time corresponding to the previous frame state description to the time corresponding to the next frame state description, where each frame retains the original timestamp and pixel data.

[0068] In one embodiment, for the state breakpoint interval of suspected abnormal worker movement in the factory workshop monitoring video, the linear time interpolation frame extraction algorithm inserts intermediate frames in the interval at a density of three times the basic analysis frame rate, resulting in a continuation verification frame group containing twelve verification frames for subsequent fine comparison.

[0069] After the above-mentioned time-interpolation frame processing, a set of receiving verification frames is formed, which provides input data with higher temporal resolution for the next step of generating the state transition corridor. This directly supports the accurate calculation of the corridor occupancy ratio sequence and prepares the necessary basic frame sequence for the determination of the continuous evolution process.

[0070] S302: Generate a state transition corridor based on the target occupancy area in the previous frame state description and the target occupancy area in the subsequent frame state description.

[0071] After the generation of the verification frame group is completed, in order to establish a reasonable motion path reference for the target from the previous frame to the next frame in real-time video stream analysis and avoid false continuity judgments caused by lighting fluctuations or lens shake, the real-time video stream analysis method needs to construct a state transition corridor based on the target occupancy area of ​​the previous and next frames. This geometric construction process provides a spatial constraint benchmark for subsequent occupancy ratio calculation, ensuring that only the target behavior that actually moves along a reasonable path will be judged as continuous evolution, thereby optimizing the allocation of analysis focus in a resource-constrained environment.

[0072] The target occupancy regions in the previous frame state description and the target occupancy regions in the subsequent frame state description are extracted. A state transition corridor is constructed by combining mathematical morphological dilation operation with connected component analysis. Mathematical morphological dilation operation refers to the operation of expanding the boundary pixels of the region outward. Connected component analysis refers to the processing method of identifying and merging adjacent pixel sets. The state transition corridor refers to the spatial channel region composed of the target occupancy region in the previous frame, the target occupancy region in the subsequent frame, and the minimum connected region connecting the outer contours of the two. Specifically, after performing morphological dilation processing on the target occupancy regions in the previous and subsequent frames respectively, the union is taken, and then the internal holes are filled to form a complete corridor region.

[0073] In one embodiment, for the state breakpoint interval of a pedestrian moving from the warning zone to the passage in a road surveillance video, the state transition corridor is presented as an elliptical passage area that smoothly extends from the human body occupancy area in the previous frame to the occupancy area in the next frame, which is used to constrain the rationality of the target's position in subsequent intermediate frames.

[0074] After the above state transition corridor generation process, a spatial reference for verifying the target motion path is obtained, providing a fixed reference area for the subsequent frame-by-frame calculation of the corridor occupancy ratio sequence, thus seamlessly connecting the joint analysis process of deviation and sway.

[0075] S303: Extract the intermediate target occupancy area from each intermediate video frame in the receiving verification frame group, calculate the corridor occupancy ratio, form a corridor occupancy ratio sequence, and construct a target occupancy progressive sequence to obtain the transition corridor occupancy backfill deviation.

[0076] After the state transition corridor is constructed, in order to quantify whether the intermediate frame target transitions smoothly along the expected path, the real-time video stream analysis method needs to calculate the corridor occupancy ratio and accumulate the deviation with the ideal progressive sequence. This process avoids the limitations of simple shape comparison by measuring the area matching degree, so that subtle abnormal behaviors can be accurately captured, thereby guiding the analysis resources more effectively to the event intervals that really need attention and improving the stability of real-time video stream analysis.

[0077] In each intermediate video frame of the receiving verification frame group, an intermediate target occupancy region is extracted. The intermediate target occupancy region refers to the set of pixels surrounded by the target outline obtained by target recognition in the intermediate frame. The corridor occupancy ratio of the intermediate target occupancy region falling into the state transition corridor is calculated. The corridor occupancy ratio is obtained by dividing the intersection area of ​​the intermediate target occupancy region and the state transition corridor by the total area of ​​the state transition corridor. Then, a corridor occupancy ratio sequence is formed according to the time order of the receiving verification frame group. At the same time, a target occupancy progressive sequence is constructed based on the target occupancy regions in the state description of the previous frame and the target occupancy regions in the state description of the subsequent frame. The target occupancy progressive sequence refers to the reference sequence in which the target occupancy ratio increases linearly from zero to one under ideal conditions. The corridor occupancy ratio sequence and the target occupancy progressive sequence are aligned frame by frame. The occupancy deviation value of each aligned position is accumulated to obtain the transition corridor occupancy backfill deviation. The transition corridor occupancy backfill deviation is obtained by multiplying the absolute difference between the actual occupancy ratio and the ideal occupancy ratio of each aligned position by the corresponding time interval and then summing them. For example, suppose a total of m intermediate video frames are obtained within the state breakpoint interval. Let Mk be the target occupancy area of ​​the k-th intermediate video frame and C be the state transition corridor. Then, the corridor occupancy ratio Rk of the k-th intermediate video frame can be expressed as Rk = intersection area (Mk, C) / area (C). The ideal target occupancy progression value Pk is preferably Pk = k / (m+1). Furthermore, the projection of the target centroid of the k-th intermediate video frame onto the line connecting the centroids of the preceding and following frames can be normalized and used as the actual progression value Ak. Accumulating |Ak-Pk| of each intermediate frame in chronological order yields the transition corridor occupancy backfill deviation E.

[0078] In one embodiment, for the breakpoint interval where a vehicle slowly enters a sensitive area in a parking lot monitoring video, the corridor occupancy ratio sequence shows a steady upward trend with a small deviation from the ideal progressive sequence, thus obtaining a low amount of backfilling deviation in the transition corridor occupancy.

[0079] After the above calculations, the backfill deviation of the transition corridor was obtained, which provided the first quantitative feature for the subsequent calculation of the main outline bearing the swing and the joint input model, thus establishing the basis for the determination of the area continuity of the continuous evolution process.

[0080] S304: Extract the target orientation bounding rectangle for each intermediate video frame in the acceptance verification frame group, calculate the principal axis direction, and generate a sequence of pose angle differences to obtain the main contour acceptance swing amount.

[0081] After the calculation of the deviation of the backfilling of the transition corridor is completed, in order to further examine the consistency of the target's posture and direction of motion and avoid the misjudgment of abrupt posture changes or reverse swings as continuous behavior, the real-time video stream analysis method needs to perform directional geometric analysis on the intermediate frames and accumulate the swing index. This process enhances the sensitivity to weak motion behavior through angle deviation measurement, ensuring that slow approach or subtle anomalies can be correctly preserved.

[0082] In each intermediate video frame of the verification frame group, a target-oriented bounding rectangle is extracted. This bounding rectangle refers to the minimum area rotation rectangle enclosing the target contour. The angle between the principal axis of the bounding rectangle (i.e., the angle between its long side and the horizontal coordinate axis) and the target centroid displacement direction is calculated. A pose angle difference sequence is generated by combining this with the target centroid displacement direction between adjacent intermediate video frames. This pose angle difference refers to the angular difference between the principal axis and the centroid displacement direction. The directional flip and jump values ​​in the pose angle difference sequence are accumulated to obtain the main contour bearing sway amount. This main contour bearing sway amount is obtained by summing the absolute values ​​of the pose angle differences between adjacent frames and then adding the complete rotation angle reference after the number of directional flips. For example, for each intermediate video frame, a target-oriented bounding rectangle is extracted, and the angle theta_k of its long side relative to the horizontal coordinate axis is calculated. Simultaneously, the motion direction angle phi_k is calculated based on the displacement vector of the target centroid in adjacent intermediate video frames. The pose angle difference is preferably expressed as delta_k = min(|theta_k - phi_k|, 2pi - |theta_k - phi_k|). If the motion direction reverses between two adjacent frames, a direction flip is recorded. The delta_k values ​​are accumulated and the direction flip penalty term lambda×N is added to obtain the swing amount W of the main contour, where N is the number of direction flips and lambda is the preset penalty coefficient.

[0083] In one embodiment, for the breakpoint interval of a person slowly approaching the access control in an indoor security video, the angle difference between the main axis direction and the movement direction remains stable, and the number of flips is zero. Therefore, the main contour receives a low amount of swing.

[0084] After the above extraction and accumulation processing, the swing amount of the main contour was obtained, which provides posture continuity features for the next step of dimensional normalization and model input, thus forming a complete evidence chain for determining continuous evolution together with the area deviation.

[0085] S305: After normalizing the dimensions of the backfill deviation of the transition corridor and the sway of the main contour, input them into the acceptance discrimination model to generate the transition judgment result.

[0086] After integrating the two aforementioned deviations, in resource-constrained real-time video stream analysis scenarios, to ultimately determine whether the state breakpoint interval belongs to a continuous evolution process, the real-time video stream analysis method needs to jointly input the area deviation and attitude sway characteristics into the model for comprehensive decision-making. This process overcomes the limitations of a single indicator through model fusion, thereby accurately blocking perturbation segments from entering the event analysis model and reserving computational power for the evolution of real events. The deviation of the transition corridor occupancy backfill and the sway of the main contour are respectively processed by dimension normalization. Dimension normalization refers to scaling using the maximum deviation value of the scene reference to unify the magnitude. Then, the dimension-normalized deviation of the transition corridor occupancy backfill and the dimension-normalized sway of the main contour are input into the acceptance discrimination model.

[0087] The acceptance discrimination model is constructed using a support vector machine classification model based on radial basis function kernels. The construction process first collects a large number of historical video clips as training samples. Among them, the continuously evolving clips and the perturbation clips are manually labeled to form a labeled dataset. Then, the deviation of the transition corridor occupancy and the swing of the main contour are extracted from each sample to form a two-dimensional feature vector as input. During the model training stage, the five-fold cross-validation method combined with grid search is used to optimize the penalty coefficient C and the kernel parameter gamma to obtain the optimal classification boundary.

[0088] In one embodiment, the acceptance discrimination model is finally trained and optimized using parameter settings of C = 10.0 and gamma = 0.1. After training, the model maps the decision function to an acceptance confidence coefficient through Platt Scaling. The acceptance confidence coefficient refers to the confidence probability that the model determines that there is a continuous evolution process between the state descriptions of the previous and subsequent frames, and its value ranges from 0 to 1. After the acceptance discrimination model outputs the acceptance confidence coefficient, it is compared with a preset acceptance threshold. The preset acceptance threshold is defined as 0.75, which is predetermined based on the continuity statistics of historical events. When the acceptance confidence coefficient is higher than the preset acceptance threshold, it is determined that there is a continuous evolution process between the state descriptions of the previous and subsequent frames. When the acceptance confidence coefficient is not higher than the preset acceptance threshold, it is determined that there is no continuous evolution process between the state descriptions of the previous and subsequent frames, and a transitional judgment result corresponding to the state breakpoint interval is generated. In one embodiment, for the state breakpoint interval caused by a sudden change in illumination in the surveillance video, the acceptance confidence coefficient is only 0.42, which is lower than the preset acceptance threshold, and a transitional judgment result indicating that there is no continuous evolution process is ultimately generated.

[0089] After the above joint judgment process, the transition judgment result corresponding to each state breakpoint record is generated, which provides a direct decision basis for subsequent steps to read the transition judgment result one by one and selectively call the event analysis model or mark the disturbance fragment accordingly, thereby realizing the precise allocation of analysis resources.

[0090] The real-time video stream analysis method interpolates frames within the state breakpoint interval of each state breakpoint record to obtain a transition verification frame group located between the state descriptions of the previous and subsequent frames. Based on the target occupancy regions in the state descriptions of the previous and subsequent frames, a state transition corridor is generated. For each intermediate video frame in the transition verification frame group, the intermediate target occupancy region is extracted, and the corridor occupancy ratio is calculated to form a corridor occupancy ratio sequence. At the same time, a target occupancy progression sequence is constructed to obtain the transition corridor occupancy backfill deviation. For each intermediate video frame, the target orientation bounding rectangle is extracted, the principal axis direction is calculated, and a pose angle difference sequence is generated to obtain the main contour acceptance swing amount. After normalizing the dimensions of both, they are input into the acceptance discrimination model to output the acceptance confidence coefficient, which is compared with a preset acceptance threshold to generate the transition judgment result corresponding to the state breakpoint interval, thus completing the accurate verification of the semantic continuity of the state breakpoint interval.

[0091] Once the transition judgment results recorded at each state breakpoint clearly characterize whether the interval is a continuous evolution process, the real-time video stream analysis method needs to perform differentiated processing on the video segments accordingly. This allows for the precise application of deep event analysis capabilities to continuous intervals with business significance, while efficiently filtering out irrelevant disturbances. In scenarios where multiple real-time video streams are accessed in parallel and resources are limited, the effectiveness of the analysis is maximized, computational power is avoided from being consumed by non-event segments, and the timeliness and reliability of alarm judgment are improved.

[0092] S401: Read the transition judgment results corresponding to each state breakpoint record one by one.

[0093] After the transition judgment results corresponding to each state breakpoint record have been generated through the aforementioned steps, in the intelligent analysis scenario where multiple real-time video streams are accessed in parallel, since limited inference resources cannot support complete event analysis of all video segments, the real-time video stream analysis method must read the transition judgment results corresponding to each state breakpoint record to achieve selective processing. This process ensures the immediacy of subsequent decisions through efficient index traversal, so that the event analysis model only operates on the real continuous evolution interval, avoiding irrelevant disturbances from occupying computing power, and significantly improving the relevance and overall stability of real-time video stream analysis results in a resource-constrained environment.

[0094] Each state breakpoint record stored in the state sequence table is read one by one in chronological order, and the corresponding transition judgment result is extracted. The transition judgment result refers to the binary identifier record that represents whether there is a continuous evolution process output by the previous steps, as well as the associated succession confidence coefficient value. The reading operation adopts a database sequential query mechanism, and only the transition judgment result field is extracted to minimize memory overhead.

[0095] In one embodiment, for the four video streams of a large parking lot monitoring system, the real-time video stream analysis method sequentially reads the transition determination result of the twelfth state breakpoint record. The result shows that the acceptance confidence coefficient is 0.82, which is used for subsequent branch decisions.

[0096] After the above reading and processing, the transitional judgment result for branch processing is obtained, which provides a direct decision basis for the next step of judging the existence of a continuous evolution process, thus seamlessly connecting the selective invocation of the event analysis model and preparing the necessary prerequisite data for the orderly construction of the final result sequence list.

[0097] S402: When the transition determination result indicates a continuous evolution process, the event analysis model is invoked to jointly analyze the state breakpoint interval and the previous frame state description and the subsequent frame state description that are connected to the beginning and end of the state breakpoint interval, and the event determination result containing the state breakpoint interval is output.

[0098] Following the reading of the transition judgment results, in order to perform deep event recognition only on semantically continuous intervals in real-time video stream analysis and avoid wasting limited resources on perturbed segments, the real-time video stream analysis method needs to call the event analysis model for joint analysis based on the transition judgment result branch. This process ensures the contextual integrity of the event judgment by fusing the state descriptions of the preceding and following frames with the breakpoint intervals, thereby accurately outputting risk-related event types and alarm information, effectively improving the reliability of the analysis and the timeliness of the event response in multi-parallel scenarios.

[0099] When the transition judgment result represents a continuous evolution process, the event analysis model is invoked to jointly analyze the state breakpoint interval and the preceding and following frame state descriptions that connect to the beginning and end of the state breakpoint interval. The event analysis model refers to an event analysis model that uses a temporal convolutional network combined with an attention mechanism. The model construction process first builds a multi-layer temporal convolutional network as the backbone feature extraction module, which includes three dilated convolutional layers. The kernel size of each convolutional layer is set to 3, and the dilation coefficients are 1, 2, and 4 respectively to capture the dependencies of different time scales. At the same time, a multi-head attention mechanism module is embedded at the end of the network. Through self-attention calculation, the intermediate frame sequence within the state breakpoint interval is weighted for key behavioral segments, and the number of attention heads is set to 8. The model input integrates the target contour, target centroid, target occupancy area, scene boundary, scene region division result, and the positional relationship of the target relative to the scene boundary from the preceding and following frame state descriptions into an initial feature vector. This initial feature vector is then input into the network together with the temporal features of the intermediate frame sequence within the state breakpoint interval. Finally, the output is a structured event judgment result record containing event type, risk level, and alarm information.

[0100] The training and optimization process of the event analysis model uses an labeled historical video sample set for iterative training. The event type of each video in the sample set is manually labeled as an intrusion event, an abnormal behavior event, or a normal event. Then, the cross-entropy loss function is used in combination with the Adam optimizer to carry out end-to-end training. The number of training rounds is set to 100 rounds, the initial learning rate is set to 0.001, and the learning rate decay is implemented after the 50th round to improve convergence stability.

[0101] In one embodiment, for the continuous evolution of the state of a worker slowly approaching dangerous equipment in a factory workshop monitoring video, the event analysis model combines the target's relative position to the safety line in the previous and subsequent frames with the movement trajectory within the breakpoint interval to output an intrusion event judgment result with a high risk level.

[0102] Through the above joint analysis and processing, a complete record containing the event determination results is formed, which provides selective output for subsequent branch processing without continuous evolution process and final result storage, thereby directly supporting the precise allocation of analysis resources.

[0103] S403: When the transition determination result indicates that there is no continuous evolution process, block the state breakpoint interval from entering the event analysis model, mark the state breakpoint interval as a disturbance segment, and output the disturbance filtering result corresponding to the state breakpoint interval.

[0104] After the transition determination result is read, in order to block irrelevant screen fluctuations from interfering with the event analysis model and save valuable inference resources, the real-time video stream analysis method needs to perform blocking and marking operations based on the transition determination result. This process ensures that only continuously evolving intervals enter deep analysis through strict conditional branching, thereby concentrating computing power on real business semantic changes. In a resource-constrained multi-channel video stream environment, this significantly reduces the false alarm rate and improves the overall processing efficiency.

[0105] When the transition determination result indicates that there is no continuous evolution process, the state breakpoint interval is blocked from entering the event analysis model. At the same time, the state breakpoint interval is marked as a disturbance segment. A disturbance segment refers to a non-semantic continuous video segment caused by lighting fluctuations, lens shake, or changes in compression state. The blocking operation skips the model call through condition judgment logic and directly generates the disturbance filtering result. The disturbance filtering result refers to the filter identifier record corresponding to the state breakpoint interval, including the disturbance type label and time interval information.

[0106] In one embodiment, for the state breakpoint interval caused by shadow swing in indoor security video, the transition determination result shows that there is no continuous evolution process that causes the interval to be blocked and marked as a disturbance segment, and the corresponding disturbance filtering result is output for subsequent log recording.

[0107] After the above blocking and marking processes, the filtering of perturbation segments is completed, providing a complete set of event judgment results and perturbation filtering results for the next step of writing the result sequence table. This ensures that the output sequence retains only the valid analysis content and lays the foundation for the final stability of the real-time video stream analysis method.

[0108] S404: Write the event determination result and the disturbance filtering result into the result sequence table respectively, and output them according to the time position of the state breakpoint interval in the real-time video stream.

[0109] After integrating the results of the aforementioned branch processing, in order to achieve structured output and traceable management of the entire real-time video stream, the real-time video stream analysis method needs to write the two types of results into a result sequence table and output them according to time position. This encapsulation and sorting process establishes the final analysis result link, so that the alarm information and filtering records of multiple video streams can be presented in an orderly manner. Under limited resource conditions, the entire closed loop from state breakpoint identification to event determination is completed, improving the completeness and credibility of the real-time video stream analysis results.

[0110] The event determination results and disturbance filtering results are written to a result sequence table. The result sequence table is a database structure indexed by the temporal position of the state breakpoint interval in the real-time video stream, supporting fast retrieval and sequential output. The write operation uses a timestamp alignment mechanism, and the output is performed according to the temporal position of the state breakpoint interval in the real-time video stream. The output format is a standardized record sequence containing the event type, alarm time, and filter identifier. Each record in the result sequence table includes at least: video source identifier, start and end times of the state breakpoint interval, transition determination result, continuity confidence coefficient, event type, risk level, alarm information, and disturbance type. For state breakpoint intervals determined to be continuous evolution processes, the event determination result field is written; for state breakpoint intervals determined to be disturbance segments, the disturbance type and filter identifier fields are written.

[0111] In one embodiment, for continuously monitored road traffic video streams, all event determination results, such as vehicle illegal parking alarms and disturbance filtering results, such as light and shadow fluctuation segments, are written into a result sequence table in chronological order, and finally a complete analysis report is output in the order of the original video timeline.

[0112] After the above writing and output processing, the core results of the real-time video stream analysis method were generated, providing the final guarantee for resource optimization and event determination stability in the scenario of parallel access of multiple real-time video streams.

[0113] The real-time video stream analysis method reads the transition judgment results corresponding to each state breakpoint record one by one. When the transition judgment result indicates that there is a continuous evolution process, it calls the event analysis model to jointly analyze the state breakpoint interval and the state descriptions of the preceding and following frames that are connected to the state breakpoint interval, and outputs the event judgment result containing the state breakpoint interval. When the transition judgment result indicates that there is no continuous evolution process, it blocks the state breakpoint interval from entering the event analysis model and marks the state breakpoint interval as a disturbance segment, outputting the disturbance filtering result corresponding to the state breakpoint interval. Finally, the event judgment result and the disturbance filtering result are written into the result sequence table respectively, and output according to the time position of the state breakpoint interval in the real-time video stream, realizing the presentation of complete intelligent analysis results of the entire real-time video stream.

[0114] Please see Figure 6 This invention provides a real-time video stream analysis system, comprising: Timed frame capture module: The real-time video stream is captured at timed intervals according to the basic analysis frame rate to obtain the basic analysis frame sequence. For each basic analysis frame, target recognition and scene occupancy recognition are performed to generate a frame state description and write it into the state sequence table. Breakpoint identification module: Reads the state descriptions of adjacent previous frames and subsequent frames in the time order of the state sequence table, performs overlap comparison, displacement comparison and continuity comparison on the positional relationship between the target's occupied area, the target's centroid and the target's position relative to the scene boundary, and extracts the state breakpoint interval when the continuity condition is not met and forms a state breakpoint record. Continuous verification module: Interpolate frames within the state breakpoint interval in each state breakpoint record to obtain a set of acceptance verification frames. Generate a state transition corridor based on the target occupancy area. Calculate the backfill deviation of the transition corridor and the main contour acceptance swing amount, input them into the acceptance discrimination model to generate the transition judgment result. Event Judgment Module: Based on the transition judgment result, when there is a continuous evolution process, the event analysis model is called to jointly analyze the state breakpoint interval and the state description of the preceding and following frames and output the event judgment result. When there is no breakpoint, the event is blocked and marked as a disturbance segment and the disturbance filtering result is output. The result is written into the result sequence table and output according to the time position.

[0115] Specifically, the above are merely preferred embodiments of this application and are not intended to limit this application.

[0116] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0117] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A real-time video stream analysis method, characterized in that, Including the following steps: S1: The real-time video stream is periodically captured according to the basic analysis frame rate to obtain the basic analysis frame sequence. For each basic analysis frame, target recognition and scene occupancy recognition are performed to generate a frame state description and write it into the state sequence table. S2: Read the state descriptions of adjacent previous frames and subsequent frames in the time order of the state sequence table, and perform overlap comparison, displacement comparison and continuity comparison on the positional relationship between the target's occupied area, the target's centroid and the target's relative to the scene boundary. When the continuity condition is not met, it is truncated as a state breakpoint interval and a state breakpoint record is formed. S3: Interpolate frames within the state breakpoint interval in each state breakpoint record to obtain a set of acceptance verification frames. Generate a state transition corridor based on the target occupancy area. Calculate the backfill deviation of the transition corridor occupancy and the main contour acceptance swing amount and input them into the acceptance discrimination model to generate the transition judgment result. S4: Based on the transition determination result, when there is a continuous evolution process, call the event analysis model to jointly analyze the state breakpoint interval and the state description of the preceding and following frames and output the event determination result. When there is no such process, block and mark it as a disturbance segment and output the disturbance filtering result. Write the result into the result sequence table and output it according to the time position.

2. The real-time video stream analysis method according to claim 1, characterized in that, Step S1 includes the following: The generation of the basic analysis frame sequence adopts a video decoder that triggers a frame capture mechanism at equal intervals on the time axis to decompress the real-time video stream and extract key image data. The resulting frame images are strictly sorted according to the extraction timestamps, and each frame retains the original timestamp and video source identifier.

3. The real-time video stream analysis method according to claim 2, characterized in that, Step S2 includes the following: The target recognition uses the YOLOv5 target detection model, which locates potential target objects through a feature extraction network and generates target contours using a boundary regression branch. At the same time, it calculates the target centroid and the target occupancy region. The scene occupancy recognition uses the DeepLabv3 semantic segmentation model, which performs pixel-level classification through multi-scale feature fusion and outputs the scene boundary, scene region division results, and the positional relationship of the target relative to the scene boundary.

4. The real-time video stream analysis method according to claim 1, characterized in that, Step S2 also includes the following: Overlap comparison calculates the ratio of the number of intersection pixels to the number of union pixels of the target-occupied area in the state description of the previous frame and the state description of the next frame. Displacement comparison calculates the Euclidean distance between the centroid coordinates of the target in the previous and next frames. Connection comparison calculates the magnitude of the change in the positional relationship vector of the target relative to the scene boundary in the previous and next frames.

5. The real-time video stream analysis method according to claim 4, characterized in that, Step S2 also includes the following: When the overlapping relationship of the target's occupied area does not meet the continuous extension condition, the displacement relationship of the target's centroid does not meet the continuous migration condition, and the positional relationship of the target relative to the scene boundary does not meet the continuous insertion condition, the original video segment between the two corresponding basic analysis frames is extracted as a state breakpoint interval and associated with the state description of the previous frame and the state description of the next frame to form a state breakpoint record.

6. The real-time video stream analysis method according to claim 1, characterized in that, Step S3 includes the following: The time-interpolation frame extraction uses a linear time interpolation frame extraction algorithm to extract the intermediate frame sequence for each state breakpoint interval according to a preset interpolation multiple to form a receiving verification frame group. The receiving verification frame group is located between the previous frame state description and the next frame state description and retains the original timestamp and pixel data.

7. The real-time video stream analysis method according to claim 6, characterized in that, Step S3 also includes the following: The state transition corridor is constructed based on the target occupancy regions in the previous frame state description and the subsequent frame state description through mathematical morphological dilation operations combined with connected component analysis. It consists of the target occupancy region in the previous frame, the target occupancy region in the subsequent frame, and the minimum connected region connecting the outer contours of the two.

8. A real-time video stream analysis method according to claim 6, characterized in that, Step S3 also includes the following: For each intermediate video frame in the acceptance verification frame group, the intermediate target occupancy area is extracted, the corridor occupancy ratio is calculated to form a corridor occupancy ratio sequence, and the target occupancy progressive sequence is constructed and the occupancy deviation value is accumulated to obtain the transition corridor occupancy backfill deviation. For each intermediate video frame, the target orientation bounding rectangle is extracted, the principal axis direction is calculated to generate a posture angle difference sequence, and the main contour acceptance swing amount is accumulated.

9. A real-time video stream analysis method according to claim 1, characterized in that, Step S4 includes the following: The event analysis model performs joint analysis on the state breakpoint interval and the state descriptions of the preceding and following frames to output the event determination results. The result sequence table is indexed by time position and outputs a standardized record sequence containing the event determination results and perturbation filtering results.

10. A real-time video stream analysis system, used to implement the real-time video stream analysis method according to any one of claims 1-9, characterized in that, include: Timed frame capture module: The real-time video stream is captured at timed intervals according to the basic analysis frame rate to obtain the basic analysis frame sequence. For each basic analysis frame, target recognition and scene occupancy recognition are performed to generate a frame state description and write it into the state sequence table. Breakpoint identification module: Reads the state descriptions of adjacent previous frames and subsequent frames in the time order of the state sequence table, performs overlap comparison, displacement comparison and continuity comparison on the positional relationship between the target's occupied area, the target's centroid and the target's position relative to the scene boundary, and extracts the state breakpoint interval when the continuity condition is not met and forms a state breakpoint record. Continuous verification module: Interpolate frames within the state breakpoint interval in each state breakpoint record to obtain a set of acceptance verification frames. Generate a state transition corridor based on the target occupancy area. Calculate the backfill deviation of the transition corridor and the main contour acceptance swing amount, input them into the acceptance discrimination model to generate the transition judgment result. Event Judgment Module: Based on the transition judgment result, when there is a continuous evolution process, the event analysis model is called to jointly analyze the state breakpoint interval and the state description of the preceding and following frames and output the event judgment result. When there is no breakpoint, the event is blocked and marked as a disturbance segment and the disturbance filtering result is output. The result is written into the result sequence table and output according to the time position.

Images