Video stream event detection and analysis method, device, medium and program product

Abstract

The invention belongs to the field of big data processing, and particularly relates to a video stream event detection and analysis method and device, a medium and a program product, an event in a video stream is detected in real time, the event duration of the event in the video stream is determined, whether the event is a target event or not is determined according to the event duration, and if the event is the target event, the video stream event is detected. If yes, obtaining event triggering time information of the target event, determining a plurality of video frames corresponding to the target event based on the event triggering time information, determining a target area of the target event in each video frame according to spatial distribution characteristics of the target event in the plurality of video frames, and performing semantic analysis on the target event in the target area; according to the method, the target event needing semantic analysis is screened based on the event duration, and the target area is accurately positioned in combination with the trigger time and spatial distribution characteristics, so that the efficiency of video stream event detection and analysis is improved.

Description

Technical Field

[0001] This application relates to the field of big data processing, and in particular to a method, device, medium, and program product for video stream event detection and analysis. Background Technology

[0002] In the field of real-time video stream analytics, scenarios such as security monitoring, intelligent transportation, industrial inspection, and smart homes all require the detection and semantic analysis of various events in video streams to achieve intelligent event recognition, early warning, and decision-making. Video streams are characterized by massive data volumes, complex event types, and high real-time requirements. They include both critical events requiring in-depth analysis (such as traffic violations, abnormal behavior, and equipment malfunctions) and a large number of invalid events with no analytical value (such as slight image shaking, brief changes in lighting, and accidental object occlusion).

[0003] In existing technologies, the analysis of video stream events mainly involves identifying all triggered events in the video stream using detection algorithms, and then performing undifferentiated semantic analysis on all events.

[0004] However, existing technologies first perform full detection of video stream events and then perform undifferentiated semantic analysis, which cannot meet the real-time requirements of the scenario and results in the ineffective consumption of analysis resources, leading to low efficiency in video stream event detection and analysis. Summary of the Invention

[0005] This application provides video stream event detection and analysis methods, devices, media, and program products to solve the problem of low efficiency in video stream event detection and analysis.

[0006] In a first aspect, embodiments of this application provide a video stream event detection and analysis method, including:

[0007] Real-time detection of events in the video stream to determine the duration of the events in the video stream;

[0008] Based on the duration of the event, determine whether the event is a target event, where the target event is an event that requires semantic analysis;

[0009] If the event is a target event, then obtain the event trigger time information of the target event;

[0010] Based on the event triggering time information, multiple video frames corresponding to the target event are determined, and the target region of the target event in each video frame is determined according to the spatial distribution characteristics of the target event in the multiple video frames.

[0011] Semantic analysis is performed on the target events within the target area.

[0012] As an optional implementation, determining whether the event is a target event based on the duration of the event includes:

[0013] Obtain the video scene type corresponding to the video stream;

[0014] According to the video scene type, the event duration threshold associated with the video scene type is retrieved from the preset threshold library. The preset threshold library stores the mapping relationship between each video scene type and its corresponding event duration threshold.

[0015] If the duration of the event is greater than or equal to the event duration threshold, then the event is determined to be the target event;

[0016] If the duration of the event is less than the event duration threshold, then the event is determined to be a non-target event.

[0017] As an optional implementation, obtaining the event trigger time information of the target event includes:

[0018] Obtain the start time of the target event in the video stream and determine the start frame time of the event;

[0019] Real-time detection of the duration of the target event in the video stream to determine the duration of the event;

[0020] Based on the event start frame time and the event duration, the event triggering event information of the target event is generated.

[0021] As an optional implementation, determining the multiple video frames corresponding to the target event based on the event trigger time information includes:

[0022] Based on the event trigger time information, extract a target event video stream segment from the video stream that matches the event trigger time information;

[0023] Frame parsing is performed on the target event video stream segment to extract multiple video frames corresponding to the target event.

[0024] As an optional implementation, determining the target region of the target event in each video frame based on the spatial distribution characteristics of the target event in the plurality of video frames includes:

[0025] Extract the pixel distribution features, contour features, and position features of the target event in each of the multiple video frames to determine the spatial distribution features of the target event;

[0026] Cluster analysis and contour fitting are performed on the spatial distribution characteristics of the target event in each video frame to determine the pixel coverage range of the target event in each video frame;

[0027] Based on the pixel coverage in all video frames, the target region of the target event in each video frame is determined.

[0028] As an optional implementation, the semantic analysis of the target events within the target region includes:

[0029] Extract the visual features of the target event within the target area, the visual features including event subject features, action features, and scene association features;

[0030] The visual features are analyzed and semantically mapped to generate a structured semantic description corresponding to the target event. The structured semantic description includes the event subject, event action, event occurrence scene, and event association relationship.

[0031] As an optional implementation, after performing semantic analysis on the target events within the target region, the method further includes:

[0032] The structured semantic description of the target event is converted into vectorized semantic features, and the vectorized semantic features are associated with the event trigger time information of the target event and the target region in each video frame, and stored in the video event library;

[0033] Upon receiving a natural language query request, the time range constraint in the natural language query request is parsed, and candidate events are selected from the video event library based on the time range constraint.

[0034] Calculate the similarity between the vectorized features of the natural language query request and the vectorized semantic features of each candidate event, sort them from high to low according to the similarity and output the search results.

[0035] Secondly, embodiments of this application provide a video stream event detection and analysis apparatus, comprising:

[0036] The determination module is used to detect events in the video stream in real time and determine the duration of the events in the video stream.

[0037] The determining module is further configured to determine whether the event is a target event based on the duration of the event, wherein the target event is an event that requires semantic analysis;

[0038] The acquisition module is also used to acquire the event triggering time information of the target event if the event is a target event;

[0039] The determining module is further configured to determine multiple video frames corresponding to the target event based on the event triggering time information, and determine the target region of the target event in each video frame according to the spatial distribution characteristics of the target event in the multiple video frames;

[0040] The analysis module also performs semantic analysis on the target events within the target area.

[0041] As an optional implementation, the video stream event detection and analysis device further includes: a retrieval module;

[0042] The acquisition module is also used to acquire the video scene type corresponding to the video stream;

[0043] The retrieval module is used to retrieve the event duration threshold associated with the video scene type from a preset threshold library according to the video scene type. The preset threshold library stores the mapping relationship between each video scene type and its corresponding event duration threshold.

[0044] The determining module is further configured to determine the event as a target event if the duration of the event is greater than or equal to the duration threshold of the event.

[0045] The determining module is further configured to determine that the event is a non-target event if the duration of the event is less than the event duration threshold.

[0046] As an optional implementation, the video stream event detection and analysis device further includes: a generation module;

[0047] The determining module is further configured to obtain the start time of the target event in the video stream and determine the start frame time of the event;

[0048] The determining module is also used to detect the duration of the target event in the video stream in real time and determine the duration of the event;

[0049] The generation module is further configured to generate event triggering information for the target event based on the event start frame time and the event duration.

[0050] As an optional implementation, the video stream event detection and analysis device further includes: a capture module and an extraction module;

[0051] The interception module is further configured to, based on the event triggering time information, intercept a target event video stream segment from the video stream that matches the event triggering time information;

[0052] The extraction module is also used to perform frame parsing on the target event video stream segment and extract multiple video frames corresponding to the target event.

[0053] As an optional implementation, the determining module is further configured to extract the pixel distribution features, contour features, and position features of the target event in each of the plurality of video frames, and determine the spatial distribution features of the target event;

[0054] The determining module is further configured to perform cluster analysis and contour fitting on the spatial distribution characteristics of the target event in each video frame to determine the pixel coverage range of the target event in each video frame;

[0055] The determining module is further configured to determine the target region of the target event in each video frame based on the pixel coverage range in all video frames.

[0056] As an optional implementation, the extraction module is further configured to extract visual features of the target event within the target area, the visual features including event subject features, action features, and scene association features;

[0057] The generation module is further configured to perform feature analysis and semantic mapping on the visual features to generate a structured semantic description corresponding to the target event. The structured semantic description includes the event subject, event action, event occurrence scene, and event association relationship.

[0058] As an optional implementation, the video stream event detection and analysis device further includes: a storage module and a filtering module;

[0059] The storage module is used to convert the structured semantic description of the target event into vectorized semantic features, and associate the vectorized semantic features with the event triggering time information of the target event and the target region in each video frame, and store them in the video event library;

[0060] The filtering module is used to parse the time range constraints in the natural language query request when a natural language query request is received, and filter candidate events from the video event library based on the time range constraints.

[0061] The analysis module is also used to calculate the similarity between the vectorized features of the natural language query request and the vectorized semantic features of each candidate event, sort them from high to low according to the similarity and output the search results.

[0062] Thirdly, embodiments of this application provide a video stream event detection and analysis device, including: a receiver, a transmitter, a memory, and a processor;

[0063] Receiver, used to receive instructions and data;

[0064] A transmitter is used to send commands and data;

[0065] The memory stores computer-executed instructions;

[0066] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0067] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0068] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0069] The video stream event detection and analysis method provided in this application effectively distinguishes between target events with analytical value and transient, accidental, or invalid interference events by introducing event duration as a filtering condition. This avoids performing computationally expensive semantic analysis on all detected events, thereby reducing system resource consumption. At the same time, it accurately locates the corresponding video frame by combining event trigger time information and accurately delineates the target region by utilizing the spatial distribution characteristics of the target event in multiple frames. This allows semantic analysis to focus on the truly relevant spatiotemporal range, which not only improves the accuracy and relevance of the analysis but also enhances the real-time performance and efficiency of the overall video stream processing system. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0071] Figure 1 Flowchart of the video stream event detection and analysis method provided in this application Figure 1 ;

[0072] Figure 2 Flowchart of the video stream event detection and analysis method provided in this application Figure 2 ;

[0073] Figure 3 A schematic diagram of the video stream event detection and analysis device provided in this application;

[0074] Figure 4 A schematic diagram of the structure of the video stream event detection and analysis device provided in this application.

[0075] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] In the field of real-time video stream analytics, applications such as security monitoring, intelligent transportation, industrial inspection, and smart homes generally rely on the accurate identification and semantic understanding of various events in videos. The video streams in these scenarios are typically characterized by large data volumes, diverse event types, and extremely high requirements for response speed. They contain both critical events with analytical value (such as traffic violations, abnormal behaviors, or equipment malfunctions) and a large amount of meaningless interference information (such as slight image jitter, brief changes in lighting, or occasional occlusions).

[0078] Current mainstream technologies typically employ a "full detection first, then unified analysis" process: that is, firstly, all triggered events in the video stream are identified through detection algorithms, and then semantic analysis is performed on these events without distinction.

[0079] However, this one-size-fits-all approach ignores the significant differences in business value and urgency among various events, making it difficult to meet the demands for high timeliness and efficient resource utilization in practical applications. Due to the lack of a dynamic mechanism for determining the importance of events, existing methods not only waste significant computing resources but may also delay response times to critical events by processing redundant information, thus reducing the overall efficiency and practicality of video stream event detection and analysis.

[0080] To address the aforementioned issues, this application provides a video stream event detection and analysis method. First, detected events in the video stream are preliminarily screened based on their duration to determine if they belong to target events requiring further processing (i.e., events with semantic analysis value). If the conditions are met, the trigger time information of the event is used to locate multiple keyframes corresponding to it in the video stream. Furthermore, the spatial distribution characteristics of the event in these frames (such as position, shape, and motion trajectory) are combined to precisely delineate the target region in each frame. Finally, semantic analysis is performed only on the content within these target regions. Keyframes refer to representative image frames selected from the time series of the video stream that can characterize the entire process of the target event's occurrence, development, and termination. These frames are not all continuous frames in the video stream, but rather a set of discrete frames that have been screened, contain the core visual information of the target event, and effectively reflect the spatiotemporal evolution characteristics of the event. These frames are used for subsequent spatial localization and semantic analysis to avoid computational redundancy on invalid frames. This method effectively eliminates transient or irrelevant interference events, avoiding computational redundancy caused by full-frame or full-time-period analysis, thereby improving the accuracy and efficiency of video event detection and semantic understanding.

[0081] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0082] Figure 1 Flowchart of the video stream event detection and analysis method provided in this application Figure 1 The execution entity in this embodiment is, for example, a video stream event detection and analysis system. Figure 1 As shown, the method includes:

[0083] S101: Real-time detection of events in the video stream to determine the duration of each event in the video stream.

[0084] In this context, a video stream is a dynamic data stream composed of a series of consecutive image frames arranged in chronological order. The duration of an event is the length of time from the start of the event to its end.

[0085] Specifically, each frame of the video stream is analyzed and processed to identify the start and end markers of the event. For example, using object detection algorithms in computer vision, the event is marked as starting when a specific object is detected, and as ending when the object disappears or the behavior ends. The duration of the event is obtained by calculating the time difference between the start and end times.

[0086] For example, in a video stream monitoring a traffic intersection, events of vehicles running red lights can be detected in real time. When a vehicle enters the intersection while the red light is on, the event is marked as beginning; when the vehicle has completely exited the intersection, the event is marked as ending. By recording these two moments, the duration of the event of the vehicle running the red light can be calculated.

[0087] S102: Based on the duration of the event, determine whether the event is a target event. A target event is an event that requires semantic analysis.

[0088] Specifically, one or more threshold ranges for the duration of events are preset, and the duration of detected events is compared with these threshold ranges. If the duration of an event falls within the preset target threshold range, the event is determined to be a target event; otherwise, it is determined to be a non-target event.

[0089] For example, in a traffic intersection video stream, suppose a red-light violation lasting more than 3 seconds is defined as a target event. If a vehicle is detected running a red light for 5 seconds, since 5 seconds is within the set target threshold, the event is determined to be a target event; if another vehicle runs a red light for 2 seconds, which is not within the target threshold, it is determined to be a non-target event.

[0090] S103: If the event is the target event, then obtain the event trigger time information of the target event.

[0091] Specifically, once an event is identified as the target event, precise time information is extracted from the relevant data recording the start of the event. This can be achieved by using the timestamp system of the video stream and combining it with the moment the event was detected to obtain the event trigger time information, typically expressed in the form of year, month, day, hour, minute, and second.

[0092] For example, in a traffic intersection video stream, after determining that a vehicle running a red light is the target event, checking the time record of the video stream reveals that the timestamp corresponding to the start time of the event is displayed as 10:30:15 on May 10, 2024. This time information is the event trigger time information of the target event.

[0093] S104: Based on the event trigger time information, determine the multiple video frames corresponding to the target event, and determine the target region of the target event in each video frame according to the spatial distribution characteristics of the target event in the multiple video frames.

[0094] Spatial distribution characteristics refer to the spatial features of the target event, such as its position, size, and shape, within these video frames.

[0095] Specifically, based on the event trigger time information, the corresponding time period is located in the video stream, and multiple video frames within that time period are extracted. Then, using techniques such as object detection or image segmentation, the spatial information of the target event, such as its position and shape, in these video frames is analyzed to determine the target region that accurately covers the main content of the target event.

[0096] For example, in the case of a vehicle running a red light, 10 video frames before and after the event are extracted based on the event trigger time information. Image analysis reveals that the vehicle is mainly concentrated in the center-left position of these 10 frames and is rectangular in shape. Therefore, the rectangular area containing the vehicle is delineated as the target area in each frame.

[0097] S105: Perform semantic analysis on target events within the target area.

[0098] Semantic analysis is the process of deeply understanding and interpreting the meaning, behavior, and state conveyed by target events within a target area.

[0099] Specifically, techniques such as natural language processing and semantic understanding from computer vision are used to perform feature extraction and model analysis on image information within the target area. For example, deep learning models are used to identify objects and actions within the target area, and contextual information is combined to understand the semantic meaning of events.

[0100] For example, in response to a red light violation incident within a previously identified target area, semantic analysis technology can be used to identify the vehicle's color and model, determine the vehicle's direction of travel and speed when it ran the red light, and ultimately arrive at a semantically clear conclusion such as "a red sedan ran a red light from east to west."

[0101] The video stream event detection and analysis method provided in this application performs real-time detection of events in the video stream, determines the duration of each event in the video stream, and determines whether the event is a target event based on the duration. A target event is an event that requires semantic analysis. If the event is a target event, the method obtains the event trigger time information, determines multiple video frames corresponding to the target event based on the event trigger time information, and determines the target region of the target event in each video frame based on the spatial distribution characteristics of the target event in the multiple video frames. Semantic analysis is then performed on the target event within the target region. This method improves the efficiency of video stream event detection and analysis by filtering target events requiring semantic analysis based on event duration and accurately locating the target region by combining its trigger time and spatial distribution characteristics, avoiding redundant processing of invalid events.

[0102] Figure 2 Flowchart of the video stream event detection and analysis method provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, a video stream event detection and analysis method is described in detail, which includes:

[0103] S201: Perform real-time detection of events in the video stream to determine the duration of each event in the video stream.

[0104] Step S201 is similar to step S101, and will not be described again here.

[0105] S202: Obtain the video scene type corresponding to the video stream.

[0106] Among them, video scene type refers to the classification of the actual physical environment and application scenario captured by the video stream, reflecting the shooting background and monitoring / collection purpose of the video stream. Different scene types correspond to different event analysis requirements and threshold standards.

[0107] Specifically, a video scene type classification library is pre-established, containing feature templates for various typical scenes (such as traffic intersections, school playgrounds, shopping mall lobbies, highways, etc.); scene recognition algorithms are used to extract key scene features from the video stream (such as background elements, dynamic target types, environmental markers, etc.); the extracted scene features are matched with feature templates in the classification library, and combined with scene feature similarity judgment rules to determine the unique video scene type corresponding to the video stream; if there are ambiguous scenes, the location of the video stream's acquisition device and preset parameters can be used to assist in the judgment.

[0108] For example, a traffic intersection surveillance video stream is used to determine its video scene type. Using a scene recognition algorithm, background features (traffic lights, stop lines, intersection signs, zebra crossings) and dynamic target features (vehicles, pedestrians, non-motorized vehicles) are extracted from the video stream. These features are then matched against a pre-defined scene type classification library. Since the extracted features have a 98% similarity to the "traffic intersection" scene template, far exceeding the similarity threshold (80%), the video scene type corresponding to this video stream is determined to be "traffic intersection".

[0109] S203: Based on the video scene type, retrieve the event duration threshold associated with the video scene type from the preset threshold library. The preset threshold library stores the mapping relationship between each video scene type and its corresponding event duration threshold.

[0110] The preset threshold library is a pre-built structured database that stores a one-to-one mapping relationship between each video scene type and the corresponding event duration threshold.

[0111] Specifically, firstly, a preset threshold library is constructed and stored according to video scene type. Each scene type corresponds to at least one event duration threshold (thresholds can be further subdivided according to event type). The mapping relationship between each scene type and the threshold is clearly defined and an index is established. After determining the scene type of the video stream, the preset threshold library is quickly queried through the index based on the scene type as the search keyword. The event duration threshold that is precisely associated with the scene type is retrieved. If the scene type corresponds to multiple thresholds, the matching thresholds need to be further filtered in combination with the currently detected event type.

[0112] For example, a preset threshold library stores the mapping relationship between various scene types and event duration thresholds. The "traffic intersection" scene corresponds to two core event thresholds: a red-light violation threshold of 3 seconds and a vehicle illegal parking threshold of 10 seconds. The current video stream scene type is "traffic intersection," and the detected event is a vehicle running a red light. Therefore, the event duration threshold associated with the red-light violation event in the "traffic intersection" scene is retrieved from the threshold library, and it is set to 3 seconds.

[0113] S204: If the duration of an event is greater than or equal to the event duration threshold, then the event is determined to be the target event.

[0114] Specifically, the process involves obtaining the determined duration of the event and the corresponding event duration threshold for the scene; comparing the two values ​​and executing the judgment logic: if the event duration is greater than or equal to the event duration threshold, it indicates that the event has sufficient duration and is determined to be the target event; after the judgment is completed, relevant information about the event (duration, scene type, threshold, etc.) is recorded.

[0115] For example, the duration of a vehicle running a red light is 5.5 seconds, and the threshold for the duration of a red light running event in the "traffic intersection" scenario is 3 seconds. By comparing the values, 5.5 seconds ≥ 3 seconds, which meets the criteria for determining a target event. Therefore, the vehicle running a red light event is determined to be the target event.

[0116] S205: If the duration of an event is less than the event duration threshold, the event is determined to be a non-target event.

[0117] Non-target events refer to events whose duration is less than the event duration threshold of the corresponding scenario. These events are short-lived and do not require subsequent semantic analysis, storage, or retrieval. They are usually occasional, momentary, and invalid events.

[0118] Specifically, the process involves obtaining the determined duration of the event and the corresponding event duration threshold for the scene; comparing the two values ​​and executing the judgment logic: if the event duration is less than the event duration threshold, it indicates that the event duration is too short and has no analytical significance, and the event is judged as a non-target event; after the judgment is completed, only the basic information of the event is recorded (no need to retain frame data), without triggering subsequent processes, and the process directly returns to the event detection stage to continue detecting the next event in the video stream.

[0119] For example, in a traffic intersection surveillance video stream, if another vehicle is detected running a red light, and its front wheels cross the stop line before immediately stopping and reversing back inside the stop line, the duration of this event is recorded as 1.2 seconds. The threshold for red light running events is 3 seconds. Since 1.2 seconds is less than 3 seconds, the event is determined to be a non-target event. Only a brief record of the event's occurrence time is made, without performing subsequent time localization, semantic analysis, or other operations, and the behavior of subsequent vehicles is continued to be detected.

[0120] S206: If the event is a target event, then obtain the start time of the target event in the video stream and determine the start frame time of the event.

[0121] Among them, the event start frame time refers to the specific video frame number and time corresponding to the start time of the target event. It is the basis for locating the starting position of the target event in the video stream and is related to the frame rate of the video stream.

[0122] Specifically, once an event is identified as a target event, the timestamp of the video stream corresponding to the recorded start time of the event is retrieved to extract the precise start time of the event. Combined with the frame rate parameters of the video stream (such as 25 frames per second or 30 frames per second), the video frame number corresponding to the start time of the event is calculated to determine the start frame time of the event. At the same time, the image data of the start frame is stored, and the event start flag in the start frame is marked.

[0123] For example, for a vehicle running a red light, after the event is identified as a target event, the timestamp of the video stream at the start time of the event is retrieved, and the start time of the event is obtained as 10:30:15.0 on May 10, 2024. It is known that the frame rate of the video stream of the traffic intersection monitoring is 10 frames / second (each frame interval is 0.1 seconds), and the video frame number corresponding to 10:30:15.0 is calculated to be frame number 1030150. Therefore, the start frame time of the target event is determined to be 10:30:15.0 on May 10, 2024, corresponding to frame number 1030150.

[0124] S207: Real-time detection of the duration of a target event in the video stream to determine the duration of the event.

[0125] The duration of an event refers to the complete time interval from the start to the end of the target event.

[0126] Specifically, after determining the start frame time of the event, real-time frame parsing and event detection are performed on the video stream to continuously track the development process of the target event; relying on behavior recognition algorithms, it is determined in real time whether the target event has reached the end mark; when the event end mark is detected (such as the target disappearing or the behavior ending), the timestamp and video frame number corresponding to the event end time are recorded; the event start time and event end time are associated to form a complete event duration, clarifying the time span of the target event in the video stream, and verifying that the time difference of the event duration is consistent with the determined event duration.

[0127] For example, regarding the aforementioned target event of a vehicle running a red light, after determining the starting frame time as 10:30:15.0 (frame number 1030150), the vehicle's behavior continues to be monitored in real time. When it is detected that the rear wheels of the vehicle have completely crossed the stop line at the intersection, and the red light running behavior ends, the event end time is recorded as 10:30:20.5 on May 10, 2024, corresponding to frame number 1030205. Thus, the event duration of the target event is determined to be "10:30:15.0 on May 10, 2024 - 10:30:20.5 on May 10, 2024", and the time difference of this period is verified to be 5.5 seconds.

[0128] S208: Generate event triggering information for the target event based on the event start frame time and event duration.

[0129] Specifically, the event start frame time (including timestamp and frame number) and the event duration (start time - end time) are retrieved and integrated to generate event trigger time information.

[0130] For example, regarding the aforementioned incident of a vehicle running a red light, the relevant time parameters are retrieved: the event start frame time is "May 10, 2024, 10:30:15.0 (frame number 1030150)", the event duration is "May 10, 2024, 10:30:15.0 - May 10, 2024, 10:30:20.5", and the event duration is "5.5 seconds". Following a structured format, the event trigger time information for this target event is generated as follows: {Event start time: 2024-05-10 10:30:15.0, start frame number: 1030150, event duration: 2024-05-10 10:30:15.0 - 2024-05-10 10:30:20.5, event duration: 5.5 seconds}.

[0131] S209: Based on the event trigger time information, extract the target event video stream segment from the video stream that matches the event trigger time information.

[0132] Specifically, the event duration (start time - end time) is extracted from the event trigger time information and used as the time range for video clip extraction. Based on video stream extraction technology, the corresponding video clip is accurately extracted from the complete video stream according to this time range. During the extraction process, the frame rate and resolution of the video clip are ensured to be consistent with the original video stream, and the key frames of the target event are not lost. After the extraction is completed, the video clip is associated and marked with the event trigger time information.

[0133] For example, in the aforementioned incident of a vehicle running a red light, the duration of the time period "10:30:15.0 - 10:30:20.5" in the event trigger time information is extracted. Using a video capture tool, a video segment within this time range is accurately extracted from the complete monitoring video stream of the traffic intersection. The segment is 5.5 seconds long and covers the complete process of the vehicle's front wheels crossing the line, running the red light, and the rear wheels leaving the intersection. After the extraction is completed, the video segment is marked as "2024-05-10 10:30:15.0 Vehicle running a red light segment" and stored in association with the event trigger time information.

[0134] S210: Perform frame parsing on the target event video stream segment and extract multiple video frames corresponding to the target event.

[0135] Among them, the multiple video frames corresponding to the target event refer to the key image frames extracted from the video stream segments that can fully present the development process of the target event, including the event start frame, development process frames, and end frame. Key image frames refer to single static images in the video stream segments that, after sharpness detection and noise filtering, can clearly show the subject of the target event and its behavioral changes; these frames are the basic units constituting the temporal evolution of the event, and their selection criteria include meeting image quality standards (no blur, no serious noise) and being evenly distributed on the timeline to cover the entire life cycle of the event.

[0136] Specifically, a frame parsing algorithm is used to decompose the target event video stream segment frame by frame, obtaining all single-frame images in the segment. Each frame undergoes sharpness detection and noise filtering to remove blurry, stuttering, or excessively noisy invalid frames. Based on the event duration and frame rate, multiple key video frames are extracted evenly (ensuring coverage of the entire event's start, development, and end). The number of extracted frames can be adjusted according to the event duration (typically one frame every 1-2 seconds). These key video frames are then sorted chronologically, each frame's corresponding time point is marked, and they are associated with and stored in relation to the video stream segment and event trigger time information. A key video frame is an independent frame in a dynamic image sequence extracted from the target event video stream segment at preset time intervals or event state change points, carrying key semantic information about the target event. Unlike key image frames, which focus on the static attributes of a single frame, key video frames emphasize their sequential attributes on the video timeline; that is, when these frames are arranged chronologically, they can reconstruct the complete action trajectory and state transitions of the target event.

[0137] For example, the 5.5-second video clip of a vehicle running a red light is analyzed. This clip has a frame rate of 10 frames per second and contains a total of 55 frames. Through sharpness detection, 3 blurry frames (frame numbers 1030152, 1030180, and 1030203) are removed. Following the rule of extracting 1 frame every 0.5 seconds, 11 key video frames are evenly extracted, corresponding to times 10:30:15.0, 10:30:15.5, ..., 10:30:20.5, covering the complete process of the vehicle crossing the line, driving, and exiting the intersection. These 11 video frames are sorted in chronological order, and the time corresponding to each frame is marked and associated with the video clip for storage.

[0138] S211: Extract the pixel distribution features, contour features, and position features of the target event in each of the multiple video frames to determine the spatial distribution features of the target event.

[0139] Among them, pixel distribution features refer to the distribution pattern of grayscale and color values ​​of different pixels in a single frame image of a target event, reflecting the visual attributes of the target event such as color and brightness.

[0140] Contour features refer to the edge contour shape, contour size, contour complexity, and other features of a target event in a single frame image, which are used to distinguish the target event from the background.

[0141] Location features refer to the specific coordinates of a target event in a single frame image, its offset relative to the background, and other characteristics that reflect the spatial location changes of the target event.

[0142] Spatial distribution features integrate the pixel distribution, contour, and position features of each frame to form the overall spatial distribution pattern of the target event, reflecting the spatial morphology and positional change trend of the target event.

[0143] Specifically, for each extracted key video frame, an image feature extraction algorithm is used to extract the pixel distribution features (grayscale histogram, color histogram), contour features (contour point coordinates, contour perimeter, contour area), and position features (target center coordinates, position offset relative to the stop line / traffic light) of the target event. The three types of features of each frame are standardized to eliminate inter-frame differences. The three types of features of all key frames are integrated to analyze the spatial morphological changes and position movement patterns of the target event, and finally determine the spatial distribution features of the target event to form a feature set.

[0144] For example, based on the 11 key video frames extracted above, three types of features were extracted for the vehicle running a red light: In terms of pixel distribution, the color histogram of the vehicle area is predominantly red (the vehicle is red), with grayscale values ​​concentrated between 150 and 200; in terms of contour features, the vehicle contour is rectangular, with a perimeter of approximately 800 pixels and an area of ​​approximately 40,000 pixels; in terms of position, from 10:30:15.0 to 10:30:20.5, the vehicle's center coordinates gradually moved from (300, 400) to (800, 420), generally moving to the right side of the screen (the direction of exiting the intersection). Integrating these three types of features, the spatial distribution characteristics of the target event were determined as follows: a red rectangular target, moving from the lower left side of the screen (near the stop line) to the right, with a stable contour size and uniform color distribution.

[0145] S212: Perform cluster analysis and contour fitting on the spatial distribution characteristics of target events in each video frame to determine the pixel coverage of target events in each video frame.

[0146] Cluster analysis refers to grouping the spatial distribution features (pixels, contours, and positions) of target events in each video frame according to their similarity, eliminating abnormal feature points, retaining the core features of the target events, and ensuring the consistency of features.

[0147] Contour fitting refers to constructing a closed region that can completely enclose the contour of a target event based on its contour features, using fitting algorithms (such as rectangle fitting and polygon fitting), thereby optimizing the integrity and accuracy of the contour.

[0148] Pixel coverage refers to the smallest pixel area in a single frame of an image that can completely encompass the target event, reflecting the spatial extent of the target event in that frame.

[0149] Specifically, clustering analysis algorithms (such as K-means clustering) are used to cluster the spatial distribution features of target events in each video frame, eliminating abnormal feature points caused by image noise and background interference, and retaining the core features of the target events. For the clustered contour features in each frame, a contour fitting algorithm is used to construct a closed fitting region that can completely wrap the contour of the target event (preferably using the fitting method with the highest matching degree to the target contour). Based on the pixel coordinates of the fitting region, the pixel coverage range of the target event in that frame is determined, and the minimum and maximum pixel coordinates of the coverage range are specified to ensure that the coverage range can completely contain the target event, without omitting the core area or including too many background pixels.

[0150] For example, for the 11 key video frames mentioned above, K-means clustering analysis was performed on the spatial distribution features of the vehicles to remove three abnormal contour points caused by background interference (shadows of roadside trees), while retaining the core contour features of the vehicles. Rectangular fitting was performed on the vehicle contours in each frame to construct a rectangular fitting region that could completely wrap around the vehicle. Taking the frame corresponding to 10:30:15.0 as an example, the pixel coordinate range of the fitted rectangular region is (250, 350) - (350, 450), that is, the pixel coverage range of the vehicle in this frame is 250-350 pixels on the x-axis and 350-450 pixels on the y-axis. Similarly, the pixel coverage range of the vehicles in the remaining 10 frames was extracted. The coverage range of each frame shifted slightly with the movement of the vehicle position, but all of them could completely wrap around the main body of the vehicle.

[0151] S213: Based on the pixel coverage in all video frames, determine the target area of ​​the target event in each video frame.

[0152] Specifically, the pixel coverage range of the target event in each video frame is retrieved; the pixel coverage range of each frame is fine-tuned and optimized, and a small number of background pixels (such as road surface and shadows at the edge) are removed from the coverage range to ensure that the coverage range only contains the main body of the target event; the optimized pixel coverage range is used as the target area of ​​the target event in that frame, and the pixel coordinate range of the target area is defined; at the same time, the correspondence between the target area and the target event in each frame is marked, and the target area is associated with and stored with the video frame and spatial distribution features.

[0153] For example, based on the pixel coverage of the vehicle in the aforementioned 11 frames, the target region for each frame is determined. Taking the frame corresponding to 10:30:15.0 as an example, its pixel coverage is (250,350)-(350,450). After fine-tuning and removing the road background with an edge of 5 pixels, the optimized target region pixel coordinate range is (255,355)-(345,445). This region only contains the red vehicle body, excluding roadside shadows and road textures. Similarly, the pixel coverage of the remaining 10 frames is fine-tuned to determine the target region for each frame. The target region of each frame shifts synchronously with the movement of the vehicle, always accurately encompassing the vehicle body, providing a focusing range for subsequent extraction of the vehicle's visual features.

[0154] S214: Extract the visual features of the target event within the target area. The visual features include event subject features, action features, and scene association features.

[0155] Among them, visual features refer to the set of visual information within the target area that can reflect the essential attributes of the target event. They are the core basis for semantic analysis and are divided into three categories: event subject features (attribute features of the core subject of the target event), action features (behavioral action features of the subject of the target event), and scene association features (association features between the target event and surrounding scene elements).

[0156] Specifically, focusing on the target area of ​​each video frame, a multi-dimensional visual feature extraction algorithm is used to extract three types of visual features: event subject features, which extract attributes such as the color, shape, size, and type of the subject; action features, which extract features such as the motion trajectory, speed, and action state of the subject through inter-frame comparison; and scene association features, which extract features such as the relative position and association relationship between the subject and surrounding scene elements (such as traffic lights, stop lines, and intersection signs). The three types of visual features are integrated to form a visual feature set of the target event, ensuring the completeness and accuracy of the features.

[0157] For example, for the vehicle target area in each of the above frames, three types of visual features are extracted: event subject features, the vehicle color is red, the model is a sedan, the body length is approximately 4.5 meters, and the width is approximately 1.8 meters; action features, the vehicle's trajectory is from east to west (from left to right of the frame), the speed is approximately 30 km / h, and the action state is constant speed driving without stopping or turning; scene association features, when the vehicle is moving, the surrounding traffic lights are red, the vehicle's initial position is inside the stop line (illegally crossing the line), it crosses the zebra crossing during its journey, and finally exits the intersection, showing a correlation with the traffic lights, stop line, and zebra crossing. Integrating these three types of features forms a visual feature set for the target event.

[0158] S215: Perform feature analysis and semantic mapping on visual features to generate a structured semantic description of the target event. The structured semantic description includes the event subject, event action, event occurrence scene, and event relationship.

[0159] Feature analysis refers to the in-depth interpretation, classification and integration of extracted visual features to uncover the event meaning, behavioral logic and correlations behind the visual features.

[0160] Semantic mapping refers to mapping visual features (unstructured image information) into structured natural language descriptions, establishing a correspondence between visual features and semantic concepts.

[0161] Structured semantic description refers to the use of natural language with a fixed format and clear classification to comprehensively describe a target event, including four core elements: event subject, action, scene, and relationship.

[0162] Specifically, feature analysis is performed on the visual feature set to classify and interpret the event subject features, action features, and scene association features, clarifying the semantic meaning of each feature; a mapping relationship between visual features and semantic concepts is established (e.g., "red, car shape" maps to "red car", "red light, crossing the line" maps to "running a red light"); and according to a preset structured format, the four elements of the event subject, event action, event scene, and event association are integrated to generate a structured semantic description.

[0163] For example, the visual features of the aforementioned red-light running incident are analyzed and semantically mapped: the subject of the incident is "a red sedan" (derived from subject feature mapping); the action of the incident is "driving at a constant speed from east to west through a red light for 5.5 seconds" (derived from action feature mapping); the scene of the incident is "traffic intersection" (derived from scene association feature mapping); the relationship between the incident and the incident is "the traffic light was red while driving, the vehicle initially crossed the stop line, crossed the zebra crossing during the journey, and finally exited the intersection" (derived from scene association feature mapping). After integration, a structured semantic description is generated: "Subject of the incident: a red sedan; Action of the incident: driving at a constant speed of approximately 30 km / h from east to west through a red light for 5.5 seconds; Scene of the incident: traffic intersection; Relationship between the incident and the incident: the traffic light was red during the journey, the vehicle initially crossed the stop line, crossed the zebra crossing, and then exited the intersection, constituting a violation of running a red light."

[0164] S216: Convert the structured semantic description of the target event into vectorized semantic features, associate the vectorized semantic features with the event trigger time information of the target event and the target region in each video frame, and store them in the video event library.

[0165] Among them, vectorized semantic features refer to the use of natural language processing technology to convert structured semantic descriptions (natural language) into vector forms (numerical vectors) that can be recognized and computed by computers. Each dimension of the vector corresponds to a semantic feature, which is used for subsequent similarity calculation and event retrieval.

[0166] A video event library is a structured database used to store information related to all target events. Its core storage includes vectorized semantic features, event trigger time information, target areas, and other data. It also features associated storage and fast retrieval capabilities.

[0167] Specifically, semantic vectorization algorithms (such as Word2Vec and BERT) are used to convert the structured semantic description of the target event into fixed-dimensional vectorized semantic features, ensuring that the vectors can accurately represent semantic information; the event triggering time information of the target event and the target region data in each video frame are retrieved; the relationship between the three is established (the vectorized semantic features serve as the core index, associating the corresponding time information and target region); and all the associated data are stored in the video event library according to the preset data format, while indexes (time index and semantic vector index) are established.

[0168] For example, the structured semantic description of the aforementioned vehicle running a red light event is converted into 128-dimensional vectorized semantic features (vector values ​​such as [0.12, 0.35, ..., 0.28]) using the BERT algorithm; the event trigger time information (start time, duration, etc.) and target region data (pixel coordinate range) from 11 video frames are retrieved; the relationship between the three is established, using the vectorized semantic features as the core index to associate the corresponding time information and target region; according to the data format requirements, the associated data is stored in the video event database, and a time index (2024-05-10 10:30:15.0) and a semantic vector index are established for subsequent natural language retrieval.

[0169] S217: Upon receiving a natural language query request, parse the time range constraints in the natural language query request and filter candidate events from the video event library based on the time range constraints.

[0170] Among them, a natural language query request refers to a query statement entered by the user in natural language form to retrieve target events in the video event library, which includes query intent and constraints (such as time range, event type, etc.).

[0171] Time range constraints refer to the explicit or implicit time interval of the event in the natural language query request (such as "around 10:30 on May 10, 2024" or "10:30-10:31 on May 10, 2024"), and are the core constraints for screening candidate events.

[0172] Candidate events refer to all target events selected from the video event library whose event trigger time information meets the user's query time range constraints.

[0173] Specifically, semantic parsing is performed on the received natural language query requests to extract the time range constraints in the query; the extracted time range constraints are standardized (converted to the format of "start time-end time"), and ambiguous time expressions (such as "around" or "nearby") are processed and expanded into reasonable time intervals; relying on the time index of the video event library, all target events whose event trigger time information (start time or duration) falls within the standardized time range are retrieved; these events are used as candidate events, and their vectorized semantic features, time information, target areas, and other data are extracted.

[0174] For example, a user inputs a natural language query request: "Query red light running incidents at traffic intersections between 10:30 AM and 10:31 AM on May 10, 2024". Using a natural language parsing model, the time range constraint is extracted as "10:30-10:31 AM on May 10, 2024", which is standardized to "2024-05-10 10:30:00.0 - 2024-05-10 10:31:00.0". Based on the time index of the video event database, target events whose trigger time information falls within this interval are retrieved. The aforementioned red car running a red light incident (triggered from 10:30:15.0 to 10:30:20.5) meets the criteria and is selected as a candidate event. Simultaneously, the vectorized semantic features, time information, target area, and other data of this candidate event are extracted.

[0175] S218: Calculate the similarity between the vectorized features of the natural language query request and the vectorized semantic features of each candidate event, sort them from high to low according to the similarity and output the search results.

[0176] The similarity refers to the degree of matching between the vectorized features of the natural language query request and the vectorized semantic features of the candidate event. It is calculated using similarity algorithms (such as cosine similarity and Euclidean distance). The closer the value is to 1, the higher the matching degree; the closer the value is to 0, the lower the matching degree.

[0177] The search results refer to the set of results that are output after candidate events are sorted from high to low similarity, and contain the core information of the events (structured semantic description, time information, target region, etc.).

[0178] Specifically, a similarity calculation algorithm (using cosine similarity) is employed to calculate the similarity between the vectorized features of the natural language query request (derived through the same vectorization algorithm) and the vectorized semantic features of each candidate event, obtaining a similarity value between each candidate event and the query request. All candidate events are then sorted according to their similarity values ​​from highest to lowest. Candidate events with similarity values ​​higher than a preset threshold (e.g., 0.7) are selected, and their structured semantic descriptions, event trigger time information, target regions in each frame, video stream segments, and other core information are extracted and organized into a standardized search result format. The sorted search results are then output to the user, ensuring that the results are clear, accurate, and relevant to the user's query intent.

[0179] For example, for the user's natural language query request, the query request is converted into a vectorized feature with the same dimensions (128 dimensions) as the candidate event using the BERT algorithm; the cosine similarity algorithm is used to calculate the similarity between the vectorized feature of the query and the vectorized semantic feature of the candidate event of the red car running a red light, and the similarity value is 0.92 (close to 1, extremely high matching degree); since this candidate event is the only event that meets the time range constraint, it is used as the only search result, and the core information is organized as required: "Search result (similarity: 0.92): Event subject: a red car; Event action: driving from east to west at a constant speed of about 30 km / h through a red light, lasting for 5.5 seconds; Event scene: traffic intersection; Event trigger time: May 10, 2024, 10:30:15.0 - 10:30:20.5; Target area: the rectangular area that accurately wraps the main body of the vehicle in each frame; Associated video clip: 5.5-second red light running clip"; the result is output to the user to complete the search.

[0180] The video stream event detection and analysis method provided in this application detects events in the video stream in real time. First, it determines the duration of the event and retrieves the corresponding duration threshold from a preset threshold library based on the current video scene type to determine whether the event is a target event. For instances identified as target events, the system obtains the start time and determines the event's starting frame and duration, generating precise event trigger time information, and then extracts the corresponding target event video segment. Subsequently, the segment is parsed frame by frame, extracting spatial features such as pixel distribution, contour, and position of the target event in each frame. The target area in each frame is determined through clustering and contour fitting. The system first extracts visual features, including subject, action, and scene association, from the domain. These features are then semantically mapped to generate structured semantic descriptions and transformed into vectorized semantic features. These features are then associated with time information and target regions and stored in the video event database. When a natural language query is received, the system parses the temporal constraints, filters candidate events from the event database, and sorts and outputs the results by calculating the similarity between the query statement vector and the semantic vectors of the candidate events. The entire process uses the duration of the event as the filtering criterion and combines spatiotemporal features to accurately focus on the analysis object, effectively avoiding redundant processing of short-lived and invalid events, and improving the overall efficiency of video stream event detection and semantic analysis.

[0181] Figure 3 This is a schematic diagram of the structure of the video stream event detection and analysis device provided in this application, as shown below. Figure 3 As shown, the video stream event detection and analysis device 300 provided in this embodiment includes:

[0182] The determination module 301 is used to detect events in the video stream in real time and determine the duration of the event in the video stream;

[0183] The determination module 301 is also used to determine whether an event is a target event based on the duration of the event. A target event is an event that needs to be semantically analyzed.

[0184] The acquisition module 302 is also used to acquire the event trigger time information of the target event if the event is a target event;

[0185] The determining module 301 is also used to determine multiple video frames corresponding to the target event based on the event triggering time information, and to determine the target region of the target event in each video frame according to the spatial distribution characteristics of the target event in the multiple video frames.

[0186] Analysis module 303 also performs semantic analysis on target events within the target area.

[0187] As an optional implementation, the video stream event detection and analysis device further includes: a retrieval module 304;

[0188] The acquisition module 302 is also used to acquire the video scene type corresponding to the video stream;

[0189] The retrieval module 304 is used to retrieve the event duration threshold associated with the video scene type from a preset threshold library according to the video scene type. The preset threshold library stores the mapping relationship between each video scene type and its corresponding event duration threshold.

[0190] The determination module 301 is also used to determine the event as the target event if the duration of the event is greater than or equal to the event duration threshold.

[0191] The determination module 301 is also used to determine that the event is a non-target event if the duration of the event is less than the event duration threshold.

[0192] As an optional implementation, the video stream event detection and analysis device further includes: a generation module 305;

[0193] The determination module 301 is also used to obtain the start time of the target event in the video stream and determine the start frame time of the event;

[0194] The determination module 301 is also used to detect the duration of the target event in the video stream in real time and determine the duration of the event.

[0195] The generation module 305 is also used to generate event triggering information of the target event based on the event start frame time and the event duration.

[0196] As an optional implementation, the video stream event detection and analysis device further includes: a capture module 306 and an extraction module 307;

[0197] The interception module 306 is also used to intercept a target event video stream segment from the video stream that matches the event trigger time information based on the event trigger time information;

[0198] The extraction module 307 is also used to perform frame parsing on the target event video stream segment and extract multiple video frames corresponding to the target event.

[0199] As an optional implementation, the determining module 301 is further used to extract the pixel distribution features, contour features and position features of the target event in each of the multiple video frames, and to determine the spatial distribution features of the target event;

[0200] The determination module 301 is also used to perform cluster analysis and contour fitting on the spatial distribution characteristics of target events in each video frame to determine the pixel coverage range of target events in each video frame.

[0201] The determination module 301 is also used to determine the target region of the target event in each video frame based on the pixel coverage in all video frames.

[0202] As an optional implementation, the extraction module 307 is also used to extract visual features of target events within the target area, including event subject features, action features, and scene association features;

[0203] The generation module 303 is also used to perform feature analysis and semantic mapping on visual features to generate a structured semantic description corresponding to the target event. The structured semantic description includes the event subject, event action, event occurrence scene, and event relationship.

[0204] As an optional implementation, the video stream event detection and analysis device further includes: a storage module 308 and a filtering module 309;

[0205] Storage module 308 is used to convert the structured semantic description of the target event into vectorized semantic features, and associate the vectorized semantic features with the event trigger time information of the target event and the target region in each video frame, and store them in the video event library;

[0206] The filtering module 309 is used to parse the time range constraints in the natural language query request when a natural language query request is received, and filter candidate events from the video event library based on the time range constraints.

[0207] The analysis module 303 is also used to calculate the similarity between the vectorized features of the natural language query request and the vectorized semantic features of each candidate event, sort them from high to low according to the similarity and output the search results.

[0208] Figure 4 A schematic diagram of the video stream event detection and analysis device provided in this application. Figure 4 As shown, this application provides a video stream event detection and analysis device 400, which includes: a receiver 401, a transmitter 402, a processor 403, and a memory 404.

[0209] Receiver 401 is used to receive instructions and data;

[0210] Transmitter 402 is used to send commands and data;

[0211] Memory 404 is used to store instructions executed by the computer;

[0212] Processor 403 is used to execute computer execution instructions stored in memory 404 to implement the various steps performed by the video stream event detection and analysis method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the video stream event detection and analysis method.

[0213] Alternatively, the memory 404 can be either standalone or integrated with the processor 403.

[0214] When the memory 404 is set up independently, the electronic device also includes a bus for connecting the memory 404 and the processor 403.

[0215] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video stream event detection and analysis method performed by the video stream event detection and analysis device described above.

[0216] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0217] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting and analyzing video stream events, characterized in that, include: Real-time detection of events in the video stream to determine the duration of the events in the video stream; Based on the duration of the event, determine whether the event is a target event, where the target event is an event that requires semantic analysis; If the event is a target event, then obtain the event trigger time information of the target event; Based on the event triggering time information, multiple video frames corresponding to the target event are determined, and the target region of the target event in each video frame is determined according to the spatial distribution characteristics of the target event in the multiple video frames. Semantic analysis is performed on the target events within the target area.

2. The method according to claim 1, characterized in that, The step of determining whether an event is a target event based on the duration of the event includes: Obtain the video scene type corresponding to the video stream; According to the video scene type, the event duration threshold associated with the video scene type is retrieved from the preset threshold library. The preset threshold library stores the mapping relationship between each video scene type and its corresponding event duration threshold. If the duration of the event is greater than or equal to the event duration threshold, then the event is determined to be the target event; If the duration of the event is less than the event duration threshold, then the event is determined to be a non-target event.

3. The method according to claim 1, characterized in that, The step of obtaining the event trigger time information of the target event includes: Obtain the start time of the target event in the video stream and determine the start frame time of the event; Real-time detection of the duration of the target event in the video stream to determine the duration of the event; Based on the event start frame time and the event duration, the event triggering event information of the target event is generated.

4. The method according to claim 1, characterized in that, The step of determining multiple video frames corresponding to the target event based on the event trigger time information includes: Based on the event trigger time information, extract a target event video stream segment from the video stream that matches the event trigger time information; Frame parsing is performed on the target event video stream segment to extract multiple video frames corresponding to the target event.

5. The method according to claim 1, characterized in that, The step of determining the target region of the target event in each video frame based on the spatial distribution characteristics of the target event in the multiple video frames includes: Extract the pixel distribution features, contour features, and position features of the target event in each of the multiple video frames to determine the spatial distribution features of the target event; Cluster analysis and contour fitting are performed on the spatial distribution characteristics of the target event in each video frame to determine the pixel coverage range of the target event in each video frame; Based on the pixel coverage in all video frames, the target region of the target event in each video frame is determined.

6. The method according to claim 1, characterized in that, The semantic analysis of the target events within the target area includes: Extract the visual features of the target event within the target area, the visual features including event subject features, action features, and scene association features; The visual features are analyzed and semantically mapped to generate a structured semantic description corresponding to the target event. The structured semantic description includes the event subject, event action, event occurrence scene, and event association relationship.

7. The method according to claim 6, characterized in that, After performing semantic analysis on the target events within the target area, the method further includes: The structured semantic description of the target event is converted into vectorized semantic features, and the vectorized semantic features are associated with the event trigger time information of the target event and the target region in each video frame, and stored in the video event library; Upon receiving a natural language query request, the time range constraint in the natural language query request is parsed, and candidate events are selected from the video event library based on the time range constraint. Calculate the similarity between the vectorized features of the natural language query request and the vectorized semantic features of each candidate event, sort them from high to low according to the similarity and output the search results.

8. A video stream event detection and analysis device, characterized in that, include: The determination module is used to detect events in the video stream in real time and determine the duration of the events in the video stream. The determining module is further configured to determine whether the event is a target event based on the duration of the event, wherein the target event is an event that requires semantic analysis. The acquisition module is also used to acquire the event triggering time information of the target event if the event is a target event; The determining module is further configured to determine multiple video frames corresponding to the target event based on the event triggering time information, and determine the target region of the target event in each video frame according to the spatial distribution characteristics of the target event in the multiple video frames; The analysis module also performs semantic analysis on the target events within the target area.

9. A video stream event detection and analysis device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images