A video event detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a video event detection method and device, electronic equipment and storage medium, which are applied to the field of video monitoring, and the method comprises: acquiring original perception data collected by a visual sensor, an audio sensor, an object state tracking module and an Internet of Things sensor, and generating a structured behavior change event according to the original perception data, wherein the structured behavior change event at least includes a personnel behavior detection result, an object state detection result, an audio anomaly detection result and an Internet of Things state detection result; generating an event importance score of the structured behavior change event for a priority dimension, a time dimension, a scene dimension, a personnel dimension and a location dimension; and controlling an image acquisition device to acquire a complete event video of the structured behavior change event based on the event importance score control chart. The embodiments of the present application can reduce the number of wireless videos, improve the effectiveness of video monitoring, enhance the monitoring quality, and improve the storage effect.

Description

Technical Field

[0001] This invention relates to the field of video surveillance technology, and in particular to a video event detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Home surveillance is a lightweight security sensing system designed for home scenarios. It uses home cameras as the core hardware carrier and can be paired with various sensing devices such as human infrared sensors, microphones, and Internet of Things (IoT) posture sensors. Relying on local storage, home gateways, and cloud platforms, it can realize functions such as real-time viewing of the home environment, video recording, and abnormal event alerts. It is mainly used in civilian scenarios such as home care, property security, elderly and infant monitoring, and pet observation. From a technical perspective, existing home surveillance systems on the market can be divided into three mainstream solutions: The first type is the 24-hour continuous recording + simple motion trigger mode commonly used by traditional devices such as Hikvision, Ring, and Nest Cam. 24-hour recording generates a large amount of redundant data and consumes high storage resources. Furthermore, basic motion triggering based on pixel changes and infrared sensing is difficult to distinguish between effective risk events such as elderly people falling, strangers entering, and babies crying, and irrelevant interference such as curtains swaying, pets moving, and changes in light and shadow, resulting in significant false triggering and missed triggering problems. The second type is a multimodal fusion abnormal behavior recognition solution that simultaneously collects video and audio monitoring data, extracts time-series event feature points and completes quality scoring, selects high-scoring feature points as time anchors, and adaptively matches the fusion time window of another data stream according to the anchor data type. The fusion of dual-modal features obtains a preliminary judgment result. If the confidence is insufficient, cross-modal causal logic verification is used to output an accurate abnormal identification conclusion. The third type is a technical route that combines low-power dual-mode recording with event confidence assessment. Existing low-power products can only simply switch between monitoring and recording power levels, and the triggering logic does not combine behavior recognition. Therefore, existing home surveillance systems have simple triggering logic, making it difficult to distinguish between valid events and irrelevant interference, resulting in a large number of invalid recordings and failing to retain footage preceding the event. Mainstream multimodal solutions only fuse raw audio and video, without fusing based on behavioral features; the recording quality is fixed, they do not differentiate between the importance of events, lack priority determination when multiple events occur concurrently, and their storage and evidence retrieval effects are poor. Summary of the Invention

[0003] This invention provides a video event detection method, device, electronic device, and storage medium to solve the problems of simple triggering logic in home surveillance, resulting in too many invalid recordings, inability to integrate surveillance audio and video data based on behavioral characteristics, and thus fixed surveillance image quality and poor storage performance.

[0004] According to one aspect of the present invention, a video event detection method is provided, wherein the method includes: The system acquires raw perception data collected by a visual sensor, an audio sensor, an object state tracking module, and an IoT sensor, and generates structured behavior change events based on the raw perception data. The structured behavior change events include at least human behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results. The structured behavioral change events are assigned an event importance score based on priority, time, scenario, personnel, and location dimensions. Based on the event importance score, the image acquisition device acquires the complete event video of the structured behavioral change event.

[0005] According to another aspect of the present invention, a video event detection apparatus is provided, wherein the apparatus comprises: The data acquisition module is used to acquire raw perception data collected by the visual sensor, audio sensor, object state tracking module and IoT sensor, and generate structured behavior change events based on the raw perception data. The structured behavior change events include at least human behavior detection results, object state detection results, audio anomaly detection results and IoT state detection results. The scoring acquisition module is used to generate event importance scores for the structured behavioral change events based on priority, time, scenario, personnel, and location dimensions. A complete video acquisition module is used to control the image acquisition device to acquire complete event videos of the structured behavioral change events based on the event importance score.

[0006] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video event detection method according to any embodiment of the present invention.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video event detection method according to any embodiment of the present invention.

[0008] The technical solution of this invention collects raw perception data through multiple types of sensors, such as visual sensors, audio sensors, object state tracking modules, and IoT sensors, and generates structured behavior change events including personnel behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results. The structured behavior change events are evaluated in multiple dimensions according to priority, time, scene, personnel, and location to obtain an event importance score. Based on the event importance score, the image acquisition device is controlled to obtain a complete event video of the structured behavior change event. This invention, through the fusion of multimodal sensory data, generates unified and standardized event information covering multiple dimensions of people, objects, sounds, and devices. This improves the comprehensiveness of event information dimensions and avoids identification defects in subsequent processes caused by single event information. By comprehensively calculating the event importance score based on multiple dimensions such as priority, time, scene, personnel, and location, the importance of events can be automatically distinguished, solving the problem of no recording priority for multiple concurrent events. The image acquisition and recording strategy is dynamically adjusted according to the event importance score, allowing for the retention of complete video before and after the event as needed, supplementing the pre-event context, while achieving differentiated recording, saving storage space, and improving the completeness of evidence collection and storage utilization.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a video event detection method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of another video event detection method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of another video event detection method provided in Embodiment 3 of the present invention; Figure 4 This is a flowchart illustrating a video event detection method according to Embodiment 4 of the present invention; Figure 5 This is an example diagram of a video saving process provided in Embodiment 4 of the present invention; Figure 6This is a schematic diagram of the structure of a video event detection system according to Embodiment 4 of the present invention; Figure 7 This is a schematic diagram of the structure of a video event detection device according to Embodiment 5 of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device that implements the video event detection method of this invention. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] Example 1 Figure 1 This is a flowchart of a video event detection method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where home surveillance video equipment triggers control. The method can be executed by a video event detection device, which can be implemented in hardware and / or software. The video event detection device can be configured in the home surveillance equipment or the server of the home surveillance equipment. Figure 1 As shown, the method includes: Step 110: Acquire raw perception data collected by the visual sensor, audio sensor, object state tracking module and IoT sensor, and generate structured behavior change events based on the raw perception data. The structured behavior change events include at least the human behavior detection results, object state detection results, audio anomaly detection results and IoT state detection results.

[0015] Among them, visual sensors can be devices deployed in a home environment for video capture, including PTZ cameras, low-power human body sensing cameras, smart door lock cameras, etc.; audio sensors can be devices for collecting sound within the home, including microphones, noise detection sensors, door and window acoustic sensors, Bluetooth built-in microphones, etc.; object state tracking modules can be devices for tracking and detecting the state of various objects within the home, which can be implemented using hardware such as image recognition, infrared ranging sensors, or laser ranging sensors; and IoT sensors can be IoT devices that sense the state of the environment, equipment, and human beings, including formaldehyde gas sensors, gas sensors, water immersion sensors, smart sockets, electricity meters, water meters, fall sensors, mattress sensors, etc.

[0016] Specifically, video sensors, audio sensors, object state tracking modules, and IoT sensors can be deployed within the home environment to collect various information about the user's home environment, thereby obtaining raw sensor data. This raw sensor data can be multimodal. The collected raw sensor data can be processed to identify and process information such as human behavior, object state, audio state, and the state of various devices during an event. The collection of human behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results corresponding to a specific event can be considered as a structured behavioral change event. This structured behavioral change event can reflect the members, objects, sounds, and IoT devices involved in an event within the home environment. For example, a structured behavioral change event could be a baby crying event, an elderly person falling event, or a stranger visiting event.

[0017] Step 120: Generate a structured event importance score for behavioral change events based on priority, time, scenario, personnel, and location dimensions.

[0018] The priority dimension can be a pre-set basic hazard level for the behavior type. Different priorities can be preset according to various common events within the home; for example, high-risk events such as falls, intruders, and gas leaks have higher priorities, while disruptive events such as pets moving or changes in light and shadow have lower priorities. The time dimension can categorize the importance of events according to the time period in which they occur. For example, the importance of the same event occurring during the day and night differs, as does the importance of the same event occurring during the same time period on weekdays and weekends. Different basic importance scores can be assigned to events at different times. The scenario dimension can be various preset scenarios where events occur, such as holiday mode, work mode, and solitary living mode. The importance of the same event can differ in different scenario modes. The location dimension can indicate the location where the event occurs; the importance of events occurring in different locations can differ. For example, locations can include the front door, the elderly person's bedroom, the nursery, the storage room, and the area where valuables are stored. The importance of an item being stolen at the front door may differ from that of an item being stolen from the area where valuables are stored. The personnel dimension can represent the people involved in the event; for example, the importance of an adult falling and an infant falling may differ.

[0019] In this embodiment of the invention, the importance of structured behavioral change events can be evaluated separately across priority, time, scene, personnel, and location dimensions. The importance scores from each dimension can be weighted and fused to obtain a unified event importance score. For example, for structured behavioral change events, quantitative features are extracted from five independent dimensions—priority, time, scene, personnel, and location—and single-dimensional sub-scores are generated. Fixed weight coefficients are pre-configured for each dimension, and the five sub-scores are weighted, summed, and normalized to output a unified comprehensive event importance score in the range of 0 to 1. This embodiment of the invention comprehensively assesses the importance of events through multi-dimensional quantitative evaluation, moving beyond simply judging event risk based on a single behavior. This allows the importance score to align with real-world family security needs and improves the reliability of video surveillance triggering.

[0020] Step 130: Based on the event importance score, control the image acquisition device to acquire complete event videos of structured behavioral change events.

[0021] The image acquisition device can be a device used to capture video of events occurring within the home. The image acquisition device can include cameras or video recorders, etc. The image acquisition device can be configured with multiple acquisition modes, such as high-quality continuous recording mode, low-quality continuous recording mode, high-quality circular buffer recording mode, low-quality wake-up buffer recording mode, etc. The image acquisition device can set different video acquisition modes according to different event importance scores.

[0022] In this embodiment of the invention, the control mode of the image acquisition device can be determined according to the importance score of the event. The image acquisition device can be controlled according to the control mode to acquire the time-frequency data corresponding to the structured behavior event according to the image quality and / or acquisition mode of the corresponding time importance score, thereby obtaining the complete event video of the structured behavior change event.

[0023] In this embodiment of the invention, raw perception data is collected by multiple types of sensors, including visual sensors, audio sensors, object state tracking modules, and IoT sensors. This generates structured behavior change events, including personnel behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results. The structured behavior change events are evaluated from multiple dimensions—priority, time, scene, personnel, and location—to obtain an event importance score. Based on this event importance score, the image acquisition device is controlled to obtain a complete event video of the structured behavior change event. This invention, through the fusion of multimodal sensory data, generates unified and standardized event information covering multiple dimensions of people, objects, sounds, and devices. This improves the comprehensiveness of event information dimensions and avoids identification defects in subsequent processes caused by single event information. By comprehensively calculating the event importance score based on multiple dimensions such as priority, time, scene, personnel, and location, the importance of events can be automatically distinguished, solving the problem of no recording priority for multiple concurrent events. The image acquisition and recording strategy is dynamically adjusted according to the event importance score, allowing for the retention of complete video before and after the event as needed, supplementing the pre-event context, while achieving differentiated recording, saving storage space, and improving the completeness of evidence collection and storage utilization.

[0024] Example 2 Figure 2 This is a flowchart of another video event detection method provided in Embodiment 2 of the present invention. This embodiment of the present invention describes the generation process of structured behavior change events. See [link to documentation]. Figure 2 The method provided in this embodiment of the invention specifically includes the following steps: Step 210: Acquire raw perception data collected by the vision sensor, audio sensor, object state tracking module and IoT sensor.

[0025] Step 220: Perform pose estimation on the video data in the original sensing data to obtain the personnel behavior detection result; perform audio anomaly detection on the audio data in the original sensing data to obtain the audio anomaly detection result; perform object state recognition on the object state tracking data in the original sensing data to obtain the object state detection result; and parse the IoT state data in the original sensing data to obtain the IoT state detection result.

[0026] In this embodiment of the invention, the raw sensing data may include video data, audio data, object state tracking data, IoT state data, etc. Human pose estimation can be performed on the video data to output human behavior detection results; abnormal acoustic recognition can be performed on the audio data to output audio anomaly detection results; state judgment can be performed on the object state tracking data to output object state detection results; and IoT sensor data can be parsed and decoded to output IoT device state detection results. These four types of detection results can be combined to form a structured behavior change event. That is, a structured behavior change event can be composed of human behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results, etc. The detection processes for body state detection, audio anomaly detection, and IoT state detection are not detailed here. For example, for human behavior detection, algorithms such as YOLO-Pose and MoveNet can be used to directly obtain key points of the human body in video data. By analyzing the temporal changes of these key points, behaviors such as falls, loitering, and intrusions can be identified. For audio anomaly detection, methods such as pure volume threshold detection, voiceprint template matching, and audio classification AI can be used to detect audio anomalies in the audio data, thereby obtaining audio anomaly detection results. For object state tracking data, single or multi-target tracking solutions can be used, such as YOLO series object detection + SORT / DeepSORT / ByteTrack tracking to continuously output the coordinate trajectory of objects or people as object state tracking data. For IoT state data, methods such as fixed threshold rule parsing, physical model standardization parsing, or temporal statistics can be used to obtain the operating status of different IoT devices, thereby obtaining IoT state detection results.

[0027] Step 230: Obtain the type priority lookup table and generate a basic priority score based on the event type and type priority lookup table of the structured behavior change event; obtain the event time lookup table and generate a time context score based on the event time and event time difference table of the structured behavior change event; obtain the event scenario lookup table and generate a personnel context score based on the event personnel status and event scenario lookup table of the structured behavior change event; obtain the event location lookup table and generate a location context score based on the event location, event location lookup table, and event location of the structured behavior change event.

[0028] In this embodiment of the invention, score lookup tables for various evaluation dimensions of structured behavior change events can be pre-configured. These lookup tables may include type priority lookup tables, event time lookup tables, event scene lookup tables, and event location lookup tables, etc. The aforementioned lookup tables can be configured with corresponding basic importance scores for different dimensions of the event. The type priority lookup table can be retrieved based on the event type corresponding to the structured behavior change event to obtain a basic priority score; the event time lookup table can be matched based on the event occurrence time to calculate a time dimension context score; the event scene lookup table can be retrieved in conjunction with the status of personnel associated with the event to output a personnel dimension context score; and the event location lookup table can be retrieved based on the event occurrence area to generate a location dimension context score, etc. It is understood that the method of generating importance scores for each dimension of structured behavior change events is not limited to querying configuration tables. It can also be achieved through expert systems or rule engines to generate event importance scores for corresponding priority, time, scene, personnel, and location dimensions. Of course, it can also be achieved through machine learning classification or regression scoring models.

[0029] Step 240: Obtain the sum of the basic priority score, time context score, personnel context score, and location context score, and use the normalized value of the sum of scores as the event importance score.

[0030] Specifically, the sum of the basic priority score, time context score, personnel context score, and location context score can be obtained. This sum of scores can be normalized so that the normalized value of the sum of scores is between 0 and 1, and this normalized value can be used as the event importance score. For example, the ratio of this sum of scores to the theoretical total score can be used as the above normalized value.

[0031] Step 250: Based on the event importance score, control the image acquisition device to acquire complete event videos of structured behavioral change events.

[0032] In this embodiment of the invention, raw perception data collected by a visual sensor, an audio sensor, an object state tracking module, and a physical network sensor are acquired. Human pose estimation is performed on video data to output human behavior detection results. Abnormal acoustic recognition is performed on audio data to output audio anomaly detection results. State judgment is performed on object state tracking data to output object state detection results. IoT sensor data is parsed and decoded to output IoT device state detection results. Four scoring tables (pre-stored for type priority, event time, event scenario, and event location) are used to determine the basic priority score, time context score, human context score, and location context score corresponding to the event type, event time, event scenario, and event location of the structured behavior change event. A normalized value of the sum of these scores is generated as the event importance score. Based on the event importance score, an image acquisition device is controlled to acquire the complete event video of the structured behavior change event. The embodiments of this invention use a pre-defined lookup table to complete the quantitative scoring of each dimension, unify the evaluation standards of each dimension, and have a simple overall calculation logic that can be adapted to real-time calculation of low-computing-power hardware at the edge. It scores independently in layers based on event type, time of occurrence, target personnel, and event location. All types of risk characteristics can be quantified and flexibly configured, providing standardized sub-item numerical basis for the integrated calculation of the comprehensive event importance score.

[0033] Example 3 Figure 3 This is a flowchart of another video event detection method provided in Embodiment 3 of the present invention. This embodiment of the present invention describes the process of acquiring complete event video. See [link to documentation]. Figure 3 The method provided in this embodiment of the invention specifically includes the following steps: Step 310: Acquire raw perception data collected by the visual sensor, audio sensor, object state tracking module and IoT sensor, and generate structured behavior change events based on the raw perception data. The structured behavior change events include at least the human behavior detection results, object state detection results, audio anomaly detection results and IoT state detection results.

[0034] Step 320: Generate a structured event importance score for behavioral change events based on priority, time, scenario, personnel, and location dimensions.

[0035] Step 330: Determine that the remaining system storage space for the complete event video meets the minimum threshold, and determine that the image acquisition device is not currently in recording mode.

[0036] The remaining system storage space can be the remaining storage space currently used to store complete event videos of various events.

[0037] In this embodiment of the invention, before controlling the image acquisition device to acquire the complete event video, the remaining storage space of the system storing the complete event video can be obtained. It is understood that the storage space for storing the complete event video can be located on a local or remote server. It can be determined whether the remaining storage space of the system is greater than a minimum threshold. If so, it is determined that the acquired complete event video can still be stored. If not, the triggering of the image acquisition device to acquire the complete event video is stopped. At the same time, it can also be determined whether the image acquisition device is currently in recording mode. If the image acquisition device is not in recording mode, the next step is entered to prevent the image acquisition device from being triggered repeatedly.

[0038] Step 340: Obtain the scoring trigger threshold of the image acquisition device, and obtain the comparison result between the event importance score and the scoring trigger threshold. The scoring trigger threshold is determined based on the base threshold and the dynamic adjustment amount corresponding to the current false trigger rate of the image acquisition device.

[0039] The scoring trigger threshold can be the minimum event importance score required to trigger the image acquisition device to capture event video. The threshold of the scoring trigger threshold can be adaptively adjusted. The current false trigger rate of the image acquisition device can be obtained, and the corresponding dynamic adjustment amount can be determined according to the value of the false trigger rate. The sum of the dynamic adjustment amount and the basic threshold can be used as the scoring trigger threshold. The scoring trigger threshold can be adjusted up or down by the false trigger rate to prevent the image acquisition device from being triggered erroneously.

[0040] In this embodiment of the invention, a scoring trigger threshold determined by a dynamic adjustment amount corresponding to a base threshold and the current false trigger rate of the image acquisition device can be obtained. The obtained event importance score and the scoring trigger threshold can be compared to obtain a comparison result. The comparison result may include an event importance score greater than or equal to the scoring trigger threshold, or an event importance score less than the scoring trigger threshold.

[0041] Step 350: Based on the comparison results, control the image acquisition device to acquire the complete event video.

[0042] In this embodiment of the invention, raw perception data is collected through multiple types of sensors, including visual sensors, audio sensors, object state tracking modules, and IoT sensors. This generates structured behavioral change events, including personnel behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results. The structured behavioral change events are evaluated from multiple dimensions—priority, time, scene, personnel, and location—to obtain an event importance score. The remaining system storage space for the complete event video and the current recording status of the image acquisition device are determined. A scoring trigger threshold for the image acquisition device is obtained, and a comparison result between the event importance score and the scoring trigger threshold is acquired. Based on the comparison result between the event importance score and the scoring trigger threshold, the image acquisition device is controlled to acquire the complete event video. This invention, through the fusion of multimodal sensory data, generates unified and standardized event information covering multiple dimensions of people, objects, sounds, and devices. This improves the comprehensiveness of event information dimensions and avoids identification defects in subsequent processes caused by single event information. By comprehensively calculating the event importance score based on multiple dimensions such as priority, time, scene, personnel, and location, the importance of events can be automatically distinguished, solving the problem of no recording priority for multiple concurrent events. The image acquisition and recording strategy is dynamically adjusted according to the event importance score, allowing for the retention of complete video before and after the event as needed, supplementing the pre-event context, while achieving differentiated recording, saving storage space, and improving the completeness of evidence collection and storage utilization.

[0043] In some embodiments of the invention, controlling the image acquisition device to acquire a complete event video based on the comparison result includes: If the comparison result indicates that the event importance score is greater than the score trigger threshold, then the recording mode corresponding to the event importance score is obtained. The recording mode includes at least one of circular buffer recording and monitoring mode recording. The image acquisition device is controlled to acquire real-time recording segments of the complete time video according to the recording mode until the structured behavior change event meets the termination condition. The pre-recorded segments of the structured behavior change event are extracted according to the trigger time of the image acquisition device. The pre-recorded segments and real-time recording segments are merged into a complete event video of the structured behavior change event. The termination condition includes at least one of the following: the person's static posture time is greater than a first time threshold, the event's silence duration is greater than a second time threshold, and the IoT controlled device is in a specified state.

[0044] Among them, the monitoring mode recording can be a lightweight acquisition mode in which the device is in normal operation, and a recording mode that continuously acquires images with low frame rate, low bit rate and low resolution. The circular buffer recording can be a recording mode that uses a circular buffer to continuously cache event video, thereby achieving high-quality recording.

[0045] In this embodiment of the invention, the obtained event importance score can be compared with a pre-set score trigger threshold. If the comparison result shows that the event importance score is higher than the score trigger threshold, the recording mode corresponding to the score range of the event importance score is retrieved. Specifically, at least one combination of monitoring mode recording and circular buffer recording can be used to perform image acquisition. Based on the selected recording mode, the image acquisition device is driven to continuously acquire real-time recording segments during the event occurrence process, without interruption, until the current structured behavior change event meets the preset termination condition. The system uses the trigger start time of the image acquisition device as the dividing point, reads and extracts the cached pre-recorded segments before the event occurrence from the circular buffer; then, the pre-recorded segments carrying the preceding context are spliced ​​and fused with the real-time recording segments acquired during the event occurrence to generate a complete event video containing complete temporal information. Specifically, the termination condition for determining the termination of the event is set with multiple judgment criteria, and the recording can be terminated if at least one of them is met: the duration of the person's posture remaining still exceeds a first time threshold, the duration of silence without detected audio abnormalities exceeds a second time threshold, and various IoT controlled sensors return to a normal specified steady state.

[0046] Based on the above embodiments of the invention, the method further includes: generating event metadata based on the complete event video, structured behavioral change events, and event importance scores, wherein the event metadata includes event identifier, event type, start and end time, image acquisition device identifier, location identifier, score value, score factor contribution, recording duration, associated personnel, associated objects, and privacy level; establishing the association between the complete event video and the event metadata, and persistently storing the complete event video and the event metadata.

[0047] In this embodiment of the invention, event attribute information such as event identifier, event type, start and end time, image acquisition device identifier, location identifier, score value, score factor contribution, recording duration, associated personnel, associated objects, and privacy level can be extracted from the acquired complete event video, structured behavioral change events, and event importance scores as event metadata. An association can be established between the event metadata and the corresponding complete event video. For example, this association can be a metadata index for adding event metadata to the complete event video, or the same identifier information can be established for both the complete event video and the event metadata. The constructed event metadata and the complete event video can be persistently stored. It is understood that the persistent storage areas for the event metadata and the complete event video can be the same or different. For example, the event metadata can be written to a database, while the complete event video can be written to a specified video file path.

[0048] Example 4 Figure 4This is a flowchart illustrating a video event detection method according to Embodiment 4 of the present invention. Existing home monitoring triggering schemes in this embodiment of the present invention have the following problems: Firstly, devices such as Hikvision and Ring use 24-hour continuous recording or simple motion detection, resulting in redundant videos, high storage costs, and reliance solely on pixels and infrared to determine motion, failing to distinguish between high-risk events and environmental interference, leading to serious false triggers and missed triggers. Secondly, the second type only integrates audio and video dual-dimensional data to identify anomalies, lacking support from object and IoT sensor information, resulting in limited identification coverage. Thirdly, although the third type has dual recording power consumption levels, mode switching relies solely on simple trigger signals, without multi-dimensional comprehensive quantitative scoring of events, and lacks intelligent hierarchical storage. For the above problems, see [link to relevant documentation]. Figure 4 This invention provides a video event detection method, which includes the following steps: Step 1: Multimodal Behavior Perception and Structured Event Generation Based on a multimodal behavior perception module, it simultaneously accesses four types of raw perception data from visual sensors, audio sensors, object state tracking modules, and IoT sensors in a home setting, covering full-dimensional perception signals of people, audio, objects, and home appliances. It can perform specialized analysis and processing on each type of raw data: outputting personnel behavior detection results through video pose estimation, outputting abnormal noise detection results through audio anomaly analysis, outputting object movement detection results through object trajectory and state recognition, and outputting home appliance state detection results through IoT data decoding and analysis. By integrating these four types of detection results, eliminating invalid and interfering data, it generates standardized and structured home behavior change events, providing complete and standardized basic event data for subsequent risk scoring and recording decisions.

[0049] Step 2: Collection of multi-dimensional scoring factors and scene context The scoring calculation module can receive structured behavioral change events output by the sensing module in real time, while simultaneously synchronizing the family scene context of these structured behavioral change events. The specific content collected in this family scene context includes: the time information of the event, the spatial location information of the event, the identities and activity status of family members present, and the current operating mode of the system. It also reads auxiliary parameters such as the remaining storage space of the system and the working status of the image acquisition device to subsequently determine whether the underlying system allows the acquisition of the complete event video.

[0050] Step 3: Calculation of Five-Dimensional Weighted Normalized Comprehensive Score The scoring calculation module retrieves four preset scoring tables based on priority type, event time, event scenario, and event location, respectively, to calculate the basic priority score, time context score, personnel context score, and location context score. Based on preset dimension weighting coefficients, the four sub-scores are weighted and summed to obtain the initial comprehensive event score. Finally, a maximum-minimum normalization algorithm maps the initial total score to a standard numerical range of 0.0–1.0, outputting a standardized comprehensive event importance score, enabling risk quantification and benchmarking for different types and scenarios of events.

[0051] Step 4: Decision-making triggered by multiple conditions The trigger decision module executes a multi-dimensional linkage verification mechanism to comprehensively determine the eligibility of the current event for recording. The verification conditions include four core indicators: whether the event importance score is higher than a preset trigger threshold, whether the system's remaining storage space meets the recording storage requirements, whether the device is currently in an idle, non-recording state, and whether the trigger interval between this event and the previous valid recording event meets the anti-duplicate triggering rules. If any condition is not met, the event is deemed an invalid trigger, and the recording process is abandoned. If all four conditions pass the verification, it is determined to be a valid risk event, and the subsequent video acquisition and recording process begins. It is understandable that if the current scenario involves multiple concurrent events, concurrent scheduling can be performed based on the obtained event importance scores, and video feeds for events can be switched or merged based on the differences between the importance scores of each event.

[0052] Step 5: Precise extraction of pre-recorded segments from the circular buffer. The system has a built-in circular buffer that continuously caches real-time audio and video footage, always retaining historical video data for a fixed duration. Upon triggering recording, the system immediately initiates a pre-recording extraction mechanism. Using the current event trigger moment as the time node, it employs a precise "write pointer forward N frames" extraction algorithm to backtrack and extract a pre-cached video segment of a preset duration (typically 30 seconds) as the event pre-recording segment. This completely preserves the preceding behavioral footage before the event occurs, overcoming the technical deficiency of traditional trigger-based recording methods that lack prior context.

[0053] Step 6: Dynamically switch recording modes and stitch together the complete video. Based on the current event importance score range, the system adaptively matches the corresponding recording mode, dynamically switching between low-power monitoring mode and high-definition recording mode, and initiates a high-quality real-time recording process from the image acquisition device, continuously acquiring real-time video clips of the entire event occurrence. Real-time recording continues until the monitored scene meets any of the preset termination conditions, including: the duration of personnel remaining still exceeds a first time threshold, the duration of on-site audio silence exceeds a second time threshold, or the IoT-controlled device returns to a normal steady-state state. After recording terminates, the pre-recorded cached clips and the entire real-time recorded clips are time-aligned and seamlessly stitched together to create a complete event video containing full-time information before, during, and after the event.

[0054] Step 7: Structured Metadata Generation and Dual-End Index Storage The event segment indexing module performs structured information parsing on the synthesized complete event video, automatically generating corresponding event metadata. This metadata includes key semantic information such as event type, risk score, occurrence time, location, participants, start and end duration, and any unusual content. The structured metadata is written into the home video semantic database, enabling events to be searchable, traceable, and categorized. Simultaneously, the storage path and file number of the complete event video are written to the local file system, completing the association and archiving of video files and semantic data, balancing video storage integrity with convenient future retrieval.

[0055] Further, see Figure 5 The system discloses a complete video storage chain, relying on a built-in circular buffer to cyclically cache real-time audio and video frames, continuously storing historical bitstream data of a fixed duration. When an event trigger signal is detected, the system synchronously starts the pre-recording extraction logic. Using the trigger timestamp as the segmentation node, it employs a write pointer-forwarding N-frame precise extraction algorithm to read forward 30 seconds of cached data of a preset duration to generate pre-recorded segments, completely restoring the behavior before the event occurred. It also initiates a high-quality real-time recording process from the image acquisition device, continuously acquiring real-time video segments of the entire event process. The real-time video segments and pre-recorded segments are merged into a complete event video. Structured semantic analysis is performed on the stitched complete event recording, automatically generating a standardized event metadata dataset. The metadata fields include key semantic information such as event category, risk score, event time, monitoring location, target personnel, recording start and end duration, and description of abnormal behavior. Structured metadata is persistently written to the family video semantic database, supporting multi-dimensional event retrieval, full-link tracing, and hierarchical classification management. At the same time, the storage path of the complete video file, the unique file number, and other index information are synchronously written to the local file system, establishing a binding and archiving mechanism between video entity files and semantic metadata. This greatly improves the convenience of later retrieval while ensuring the complete preservation of the original audio and video data.

[0056] The system provided in this invention can trigger a home monitoring system through video events showing behavioral changes. This system can be achieved through the following technical features: 1. Multi-dimensional behavioral importance scoring mechanism: The comprehensive importance score is obtained by combining the basic priority of the event (10 levels such as baby crying / falling / stranger / object moving, etc.) + time context weighting (morning peak / evening peak / late night) + scene context weighting (night mode / holiday mode / solo living mode) + personnel context weighting (baby alone / elderly living alone / child unattended) + location context weighting (dangerous area / key attention area / entrance area).

[0057] 2. Scoring-driven pre-recording buffer and dual-mode recording collaborative mechanism: The circular buffer continuously writes the most recent 30 seconds of video in monitoring mode; only when the importance score reaches the preset threshold is the historical video extracted from the buffer and the recording mode is switched to high-quality recording; realizing "behavior prediction-driven context capture".

[0058] 3. Multi-event concurrent scheduling and privacy control linkage mechanism: When processing multiple events concurrently, the switching / merging is determined based on the score difference; privacy control based on the family member role classification (parent / child / visitor) is executed synchronously.

[0059] See Figure 6The system provided in this embodiment of the invention can realize video event detection through the following 5 core modules: (1) Multimodal behavior perception module. It receives raw perception data from visual sensors, audio sensors, object state tracking modules and IoT sensor arrays, and outputs structured behavior change signals through four sub-modules: personnel behavior detection (abnormal posture / abnormal trajectory / timeout of stillness), object state detection (position change / posture change), audio abnormality detection (crying / shouting / glass breakage / abnormal sound), and IoT state detection (door open / light switch / human infrared). (2) Multidimensional score calculation module. It receives behavior change signals, calculates the comprehensive importance score of the event by weighting it according to the five dimensions of "basic priority + time context + scene context + personnel context + location context", and normalizes it to the range of 0.0-1.0; the scoring factors can be adjusted by the family administrator according to the scenario configuration. (3) Trigger decision module. Based on the output of the score calculation module, it executes the trigger decision logic (score ≥ preset threshold + sufficient storage + not in recording + trigger interval compliance), outputs the recording decision signal to the pre-recording buffer and dual-mode recording module, and synchronously executes the privacy control of family member role classification. (4) Pre-recording buffer and dual-mode recording module. The circular buffer continuously writes the most recent 30 seconds of video (overwriting the oldest frame) in monitoring mode; after the trigger decision module outputs a trigger signal, it extracts the pre-recorded content from the buffer, switches to recording mode (1920×1080 / 30fps) for high-quality recording, and splices the pre-recorded segments with the real-time recorded segments to output a complete event video. (5) Event segment indexing module. After each event recording is completed, it automatically generates structured event metadata (event identifier / event type / start and end time / camera identifier / location identifier / rating value / recording duration / list of associated people / list of associated objects / privacy level) and writes it to the home video semantic database.

[0060] The specific implementations of the core modules provided in this embodiment of the invention can be as follows: Module 1, the Multimodal Behavior Perception Module, receives raw data from multiple types of sensors, detects behavioral change signals, and outputs structured behavioral change events. Its inputs include: visual / audio / object state tracking (from B2) / IoT sensor data (door magnets / lights / human infrared). The Multimodal Behavior Perception Module can include four sub-modules that detect behaviors in parallel: 1.1 Personnel Behavior Detection (pose estimation outputs fall / timeout / abnormal activity, associating with family members through facial recognition); 1.2 Object State Detection (tracks object position / pose changes, outputting object movement events); 1.3 Audio Anomaly Detection (crying / glass breakage / abnormal sound recognition model); 1.4 IoT State Detection (door magnets / lights / human infrared state changes). The output of the Multimodal Behavior Perception Module can be structured behavioral change events (event type / event source / timestamp / confidence level / associated entity). The Multimodal Behavior Perception Module can submit the output structured events to Module 2.

[0061] Module 2: The multi-dimensional scoring calculation module integrates five scoring factors to calculate the overall importance score of an event, normalizing it to the range of 0.0–1.0. The inputs to the multi-dimensional scoring calculation module include structured events from the multimodal behavior perception module and the family scene context (time / location / family member status / system mode), etc.

[0062] Specifically, the multi-dimensional scoring calculation module can include the scoring calculation process for the following dimensions: Factor 2.1 is used to calculate the basic priority. Based on the event type, a table is looked up (baby crying 0.9 / falling 0.95 / stranger 0.85 / important object moving 0.6 / door opening 0.5 / ordinary activity 0.3 / pet 0.1 / irrelevant movement 0.05) to obtain S_base.

[0063] Factor 2.2 time context weighted calculation, S_time is obtained based on the current time period (morning peak +0.15 / evening peak +0.20 / late night +0.25 / other 0).

[0064] Factor 2.3 scene context weighted calculation, S_context is obtained according to system mode (night +0.10 / holiday +0.05 / living alone +0.10 / quiet -0.05).

[0065] Factor 2.4 is a context-weighted calculation of the person's situation. S_person is obtained based on the status of family members (infant living alone +0.25 / elderly living alone +0.20 / child unattended +0.20 / stranger vs family member +0.15).

[0066] The location context is weighted by factor 2.5, and S_location is obtained based on the event area (dangerous area kitchen / stairs +0.10 / key attention area +0.05 / entrance area +0.08 / normal 0).

[0067] Total score calculation: S_total = S_base + S_time + S_context + S_person + S_location; Normalization: S_normalized = min(S_total, 1.0).

[0068] The multi-dimensional scoring calculation module can output an overall event importance score (0.0-1.0 floating-point number) and a detailed breakdown of each factor's contribution (for debugging and interpretability), and submit the overall event importance score to the trigger decision module, which is module 3. The above scoring factors can be configured and adjusted by the family administrator.

[0069] Module 3: Trigger Decision Module, used to execute multi-condition trigger decisions based on the scoring calculation results and output a recording decision signal. The trigger decision module takes as input the scoring value from the multi-dimensional scoring calculation module, the system storage status, the current recording status, and the last trigger timestamp, and makes a decision based on the following conditions: Condition 3.1 Necessary condition one: The score S_normalized ≥ the preset threshold T (typical value 0.6); if not, the triggering is abandoned.

[0070] Condition 3.2 Necessary Condition 2: Sufficient system storage space (remaining space > minimum threshold); otherwise, abandon triggering.

[0071] Condition 3.3 Necessary condition three: Not currently recording (to avoid repeated triggering); if so, switch to multi-event concurrent scheduling (M3 processing).

[0072] Condition 3.4 Optional condition: Trigger interval > minimum interval (typical value 30 seconds, to prevent jitter); if not, abandon triggering.

[0073] Condition 3.5 Threshold Adaptation: If the false trigger rate is >0.30, the threshold is increased by 0.05; if the missed trigger rate is >0.10, the threshold is decreased by 0.05.

[0074] The trigger decision module can output recording decision signals (trigger / abandon / schedule to M3) and decision criteria (whether each condition is met). The trigger decision module can drive the pre-recording buffer and dual-mode recording module based on the output, wherein the above thresholds are adaptively triggered by feedback from the home administrator or system statistics.

[0075] Module 4: Pre-recording buffer and dual-mode recording module, maintaining continuous monitoring mode + triggered recording mode. When triggered, it extracts the pre-recording buffer content and splices it for output. It can accept the recording decision signal from the trigger decision module and the camera acquisition hardware as input.

[0076] The aforementioned pre-recording buffer and dual-mode recording module may include the following modules: Module 4.1 Circular Buffer: This circular buffer has a capacity of 30 seconds × 5fps = 150 frames (640×360), and can be written in a loop using "write pointer + overwrite the oldest frame". Module 4.2 Monitoring Mode (Continuous 640×360 / 5fps / ~200kbps write to buffer, real-time analysis only, no writing to video file); Module 4.3 Recording mode, when triggered, switches to 1920×1080 / 30fps / ~8Mbps to write video files; Module 4.4 Recording and Merging (When triggered, the pre-recorded segments are extracted according to the "write pointer advance N frames" algorithm, spliced ​​with the real-time recorded segments, and the complete event video is output); 4.5 Continuous Event Processing (Continuous recording until the end condition: personnel are still for more than 5 minutes / crying is silent for more than 30 seconds / abnormal sounds are quiet for more than 1 minute / door is closed, take the min (end condition, maximum duration of 5 minutes)).

[0077] The pre-recording buffer and dual-mode recording module can output: a complete event video file + metadata (recording duration / start and end time / trigger reason), and submit the output to module 5.

[0078] Module 5: Event Segment Index Module. After each event recording, it automatically generates structured event metadata and writes it to the home video semantic database. The complete event video file, structured events from the multimodal behavior perception module, and score details from the scoring calculation module can be input into the Event Segment Index Module. It can perform the following steps: Step 5.1 Metadata generation: The assembled structured fields of this metadata may include: event identifier (timestamp + camera identifier + serial number) / event type / start and end time / camera identifier / location identifier / score value / contribution of each factor / recording duration / list of associated people / list of associated objects / privacy level.

[0079] Step 5.2 Database writing: Metadata is written to the family video semantic database through the unified application interface, and the event video file path is written to the video_segments table of the relational database.

[0080] The event fragment index module can output: structured event metadata (already written to the video semantic database) + video file path (already written to the file system).

[0081] Let's take an infant-triggered recording scenario as an example: Scenario Description: A three-bedroom, two-living-room family home with six cameras deployed (living room, entryway, kitchen, master bedroom, children's room, and balcony). The permanent residents include two adults, a six-month-old infant, and an elderly person. The system is continuously running. At 3:00 PM, the infant is sleeping alone in the children's room.

[0082] The system triggers input: the audio module detects a baby crying at 15:30:00 (lasting 2 seconds, recognition confidence level 0.92), and the personnel behavior detection module simultaneously detects that the baby is in the nursery (associated with the baby's identity). The system's processing flow may include the following steps: Step 1: Multimodal Behavior Perception. The personnel behavior detection module outputs "Baby Crying Event" (Event Type = Baby Crying, Timestamp = 15:30:00, Associated Person = Baby, Associated Location = Children's Room, Confidence = 0.92).

[0083] Step 2: Scoring Factor Collection. The scoring calculation module receives the event and simultaneously collects the context: current time 15:30 (afternoon, S_time=0.0); system mode = non-night / non-holiday / non-single-person (S_context=0.0); family member status = baby alone (S_person=+0.25); event area = children's room (baby activity area, S_location=+0.05).

[0084] Step 3: Calculate the multi-dimensional score. S_base=0.9 (look up the "Baby Crying" table); S_time=0.0; S_context=0.0; S_person=0.25; S_location=0.05; S_total=0.9+0.0+0.0+0.25+0.05=1.20; S_normalized=min(1.20,1.0)=1.0.

[0085] Step 4 triggers a decision. If the following conditions are met: score 1.0 ≥ threshold 0.6, sufficient storage, not currently recording, and trigger interval > 30 seconds, then determine to trigger recording.

[0086] Step 5: Pre-recorded buffer extraction. Extract 30 seconds of monitoring mode video (including the baby falling asleep) from the circular buffer of the children's room camera using the algorithm of "write pointer forward 150 frames (30 seconds × 5fps)".

[0087] Step 6: Switching Recording Mode and Stitching. Switch the children's room camera to recording mode (1920). Start high-quality recording at 1080 / 30fps; splice the pre-recorded segment (15:29:30—15:30:00) with the live recording segment (15:30:00—continuous); end the continuous event judgment: baby crying silence > 30 seconds → end recording at 15:31:45; take min(end condition, 5 minutes maximum duration) = 1 minute 45 seconds.

[0088] Step 7: Metadata Index Writing. The event fragment index module generates structured metadata: Event Identifier = EVT-20260603-153000-Baby Crying-Children's Room-001; Event Type = Baby Crying; Start and End Time = 15:29:30—15:31:45; Rating = 1.0; Associated Person = [Baby]; Associated Location = Children's Room; Privacy Level = restricted (Baby details anonymized); written to the video semantic database; video file path written to the file system.

[0089] Example of system output: Full event video: 1 minute 45 seconds (30 seconds pre-recorded + 1 minute 15 seconds real-time recording), 1920×1080 / 30fps; Event metadata: already written to the video semantic database, event identifier EVT-20260603-153000-baby crying-children's room-001; Privacy controls: Parents have full access; children / guests can only access events marked "is_private=false" and not related to infants. Response time: from the detection of crying to the start of recording < 200 milliseconds (score calculation < 50ms + pre-recording extraction < 100ms + mode switching < 100ms).

[0090] Taking an elderly person falling as an example to trigger a recording scenario Scenario Description: A three-bedroom, two-living-room family home with six cameras deployed (living room, entryway, kitchen, master bedroom, children's room, and balcony). The permanent residents include two adults, a six-month-old infant, and an elderly person. The system is continuously running. At 3:00 PM, the infant is sleeping alone in the children's room.

[0091] System input: Continuous input: Six cameras continuously write to their respective circular buffers at 640×360 / 5fps; IoT sensors continuously report.

[0092] Trigger input: At 20:15:30, the personnel behavior detection module detected a "head height < knee" posture (fall detection) through posture estimation, and the person's identity was associated with an elderly person (confidence 0.95). The system's processing flow may include the following steps: Step 1: Multimodal Behavior Perception. The personnel behavior detection module outputs "Personnel Fall Event" (Event Type = Personnel Fall, Timestamp = 20:15:30, Associated Person = Elderly Person, Associated Location = Living Room, Confidence = 0.95).

[0093] Step 2: Scoring Factor Collection. The scoring calculation module receives the event and simultaneously collects the context: current time 20:15 (evening peak, S_time=+0.20); system mode = non-night / non-holiday / non-alone living (S_context=0.0); family member status = elderly (associated person is elderly, S_person=+0.20 related to elderly living alone); event area = living room (normal area, S_location=0.0).

[0094] Step 3: Calculate the multi-dimensional score. S_base=0.95 (look up the "Personnel Fall" category in the table); S_time=0.20; S_context=0.0; S_person=0.20; S_location=0.0; S_total=0.95+0.20+0.0+0.20+0.0=1.35; S_normalized=min(1.35,1.0)=1.0.

[0095] Step 4 triggers a decision. If the following conditions are met: score 1.0 ≥ threshold 0.6; sufficient storage; not currently recording; trigger interval > 30 seconds, then determine to trigger recording and push an alarm notification.

[0096] Step 5: Pre-recorded buffer extraction. Extract 30 seconds of monitoring mode video from the circular buffer of the living room camera, from 20:15:00 to 20:15:30 (including the elderly person's walking process, which can be observed whether there was any swaying before the fall) using the "write pointer advances 150 frames" algorithm.

[0097] Step 6: Recording Mode Switching and Stitching. Switch the living room camera to recording mode (1920×1080 / 30fps) and start high-quality recording; stitch the pre-recorded clips with the live recording clips; continuous event end determination: elderly person stands up OR 5 minutes maximum duration is reached → recording ends at 20:20:30 (the elderly person did not stand up in time, reaching the 5-minute limit); at the same time, push alarm notifications to family members' mobile phones.

[0098] Step 7: Write metadata index. The event fragment index module generates structured metadata: Event ID = EVT-20260603-201530-Person Fall-Living Room-001; Event Type = Person Fall; Start and End Time = 20:15:00—20:20:30; Rating = 1.0; Associated Person = [Elderly Person]; Associated Location = Living Room; Privacy Level = restricted (elderly person details are anonymized); Write to semantic database; Trigger alarm: Push to family members' mobile phones.

[0099] Example of system output: Full event video: 5 minutes 30 seconds (30 seconds pre-recorded + 5 minutes real-time recording), 1920×1080 / 30fps; Event metadata: already written to the semantic database, event identifier EVT-20260603-201530-person fall-living room-001; Alarm notification: Pushed to family members' mobile phones; Response time: <200 milliseconds from fall detection to start recording; <500 milliseconds from fall detection to alarm push.

[0100] Example 5 Figure 7 This is a schematic diagram of the structure of a video event detection device according to Embodiment 5 of the present invention, as shown below. Figure 7 As shown, the device includes: The data acquisition module 410 is used to acquire raw perception data collected by the vision sensor, audio sensor, object state tracking module and IoT sensor, and generate structured behavior change events based on the raw perception data. The structured behavior change events include at least the human behavior detection results, object state detection results, audio anomaly detection results and IoT state detection results.

[0101] The scoring module 420 is used to generate structured event importance scores for behavioral change events based on priority, time, scenario, personnel, and location dimensions.

[0102] The complete video acquisition module 430 is used to control the image acquisition device to acquire complete event videos of structured behavioral change events based on event importance scoring.

[0103] In this embodiment of the invention, a data acquisition module acquires raw perception data collected by various sensors, including visual sensors, audio sensors, object state tracking modules, and IoT sensors, and generates structured behavior change events that include personnel behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results. A scoring acquisition module evaluates the structured behavior change events from multiple dimensions, including priority, time, scene, personnel, and location, to obtain an event importance score. A complete video acquisition module controls the image acquisition device based on the event importance score, thereby acquiring a complete event video of the structured behavior change event. This invention, through the fusion of multimodal sensory data, generates unified and standardized event information covering multiple dimensions of people, objects, sounds, and devices. This improves the comprehensiveness of event information dimensions and avoids identification defects in subsequent processes caused by single event information. By comprehensively calculating the event importance score based on multiple dimensions such as priority, time, scene, personnel, and location, the importance of events can be automatically distinguished, solving the problem of no recording priority for multiple concurrent events. The image acquisition and recording strategy is dynamically adjusted according to the event importance score, allowing for the retention of complete video before and after the event as needed, supplementing the pre-event context, while achieving differentiated recording, saving storage space, and improving the completeness of evidence collection and storage utilization.

[0104] In some embodiments of the invention, the data acquisition module 410 includes an event recognition unit for at least one of the following: Pose estimation is performed on the video data within the raw perception data to obtain the human behavior detection results; Audio anomaly detection is performed on the audio data within the raw sensory data to obtain audio anomaly detection results; Object state recognition is performed on the object state tracking data within the raw sensing data to obtain object state detection results; The IoT state data within the raw sensing data is analyzed to obtain IoT state detection results.

[0105] In some embodiments of the invention, the scoring acquisition module 420 is specifically used for: acquiring a type priority lookup table, generating a basic priority score according to the event type and type priority lookup table of the structured behavior change event; acquiring an event time lookup table, generating a time context score according to the event time and event time difference table of the structured behavior change event; acquiring an event scenario lookup table, generating a personnel context score according to the event personnel status and event scenario lookup table of the structured behavior change event; acquiring an event location lookup table, generating a location context score according to the event location, event location lookup table, and event location of the structured behavior change event; acquiring the sum of the basic priority score, time context score, personnel context score, and location context score, and using the normalized value of the score sum as the event importance score.

[0106] In some embodiments of the invention, the complete video acquisition module 430 includes: The trigger threshold unit is used to obtain the scoring trigger threshold of the image acquisition device and to obtain the comparison result between the event importance score and the scoring trigger threshold. The scoring trigger threshold is determined based on the base threshold and the dynamic adjustment amount corresponding to the current false trigger rate of the image acquisition device.

[0107] The acquisition trigger unit is used to control the image acquisition device to acquire the complete event video based on the comparison result.

[0108] In some embodiments of the invention, the complete video acquisition module 430 further includes: The status determination unit is used to determine whether the remaining system storage space for the complete event video meets the minimum threshold, and to determine whether the image acquisition device is currently not in recording mode.

[0109] In some embodiments of the invention, the acquisition triggering unit is specifically used to: determine that the comparison result is that the event importance score is greater than the score trigger threshold, then acquire the recording mode corresponding to the event importance score, wherein the recording mode includes at least one of circular buffer recording and monitoring mode recording; control the image acquisition device to acquire real-time recording segments of the complete time video according to the recording mode until the structured behavior change event meets the termination condition; extract the pre-recorded segments of the structured behavior change event according to the trigger time of the image acquisition device; merge the pre-recorded segments and the real-time recording segments into a complete event video of the structured behavior change event; wherein the termination condition includes at least one of the following: the person's posture still time is greater than a first time threshold, the event silence duration is greater than a second time threshold, and the IoT controlled device is in a specified state.

[0110] In some embodiments of the invention, the method further includes: a persistence module, used to generate event metadata based on the complete event video, structured behavioral change events, and event importance scores, wherein the event metadata includes event identifier, event type, start and end time, image acquisition device identifier, location identifier, score value, score factor contribution, recording duration, associated personnel, associated objects, and privacy level; establish the association between the complete event video and the event metadata, and persistently store the complete event video and the event metadata.

[0111] The video event detection device provided in the embodiments of the present invention can execute the video event detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0112] Example 6 Figure 8This is a schematic diagram of the structure of an electronic device implementing the video event detection method of this invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0113] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video event detection methods.

[0116] In some embodiments, the video event detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video event detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video event detection method by any other suitable means (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0122] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0123] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0124] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video event detection method, characterized in that, The method includes: The system acquires raw perception data collected by a visual sensor, an audio sensor, an object state tracking module, and an IoT sensor, and generates structured behavior change events based on the raw perception data. The structured behavior change events include at least human behavior detection results, object state detection results, audio anomaly detection results, and IoT state detection results. The structured behavioral change events are assigned an event importance score based on priority, time, scenario, personnel, and location dimensions. Based on the event importance score, the image acquisition device acquires the complete event video of the structured behavior change event.

2. The method according to claim 1, characterized in that, The generation of structured behavior change events based on the original perceived data includes: Pose estimation is performed on the video data within the original perception data to obtain the personnel behavior detection results; Audio anomaly detection is performed on the audio data within the original perceptual data to obtain the audio anomaly detection result; Object state recognition is performed on the object state tracking data within the original sensing data to obtain the object state detection result; The IoT state data within the original sensing data is analyzed to obtain the IoT state detection result.

3. The method according to claim 1, characterized in that, The generation of event importance scores for the structured behavioral change events based on priority, time, scenario, personnel, and location dimensions includes: Obtain the type priority lookup table, and generate a basic priority score according to the event type of the structured behavior change event and the type priority lookup table; Obtain the event time lookup table, and generate a time context score based on the event time of the structured behavior change event and the event time difference table; Obtain the event scenario scoring table, and generate a personnel context score based on the event personnel status of the structured behavior change event and the event scenario scoring table; Obtain the event location lookup table, and generate a location context score based on the event location of the structured behavior change event, the event location lookup table, and the event location; Obtain the sum of the basic priority score, the time context score, the personnel context score, and the location context score, and use the normalized value of the sum of the scores as the event importance score.

4. The method according to claim 1, characterized in that, The image acquisition device, based on the event importance score, acquires a complete event video of the structured behavioral change event, including: The scoring trigger threshold of the image acquisition device is obtained, and the comparison result between the event importance score and the scoring trigger threshold is obtained, wherein the scoring trigger threshold is determined based on a base threshold and a dynamic adjustment amount corresponding to the current false trigger rate of the image acquisition device; Based on the comparison result, the image acquisition device is controlled to acquire the complete event video.

5. The method according to claim 4, characterized in that, Before controlling the image acquisition device to acquire the complete event video based on the comparison result, the method further includes: The system storage space remaining for the complete event video meets a minimum threshold, and the image acquisition device is currently not in recording mode.

6. The method according to claim 4 or 5, characterized in that, The step of controlling the image acquisition device to acquire the complete event video based on the comparison result includes: If the comparison result is determined to be that the event importance score is greater than the score trigger threshold, then the recording mode corresponding to the event importance score is obtained, wherein the recording mode includes at least one of circular buffer recording and monitoring mode recording; The image acquisition device is controlled to acquire real-time recording segments of the complete time video according to the recording mode until the structured behavior change event meets the termination condition. Extract the pre-recorded segments of the structured behavior change event according to the trigger time of the image acquisition device; The pre-recorded segments and the real-time recorded segments are merged into the complete event video of the structured behavior change event; The termination conditions include at least one of the following: the person's static posture time is greater than a first time threshold, the event's silence duration is greater than a second time threshold, and the IoT controlled device is in a specified state.

7. The method according to claim 1, characterized in that, Also includes: Event metadata is generated based on the complete event video, the structured behavior change event, and the event importance score. The event metadata includes event identifier, event type, start and end time, image acquisition device identifier, location identifier, score value, score factor contribution, recording duration, associated personnel, associated objects, and privacy level. Establish the association between the complete event video and the event metadata, and persistently store the complete event video and the event metadata.

8. A video event detection device, characterized in that, The device includes: The data acquisition module is used to acquire raw perception data collected by the visual sensor, audio sensor, object state tracking module and IoT sensor, and generate structured behavior change events based on the raw perception data. The structured behavior change events include at least human behavior detection results, object state detection results, audio anomaly detection results and IoT state detection results. The scoring acquisition module is used to generate event importance scores for the structured behavioral change events based on priority, time, scenario, personnel, and location dimensions. A complete video acquisition module is used to control the image acquisition device to acquire complete event videos of the structured behavioral change events based on the event importance score.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video event detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video event detection method as described in any one of claims 1-7.

Images