A Home Security AI Monitoring Method and System Based on Multimodal Algorithm Fusion

Abstract

This invention discloses a home security AI monitoring method and system based on multimodal algorithm fusion, specifically relating to the field of smart home monitoring technology. It involves: collecting raw video and audio data from a home monitoring scenario; using a convolutional neural network model for target recognition to generate bounding box location information of the monitored object; extracting behavioral features of the monitored object using a long short-term memory neural network; extracting features from the audio frame sequence and identifying the corresponding sound event type using an event classification model; synchronously correcting the target location information, behavioral features, and sound event type to generate a joint behavioral feature vector; identifying abnormal behavior based on the joint feature vector to determine whether the monitored object exhibits predefined risky behavior and outputting the behavior recognition result; generating a home security alarm command based on the behavior recognition result and pushing the alarm content to the user terminal, thereby improving the accuracy and real-time performance of the home monitoring system.

Description

Technical Field

[0001] This invention relates to the field of smart home monitoring technology, and more specifically, to a home security AI monitoring method and system based on multimodal algorithm fusion. Background Technology

[0002] Home security monitoring technologies commonly employ multimodal data fusion methods combining video images and audio signals. However, existing multimodal fusion algorithms, when processing data from different modalities, suffer from inconsistencies in time scale between single-frame target detection and continuous-frame behavior analysis. This leads to a lag in the location update of monitored objects in fast-moving scenarios compared to behavioral feature analysis. This time-series mismatch, or cross-modal data asynchrony, can easily cause normal household activities to be incorrectly identified as risky or abnormal behavior, severely reducing the reliability of the monitoring system and the user experience. Summary of the Invention

[0003] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a home security AI monitoring method and system based on multimodal algorithm fusion to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A home security AI monitoring method based on multimodal algorithm fusion includes the following steps:

[0006] S1. Collect raw video image data and raw audio data from the home monitoring scene, and perform image frame extraction and temporal framing respectively to generate video image frame sequence and audio frame sequence.

[0007] S2. Based on a convolutional neural network, target recognition is performed on the video image frame sequence to generate bounding box location information of the monitored object in each frame image;

[0008] S3. Based on the bounding box position information of the monitored object in each frame of the image, extract the behavioral features of the monitored object through a long short-term memory neural network;

[0009] S4. Perform feature extraction and event recognition on the audio frame sequence, and combine the event classification model to identify the corresponding sound event type;

[0010] S5. Synchronously correct the bounding box position information, behavioral features and sound event types to generate a joint behavioral feature vector;

[0011] S6. Based on the joint behavioral feature vector, identify abnormal behavior, use the event classification model to determine whether the current monitored object has predefined risk behaviors, and output the behavior identification results;

[0012] S7. Generate corresponding home security alarm commands based on behavior recognition results and push them to the user interface via mobile terminal.

[0013] In a preferred embodiment, S1 specifically refers to:

[0014] Collect raw video and audio data within the home's internal monitoring area;

[0015] Image frames are extracted from the original video image data at set time intervals, and a video image frame sequence is constructed in chronological order.

[0016] The original audio data is divided into time-domain frames according to fixed frame length and frame shift parameters, and each audio frame is associated with a corresponding time index to generate an audio frame sequence.

[0017] In a preferred embodiment, S2 specifically refers to:

[0018] A convolutional neural network model is used to extract spatial feature information from each image frame in a video image frame sequence;

[0019] Based on spatial feature information in the image frame, detect whether there is a predefined monitoring object in the image frame;

[0020] When a predefined monitored object exists within an image frame, the boundary position parameters of the monitored object in the image frame are calculated, and the bounding box position information of the monitored object in each frame image is output.

[0021] In a preferred embodiment, S3 specifically refers to:

[0022] Based on the bounding box location information of the monitored object, a temporal location data sequence of the monitored object is constructed according to the temporal order of the image frames;

[0023] Long Short-Term Memory Neural Network is used to perform sequence modeling on the temporal location data sequence of the monitored object, and to extract the motion trajectory features and attitude change features of the monitored object in continuous image frames;

[0024] The motion trajectory and posture change features of the monitored object in consecutive image frames are used as the behavioral features of the monitored object.

[0025] In a preferred embodiment, S4 specifically refers to:

[0026] Extract the audio signal features of each audio frame in the audio frame sequence;

[0027] The audio signal features are input into the event classification model to classify and identify the sound events contained in the audio frame sequence, identify the category of the sound event, and output the corresponding sound event type.

[0028] In a preferred embodiment, the audio signal characteristics include Mel frequency cepstral coefficients, short-time energy, and short-time zero-crossing rate.

[0029] In a preferred embodiment, S5 specifically refers to:

[0030] A time alignment mechanism is constructed using the timestamps of video image frame sequences and audio frame sequences as reference benchmarks;

[0031] The bounding box position information and the temporal information of behavioral features in the image frame sequence are synchronized and aligned with the temporal information of sound event types in the audio frame sequence through a time alignment mechanism.

[0032] The bounding box position information after synchronization and alignment, the behavioral characteristics of the monitored object, and the sound event type are fused to generate a joint behavioral feature vector.

[0033] In a preferred embodiment, S6 specifically refers to:

[0034] Input the joint behavioral feature vector into the event classification model;

[0035] The event classification model classifies and identifies the behavior of monitored objects based on joint behavioral feature vectors, and identifies whether the current behavior of the monitored objects belongs to predefined risk behaviors.

[0036] Based on the classification and recognition results of the event classification model, the behavior recognition results of the monitored objects are generated.

[0037] In a preferred embodiment, S7 specifically refers to:

[0038] Based on the behavior recognition results of the monitored object, determine whether the current behavior of the monitored object is a risky behavior;

[0039] When the current behavior of the monitored object is considered risky, a home security alarm command corresponding to the current behavior of the monitored object is generated.

[0040] Home security alarm commands are pushed to the user's mobile device and displayed on the user's mobile device's user interface.

[0041] On the other hand, the present invention provides a home security AI monitoring system based on multimodal algorithm fusion, comprising:

[0042] By jointly acquiring and processing video images and audio signals in a timely manner, and combining convolutional neural networks and long short-term memory neural networks, collaborative perception of the spatial location and temporal behavior of monitored objects is achieved. Simultaneously, audio event recognition enhances the responsiveness to changes in environmental sound, and a synchronization correction mechanism eliminates temporal mismatch between image and audio modalities, improving the consistency of cross-modal data fusion. Abnormal behavior recognition accurately identifies risky behaviors and promptly pushes alarm information to user terminals, achieving intelligent early warning and response to home security incidents, demonstrating good real-time performance, accuracy, and practicality.

[0043] The technical effects and advantages of the home security AI monitoring method and system based on multimodal algorithm fusion proposed in this invention are as follows:

[0044] Data acquisition module: Acquires raw video image data and raw audio data in home monitoring scenarios, and performs image frame extraction and temporal framing to generate video image frame sequences and audio frame sequences.

[0045] Target recognition module: Based on convolutional neural network, target recognition is performed on video image frame sequences to generate bounding box location information of the monitored object in each frame image;

[0046] Behavior analysis module: Based on the bounding box position information of the monitored object in each frame of image, the behavioral features of the monitored object are extracted through a long short-term memory neural network;

[0047] Sound recognition module: performs feature extraction and event recognition on audio frame sequences, and combines an event classification model to identify the corresponding sound event type;

[0048] Feature fusion module: Synchronously corrects bounding box location information, behavioral features and sound event types to generate a joint behavioral feature vector;

[0049] Anomaly detection module: Based on joint behavioral feature vectors, it identifies abnormal behavior, uses an event classification model to determine whether the current monitored object has predefined risky behavior, and outputs the behavior detection results;

[0050] Alarm push module: Generates corresponding home security alarm commands based on behavior recognition results and pushes them to the user interface via mobile terminal. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of a home security AI monitoring method based on multimodal algorithm fusion according to the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of a home security AI monitoring system based on multimodal algorithm fusion according to the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1: Figure 1 This invention presents a home security AI monitoring method based on multimodal algorithm fusion, which includes the following steps:

[0055] S1. Collect raw video image data and raw audio data from the home monitoring scene, and perform image frame extraction and temporal framing respectively to generate video image frame sequence and audio frame sequence.

[0056] S2. Based on a convolutional neural network, target recognition is performed on the video image frame sequence to generate bounding box location information of the monitored object in each frame image;

[0057] S3. Based on the bounding box position information of the monitored object in each frame of the image, extract the behavioral features of the monitored object through a long short-term memory neural network;

[0058] S4. Perform feature extraction and event recognition on the audio frame sequence, and combine the event classification model to identify the corresponding sound event type;

[0059] S5. Synchronously correct the bounding box position information, behavioral features and sound event types to generate a joint behavioral feature vector;

[0060] S6. Based on the joint behavioral feature vector, identify abnormal behavior, use the event classification model to determine whether the current monitored object has predefined risk behaviors, and output the behavior identification results;

[0061] S7. Generate corresponding home security alarm commands based on behavior recognition results and push them to the user interface via mobile terminal.

[0062] S1. Collect raw video image data and raw audio data from the home monitoring scene, and perform image frame extraction and temporal framing to generate video image frame sequences and audio frame sequences, including:

[0063] Collect raw video and audio data within the home's internal monitoring area;

[0064] The monitored area within a home refers to the home spaces that require security monitoring, such as the living room, bedroom, and kitchen. Raw video image data is acquired by installing high-definition cameras within the monitored area to capture video footage in real time and save the captured video as digital image signals. Raw video image data includes pixel matrix information, color information, and timestamp information. The image signal is typically stored in an uncompressed digital encoding format to ensure the integrity of the video information. For example, installing a high-definition camera in the living room can achieve a shooting frequency of 25 frames per second and a video resolution of 1920 pixels × 1080 pixels, ensuring clear image quality and complete detail. Raw audio data is acquired by installing microphones within the same monitored area to sample and digitize sound signals from the home environment in real time. Raw audio data includes the amplitude, frequency, and sampling rate information of the audio signal. The audio acquisition device can be a directional or omnidirectional microphone to capture sound events within the area, such as conversations among family members, sounds of pets, the opening and closing of doors and windows, or the sound of breaking glass in an accident. The microphone device can sample at a frequency of 16,000 Hz, continuously sampling the sound waveform in digital form to ensure that the audio information truly reflects the sound environment inside the home.

[0065] Image frames are extracted from the original video image data at set time intervals, and a video image frame sequence is constructed in chronological order.

[0066] The set time interval refers to the fixed interval used when extracting frames from the original video image signal frame by frame. This ensures that the extracted image frame sequence has a stable and uniform time interval structure. When the video acquisition frequency of the camera device is 25 frames per second, the set image frame extraction interval is one frame extracted every 0.04 seconds, that is, the first frame, the second frame, the third frame are extracted continuously, and so on, to complete the frame extraction step by step. This ensures that each extracted image frame has independent and accurate timestamp information. Each extracted video image frame retains the pixel data, color information, and spatial structure of the original image data for target detection and recognition processing. For example, a camera device installed in a family living room captures video at 25 frames per second. Through time interval extraction, 25 independent image frames can be generated per second. This image frame sequence can be used for the identification, location, and behavior trajectory analysis of monitored targets.

[0067] The original audio data is divided into time-domain frames according to fixed frame length and frame shift parameters, and each audio frame is associated with a corresponding time index to generate an audio frame sequence.

[0068] A fixed frame length parameter represents the duration of the audio signal data contained in each audio frame, while a fixed frame shift parameter represents the fixed length of the time window shift between each audio frame. Frame length and frame shift parameters ensure the temporal continuity and overlap of the audio frame sequence. For example, the frame length can be set to 20 milliseconds (0.02 seconds), and the frame shift can be set to 10 milliseconds (0.01 seconds). This ensures that each audio frame in the audio frame sequence overlaps with its adjacent frames by 10 milliseconds, effectively preventing the interruption of continuous sound events and making the audio feature extraction process more accurate. Associating each audio frame with its corresponding time index means that each audio frame has a clear and unique timestamp, ensuring that the audio frame sequence maintains temporal synchronization with the video image frame sequence. For example, in home surveillance, when the sound of doors and windows opening and closing occurs, the audio frame sequence can be used to correspond to the time point of the sound event.

[0069] S2. Based on a convolutional neural network, target recognition is performed on the video image frame sequence to generate the bounding box location information of the monitored object in each frame image, including:

[0070] A convolutional neural network model is used to extract spatial feature information from each image frame in a video image frame sequence;

[0071] Each image frame in a video image frame sequence refers to an independent image frame data extracted and saved from the original video image data at set time intervals. Each image frame contains video content, pixel matrix information, color information, and the precise timestamp information corresponding to the image frame. Spatial feature information is the key visual feature in video image frames, including but not limited to the edge contours, shapes, color distributions, texture information, background information, and spatial positional relationships between visual objects in the image. Spatial feature information describes the visual appearance structure information of objects and scenes in video image frames. A convolutional neural network model is used to extract spatial feature information. A convolutional neural network model is a deep learning neural network model whose network structure consists of convolutional layers, pooling layers, and fully connected layers, and can extract and represent spatial feature information from the input video image frame data. For example, each independent image frame in a video image frame sequence can be used as input data for a convolutional neural network model. A convolutional layer performs convolution operations on the input image, sliding a convolutional kernel of a specific size across all pixel regions of the input image to obtain local visual feature information of the input image frame. Then, a pooling layer reduces the dimensionality and compresses the feature map output by the convolutional layer to obtain a spatial feature information representation. Finally, a fully connected layer integrates and outputs the spatial feature information representation to form a high-dimensional spatial feature information vector, representing the overall visual content feature information of the input video image frame. For instance, in practical home surveillance applications, the convolutional kernel size of the convolutional neural network model can be set to 3×3 pixels or 5×5 pixels. Convolutional layers and pooling layers are stacked multiple times, for example, using an alternating structure of 5 convolutional layers and 5 pooling layers, to capture detailed feature information of the visual content in each video image frame. The resulting video image frame spatial feature information vector can be used for monitoring object recognition.

[0072] Based on spatial feature information in the image frame, detect whether there is a predefined monitoring object in the image frame;

[0073] Predefined monitoring objects refer to the types of visual targets that need to be focused on and identified in a pre-determined home monitoring scenario, including but not limited to family members, visitors, strangers, pets, and other human or animal individuals. A convolutional neural network model scans and detects all visual regions in each image frame using spatial feature information to determine whether a visual target belonging to the predefined monitoring object exists in the video image frame. The monitoring object detection method involves performing visual target classification calculations based on spatial feature information. Specifically, the extracted spatial feature information is input into a visual target classification network pre-trained through supervised learning. This network records and stores standard feature patterns for various monitoring object categories and can compare the input spatial feature information with these standard feature patterns to determine whether a feature pattern matching the predefined monitoring object exists in the input video image frame. The structure of the visual target classification network includes several fully connected layers and a softmax classification layer. The softmax classification layer outputs the probability of the presence of a monitoring object category. For example, after a video image frame is input into the visual target classification network, the network outputs a 98% probability of a "family member" monitoring object and a 2% probability of no monitoring object, indicating the presence of a monitoring object in the image frame.

[0074] When a predefined monitored object exists within an image frame, calculate the boundary position parameters of the monitored object in the image frame and output the bounding box position information of the monitored object in each frame image.

[0075] The boundary position parameters of the monitored object are the bounding box data representing the object's location. The bounding box is the smallest rectangular frame surrounding the monitored object, indicating its spatial position within the current image frame. The boundary position parameters include the two-dimensional pixel coordinates of the top-left and bottom-right corners of the bounding box, representing the spatial range of the monitored object within the image frame. The calculation method for the boundary position parameters includes target localization and regression techniques based on convolutional neural networks. These techniques predict the position coordinates of the visual features of the monitored object within the spatial feature information of the image frame, thus obtaining the bounding box position information. For example, in a video image frame of a family living room monitoring scene, after detecting a family member as a monitored object, the calculated bounding box position information is: top-left corner coordinates (100, 150), bottom-right corner coordinates (400, 600), indicating that the family member occupies the spatial region from pixel position 100th row, 150th column to pixel position 400th row, 600th column in the current image frame. The output bounding box position information is marked on the video image frame, enabling behavioral feature extraction and abnormal behavior analysis to utilize the spatial location and activity area information of the monitored object.

[0076] S3. Based on the bounding box location information of the monitored object in each frame of the image, extract the behavioral features of the monitored object through a long short-term memory neural network, including:

[0077] Based on the bounding box location information of the monitored object, a temporal location data sequence of the monitored object is constructed according to the temporal order of the image frames;

[0078] The temporal order of image frames refers to the sequential structure formed by extracting each image frame from a video image frame sequence at fixed time intervals. Each image frame has a timestamp to indicate its actual location in the monitored scene at the time of capture. For example, the timestamp of frame 1 is 10.00 seconds, frame 2 is 10.04 seconds, frame 3 is 10.08 seconds, and so on, maintaining the same temporal order. Based on the bounding box position information of the monitored object output within each image frame, the bounding box position information of the monitored object's spatial location in consecutive image frames is arranged and organized into a continuous temporal position data sequence according to the temporal order of the video image frames. The temporal position data sequence is a collection of bounding box two-dimensional coordinate points with a temporal order. Each data point in the sequence is associated with a corresponding image frame and has a definite time identifier. For example, in a living room scene inside a home, a family member entering the monitoring area is continuously filmed. In the first frame (timestamp 10.00 seconds), the bounding box coordinates of the monitored object are top left (120, 180) and bottom right (480, 720); in the second frame (10.04 seconds), the bounding box coordinates are top left (130, 190) and bottom right (490, 730); in the third frame (10.08 seconds), the bounding box coordinates are top left (140, 210) and bottom right (500, 750), and so on. The spatial location data is arranged in the order of the timestamps to form a complete and continuous temporal location data sequence, which expresses the spatial movement trajectory information of the monitored object over time.

[0079] Long Short-Term Memory Neural Network is used to perform sequence modeling on the temporal location data sequence of the monitored object, and to extract the motion trajectory features and attitude change features of the monitored object in continuous image frames;

[0080] Long Short-Term Memory (LSTM) neural networks are deep learning neural network models based on time series modeling. The model structure includes multiple network units such as an input layer, multiple hidden layers, memory cells, and an output layer. The memory cells can store and transmit the temporal feature information of historical data through a gating mechanism, making it suitable for processing the positional information sequence data of monitored objects over continuous time. The bounding box position information of the monitored object in the temporal position data sequence is sequentially input into the input layer of the LSM. The network processes the input bounding box position information frame by frame through multiple layers of memory cell units. Specifically, it tracks the spatial position change trend of the monitored object in consecutive image frames and establishes a temporal series relationship model of position change. Within each time step, the memory cell units record and store the current position, historical position, and position change trend of the monitored object, and extract sequence features from the position change information. Motion trajectory features include spatial direction, velocity, acceleration, and trajectory curve shape of the monitored object's continuous positional changes. For example, by inputting bounding box position information from three consecutive frames—frame 1 (top left, 120, 180), frame 2 (top left, 130, 190), and frame 3 (top left, 140, 210)—the network can calculate and extract motion trajectory features such as the monitored object's movement direction from top left to bottom right and an average speed of approximately 14.1 pixels per frame. Posture change features represent the trend of the monitored object's bounding box position, shape, and proportions across consecutive image frames. For instance, if the monitored object's bounding box is 360 pixels wide and 540 pixels high in frame 1, 360 pixels wide and 545 pixels high in frame 2, and 360 pixels wide and 550 pixels high in frame 3, it indicates that the monitored object may be gradually changing from a sitting to a standing posture. These motion trajectory features and posture change features are extracted through sequence modeling using a long short-term memory neural network model, forming the key foundational data for analyzing the behavior of monitored objects in consecutive image frames.

[0081] The motion trajectory features and posture change features of the monitored object in consecutive image frames are used as the behavioral features of the monitored object.

[0082] The behavioral characteristics of a monitored object refer to key feature parameters that describe its behavioral state and movement trends in a home surveillance scenario, including movement trajectory characteristics and posture change characteristics. In home security monitoring analysis, these behavioral characteristics are used for abnormal behavior identification and classification, enabling the detection and early warning of security threats or abnormal activities in the home surveillance scenario. For example, the movement trajectory characteristics of a monitored object can indicate whether it is continuously pacing, running quickly, or moving slowly; posture change characteristics can indicate abnormal postures such as falling, squatting, lying down, or suddenly standing up.

[0083] S4. Perform feature extraction and event recognition on the audio frame sequence, and combine the event classification model to identify the corresponding sound event type, including:

[0084] Extract the audio signal features of each audio frame in the audio frame sequence;

[0085] Audio signal characteristics refer to a series of digital parameters or feature vectors that can be used in the field of sound analysis to characterize the properties and characteristics of sound signals. They can accurately express the time-domain or frequency-domain characteristics of sound signals, including Mel frequency cepstral coefficients, short-time energy, and short-time zero-crossing rate.

[0086] Mel frequency cepstral coefficients (MSCs) effectively simulate the nonlinear characteristics of human ear perception of sound frequencies and are widely used in sound recognition tasks. The extraction method for MSCs is as follows: First, a Fast Fourier Transform (FFT) is performed on the audio frame to convert the time-domain sound signal into a frequency-domain signal. The frequency-domain signal is then filtered using a Mel filter bank, which consists of multiple overlapping triangular filters. The center frequencies of these filters are determined based on the laws of human auditory perception. Finally, a logarithmic operation is performed on the sound energy after Mel filtering, followed by a Discrete Cosine Transform (DCT), to obtain the MSCs. Specifically, 13-dimensional or 26-dimensional MSC feature vectors are typically extracted.

[0087] Short-time energy (SFE) characteristics represent the signal energy of a sound signal within a short time window, reflecting the changing trend of the audio signal's intensity and effectively distinguishing between silence, background noise, and sudden sound events. Specifically, SFE is calculated by squaring the amplitude values ​​of each sampling point of the audio signal within a given audio frame and then summing the squares of the instantaneous amplitude values. For example, when a family member suddenly speaks in the monitored area or a pet barks, the SFE within the audio frame will significantly increase; in quiet environments, the SFE remains within a lower range. By calculating the SFE characteristics of each audio frame, information on sound intensity and its changing trends can be provided for sound event classification models.

[0088] The short-time zero-crossing rate (SCR) feature specifically represents the number of times a sound signal crosses the zero-value horizontal line within a short time window, used to describe the degree of frequency change in an audio signal. Specifically, the SCR is calculated as follows: within an audio frame, consecutive sampling points of the audio signal are evaluated. When the sign of two consecutive sampling points changes (from positive to negative or vice versa), it is considered a zero-crossing event. The total number of zero-crossing events within the audio frame is counted to obtain the SCR of the audio frame. For example, the sound produced when a door or window is suddenly closed is usually a high-frequency and abrupt sound, in which case the SCR will show a significant increase; while low-frequency background noise shows a lower SCR. Therefore, extracting the SCR feature can distinguish the frequency characteristics of different types of sound events, providing feature support for the accurate classification of sound events.

[0089] The audio signal features are input into the event classification model to classify and identify the sound events contained in the audio frame sequence, identify the category of the sound event, and output the corresponding sound event type.

[0090] The event classification model is a sound classification model trained through supervised learning. During the training phase, it is trained using labeled data, which consists of data samples containing typical sound events in home monitoring scenarios, such as human speech, pet barking, door and window opening and closing sounds, glass breaking sounds, and falling sounds. Through training, the model can master and memorize the typical feature patterns of various sound events in order to distinguish the event category of the input audio frame.

[0091] The Mel-frequency cepstral coefficients, short-time energy, and short-time zero-crossing rate features extracted from each audio frame are normalized and vectorized before being fed into the event classification model as input data. The event classification model performs classification on the feature vector of each input audio frame, comparing and matching the sound events contained in the current audio frame with standard sound event feature patterns stored within the model to determine which predefined sound category the current sound event belongs to. For example, when the Mel-frequency cepstral coefficients of an audio frame exhibit a high mid-to-low frequency energy distribution, high short-time energy, and a low short-time zero-crossing rate, the classification model outputs a result with a 95% probability of "human speech," thus identifying the sound event category corresponding to the audio frame as "human speech" and outputting the category information for this sound event.

[0092] Through the above audio signal feature extraction and event classification model analysis, the final classification and recognition results of all sound event types in the audio frame sequence are generated, including the category information of the sound event corresponding to each audio frame, providing sound event information support for the analysis of abnormal behavior in home security monitoring based on multimodal fusion.

[0093] S5. Synchronously correct the bounding box position information, behavioral features, and sound event types to generate a joint behavioral feature vector, including:

[0094] A time alignment mechanism is constructed using the timestamps of video image frame sequences and audio frame sequences as reference benchmarks;

[0095] Constructing a time alignment mechanism refers to using the timestamp information of video image frame sequences and audio frame sequences as a unified reference standard. By matching the correspondence between the video image frame sequences and audio frame sequences in the time dimension, it ensures that the video image data and audio data can correspond to the same time position in the monitoring scene. Specifically, a time alignment algorithm is used, such as a time alignment algorithm based on linear interpolation or a time alignment algorithm based on dynamic time warping. Taking the time alignment algorithm based on linear interpolation as an example, the alignment process includes: when the timestamp corresponding to a certain image frame in the video image frame sequence is 10.04 seconds, the corresponding audio data position in the audio frame sequence is found. Through linear interpolation, the audio feature vector corresponding to the 10.04-second timestamp is calculated and inserted into the audio frame sequence; when the timestamp corresponding to a certain audio frame data in the audio frame sequence is 10.02 seconds, the corresponding position in the video image frame sequence is found, and the time alignment relationship between the video image frame sequence and the audio frame sequence is achieved through interpolation. The above time alignment mechanism construction process ensures the time correspondence between video image frame sequences and audio frame sequences, providing a time synchronization basis for multimodal data fusion processing.

[0096] The bounding box position information and the temporal information of behavioral features in the image frame sequence are synchronized and aligned with the temporal information of sound event types in the audio frame sequence through a time alignment mechanism.

[0097] By aligning the bounding box location information and behavioral features in the video image frame sequence with the temporal position of the sound event type in the audio frame sequence, a synchronization relationship between video image data and audio data in the temporal dimension is established. For example, the time alignment mechanism determines that the bounding box position, motion trajectory features, and posture change features of the monitored object at the 10.04-second timestamp in the video image frame sequence are at the same temporal position as the sound event type "human speech" at the 10.04-second timestamp in the audio frame sequence, thus achieving synchronous alignment between video image data and sound event data.

[0098] The bounding box position information after synchronization and alignment, the behavioral characteristics of the monitored object, and the sound event type are fused to generate a joint behavioral feature vector.

[0099] Feature fusion refers to the data-level fusion and integration of multiple feature information generated from different modal data sources, such as bounding box position information and behavioral features in the visual modality, and sound event type information in the auditory modality, to form a joint behavioral feature vector. The joint behavioral feature vector expresses the multimodal feature fusion information of a home surveillance scene at a specific time location. Multimodal feature fusion methods are employed, such as feature vector concatenation fusion or feature weight fusion. Taking feature vector concatenation fusion as an example, the fusion process is as follows: the bounding box position information and behavioral features extracted from the video image modality are represented in vector form, such as bounding box position (120, 180, 480, 720), motion trajectory (direction 0.75, speed 14.1), and posture features (height-to-width ratio 1.5), forming a video image feature vector; the sound event type is converted into a one-hot encoded feature vector representation, for example, "human speech" is converted into a feature vector (1, 0, 0, 0, 0), forming a sound event feature vector. These two types of feature vectors are then concatenated and fused at the same timestamp. For example, the video image feature vector and the sound event feature vector are concatenated to form a joint behavioral feature vector (120,180,480,720,0.75,14.1,1.5,1,0,0,0,0), which expresses the fusion information of multimodal data at a specific monitoring scene and time location.

[0100] S6. Based on the joint behavioral feature vector, perform abnormal behavior identification, use an event classification model to determine whether the currently monitored object exhibits predefined risky behavior, and output the behavior identification results, including:

[0101] Input the joint behavioral feature vector into the event classification model;

[0102] The event classification model is a multi-class classification model pre-trained using supervised learning. Supervised learning means that during training, the model uses labeled training data (standard training samples with category labels) to learn and optimize, mastering and recording the pattern information corresponding to different category feature vectors, and classifying unknown data during the classification process. A multi-class model means that the model can simultaneously distinguish and identify multiple categories from the input joint behavioral feature vector; that is, the model is not limited to single-class or binary classification, but simultaneously outputs probabilities and identifies multiple possible categories. For example, the event classification model uses a large amount of standard sample data from home surveillance scenarios for supervised learning training. The standard sample data includes training samples with various labeled categories, such as normal family member activity behavior samples, pet activity behavior samples, stranger intrusion behavior samples, abnormal loitering behavior samples, and fall or strenuous activity behavior samples. Each sample data is expressed as a joint behavioral feature vector and labeled with a specific behavioral category. The event classification model learns typical joint feature patterns of different behavior categories through training. For example, falling behavior corresponds to high posture change feature values ​​and a sudden increase in short-term sound energy features; stranger intrusion behavior corresponds to the category of unfamiliar monitored objects, abnormal movement trajectory features, and sound event types. After training, the event classification model can effectively distinguish the above multiple behavior category patterns, realizing the ability to identify and classify various abnormal behaviors in home security monitoring scenarios.

[0103] The event classification model classifies and identifies the behavior of monitored objects based on joint behavioral feature vectors, and identifies whether the current behavior of the monitored objects belongs to predefined risk behaviors.

[0104] Predefined risky behaviors refer to a series of abnormal behavioral categories that are predefined in a home security monitoring scenario and pose a threat to home safety or may lead to dangerous consequences. These include, but are not limited to, falling behavior, intrusion by strangers, strenuous exercise or conflict behavior, abnormal pet behavior, and accidental damage to household items, which may threaten the safety of family members or property. The event classification model classifies and identifies each input joint behavioral feature vector. Specifically, it first extracts visual and auditory feature information from the joint behavioral feature vector and inputs it into the model's multi-classification output layer. The similarity or probability output value between the joint behavioral feature vector and the different category standard feature patterns stored internally by the model is calculated. Based on the probability output value, the event classification model determines the most likely behavioral category to which the monitored object's behavior represented by the current joint behavioral feature vector belongs. For example, when the joint behavioral feature vector clearly indicates that the bounding box position of the monitored object is rapidly decreasing in consecutive image frames (the height ratio is significantly reduced), the motion trajectory features show a clear trend of rapid downward movement, and the sound event type is identified as "falling sound" with a significant short-term increase in energy, the event classification model outputs a joint behavioral feature vector with a 98% probability of belonging to the "falling behavior" risk behavior. However, when the joint behavioral feature vector indicates that the monitored object's position and motion trajectory are normal, the sound event type is identified as normal family conversation with low sound intensity, the event classification model outputs a joint behavioral feature vector with an extremely low probability of belonging to the risk behavior category, thus classifying the current behavior as normal family activity.

[0105] Based on the classification and recognition results of the event classification model, the behavior recognition results of the monitored objects are generated.

[0106] The behavior recognition result of the monitored object is the output information of the event classification model after classifying and recognizing the joint behavior feature vector. It indicates which defined behavior category the monitored object's current behavior belongs to, i.e., whether it belongs to a risky behavior category. Specifically, when the probability that the event classification model outputs that the current joint behavior feature vector of the monitored object belongs to a risky behavior category exceeds a predefined threshold (e.g., the probability exceeds 95%), the current monitored object's behavior is determined to belong to a risky behavior category, and the behavior recognition result is output as risky behavior category information, such as "falling behavior" or "unknown person intrusion behavior," etc. When the probability that the current behavior belongs to a risky behavior category is lower than the defined threshold, the current monitored object's behavior is determined to belong to normal non-risk behavior, and the behavior recognition result is output as "normal activity behavior" or other non-risk behavior category. The behavior recognition result serves as an important basis for abnormal behavior response or security warning of the home security monitoring system. It can be used to promptly issue alarms, notify users, or initiate other corresponding emergency response procedures to ensure that the home security situation is handled and responded to in a timely manner.

[0107] S7. Generate corresponding home security alarm commands based on the behavior recognition results, and push them to the user interface via the mobile terminal, including:

[0108] Based on the behavior recognition results of the monitored object, determine whether the current behavior of the monitored object is a risky behavior;

[0109] When the behavior identification result of the monitored object is classified as "falling behavior" or "unauthorized intrusion," the current behavior of the monitored object is determined to be risky. When the behavior identification result of the monitored object is "normal activity behavior" or other non-risk behavior categories, the current behavior of the monitored object is determined to be non-risky. For example, in a home monitoring scenario, if the event classification model outputs the behavior identification result of the monitored object as "falling behavior," then the current behavior of the monitored object is immediately determined to be risky. If the event classification model outputs the behavior identification result of the monitored object as "normal household activity behavior," then the current behavior of the monitored object is determined to be non-risky, thus making a risky behavior determination.

[0110] When the current behavior of the monitored object is considered risky, a home security alarm command corresponding to the current behavior of the monitored object is generated.

[0111] When the current behavior of a monitored object is determined to be a risky behavior category, a corresponding risk alarm operation will be immediately executed, specifically by generating a home security alarm command. A home security alarm command is a specific data structure or information message used to inform the user of a security risk behavior that has occurred within the home's monitored area, enabling the user to promptly understand the situation and take appropriate preventative or emergency measures. Home security alarm commands include data such as alarm category information, alarm level information, alarm occurrence time information, alarm occurrence location information, and specific characteristics of the monitored object, allowing users to quickly understand the details of the alarm event. For example, when the monitored object's behavior is "falling," the generated home safety alarm command will indicate the alarm category as "person falling," the alarm level as "emergency alarm," the alarm occurrence time as "May 10, 2024, 10:04:00," and the alarm location as "family living room area." It will also provide the boundary coordinates of the monitored object, such as "top left (120, 180), bottom right (480, 720)," allowing users to quickly and accurately grasp the situation. When the monitored object's behavior is "unknown person intrusion," the generated home safety alarm command will indicate the alarm category as "stranger intrusion," the alarm level as "high danger alarm," and clearly display the alarm occurrence time and location, such as "May 10, 2024, 11:20:00, family bedroom area," so that users can quickly understand and assess the home safety risk. The generation process of these home safety alarm commands is based on the risk category of the monitored object's current behavior to ensure the accuracy and specificity of the alarm information.

[0112] Push home security alarm commands to the user's mobile terminal device and display the home security alarm commands on the user's mobile terminal device's user interface;

[0113] Once generated, home security alarm commands are immediately pushed out via wireless communication technology to the user's personal mobile device. These devices include smartphones, tablets, and other devices with wireless communication capabilities and user interface displays. The wireless communication technology utilizes mobile network communication methods, including but not limited to Wi-Fi or 4G / 5G cellular network communication technologies. Specifically, after the home security alarm command is generated, it is sent and pushed to the user's personal mobile device via the wireless communication network through the internal communication module of the home security AI monitoring system. The user's personal mobile terminal device has the accompanying application software for the home security AI monitoring system installed. The user's mobile terminal device receives home security alarm commands through this application software, and the content of the alarm commands is displayed in a timely manner through the application's user interface. For example, it may display "Alarm Type: Person Falls; Alarm Level: Emergency; Time: May 10, 2024, 10:04:00; Location: Living Room; Monitoring Object Location: (120, 180) - (480, 720)". Through the user interface of the mobile terminal device, the user can quickly see the details of the home security alarm command and the on-site situation, and can immediately take corresponding measures, such as quickly calling emergency rescue or checking the monitoring screen to determine the actual situation at home. Simultaneously, the display of home security alarm commands on the user's mobile terminal device can be achieved through pop-ups, sound prompts, or vibration alerts, ensuring that the user can promptly pay attention to and understand the alarm content, maximizing the guarantee that home security risks receive timely attention and handling.

[0114] Example 2: The difference between Example 2 and Example 1 is that this example introduces a home security AI monitoring system based on multimodal algorithm fusion.

[0115] Figure 2 A schematic diagram of a home security AI monitoring system based on multimodal algorithm fusion is provided. The home security AI monitoring system based on multimodal algorithm fusion includes:

[0116] Data acquisition module: Acquires raw video image data and raw audio data in home monitoring scenarios, and performs image frame extraction and temporal framing to generate video image frame sequences and audio frame sequences.

[0117] Target recognition module: Based on convolutional neural network, target recognition is performed on video image frame sequences to generate bounding box location information of the monitored object in each frame image;

[0118] Behavior analysis module: Based on the bounding box position information of the monitored object in each frame of image, the behavioral features of the monitored object are extracted through a long short-term memory neural network;

[0119] Sound recognition module: performs feature extraction and event recognition on audio frame sequences, and combines an event classification model to identify the corresponding sound event type;

[0120] Feature fusion module: Synchronously corrects bounding box location information, behavioral features and sound event types to generate a joint behavioral feature vector;

[0121] Anomaly detection module: Based on joint behavioral feature vectors, it identifies abnormal behavior, uses an event classification model to determine whether the current monitored object has predefined risky behavior, and outputs the behavior detection results;

[0122] Alarm push module: Generates corresponding home security alarm commands based on behavior recognition results and pushes them to the user interface via mobile terminal.

[0123] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0124] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0125] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0126] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0128] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0130] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0132] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A home security AI monitoring method based on multimodal algorithm fusion, characterized in that, Includes the following steps: S1. Collect raw video image data and raw audio data from the home monitoring scene, and perform image frame extraction and temporal framing respectively to generate video image frame sequence and audio frame sequence. S2. Based on a convolutional neural network, target recognition is performed on the video image frame sequence to generate the bounding box location information of the monitored object in each frame image, specifically: A convolutional neural network model is used to extract spatial feature information from each image frame in a video image frame sequence; Based on spatial feature information in the image frame, detect whether there is a predefined monitoring object in the image frame; When a predefined monitored object exists within an image frame, calculate the boundary position parameters of the monitored object in the image frame and output the bounding box position information of the monitored object in each frame image. S3. Based on the bounding box position information of the monitored object in each frame of the image, the behavioral features of the monitored object are extracted through a long short-term memory neural network, specifically: Based on the bounding box location information of the monitored object, a temporal location data sequence of the monitored object is constructed according to the temporal order of the image frames; Long Short-Term Memory Neural Network is used to perform sequence modeling on the temporal location data sequence of the monitored object, and to extract the motion trajectory features and attitude change features of the monitored object in continuous image frames; The motion trajectory features and posture change features of the monitored object in consecutive image frames are used as the behavioral features of the monitored object. S4. Perform feature extraction and event recognition on the audio frame sequence, and combine the event classification model to identify the corresponding sound event type; S5. Synchronously correct the bounding box location information, behavioral features, and sound event types to generate a joint behavioral feature vector, specifically: A time alignment mechanism is constructed using the timestamps of video image frame sequences and audio frame sequences as reference benchmarks; The bounding box position information and the temporal information of behavioral features in the image frame sequence are synchronized and aligned with the temporal information of sound event types in the audio frame sequence through a time alignment mechanism. The bounding box position information after synchronization and alignment, the behavioral characteristics of the monitored object, and the sound event type are fused to generate a joint behavioral feature vector. S6. Based on the joint behavioral feature vector, identify abnormal behavior, use the event classification model to determine whether the current monitored object has predefined risk behaviors, and output the behavior identification results; S7. Generate corresponding home security alarm commands based on behavior recognition results and push them to the user interface via mobile terminal.

2. The home security AI monitoring method based on multimodal algorithm fusion according to claim 1, characterized in that, S1, specifically: Collect raw video and audio data within the home's internal monitoring area; Image frames are extracted from the original video image data at set time intervals, and a video image frame sequence is constructed in chronological order. The original audio data is divided into time-domain frames according to fixed frame length and frame shift parameters, and each audio frame is associated with a corresponding time index to generate an audio frame sequence.

3. The home security AI monitoring method based on multimodal algorithm fusion according to claim 2, characterized in that, S4, specifically: Extract the audio signal features of each audio frame in the audio frame sequence; The audio signal features are input into the event classification model to classify and identify the sound events contained in the audio frame sequence, identify the category of the sound event, and output the corresponding sound event type.

4. The home security AI monitoring method based on multimodal algorithm fusion according to claim 3, characterized in that, Audio signal characteristics include Mel frequency cepstral coefficients, short-time energy, and short-time zero-crossing rate.

5. A home security AI monitoring method based on multimodal algorithm fusion according to claim 4, characterized in that, S6, specifically: Input the joint behavioral feature vector into the event classification model; The event classification model classifies and identifies the behavior of monitored objects based on joint behavioral feature vectors, and identifies whether the current behavior of the monitored objects belongs to predefined risk behaviors. Based on the classification and recognition results of the event classification model, the behavior recognition results of the monitored objects are generated.

6. A home security AI monitoring method based on multimodal algorithm fusion according to claim 5, characterized in that, S7, specifically: Based on the behavior recognition results of the monitored object, determine whether the current behavior of the monitored object is a risky behavior; When the current behavior of the monitored object is considered risky, a home security alarm command corresponding to the current behavior of the monitored object is generated. Home security alarm commands are pushed to the user's mobile device and displayed on the user's mobile device's user interface.

7. A home security AI monitoring system based on multimodal algorithm fusion, used to implement the home security AI monitoring method based on multimodal algorithm fusion as described in any one of claims 1-6, characterized in that, include: Data acquisition module: Acquires raw video image data and raw audio data in home monitoring scenarios, and performs image frame extraction and temporal framing to generate video image frame sequences and audio frame sequences. Target recognition module: Based on convolutional neural network, target recognition is performed on video image frame sequences to generate bounding box location information of the monitored object in each frame image; Behavior analysis module: Based on the bounding box position information of the monitored object in each frame of image, the behavioral features of the monitored object are extracted through a long short-term memory neural network; Sound recognition module: performs feature extraction and event recognition on audio frame sequences, and combines an event classification model to identify the corresponding sound event type; Feature fusion module: Synchronously corrects bounding box location information, behavioral features and sound event types to generate a joint behavioral feature vector; Anomaly detection module: Based on joint behavioral feature vectors, it identifies abnormal behavior, uses an event classification model to determine whether the current monitored object has predefined risky behavior, and outputs the behavior detection results; Alarm push module: Generates corresponding home security alarm commands based on behavior recognition results and pushes them to the user interface via mobile terminal.

Images