A video stream event analysis and wearable device real-time alarm system

Abstract

This invention relates to the field of video stream event analysis technology, specifically a video stream event analysis and wearable device real-time alarm system. The invention segments video streams and identifies event segments with significant spatiotemporal variations through keyframe recognition and frequency domain feature extraction, thereby improving the efficiency of event analysis. Based on a recurrent neural network model, it identifies and evaluates abnormal events in the video through precise temporal data modeling and event label matching. Furthermore, through an adversarial joint training module, it jointly optimizes the event recognition model and the matching result analysis model using a generative adversarial network, making the model more robust and generalizing in complex environments, thus improving the accuracy of event recognition. Alarm information is generated based on the reliability probability of event labels, providing timely feedback to users. This invention improves the real-time performance and accuracy of event detection through the close integration of video stream event analysis and wearable devices.

Description

Technical Field

[0001] This invention relates to the field of video stream event analysis technology, specifically to a video stream event analysis and wearable device real-time alarm system. Background Technology

[0002] With the development of intelligent monitoring and security technologies, video stream-based anomaly detection and alarm systems are gradually being widely used. Traditional monitoring systems detect events through static monitoring and manual analysis, but as the amount of video data continues to increase, the efficiency and accuracy of manual analysis are becoming bottlenecks. Furthermore, traditional monitoring systems can typically only detect events with known patterns and cannot accurately identify new and unknown anomalies in ever-changing real-world scenarios.

[0003] To address this challenge, modern video stream event analysis systems have begun to incorporate deep learning and computer vision technologies to perform real-time event recognition in an automated and intelligent manner. These systems can quickly extract features, analyze events, and generate alerts from large-scale video stream data. However, existing video stream event analysis systems are often limited by the amount of data, especially since processing real-time video streams requires efficient computing and storage resources, resulting in high latency and an inability to respond promptly at critical moments.

[0004] Secondly, existing alarm systems often rely on fixed alarm rules, such as threshold judgment based on event matching tags, which cannot flexibly adjust alarm responses. Traditional alarm systems fail to effectively consider alarm levels and users' actual needs, resulting in serious false alarms and missed alarms.

[0005] To address the aforementioned issues, it is necessary to propose a video stream event analysis and wearable device real-time alarm system. Summary of the Invention

[0006] The purpose of this invention is to solve the problems existing in the background art, and to propose a video stream event analysis and wearable device real-time alarm system.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A video stream event analysis and wearable device real-time alarm system includes a video stream acquisition module, an event analysis module, an analysis result scoring module, an adversarial joint training module, a signal matching and alarm module, and a wearable device execution response module.

[0009] The video stream acquisition module acquires raw video signals and entropy coding compression algorithm to encode the acquired raw video signals in real time to obtain standard video stream data. Keyframe recognition algorithm extracts feature frames from the standard video stream data. The raw video signal is then segmented using these feature frames to obtain several video slices.

[0010] It can access webcams and CCTV cameras in real time and capture raw video signals at preset frame rates and video encoding formats.

[0011] In a preferred embodiment of the present invention, the acquired raw video signal is encoded in real time using an entropy coding compression algorithm to obtain standard video stream data. The specific process is as follows:

[0012] The original video signal is transformed to the frequency domain using Discrete Cosine Transform (DCT), yielding the frequency domain coefficients of each image frame. All frequency domain coefficients of each image frame are then arranged according to their frequency domain indices to obtain a frequency domain coefficient matrix. Finally, the frequency domain coefficient matrices corresponding to each image frame are arranged according to their image frame indices to obtain the standard video stream data.

[0013] As a preferred embodiment of the present invention, feature frames are extracted from standard video stream data using a keyframe recognition algorithm. The specific process is as follows:

[0014] In standard video stream data, the sum of the upper diagonal portion of the frequency domain coefficient matrix corresponding to each image frame is calculated to obtain the high-frequency feature values ​​corresponding to each image frame. For the high-frequency feature values ​​of each adjacent image frame, the coefficient of variation of the high-frequency features for each image frame is calculated.

[0015] If the coefficient of variation of a high-frequency feature of an image frame is found to be greater than a preset threshold, the image frame is determined to be a feature frame.

[0016] The original video signal is segmented based on feature frames to obtain several video slices. The segmentation rules are as follows:

[0017] If there are several consecutive feature frames, that is, if the image frame indices of multiple feature frames are an arithmetic sequence with a common difference of 1, then the consecutive feature frames are grouped into the same video slice.

[0018] All image frames between two non-contiguous feature frames are grouped into the same video slice;

[0019] All generated video slices are sent to the event analysis module and the adversarial joint training module.

[0020] The event analysis module inputs the received video slices into the event recognition model based on a recurrent neural network to obtain event matching labels for each feature frame.

[0021] Event matching labels for each video slice are calculated using an event recognition model based on a recurrent neural network. The preset event matching labels point to: no abnormal situation, traffic collision, running a red light, speeding, people being hit, fire, or damage to infrastructure structures.

[0022] Send the event recognition model to the adversarial joint training module.

[0023] The analysis results scoring module inputs the event matching labels of each video slice into the matching results analysis model based on a recurrent neural network to obtain the reliability probability of the event matching labels.

[0024] The credibility of the matching results is obtained by using a recurrent neural network-based matching result analysis model to score the credibility of the image frames in the video slices and the event matching labels output by the corresponding event analysis module.

[0025] The label confidence level ranges from 0 to 1 and is used to evaluate whether the event matching labels output by the event analysis module are reliable. A label confidence level of 1 means that the event matching label describes the image frame completely accurately; a label confidence level of 0 means that the event matching label describes the image frame completely incorrectly.

[0026] The matching result analysis model is sent to the adversarial joint training module.

[0027] The adversarial joint training module uses a GA generative adversarial network to jointly train the event recognition model and the matching result analysis model, and sends the model parameters of the trained event recognition model to the signal matching and alarm module.

[0028] The Gan generative adversarial network consists of a generator G and a discriminator D.

[0029] Generator G contains several event recognition models. Each model generates pseudo-event matching labels based on video slices, simulating the event recognition model's operation on event matching labels for image frames within the video slices. The pseudo-event matching labels and their corresponding video frames from the video slices are then sent to discriminator D. The goal of generator G is to make the discriminator unable to distinguish between the generated pseudo-event matching labels and real event matching labels. Each event recognition model generates pseudo-event matching labels based on video slices, simulating the event recognition model's operation on event matching labels for image frames within the video slices.

[0030] The loss function of generator G optimizes the set of training parameters E1 of each event recognition model in the generator by minimizing the probability that the discriminator determines whether the label is real;

[0031] The discriminator D comprises a matching result analysis model and a set of preset standard video slices. It performs label confidence calculations on all pseudo-event matching labels and their corresponding image frames, and compares the confidence calculation results with the confidence calculation results of the preset standard video slices and their corresponding real event matching labels. The discriminator's goal is to correctly distinguish between real event matching labels and pseudo-event matching labels.

[0032] The loss function of discriminator D is optimized by maximizing the difference between the generated label and the real label;

[0033] In a preferred embodiment of the present invention, adversarial training is performed between a generator G and a discriminator D, allowing the generator and discriminator to continuously optimize each other. This adversarial training is achieved by minimizing the loss functions of both the generator G and the discriminator D. The generator continuously generates data with a level of credibility closer to the true labels, while the discriminator continuously improves its ability to identify fake data.

[0034] In a preferred embodiment of the present invention, in each training iteration, the generator and discriminator optimize the set of parameters to be trained using the backpropagation algorithm until the loss function of the generator G and the loss function of the discriminator D both reach their minimum values. At this point, the set of parameters to be trained for each event recognition model is saved, and this is marked as training complete. The set of model parameters of the trained event recognition model is then sent to the signal matching and alarm module.

[0035] The signal matching and alarm module saves the model parameters of the trained event recognition model. Video slices output in real-time from the video stream acquisition module are input into the trained event recognition model to obtain event matching labels for each feature frame in each video slice. The percentage of repeated occurrences of the same event matching label within the same video slice is calculated to obtain event matching information, including alarm content and alarm level.

[0036] Record the event matching labels for each feature frame in each video slice, and count the event matching labels that appear most frequently.

[0037] The specific process of matching alarm information based on events is as follows:

[0038] If the most frequently occurring event is identified as having no abnormality, no alarm information will be generated, and no subsequent alarm information level matching will be performed.

[0039] If the most frequently occurring event matching tag is not 0, then the alarm information content is matched according to the accident type pointed to by the event matching tag.

[0040] As a preferred embodiment of the present invention, the alarm information level is obtained based on event matching information, and the specific process is as follows:

[0041] If the most frequently occurring event matching tag accounts for 80% or more of all event matching tags, then the alarm level is set to Level 3.

[0042] If the most frequently occurring event matching tag accounts for less than 80% but greater than or equal to 50% of all event matching tags, then the alarm information level is set to level two.

[0043] If the most frequently occurring event matching tag accounts for less than 50% but greater than or equal to 30% of all event matching tags, then the alarm level is set to Level 1.

[0044] The wearable device's response module triggers an automatic response mechanism based on the content and level of the received alarm information.

[0045] If the alarm information level is identified as Level 1, the time and location of the event are recorded and output through the display screen of the wearable device;

[0046] If the alarm information level is identified as level two, detailed event information and the steps the user should take will be displayed on the screen and output through the wearable device's display screen; and the corresponding video clip will be played through the wearable device.

[0047] If the alarm information level is detected as level three, the wearable device will vibrate violently and emit a high-decibel sound warning to ensure that the user's attention is drawn; and the corresponding video clip will be played through the wearable device.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] This invention provides a video stream-based event analysis and wearable device real-time alarm system. Through the collaborative work of multiple modules, it significantly improves the accuracy and real-time performance of event analysis. First, the video stream acquisition module acquires video signals using high frame rates and high-quality encoding formats, ensuring the clarity and accuracy of the video data, facilitating subsequent feature extraction and abnormal event identification. Through keyframe recognition and frequency domain feature extraction, the system can efficiently segment the video stream and identify event segments with significant spatiotemporal variations, thereby improving the efficiency of event analysis.

[0050] The event analysis module and analysis result scoring module proposed in this invention adopt a model based on recurrent neural networks. Through accurate time-series data modeling and event label matching, they can promptly identify and evaluate abnormal events in videos and provide reliable event information for subsequent alarms. The adversarial joint training module uses a generative adversarial network (GAN) to jointly optimize the event recognition model and the matching result analysis model, making the model more robust and generalizing in complex environments, thereby improving the accuracy of event recognition. Attached Figure Description

[0051] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0053] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A or solution B, or a solution that simultaneously satisfies A and B. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0054] Please see Figure 1 As shown, a video stream event analysis and wearable device real-time alarm system includes a video stream acquisition module, an event analysis module, an analysis result scoring module, an adversarial joint training module, a signal matching and alarm module, and a wearable device execution response module.

[0055] The video stream acquisition module accesses network cameras and closed-circuit television cameras in real time, acquiring raw video signals at a preset frame rate and video encoding format. The acquired raw video signals are then encoded in real time using an entropy coding compression algorithm to obtain standard video stream data. A keyframe recognition algorithm extracts feature frames from the standard video stream data to capture abnormal actions. The raw video signal is then segmented using these feature frames to obtain several video slices. Historical data from all extracted video slices is sent to the event analysis module. Finally, all real-time extracted video slices are sent to the adversarial joint training module and the signal matching and alarm module.

[0056] Access webcams and CCTV cameras in real time to capture raw video signals vi={v1, v2, v3, ..., vn} at a preset frame rate of 30fps and H.264 video encoding format.

[0057] Where i is the image frame index, i = 1, 2, ..., n;

[0058] Where v1, v2, v3, ..., vn are the image frames contained in the original video signal, and n is the total number of image frames contained in the original video signal vi.

[0059] It's important to note that in image processing, the choice of frame rate is closely related to the object's movement speed and the spatiotemporal resolution of the image. Generally, the maximum displacement of an object between two adjacent frames should not exceed the minimum size of the target object. In video stream event analysis applications, when objects such as people and vehicles are captured at 30fps, the time interval between two adjacent frames is 33.33 milliseconds. At this time, the object's displacement typically does not exceed 2 meters. Therefore, 30fps ensures sufficiently high temporal resolution, avoiding the loss or blurring of key features due to insufficient frame rate. Furthermore, the H.264 video encoding format can provide high-quality video images, especially performing well at low bitrates. It can maintain high video clarity even with lower bandwidth.

[0060] Furthermore, the acquired raw video signal is encoded in real time using an entropy coding compression algorithm to obtain standard video stream data. The specific process is as follows:

[0061] The original video signal V is transformed into the frequency domain by using Discrete Cosine Transform (DCT) to obtain the frequency domain coefficients vi(u, v) of each image frame vi. The transformation formula is as follows:

[0062]

[0063] Where x and y are the height and width components of the pixel coordinates (x, y) in each image frame vi, respectively; the value range is x=0, 1, 2, ..., M-1; y=0, 1, 2, ..., N-1; M is the image width of image frame vi, and N is the image height of image frame vi;

[0064] Among them The frequency domain coefficients obtained after transformation represent the low-frequency and high-frequency components of the image in the frequency domain, where u and v are the transformed frequency domain indices, with values ​​ranging from u = 0, 1, 2, ..., M-1; y = 0, 1, 2, ..., N-1; when the frequency domain coefficients The closer the frequency domain indices u and v are to 0, the closer they are to the low-frequency part of image frame vi, i.e., the main contour and structural information of the image, which will serve as key parameters for subsequent abnormal motion capture calculations; when the frequency domain coefficients... The further the frequency domain indices u and v are from 0, the closer they are to the high-frequency part of the image frame vi, that is, the detailed information contained in the edges and textures of the image, which can be reduced or discarded during image compression.

[0065] Among them and Let be the predefined normalized coefficients of the DCT discrete cosine transform, and let its expression be:

[0066]

[0067] Arrange all frequency domain coefficients of each image frame vi according to their frequency domain indices u and v to obtain the frequency domain coefficient matrix:

[0068]

[0069] Arrange the frequency domain coefficient matrix Vi corresponding to each image frame vi according to the image frame index i=1,2,...,n to obtain the standard video stream data Vi=V1,V2,...,Vn.

[0070] Furthermore, feature frames are extracted from standard video stream data using a keyframe recognition algorithm. The specific process is as follows:

[0071] Calculate the sum of the upper diagonal portions of the frequency domain coefficient matrix Vi corresponding to each image frame vi in ​​the standard video stream data:

[0072]

[0073] Obtain the high-frequency feature values ​​corresponding to each image frame vi. .

[0074] It should be noted that high-frequency eigenvalues This is the summation of the frequency domain coefficients in the frequency domain coefficient matrix where the frequency domain indices u and v are close to 0, containing the main contour and structural information of all images. When a significant change in the high-frequency feature values ​​of adjacent image frames is detected, it indicates that there has been a significant contour displacement, increase or decrease, or change in the object structure between these two adjacent image frames.

[0075] Furthermore, for the high-frequency feature values ​​of each adjacent image frame vi and vi+1 and By using a preset formula:

[0076]

[0077] Calculate the coefficient of variation of high-frequency features for each image frame vi. .

[0078] If the coefficient of variation of the high-frequency features of image frame vi is found to be greater than the preset threshold of 0.5, it means that the change in the high-frequency feature value from the image frame to the next image frame is greater than 50%, and image frame vi is determined to be a feature frame.

[0079] The original video signal vi = {v1, v2, v3, ..., vn} is segmented based on feature frames to obtain several video slices. The segmentation rule is as follows:

[0080] If there are several consecutive feature frames, that is, if the image frame index i of multiple feature frames is an arithmetic sequence with a common difference of 1, then the consecutive feature frames are grouped into the same video slice.

[0081] All image frames between two non-contiguous feature frames are grouped into the same video slice;

[0082] All generated video slices are sent to the event analysis module and the adversarial joint training module.

[0083] It should be noted that video slicing divides the original video stream into several smaller temporal segments by segmenting continuous feature frames. Each video slice represents a segment of an event or behavior with significant temporal and spatial variations, facilitating subsequent event analysis and anomaly detection. By extracting and segmenting feature frames, the originally long video data can be simplified into small segments with key spatiotemporal features, greatly improving analysis efficiency.

[0084] It should be further explained that video slicing identifies significant changes between image frames based on the high-frequency feature variation coefficient between frames, timely captures sudden movements or structural changes of objects in the scene, and identifies video slices containing continuous events. This not only preserves the temporal continuity of events but also effectively reduces redundant data, thereby improving the computational efficiency and response speed of subsequent analysis modules when processing video streams.

[0085] The event analysis module inputs the received video slices into the event recognition model based on a recurrent neural network to obtain event matching labels for each feature frame.

[0086] An event recognition model based on recurrent neural networks:

[0087]

[0088] Calculate event matching tags for each video slice ;

[0089] in, The hidden state of the event recognition model; where Output in hidden state The tanh activation function is used. and Let be the first and second weight matrices of the hidden state to be trained, which are used to map the features of image frame vi in ​​the input video slice and the hidden state at the previous time step to the current state, respectively. Output the hidden state from the previous loop. The hidden state bias term to be trained;

[0090] in, This is the output layer of the event recognition model, responsible for outputting the hidden states. Perform a fully connected computation and map it to an event label space, matching it with preset event matching labels. ;in For the sigmoid activation function, where and These are the weight matrix and bias term of the output layer to be trained, respectively;

[0091] Among them, the preset event matching tags include:

[0092] =0 indicates no abnormal situation;

[0093] =1 represents a traffic collision accident;

[0094] =2 indicates that a red light violation occurred;

[0095] =3 indicates that speeding has occurred;

[0096] =5 indicates that someone was hit;

[0097] =6 indicates that a fire has occurred;

[0098] =7 indicates that there is damage to the infrastructure structure;

[0099] Send the event recognition model to the adversarial joint training module.

[0100] The analysis results scoring module inputs the event matching labels of each video slice into the matching results analysis model based on a recurrent neural network to obtain the reliability probability of the event matching labels.

[0101] The model is analyzed based on the matching results from a recurrent neural network.

[0102]

[0103] The event matching label is output by the event analysis module for the image frame vi in ​​the video slice and the corresponding event. Perform a credibility score to obtain the label credibility. .

[0104] in The hidden state of the matching result analysis model; where For output in the hidden state, The tanh activation function is used. The first weight matrix for the hidden state to be trained is used to match the image frame vi in ​​the input video slice with the event labels. The combination of these is mapped to a higher-dimensional space; This is the second weight matrix for the hidden state to be trained, responsible for taking the output of the hidden state from the previous iteration. Mapped to the current state, The hidden state bias term to be trained;

[0105] in The output gate of the matching result analysis model, For the credibility of the label, For the sigmoid activation function, where and These are the weight matrix and bias term of the output layer to be trained, respectively;

[0106] Tag credibility The numerical range is 0 to 1, used to evaluate the event matching tags output by the event analysis module. Is it reliable? How trustworthy is the label? A value of 1 indicates that the event matches the tag. The description of image frame vi is completely accurate; the label reliability is high. A value of 0 indicates that the event matches the tag. The description of image frame vi is completely wrong; label credibility. A value greater than 0.8 indicates that the event matches the tag. The description of image frame vi reaches the minimum acceptable probability of credibility.

[0107] The matching result analysis model is sent to the adversarial joint training module.

[0108] The adversarial joint training module uses a GA generative adversarial network to jointly train the event recognition model and the matching result analysis model, and sends the model parameters of the trained event recognition model to the signal matching and alarm module.

[0109] The Gan generative adversarial network consists of a generator G and a discriminator D.

[0110] Generator G contains several event recognition models. Each model generates pseudo-event matching labels based on video slices, simulating the event recognition model's operation on event matching labels for image frames within the video slices. The pseudo-event matching labels and their corresponding video frames from the video slices are then sent to discriminator D. The goal of generator G is to make the discriminator unable to distinguish between the generated pseudo-event matching labels and real event matching labels. Each event recognition model generates pseudo-event matching labels based on video slices, simulating the event recognition model's operation on event matching labels for image frames within the video slices.

[0111] The loss function of generator G optimizes the set of training parameters E1 of each event recognition model in the generator by minimizing the probability that the discriminator determines whether a label is real. Its loss function is:

[0112]

[0113] Where E1 is the set of parameters to be trained for each event recognition model, E1={ , , , , }; that is, the hidden state of each event recognition model, the first weight matrix, the second weight matrix, the weight matrix of the output layer to be trained, and the bias term of the output layer to be trained;

[0114] Among them This indicates that the pseudo-events output by the generator match the tags.

[0115] in This indicates that the discriminator matches the generated pseudo-events with tags. The discrimination result is in the range of 0 to 1, where P(vi) is the set of inputs to the generator, i.e., all video slices.

[0116] The discriminator D comprises a matching result analysis model and a set of preset standard video slices. It performs label confidence calculations on all pseudo-event matching labels and their corresponding image frames vi, and compares the confidence calculation results with the confidence calculation results of the preset standard video slices and their corresponding real event matching labels. The discriminator's goal is to correctly distinguish between real event matching labels and pseudo-event matching labels.

[0117] The loss function of the discriminator D is optimized by maximizing the difference between the generated label and the real label. Its loss function is:

[0118]

[0119] E2 is the set of parameters to be trained for the matching result analysis model, E2={ , , , , }, including the hidden state of the matching result analysis model, the first weight matrix, the second weight matrix, the weight matrix of the output layer to be trained, and the bias terms of the output layer to be trained;

[0120] Where x is a preset standard video slice, and G(x) is the discrimination result of the discriminator on the standard video slice x;

[0121] Adversarial training is performed between the generator G and the discriminator D, allowing the generator and discriminator to continuously optimize each other by minimizing the loss function of the generator G. The loss function of discriminator D Implement adversarial training between the generator and the discriminator. The generator continuously generates data with a level of credibility closer to the real labels, while the discriminator continuously improves its ability to identify fake data.

[0122] Furthermore, in each training iteration, the generator and discriminator are optimized using the backpropagation algorithm on the sets E1 and E2 of parameters to be trained, until the loss function of the generator G is optimized. The loss function of discriminator D Once all parameters reach their minimum values, the set E1 containing the training parameters of each event recognition model is saved, marking the training as complete. The set E1 containing the model parameters of the trained event recognition models is then sent to the signal matching and alarm module.

[0123] It's important to note that Generative Adversarial Networks (GANs) can generate high-quality data samples in an unsupervised environment through adversarial training. The game-like mechanism between the generator and discriminator allows the model to self-optimize during training without relying on large amounts of labeled data. Traditional machine learning models typically depend on manually labeled, large-scale datasets, while GANs, through adversarial training, can process unlabeled data more efficiently, reducing the cost and time of data labeling.

[0124] The signal matching and alarm module saves the model parameters of the trained event recognition model. Video slices output in real-time from the video stream acquisition module are input into the trained event recognition model to obtain event matching labels for each feature frame in each video slice. The percentage of repeated occurrences of the same event matching label within the same video slice is calculated to obtain event matching information, including alarm content and alarm level.

[0125] Record the event matching labels for each feature frame in each video slice, and count the event matching labels that appear most frequently.

[0126] The specific process of matching alarm information based on events is as follows:

[0127] If the most frequently occurring event is identified as matching the tag If the value is 0, no alarm information will be generated, and no subsequent alarm information level matching will be performed.

[0128] If the most frequently occurring event is identified as matching the tag =1, then a traffic collision alarm message is generated; if the most frequently occurring event matching tag is identified, then... =2, then a red light violation warning message is generated; if the most frequently occurring event is identified, the matching tag is =3, then an overspeed alarm message is generated; if the most frequently occurring event is identified, the matching tag is =5, then generate a personnel collision alarm message; if the most frequently occurring event matching tag is identified as =6, then a fire alarm message is generated; if the most frequently occurring event matching tag is identified as If the value is 7, an infrastructure structure damage alarm message will be generated.

[0129] Furthermore, the alarm information level is obtained based on the event matching information. The specific process is as follows:

[0130] If the most frequently occurring event matching tag accounts for 80% or more of all event matching tags, then the alarm level is set to Level 3.

[0131] If the most frequently occurring event matching tag accounts for less than 80% but greater than or equal to 50% of all event matching tags, then the alarm information level is set to level two.

[0132] If the most frequently occurring event matching tag accounts for less than 50% but greater than or equal to 30% of all event matching tags, then the alarm level is set to Level 1.

[0133] The wearable device's response module triggers an automatic response mechanism based on the content and level of the received alarm information.

[0134] If the alarm information level is identified as Level 1, the time and location of the event are recorded and output through the display screen of the wearable device;

[0135] If the alarm information level is identified as level two, detailed event information and the steps the user should take will be displayed on the screen and output through the wearable device's display screen; and the corresponding video clip will be played through the wearable device.

[0136] If the alarm information level is detected as level three, the wearable device will vibrate violently and emit a high-decibel sound warning to ensure that the user's attention is drawn; and the corresponding video clip will be played through the wearable device.

Claims

1. A video stream event analysis and wearable device instant alarm system, comprising a video stream acquisition module, an event analysis module, an analysis result scoring module and an adversarial joint training module, characterized in that: the video stream acquisition module acquires original video signals, encodes the acquired original video signals in real time through an entropy encoding compression algorithm to obtain standard video stream data; and performs feature frame extraction from the standard video stream data through a key frame recognition algorithm; the original video signals are cut into several video slices through the feature frames; the event analysis module inputs the received video slices into an event recognition model based on a recurrent neural network to obtain event matching labels of each feature frame; the analysis result scoring module inputs the event matching labels of each video slice into a matching result analysis model based on a recurrent neural network to obtain a reliability probability of the event matching labels; the adversarial joint training module jointly trains the event recognition model and the matching result analysis model through a Gan generative adversarial network, and sends the model parameters of the trained event recognition model to a signal matching and alarm module; the specific process of jointly training the event recognition model and the matching result analysis model through the Gan generative adversarial network is that: a generator G and a discriminator D are generated, the generator G includes several event recognition models, each event recognition model generates a pseudo event matching label according to a video slice to simulate the event matching label operation of the event recognition model on the image frames in the video slice; and the pseudo event matching label and the video frames corresponding to the video slice are sent to the discriminator D; the goal of the generator G is to make the discriminator unable to distinguish between the generated pseudo event matching label and the real event matching label; each event recognition model generates a pseudo event matching label according to a video slice to simulate the event matching label operation of the event recognition model on the image frames in the video slice. It also includes a signal matching and alarm module and a wearable device execution response module:

2. The video stream event analysis and wearable device instant alert system of claim 1, wherein, the signal matching and alarm module saves the model parameters of the trained event recognition model; inputs the video slices output in real time by the video stream acquisition module into the trained event recognition model to obtain event matching labels of each feature frame in each video slice; and obtains event matching information including alarm information content and alarm information level by counting the repeated occurrence proportion of the same event matching label in the same video slice; the wearable device execution response module triggers a corresponding automatic response mechanism according to the received alarm information content and alarm information level. The specific process of encoding the acquired original video signals in real time through the entropy encoding compression algorithm is that:

3. The video stream event analysis and wearable device instant alert system of claim 2, wherein, each frame of the original video signals is converted to the frequency domain through DCT discrete cosine transform to obtain frequency domain coefficients of each image frame; all frequency domain coefficients of each image frame are arranged according to the frequency domain index to obtain a frequency domain coefficient matrix; and the frequency domain coefficient matrix corresponding to each image frame is arranged according to the image frame index to obtain the standard video stream data. The specific process of extracting feature frames from the standard video stream data through the key frame recognition algorithm is that:

4. The video stream event analysis and wearable device instant alert system of claim 3, wherein, ​ The sum of the upper diagonal part of the frequency domain coefficient matrix corresponding to each image frame in the standard video stream data is calculated to obtain a high-frequency eigenvalue corresponding to each image frame; and a high-frequency eigenvalue variation coefficient of each image frame is calculated according to the high-frequency eigenvalues of adjacent image frames. If the high-frequency eigenvalue variation coefficient of the image frame is greater than a preset threshold, it is determined that the image frame is a feature frame.

5. The video streaming event analysis and wearable device instant alert system of claim 1, wherein, The specific process of splitting the original video signal based on the feature frame is as follows: If there are several continuous feature frames, that is, the image frame indexes of the multiple feature frames form an arithmetic sequence with a common difference of 1, the continuous feature frames are classified into the same video slice; All image frames between the two discontinuous feature frames are classified into the same video slice; All the generated video slices are sent to the event analysis module and the adversarial joint training module.

6. The video streaming event analysis and wearable device instant alert system of claim 1, wherein, The reliability probability of the event matching label is specifically: The numerical range of the label credibility is 0 to 1, which is used to evaluate whether the event analysis module output event matching label is reliable. The label credibility of 1 represents that the event matching label is completely accurate in describing the image frame; and the label credibility of 0 represents that the event matching label is completely wrong in describing the image frame.

7. The video streaming event analysis and wearable device instant alert system of claim 1, wherein, The generator and the discriminator are specifically: The generator G includes a plurality of event recognition models. Each event recognition model generates a pseudo event matching label according to the video slice to simulate the event recognition model operation on the event matching label of the image frame in the video slice. The pseudo event matching label and the video frame corresponding thereto are sent to the discriminator D. The goal of the generator G is to make the discriminator unable to distinguish between the generated pseudo event matching label and the real event matching label. Each event recognition model generates a pseudo event matching label according to the video slice to simulate the event recognition model operation on the event matching label of the image frame in the video slice. The loss function of the generator G optimizes the set E1 composed of the to-be-trained parameters of each event recognition model in the generator by minimizing the probability that the discriminator determines that the label is real. The discriminator D includes a matching result analysis model and a group of preset standard video slices. The label credibility of all pseudo event matching labels and the corresponding image frames is calculated, and the credibility calculation result is compared with the credibility calculation result of the preset standard video slice and the corresponding real event matching label. The goal of the discriminator is to correctly distinguish between the real event matching label and the pseudo event matching label. The loss function of the discriminator D is optimized by maximizing the difference between the generated label and the real label. The generator G and the discriminator D are trained in an adversarial manner.

8. The video stream event analysis and wearable device instant alert system of claim 7, wherein, The specific process of training the generator G and the discriminator D in an adversarial manner is as follows: The generator and the discriminator are constantly optimized, and the adversarial training of the generator and the discriminator is realized by minimizing the loss function of the generator G and the loss function of the discriminator D. The generator will constantly generate data closer to the credibility of the real label, and the discriminator will constantly improve the ability to identify pseudo data. In each training iteration, the generator and the discriminator are allowed to optimize the set of to-be-trained parameters through a back propagation algorithm until the loss function of the generator G and the loss function of the discriminator D are minimized, at which time the set of to-be-trained parameters of each event recognition model is saved, denoted as training completion; the set of model parameters of the event recognition model after training completion is sent to the signal matching and alarming module.

Images