Edge-cloud collaborative real-time video analysis method, device and medium

Abstract

The invention provides an edge-cloud collaborative real-time video analysis method and device and a medium, and belongs to the technical field of security monitoring, and the method comprises the steps: obtaining real-time video data, and analyzing non-critical events in the real-time video data at an edge end through employing a lightweight deep learning inference engine; when the real-time video data relates to a key event, the edge end synchronizes the real-time video data subjected to low-delay processing to the cloud end according to a cloud-edge collaborative optimization strategy; and processing key events in the real-time video data at the cloud according to natural language analysis in combination with a video understanding strategy, and feeding back a processing result. According to the technical scheme, the problems that the real-time processing capability of the video is insufficient and the video content is difficult to analyze quickly and accurately due to the fact that the prior art depends on centralized processing of a cloud side can be solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of security monitoring, and particularly relates to a real-time video analysis method, device and medium based on edge-cloud cooperation. BACKGROUND

[0002] Real-time video analysis technology is a branch of computer image vision technology applied in the field of security monitoring, and is an intelligent monitoring technology based on target behavior. Unlike traditional mobile detection technology, real-time video analysis first separates the background and target in the scene, identifies the real target, removes background interference (such as tree leaf shaking, water wave or light change, etc.), and then analyzes and tracks the target behavior appearing in the camera scene. Real-time video analysis technology can accurately identify the real active target in the video.

[0003] Most of the existing real-time video analysis technology relies on centralized processing of the cloud, and only simple data collection and preprocessing functions can be performed at the edge. For example, the prior art CN119629388A discloses a real-time video analysis method based on dynamic detection and cloud AI combination, which includes: receiving an application request from a connection address of a current monitoring camera, providing a usable connection address for the current monitoring camera after authorization authentication, so that the current monitoring camera and the streaming media server establish a long connection; uploading the real-time captured video clip to the object storage; reporting the return information corresponding to the screenshot, the information of the current monitoring camera and the time of the picture change to the dispatch server for dispatch to one or more corresponding analysis servers for analysis; receiving the analysis result of the one or more corresponding analysis servers, and notifying the video APP of the analysis result, which includes the AI recognition analysis result, AI event and event start time.

[0004] However, because the existing real-time video analysis technology relies on centralized processing of the cloud, the real-time processing capability of the video is insufficient, and it is difficult to realize fast and accurate analysis of the video content. SUMMARY

[0005] The technical solution described below provides a real-time video analysis method, device and medium based on edge-cloud cooperation, which mainly aims to solve the problem that the existing technology relies on centralized processing of the cloud, resulting in insufficient real-time processing capability of the video, and it is difficult to realize fast and accurate analysis of the video content.

[0006] According to a first aspect of the application, the embodiments of the application provide a real-time video analysis method based on edge-cloud cooperation, which includes: obtaining real-time video data, and using a lightweight deep learning inference engine to analyze non-key events in the real-time video data at the edge; When the real-time video data relates to a key event, the edge end synchronizes the low-delay processed real-time video data to the cloud end according to a cloud-edge collaborative optimization strategy; In the cloud end, the key event in the real-time video data is processed according to a natural language analysis combined with a video understanding strategy, and the processing result is fed back.

[0007] Preferably, the real-time video analysis method uses a lightweight deep learning inference engine to analyze non-key events in the real-time video data at the edge end, including: A spatiotemporal information processing model is constructed using a time series convolution network and a three-dimensional convolutional neural network at the edge end; According to a model compression technique, the spatiotemporal information processing model is compressed, and the compressed spatiotemporal information processing model is introduced into a lightweight convolutional neural network architecture to obtain a lightweight deep learning inference engine; According to a hardware acceleration and parallel computing strategy, the lightweight deep learning inference engine is accelerated and processed in parallel to enhance the analysis capability of the real-time video data.

[0008] Preferably, the real-time video analysis method uses a time series convolution network and a three-dimensional convolutional neural network to construct a spatiotemporal information processing model, including: The time series convolution network includes causal convolution, multi-layer dilated convolution, and residual connection; wherein, the input of the causal convolution kernel is limited to the current time step and the previous time step to ensure the time series causal relationship; multi-layer dilated convolution is introduced into the time series convolution network, wherein the dilated factor of each dilated convolution layer is different, and the time series features in the real-time video data are extracted by the dilated factor; residual connection is used in the time series convolution network to connect the multi-layer dilated layers to stabilize the training of the time series convolution network; The time dimension is introduced into the three-dimensional convolutional neural network to analyze the continuous frames of the real-time video data and capture the motion information in the real-time video data through the time dimension; the robustness of the three-dimensional convolutional neural network to spatial changes is enhanced through the time translation invariance of the time dimension; the behavior boundary and type in the event are determined through the context analysis of the continuous frames in the real-time video data.

[0009] Preferably, the real-time video analysis method compresses the spatiotemporal information processing model according to a model compression technique, and introduces the compressed spatiotemporal information processing model into a lightweight convolutional neural network architecture to obtain a lightweight deep learning inference engine, including: The spatiotemporal information processing model is compressed using pruning, quantization, and knowledge distillation techniques, and the attention module and dynamic inference mechanism in the information processing model are retained; The spatio-temporal information processing model after model compression is introduced into the lightweight convolutional neural network architecture; wherein, The attention module is introduced into the lightweight convolutional neural network architecture, and the dynamic inference mechanism is activated according to the dynamic depth and dynamic width of the lightweight convolutional neural network architecture.

[0010] Preferably, in the real-time video analysis method, when the real-time video data involves a key event, the edge end synchronizes the low-delay processed real-time video data to the cloud end according to the cloud-edge collaborative optimization strategy, including: According to the hierarchical task scheduling and data transmission strategy, a data synchronization mechanism of the edge end and the cloud end is constructed; Low-delay data is extracted from the real-time video data through data compression technology; The low-delay data is synchronized to the cloud end through the data synchronization mechanism.

[0011] Preferably, in the real-time video analysis method, when the real-time video data involves a key event, the edge end synchronizes the low-delay processed real-time video data to the cloud end according to the cloud-edge collaborative optimization strategy, including: At the edge end, the attention mechanism of the lightweight deep learning inference engine is used to capture the key frames and dynamic change information of the real-time video data; Through the pre-trained Seq2Seq model, the context vector is generated according to the time sequence characteristics of the real-time video data, and the context vector is converted into descriptive text; The key frames and dynamic change information and the descriptive text are synchronized to the cloud end through the cloud-edge collaborative optimization strategy.

[0012] Preferably, in the real-time video analysis method, the cloud end processes the key event in the real-time video data according to the natural language analysis combined with the video understanding strategy, and feeds back the processing result, including: Using automatic description generation technology, the target behavior and scene information reflected by the key frames and dynamic changes are located and identified, and the descriptive text corresponding to the key event is automatically recorded and stored in combination with the target behavior and scene information; Integrating speech recognition technology, the query language is converted into a query instruction in the form of text; in combination with the semantic understanding ability of the natural language processing model, the query instruction is used to realize deep semantic search of the real-time video data; the vector space model and cosine similarity are used to quantify the similarity between the query instruction in the form of text and the real-time video data, so as to improve the accuracy of deep semantic search; In combination with the emotion analysis and social behavior recognition technology, the voice text and visual signals in the real-time video data are analyzed to identify the emotional state and social behavior pattern of the characters; facial expression recognition and voice emotion recognition technology are used to interpret the human behavior in the real-time video data in combination with the behavior analysis algorithm.

[0013] Preferably, the real-time video analysis method, in combination with the emotion analysis and social behavior recognition technology, analyzes the voice text and visual signals in the real-time video data to identify the emotional state and social behavior pattern of the characters, comprises: using the emotion analysis model to perform text emotion analysis on the voice text to obtain text emotion features, and analyzing the facial expression and behavior features of the video signals; using the multi-modal fusion technology to fuse the text emotion features with the facial expression and behavior features; using the graph neural network in combination with the time sequence features of the real-time video data; according to the emotional state space vector predefined by the social behavior recognition technology, matching the text emotion features with the facial expression and behavior features according to the time sequence features to identify the emotional state and social behavior pattern of the characters.

[0014] According to the second aspect of the present application, the present application also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executed by the processor, wherein the processor implements the edge-cloud collaborative real-time video analysis method provided by any of the above technical solutions when executing the program.

[0015] According to the third aspect of the present application, the present application also provides a computer storage medium having a computer program stored thereon, wherein the computer program is executed to implement the edge-cloud collaborative real-time video analysis method provided by any of the above technical solutions.

[0016] The technical solution of the present application has at least the following technical effects: The edge-cloud collaborative real-time video analysis method provided in the application can not only obtain real-time video data at the edge, but also use a lightweight deep learning inference engine to determine whether the real-time video data involves a key event, and if it is a non-key event, the lightweight deep learning inference engine at the edge is used to analyze the non-key event in the real-time video data. In this way, the real-time video data can be processed at the edge, reducing the computing power pressure of cloud video processing. When the real-time video data involves a key event, the edge synchronizes the low-latency processed real-time video data to the cloud according to the cloud-edge collaborative optimization strategy. In this way, through cloud-edge collaborative optimization, seamless collaboration between the edge and the cloud is ensured, and when an event occurs, the edge device transmits key data to the cloud in real time through low-latency data transmission. The cloud system is responsible for deep analysis and returns the processing result, generating a safety warning. In this way, through the cloud-edge collaborative optimization strategy, including an efficient data synchronization mechanism, data compression technology and a communication protocol, network bandwidth pressure can be reduced and the reliability of data processing can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the application, constitute a part of this application and illustrate embodiments of the application and together with the description serve to explain the application. In the drawings: Figure 1 A flowchart of an edge-cloud collaborative real-time video analysis method provided for the embodiments of the application is shown in the figure. Figure 2 A flowchart of a non-key event analysis method provided for the embodiments of the application is shown in the figure. Figure 1 A flowchart of a data synchronization method of a cloud-edge collaborative optimization strategy provided for the embodiments of the application is shown in the figure. Figure 3 Figure 1 A flowchart of a cloud key event processing method provided for the embodiments of the application is shown in the figure. Figure 4 A structural diagram of an electronic device provided for the embodiments of the application is shown in the figure. Figure 1 Figure 5 A structural diagram of an electronic device provided for the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the overall concept of the application, the following will be described in detail with reference to the accompanying drawings.

[0019] ​​Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0020] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a variety of ways beyond the specific embodiments described herein without departing from the spirit or scope of the present application. It is to be understood that features of the present application and features in each embodiment can be combined with each other without conflict.

[0021] In the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. In the description of the specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0022] The prior art has the following defects: Existing real-time video analysis technology relies on centralized processing in the cloud, resulting in insufficient real-time processing capability of video, and it is difficult to achieve fast and accurate analysis of video content.

[0023] To solve the above problems, the following embodiments of the present application provide an edge-cloud collaborative real-time video analysis scheme. In view of the problem that existing real-time video analysis technology relies on centralized processing in the cloud, resulting in insufficient real-time processing capability of video, and it is difficult to achieve fast and accurate analysis of video content, the technical scheme of the present application can integrate certain computing power and deep learning inference engine in the edge, process non-key events in real-time video data in the edge, and synchronize real-time video data to the cloud for processing according to the cloud-edge collaborative strategy when key events are involved. Through the above-mentioned manner, a lightweight deep learning model can be deployed on the edge device to perform preliminary event detection and target tracking; at the same time, high-performance computing resources are deployed in the cloud to process complex tasks. The layered task scheduling and efficient data transmission strategy are adopted to ensure the smoothness and efficiency of edge-cloud collaboration.

[0024] To achieve the above object, see Figure 1 , Figure 1 The flowchart of the edge-cloud collaborative real-time video analysis method provided by the embodiment of the application is shown in Figure 1 The edge-cloud collaborative real-time video analysis method comprises the following steps: S110: acquiring real-time video data, and using a lightweight deep learning inference engine to analyze non-key events in the real-time video data at the edge.

[0025] An optimized deep learning inference engine is deployed at the edge computing node, which can realize low-delay processing of video data. The edge device is responsible for preliminary analysis of high-frequency and simple events, i.e., the above-mentioned non-key events (such as object appearance and personnel counting), to ensure that the non-key events can be responded to in a timely manner when they occur locally. For complex tasks, i.e., key events (such as face recognition or large-scale behavior analysis), the tasks are allocated to the cloud for deep calculation. Through the above-mentioned manner, the edge device can be given certain event processing capability, and the transmission pressure of video data and the video data processing pressure of the cloud device can be reduced.

[0026] Specifically, as a preferred embodiment, as shown in Figure 2 The real-time video analysis method comprises the following steps: S111: using a time series convolution network and a three-dimensional convolutional neural network to construct a space-time information processing model at the edge; S112: model compression is performed on the space-time information processing model according to a model compression technology, and the model compressed space-time information processing model is introduced into a lightweight convolutional neural network architecture to obtain a lightweight deep learning inference engine.

[0027] The model compression technology herein includes but is not limited to network pruning, quantization, and knowledge distillation. The lightweight convolutional neural network architecture can be MobileNet and ShuffleNet, which reduces the demand for computing resources and improves the efficiency of model operation.

[0028] S113: hardware acceleration and parallel processing are performed on the lightweight deep learning inference engine according to a hardware acceleration and parallel computing strategy to enhance the analysis capability of real-time video data. The hardware acceleration strategy includes GPU acceleration and TPU acceleration, thereby enhancing the processing capability of real-time video analysis.

[0029] The technical scheme provided by the embodiments of the present application is used to enhance the capturing capability of the time sequence dynamic characteristics in the video sequence, and uses a time convolution network (TCN) and a three-dimensional convolution neural network (3D CNN) to model a spatio-temporal information processing model of a video frame. The time convolution network (TCN) can effectively process the time dependence in long time sequence video data, and the 3D CNN further improves the accuracy and robustness of behavior recognition and anomaly detection by introducing the information of the time dimension. In addition, the model compression technology, such as network pruning, quantization and knowledge distillation, is used to greatly reduce the parameters and the calculation complexity of the model, and a lightweight convolution neural network architecture is further introduced, so as to reduce the demand for computing resources and improve the operation efficiency of the model.

[0030] Specifically, as a preferred embodiment, the real-time video analysis method comprises the following steps: The time convolution network comprises a causal convolution, a multi-layer dilated convolution and a residual connection. The input of the causal convolution is limited to the current time step and the previous time step to ensure the time sequence causality. The multi-layer dilated convolution is introduced into the time convolution network, wherein the dilated factors of the dilated convolution layers are different, and the time sequence characteristics in the real-time video data are extracted by the multi-layer dilated convolution according to the dilated factors. The residual connection is used in the time convolution network to connect the multi-layer dilated layers to stabilize the training of the time convolution network. The causal convolution is used to ensure the time sequence correctness. Because the ordinary convolution uses the past and future information to predict the current point when processing the sequence, which is not allowed in video analysis (you cannot use the future frame to predict the state of the current frame). In order to solve this problem, the causal convolution is provided in the embodiments of the present application, which solves this problem by strictly limiting the input of the convolution kernel to the current time step and the previous time step. From the structure, it is ensured that the output at the t time only depends on the input at the t time and before, and does not depend on any future input. This means that when the model processes the 100th frame, it can only "see" the information of the 0th to 100th frames, and does not know what will happen to the 101th frame. This conforms to the causal relationship of video understanding.

[0031] The multi-layer dilated convolution can expand the receptive field exponentially, so as to capture the long-term dependence relationship of the video sequence. This is the most critical feature of the time convolution network, and the above method can solve the problem of limited receptive field of the traditional convolution network. The multi-layer dilated convolution performs "interval sampling" between the inputs of the convolution kernel, which is controlled by a dilated factor. The dilated factor defines the interval between the values when the convolution kernel processes the data. Specifically: The dilated factor d = 1 is the ordinary convolution, and each input is processed.

[0032] Dilation factor d = 2: the kernel skips one input for every input it processes, the receptive field is expanded.

[0033] Dilation factor d = 4: skip three inputs, the receptive field is further expanded.

[0034] The embodiment of the application can obtain a very large receptive field in a very efficient way by stacking multiple layers of dilated convolution and exponentially increasing the dilation factor (for example, 1, 2, 4, 8, 16,... for each layer, respectively).

[0035] Because the bottom network (small dilation factor) of the multi-layer dilated convolution focuses on short-term and local patterns, such as small optical flow changes between adjacent frames and the beginning of a gesture, and the high-level network (large dilation factor) focuses on long-term and global patterns, such as a complete "basketball shooting" action sequence from dribbling, passing to the whole process of shooting. In this way, the time convolution network TCN can capture the time dependence in a video clip of several seconds to several minutes or even longer.

[0036] Using residual connections can stabilize the training of deep networks, and the time convolution network TCN borrows the idea of ResNet, and each block contains a residual connection inside. This means that information can be directly "jumped" from one layer to the next layer.

[0037] In addition, the time dimension is introduced into the three-dimensional convolutional neural network, the continuous frames of real-time video data are analyzed through the time dimension, the motion information in the real-time video data is captured, the robustness of the three-dimensional convolutional neural network to spatial changes is enhanced through the time translation invariance of the time dimension, and the behavior boundary and type in the event are determined through the context analysis of the continuous frames of the real-time video data.

[0038] In the technical scheme provided by the embodiment of the application, the three-dimensional convolutional neural network 3D CNN can expand the convolution kernel from 2D (height x width) to 3D (height x width x time). This 3D CNN kernel slides on a space-time cube stacked by continuous video frames. In the above manner, a convolution operation can extract features from multiple adjacent frames at the same time. The introduction of the three-dimensional convolutional neural network can improve the accuracy and robustness of behavior recognition and anomaly detection. Specifically: 1. Capture motion information and improve accuracy. This is the most direct advantage of 3D CNN. Behavior is essentially the change of posture in space and time. 3D CNN can determine the relative motion between objects by analyzing continuous frames, thereby identifying specific behavior actions. 3D CNN can directly learn the motion pattern itself.

[0039] 2. Robustness to spatial variations. 3D CNNs are naturally spatial-temporally translation-invariant. This means that as long as a motion pattern appears in the same form in space and time, regardless of where in the video it occurs, and from which frame it starts, a 3D CNN can identify it with high probability. Spatial translation invariance: inherited from 2D CNNs, a "wave" can be recognized regardless of whether it is on the left or right of the frame. Temporal translation invariance: the "wave" action can be captured by the model regardless of whether it starts from the 1st second or the 5th second. This improves the robustness to changes in the speed and rhythm of behavior execution.

[0040] 3. Reducing ambiguity with contextual information. 3D CNNs can effectively eliminate ambiguity by observing temporal context. By analyzing the dynamic changes within a short sequence, 3D CNNs can more accurately define the boundaries and categories of behaviors.

[0041] 4. Robustness to noise and occlusion. In complex scenes (such as crowded streets), target objects may be temporarily occluded, or the image quality of a certain frame may be blurred or too dark. 3D CNNs deal with a sequence of frames rather than isolated frames. If the information of a certain frame is lost due to occlusion or noise, the model can recover and complete the information from its previous and subsequent frames. Its prediction is based on the overall evidence of a spatio-temporal segment rather than a "snapshot" of a single instant, so it is more stable.

[0042] In addition, as a preferred embodiment, the real-time video analysis method described above, S112: model compression is performed on the spatio-temporal information processing model according to a model compression technique, and the model compressed spatio-temporal information processing model is introduced into a lightweight convolutional neural network architecture to obtain a lightweight deep learning inference engine, comprising: using pruning, quantization and knowledge distillation techniques to compress the spatio-temporal information processing model, and retaining the attention module and dynamic inference mechanism in the information processing model; building a lightweight convolutional neural network architecture on the edge, and introducing the model compressed spatio-temporal information processing model into the lightweight convolutional neural network architecture; wherein, introducing an attention module into the lightweight convolutional neural network architecture, and activating a dynamic inference mechanism according to the dynamic depth and dynamic width of the lightweight convolutional neural network architecture.

[0043] The technical scheme provided by the embodiments of the present application, in view of the calculation capability limitation of the edge device, adopts a series of model compression technologies such as network pruning (Pruning), quantization (Quantization) and knowledge distillation (Knowledge Distillation), greatly reduces the parameter quantity and calculation complexity of the model. A lightweight convolutional neural network architecture such as MobileNet and ShuffleNet can be further introduced to reduce the calculation resource requirement and improve the model running efficiency. Through hardware acceleration (GPU / TPU) and parallel computing strategy, the processing capability of real-time video analysis is enhanced to ensure fast inference and real-time response on the edge device. Through the above-mentioned manner, model lightweight and calculation optimization can be realized.

[0044] Figure 1 The technical scheme provided by the embodiments shown, after the step S110 of acquiring real-time video data and using a lightweight deep learning inference engine to analyze non-key events in the real-time video data at the edge, further comprises: S120: When the real-time video data involves a key event, the edge synchronizes the low-delay processed real-time video data to the cloud according to the cloud-edge collaborative optimization strategy.

[0045] The technical scheme provided by the embodiments of the present application, the cloud-edge collaborative optimization strategy includes a data synchronization mechanism and a data compression technology, specifically as Figure 3 As shown in a preferred embodiment, the step S120: When the real-time video data involves a key event, the edge synchronizes the low-delay processed real-time video data to the cloud according to the cloud-edge collaborative optimization strategy, comprises: S121: According to the hierarchical task scheduling and data transmission strategy, a data synchronization mechanism of the edge and the cloud is constructed; S122: Low-delay data is extracted from the real-time video data through a data compression technology; S123: The low-delay data is synchronized to the cloud through the data synchronization mechanism.

[0046] The hierarchical task scheduling and data transmission strategy can realize seamless collaboration between the edge and the cloud. When an event occurs, the edge device can compress the real-time video data through a data compression technology to obtain low-delay data (including behavior pattern recognition and audio signals, etc.), and transmit the low-delay data to the cloud in real time through a data synchronization mechanism. The cloud system is responsible for deep analysis of the low-delay data such as behavior pattern recognition and abnormal event prediction, and returns the processing result to generate a corresponding safety warning. The embodiments of the present application adopt an efficient data synchronization mechanism, data compression technology and communication protocol to reduce network bandwidth pressure and improve data processing reliability.

[0047] In addition, as a preferred embodiment, as shown in Figure 3 When the real-time video data involves a key event, the edge end synchronizes the low-delay processed real-time video data to the cloud end according to the cloud-edge collaborative optimization strategy before the real-time video data is synchronized to the cloud end, including: S124: At the edge end, the attention mechanism of the lightweight deep learning inference engine is used to capture the key frames and dynamic change information of the real-time video data.

[0048] S125: Through the pre-trained Seq2Seq model, the context vector is generated according to the time sequence characteristics of the real-time video data, and the context vector is converted into descriptive text.

[0049] S126: The key frames and dynamic change information and the descriptive text are synchronized to the cloud end through the cloud-edge collaborative optimization strategy.

[0050] Wherein, the core of the Seq2Seq model is that an encoder encodes a sequence into a fixed-size context vector, and then a decoder decodes the vector into another sequence. In the generation process of the video description text, the encoder is responsible for “understanding” the video content. Its input is the feature sequence of the video, and the output is one or more context vectors containing the overall information of the video. The decoder is responsible for “generating” the descriptive text. It generates natural language sentences word by word according to the context information provided by the encoder. The detailed workflow is as follows: First, video feature extraction: this is the most critical preparation step. Raw video pixels cannot be directly processed by the model, and the application needs to be converted into a more meaningful feature sequence. The embodiments of the application mainly have two methods: a) Feature based on 3D CNN: the purpose is to capture appearance and short-term temporal dynamics (motion information). 3D CNN: directly use the 3D CNN pre-trained on large video datasets (such as Kinetics). Divide the video into equal-length segments, input into the 3D CNN, and extract the feature vector of each segment. These vector sequences constitute the spatiotemporal feature representation of the video.

[0051] b) Encoding features based on RNN: the purpose is to pre-encode the feature sequence before inputting into the Seq2Seq encoder, in order to better fuse the time information. Specifically, the CNN feature sequence (for example, [f1, f2,...,fn]) extracted above is input into an RNN (such as LSTM or GRU). The last hidden state (or the weighted average of all hidden states) of the RNN can be used as a compact representation of the entire video. The final result is to obtain one or more feature vectors or feature sequences that can represent the video content. Second, the encoder understands the video in the classic Seq2Seq model, the encoder is usually an RNN. Its input: the video feature sequence V = [v1, v2,..., vn] obtained in the first step. The processing process is as follows: the encoder RNN (usually LSTM or GRU) processes each feature step by step and updates its internal hidden state. The output final hidden state: after processing the entire sequence, the final hidden state h_n of the encoder is regarded as a context vector representing the entire video content. All hidden states: in more advanced attention mechanisms, the hidden state [h1, h2,..., hn] of the encoder at each step is retained because they encode the information of different parts of the video.

[0052] Finally, the decoder generates text, the decoder is another RNN, whose task is a language model, responsible for generating words. The initial state includes: the initial hidden state s0 of the decoder is usually set to the final hidden state h_n of the encoder (i.e. the context vector). In this way, the decoder "knows" the video content when it starts generating.

[0053] The technical scheme provided by the embodiment of the application, the automatic description generation technology of video content: we will use the encoder-decoder architecture based on Transformer (i.e. the above Seq2Seq model), use the attention mechanism to capture the key frames and dynamic changes in the video, realize the mapping from visual data to natural language. By training the Seq2Seq model, the model will learn how to convert the visual features of the video into descriptive text, and generate coherent and information-rich language descriptions. In security monitoring, the automatic video description generation technology can automatically record the key events of the monitoring scene and summarize them in the form of text, which is convenient for quick browsing and post-analysis, thereby improving the readability and usability of monitoring data.

[0054] Figure 1 The technical scheme provided by the embodiment shown, in the above step S120: when the real-time video data involves a key event, the edge end synchronizes the low-delay processed real-time video data to the cloud end according to the cloud-edge collaborative optimization strategy, and then further comprises: S130: processing the key event in the real-time video data according to the natural language analysis combined with the video understanding strategy in the cloud end, and feeding back the processing result.

[0055] The cloud system is responsible for deep analysis such as behavior pattern recognition and abnormal event prediction, returns the processing result and generates the corresponding security warning.

[0056] Specifically, as a preferred embodiment, as shown in Figure 4As shown, the real-time video analysis method described above, step S130: according to natural language analysis combined with video understanding strategy, process the key events in real-time video data in the cloud, and feedback the processing results, including: S131: using automatic description generation technology, locating and identifying key frames and dynamic changes reflecting target behavior and scene information, and automatically recording and storing descriptive text corresponding to key events combined with target behavior and scene information.

[0057] S132: integrate speech recognition technology to convert query language into text form query instruction; combined with the semantic understanding ability of natural language processing model, use the query instruction to realize the deep semantic search of real-time video data; use vector space model and cosine similarity to quantify the similarity between text form query instruction and real-time video data, to improve the accuracy of deep semantic search.

[0058] S133: combined with sentiment analysis and social behavior recognition technology, analyze the voice text and visual signals in real-time video data, identify the emotional state and social behavior mode of the characters; use facial expression recognition and speech emotion recognition technology, combined with behavior analysis algorithm, to interpret human behavior in real-time video data.

[0059] In the technical scheme provided by the embodiments of the application, the language-based video retrieval and analysis system: research will integrate speech recognition technology to convert the oral query of security personnel into text, and combine the semantic understanding technology in NLP, such as dependency syntax analysis and entity recognition, to realize the deep semantic search of video content. In addition, vector space model (Vector Space Model) and cosine similarity (Cosine Similarity) are used to quantify the similarity between text and video content, to realize accurate video retrieval. Language-based video retrieval technology will allow security personnel to directly access relevant video clips through natural language query, greatly improving the efficiency of event tracking and evidence collection, especially when it is necessary to quickly locate specific events from a large amount of monitoring data. In addition, sentiment and social behavior analysis of video content: combined with sentiment analysis (Sentiment Analysis) and behavior recognition technology, research will develop a system that can identify the emotional state and social behavior mode of characters by analyzing the voice, text and visual signals in the video. Use facial expression recognition (Facial Expression Recognition) and speech emotion recognition (Speech Emotion Recognition) technology to interpret the emotional state of characters in real-time video data, and use behavior recognition technology to identify social behavior patterns such as hand gestures and body language. These technologies will provide security personnel with a deeper understanding of the events captured in the video, enabling them to make more informed decisions and improve the overall efficiency of their work. By combining the Emotion Recognition) technology with behavior analysis algorithms, the system will be able to interpret human behavior in videos more thoroughly. In security monitoring, emotion and social behavior analysis technology can help monitor the safety situation of public places in real time, identify tense, conflict or other abnormal emotional expressions in the video, and provide support for preventive measures.

[0060] Among them, as a preferred embodiment, the step S133: combining emotion analysis and social behavior recognition technology, analyzing the voice text and visual signals in the real-time video data, identifying the emotional state and social behavior mode of the person, includes: using an emotion analysis model to perform text emotion analysis on the voice text to obtain text emotion features, and analyzing the facial expression and behavior features of the video signal; using a multi-modal fusion technology to fuse the text emotion features with the facial expression and behavior features; using a graph neural network combined with the time sequence features of the real-time video data; according to the emotion state space vector predefined by the social behavior recognition technology, matching the text emotion features and the facial expression and behavior features according to the time sequence features, identifying the emotional state and social behavior mode of the person.

[0061] Since the information of a single modality is incomplete and prone to misjudgment. For example, a person can be smiling (vision), but the tone is cold (voice), and the content of the words is official (text); therefore, the embodiments of the present application need to accurately judge the true emotional state and social intention by fusing multiple signals and placing them in a specific social context. First, the voice is converted into text, and then text sentiment analysis is performed, including sentiment orientation, emotion category, and potential intention, etc. The above text sentiment analysis is mainly based on a Transformer model, such as BERT or RoBERTa, and performs sentiment analysis fine-tuning. And it can also analyze paralanguage features, such as pitch, speech rate, volume, and voice quality, etc., so as to realize the deep classification of language features. In addition, for visual signal analysis, the present application analyzes facial expressions and body posture and gesture analysis, which can encode and classify facial key points through a convolutional neural network (such as VGG, ResNet) or a special model (such as DAN) to determine macro expressions and micro expressions. It can also obtain body posture and gesture features through pose estimation algorithms. Finally, the above multiple modal features are fused through a multi-modal feature fusion strategy, including feature-level fusion, decision-level fusion, and model-level fusion, etc. Among them, the model-level fusion includes using a more advanced architecture (such as Transformer) to simulate cross-modal attention. For example, the system can learn to pay more attention to facial expressions and speech tone when a person says something, rather than the text content itself. The above model-level fusion can be realized by using a multi-modal Transformer and a cross-modal attention mechanism. In summary, the embodiments of the present application can dynamically and selectively weigh the importance of different modalities, improving the recognition performance of character emotional state and social behavior patterns.

[0062] In summary, the edge-cloud collaborative real-time video analysis method provided by the embodiments of the present application can not only obtain real-time video data at the edge, but also use a lightweight deep learning inference engine to determine whether the real-time video data involves a key event. If it is a non-key event, the lightweight deep learning inference engine at the edge is used to analyze the non-key event in the real-time video data. In this way, the real-time video data can be processed at the edge, reducing the algorithmic pressure of cloud video processing. When the real-time video data involves a key event, the edge synchronizes the low-latency processed real-time video data to the cloud according to the cloud-edge collaborative optimization strategy. In this way, through cloud-edge collaborative optimization, seamless collaboration between the edge and the cloud is ensured, and when an event occurs, the edge device transmits the key data to the cloud in real time through low-latency data transmission. The cloud system is responsible for deep analysis and returns the processing result, generating a security warning. In this way, through the cloud-edge collaborative optimization strategy, including efficient data synchronization mechanism, data compression technology and communication protocol, the network bandwidth pressure can be reduced and the reliability of data processing can be improved.

[0063] In addition, the following embodiments of this application provide product embodiments, the beneficial effects of which are the same as those of the edge-cloud collaborative real-time video analysis method provided in the above embodiments, and other technical features in the product embodiments are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0064] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device includes: A memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the program, implements a real-time video analytics method for edge-cloud collaboration as provided by any of the above technical solutions.

[0065] like Figure 5 As shown, the electronic device can include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 1002 or a program loaded from a storage device 1003 into a random access memory RAM 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 can operate the electronic device to exchange data with other devices via wireless or wired communication. Although the diagram shows a model building device with various systems, it should be understood that it is not required to implement or have all of the systems shown. It is possible to implement or have more or fewer systems alternatively.

Claims

1. An edge-cloud collaborative real-time video analytics method, characterized in that, The method comprises: acquiring real-time video data, and using a lightweight deep learning inference engine to analyze non-key events in the real-time video data at an edge side; when the real-time video data involves a key event, the edge side synchronizes low-delay processed real-time video data to a cloud side according to a cloud-edge collaborative optimization strategy; processing the key event in the real-time video data according to a natural language analysis combined with video understanding strategy at the cloud side, and feeding back a processing result.

2. The method of claim 1, wherein, The method of using a lightweight deep learning inference engine to analyze non-key events in the real-time video data at the edge side comprises: using a time series convolution network and a three-dimensional convolution neural network to construct a space-time information processing model at the edge side; model compression is performed on the space-time information processing model according to a model compression technique, the space-time information processing model after model compression is introduced into a lightweight convolution neural network architecture, and the lightweight deep learning inference engine is obtained; hardware acceleration and parallel processing are performed on the lightweight deep learning inference engine according to a hardware acceleration and parallel computing strategy, so as to enhance the analysis capability of the real-time video data.

3. The method of claim 2, wherein, The method of using a time series convolution network and a three-dimensional convolution neural network to construct a space-time information processing model comprises: the time series convolution network comprises causal convolution, multi-layer dilated convolution and residual connection; wherein, the input of the convolution kernel of the causal convolution is limited to the current time step and the previous time step to ensure the time series causal relationship; multi-layer dilated convolution is introduced into the time series convolution network, wherein the dilated factor of each dilated convolution layer is different, and the dilated factor is used to control the multi-layer dilated convolution to extract the time series features in the real-time video data; residual connection is used in the time series convolution network to connect the multi-layer dilated layers, so as to stabilize the training of the time series convolution network; a time dimension is introduced into the three-dimensional convolution neural network, the continuous frames of the real-time video data are analyzed through the time dimension, the motion information in the real-time video data is captured, the robustness of the three-dimensional convolution neural network to spatial changes is enhanced through the time translation invariance of the time dimension, and the behavior boundary and type in the event are determined through the context analysis of the continuous frames of the real-time video data.

4. The method of claim 2, wherein, The method of performing model compression on the space-time information processing model according to a model compression technique, and introducing the space-time information processing model after model compression into a lightweight convolution neural network architecture to obtain the lightweight deep learning inference engine comprises: model compression is performed on the space-time information processing model using pruning, quantization and knowledge distillation techniques, and the attention module and dynamic inference mechanism in the space-time information processing model are retained; a lightweight convolution neural network architecture is built at the edge side, and the space-time information processing model after model compression is introduced into the lightweight convolution neural network architecture; wherein, the attention module is introduced into the lightweight convolution neural network architecture, and the dynamic inference mechanism is activated according to the dynamic depth and dynamic width of the lightweight convolution neural network architecture.

5. The method of claim 1, wherein, When the real-time video data involves a key event, the edge end synchronizes the low-delay processed real-time video data to the cloud end according to a cloud-edge collaborative optimization strategy, including: According to the hierarchical task scheduling and data transmission strategy, the data synchronization mechanism of the edge end and the cloud end is constructed; Through data compression technology, low-delay data is extracted from the real-time video data; Through the data synchronization mechanism, the low-delay data is synchronized to the cloud end.

6. The method of claim 1, wherein, Before the edge end synchronizes the low-delay processed real-time video data to the cloud end according to the cloud-edge collaborative optimization strategy when the real-time video data involves a key event, including: At the edge end, the attention mechanism of the lightweight deep learning inference engine is used to capture the key frames and dynamic change information of the real-time video data; Through the pre-trained Seq2Seq model, the context vector is generated according to the time sequence characteristics of the real-time video data, and the context vector is converted into descriptive text; The key frames and dynamic change information and the descriptive text are synchronized to the cloud end through the cloud-edge collaborative optimization strategy.

7. The method of claim 6, wherein, At the cloud end, the key event in the real-time video data is processed according to the natural language analysis combined with the video understanding strategy, and the processing result is fed back, including: Using automatic description generation technology, the target behavior and scene information reflected by the key frames and dynamic changes are located and identified, and the key event corresponding to the descriptive text is automatically recorded and stored in combination with the target behavior and scene information; Integrating speech recognition technology, the query language is converted into a query instruction in the form of text; combined with the semantic understanding ability of the natural language processing model, the query instruction is used to realize the deep semantic search of the real-time video data; the vector space model and cosine similarity are used to quantify the similarity between the query instruction in the form of text and the real-time video data, so as to improve the accuracy of the deep semantic search; Combined with emotion analysis and social behavior recognition technology, the voice text and visual signal in the real-time video data are analyzed, the emotional state and social behavior mode of the characters are identified; using facial expression recognition and voice emotion recognition technology, combined with behavior analysis algorithm, the human behavior in the real-time video data is interpreted.

8. The method of claim 7, wherein, In combination with emotion analysis and social behavior recognition technology, the voice text and visual signal in the real-time video data are analyzed, the emotional state and social behavior mode of the characters are identified, including: Using emotion analysis model to perform text emotion analysis on the voice text to obtain text emotion features, and analyzing the facial expression and behavior features of the video signal; Using multi-modal fusion technology to fuse the text emotion features and the facial expression and behavior features; Using graph neural network combined with the time sequence characteristics of the real-time video data; According to the emotion state space vector predefined by the social behavior recognition technology, the text emotion features and facial expression and behavior features are matched according to the time sequence characteristics, and the emotional state and social behavior mode of the characters are identified.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and run on the processor, characterized in that, The processor executes the program to realize the edge-cloud collaborative real-time video analysis method of any one of claims 1 to 8.

10. A computer storage medium having stored thereon a computer program, characterized in that The computer program, when executed, implements the edge-cloud collaborative real-time video analytics method of any one of claims 1 to 8.

Images