Personnel behavior detection method based on multi-modal data and related equipment

Abstract

The embodiment of the invention provides a personnel behavior detection method based on multi-modal data and related equipment, and is applied to the technical field of image recognition processing. A to-be-processed video is acquired, and a to-be-processed image frame is extracted from the to-be-processed video. And extracting multi-modal data from the to-be-processed image frame by using a target detection algorithm, a target tracking algorithm and a human body posture estimation algorithm, wherein the multi-modal data at least comprises a personnel detection frame, a personnel image and a personnel posture feature map. And inputting the personnel detection frame, the personnel image and the personnel posture feature map into a pre-constructed abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model. Image data is obtained from the video data, the personnel action has continuity, and the image data is close to an actual website scene; a multi-modal data fusion abnormal behavior detection model is used to make up for the data limitation and fusion suitability among the personnel image, the detection frame and the attitude feature, so as to improve the behavior abnormality recognition accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition and processing, and in particular to a method for detecting human behavior based on multimodal data and related equipment. Background Art

[0002] Abnormal behavior detection is applied to all walks of life to ensure the safety of industry services. For example, in video surveillance and security scenarios, in areas covered by surveillance cameras, abnormal behavior detection can promptly detect abnormal events, such as intrusion by outsiders and bad behavior, thereby strengthening security prevention and management; in elderly care scenarios, hospitals, nursing homes and other places use abnormal behavior detection technology to detect the activities of the elderly, such as falls, long periods of immobility, etc., and provide timely assistance; in traffic safety scenarios, abnormal behavior detection technology can be used to detect the driver's behavior and determine whether the driver is fatigued driving, illegal operations, etc., to reduce the occurrence of traffic accidents. It can be seen that abnormal behavior detection can be applied to various scenarios, aiming to promptly discover and respond to potential security risks. Similarly, in scenarios such as banks and financial institutions, it is crucial to use abnormal behavior detection technology to detect abnormal behavior of personnel in outlets in real time. In terms of financial security, it helps to protect the asset security of other customers and prevent bad incidents; in terms of customer safety, it can promptly alert branch staff of abnormal situations, such as fighting or customer fainting, thereby protecting customers' personal safety in real time.

[0003] Currently, a dual-camera method for identifying abnormal behavior of bank insiders is used to detect abnormal behavior. The camera shoots the movements of people at eye level, and data collection has limitations and cannot fit the actual application scenarios. The two required images obtained by dual cameras separately cannot show the continuity of the behavior. The feature processing of a single image discards the correlation information between the previous and next frame images, limiting the ability to understand and analyze objects in dynamic scenes.

[0004] You can also use human skeleton features or image data for abnormal behavior detection. Although single-modal data is relatively simple and efficient to process, the feature information that can be extracted from single-modal data is limited and cannot provide comprehensive feature representation, which limits the generalization ability of the model. At the same time, single-modal data is not enough to support decision-making and judgment of complex tasks. The data is affected by noise and abnormal changes, which can easily lead to unstable detection results.

[0005] Another method for detecting abnormal behavior based on multi-feature fusion can be used. However, when fusing human posture features and human facial features, this method simply concatenates them by dimension and uses a basic residual network for final classification, without considering the adaptability of fusion in different periods and in different ways.

[0006] It can be seen that the currently used behavior anomaly detection methods have the problem of low accuracy in identifying behavior anomalies and are unable to accurately identify abnormal behaviors of people. Summary of the invention

[0007] In view of this, the present application provides a personnel behavior detection method and related equipment based on multimodal data, which obtains image data from video data, the personnel movements are continuous, and the image data is close to the actual network scene; an abnormal behavior detection model using multimodal data fusion is used to make up for the data limitations and fusion adaptability between personnel images, detection frames and posture features, so as to improve the accuracy of behavior abnormality recognition.

[0008] To solve the above problems, the technical solutions provided by this application are as follows:

[0009] In a first aspect, an embodiment of the present application provides a method for detecting human behavior based on multimodal data, the method comprising:

[0010] In response to acquiring a video to be processed, extracting an image frame to be processed from the video to be processed;

[0011] Extracting multimodal data from the image frame to be processed by using a target detection algorithm, a target tracking algorithm, and a human posture estimation algorithm, wherein the multimodal data at least includes a person detection frame, a person image, and a person posture feature map;

[0012] The person detection frame, the person image and the person posture feature map are input into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model.

[0013] In a possible implementation, the abnormal behavior detection model includes a spatiotemporal graph convolutional network, a gated recurrent neural network, and an image feature extraction network;

[0014] The step of inputting the person detection frame, the person image, and the person posture feature map into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model includes:

[0015] Extracting posture features from the personnel posture feature graph using the spatiotemporal graph convolutional network;

[0016] Extracting detection frame features from the person detection frame using the gated recurrent neural network;

[0017] Splicing the posture feature and the detection frame feature to obtain an intermediate fusion feature;

[0018] extracting image features from the person image using the image feature extraction network;

[0019] Splicing the intermediate fusion feature and the image feature to obtain a target fusion feature;

[0020] The behavior detection result is obtained based on the target fusion feature.

[0021] In a possible implementation, the extracting detection frame features from the person detection frame by using the gated recurrent neural network includes:

[0022] Determining the temporal characteristics of the person detection frame using the gated recurrent neural network;

[0023] The detection frame features are obtained by performing correlation calculation through a temporal attention mechanism.

[0024] In a possible implementation, the image feature extraction network includes a three-dimensional bottleneck network and a time displacement attention module, and the step of extracting image features from the person image using the image feature extraction network includes:

[0025] Determining initial image features of the person image using the three-dimensional bottleneck module;

[0026] Inputting the initial image features into the time permutation attention module, the time permutation attention module divides the initial image features into a plurality of groups according to image channels, each group having at least one sub-feature;

[0027] The sub-features are used as input features of the first branch channel, the spatial attention mechanism, and the second branch temporal attention mechanism, respectively, to determine the output features of the first branch and the output features of the second branch;

[0028] The output features of the first branch and the output features of the second branch are added and fused, and the feature channels are shuffled and aggregated across groups to obtain the image features.

[0029] In a possible implementation manner, the step of concatenating the intermediate fusion feature and the image feature to obtain the target fusion feature includes:

[0030] The intermediate fusion features and the image features are concatenated as input features of the modal attention mechanism to obtain the target fusion features output by the modal attention mechanism.

[0031] In a possible implementation, the target detection algorithm, the target tracking algorithm and the human posture estimation algorithm are used to extract multimodal data from the image frame to be processed, and the multimodal data includes at least a person detection frame, a person image and a person posture feature map, including:

[0032] Extracting the person detection frame from the image frame to be processed by using a target detection algorithm, wherein the target detection algorithm is a single-stage detection model Yolov5;

[0033] Extracting the person image from the image frame to be processed by using the target tracking algorithm, wherein the target tracking algorithm is a multi-target tracking model DeepSORT;

[0034] The human body posture estimation algorithm is used to extract a person posture feature map from the image frame to be processed, and the human body posture estimation algorithm includes an OpenPose algorithm.

[0035] In a possible implementation, the personnel behavior detection method further includes:

[0036] If the behavior detection result is abnormal behavior, an alarm is triggered.

[0037] In a second aspect, an embodiment of the present application provides a personnel behavior detection device based on multimodal data, the personnel behavior detection device comprising:

[0038] An extraction module, configured to extract an image frame to be processed from the video to be processed in response to obtaining the video to be processed;

[0039] The extraction module is further used to extract multimodal data from the image frame to be processed by using a target detection algorithm, a target tracking algorithm and a human posture estimation algorithm, wherein the multimodal data at least includes a person detection frame, a person image and a person posture feature map;

[0040] The acquisition module is used to input the person detection frame, the person image and the person posture feature map into a pre-built abnormal behavior detection model to obtain the behavior detection result output by the abnormal behavior detection model.

[0041] In a possible implementation, the abnormal behavior detection model includes a spatiotemporal graph convolutional network, a gated recurrent neural network, and an image feature extraction network;

[0042] The acquisition module comprises:

[0043] An extraction submodule, used for extracting posture features from the personnel posture feature graph using the spatiotemporal graph convolutional network;

[0044] The extraction submodule is further used to extract detection frame features from the person detection frame using the gated recurrent neural network;

[0045] A splicing submodule, used for splicing the posture feature and the detection frame feature to obtain an intermediate fusion feature;

[0046] The extraction submodule is further used to extract image features from the person image using the image feature extraction network;

[0047] The splicing submodule is further used to splice the intermediate fusion feature and the image feature to obtain the target fusion feature;

[0048] The acquisition submodule is used to obtain the behavior detection result based on the target fusion feature.

[0049] In a possible implementation, the extraction submodule is further used to extract detection frame features from the person detection frame using the gated recurrent neural network, including:

[0050] A determination unit, configured to determine the temporal features of the person detection frame using the gated recurrent neural network;

[0051] The acquisition unit is used to perform correlation calculation through a temporal attention mechanism to obtain the detection frame feature.

[0052] In a possible implementation, the image feature extraction network includes a three-dimensional bottleneck network and a time displacement attention module, and the extraction submodule is further used to extract image features from the person image using the image feature extraction network, including:

[0053] a determining unit, configured to determine initial image features of the person image using the three-dimensional bottleneck module;

[0054] a dividing unit, configured to input the initial image features into the time permutation attention module, wherein the time permutation attention module divides the initial image features into a plurality of groups according to image channels, each group having at least one sub-feature;

[0055] The determination unit is further used to use the sub-features as input features of the first branch channel, the spatial attention mechanism, and the second branch temporal attention mechanism, respectively, to determine the output features of the first branch and the output features of the second branch;

[0056] A processing unit is used to perform addition and fusion processing on the output features of the first branch and the output features of the second branch and to perform cross-group aggregation processing on the disrupted feature channels to obtain the image features.

[0057] In a possible implementation, the splicing submodule is further used to splice the intermediate fusion feature and the image feature to obtain a target fusion feature, including:

[0058] A splicing unit is used to splice the intermediate fusion feature and the image feature as input features of the modal attention mechanism to obtain the target fusion feature output by the modal attention mechanism.

[0059] In a possible implementation, the extraction module is further used to extract multimodal data from the image frame to be processed using a target detection algorithm, a target tracking algorithm, and a human posture estimation algorithm, wherein the multimodal data includes at least a person detection frame, a person image, and a person posture feature map, including:

[0060] An extraction submodule, used for extracting the person detection frame from the image frame to be processed by using a target detection algorithm, wherein the target detection algorithm is a single-stage detection model Yolov5;

[0061] The extraction submodule is further used to extract the person image from the image frame to be processed by using the target tracking algorithm, and the target tracking algorithm is a multi-target tracking model DeepSORT;

[0062] The extraction submodule is further used to extract a person posture feature map from the image frame to be processed by using the human posture estimation algorithm, and the human posture estimation algorithm includes an OpenPose algorithm.

[0063] In a possible implementation, the personnel behavior detection method further includes:

[0064] The alarm module is used to trigger an alarm if the behavior detection result is abnormal behavior.

[0065] In a third aspect, an embodiment of the present application provides a human behavior detection device based on multimodal data, the device comprising: a processor, a memory, and a system bus;

[0066] The processor and the memory are connected via the system bus;

[0067] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes the personnel behavior detection method based on multimodal data described in the first aspect above.

[0068] In a fourth aspect, an embodiment of the present application provides a storage medium, which stores instructions. When the instructions are executed by a computer, the computer implements the personnel behavior detection method based on multimodal data described in the first aspect above.

[0069] It can be seen that this application has the following beneficial effects:

[0070] The embodiment of the present application provides a method for detecting personnel behavior based on multimodal data. First, a video to be processed is obtained, and an image frame to be processed is extracted from the video to be processed. Secondly, a target detection algorithm, a target tracking algorithm, and a human posture estimation algorithm are used to extract multimodal data from the image frame to be processed, and the multimodal data at least includes a personnel detection frame, a personnel image, and a personnel posture feature map. Finally, the personnel detection frame, the personnel image, and the personnel posture feature map are input into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model. In this way, image data is obtained from video data, the personnel movements are continuous, and the image data is close to the actual network scene; the abnormal behavior detection model fused with multimodal data is used to make up for the data limitations and fusion adaptability between the personnel image, the detection frame, and the posture features, so as to improve the accuracy of abnormal behavior recognition.

[0071] The embodiment of the present application also provides a device corresponding to the above method, which has the same beneficial effects as the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention, and for ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0073] Figure 1 A schematic diagram of a process of identifying abnormal behavior based on images taken by dual cameras in the prior art;

[0074] Figure 2 A flowchart of a method for detecting human behavior based on multimodal data provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of the positions of characteristic points of a human body posture provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of another method for detecting human behavior based on multimodal data provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of a three-dimensional bottleneck module structure provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of the structure of a time displacement attention module provided in an embodiment of the present application;

[0079] Figure 7A schematic diagram of the structure of a human behavior detection device based on multimodal data provided in an embodiment of the present application;

[0080] Figure 8 A schematic diagram of the structure of a personnel behavior detection device based on multimodal data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0082] In this application, the terms "comprises", "comprising" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0083] In the early days, in order to detect abnormal behavior, manual monitoring videos were used. However, the working time of monitoring personnel was limited, and they could not work with high concentration for a long time, resulting in low control efficiency. At the same time, manual monitoring may be affected by personal bias and subjective factors, resulting in biased monitoring results. In order to solve the shortcomings of manual monitoring, traditional machine learning algorithms can perform feature processing on video images, and then the classifier will judge abnormal behavior. This type of machine learning algorithm usually uses features such as corner points and textures to train classification models such as support vector machines and hidden Markov. However, this type of model is only suitable for simple scenes, has poor adaptability to complex scenes, and does not have good versatility. With the development of deep learning, neural networks can mine deeper features of images, not only appearance features, but also motion features. Therefore, deep learning has more advantages than traditional machine learning methods in processing image features. At the same time, the increase in model parameters and training data sets is also more conducive to improving the accuracy and versatility of neural networks.

[0084] At present, patent document CN116386145 records a method for identifying abnormal behavior of bank internal personnel based on dual cameras, see Figure 1 , using two cameras with different viewing angles to shoot people in the bank. The two viewing angles get images With image Where C is the number of image channels, and H and W are the height and width of the image, respectively. Image A is convolved through the pose mask module to extract high-dimensional image features. Among them, the pose encoding module is used to mask the non-human image part, that is, to mask the environmental data in the image to avoid excessive memory consumption during the model calculation process, which affects the speed. The pose encoding module consists of a slicing operation and two convolution operations. The sizes of the two convolution kernels are both 3×3. The two convolutions respectively extract the coarse mask information and fine mask information of the image, and finally obtain the pose mask feature. The pose encoding module that processes image B extracts features through a traditional residual network to obtain pose encoding features.

[0085] In order to extract the interactive feature information under two visual conditions, feature fusion is performed through the pose interaction module. The pose interaction module contains a spatial perception interaction attention mechanism and a multi-layer perceptron. The spatial perception matrix is ​​set to F, the pose mask feature is set to Q, and the pose encoding feature is set to K and V. Then, the spatial perception feature N can be obtained through formula (1).

[0086] N=F(QK T ) (1)

[0087] Then the pose encoding feature V and the spatial perception feature N are calculated to obtain the final spatial perception interaction feature M. For the specific calculation, please refer to formula (2).

[0088]

[0089] Among them, Softmax(·) is the activation function, D h It is a constant whose value is 256.

[0090] Finally, the spatial perception interaction feature M is passed through a multi-layer perceptron as the input of the human posture prediction module and the human action prediction module. The human posture prediction module will output the specific location of the key points of the human posture in the image, while the human action prediction module will output the predicted probability values ​​of various actions of the characters in the image.

[0091] The above-mentioned prior art uses images taken by dual cameras to identify abnormal behavior of bank personnel, see Figure 1 It can be seen that the dual cameras shoot at the perspective of looking at people, which makes data collection very limited. Data collection cannot fit the actual application scenario. Shooting from top to bottom at a bird's-eye view is more in line with the actual application scenario, and then identifying and detecting each person in the captured image one by one. At the same time, behavior should be a continuous action, not a single picture. The analysis of continuous video frames is obviously better than the detection of a single image. The feature processing of a single image discards the correlation information between the previous and next frame images, which limits the ability to understand and analyze objects in dynamic scenes.

[0092] In other existing technologies, such as CN111582092 and CN116884087, human skeleton features and image data are used for abnormal behavior detection respectively. Although single-modal data is relatively simple and efficient to process, the feature information that can be extracted from single-modal data is limited and cannot provide comprehensive feature representation, which limits the generalization ability of the model. At the same time, single-modal data is not enough to support the decision-making and judgment of complex tasks, and the data is affected by noise and abnormal changes, which can easily lead to unstable detection results. For example, CN111523559 adopts an abnormal behavior detection method based on multi-feature fusion, but when fusing human posture features and human facial features, this method simply splices them by dimension and uses the basic residual network for final classification, without considering the fusion adaptability of different periods and different methods.

[0093] It can be seen that the currently used behavior anomaly detection methods have the problem of low accuracy in identifying behavior anomalies and are unable to accurately identify abnormal behaviors of people.

[0094] In summary, the embodiments of the present application provide a personnel behavior detection method based on multimodal data and related equipment. First, a video to be processed is obtained, and an image frame to be processed is extracted from the video to be processed. Secondly, a target detection algorithm, a target tracking algorithm, and a human posture estimation algorithm are used to extract multimodal data from the image frame to be processed, and the multimodal data includes at least a personnel detection frame, a personnel image, and a personnel posture feature map. Finally, the personnel detection frame, the personnel image, and the personnel posture feature map are input into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model. In this way, image data is obtained from video data, the personnel movements are continuous, and the image data is close to the actual network scene; the abnormal behavior detection model fused with multimodal data is used to make up for the data limitations and fusion adaptability between the personnel image, the detection frame, and the posture features, so as to improve the accuracy of abnormal behavior recognition.

[0095] In order to facilitate understanding of the technical solution provided by the embodiment of the present application, the personnel behavior detection method based on multimodal data and related equipment provided by the embodiment of the present application are described below in conjunction with the accompanying drawings.

[0096] Abnormal behavior detection can be specifically divided into the following four stages: data collection and preprocessing, target detection and tracking, behavior feature extraction, abnormal behavior identification and classification. The data collection and preprocessing stage mainly collects personnel data in the network point through videos or images, and can preliminarily perform preprocessing operations such as cleaning, denoising, and cropping on the data. The target detection and tracking stage uses the target detection algorithm to identify the personnel in the video and determine their specific position in the video image. The target tracking algorithm is used to track the movement trajectory of the personnel in the video sequence to ensure the continuity of the personnel trajectory. The behavior feature extraction stage is to use the computer vision and machine learning algorithms to fuse the features extracted from the above two stages to extract higher-dimensional abstract features. The abnormal behavior identification and classification stage is to extract and classify the high-dimensional abstract features and obtain the probability information of various abnormal behaviors of the personnel in the network point.

[0097] See also Figure 2 , Figure 2 A flow chart of a method for detecting human behavior based on multimodal data provided in an embodiment of the present application. The method for detecting human behavior can be applied to a server or a system, which is not limited in the embodiment of the present application and can be selected according to actual needs. The method for detecting human behavior specifically includes S201-S203.

[0098] First, some nouns appearing in the embodiments of the present application are explained.

[0099] Object detection is the process of identifying and locating specific objects in images or videos. The object detection algorithm is the algorithm used in the aforementioned positioning process.

[0100] Target tracking is to track the position and trajectory of the target in the video sequence and obtain the motion information of the target between consecutive frames.

[0101] Human pose estimation is the process of inferring the pose and motion information of a human body from images or videos.

[0102] Multimodal data is multiple types of data from different data sources, such as video, image, sound, text, etc.

[0103] Spatio-Temporal Graph Convolutional Networks (STGCN) is a deep learning network that can be used to process image spatiotemporal data. It combines the characteristics of Graph Convolutional Networks (GCN) and Spatio-Temporal Convolutional Networks, and can capture the characteristics of data in both time and space. Graph Neural Network is a neural network model used to process graph structured data. It can effectively learn the relationship and attributes between graph nodes. It can be implemented through graph convolutional networks, emphasizing the application of convolution operations on graph data. Gated Recurrent Neural Network (GRNN) is a neural network model used to process time series data. It controls the flow of information through gating units. Convolutional Neural Networks (CNN) is a neural network model used to process image data, usually composed of convolutional layers, pooling layers and fully connected layers. Attention mechanism is a technology used to enhance the performance of neural networks, mainly used to dynamically adjust the model's attention to different parts.

[0104] In order to detect abnormal behavior of personnel in bank branches more efficiently and accurately, optimization is carried out in terms of input data and feature processing.

[0105] S201: In response to obtaining a video to be processed, extracting an image frame to be processed from the video to be processed.

[0106] The overhead video shot by the daily surveillance camera is selected as the input data of the overall method, that is, the surveillance video can be obtained in real time, and the surveillance video is the video to be processed.

[0107] Select T consecutive frames from the video to be processed to replace a single frame. The consecutive frames will provide richer and more comprehensive action information, which is more conducive to the analysis and understanding of people in dynamic scenes such as behavior detection. Where T≥2, T is a positive integer.

[0108] S202: extracting multimodal data from the image frame to be processed using a target detection algorithm, a target tracking algorithm, and a human posture estimation algorithm, wherein the multimodal data at least includes a person detection frame, a person image, and a person posture feature map.

[0109] After obtaining the video to be processed in the bank branch, it is necessary to perform real-time and accurate object analysis on the video to be processed in order to determine the exact location of the person in the video.

[0110] The multimodal data at least includes a person detection frame, a person image and a person posture feature map. In a possible implementation method, the multimodal data is extracted from the image frame to be processed using a target detection algorithm, a target tracking algorithm and a human posture estimation algorithm, including: extracting the person detection frame from the image frame to be processed using a target detection algorithm, and the target detection algorithm is a single-stage detection model (You Only Look Once v5, Yolov5); extracting the person image from the image frame to be processed using the target tracking algorithm, and the target tracking algorithm is a multi-target tracking model (Deep Simple Online and Realtime Tracking, DeepSORT); extracting the person posture feature map from the image frame to be processed using the human posture estimation algorithm, and the human posture estimation algorithm includes an OpenPose algorithm.

[0111] First, the target detection algorithm can be used to determine the position of the person in each image frame to be processed, that is, the output vector [T, x, y, w, h], where x and y represent the position of the center point of the detected person, and w and h represent the width and height of the detection frame respectively. Then, the target tracking algorithm is used to perform more detailed positioning and tracking of all detected persons in the continuous image frames to be processed to obtain the person image.

[0112] In terms of the selection of target detection algorithm and target tracking algorithm, the embodiment of the present application selected the single-stage detection model Yolov5 and the multi-target tracking model DeepSORT. In order to quickly identify and track people in the video, the single-stage detection Yolov5 model has higher detection efficiency than the two-stage model while maintaining high accuracy; and the DeepSORT target tracking model integrates the advantages of deep learning on the basis of traditional tracking methods, making the target tracking results more real-time and robust.

[0113] In order to enable the anomaly detection model to understand the human movements from multiple angles, after obtaining the human image and the human detection frame, the OpenPose algorithm is used to extract 18 human posture feature points to obtain the corresponding human posture feature map. The specific positions of the human posture feature points are as follows: Figure 3 shown.

[0114] S203: Inputting the person detection frame, the person image, and the person posture feature map into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model.

[0115] Multimodal data is used as the input of the abnormal behavior detection model. The multimodal data includes person detection frames, person images, and person posture feature maps. Among them, the person posture feature map has richer information than the common image input, see Figure 3 As shown in the figure, the human posture feature points contain information such as the position and direction of important joints such as hands and legs. These important joints can provide richer motion information and spatial relationships. At the same time, as a higher-level abstract representation, the human posture feature map can express the meaning and difference between different actions, which helps the abnormal behavior detection model to accurately understand human actions and improve the robustness and interpretability of the model.

[0116] See also Figure 4 , Figure 4 Another flow chart of a method for detecting human behavior based on multimodal data provided in an embodiment of the present application. The abnormal behavior detection model includes a spatiotemporal graph convolutional network, a gated recurrent neural network, and an image feature extraction network, which are used to process human posture feature graphs, human detection frames, and human images, respectively.

[0117] In one possible implementation, a person detection frame, a person image, and a person posture feature map are input into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model, including: extracting posture features from the person posture feature map using a spatiotemporal graph convolutional network; extracting detection frame features from the person detection frame using a gated recurrent neural network; splicing posture features and the detection frame features to obtain intermediate fusion features; extracting image features from the person image using an image feature extraction network; splicing intermediate fusion features and image features to obtain target fusion features; and obtaining a behavior detection result based on the target fusion features.

[0118] The embodiment of the present application uses a spatiotemporal graph convolutional network to process the human posture feature graph. The human posture feature points can be regarded as a graph structure. The connection relationship between the joints makes the joints affect each other. The complex multi-node relationship processing can use a graph neural network. The spatiotemporal graph convolutional network is mainly composed of 9 spatiotemporal graph convolution submodules, each of which is composed of a graph convolutional network and a spatiotemporal convolutional network. The graph convolutional network and the spatiotemporal convolutional network respectively extract the spatial features and temporal features in the human posture feature graph.

[0119] Assume that the input of the human posture feature graph is T frames, and there are N human posture feature points in each frame. Define the spatiotemporal graph as G = (V, E), where the node set V = {v ti |t=1,...T,i=1,...,N}, the i-th node feature of the t-th frame is defined as F(v ti ), including node coordinates and confidence. E is an edge set, which defines the spatial edges E between feature points. s = {v ti v tj |(i,j)∈H} and the time edge E F = {v ti v(t+1)i}. Where H represents the connection relationship of the human body joints.

[0120] The calculation process of spatial graph convolution is shown in the following formula (3).

[0121]

[0122] Among them, f out (v ti ) is the output feature of the i-th node in the t-th frame; f in (v ti ) is the input feature of the ith node in the tth frame; ρ(·) and ω(·) represent the sampling function and weight function respectively. In common image convolution, since pixels are arranged in a regular order, but the graph structure is different from the arrangement of pixels, it is necessary to divide the nodes in the graph into subsets before performing the convolution operation. The division of the subset is represented by l ti :N(v ti )→{0,...,K-1}, where N(·) represents other nodes and node v ti The distance between Z and K represents the convolution kernel size of the graph convolution. ti (v tj )=|{v tk |l ti (v tk )=l ti (v tj )}| represents the normalization of the relevant subset. Finally, based on the spatial convolution, time is introduced to define the adjacent graph nodes in space and time. The time length is defined as Γ, and the specific calculation of other nodes and node v ti The distance N under the combined action of space and time ST (v ti ) is shown in the following formula (4).

[0123]

[0124] In the feature extraction of the person detection frame, the detection frame data is related to the time series, but the detection frame data structure is relatively simple. Based on this, the embodiment of the present application can select a simple and efficient gated recurrent neural network to extract the detection frame features. Compared with the long short-term memory network, the gated recurrent neural network has a simpler network structure, a more efficient calculation process, and is more suitable for constructing simple time series data.

[0125] In one possible implementation, a gated recurrent neural network is used to extract detection frame features from a person detection frame, including: using a gated recurrent neural network to determine the temporal features of the person detection frame, performing correlation calculations through a temporal attention mechanism, and obtaining the detection frame features.

[0126] After the gated recurrent neural network processing, the temporal feature h of the person detection frame is obtained. b ={h b1 ,h b2 ,...,h be}, in order to further extract the feature relationship between each time series, the correlation calculation can be performed through the time attention mechanism. The time series feature h of the last frame be As a key frame, calculate the key frame h be With other frames bi The specific calculation is shown in formula (5).

[0127]

[0128] Among them, score(·) is the scoring function, and the specific calculation method is: W s is the weight matrix that the network needs to learn, h bi is the time series feature of the past moment. Calculate the global time series feature information h bc =∑ i α i h bi , the two features h bc With h be The splicing is performed and the activation function tanh(·) is used as the final output of the temporal attention. Finally, the posture feature and the detection frame feature are spliced ​​as the input feature of the modal attention mechanism. The calculation process of the modal attention mechanism is consistent with the calculation process of the temporal attention mechanism. The key modal feature selects the posture feature for calculation, and the length of the feature vector is 128. Because of the existence of the modal attention mechanism, as long as the final output feature vector length of any modal data is 128, it can be freely inserted into the anomaly detection model. Similarly, the modal data of the model can also be freely deleted without affecting the feature extraction structure of other modal data.

[0129] For the feature processing of personnel images, take the 3D-MobileNetV3 network input as an example. The 3D-MobileNetV3 network structure is shown in Table 1 below. The image feature extraction network includes a three-dimensional bottleneck network (3D-MobileNetV3) and a time displacement attention module.

[0130]

[0131] Table 1

[0132] The length and width of the image frame to be processed are both 224, and the number of channels is 3. The embodiment of the present application relates to feature extraction of continuous multi-frame images, and the time dimension can be set to 16, which is not limited and can be selected according to actual needs.

[0133] After determining the posture features and the detection frame features, the two features are concatenated to obtain an intermediate fusion feature. After being processed by the modal attention mechanism, the intermediate fusion feature can be concatenated with the image feature.

[0134] The original MobileNetV3 network, as a two-dimensional convolutional network, cannot process three-dimensional continuous frame image information. Therefore, the embodiment of the present application optimizes the original MobileNetV3 network and expands it into a three-dimensional version, namely, the 3D-MobileNetV3 network. The specific network structure of the 3D-MobileNetV3 network is shown in Table 1. Among them, HS and RE represent the Hardswish activation function and the ReLU activation function respectively. The core of the optimized three-dimensional MobilNetV3 network is the three-dimensional bottleneck module, see Figure 5 , Figure 5 A schematic diagram of a three-dimensional bottleneck module structure provided in an embodiment of the present application.

[0135] The three-dimensional bottleneck module is mainly composed of depthwise separable convolution and time-permuted attention module. Among them, the depthwise separable convolution divides the ordinary convolution process into two steps: three-dimensional depthwise convolution and three-dimensional pointwise convolution. The three-dimensional depthwise convolution performs independent convolution operations on each channel of the input feature, while the three-dimensional pointwise convolution uses a 1×1×1 convolution kernel to fuse the feature information of different channels. The first three-dimensional pointwise convolution expands the channel of the input feature, and the second three-dimensional pointwise convolution then performs dimensionality reduction connection on the channel to form an inverted residual structure. Compared with ordinary convolution, the depthwise separable convolution can effectively reduce the number of parameters and calculations of the model, thereby improving the computational efficiency and performance of the model.

[0136] The original bottleneck module uses a channel attention mechanism, which can only capture feature information on the channel and cannot effectively utilize spatial and temporal information. This embodiment of the application introduces a permutation attention module and introduces a temporal attention mechanism based on the module. Figure 6 , Figure 6 A schematic diagram of the structure of a time displacement attention module provided in an embodiment of the present application.

[0137] After processing the person image through the three-dimensional bottleneck module, the initial image features can be obtained, and then the required image features can be obtained based on the initial image features through the time displacement attention module.

[0138] In one possible implementation, an image feature extraction network is used to extract image features from a person image, including:

[0139] A three-dimensional bottleneck module is used to determine the initial image features of the person image; the initial image features are input into the time permutation attention module, which divides the initial image features into several groups according to the image channels, and each group has at least one sub-feature; the sub-features are used as input features of the first branch channel, the spatial attention mechanism and the second branch temporal attention mechanism respectively to determine the output features of the first branch and the output features of the second branch; the output features of the first branch and the output features of the second branch are added and fused, and the feature channels are shuffled and aggregated across groups to obtain image features.

[0140] The temporal permutation attention module mainly consists of three steps: feature grouping, mixed attention and feature aggregation. The first step of feature grouping is to divide the input feature X into G groups according to the channel. Each sub-feature X k ∈R C / G×T×H×W In the second step, the sub-features are used as input features of the first branch channel, spatial attention mechanism and the second branch temporal attention mechanism. In the channel and spatial attention mechanisms, the sub-feature X k Will be further grouped into X k1 ,X k2 ∈R C / 2G×T×H×W The specific calculation process of channel attention is shown in formula (6) and formula (7), and the specific calculation process of spatial attention is shown in formula (8).

[0141]

[0142] X′ k2 =σ(W2·GN(X k2 )+b2)·X k2 (8)

[0143] Among them, F gp (·) is the average pooling function, GN(·) is the group normalization operation, σ(·) is the Sigmoid activation function, W1, W2, b1 and b2 are the weight and bias parameters that need to be trained respectively.

[0144] After feature concatenation, the feature output of the first branch is X′ k =[X′ k1 ,X′ k2 ], the dimensions of the input feature and the output feature are the same. The second branch is for the input feature X k To calculate the temporal correlation, firstly generate the feature vector z∈R through three-dimensional average pooling 1×1×1×T , this feature vector reflects the importance of each time dimension, and the specific calculation is shown in formula (9).

[0145]

[0146] The second step is to use two fully connected layers to weight the correlation between the time dimensions. The specific calculation is shown in formula (10). The role of the hyperparameter r is to expand the middle fully connected layer to r times the original size and then shrink it to r times to avoid inconsistent dimensions in the final calculation.

[0147]

[0148] The output feature X′ of the first branch k With the output feature X of the second branch k The addition and fusion are performed, and then the feature channels are shuffled for cross-group aggregation as the output features of the time permutation attention module, that is, the image features.

[0149] In one possible implementation, the intermediate fusion features and the image features are concatenated to obtain the target fusion features, including: concatenating the intermediate fusion features and the image features as input features of the modal attention mechanism to obtain the target fusion features output by the modal attention mechanism.

[0150] The image features of the image feature extraction network and the intermediate fusion features of the non-image extraction network are spliced ​​as the input features of the modal attention mechanism, and the abnormal behavior detection results of personnel are output through the fully connected layer.

[0151] In one possible implementation, the abnormal behavior detection model may output the probability of abnormal behavior. If the probability of abnormal behavior is higher than a preset probability threshold, the behavior detection result is determined to be abnormal, and an alarm is triggered.

[0152] The video data captured by the cameras in the outlets are analyzed in real time, and the target detection and target tracking methods are used to locate the position of the personnel in the video image. The human posture estimation method is used to extract key point information and analyze the personnel movements. Finally, the extracted multimodal data is fused through the abnormal behavior detection model to identify the abnormal behavior of the personnel in the video, thus realizing the real-time identification of abnormal behavior of the personnel in the outlets.

[0153] Based on the above embodiment, the video data of the personnel in the network point is obtained and analyzed from the perspective of the top view, and the abnormal behavior detection model is constructed by using the connection between multiple frames of images. Compared with the single-frame static image, the multi-frame dynamic image contains richer personnel scene information, allowing the model to have a more comprehensive understanding of the action. Compared with the single type, simple structure, and susceptibility to noise interference of single-modal data, the information between the data is complemented by multi-modal data, which makes up for the data limitations between the personnel image and the posture features, enhances the model's understanding and analysis capabilities of the detection task, and improves the applicability and versatility of the model. At the same time, the model uses the modal attention mechanism to support custom addition and deletion of multi-modal data, and at the same time, for different modal data, the most suitable model is used for feature extraction, giving the model a high enough degree of freedom to meet the needs of various real detection scenarios. After extracting the features of each modal data, the time attention and modal attention mechanisms are introduced to calculate the correlation between the data in the time dimension and the modal dimension, and enhance the expression and generalization capabilities of the model. In the extraction of image features, the original two-dimensional convolutional network is optimized so that it can extract the image feature information of continuous frames and use deep separable convolution to reduce the computational complexity. In order to achieve the goal of being lightweight and efficient, the abnormal behavior detection model uses deep separable convolution plus spatiotemporal permutation attention mechanism on the most time-consuming image feature extraction module. The spatiotemporal attention mechanism expands the model's information perception ability in various dimensions, while reducing the computational complexity of ordinary convolution, while ensuring the model's detection accuracy as much as possible and improving the model's detection performance.

[0154] The above-mentioned embodiment of the present application provides a method for detecting human behavior based on multimodal data. Next, a human behavior detection device based on multimodal data provided in the embodiment of the present application is described, which is used to perform the above-mentioned Figure 2 The function of the personnel behavior detection device based on multimodal data is described. The structural schematic diagram of the personnel behavior detection device based on multimodal data is shown in FIG. Figure 7 As shown, it includes an extraction module 701 and an acquisition module 702 .

[0155] in,

[0156] An extraction module 701 is used for extracting an image frame to be processed from the video to be processed in response to obtaining the video to be processed;

[0157] The extraction module 701 is further used to extract multimodal data from the image frame to be processed by using a target detection algorithm, a target tracking algorithm and a human posture estimation algorithm, wherein the multimodal data at least includes a person detection frame, a person image and a person posture feature map;

[0158] The acquisition module 702 is used to input the person detection frame, the person image and the person posture feature map into a pre-built abnormal behavior detection model to obtain the behavior detection result output by the abnormal behavior detection model.

[0159] In a possible implementation, the abnormal behavior detection model includes a spatiotemporal graph convolutional network, a gated recurrent neural network, and an image feature extraction network;

[0160] The acquisition module 702 includes:

[0161] An extraction submodule, used for extracting posture features from the personnel posture feature graph using the spatiotemporal graph convolutional network;

[0162] The extraction submodule is further used to extract detection frame features from the person detection frame using the gated recurrent neural network;

[0163] A splicing submodule, used for splicing the posture feature and the detection frame feature to obtain an intermediate fusion feature;

[0164] The extraction submodule is further used to extract image features from the person image using the image feature extraction network;

[0165] The splicing submodule is further used to splice the intermediate fusion feature and the image feature to obtain the target fusion feature;

[0166] The acquisition submodule is used to obtain the behavior detection result based on the target fusion feature.

[0167] In a possible implementation, the extraction submodule is further used to extract detection frame features from the person detection frame using the gated recurrent neural network, including:

[0168] A determination unit, configured to determine the temporal features of the person detection frame using the gated recurrent neural network;

[0169] The acquisition unit is used to perform correlation calculation through a temporal attention mechanism to obtain the detection frame feature.

[0170] In a possible implementation, the image feature extraction network includes a three-dimensional bottleneck network and a time displacement attention module, and the extraction submodule is further used to extract image features from the person image using the image feature extraction network, including:

[0171] a determining unit, configured to determine initial image features of the person image using the three-dimensional bottleneck module;

[0172] a dividing unit, configured to input the initial image features into the time permutation attention module, wherein the time permutation attention module divides the initial image features into a plurality of groups according to image channels, each group having at least one sub-feature;

[0173] The determination unit is further used to use the sub-features as input features of the first branch channel, the spatial attention mechanism, and the second branch temporal attention mechanism, respectively, to determine the output features of the first branch and the output features of the second branch;

[0174] A processing unit is used to perform addition and fusion processing on the output features of the first branch and the output features of the second branch and to perform cross-group aggregation processing on the disrupted feature channels to obtain the image features.

[0175] In a possible implementation, the splicing submodule is further used to splice the intermediate fusion feature and the image feature to obtain a target fusion feature, including:

[0176] A splicing unit is used to splice the intermediate fusion feature and the image feature as input features of the modal attention mechanism to obtain the target fusion feature output by the modal attention mechanism.

[0177] In a possible implementation, the extraction module 701 is further used to extract multimodal data from the image frame to be processed using a target detection algorithm, a target tracking algorithm, and a human posture estimation algorithm, wherein the multimodal data includes at least a person detection frame, a person image, and a person posture feature map, including:

[0178] An extraction submodule, used for extracting the person detection frame from the image frame to be processed by using a target detection algorithm, wherein the target detection algorithm is a single-stage detection model Yolov5;

[0179] The extraction submodule is further used to extract the person image from the image frame to be processed by using the target tracking algorithm, and the target tracking algorithm is a multi-target tracking model DeepSORT;

[0180] The extraction submodule is further used to extract a person posture feature map from the image frame to be processed by using the human posture estimation algorithm, and the human posture estimation algorithm includes an OpenPose algorithm.

[0181] In a possible implementation, the personnel behavior detection method further includes:

[0182] The alarm module is used to trigger an alarm if the behavior detection result is abnormal behavior.

[0183] The embodiment of the present application provides a personnel behavior detection device based on multimodal data, including an extraction module and an acquisition module. The extraction module is used to extract the image frame to be processed from the video to be processed in response to acquiring the video to be processed; it is also used to extract multimodal data from the image frame to be processed using a target detection algorithm, a target tracking algorithm and a human posture estimation algorithm, and the multimodal data at least includes a personnel detection frame, a personnel image and a personnel posture feature map. The acquisition module is used to input the personnel detection frame, the personnel image and the personnel posture feature map into a pre-built abnormal behavior detection model to obtain the behavior detection result output by the abnormal behavior detection model. In this way, image data is obtained from video data, the personnel action is continuous, and the image data is close to the actual network scene; the abnormal behavior detection model using multimodal data fusion makes up for the data limitations and fusion adaptability between the personnel image, the detection frame and the posture features, so as to improve the accuracy of abnormal behavior recognition.

[0184] Based on the method for detecting human behavior based on multimodal data provided in the above method embodiment, the present application embodiment also provides a human behavior detection device based on multimodal data, see Figure 8 , the device includes: a processor, a memory, and a system bus;

[0185] The processor and the memory are connected via the system bus;

[0186] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes the personnel behavior detection method based on multimodal data described in any one of the above embodiments.

[0187] Based on the method embodiment provided above, a method for detecting human behavior based on multimodal data is provided. The embodiment of the present application further provides a storage medium, wherein the storage medium stores instructions. When the instructions are executed by a computer, the computer implements the method for detecting human behavior based on multimodal data described in any one of the embodiments above.

[0188] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0189] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0190] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting human behavior based on multimodal data, characterized in that: The personnel behavior detection method comprises: In response to acquiring a video to be processed, extracting an image frame to be processed from the video to be processed; Extracting multimodal data from the image frame to be processed by using a target detection algorithm, a target tracking algorithm, and a human posture estimation algorithm, wherein the multimodal data at least includes a person detection frame, a person image, and a person posture feature map; The person detection frame, the person image and the person posture feature map are input into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model.

2. The personnel behavior detection method according to claim 1, characterized in that: The abnormal behavior detection model includes a spatiotemporal graph convolutional network, a gated recurrent neural network, and an image feature extraction network; The step of inputting the person detection frame, the person image, and the person posture feature map into a pre-built abnormal behavior detection model to obtain a behavior detection result output by the abnormal behavior detection model includes: Extracting posture features from the personnel posture feature graph using the spatiotemporal graph convolutional network; Extracting detection frame features from the person detection frame using the gated recurrent neural network; Splicing the posture feature and the detection frame feature to obtain an intermediate fusion feature; extracting image features from the person image using the image feature extraction network; Splicing the intermediate fusion feature and the image feature to obtain a target fusion feature; The behavior detection result is obtained based on the target fusion feature.

3. The personnel behavior detection method according to claim 2, characterized in that: The extracting detection frame features from the person detection frame by using the gated recurrent neural network includes: Determining the temporal characteristics of the person detection frame using the gated recurrent neural network; The detection frame features are obtained by performing correlation calculation through a temporal attention mechanism.

4. The personnel behavior detection method according to claim 2, characterized in that: The image feature extraction network includes a three-dimensional bottleneck network and a time displacement attention module, and the extracting image features from the person image using the image feature extraction network includes: Determining initial image features of the person image using the three-dimensional bottleneck module; Inputting the initial image features into the time permutation attention module, the time permutation attention module divides the initial image features into a plurality of groups according to image channels, each group having at least one sub-feature; The sub-features are used as input features of the first branch channel, the spatial attention mechanism, and the second branch temporal attention mechanism, respectively, to determine the output features of the first branch and the output features of the second branch; The output features of the first branch and the output features of the second branch are added and fused, and the feature channels are shuffled and aggregated across groups to obtain the image features.

5. The personnel behavior detection method according to claim 2, characterized in that: The step of splicing the intermediate fusion feature and the image feature to obtain a target fusion feature includes: The intermediate fusion features and the image features are concatenated as input features of the modal attention mechanism to obtain the target fusion features output by the modal attention mechanism.

6. The personnel behavior detection method according to claim 1, characterized in that: The method comprises extracting multimodal data from the image frame to be processed by using a target detection algorithm, a target tracking algorithm and a human posture estimation algorithm, wherein the multimodal data at least includes a person detection frame, a person image and a person posture feature map, including: Extracting the person detection frame from the image frame to be processed by using a target detection algorithm, wherein the target detection algorithm is a single-stage detection model Yolov5; Extracting the person image from the image frame to be processed by using the target tracking algorithm, wherein the target tracking algorithm is a multi-target tracking model DeepSORT; The human body posture estimation algorithm is used to extract a person posture feature map from the image frame to be processed, and the human body posture estimation algorithm includes an OpenPose algorithm.

7. The personnel behavior detection method according to claim 1, characterized in that: The personnel behavior detection method also includes: If the behavior detection result is abnormal behavior, an alarm is triggered.

8. A personnel behavior detection device based on multimodal data, characterized in that: The personnel behavior detection device comprises: An extraction module, configured to extract an image frame to be processed from the video to be processed in response to obtaining the video to be processed; The extraction module is further used to extract multimodal data from the image frame to be processed by using a target detection algorithm, a target tracking algorithm and a human posture estimation algorithm, wherein the multimodal data at least includes a person detection frame, a person image and a person posture feature map; The acquisition module is used to input the person detection frame, the person image and the person posture feature map into a pre-built abnormal behavior detection model to obtain the behavior detection result output by the abnormal behavior detection model.

9. A personnel behavior detection device based on multimodal data, characterized in that: The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the personnel behavior detection method based on multimodal data as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed by a computer, the computer implements the personnel behavior detection method based on multimodal data as described in any one of claims 1 to 7.

Images