A smart community passenger behavior analysis system

Abstract

The present invention relates to the field of behavior analysis technology, and discloses a smart community passenger behavior analysis system, including: a data acquisition and preprocessing module, which is used to obtain monitoring video data, and save candidate videos with personnel, preprocess the candidate videos, and obtain a first feature sequence; a skeleton feature extraction module, which is used to construct a skeleton spatiotemporal graph according to the first feature sequence, and extract high-dimensional skeleton features through an improved spatiotemporal graph neural network; an environmental feature extraction module, which is used to extract environmental features from the first feature sequence; a feature fusion module, which is used to fuse high-dimensional skeleton features with environmental features to obtain fused features; an identification and early warning module, which is used to classify the fused features using a fully connected layer, obtain the probability of personnel behavior, and generate early warning measures based on the probability of personnel behavior. The present invention reduces dependence on security personnel and reduces the manpower cost of community operation through automated behavior analysis and anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of behavior analysis, and specifically relates to a smart community passenger behavior analysis system. Background Art

[0002] In the security management of smart communities, traditional monitoring systems mainly rely on security personnel to observe and judge multiple camera images in real time. However, as the community environment becomes increasingly complex, this model has gradually exposed its limitations. First, security personnel have limited energy and it is difficult to take care of multiple monitoring areas at the same time, resulting in potential safety hazards being overlooked. Second, long-term concentration can easily cause visual fatigue, reduce the efficiency and accuracy of identifying abnormal behaviors, and events such as illegal intrusions, climbing over fences, or gathering to cause trouble cannot be handled in a timely manner. In addition, the traditional system has a slow response speed, and there is a time difference from discovering anomalies to taking action, which affects the effective control of events. Furthermore, due to the lack of intelligent analysis capabilities, a large number of surveillance videos have not been effectively utilized, missing the opportunity to optimize security management strategies through data analysis. High labor costs also bring economic pressure to community operations. Summary of the invention

[0003] The present invention provides a smart community passenger behavior analysis system to solve the technical problems in related technologies such as multi-channel monitoring that relies on manual observation and judgment and is difficult to fully cover, is susceptible to visual fatigue, has low efficiency in identifying abnormal behaviors, and has high labor costs.

[0004] The present invention provides a smart community passenger behavior analysis system, comprising:

[0005] A data acquisition and preprocessing module is used to obtain surveillance video data from the surveillance management platform in real time, and use a lightweight target detection model to detect in real time whether there are people in the video frames of the surveillance video data, save the video frames with people and their subsequent video frames until the people disappear, use the saved video frames as candidate videos, and preprocess the candidate videos to obtain a first feature sequence;

[0006] A skeleton feature extraction module, which is used to construct a skeleton spatiotemporal graph according to the first feature sequence, and extract high-dimensional skeleton features through an improved spatiotemporal graph neural network; wherein the skeleton spatiotemporal graph includes: nodes, edges and initial features of nodes, a mapping relationship is established between nodes and joints of the human body, and the initial features of nodes are two-dimensional coordinates of nodes that establish a mapping relationship with joints;

[0007] An environmental feature extraction module, which is used to use a pre-trained convolutional neural network model as a feature extractor to extract and process features from the first feature sequence to obtain environmental features;

[0008] A feature fusion module is used to fuse high-dimensional skeleton features with environmental features to obtain fused features;

[0009] The recognition and warning module is used to classify the fused features using a fully connected layer to obtain the probability of personnel behavior and generate warning measures based on the probability of personnel behavior.

[0010] Furthermore, video frames are extracted from the surveillance video data at preset time intervals, and a lightweight target detection model is used to perform real-time detection on the video frames, wherein K adjacent video frames are packaged into a batch and input into the target detection model for personnel identification, where K is a custom parameter.

[0011] Further, the candidate video is preprocessed to obtain a first feature sequence, and the specific steps include:

[0012] Step S201: use the video frame of the candidate video

[0013] Processing, in which represents the Gaussian function value, represents the standard deviation of the Gaussian filter, x and y represent the coordinates of the video frame image pixels, Represents the edge direction angle, that is, the gradient direction angle of the pixel value change. Indicates the weight coefficient for adjusting the vertical direction, with a value range of 0 to 1. The entire video frame image is processed according to the Gaussian function value to obtain a denoised image. ;

[0014] Step S202, use The denoised image is subjected to nonlinear sharpening processing, where represents the image after nonlinear sharpening processing, represents the video frame of the candidate video, k represents the sharpening strength coefficient, and max represents the maximum value operation;

[0015] Step S203: using the brightness formula to sharpen the image after nonlinear sharpening Converted to grayscale, the brightness formula is: ,in, Represents a grayscale image, , and Represent the red, green, and blue channel values ​​of the pixel respectively;

[0016] Step S204: normalize the pixel values ​​of the processed video frame using a maximum-minimum normalization method to obtain a first feature sequence.

[0017] Further, the video frames in the first feature sequence are detected frame by frame using a posture estimation algorithm to extract the two-dimensional coordinates of the joint points of the personnel, and the two-dimensional coordinates of the joint points are used as the initial features of the nodes that establish a mapping relationship with the joint points;

[0018] The edges of the skeleton spatiotemporal graph include: spatial edges connecting the joints of the human body in the same video frame according to the human anatomical structure, and temporal edges connecting the same joints in adjacent video frames.

[0019] Furthermore, high-dimensional skeleton features are extracted through an improved spatiotemporal graph neural network. The specific improvements include: introducing nonlinear functions in the spatial convolution part, introducing gated recurrent units in the temporal convolution part, and adding feature enhancement items of the nodes themselves; the calculation formula of the improved spatiotemporal graph neural network is:

[0020] ;

[0021] in, Represents the high-dimensional skeleton features output by node v in the tth video frame, represents the initial features of node u in the tth video frame, represents the neighbor set in the jth subregion of node v, where the neighbor nodes of node v are divided into different groups according to the anatomical structure of the skeleton, each group represents a subregion of node v, J represents the number of subregions of node v, j represents the subregion index of node v, represents the weight parameter of the jth spatial convolution kernel, Represents the adjacency matrix value between nodes v and u, that is, the connection relationship between nodes v and u. If there is an edge between nodes v and u, the adjacency matrix value is 1, otherwise it is 0. represents the nonlinear transformation weight matrix, represents the nonlinear transformation bias parameter, represents the normalization coefficient of nodes v and u, T represents the number of video frames, represents the weight parameter of the temporal convolution kernel, represents the time frame after the shift, represents the time offset, GRU represents the use of gated recurrent units to process the initial features of nodes in the time dimension, represents the self-characteristic proportionality coefficient, represents the self-feature weight parameter, ReLU represents the self-feature bias parameter and ReLU represents the ReLU activation function.

[0022] Furthermore, a pre-trained model is used to extract environmental features, wherein the pre-trained model includes a convolution layer and a pooling layer, the convolution layer inputs a video frame of a first feature sequence and outputs a feature map, and the pooling layer is used to convert the feature map into environmental features through global average pooling.

[0023] Furthermore, the high-dimensional skeleton features are fused with the environmental features to obtain the fused features. The fusion calculation formula is:

[0024] ;

[0025] in, represents the fusion feature, represents the activation function, represents the weight coefficient of skeleton feature transformation, represents the weight coefficient of environmental feature transformation, and Represent the weight matrix and bias vector of the i-th skeleton feature transformation, respectively. and Represent the weight matrix and bias vector of the d-th environmental feature transformation, respectively. represents the coefficient of the regularization term, represents the weight matrix of the regularization term, represents the regularization term, Represents the original skeleton feature vector, which contains the human body posture and joint positions extracted from the skeleton data. Represents the original environment feature vector, which contains the scene layout and object positions extracted from the environment data. N and M represent the number of skeleton feature transformations and environment feature transformations, respectively. i and d represent index values, respectively.

[0026] Furthermore, the human behaviors include: normal behavior, climbing over fences, fighting, falling and climbing; the fused features are input into the fully connected layer, in which each fused feature has a weight connection with the output node, and the score of each behavior category is obtained by calculating the product of the fused feature and the weight matrix and adding the bias parameter; the score of each behavior category is converted into a probability value between 0 and 1 using the Softmax function, and the sum of the probabilities of all categories is 1, wherein each behavior category will obtain a probability value, and according to the size of the probability value, the corresponding behavior category is selected as the final classification result; wherein the calculation formula of the fully connected layer is: , where Q represents the probability of each person's behavior, and the sum of the probabilities is 1, and W represents the weight matrix of the fully connected layer. Represents the bias parameters of the fully connected layer.

[0027] The beneficial effects of the present invention are as follows: the present invention uses a lightweight target detection model and an improved spatiotemporal graph neural network to analyze the behavior of people in the monitoring screen in real time, quickly identify abnormal situations and trigger early warnings, and significantly improve the response speed of community safety management;

[0028] The present invention combines the extracted high-dimensional skeleton features with environmental features to accurately identify abnormal behaviors such as climbing over fences, fighting, and falling, adapting to complex scenarios and ensuring the high sensitivity and accuracy of the monitoring system to potential threats.

[0029] The present invention gets rid of the inefficient mode of traditional manual monitoring, reduces the dependence on security personnel and reduces the manpower cost of community operation through automated behavior analysis and anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a module schematic diagram of a smart community passenger behavior analysis system of the present invention;

[0031] Figure 2 It is a flow chart of preprocessing candidate videos of the present invention.

[0032] In the figure: data collection and preprocessing module 101, skeleton feature extraction module 102, environmental feature extraction module 103, feature fusion module 104 and recognition and warning module 105. DETAILED DESCRIPTION

[0033] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0034] like Figure 1-Figure 2 As shown, a smart community passenger behavior analysis system includes:

[0035] The data acquisition and preprocessing module 101 is used to obtain monitoring video data from the monitoring management platform in real time, and use a lightweight target detection model to detect in real time whether there is a person in the video frame of the monitoring video data, save the video frame with the person and its subsequent video frames until the person disappears, use the saved video frame as a candidate video, and preprocess the candidate video to obtain a first feature sequence;

[0036] A skeleton feature extraction module 102 is used to construct a skeleton spatiotemporal graph according to the first feature sequence, and extract high-dimensional skeleton features through an improved spatiotemporal graph neural network; wherein the skeleton spatiotemporal graph includes: nodes, edges, and initial features of nodes, and mapping relationships are established between nodes and joints of the human body, for example, shoulders, elbows, and knees, and the initial features of nodes are two-dimensional coordinates of nodes that establish mapping relationships with joints;

[0037] An environmental feature extraction module 103 is used to use a pre-trained convolutional neural network model as a feature extractor to extract and process features from the first feature sequence to obtain environmental features;

[0038] A feature fusion module 104 is used to fuse the high-dimensional skeleton features with the environmental features to obtain fused features;

[0039] The identification and warning module 105 is used to classify the fused features using a fully connected layer to obtain the probability of personnel behavior and generate warning measures based on the probability of personnel behavior.

[0040] In one embodiment of the present invention, video frames are extracted from surveillance video data at preset time intervals, and a YOLOv5n target detection model is used for real-time identification of personnel. Specifically, the size of the video frames is adjusted to the same size, and K adjacent video frames are packaged into a batch and input into the YOLOv5n target detection model together, and the category, confidence and bounding box coordinates of the detected target object are output. When the category of the target object is a person and the confidence exceeds a preset confidence threshold, the current video frame and its subsequent video frames are saved until the person disappears to form a candidate video, wherein the preset time interval is set to 10ms, K is a custom parameter, and K is set to 10.

[0041] In one embodiment of the present invention, the candidate video is preprocessed to obtain a first feature sequence, and the specific steps include:

[0042] Step S201: use the video frame of the candidate video

[0043] Processing is performed to smooth the noise of the video frame image, where represents the Gaussian function value, represents the standard deviation of the Gaussian filter, x and y represent the coordinates of the video frame image pixels, Represents the edge direction angle, that is, the gradient direction angle of the pixel value change. Indicates the weight coefficient for adjusting the vertical direction, with a value range of 0 to 1. The entire video frame image is processed according to the Gaussian function value to obtain a denoised image. ;

[0044] Step S202, use The denoised image is subjected to nonlinear sharpening processing to improve the clarity of the image by enhancing the edges and details of the image. represents the image after nonlinear sharpening processing, represents the video frame of the candidate video, k represents the sharpening strength coefficient, and max represents the maximum value operation;

[0045] Step S203: using the brightness formula to sharpen the image after nonlinear sharpening Converted to grayscale, the brightness formula is: ,in, Represents a grayscale image, , and Represent the red, green, and blue channel values ​​of the pixel respectively;

[0046] Step S204: normalize the pixel values ​​of the processed video frame using a maximum-minimum normalization method to obtain a first feature sequence.

[0047] In one embodiment of the present invention, a posture estimation algorithm is used to detect the video frames in the first feature sequence frame by frame, extract the two-dimensional coordinates of the joint points of the personnel, and use the two-dimensional coordinates of the joint points as the initial features of the nodes that establish a mapping relationship with the joint points;

[0048] The edges of the skeleton spatiotemporal graph include: spatial edges connecting the joints of the human body in the same video frame according to the human anatomical structure, and temporal edges connecting the same joints in adjacent video frames.

[0049] In one embodiment of the present invention, the HRNet posture estimation algorithm is used to extract the two-dimensional coordinates of the joint points of a person, and the specific steps include:

[0050] Step S301, using a multi-resolution parallel convolutional network to simultaneously process high-resolution and low-resolution features, and perform feature fusion to obtain a feature map;

[0051] Step S302, generate a heat map of each joint point by performing a convolution operation. The value of each pixel on the heat map represents the probability that the position is the joint point k. The formula for the convolution operation is: ,in, represents the heat map of the kth joint point, represents the convolution kernel used to generate the k-th joint point heat map, and F represents the feature map;

[0052] Step S303, using the centroid method to extract the two-dimensional coordinates of the joint points from the heat map, the calculation formula of the centroid method is: ,in, represents the two-dimensional coordinates of the kth joint point, and x and y represent the horizontal and vertical coordinates of the pixel in the feature map.

[0053] In one embodiment of the present invention, high-dimensional skeleton features are extracted by an improved spatiotemporal graph neural network. The specific improvements include: introducing a nonlinear function in the spatial convolution part, introducing a gated recurrent unit in the temporal convolution part, and adding feature enhancement items of the node itself; wherein the calculation formula of the improved spatiotemporal graph neural network is:

[0054] ;

[0055] in, Represents the high-dimensional skeleton features output by node v in the tth video frame, represents the initial features of node u in the tth video frame, represents the neighbor set in the jth subregion of node v, where the neighbor nodes of node v are divided into different groups according to the anatomical structure of the skeleton, each group represents a subregion of node v, and the anatomical structure of the skeleton represents the natural joint connection relationship of the human body. For example, for the shoulder node, subregion 1 is the neck, and subregion 2 is the elbow. J represents the number of subregions of node v, and j represents the subregion index of node v. represents the weight parameter of the jth spatial convolution kernel, Represents the adjacency matrix value between nodes v and u, that is, the connection relationship between nodes v and u. If there is an edge between nodes v and u, the adjacency matrix value is 1, otherwise it is 0. represents the nonlinear transformation weight matrix, represents the nonlinear transformation bias parameter, represents the normalization coefficient of nodes v and u, T represents the number of video frames, represents the weight parameter of the temporal convolution kernel, represents the time frame after the shift, represents the time offset, GRU represents the use of gated recurrent units to process the initial features of nodes in the time dimension, represents the self-characteristic proportionality coefficient, represents the self-feature weight parameter, ReLU represents the self-feature bias parameter and ReLU represents the ReLU activation function.

[0056] By introducing the nonlinear transformation function , which can make node features undergo nonlinear transformation before convolution to enhance the expressive power; using gated recurrent units to process the initial node features in the time dimension can capture complex time series relationships; adding feature enhancement items of the node itself , which can enhance the importance of each node’s own characteristics and improve the understanding of local behavior.

[0057] In one embodiment of the present invention, a ResNet model pre-trained on a Places365 dataset is used to extract environmental features, wherein the Places365 dataset is a dataset specifically used for scene classification, and the ResNet model includes a convolutional layer and a pooling layer. Specifically, the fully connected layer of the last layer of the ResNet model is removed, and the convolutional layer and the pooling layer are retained, and the feature map output by the convolutional layer is converted into an environmental feature of a fixed length through global average pooling.

[0058] In one embodiment of the present invention, the high-dimensional skeleton features are fused with the environmental features to obtain fused features. The calculation formula for the fusion is:

[0059] ;

[0060] in, represents the fusion feature, represents the activation function, represents the weight coefficient of skeleton feature transformation, represents the weight coefficient of environmental feature transformation, and Represent the weight matrix and bias vector of the i-th skeleton feature transformation, respectively. and Represent the weight matrix and bias vector of the d-th environmental feature transformation, respectively. represents the coefficient of the regularization term, represents the weight matrix of the regularization term, represents the regularization term, Represents the original skeleton feature vector, which contains the human body posture and joint positions extracted from the skeleton data. Represents the original environment feature vector, which contains the scene layout and object positions extracted from the environment data. N and M represent the number of skeleton feature transformations and environment feature transformations, respectively. i and d represent index values, respectively.

[0061] In one embodiment of the present invention, the human behavior includes: normal behavior, climbing over a fence, fighting, falling, and climbing; the fused features are input into a fully connected layer, in which each fused feature has a weight connection with an output node, and the score of each behavior category is obtained by calculating the product of the fused feature and the weight matrix and adding a bias parameter; the score of each behavior category is converted into a probability value between 0 and 1 using a Softmax function, and the sum of the probabilities of all categories is 1, wherein each behavior category will obtain a probability value, and according to the size of the probability value, the corresponding behavior category is selected as the final classification result, wherein the calculation formula of the fully connected layer is: , where Q represents the probability of each person's behavior, and the sum of the probabilities is 1, and W represents the weight matrix of the fully connected layer. Represents the bias parameters of the fully connected layer.

[0062] In one embodiment of the present invention, according to the probabilities of various human behaviors output by the fully connected layer combined with preset thresholds, corresponding early warning measures are taken for different behaviors; when the probability of normal behavior is greater than 0.8, the behavior is judged to be normal and no measures are taken; when the probability of climbing over the fence is greater than 0.5, the fence alarm is triggered and the security personnel are notified; when the probability of fighting exceeds 0.5, the monitoring picture of the location is enlarged, and the security personnel are notified to arrive quickly; when the probability of falling exceeds 0.5, the monitoring picture is pushed to the security personnel, and it is determined whether they need to arrive based on the actual situation; when the probability of climbing behavior exceeds 0.5, a high-altitude alarm is issued through a nearby speaker, and the security personnel are notified.

[0063] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are within the protection of the present embodiment.

Claims

1. A smart community passenger behavior analysis system, characterized in that: include: The data acquisition and preprocessing module is used to obtain surveillance video data from the surveillance management platform in real time, and use a lightweight target detection model to detect in real time whether there are people in the video frames of the surveillance video data, save the video frames with people and their subsequent video frames until the people disappear, use the saved video frames as candidate videos, and preprocess the candidate videos to obtain the first feature sequence. The specific steps include: Step S201: use the video frame of the candidate video Processing, in which represents the Gaussian function value, represents the standard deviation of the Gaussian filter, x and y represent the coordinates of the video frame image pixels, Represents the edge direction angle, that is, the gradient direction angle of the pixel value change. Indicates the weight coefficient for adjusting the vertical direction, with a value range of 0 to 1. The entire video frame image is processed according to the Gaussian function value to obtain a denoised image. ; Step S202, use The denoised image is subjected to nonlinear sharpening processing, where represents the image after nonlinear sharpening processing, represents the video frame of the candidate video, k represents the sharpening strength coefficient, and max represents the maximum value operation; Step S203: using the brightness formula to sharpen the image after nonlinear sharpening Converted to grayscale, the brightness formula is: ,in, Represents a grayscale image, , and Represent the red, green, and blue channel values ​​of the pixel respectively; Step S204, using a maximum-minimum normalization method to normalize the pixel values ​​of the processed video frame to obtain a first feature sequence; The skeleton feature extraction module is used to construct a skeleton spatiotemporal graph according to the first feature sequence, and extract high-dimensional skeleton features through an improved spatiotemporal graph neural network; wherein the skeleton spatiotemporal graph includes: nodes, edges and initial features of nodes, a mapping relationship is established between nodes and joints of the human body, and the initial features of nodes are the two-dimensional coordinates of nodes that establish a mapping relationship with joints. The specific improvements include: introducing nonlinear functions in the spatial convolution part, introducing gated recurrent units in the temporal convolution part, and adding feature enhancement items of the nodes themselves; the calculation formula of the improved spatiotemporal graph neural network is: ； in, Represents the high-dimensional skeleton features output by node v in the tth video frame, represents the initial features of node u in the tth video frame, represents the neighbor set in the jth subregion of node v, where the neighbor nodes of node v are divided into different groups according to the anatomical structure of the skeleton, each group represents a subregion of node v, J represents the number of subregions of node v, j represents the subregion index of node v, represents the weight parameter of the jth spatial convolution kernel, Represents the adjacency matrix value between nodes v and u, that is, the connection relationship between nodes v and u. If there is an edge between nodes v and u, the adjacency matrix value is 1, otherwise it is 0. represents the nonlinear transformation weight matrix, represents the nonlinear transformation bias parameter, represents the normalization coefficient of nodes v and u, T represents the number of video frames, represents the weight parameter of the temporal convolution kernel, represents the time frame after the shift, represents the time offset, GRU represents the use of gated recurrent units to process the initial features of nodes in the time dimension, represents the self-characteristic proportionality coefficient, represents the self-feature weight parameter, Represents the self-feature bias parameter, ReLU represents the ReLU activation function; An environmental feature extraction module, which is used to use a pre-trained convolutional neural network model as a feature extractor to extract and process features from the first feature sequence to obtain environmental features; The feature fusion module is used to fuse the high-dimensional skeleton features with the environmental features to obtain the fusion features. The calculation formula for the fusion is: ； in, represents the fusion feature, represents the activation function, represents the weight coefficient of skeleton feature transformation, represents the weight coefficient of environmental feature transformation, and Represent the weight matrix and bias vector of the i-th skeleton feature transformation, respectively. and Represent the weight matrix and bias vector of the d-th environmental feature transformation, respectively. represents the coefficient of the regularization term, represents the weight matrix of the regularization term, represents the regularization term, Represents the original skeleton feature vector, which contains the human body posture and joint positions extracted from the skeleton data. represents the original environment feature vector, which contains the scene layout and object positions extracted from the environment data. N and M represent the number of skeleton feature transformations and environment feature transformations, respectively. i and d represent index values, respectively. The recognition and warning module is used to classify the fused features using a fully connected layer to obtain the probability of personnel behavior and generate warning measures based on the probability of personnel behavior.

2. According to claim 1, a smart community passenger behavior analysis system is characterized in that: Video frames are extracted from the surveillance video data at preset time intervals, and a lightweight target detection model is used to perform real-time detection on the video frames. K adjacent video frames are packaged into a batch and input into the target detection model for personnel identification. K is a custom parameter.

3. According to claim 1, a smart community passenger behavior analysis system is characterized in that: Using a posture estimation algorithm to detect the video frames in the first feature sequence frame by frame, extracting the two-dimensional coordinates of the joint points of the personnel, and using the two-dimensional coordinates of the joint points as the initial features of the nodes that establish a mapping relationship with the joint points; The edges of the skeleton spatiotemporal graph include: spatial edges connecting the joints of the human body in the same video frame according to the human anatomical structure, and temporal edges connecting the same joints in adjacent video frames.

4. According to claim 1, a smart community passenger behavior analysis system is characterized in that: Environmental features are extracted using a pre-trained model, wherein the pre-trained model includes a convolution layer and a pooling layer, the convolution layer inputs a video frame of a first feature sequence and outputs a feature map, and the pooling layer is used to convert the feature map into environmental features through global average pooling.

5. According to claim 1, a smart community passenger behavior analysis system is characterized in that: The behaviors of the personnel include: normal behavior, climbing over fences, fighting, falling and climbing; the fusion features are input into the fully connected layer, in which each fusion feature has a weight connection with the output node, and the score of each behavior category is obtained by calculating the product of the fusion feature and the weight matrix and adding the bias parameter; the score of each behavior category is converted into a probability value between 0 and 1 using the Softmax function, and the sum of the probabilities of all categories is 1, where each behavior category will get a probability value, and according to the size of the probability value, the corresponding behavior category is selected as the final classification result; wherein, the calculation formula of the fully connected layer is: , where Q represents the probability of each person's behavior, and the sum of the probabilities is 1, and W represents the weight matrix of the fully connected layer. Represents the bias parameters of the fully connected layer.

Images