A Group Behavior Recognition Method Based on Sensor Data with Multi-Level Feature Enhancement

Abstract

This invention belongs to the field of sensor data group behavior recognition technology, and discloses a sensor data group behavior recognition method based on multi-level feature enhancement. The method includes: segmenting individual local location sensor data through a sliding window, inputting the data into an individual feature extraction network to obtain individual behavior features; obtaining group position interaction features from the location data of each individual, fusing individual behavior features and group position interaction features to obtain individual behavior enhancement features, and concatenating them to obtain group behavior features; using a spatiotemporal feature enhancement module to obtain group behavior enhancement features, and outputting the group behavior recognition result through a fully connected layer. This invention not only fully utilizes sensor data information and overcomes the difficulty of group feature representation, but also improves the accuracy and robustness of group behavior recognition by performing multi-level enhancement of individual and group features, effectively representing the spatiotemporal information of groups.

Description

Technical Field

[0001] This invention belongs to the field of sensor data group behavior recognition technology, specifically relating to a sensor data group behavior recognition method based on multi-level feature enhancement. Background Technology

[0002] Group behavior recognition based on sensor data is a promising and promising research field. It involves understanding the overall behavior of interactions among multiple individuals and has wide applications in various fields, such as urban planning, traffic management, and sociological research. Group behavior recognition in complex environments is a challenging task. This is because group behavior is not simply the sum of individual behaviors, but requires inference from individual behaviors and the interactions between individuals. Therefore, it needs to identify individual behaviors within a group and consider the complex interactions between individuals to conduct bottom-up behavioral analysis.

[0003] In recent years, with the rapid development of the Internet of Things (IoT) industry and the widespread adoption of technologies such as wireless sensor networks and wearable devices, the extensive application of various sensors has provided new data sources for group behavior recognition research. Simultaneously, the widespread application of technologies such as machine learning and deep learning has led to significant progress in group behavior recognition based on sensor data. However, most current group behavior research still focuses on the fields of vision and image processing. Group behavior recognition based on sensor data is less costly than video and image-based methods, is not geographically restricted, and has less privacy intrusion. Currently, smart terminal devices have integrated a wealth of sensor devices, including accelerometers, magnetometers, gyroscopes, and GPS, which makes the use of smart terminal devices for group behavior recognition feasible.

[0004] However, despite the widespread application of various sensors and deep learning technologies providing data and theoretical support for group behavior recognition research, this field still faces several challenges. First, the extraction of group features remains a complex problem. Group behavior often involves the interaction of multiple individuals, thus requiring effective feature extraction methods. Extracting key information from sensor data to describe overall behavior remains a challenging task. Second, group behavior is typically dynamic and uncertain, necessitating consideration of temporal and spatial characteristics. Summary of the Invention

[0005] The purpose of this invention is to provide a sensor data group behavior recognition method based on multi-level feature enhancement. This method not only fully mines sensor data information and overcomes the difficulty of group feature representation, but also improves the accuracy and robustness of group behavior recognition by performing multi-level enhancement on individual and group features. It can effectively represent the spatiotemporal information of the group.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for identifying group behavior based on sensor data using multi-level feature enhancement, comprising:

[0008] Step 1: For each individual, the continuous sensor data collected by the sensor at the local location of the individual is segmented by a sliding window to obtain the sensor data at the local location of the individual. The sensor data at different local locations of the same individual are then input into the individual feature extraction network to obtain the individual's behavioral features.

[0009] Step 2: Preprocess the location data of each individual to obtain group location interaction information, characterize the group location interaction information to obtain group location interaction features, fuse the individual behavior features of an individual and the group location interaction features to obtain the individual behavior enhancement features of that individual, and splice the individual behavior enhancement features of all individuals in the group to obtain the group behavior features.

[0010] Step 3: Construct a spatiotemporal feature enhancement module using a CAMLP-mixer network. Use the spatiotemporal feature enhancement module to enhance the group behavior features to obtain enhanced group behavior features. Then, output the group behavior recognition result through a fully connected layer.

[0011] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0012] Preferably, the individual feature extraction network includes a spatiotemporal graph convolutional network and a bidirectional long short-term memory neural network. First, the spatiotemporal graph convolutional network is used to extract temporal and spatial features from sensor data at different local locations to obtain local individual behavior features. Then, after feature fusion of local individual behavior features at different local locations of the same individual, the bidirectional long short-term memory neural network is used to obtain individual behavior features.

[0013] Preferably, the spatiotemporal graph convolutional network uses a graph convolutional network for spatial feature extraction, and the extraction process is as follows:

[0014] z1=ReLU(Conv2d(x·A)+down(x))

[0015] In the formula, z1 represents the extracted spatial features, ReLU(·) represents the ReLU activation function, Conv2d(·) represents 2D convolution, x represents the input data tensor, A represents the adjacency matrix in the graph convolutional network, x·A represents the input data tensor x multiplied by the adjacency matrix A, and down(·) represents the downsampling operation.

[0016] The spatiotemporal graph convolutional network uses a temporal convolutional network for temporal feature extraction, and the extraction process is as follows:

[0017] z2 = ReLU(Conv1d(z1,1))

[0018] In the formula, z2 is the extracted temporal feature, and Conv1d(z1,1) represents a 1D convolution operation on the spatial feature z1 with a kernel size of 1.

[0019] Preferably, the preprocessing of each individual's location data to obtain group location interaction information includes:

[0020] For a unit time step, the location data of each individual is converted into a value within the standard range of [0,1] through a normalization operation to obtain normalized location data. Then, the normalized location data is scaled to obtain scaled location data. Finally, the scaled location data of all individuals are marked on a grid of a specific size to obtain grid information representing the spatial relationship and location distribution of the group in a time step.

[0021] The location data at all time steps are processed to obtain the grid information corresponding to each time step. The grid information of all time steps is then stitched together to obtain the group location interaction information.

[0022] Preferably, the process of characterizing the group location interaction information to obtain group location interaction features includes:

[0023] The group's positional interaction information is extracted by using a 3D convolutional neural network, and then the output of the 3D convolutional neural network is modeled by a bidirectional long short-term memory neural network to obtain the group's positional interaction features.

[0024] Preferably, the individual behavioral features of the fused individual and the group positional interaction features are used to obtain the individual behavioral enhancement features of that individual, and the group behavioral features are obtained by splicing together the individual behavioral enhancement features of all individuals in the group, including:

[0025]

[0026]

[0027]

[0028]

[0029] In the formula, Let Q represent the weight matrix of the i-th individual during fusion. i K i V i yes and The result of performing a dot product and then multiplying it by the corresponding weight matrix is ​​d. k For K i The dimension size, softmax is the activation function. F represents the individual behavioral characteristics of the i-th individual. inter Indicates the location interaction characteristics of the group. K represents i The transpose of F * This represents the characteristics of group behavior, where n is the total number of individuals in the population.

[0030] Preferably, the spatiotemporal feature enhancement module includes multiple cascaded CAMLP-mixer networks, and the operation of each CAMLP-mixer network is represented as follows:

[0031] F = x(E, CA(E)) + E

[0032] In the formula, F represents the output feature of the CAMLP-mixer network, x(·,·) represents the element-wise multiplication operation, E represents the result of the Mix operation, and CA(·) represents the channel attention processing operation.

[0033] The calculation process for the Mix operation result is as follows:

[0034] Mix=Norm(σ((F * ) T W t1 W t2 )

[0035] In the formula, Mix is ​​the result of the Mix operation, Norm(·) is the NORM function, σ(·) represents the ReLU activation function, and F * W represents the characteristics of group behavior. t1 and W t2 This represents a 2-layer MLP operation;

[0036] The calculation process for the channel attention processing operation is as follows:

[0037]

[0038]

[0039] In the formula, x kjLet k represent the influence of the k-th channel on the j-th channel. Linear transformations are performed on the input feature E to obtain feature maps B, C, and D. The indices k and j in feature maps B, C, and D represent different channels in these feature maps, and k ≠ j, i.e., B... k C represents the k-th channel in feature map B. j D represents the j-th channel in feature map C. k Let M represent the k-th channel in feature map D, N represent the total number of channels in input feature E, and M represent the total number of channels in input feature E. j Let β be the weight of the j-th channel in the final attention map, and β be the scale parameter. For x kj The transpose matrix, E k This refers to the k-th channel in the input feature E.

[0040] This invention provides a sensor data group behavior recognition method based on multi-level feature enhancement. It uses a location-level feature enhancement module based on spatial location interaction and a spatiotemporal feature enhancement module based on CAMLP-mixer to recognize group behavior. It fully explores the interaction relationship between individuals to represent group features at the group level. At the same time, it uses the spatiotemporal feature enhancement module to realize the information propagation in the spatial and temporal domains of the group. It can achieve high recognition accuracy and robustness even when the number of participants is random. Attached Figure Description

[0041] Figure 1 This is a flowchart of the sensor data group behavior recognition method based on multi-level feature enhancement according to the present invention;

[0042] Figure 2 This is a schematic diagram of the location-level feature enhancement module according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the spatiotemporal feature enhancement module in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0046] Group behavior recognition based on sensor data refers to acquiring individual behavioral information by collecting sensor data, then using this data as input to ultimately output the overall group behavior. These sensors typically include accelerometers and gyroscopes, whose activity varies depending on the human body's state. By collecting individual behavioral data from different body parts using these sensors, while also considering individual positional coordinates, the information from the sensor data can be fully utilized to improve the accuracy and robustness of group behavior recognition.

[0047] Currently, various smart devices such as smartphones and smartwatches are widely used. These devices have built-in sensors, such as accelerometers, magnetometers, gyroscopes, and GPS. This example uses accelerometer and gyroscope sensors located in the wrist and lower limb pockets to illustrate the processing of sensor data for group behavior recognition.

[0048] like Figure 1 As shown in the figure, this embodiment provides a method for group behavior recognition based on sensor data with multi-level feature enhancement, which includes the following steps:

[0049] Step 1: For each individual, the continuous sensor data collected by the sensor at the local location of the individual is segmented by a sliding window to obtain the individual's local location sensor data. The sensor data at different local locations of the same individual are then input into the individual feature extraction network to obtain the individual's behavioral features.

[0050] Individuals collect continuous data from sensors at different local locations. This continuous sensor data needs to be segmented into individual local location sensor data. In a specific example, a sliding window with a window size of 200 is used. A window size of 200 means that, with the sensor sampling data at a frequency of 50Hz, each input data point represents 4 seconds of data, or 50 data points sampled per second. Data within a consecutive four seconds is considered as one input data point. A preset overlap ratio is also considered when segmenting the data. For example, a window size of 200 is selected with 50% data overlap segmentation. 50% data overlap means that some data is reused in the segmentation. Assuming that each window size of the current input data is 200, the last 100 data points are also simultaneously used as the first 100 data points of the next input data point.

[0051] Continuous data collected by sensors at different local locations is processed to obtain individual sensor data for those locations. For example, smart devices located on the wrist and lower limb pockets have accelerometers and gyroscopes, each with x, y, and z-axis data, resulting in 6-dimensional data (i.e., 6 channels) for each local location. By segmenting this continuous sensor data, individual local location sensor data is obtained, represented as l = {l1, l2, ..., l6}, a 6-dimensional data vector, where l1 is the first dimension after segmentation, and so on.

[0052] In this embodiment, each individual has two sets of local position sensor data. If sensors also collect data from other parts of the individual's body, then each individual can have more local position sensor data. Furthermore, a single body part can be equipped with more sensors; this embodiment does not limit the number or dimensions of local position sensor data for each individual.

[0053] This invention identifies group behavior by collecting sensor data from all individuals in the group and inputting it into a network. It assumes the group contains a maximum of 5 individuals, but in reality, the group size is not fixed. Therefore, the following embodiments use a group of 5 individuals as an example for illustration.

[0054] The network model constructed in this embodiment includes an individual feature extraction network, a location-level feature enhancement module, and a spatiotemporal feature enhancement module.

[0055] In one specific embodiment, the individual feature extraction network is used to process the local location sensor data of each individual in the group to obtain individual behavioral features, including:

[0056] For each individual, continuous sensor data collected from sensors at local locations is segmented using a sliding window. This segmentation yields individual local location sensor data. The individual's wrist sensor data and lower limb pocket sensor data are then input into a cascaded spatiotemporal graph convolutional network (GCN-TCN), comprising a graph convolutional network (GCN) and a temporal convolutional network (TCN). The 6-dimensional data from each individual's local sensors are used as nodes in the GCN and input to the graph convolutional network. A single layer of the graph convolutional network is used, and its aggregation function is applied to convolve the neighborhood dimensions of each dimension to update the center dimension. This results in six features containing latent semantic relationships between dimensions. A ReLU activation function is then applied for nonlinear transformation, and finally, the temporal convolutional network is used to represent the temporal features of the sensor data, thus obtaining the local individual behavioral features. (Wrist features) and (physical characteristics), then and Feature concatenation is performed, and the data is input into a bidirectional long short-term memory neural network (BLSTM) to obtain the individual behavioral features F. per .

[0057] Individual feature extraction networks can be represented by the following formula:

[0058] z1=ReLU(Conv2d(x·A)+down(x))

[0059] z2 = ReLU(Conv1d(z1,1))

[0060]

[0061]

[0062] z1 represents the result after processing by the Graph Convolutional Network (GCN), and x is the input data tensor, x∈R. N×C×T×V N represents the batch size, C represents the number of channels in the input data (6 in this embodiment), T represents the number of time steps in the time series data (200 in this embodiment), and V represents the number of nodes in the graph structure (6 in this embodiment). Matrix A represents the adjacency matrix in the Graph Convolutional Network (GCN), which performs convolution operations on the data in the graph and can be automatically learned during network training. The convolution operation in GCN is a local information aggregation method based on the adjacency matrix, allowing nodes to share information with their neighboring nodes. By multiplying the input tensor x with the adjacency matrix A, feature propagation and information transfer are performed, aggregating the feature information of each node to its neighboring nodes, and finally generating the convolutional output. The convolution used in GCN is 2D convolution, i.e., the Conv2d operation in the formula, with a kernel size of 1. The convolutional output is then residually concatenated with the original data to allow gradient propagation to be faster. Here, down represents the downsampling operation on the original data. Finally, the output z1 of the graph convolutional network is obtained through the ReLU activation function.

[0063] z2 represents the result after processing by the Temporal Convolutional Network (TCN), and Conv1d represents performing a 1D convolution on z1 with a kernel size of 1, followed by a ReLU activation function to capture the temporal dependencies and features in the sensor data.

[0064] By performing a residual connection on z1 and z2, the local behavioral characteristics of the individual can be obtained. Where o∈{body,waist}, represents the individual local behavioral features from the body and wrist. Finally, the individual local behavioral features from the wrist... and individual local behavioral characteristics from the body Feature concatenation is performed, and the final individual behavioral features F can be obtained through a bidirectional long short-term memory neural network (BLSTM network). per For ease of distinction, this embodiment adopts... This represents the individual behavioral characteristics of the i-th individual.

[0065] Step 2: Preprocess the location data of each individual to obtain group location interaction information, characterize the group location interaction information to obtain group location interaction features, fuse the individual behavior features of an individual and the group location interaction features to obtain the individual behavior enhancement features of that individual, and splice the individual behavior enhancement features of all individuals in the group to obtain the group behavior features.

[0066] like Figure 2 As shown, within a unit time step, each individual in the group has its own spatial location (during dataset collection, the change in location of each individual over time is simulated by software, specifically referring to (x, y), i.e., coordinate data on the x and y axes), represented by black dots in the coordinate system. For a unit time step, the location-level feature enhancement module first processes the coordinates of each individual in the group. First, a normalization operation is performed, converting the original location data coordinates into values ​​within the standard range of [0, 1]. Specifically, the minimum value (min) of the location coordinate data is first calculated. coord and maximum value max coord Then, by subtracting min from each position coordinate... coord Divide by max coord -min coord The difference is used to normalize the position coordinate data to the range [0,1].

[0067] The normalization process can be represented by the following formula:

[0068]

[0069] The original location coordinates of each individual are denoted as coord, and the normalized location data is denoted as norm. coord min coord The maximum value represents the minimum position coordinate in the group. coord This represents the largest position coordinate in the group. The position coordinates of each individual are normalized by subtracting the smallest position coordinate from the original coordinates of the group, and then dividing by the difference between the largest and smallest position coordinates. Next, a scaling operation is performed on the normalized position coordinate data. The purpose of scaling is to map the normalized values ​​to a larger range, in this example, the range [0, 9]. Specifically, this step involves multiplying each normalized position coordinate by a scaling factor (scale).factor In this example, the scaling factor is set to 9.

[0070] Scaling can be represented by the following formula:

[0071] scaled coord =norm coord ×scale factor

[0072] scaled coord This refers to the scaled position data. Then, the processed position information is marked on a grid of specific dimensions (the grid size corresponds to the scaling range; the grid is a square grid, and the side length of the grid is the length of the range of the scaled values. In this embodiment, the range of the scaled values ​​is [0,9], i.e., the length is 10, meaning this embodiment uses a 10×10 grid). The group position interaction information within a unit time step is represented as M∈R. H×W Where H and W are both 10, representing the spatial relationships and distribution of the population at each time step, the positional information of each individual in the population over the entire time step can be represented as L∈R. N×C×W×H×T Where N is the number of individuals in the group, T is the time step (200 in this example), C is the number of channels (12 channels including those from the wrist and body), and W and H are the grid size at each time point (10×10 in this example). If all the position coordinates of an individual are 0, then all the processed position data are set to 0, indicating that this person does not exist.

[0073] Then, a 3D convolutional neural network is used to extract features from the processed location data, and BLSTM is used to model the temporal information to obtain the group location interaction features. The 3D convolutional neural network used includes two 3D convolutional layers and two pooling layers, where the kernel size of each convolutional layer is set to 3. The final group location interaction features can be obtained by the following formula:

[0074] F inter =BLSTM(Conv3D(Conv3D(L,W,b)))

[0075] Where L is the network input, representing the group location interaction information, W is the convolution kernel weight, b is the bias term, Conv3D represents a 3D convolutional layer, and the group location interaction features F are obtained by performing two 3D convolution operations and then using a bidirectional long short-term memory neural network (BLSTM network) on the results. interThe obtained group location interaction features and individual behavior features are fused using attention-weighted features to obtain enhanced individual behavior features, denoted as individual behavior enhanced features. Finally, all enhanced individual features are concatenated to obtain the group behavior features. The attention-weighted fusion process is expressed by the following formula:

[0076]

[0077]

[0078]

[0079]

[0080] in, Q is the weight matrix that the i-th individual needs to learn during weighted fusion. i K i V i yes and F inter The result of multiplying the dot product with the corresponding weight matrix represents the weight obtained when the i-th individual feature is fused with the location interaction feature. k For K i The size of the dimension. Representation matrix K i The transpose of F * F represents the group behavior feature obtained by weighted fusion of n individual behavioral features and group location interaction features, followed by concatenation. * ∈R B×C×T Where B is the training batch size, C is the number of channels, and T is the time step. Softmax is a commonly used activation function that normalizes a numerical vector into a probability distribution vector, where the sum of the probabilities is 1. F represents the individual behavioral characteristics of the i-th individual. inter This represents the group's location interaction characteristics, where n is the total number of individuals in the population.

[0081] Step 3: A spatiotemporal feature enhancement module is constructed using a CAMLP-mixer network to enhance the group behavior features, resulting in enhanced group behavior features. These enhanced features are then processed by a fully connected layer to output the group behavior recognition result. To enable the network to automatically select important individuals and increase their contribution to group activity classification while ignoring redundant information, a CAMLP-mixer network structure is proposed. This allows the group enhancement module to focus on the relationships between features and learn the importance of different features.

[0082] like Figure 3As shown, this embodiment proposes a CAMLP-mixer (Channel Attention based MLP-mixer) network as the core structure of the spatiotemporal feature enhancement module. The CAMLP-mixer network allows global spatial and temporal feature communication between different individuals in a group through a multilayer perceptron (MLP). It then overlays the global information between different individuals onto the long-term dependencies of the entire group. In other words, the CAMLP-mixer network achieves information propagation in both the spatial and temporal domains within the group, thereby obtaining an enhanced group feature representation. Specifically, different individuals have varying degrees of correlation for the task of group activity recognition. The CAMLP-mixer network can automatically select important individuals and enhance their contribution to group behavior classification while ignoring redundant information. The channel attention mechanism added to the network selectively emphasizes interdependent individual feature channels by integrating relevant features from the group features, thus contributing to more accurate classification results.

[0083] The spatiotemporal feature enhancement module includes three cascaded CAMLP-mixers. By fusing attention-weighted features from group location interaction features and individual features, it obtains location-enhanced individual behavior features. Finally, all these enhanced individual features are concatenated to form the group behavior features. These group behavior features are used as input to the spatiotemporal feature enhancement module. After obtaining the enhanced group behavior features, a fully connected layer is applied to obtain the group behavior classification result. The CAMLP-mixer consists of components from two cascaded MLPs, and its basic unit can be represented by the following formula:

[0084] Mix=Norm(σ((F * ) T W t1 W t2 )

[0085] Among them, (F * ) T This indicates that the output of the location-level feature enhancement module is transposed, where σ is the ReLU activation function and W... t1 and W t2 This indicates a 2-layer MLP operation.

[0086] CAMLP-mixer can be represented by the following formula:

[0087] F = x(E, CA(E)) + E

[0088] E represents the result of a mix operation on the group features, CA is the channel attention process, x represents the element-wise multiplication of the results of E and CA, and F is the enhanced group feature used for group behavior recognition. By incorporating channel attention, the network can automatically filter out key individuals and enhance their influence on group behavior classification. Each channel mapping of high-level features can be viewed as a response to a specific category, and there are correlations between different semantic responses. By leveraging the interdependencies between channel mappings, we can emphasize interrelated feature mappings and improve the feature representation of specific semantics.

[0089] The CA processing procedure can be represented by the following formula:

[0090]

[0091]

[0092] First, linear transformations are performed on the input feature E to obtain feature maps B, C, and D. Matrix multiplication is then performed between B and C. Finally, a softmax layer is applied to compute the channel attention map X, where X∈R. N×N N is the number of channels in the feature, x kj The influence of the k-th channel on the j-th channel was measured. x represents kj The transpose of x kj transpose and D k Perform matrix multiplication, then multiply the result by a scaling parameter β, which gradually learns a weight value from 0, and then perform E... k The element-wise summation operation is performed to obtain the final output attention map M. The final feature of each channel is a weighted sum of all channels and the original features, thus modeling the long-range semantic dependencies between feature maps, which helps to improve the discriminability of features.

[0093] It should be noted that when training the network model, the training sample data are all labeled, including individual behavior labels and group behavior labels, so that they can be used to calculate the loss function, which will not be elaborated here.

[0094] This invention conducts experiments on a group behavior dataset constructed using the UT-Data dataset and a self-created GARSENSORS dataset to verify the effectiveness of the proposed method on both datasets and compare it with traditional methods for processing time-series data. Considering that the number of people in a group is not constant in reality, the experiment sets the group size to a random 3-5 people to effectively verify the robustness of the network. The accuracy, precision, recall, and average F1 score are measured after 10 repetitions using five-fold cross-validation to evaluate the recognition performance. The experimental results are shown in Table 1.

[0095] Table 1 Experimental Results

[0096]

[0097] In Table 1, CNN represents using only convolutional neural networks for feature extraction and then completing the recognition task; BLSTM represents using only bidirectional long short-term memory networks for recognition; CNN+BLSTM represents using convolutional neural networks first and then bidirectional long short-term memory networks for recognition; GCN represents using only graph convolutional networks for feature extraction and then completing the recognition task. TJAMSD is the method proposed in the paper "Two-Domain Joint Attention Mechanism Based on Sensor Data for Group Activity Recognition" (R.Huan, A.Bo, J.Shu, P.Chen and R.Liang, "Two-Domain Joint Attention Mechanism Based on Sensor Data for Group Activity Recognition," in IEEE Transactions on Instrumentation and Measurement, vol.72, pp.1-15, 2023, Art no.2507315, doi:10.1109 / TIM.2023.3246469.).

[0098] The results of experiments on five methods across two datasets show that, on the UT-Data dataset, the accuracy of the proposed method reaches 91.85%, which is the best result to date. Compared to TJAMSD, another group behavior recognition method based on sensor data, the accuracy of the proposed method is improved by 8.59%. Furthermore, the precision, recall, and F1 score of the proposed method are improved by 6.11%, 8.17%, and 6.77%, respectively, compared to the TJAMSD method. On the GARSENSORS dataset, the accuracy of the proposed method reaches 96.59%, with improvements of 0.89%, 1.58%, 1.6%, and 1.57% in precision, recall, and F1 score, respectively, compared to the TJAMSD method.

[0099] In summary, the method of this invention has certain advantages over other algorithms. This method can more comprehensively understand the group behavior reflected in sensor data and efficiently extract group features from sensor data, thereby significantly improving the accuracy and robustness of group behavior recognition.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for recognizing group behavior based on sensor data using multi-level feature enhancement, characterized in that, The sensor data group behavior recognition method based on multi-level feature enhancement includes: Step 1: For each individual, the continuous sensor data collected by the sensor at the local location of the individual is segmented by a sliding window to obtain the sensor data at the local location of the individual. The sensor data at different local locations of the same individual are then input into the individual feature extraction network to obtain the individual's behavioral features. Step 2: Preprocess the location data of each individual to obtain group location interaction information, characterize the group location interaction information to obtain group location interaction features, fuse the individual behavior features of an individual and the group location interaction features to obtain the individual behavior enhancement features of that individual, and splice the individual behavior enhancement features of all individuals in the group to obtain the group behavior features. Step 3: Construct a spatiotemporal feature enhancement module using a CAMLP-mixer network, enhance the group behavior features using the spatiotemporal feature enhancement module to obtain group behavior enhanced features, and then output the group behavior recognition result through a fully connected layer; The spatiotemporal feature enhancement module includes multiple cascaded CAMLP-mixer networks, and the operation of each CAMLP-mixer network is represented as follows: In the formula, The output features of the CAMLP-mixer network, This indicates the element-wise dot product operation. express Operation results This indicates channel attention processing operations; Among them, the The calculation process for the operation result is as follows: In the formula, for Operation results For NORM functions, Represents the ReLU activation function. Indicates characteristics of group behavior. and This represents a 2-layer MLP operation; The calculation process for the channel attention processing operation is as follows: In the formula, Indicates the first The first channel is for the first The influence of each channel on input features The feature maps are obtained by performing linear transformations respectively. , , Feature map , , subscript , These feature maps represent different channels, and ,Right now Representation of feature map The first in One channel, Representation of feature map The first in One channel, Representation of feature map The first in One channel, Representing input features Total number of channels For the final attention graph, the first The weight of each channel, For scale parameters, for The transpose of the matrix, Input features The first in One channel.

2. The method for group behavior recognition based on sensor data using multi-level feature enhancement according to claim 1, characterized in that, The individual feature extraction network includes a spatiotemporal graph convolutional network and a bidirectional long short-term memory neural network. First, the spatiotemporal graph convolutional network is used to extract temporal and spatial features from sensor data at different local locations to obtain local individual behavior features. Then, after feature fusion of local individual behavior features at different local locations of the same individual, the bidirectional long short-term memory neural network is used to obtain the individual behavior features.

3. The sensor data group behavior recognition method based on multi-level feature enhancement according to claim 2, characterized in that, The spatiotemporal graph convolutional network uses a graph convolutional network for spatial feature extraction. The extraction process is as follows: In the formula, For the extracted spatial features, Represents the ReLU activation function. Represents 2D convolution. Represents the input data tensor. This represents the adjacency matrix in a graph convolutional network. Represents the input data tensor Adjacency Matrix Multiply, Indicates a downsampling operation; The spatiotemporal graph convolutional network uses a temporal convolutional network for temporal feature extraction, and the extraction process is as follows: In the formula, For the extracted time features, Indicates spatial features Perform a 1D convolution operation with a kernel size of 1.

4. The method for group behavior recognition based on sensor data using multi-level feature enhancement according to claim 1, characterized in that, The process of preprocessing the location data of each individual to obtain group location interaction information includes: For a unit time step, the location data of each individual is converted into a value within the standard range of [0,1] through a normalization operation to obtain normalized location data. Then, the normalized location data is scaled to obtain scaled location data. Finally, the scaled location data of all individuals are marked on a grid of a specific size to obtain grid information representing the spatial relationship and location distribution of the group in a time step. The location data at all time steps are processed to obtain the grid information corresponding to each time step. The grid information of all time steps is then stitched together to obtain the group location interaction information.

5. The sensor data group behavior recognition method based on multi-level feature enhancement according to claim 1, characterized in that, The process of characterizing group location interaction information to obtain group location interaction features includes: The group's positional interaction information is extracted by using a 3D convolutional neural network, and then the output of the 3D convolutional neural network is modeled by a bidirectional long short-term memory neural network to obtain the group's positional interaction features.

6. The method for group behavior recognition based on sensor data using multi-level feature enhancement according to claim 1, characterized in that, The individual behavioral features of the fused individual and the group positional interaction features are used to obtain the individual behavioral enhancement features of that individual. The group behavioral features are obtained by splicing together the individual behavioral enhancement features of all individuals in the group, including: In the formula, , , This represents the weight matrix for the i-th individual during fusion. , , yes and The result of performing a dot product operation and then multiplying it by the corresponding weight matrix. for Dimension size, It is an activation function. This represents the individual behavioral characteristics of the i-th individual. Indicates the location interaction characteristics of the group. express The transpose of the matrix, As a characteristic of group behavior, This represents the total number of individuals in the population.

Images