A multi-instance behavior recognition method and system based on an attention mechanism

Abstract

The application provides a multi-instance behavior recognition method based on an attention mechanism, comprising the following steps: obtaining known behavior data, generating a plurality of behavior data packets, marking instances of the behavior data packets, and constructing a training set; constructing a recognition model and training the recognition model by using the training set; the recognition model comprises a feature extractor, an encoder and a classifier, wherein the encoder uses a multi-head self-attention mechanism to assign weights to instances in the behavior data packet; and the recognition model is used for human behavior recognition. The application also provides a multi-instance behavior recognition system based on an attention mechanism and a data processing device for realizing multi-instance behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to a multi-instance behavior recognition method and system based on an attention mechanism. BACKGROUND

[0002] Human behavior recognition technology is widely used in biomedical, disease rehabilitation and other fields. Due to the low cost, non-invasive and other characteristics of wearable sensors, wearable sensors are widely used in human behavior recognition activities. In recent years, with the popularity of wearable devices, human behavior recognition technology based on wearable devices provides an efficient and convenient solution for long-term health monitoring. At present, many wearable behavior recognition methods are modeled based on a supervised manner. This supervised modeling method needs to collect a large amount of labeled data for training, and the fine-grained labeling of data is a very tedious work.

[0003] The lack of label granularity makes the human behavior recognition problem become a weakly supervised learning problem, specifically an imprecise supervision problem in weakly supervised classification, and multi-instance learning (MIL) is a typical solution to this problem. Unlike fully supervised learning, MIL only needs to label instance bags, not all instances. If all instances in the bag are negative, the bag can be labeled as negative. Correspondingly, if at least one instance is positive, the bag is labeled as positive. Dietterich et al. first applied multi-instance learning to the drug activity prediction problem. At the same time, some classic machine learning methods are used for multi-instance learning, such as kNN, EM-DD, MI-SVM, miGraph, MIForests, MIBoosting. In recent years, with the rise of deep learning, multi-instance learning methods based on deep neural networks have received widespread attention. Maximilian et al. proposed a deep neural network based on attention to fully parameterize the bag label probability, describing the MIL problem as learning the Bernoulli distribution of bag labels.

[0004] Currently, there are two main ways to label wearable data: behavior data labeling under specific conditions and behavior data labeling in daily life.

[0005] 1. The behavior data labeling under specific conditions has the following problems: the labeling process is tedious and time-consuming, the boundary division problem between different activities in the behavior, and some sudden behaviors cannot be accurately simulated under laboratory conditions.

[0006] Under certain conditions, in order to obtain labeled data, volunteers wearing wearable devices need to perform a specified behavior containing one or several activities in a specific occasion within a specific time, and sensors collect data from it. People divide these data into single activities by marking the start and end time of each activity. In order to accurately obtain the corresponding label of the activity, researchers usually first record the entire behavior in a certain way (such as video recording, etc.), and then determine the accurate start and end time of each activity by repeatedly watching the record later, and finally label the activity. This is a very tedious and time-consuming process, and may cause time alignment problems.

[0007] Human behavior is a continuous and complex process, and people usually make a series of actions in a period of time. However, even in the case of video recording, the boundary between human activities is difficult to determine, and the label divided in this way is inaccurate. Modeling with inaccurate label data will affect the effect of the model.

[0008] In addition, some sudden behaviors cannot be simulated in laboratory conditions. For example, no one knows when the fall state will occur, so we cannot accurately label the real fall behavior. At the same time, due to the particularity of the fall behavior, we cannot accurately simulate it. At present, almost all public fall data sets are simulated falls rather than real falls. The model obtained by modeling with simulated fall data may not be consistent with the model of real fall.

[0009] 2. There are problems in the labeling of behavior data in daily life: it is impossible to perform fine-grained labeling, and real-time data labeling behavior seriously affects user experience.

[0010] In daily life, due to the high cost of data labeling process, individuals cannot label activities every moment. At the same time, explicit labeling in activities will greatly affect user experience and data accuracy. Therefore, a more realistic daily behavior labeling method is coarse-grained time period labeling. That is: after the corresponding occurrence, label a longer period of time before. Still taking fall as an example, we can label it in a retrospective way after the fall occurs, such as: there was a fall in a certain period of time. However, due to the short duration of the fall, and the fact that the time period contains many other actions, the labeling is not as accurate as expected. In other words, in the typical scenario of human behavior recognition, only coarse-grained label information is available, and such coarse-grained labeling data cannot be used for supervised modeling.

[0011] The existing multi-instance method has the problems of same learning weight or ignoring time correlation. Generally, MIL is more difficult than fully supervised binary classification. Because there are negative instances in both negative bags and positive bags, it is unknown which instances in the positive bag are positive instances and which are negative instances. The ordinary machine learning method is difficult to distinguish the positive and negative instances in the positive bag due to the same learning weight of the instances in the bag, resulting in poor model effect.

[0012] Most previous work on multi-instance learning mainly faces discrete instances, in which the instances in the bag are independent of each other. In order to solve the problem of spatial correlation between instances in image classification, Shao et al. proposed a new framework called Correlated MIL, which explores morphological and spatial information. However, human behavior recognition data based on sensors is typical continuous data with strong time correlation, rather than spatial correlation. As an important feature of time series data, ignoring time correlation will seriously affect the accuracy of the model. SUMMARY

[0013] In view of the model construction problem of time series data under the condition of insufficient label granularity, the present application proposes a multi-instance behavior recognition method based on attention mechanism, comprising: obtaining known behavior data, generating a plurality of behavior data bags, marking the instances of the behavior data bags, and constructing a training set; constructing a recognition model, and training the recognition model with the training set; the recognition model comprises a feature extractor, an encoder and a classifier, wherein the encoder assigns weights to the instances in the behavior data bag through a multi-head self-attention mechanism; and the recognition model is used for human behavior recognition.

[0014] The multi-instance behavior recognition method disclosed by the present application, wherein the encoder comprises a plurality of same and end-to-end connected layers, each layer comprising a multi-head self-attention layer and a fully connected feedforward network layer, and the adjacent two layers are connected through a residual network and a normalization network; the encoder comprises a feature encoder, and the feature encoder obtains multi-head self-attention information MHA(Q, K, V) from M positions of a feature set W O of the behavior data bag, performs softmax normalization on MHA(Q, K, V) to obtain an input matrix of the classifier M , MHA(Q, K, V) = Concat(Head1,..., Head O ,..., Head i ), W q , W k , W v , q , Head Oquery linear transformation to obtain an attention parameter matrix Q, W k represents a key linear transformation to W O v represents a value linear transformation to W O q represents the dimension of Q.

[0015] The multi-instance behavior recognition method, wherein the encoder further comprises a position encoder, and the position encoder obtains the position encoding PE pos O W O′ by performing feature fusion on W O′ O + αPE pos , PE pos includes PE (pos,2i) and PE (pos,2i+1) ,

[0016]

[0017]

[0018] 2i and 2i+1 represent instance dimensions, λ represents a hyperparameter for adjusting the importance of time sequence, d model represents the feature dimension, and α represents a hyperparameter for weighted fusion.

[0019] W O′ is obtained by the feature encoder

[0020] The multi-instance behavior recognition method, wherein the classifier takes as input, uses a multi-layer fully connected network for classification, and obtains a classification probability corresponding to each human behavior category; and a human behavior category with the maximum classification probability is taken as the recognition result of the human behavior.

[0021] The multi-instance behavior recognition method, wherein the feature extractor comprises: a plurality of processing channels corresponding to human behavior sensors, each processing channel uses a two-layer CNN network for feature extraction, and a ReLu function is used as an activation function between the two-layer CNN networks; and a fully connected feature fusion network that fuses features obtained by all the processing channels to form a feature set W O of the behavior data packet.

[0022] The multi-instance behavior recognition method, wherein the human behavior sensor is a wearable sensor. ​​​​

[0023] The multi-instance behavior recognition method provided by the application, wherein after the known behavior data is obtained, the known behavior data is subjected to an outlier rejection operation, a missing value completion operation and a sliding window interception operation, all data in each sliding window is taken as an instance, and a plurality of continuous instances constitute the behavior data package; if at least one instance in the behavior data package is a positive instance for a behavior, the behavior data package is marked as a positive example with respect to the behavior; if all known behavior data in the behavior data package are negative instances for the behavior, the behavior data package is marked as a negative example with respect to the behavior.

[0024] The application further provides a multi-instance behavior recognition system based on an attention mechanism, comprising: a training set construction module, configured to obtain known behavior data, generate a plurality of behavior data packages, mark instances in the behavior data packages and construct a training set; a model construction module, configured to construct a recognition model and train the recognition model by using the training set; the recognition model comprises a feature extractor, an encoder and a classifier, wherein the encoder uses a multi-head self-attention mechanism to assign weights to the instances in the behavior data package; and a behavior recognition module, configured to recognize human behaviors by using the recognition model.

[0025] The application further provides a computer readable storage medium storing computer executable instructions, characterized in that when the computer executable instructions are executed, the multi-instance behavior recognition based on the attention mechanism is realized.

[0026] The application further provides a data processing apparatus comprising the computer readable storage medium, when a processor of the data processing apparatus invokes and executes the computer executable instructions in the computer readable storage medium, the data processing apparatus realizes the multi-instance behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 FIG. 1 is a flow chart of the multi-instance behavior recognition method based on the attention mechanism of the application.

[0028] Figure 2 FIG. 2 is a structural schematic diagram of the multi-instance behavior recognition model of the application.

[0029] Figure 3 FIG. 3 is a schematic diagram of the self-attention relationship between an instance and all instances in a package of the application.

[0030] Figure 4 FIG. 4 is a schematic diagram of the data processing apparatus of the application. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given below with reference to the drawings. It should be understood that the specific implementation methods described herein are only used to explain the present application and not to limit the present application.

[0032] The purpose of the present application is to solve the problems (defects) of the prior art described above, and to propose a multi-instance behavior recognition method for coarse-grained time series data. This method solves the problem of lack of time correlation representation of intra-bag time series data in the multi-instance learning method by using position encoding to reasonably represent the time correlation. By introducing the self-attention mechanism, the weight distribution problem of intra-bag instance training is solved. By classifying the human behavior recognition problem as an imprecise supervision problem, a weakly supervised learning method is used for modeling, avoiding the problem of ambiguous boundaries between activities.

[0033] During the research of the multi-instance behavior recognition method for coarse-grained time series data, the inventors found that the prior art did not consider the time correlation of intra-bag instances and the learning weight of instances in multi-instance learning. The inventors found that this defect can be solved by introducing position encoding and attention mechanism in the transformer model. This method is based on a deep neural network algorithm, which fully utilizes the two key modules of position encoding and self-attention mechanism to solve the problem of human behavior recognition under coarse-grained labels. The former is used to embed the time correlation between instances and add the correlation between intra-bag instances to the features of the instances when aggregating in MIL; the latter is used to further assign weights to each instance to highlight the instances that can better represent the bag label, and the weight ratio of the two is determined by the correlation within the data bag. When the data correlation is strong, both feature encoding and time correlation (position encoding) are used for joint modeling; when the data correlation is weak or non-existent, the formula degenerates into a multi-instance learning model using only feature encoding.

[0034] The recognition model of the present application includes a feature fusion and extraction module, an encoder module and a classification module. In the feature fusion and extraction module, a two-layer convolutional neural network (CNN) is used to extract instance-level features, and then a fully connected layer is used for fusion; in the encoder module, the intra-bag instances are input into a deep neural network model, which is a variant of a transformer encoder, used to find key instances and represent the time series information between instances. The attention mechanism can find key instances by assigning weights to different instances. At the same time, the time series information between instances is embedded in the module through position encoding; in the classification module, a multi-layer fully connected network is used for classification to complete the prediction task.

[0035] The application utilizes position coding embedding in the transformer to model instances with time sequence relationship in the bag, compared with most discrete instance-oriented multi-instance learning models, the application fully considers the time correlation of instances in the bag and embeds it as a feature into the model, utilizes the multi-head self-attention mechanism to assign weights to the instances in the bag, in this way, the model selectively trains the instances in the bag, increases the weight training for key instances such as positive instances in the positive bag, and obtains an optimal model, and the human behavior recognition problem with coarse-grained labels is modeled as an inaccurate supervision problem, so as to fully consider the continuity of human behavior and the fuzziness of the boundary between activities, and model by using a weakly supervised learning method.

[0036] Figure 1 is a flowchart of the multi-instance behavior recognition method based on the attention mechanism of the application. As shown in Figure 1 , the multi-instance behavior recognition method based on the attention mechanism of the application comprises:

[0037] Step S1, original data acquisition

[0038] The data used by the application is derived from wearable sensors, and the sensor types include but are not limited to: accelerometer, gyroscope, magnetometer. All data are stream data containing time information.

[0039] Step S2, data preprocessing

[0040] Data preprocessing includes data cleaning, data completion, sliding windowing, and packet processing of the original data. Specifically, it includes:

[0041] 1) Clean the original data. By rejecting outliers and filling in missing values, the entire data set meets the model training requirements.

[0042] 2) According to the sensor sampling frequency, the stream data is slidingly windowed. The window size is 1s, and the sliding distance is 0.5s. All data in each window are taken as an instance, and multiple consecutive instances jointly constitute a data packet.

[0043] 3) Through coarse-grained labeling, the data is divided into positive and negative packets, wherein the data packet containing positive instances is a positive packet, and the data packet with all instances being negative instances is a negative packet.

[0044] Step S3, model construction and training

[0045] The recognition model of the application includes a feature extractor, an encoder, and a classifier. The recognition model structure is as shown in Figure 2 :

[0046] 1) Instance-level feature fusion and feature extraction by feature extractor. In order to process multi-channel and long sequence behavior data, the first step of the recognition model is feature fusion and feature extraction. Channels are composed of different sensors, and two-layer CNN networks are used for feature extraction in each channel. ReLu is selected as the activation function between the two-layer network to ensure nonlinearity. In this way, not only the context information of the original data is preserved, but also the feature dimension is reduced. In addition, since the network independently performs feature extraction, it ensures the independence between channels. Assuming that there are k instances in each bag, feature extraction is performed on each of them respectively, obtaining a 1*512 feature array, and combining the feature arrays of all instances obtains a k*512 feature matrix. Then, a fully connected network is used to fuse the features of different channels to form a bag-level feature set matrix W O .

[0047] 2) Obtain the input matrix of the classifier by the encoder. The encoder of the present application includes a feature encoder and a position encoder, the feature encoder is used to select key instances and find the order information between instances in the bag, and the position encoder uses position encoding as an additional feature to represent the time sequence; when the data correlation is strong, the recognition model simultaneously uses the position encoder and the feature encoder, when the data correlation is weak or there is no correlation, the recognition model only uses the feature encoder.

[0048] There are N identical layers in the feature encoder, which are connected in a loop. Each layer is composed of two sub-layers, one of which is a multi-head self-attention layer and the other is a fully connected feedforward network layer, which are connected through a residual network and a normalization network. The self-attention mechanism is the focus of the encoder, which can reference the context information of all instances in the bag, which is obviously different from some other time series modeling methods such as HMM. Finally, a row-wise softmax normalization function is used to process the weight of the instance. Specifically, it includes:

[0049] ① Input: W O

[0050] ② Use the self-attention mechanism to operate W O to obtain W q , W k , W v , and finally obtain the self-attention weight Attention(W q , W k , W v );

[0051] ③ Introduce multi-head attention mechanism, combine multiple Attention(W q , W k , W v ) with W OMultiplication, get multi-head attention MHA(Q, K, V);

[0052] IV. The obtained MHA(Q, K, V) is normalized by softmax to obtain the output matrix W O is consistent in dimension.

[0053] The self-attention weight can be defined as:

[0054]

[0055] Where, matrix Q, K, V represent a set of attention parameter matrix. W q represents the query linear transformation of W O to obtain matrix Q, W k represents the key linear transformation of W O to obtain matrix K, W v represents the value linear transformation of W O to obtain matrix V. d q is the dimension of the attention parameter matrix Q, which is equal to the dimension of W q . The dot product of the query is calculated with all the keys, so that the relationship between an instance and all instances in the bag can be realized, as shown in Figure 3 ;

[0056] The multi-head attention mechanism calculates the attention of different positions continuously, so that the recognition model can obtain information from different representation subspaces of different positions, defined as:

[0057]

[0058] Where M represents the number of multi-head attention, that is, the feature encoder obtains M positions of the feature set W O from the behavior data bag.

[0059] The above operation ensures that each instance can be associated with other instances, but in time series data, the time order between instances is very important. Therefore, the position encoder uses position encoding as an additional feature to represent the time order. The time series information is encoded into a form consistent with the feature encoding dimension, and it is weighted and added with the feature encoding. The calculation function of the position encoding is as follows:

[0060]

[0061] Where pos is the time order, 2i and 2i+1 represent the instance dimension, d model represents the dimension of the feature encoding. λ is a hyperparameter that adjusts the importance of the time order. PE (pos,2i) and PE(pos,2i+1) Commonly combined into PE pos .

[0062] In the case of considering the inter-instance timing correlation, it is weighted and added with the transformed feature encoding to obtain W O′ , instead of W O in formula (2)

[0063] W O′ = W O + alpha PE pos (4)

[0064] 3) Obtain the classification result through the classifier. This module uses a multi-layer fully connected network to classify the packet. Specifically, it includes:

[0065] ①Take the obtained as input, and use a two-layer fully connected network for classification to finally obtain the classification probability of each class.

[0066] ②Select the classification with the maximum probability as the final classification result of the recognition model.

[0067] 4) Train the recognition model with the data packets obtained in step S2.

[0068] Step S4, prediction classification

[0069] Use the recognition model generated in step S3 to perform prediction classification operation.

[0070] The present application uses python for modeling, which is realized in Pytorch

[39] , and all experiments are completed on an RTX3090 GPU server. In terms of evaluation indicators, considering that the data set may have an unbalanced situation, accuracy, recall, precision and F1 value are used to evaluate the classification performance.

[0071] (I) Main module setting:

[0072] Feature fusion and feature extraction module. A double-layer one-dimensional convolutional network is used to extract features for each channel, the convolution kernel size is 20, and a max pooling layer is used, and ReLu is selected as the activation function of each sub-layer. Then connect with a fully connected layer, finally flatten all instances to obtain a 1x512 feature representation.

[0073] Encoder module. During the encoding training, the dimension of each feature embedding is reduced from 512 to 64 dimensions by a fully connected layer, and then divided into 8 parts. 8 identical multi-head self-attention layers process these data in an end-to-end manner. The temporal order between instances is represented as the position encoding added to each multi-head self-attention layer. Training decay is performed using a rate of drop = 0.1.

[0074] Classification module. In the classification step, a two-layer fully connected network and softmax are used to classify and normalize the prediction scores for each class.

[0075] (ii) Experimental results:

[0076] By comparing with other multi-instance learning methods on two public data, the results show that the method of the present application achieves the best results on almost all evaluation indicators, as shown in Table 1.

[0077]

[0078] Table 1: Comparison table of experimental results

[0079] Figure 4 is a schematic diagram of the data processing apparatus of the present application. As shown in Figure 4 The computer readable storage medium of the present application stores computer executable instructions, and when the computer executable instructions are executed by the processor of the data processing apparatus, the above-mentioned multi-instance behavior recognition based on attention mechanism is realized. Those skilled in the art can understand that all or part of the steps in the above method can be instructed by a program to complete related hardware (such as processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as read-only memory, magnetic disk or optical disk, etc. All or part of the steps of the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in the form of hardware, such as by integrated circuits to implement its corresponding functions, or in the form of software function modules, such as by the processor executing the program / instructions stored in the memory to implement its corresponding functions. The embodiments of the present application are not limited to any specific form of combination of hardware and software.

[0080] The application provides a multi-instance behavior recognition method and system based on an attention mechanism, which models a human behavior recognition task as an imprecise supervised learning task, and uses two key modules of position encoding and a self-attention mechanism for human activity recognition under a coarse-grained label. The position encoding is used to embed the time correlation between instances and is added to the instance embedding when aggregated in MIL; the self-attention mechanism is used to further assign aggregation weights to each instance to highlight the instances that can better represent the bag label. Experiments conducted on public benchmark datasets show that the multi-instance behavior recognition method based on the attention mechanism is superior to other existing methods in terms of accuracy, F1-Score and other indicators.

[0081] The above embodiments are only used to illustrate the application, rather than limit the application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the application. Therefore, all equivalent technical solutions belong to the scope of the application, and the patent protection scope of the application should be defined by the claims.

Claims

1. A multi-instance behavior recognition method based on an attention mechanism, characterized in that, include: Acquire known behavioral data, generate multiple behavioral data packets, label the behavioral data packets as instances, and construct a training set; A recognition model is constructed and trained using the training set. The recognition model includes a feature extractor, an encoder, and a classifier. The encoder assigns weights to instances within the action data packet through a multi-head self-attention mechanism. The encoder includes multiple identical and interconnected layers, a feature encoder, and a position encoder. Each layer includes a multi-head self-attention layer and a fully connected feedforward network layer. Adjacent layers are connected through a residual network and a normalization network. The feature encoder extracts the feature set from this behavior data packet. Obtain multi-head self-attention information from M positions. ,Will Perform softmax normalization to obtain the input matrix of the classifier. The position encoder uses position encoding PE. pos right Perform feature fusion to obtain the fused feature set. ,Will This is obtained through the feature encoder. ; in, , , , Indicates to The query linear transformation is used to obtain the attention parameter matrix. , Indicates to The key linear transformation is used to obtain the attention parameter matrix. , Indicates to The value linear transformation is used to obtain the attention parameter matrix. , , include and , , , express The dimensions, 2i and 2i+1 represent the instance dimensions. Hyperparameters that indicate the importance of adjusting time sequence Representing feature dimension, The hyperparameters representing weighted fusion; This recognition model is used for human behavior recognition.

2. The multi-instance behavior recognition method as described in claim 1, characterized in that, The classifier is As input, a multi-layer fully connected network is used for classification to obtain... The classification probability of each human behavior category is assigned; the human behavior category with the highest classification probability is used as the recognition result.

3. The multi-instance behavior recognition method as described in claim 1, characterized in that, The feature extractor includes: Multiple processing channels corresponding to human behavior sensors, each processing channel uses a two-layer CNN network for feature extraction, and the ReLU function is used as the activation function between the two layers of the CNN network; A fully connected feature fusion network fuses the features obtained from all processing channels to form the feature set W of the behavioral data packet. O .

4. The multi-instance behavior recognition method as described in claim 3, characterized in that, This human behavior sensor is a wearable sensor.

5. The multi-instance behavior recognition method as described in claim 1, characterized in that, After obtaining the known behavior data, outlier removal, missing value completion, and sliding window truncation operations are performed on the known behavior data. All data within each sliding window is considered as one instance, and multiple consecutive instances constitute the behavior data packet. If at least one instance in the action packet is a positive instance for a certain action, then the action packet is marked as a positive instance relative to that action; If all known behavior data in the behavior packet are negative instances for that behavior, then the behavior packet is marked as a negative instance relative to that behavior.

6. A multi-instance behavior recognition system based on an attention mechanism, characterized in that, include: The training set construction module is used to acquire known behavioral data, generate multiple behavioral data packets, label the behavioral data packets as instances, and construct them into a training set. The model building module is used to build a recognition model and train the recognition model with the training set. The recognition model includes a feature extractor, an encoder, and a classifier. The encoder uses a multi-head self-attention mechanism to assign weights to instances within the behavior data packet. The encoder includes multiple identical and connected layers, a feature encoder, and a position encoder. Each layer includes a multi-head self-attention layer and a fully connected feedforward network layer. Adjacent layers are connected through a residual network and a normalization network. The feature encoder extracts the feature set from this behavior data packet. Obtain multi-head self-attention information from M positions. ,Will Perform softmax normalization to obtain the input matrix of the classifier. ; The position encoder uses position encoding PE pos right Perform feature fusion to obtain the fused feature set. ,Will This is obtained through the feature encoder. ;in, , , , Indicates to The query linear transformation is used to obtain the attention parameter matrix. , Indicates to The key linear transformation is used to obtain the attention parameter matrix. , Indicates to The value linear transformation is used to obtain the attention parameter matrix. , , include and , , , express The dimensions, 2i and 2i+1 represent the instance dimensions. Hyperparameters that indicate the importance of adjusting time sequence Representing feature dimension, The hyperparameters representing weighted fusion; The behavior recognition module is used to recognize human behavior using this recognition model.

7. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the attention-based multi-instance behavior recognition as described in any one of claims 1 to 5 is implemented.

8. A data processing apparatus comprising a computer-readable storage medium as claimed in claim 7, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, the data processing apparatus implements multi-instance behavior recognition.

Images