A human behavior recognition method and device

Abstract

The application discloses a human behavior recognition method and device, and relates to the field of human behavior recognition, and comprises the following steps: acquiring sensor data, embedding the sensor data into a high-dimensional space through preprocessing, adding position information, and obtaining a preprocessed high-dimensional space sequence; outputting sequences under different scales through local window fusion according to the high-dimensional space sequence, establishing information interaction between the sequences under different scales through an attention mechanism, obtaining a representation sequence corresponding to each scale, and aggregating the representation sequence corresponding to each scale into a total representation sequence; compressing the total representation sequence into a final feature vector, and performing behavior recognition according to the final feature vector.

Description

Technical Field

[0001] This application relates to the field of human behavior recognition technology, specifically to a human behavior recognition method and device. Background Technology

[0002] Sensor-based human behavior recognition refers to using specialized equipment to monitor and analyze human behavior patterns in specific environments to achieve purposes such as human-computer interaction, security monitoring, smart homes, and healthcare management, thereby providing convenience and safety for daily life and improving quality of life. Traditional machine learning methods have achieved good results in some human behavior recognition scenarios, but they also face challenges. These challenges stem from the fact that traditional machine learning methods often rely on manually designed and selected features, a process that typically requires domain knowledge and is tedious and time-consuming. Furthermore, due to subjective factors, manually designed features may overlook some potentially important information in the data, leading to poor performance of subsequent classifiers and limited generalization ability. With the rapid development of deep learning technology in recent years, more and more human behavior recognition research has shifted its focus from traditional machine learning methods to deep learning methods. Deep learning methods have gained attention for their ability to automatically extract features. These methods can learn the most useful features from raw sensor data without human intervention, thus reducing the burden of feature engineering.

[0003] Despite the significant advantages of deep learning methods, they still have some limitations in representation learning for behavior recognition tasks. Firstly, the mainstream methods currently applied to human behavior recognition are convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which have certain limitations in representing sensor data. CNNs, due to their fixed kernel size and limited receptive field, cannot effectively extract global features. RNNs face the major problems of vanishing and exploding gradients. Although variants of RNNs such as Long Short-Term Memory (LSTM) networks and gated recurrent units (GRUs) alleviate this problem to some extent, another limitation of the recurrent structure is that it can only process sequential data, resulting in underutilization of computational resources and low computational efficiency. Summary of the Invention

[0004] This application provides a human behavior recognition method, apparatus, device, and computer-readable storage medium, which can better learn discriminative representations from sensor data to improve the accuracy of human behavior recognition and has good application prospects.

[0005] In a first aspect, embodiments of this application provide a method for human behavior recognition, the method comprising:

[0006] Acquire sensor data, embed the sensor data into a high-dimensional space through preprocessing, and add location information to obtain a preprocessed high-dimensional space sequence;

[0007] Based on the high-dimensional spatial sequence, the sequence at different scales is output through local window fusion. Information interaction is established between the sequences at different scales through an attention mechanism to obtain the representation sequence corresponding to each scale. The representation sequences corresponding to each scale are then aggregated into a total representation sequence.

[0008] The total representation sequence is compressed into a final feature vector, and behavior recognition is performed based on the final feature vector.

[0009] In some embodiments, embedding the sensor data into a high-dimensional space through preprocessing and adding location information to obtain a preprocessed high-dimensional space sequence includes:

[0010] According to the formula:

[0011] X = preprocess(U)

[0012] We obtain a high-dimensional sequence X, where The input is sensor data.

[0013] In some embodiments, the step of fusing and outputting sequences at different scales based on the high-dimensional spatial sequence through a local window includes:

[0014] According to the formula:

[0015]

[0016] Obtain sequences {X} at different scales. (1) X (2) , ..., X (S)}, where X (i) Let i represent the sequence at the i-th scale.

[0017] In some embodiments, the step of fusing and outputting sequences at different scales based on the high-dimensional spatial sequence through a local window further includes:

[0018] According to the formula:

[0019]

[0020] Obtain the interpolation result Where ω represents the window size parameter, X (i) Let L represent the sequence at the i-th scale, where L is the sequence length;

[0021] According to the formula:

[0022]

[0023] Superimposed mining

[0024] According to the formula:

[0025]

[0026] Obtain the dimensionality-reduced sequence in For parameter matrices;

[0027] According to the formula:

[0028]

[0029] Obtain the sequence X at the (i+1)th scale. (i+1) , where LN is the layer normalization model in neural networks.

[0030] In some embodiments, the step of establishing information interaction between sequences at different scales through an attention mechanism to obtain a representation sequence corresponding to each scale includes:

[0031] According to the formula:

[0032]

[0033]

[0034] Obtain the key sequence K at the i-th scale. (i) AND value sequence V (i) ,in Let be the parameter matrix at the i-th scale;

[0035] According to the formula:

[0036]

[0037]

[0038] The query sequence Q of the first scale is obtained. (1) ,in For attention function;

[0039] According to the formula:

[0040]

[0041] Obtain the representation sequence Y corresponding to the i-th scale. (i) .

[0042] In some embodiments, the step of aggregating the representation sequences corresponding to each scale into a total representation sequence includes:

[0043] According to the formula:

[0044] Y c =concat(Y (1) Y (2) , ..., Y (S) )

[0045] Obtain the stacking result Where {Y (1) Y (2) , ..., Y (S) Let} be a set of S representation sequences, where L is the length of the representation sequence and D is the feature dimension of the representation sequence;

[0046] According to the formula:

[0047] Y o =ReLU(BN(Conv(Y) c )))

[0048] Obtain the total characterization sequence

[0049] Secondly, embodiments of this application provide a human behavior recognition device, the device comprising:

[0050] The acquisition unit is used to acquire sensor data, embed the sensor data into a high-dimensional space through preprocessing, and add location information to obtain a preprocessed high-dimensional space sequence.

[0051] The processing unit is used to output sequences at different scales by fusing them through a local window according to the high-dimensional spatial sequence, establish information interaction between the sequences at different scales through an attention mechanism, obtain the representation sequence corresponding to each scale, and aggregate the representation sequences corresponding to each scale into a total representation sequence.

[0052] The identification unit is used to compress the total representation sequence into a final feature vector and perform behavior identification based on the final feature vector.

[0053] In some embodiments, the acquisition unit is used for:

[0054] According to the formula:

[0055] X = preprocess(U)

[0056] We obtain a high-dimensional sequence X, where The input is sensor data.

[0057] Thirdly, a computer device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the method described in any one of the first aspects.

[0058] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the method described in any one of the first aspects.

[0059] The beneficial effects of the technical solutions provided in this application include:

[0060] By acquiring sensor data, the data is preprocessed and embedded into a high-dimensional space, with location information added, resulting in a preprocessed high-dimensional spatial sequence. Based on this high-dimensional spatial sequence, sequences at different scales are output through local window fusion. An attention mechanism is used to establish information interaction between these sequences at different scales, obtaining a representation sequence corresponding to each scale. These representation sequences at each scale are aggregated into a total representation sequence. The total representation sequence is compressed into a final feature vector, and behavior recognition is performed based on this final feature vector. This method allows for better learning of discriminative representations from sensor data, thereby improving the accuracy of human behavior recognition. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating a human behavior recognition method according to an embodiment of the present invention;

[0062] Figure 2 This is a flowchart illustrating the multi-scale attention process in an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of the local window merging process in an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the multi-scale aggregation process in an embodiment of the present invention;

[0065] Figure 5 This is a schematic diagram of a human behavior recognition device according to an embodiment of the present invention;

[0066] Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0068] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0069] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0070] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0071] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0073] In a first aspect, embodiments of this application provide a method for human behavior recognition.

[0074] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating a human behavior recognition method according to this application. Figure 1 As shown, the method includes:

[0075] S1. Acquire sensor data, embed the sensor data into a high-dimensional space through preprocessing, and add location information to obtain a preprocessed high-dimensional space sequence;

[0076] It's worth noting that the input sensor data is embedded into a high-dimensional space through a preprocessing module, which adds additional location information. The preprocessing module comprises two sub-modules: Convolution Embedding and Positional Encoding. The Convolution Embedding sub-module embeds the input multivariate temporal data into a high-dimensional space, and it includes four main types of basic network layers: Convolutional layers (Conv), Batch Normalization layers (BatchNorm), ReLU layers, and Max Pooling layers. The Positional Encoding sub-module adds additional location information to the sequence using a learnable position matrix.

[0077] S2. Based on the high-dimensional spatial sequence, output sequences at different scales through local window fusion, establish information interaction between the sequences at different scales through an attention mechanism to obtain the representation sequence corresponding to each scale, and aggregate the representation sequences corresponding to each scale into a total representation sequence.

[0078] S3. Compress the total representation sequence into a final feature vector, and perform behavior recognition based on the final feature vector.

[0079] Specifically, the classifier module processes the total representation sequence output by the Transformer module. The classifier module first compresses the representation sequence into a final feature vector. This final feature vector is used for action recognition. Specifically, the classifier module consists of two parts: a sequential pooling layer for sequence compression and a linear layer for generating the predicted probability distribution. This process can be described as follows:

[0080] Z class =Classifier(Y o )

[0081] in, This represents the probability that the input sample corresponds to one of the C categories.

[0082] In some embodiments, embedding the sensor data into a high-dimensional space through preprocessing and adding location information to obtain a preprocessed high-dimensional space sequence includes:

[0083] According to the formula:

[0084] X = preprocess(U)

[0085] We obtain a high-dimensional sequence X, where The input is sensor data.

[0086] See Figure 2 In some embodiments, the step of fusing and outputting sequences at different scales based on the high-dimensional spatial sequence through a local window includes:

[0087] According to the formula:

[0088]

[0089] Obtain sequences {X} at different scales. (1) X (2) , ..., X (S)}, where X (i) Let i represent the sequence at the i-th scale.

[0090] See Figure 3 In some embodiments, the step of fusing and outputting sequences at different scales based on the high-dimensional spatial sequence through a local window further includes:

[0091] According to the formula:

[0092]

[0093] Obtain the interpolation result Where ω represents the window size parameter, X (i) Let L represent the sequence at the i-th scale, where L is the sequence length;

[0094] It is worth noting that the following relationship must be satisfied:

[0095]

[0096] Where L′ represents the length of the interpolated sequence. The parameter ω controls the number of windows.

[0097] By stacking feature vectors within the same window into a single vector, the sequence is simplified. Concatenating ω feature vectors within a window along the feature dimension reduces the length of the input sequence while increasing the feature dimension.

[0098] According to the formula:

[0099]

[0100] Obtain the superposition result

[0101] According to the formula:

[0102]

[0103] Obtain the dimensionality-reduced sequence in This is the parameter matrix; it can be optimized along with other model parameters using gradient descent. Simultaneously, this parameter matrix can adaptively select important features and provide efficiency advantages for subsequent attention calculations.

[0104] According to the formula:

[0105]

[0106] Obtain the sequence X at the (i+1)th scale. (i+1) , where LN is the layer normalization model in neural networks.

[0107] In some embodiments, the step of establishing information interaction between sequences at different scales through an attention mechanism to obtain a representation sequence corresponding to each scale includes:

[0108] According to the formula:

[0109]

[0110]

[0111] Obtain the key sequence K at the i-th scale. (i) AND value sequence V (i) ,in Let be the parameter matrix at the i-th scale;

[0112] According to the formula:

[0113]

[0114]

[0115] The query sequence Q of the first scale is obtained. (1) ,in For attention function;

[0116] According to the formula:

[0117]

[0118] Obtain the representation sequence Y corresponding to the i-th scale. (i) .

[0119] See Figure 4 In some embodiments, the step of aggregating the representation sequences corresponding to each scale into a total representation sequence includes:

[0120] According to the formula:

[0121] Y c =concat(Y (1) Y (2) , ..., Y (S) )

[0122] Obtain the stacking result Where {Y (1) Y (2) , ..., Y (S) Let} be a set of S representation sequences, where L is the length of the representation sequence and D is the feature dimension of the representation sequence;

[0123] According to the formula:

[0124] Y o =ReLU(BN(Conv(Y) c )))

[0125] Obtain the total characterization sequence

[0126] Secondly, embodiments of this application also provide a human behavior recognition device.

[0127] In one embodiment, reference is made to Figure 5 , Figure 5 This is a schematic diagram of the functional modules of an embodiment of the applicant's human behavior recognition device. Figure 5 As shown, the device includes:

[0128] The acquisition unit is used to acquire sensor data, embed the sensor data into a high-dimensional space through preprocessing, and add location information to obtain a preprocessed high-dimensional space sequence.

[0129] The processing unit is used to output sequences at different scales by fusing them through a local window according to the high-dimensional spatial sequence, establish information interaction between the sequences at different scales through an attention mechanism, obtain the representation sequence corresponding to each scale, and aggregate the representation sequences corresponding to each scale into a total representation sequence.

[0130] The identification unit is used to compress the total representation sequence into a final feature vector and perform behavior identification based on the final feature vector.

[0131] In some embodiments, the acquisition unit is used for:

[0132] According to the formula:

[0133] X = preprocess(U)

[0134] We obtain a high-dimensional sequence X, where The input is sensor data.

[0135] In some embodiments, the processing unit is used to:

[0136] According to the formula:

[0137]

[0138] Obtain sequences {X} at different scales. (1) X (2) , ..., X (S)}, where X (i) Let i represent the sequence at the i-th scale.

[0139] In some embodiments, the processing unit is further configured to:

[0140] According to the formula:

[0141]

[0142] Obtain the interpolation result Where ω represents the window size parameter, X (i) Let L represent the sequence at the i-th scale, where L is the sequence length;

[0143] According to the formula:

[0144]

[0145] Obtain the superposition result

[0146] According to the formula:

[0147]

[0148] Obtain the dimensionality-reduced sequence in For parameter matrices;

[0149] According to the formula:

[0150]

[0151] Obtain the sequence X at the (i+1)th scale. (i+1) , where LN is the layer normalization model in neural networks.

[0152] In some embodiments, the processing unit is further configured to:

[0153] According to the formula:

[0154]

[0155]

[0156] Obtain the key sequence K at the i-th scale. (i) AND value sequence V (i) ,in Let be the parameter matrix at the i-th scale;

[0157] According to the formula:

[0158]

[0159]

[0160] The query sequence Q of the first scale is obtained. (1) ,in For attention function;

[0161] According to the formula:

[0162]

[0163] Obtain the representation sequence Y corresponding to the i-th scale. (i) .

[0164] In some embodiments, the processing unit is further configured to:

[0165] According to the formula:

[0166] Y c =concat(Y (1) Y (2) , ..., Y (S) )

[0167] Obtain the stacking result Where {Y (1) Y (2) , ..., Y (S) Let} be a set of S representation sequences, where L is the length of the representation sequence and D is the feature dimension of the representation sequence;

[0168] According to the formula:

[0169] Y o =ReLU(BN(Conv(Y) c )))

[0170] Obtain the total characterization sequence

[0171] Thirdly, embodiments of this application provide a human behavior recognition device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0172] Reference Figure 6 , Figure 6 This is a schematic diagram of the hardware structure for human behavior recognition involved in an embodiment of this application. In this embodiment, the human behavior recognition device may include a processor, a memory, a communication interface, and a communication bus.

[0173] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0174] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the human behavior recognition device, as well as interfaces used for interconnecting the human behavior recognition device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0175] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0176] The processor can be a general-purpose processor, which can call the human behavior recognition program stored in the memory and execute the human behavior recognition method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the human behavior recognition program is called can refer to the various embodiments of the human behavior recognition method of this application, and will not be repeated here.

[0177] Those skilled in the art will understand that Figure 6The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0178] Fourthly, embodiments of this application also provide a readable storage medium.

[0179] The present application has a human behavior recognition program stored on a readable storage medium, wherein when the human behavior recognition program is executed by a processor, it implements the steps of the human behavior recognition method described above.

[0180] The method implemented when the human behavior recognition program is executed can be referred to in various embodiments of the human behavior recognition method of this applicant, and will not be repeated here.

[0181] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0183] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for recognizing human behavior, characterized in that, The method includes: Acquire sensor data, embed the sensor data into a high-dimensional space through preprocessing, and add location information to obtain a preprocessed high-dimensional space sequence; Based on the high-dimensional spatial sequence, the sequence at different scales is output through local window fusion. Information interaction is established between the sequences at different scales through an attention mechanism to obtain the representation sequence corresponding to each scale. The representation sequences corresponding to each scale are then aggregated into a total representation sequence. The total representation sequence is compressed into a final feature vector, and behavior recognition is performed based on the final feature vector.

2. The human behavior recognition method as described in claim 1, characterized in that, The step of embedding the sensor data into a high-dimensional space through preprocessing and adding location information to obtain a preprocessed high-dimensional space sequence includes: According to the formula: X = preprocess(U) We obtain a high-dimensional sequence X, where The input is sensor data.

3. The human behavior recognition method as described in claim 2, characterized in that, The step of fusing and outputting sequences at different scales based on the high-dimensional spatial sequence through a local window includes: According to the formula: X (1) =X X (2) =LocalContextMerging(X (1) ) X (3) =LocalContextMerging(X (2) ) X (S) =LocalContextMerging(X (S-1) ) Obtain sequences {X} at different scales. (1) ,X (2) ,…,X (S) }, where X (i) Let i represent the sequence at the i-th scale.

4. The human behavior recognition method as described in claim 3, characterized in that, The step of fusing and outputting sequences at different scales based on the high-dimensional spatial sequence through a local window also includes: According to the formula: Obtain the interpolation result Where ω represents the window size parameter, X (i) Let L represent the sequence at the i-th scale, where L is the sequence length; According to the formula: Obtain the superposition result According to the formula: Obtain the dimensionality-reduced sequence in For parameter matrices; According to the formula: Obtain the sequence X at the (i+1)th scale. (i+1) , where LN is the layer normalization model in neural networks.

5. The human behavior recognition method as described in claim 4, characterized in that, The step of establishing information interaction between sequences at different scales through an attention mechanism to obtain the representation sequence corresponding to each scale includes: According to the formula: Obtain the key sequence K at the i-th scale. (i) AND value sequence V (i) ,in Let be the parameter matrix at the i-th scale; According to the formula: The query sequence Q of the first scale is obtained. (1) ,in For attention function; According to the formula: Obtain the representation sequence Y corresponding to the i-th scale. (i) .

6. The human behavior recognition method as described in claim 5, characterized in that, The step of aggregating the representation sequences corresponding to each scale into a total representation sequence includes: According to the formula: AND c =concat(Y (1) ,AND (2) ,…,AND (S) ) Obtain the stacking result Where {Y (1) ,Y (2) ,…,Y (S) Let} be a set of S representation sequences, where L is the length of the representation sequence and D is the feature dimension of the representation sequence; According to the formula: AND o =ReLU(BN(Conv(Y c ))) Obtain the total characterization sequence 7. A human behavior recognition device, characterized in that, The device includes: The acquisition unit is used to acquire sensor data, embed the sensor data into a high-dimensional space through preprocessing, and add location information to obtain a preprocessed high-dimensional space sequence. The processing unit is used to output sequences at different scales by fusing them through a local window according to the high-dimensional spatial sequence, establish information interaction between the sequences at different scales through an attention mechanism, obtain the representation sequence corresponding to each scale, and aggregate the representation sequences corresponding to each scale into a total representation sequence. The identification unit is used to compress the total representation sequence into a final feature vector and perform behavior identification based on the final feature vector.

8. A human behavior recognition device as described in claim 7, characterized in that, The acquisition unit is used for: According to the formula: X = preprocess(U) We obtain a high-dimensional sequence X, where The input is sensor data.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, performs the steps as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images