Human behavior recognition method
By using pre-trained models in human behavior recognition to generate spatiotemporal feature maps and attention heat maps, and combining decision-making and fusion with joint node semantic information, the problem of low accuracy of human behavior recognition in the prior art is solved, significantly improving recognition accuracy.
Patent Information
- Application Number
- CN202110699419.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-06-23
AI Technical Summary
The prior art human behavior recognition method based on RGB and bones has insufficient accuracy, making it difficult to effectively extract features and resist background interference.
By obtaining video frames and human joint position data, a pre-trained behavior recognition model is used to generate spatiotemporal feature maps and attention heat maps, combined with joint semantic information, local decision-making fusion is carried out to improve the accuracy of behavior recognition.
Effectively eliminate useless information, extract more discriminant spatial and temporal features, enhance the ability to characterize node characteristics, and improve the accuracy of human behavior recognition models.
Smart Images

Figure CN113591560B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer vision, and in particular relates to a method for recognizing human behavior. Background Art
[0002] As a basic problem in computer vision, human behavior recognition has broad application prospects, such as intelligent monitoring, human-computer interactive somatosensory games, video retrieval, etc., and has therefore attracted widespread attention in the industry.
[0003] In the related technology, video-based behavior recognition can be divided into human behavior recognition based on RGB data and human behavior recognition based on skeleton data according to the data modality. Specifically, in RGB-based behavior recognition, the structures of deep networks mainly include two-stream convolutional neural network (CNN) structure and 3D CNN. RGB data has rich appearance information and can better describe the interaction between people and objects, but it is easily affected by background images during collection, such as weather, lighting, camera angle and clothing, making it difficult to extract features from background images. In skeleton-based behavior recognition, recurrent neural network (RNN) is widely used to simulate the spatiotemporal changes of skeleton sequences. RNN models long-term dependencies, but the temporal modeling is performed on the original input space, so it is difficult to learn high-level features directly from the skeleton.
[0004] Obviously, the above two methods are not conducive to improving the accuracy of human behavior recognition. Summary of the invention
[0005] In order to solve the above problems existing in the prior art, the present invention provides a method for human behavior recognition. The technical problem to be solved by the present invention is achieved by the following technical solutions:
[0006] The present invention provides a human behavior recognition method, comprising:
[0007] Obtaining a preset number of video frames in the video to be processed and the position data of human joints in each video frame;
[0008] The preset number of video frames are input into a pre-trained behavior recognition model, so that after the behavior recognition model generates the spatiotemporal feature maps and attention heat maps corresponding to the preset number of video frames, it obtains the human joint point features in the video frames to determine the human joint point features containing semantic information, and after obtaining the local decision results of each human joint point feature containing semantic information, the local decision results are fused to obtain the human behavior recognition results of the video frames.
[0009] In one embodiment of the present invention, the behavior recognition model includes a feature stream, an attention stream, a local decision module and a decision fusion module.
[0010] In one embodiment of the present invention, the characteristic flow comprises a multi-fiber network.
[0011] In one embodiment of the present invention, the step of inputting the preset number of video frames into a pre-trained behavior recognition model so that the behavior recognition model obtains the human joint features in the video frames after generating the spatiotemporal feature maps and attention heat maps corresponding to the preset number of video frames to determine the human joint features containing semantic information, and after obtaining the local decision results of each human joint feature containing semantic information, fusing the local decision results to obtain the human behavior recognition results of the video frames includes:
[0012] Inputting the preset number of video frames into a pre-trained behavior recognition model, so that the feature stream extracts the spatiotemporal feature graph of the video frame, and the attention stream generates a corresponding attention heat map according to the human joint position data;
[0013] Performing bilinear operations on the spatiotemporal feature map and the attention heat map to obtain features of all human joint points in the video frame;
[0014] According to the human joint point position data and the human joint point features in each video frame, obtaining human joint point features containing semantic information;
[0015] Inputting the human joint point features containing semantic information into the local decision module to obtain a local decision result for each human joint point feature containing semantic information;
[0016] Each local decision result is input into the decision fusion module to obtain the human behavior recognition result of the video frame.
[0017] In one embodiment of the present invention, the step of inputting each local decision result into the decision fusion module to obtain the human behavior recognition result of the video frame includes:
[0018] Calculating an average value of all local decision results, and determining a human behavior recognition result of the video frame according to the average value; or,
[0019] According to preset weights, each local decision result is weighted and summed, and the human behavior recognition result of the video frame is determined according to the weighted summation result.
[0020] In one embodiment of the present invention, the step of performing a bilinear operation on the spatiotemporal feature map and the attention heat map to obtain features of all human joint points in the video frame includes:
[0021] Convert the attention heat map into a first matrix, and convert the spatiotemporal feature map into a second matrix; wherein the size of the attention heat map is M*L*H*W, the size of the spatiotemporal feature map is C*L*H*W, the first matrix includes M rows and L*H*W columns, the second matrix includes C rows and L*H*W columns, M=N×L, N represents the number of human joints in each video frame, L represents the temporal dimension of the spatiotemporal feature map, H represents the height of the spatiotemporal feature map, and W represents the width of the spatiotemporal feature map;
[0022] A bilinear operation is performed on the first matrix and the second matrix according to the following formula:
[0023] X=AB T
[0024] In the formula, A represents the first matrix, B T represents the transpose of the second matrix, and X represents the features of all human joints.
[0025] In one embodiment of the present invention, the step of obtaining human joint point features containing semantic information based on the human joint point position data and the human joint point features in each video frame includes:
[0026] The joint point type in each video frame is encoded according to the following formula to obtain the joint point type code:
[0027] z k ∈R Z ; k = 1, 2, ..., Z
[0028] Among them, z k represents the joint point encoding vector of type k, whose dimension is Z, and R represents a real number;
[0029] Perform frame index encoding on the joint points in each of the video frames according to the following formula to obtain the frame index encoding of the joint points:
[0030] q t ∈R T ,t=1,2,...,T
[0031] Among them, q t The frame index encoding vector representing the human joint point in the t-th video frame, whose dimension is T;
[0032] The joint point type code and the joint point frame index code are fused according to the following formula to obtain the human joint point features containing semantic information:
[0033]
[0034] in, Represents a splicing operation, Represents the semantic information of the joint point of type k in the t-th video frame, and its dimension is (Z+T).
[0035] In one embodiment of the present invention, the human behavior recognition model is trained by the following steps:
[0036] Acquire training samples, each of which includes a preset number of video frames to be tested, first human joint position data corresponding to the video frames to be tested, and a real behavior category corresponding to the video frames to be tested;
[0037] Inputting the training samples into a neural network model to be trained, wherein the neural network model to be trained is a preset initial neural network model;
[0038] Determine the predicted real behavior category of each training sample by using the output result of the neural network model to be trained;
[0039] Determining a first loss value according to the predicted real behavior category, the real behavior category and a preset loss function of each training sample;
[0040] Judging whether the neural network model to be trained has converged according to the first loss value; if converged, the neural network model to be trained is a trained human behavior recognition model;
[0041] If convergence has not occurred, the network parameters of the neural network model to be trained are adjusted, and the process returns to the step of inputting the training samples into the neural network model to be trained.
[0042] In one embodiment of the present invention, the preset loss function is a cross entropy loss function, and the cross entropy loss function includes a global cross entropy loss function and a local cross entropy loss function.
[0043] In one embodiment of the present invention, the joint point type includes at least one of the head, neck, left shoulder, and right shoulder.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] The present invention provides a method for human behavior recognition, which obtains a preset number of video frames in a video to be processed and the position data of human joints in each video frame, and inputs the preset number of video frames into a pre-trained behavior recognition model, so that the behavior recognition model extracts the spatiotemporal features of the video clips, and regards the human skeleton area as the attention area to generate an attention heat map. Since the skeleton point information reflects the posture characteristics of the human body, it can eliminate useless information of the target to a great extent, and then extract more discriminative spatiotemporal features; in addition, the present invention further combines the semantic information of the joint points, effectively enhances the representation ability of the joint point features, and thus improves the accuracy of the human behavior recognition model.
[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of a human behavior recognition method provided by an embodiment of the present invention;
[0048] Figure 2 is a schematic diagram of a behavior recognition model provided by an embodiment of the present invention;
[0049] Figure 3 is another flow chart of the human behavior recognition method provided by an embodiment of the present invention;
[0050] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0052] like Figure 1 As shown, the human behavior recognition method provided by the embodiment of the present invention includes:
[0053] S101, obtaining a preset number of video frames in the video to be processed and position data of human body joints in each video frame;
[0054] S102. Input a preset number of video frames into a pre-trained behavior recognition model, so that after the behavior recognition model generates spatiotemporal feature maps and attention heat maps corresponding to the preset number of video frames, it obtains the human joint features in the video frames to determine the human joint features containing semantic information, and after obtaining the local decision results of each human joint feature containing semantic information, fuses the local decision results to obtain the human behavior recognition results of the video frames.
[0055] It should be noted that the position data of the human joint points in each video frame can be obtained in advance through manual annotation or a human joint point estimation algorithm, which includes the horizontal and vertical coordinates of the human joint points in the video frame.
[0056] Specifically, if the video set to be processed is I = {I1, I2, ..., I i ,...,I V}, V represents the total number of videos, I i represents the i-th video to be processed, and the length set of the video to be processed is F = {F1, F2, ..., F i ,...,F V},F i represents the length of the i-th video to be processed, where the length of the video to be processed refers to the total number of video frames it contains. The human joint point position data set is J = {J1, J2, ..., J i ...,J V}, J i represents the human joint point data in the i-th video to be processed, J i The dimension is F i *Z*2, Z represents the number of human joints in each video frame, and 2 represents the horizontal and vertical coordinates of the position of each joint.
[0057] Further, a preset number of video frames in the video to be processed are obtained, and the preset number of video frames are scaled, randomly cropped and mean normalized to obtain video data with a dimension of T*3*224*224. Optionally, T represents the preset number of video frames selected from the video to be processed. In this embodiment, T=16. It should be understood that the human joint point position data corresponds one-to-one to the human body in the video frame. When the video frame is scaled, the size of the human body in the video is scaled in proportion, and the position of the human joint point is also scaled in proportion; when the video frame is randomly cropped, the position of the human body in the video frame changes, and the position of the human joint point is changed according to the cropping center. Obviously, the dimension of the human joint point position data is T*Z*2.
[0058] In the above step S102, after a preset number of video frames are input into a pre-trained behavior recognition model, the behavior recognition model generates spatiotemporal feature maps and attention heat maps corresponding to the preset number of video frames, and obtains the human joint features in the video frames to determine the human joint features containing semantic information, and after obtaining the local decision results of each human joint feature containing semantic information, the local decision results are fused to obtain the human behavior recognition results of the video frames.
[0059] It can be seen that the human behavior recognition method provided by the embodiment of the present invention extracts the spatiotemporal features of the video clips through a pre-trained behavior recognition model, and regards the human skeleton area as the attention area to generate an attention heat map. Since the skeleton point information reflects the posture characteristics of the human body, it can eliminate useless information of the target to a great extent, and then extract more discriminative spatiotemporal features; in addition, the present invention further combines the semantic information of joint points, effectively enhances the representation ability of joint point features, and thus improves the accuracy of the human behavior recognition model.
[0060] Figure 2 is a schematic diagram of a behavior recognition model provided by an embodiment of the present invention. Figure 2 ,The behavior recognition model includes feature stream, attention stream, local decision module and decision fusion module.
[0061] Specifically, the feature stream includes a multi-fiber network (MF-Net), which uses a multi-fiber network to extract a preset number of spatiotemporal feature maps of video frames, which can effectively reduce the number of parameters of the three-dimensional network and avoid overfitting. In this embodiment, the dimension of the generated spatiotemporal feature map is C×L×H×W, where C represents the number of channels, L represents the temporal dimension of the spatiotemporal feature map, H represents the height of the spatiotemporal feature map, and W represents the width of the spatiotemporal feature map.
[0062] According to the position data of human joints, the attention flow of the behavior recognition model obtains the attention heat map of each joint, which has the same spatiotemporal dimension as the spatiotemporal feature map. Optionally, the dimension of the attention heat map is M×L×H×W, where M represents the number of channels, L represents the temporal dimension of the attention heat map, H represents the height of the attention heat map, and W represents the width of the attention heat map; wherein, the human joint feature is realized by selecting the activation of the corresponding points of the body joints in the attention heat map, which is equivalent to assigning hard weights, that is, the weight 1 is assigned to the corresponding points of the joints, and the weight 0 is assigned to the corresponding points of the non-joints.
[0063] After determining the features of human joints containing semantic information, the set of human joint features containing semantic information is input into the local decision module. The local decision module makes decisions on the spatiotemporal features of all local joints through a fully connected layer to obtain local behavior recognition results. Then, the decision fusion module fuses all local behavior recognition results obtained by the local decision module to obtain the final behavior recognition result.
[0064] Figure 3 FIG. 2 is another flow chart of the human behavior recognition method provided by an embodiment of the present invention. Figure 3As shown, in the above step S102, a preset number of video frames are input into a pre-trained behavior recognition model, so that after the behavior recognition model generates the spatiotemporal feature graphs and attention heat maps corresponding to the preset number of video frames, the human joint features in the video frames are obtained to determine the human joint features containing semantic information, and after obtaining the local decision results of each human joint feature containing semantic information, the local decision results are fused to obtain the human behavior recognition results of the video frames, including:
[0065] S301, inputting a preset number of video frames into a pre-trained behavior recognition model, so that the feature stream extracts the spatiotemporal feature map of the video frame, and the attention stream generates a corresponding attention heat map according to the position data of the human body joints;
[0066] S302, performing bilinear operations on the spatiotemporal feature map and the attention heat map to obtain features of all human joints in the video frame;
[0067] Specifically, the attention heat map is converted into a first matrix, and the spatiotemporal feature map is converted into a second matrix; wherein the size of the attention heat map is M*L*H*W, the size of the spatiotemporal feature map is C*L*H*W, the first matrix includes M rows and L*H*W columns, the second matrix includes C rows and L*H*W columns, M=N×L, N represents the number of human joints in each video frame, L represents the temporal dimension of the spatiotemporal feature map, H represents the height of the spatiotemporal feature map, and W represents the width of the spatiotemporal feature map;
[0068] Perform bilinear operations on the first matrix and the second matrix according to the following formula:
[0069] X=AB T
[0070] In the formula, A represents the first matrix, B T represents the transpose of the second matrix, and X represents the features of all human joints.
[0071] S303, obtaining human joint point features containing semantic information according to the human joint point position data and human joint point features in each video frame;
[0072] It can be understood that, for a joint point, its semantic information consists of a spatial index and a temporal index. The spatial index can be expressed as a joint point type, and the temporal index can be expressed as a frame index.
[0073] Optionally, the semantic information of human joints is represented as:
[0074]
[0075] Among them, Q represents the semantic information set of human joints, It represents the semantic information of the joint point with joint type k in the t-th video frame, T represents the total number of video frames, and Z represents the number of human joint points in the video frame.
[0076] It is understandable that for skeletal data, intuitively the key to behavior recognition is semantic information, namely, joint type and frame index. The spatiotemporal structure of human joints is represented by semantics and dynamics together. In behavior recognition, on the one hand, two joints with the same coordinates but different semantics will convey different information. For example, for the joint above the head, if the joint is a hand joint, the behavior may be raising the hand; if the joint is a foot joint, the behavior may be kicking. On the other hand, timing information is also very important. For example, the two behaviors of sitting down and standing up are just different in the time order of the behaviors. It can be seen that the semantic information of human joints is very important.
[0077] Exemplarily, in this embodiment, semantic information may be defined according to the following steps:
[0078] First, encode the joint point type and joint point frame index respectively. Encode the joint point type in each video frame according to the following formula to obtain the joint point type code:
[0079] z k ∈R Z ; k = 1, 2, ..., Z
[0080] In the formula, z k Represents a joint point encoding vector of type k, whose dimension is Z, that is, there are Z types of joint points in the video frame, and R represents a real number; optionally, this embodiment adopts a one-hot encoding method, and sets the value of the kth position of the encoding vector to 1, and all other positions are set to 0.
[0081] Perform frame index encoding on the joint points in each video frame according to the following formula to obtain the frame index encoding of the joint points:
[0082] q t ∈R T ,t=1,2,...,T
[0083] Among them, q t A frame index encoding vector representing a human joint point in the t-th video frame, whose dimension is T; optionally, this embodiment adopts a one-hot encoding method, and sets the value of the k-th position of the encoding vector to 1, and sets all other positions to 0.
[0084] The joint point type code and the joint point frame index code are fused according to the following formula to obtain the human joint point features containing semantic information:
[0085]
[0086] in, Represents a splicing operation, Represents the semantic information of the joint point of type k in the t-th video frame, and its dimension is (Z+T).
[0087] In addition, in some other embodiments of the present application, other semantic information encoding strategies may be adopted, such as not using concatenation operations for semantic description, but directly defining an encoding vector to represent semantic information. Specifically, as shown below:
[0088]
[0089] Among them, q t The frame index encoding vector representing the human joint point in the t-th video frame has a dimension of Z × T. Similarly, one-hot encoding can be used to set the value of the ((t-1) × Z + k)th position of the encoding vector to 1 and the values of other positions to 0.
[0090] Exemplarily, the joint point type includes at least one of a head, a neck, a left shoulder, and a right shoulder.
[0091] S304, inputting the human joint point features containing semantic information into the local decision module to obtain the local decision result of each human joint point feature containing semantic information;
[0092] S305: Input each local decision result into a decision fusion module to obtain a human behavior recognition result of the video frame.
[0093] Specifically, in step S305, the decision fusion module can calculate the average value of all local decision results, and determine the human behavior recognition result of the video frame based on the average value; or, the local decision module can also perform weighted summation on each local decision result according to a preset weight, and determine the human behavior recognition result of the video frame based on the weighted summation result.
[0094] In this embodiment, the human behavior recognition model can be obtained by training a local decision network (LDNet).
[0095] Specifically, the human behavior recognition model is trained using the following steps:
[0096] Acquire training samples, each training sample including a preset number of video frames to be tested, first human joint position data corresponding to the video frames to be tested, and a real behavior category corresponding to the video frames to be tested;
[0097] Inputting the training samples into the neural network model to be trained, where the neural network model to be trained is a preset initial neural network model;
[0098] Using the output results of the neural network model to be trained, determine the predicted behavior category of each training sample;
[0099] Determine a first loss value according to the predicted behavior category, the actual behavior category and the preset loss function of each training sample;
[0100] Judging whether the neural network model to be trained has converged according to the first loss value; if converged, the neural network model to be trained is a trained human behavior recognition model;
[0101] If convergence has not occurred, the network parameters of the neural network model to be trained are adjusted, and the process returns to the step of inputting the training samples into the neural network model to be trained.
[0102] In this embodiment, the preset loss function may be a cross entropy loss function, which includes a global cross entropy loss function and a local cross entropy loss function. The global cross entropy loss function is:
[0103]
[0104] Among them, U represents the number of training samples, K represents the total number of preset behavior categories, represents the true behavior category of the jth position in the i-th video frame to be tested, represents the predicted behavior category at the jth position in the i-th video frame to be tested.
[0105] Furthermore, the local cross entropy loss function is:
[0106]
[0107] Wherein, M represents the total number of joint points in a preset number of video frames to be tested, Represents the true behavior category of the i-th sample at the j-th position of the m-th joint point.
[0108] In this embodiment, the preset loss function is the sum of the global cross entropy loss function and the local cross entropy loss function, which is helpful to improve the recognition performance of the human behavior recognition model.
[0109] Next, the recognition accuracy of the human behavior recognition method provided by the present invention is verified on the subJHMDB dataset and the Penn Action dataset.
[0110] Table 1 Recognition accuracy of semantic information modules at different joint points
[0111] method subJHMDB Penn Action LDNet(Conv5)+L(S)+D(NW) 0.839 0.965 LDNet(Conv5)+L(S)+D(NW)+SI1 0.845 0.970 LDNet(Conv5)+L(S)+D(NW)+SI2 0.837 0.957 LDNet(Conv5)+L(S)+D(W) 0.843 0.973 LDNet(Conv5)+L(S)+D(W)+SI1 0.848 0.974 LDNet(Conv5)+L(S)+D(W)+SI2 0.842 0.968
[0112] Among them, Conv4 and Conv5 represent the spatiotemporal feature maps after feature streams Conv4 and Conv5 respectively, L(S) represents the local decision module using a linear classifier with shared parameters, L(NS) represents the local decision module using a linear classifier without shared parameters, D(W) and D(NW) represent the decision fusion module using a weighted fusion strategy and an average fusion strategy respectively, LS represents local supervision, GS represents global supervision, SI1 represents the encoding strategy of encoding the joint point type and the joint point frame index respectively and then fusing them, and SI2 represents the encoding strategy of jointly encoding the joint point type and the joint point frame index.
[0113] It can be seen from Table 1 that on the subJHMDB dataset, when LDNet uses the average fusion strategy to aggregate local decision results, the recognition accuracy is 83.9%. When LDNet further adopts the SI1 encoding strategy, the recognition accuracy is increased by 0.6% to 84.5%. When LDNet is combined with the SI2 encoding strategy, the recognition accuracy decreases by 0.2%. When LDNet uses a weight-based fusion strategy to aggregate local decisions, the recognition accuracy is 84.3%. Under the weight-based fusion strategy, LDNet combines the SI1 encoding strategy, the recognition accuracy is increased by 0.5% to 84.8%. When LDNet is combined with the SI2 encoding strategy, the recognition accuracy decreases by 0.1%.
[0114] On the Penn Action dataset, when LDNet uses the average fusion strategy to aggregate local decisions, the recognition accuracy is 96.5%. LDNet further combines the SI1 encoding strategy, and the recognition accuracy is increased by 0.5% to 97.0%. LDNet combines the SI2 encoding strategy, and the recognition accuracy decreases by 0.8%. When LDNet uses a weight-based fusion strategy to aggregate local decisions, the recognition accuracy is 97.3%. Under the weight-based fusion strategy, LDNet combines the SI1 encoding strategy, and the recognition accuracy is increased by 0.1% to 97.4%, while LDNet combines the SI2 encoding strategy to 96.8%, a decrease of 0.5%.
[0115] The above results verify the effectiveness of the SI1 encoding strategy and prove that the semantic information of joint points can improve the representation ability of joint point features. When LDNet is combined with the SI2 encoding strategy, the recognition accuracy has decreased to a certain extent, whether on the subJHMDB dataset or the PennAction dataset. The reason for this phenomenon is that when the SI2 encoding strategy is used in conjunction with LDNet, the feature dimension of the joint points will be significantly increased, causing the network to overfit, thereby affecting the recognition accuracy.
[0116] In addition, this embodiment uses the SI1 encoding strategy to further verify the importance and effectiveness of semantic information on the subJHMDB dataset and the Penn Action dataset.
[0117] Table 2
[0118] method subJHMDB Penn Action LDNet(Conv5)+L(S)+D(NW) 0.839 0.965 LDNet(Conv5)+L(S)+D(NW)+SI1 0.845 0.970 LDNet(Conv5)+L(S)+D(W) 0.843 0.973 LDNet(Conv5)+L(S)+D(W)+SI1 0.848 0.974 LDNet(Conv4+5)+L(S)+D(NW) 0.847 0.977 LDNet(Conv4+5)+L(S)+D(NW)+SI1 0.851 0.980 LDNet(Conv4+5)+L(S)+D(W) 0.850 0.982 LDNet(Conv4+5)+L(S)+D(W)+SI1 0.854 0.985
[0119] Among them, Conv4+5 represents the local decision obtained by integrating the spatiotemporal feature map after the Conv4 and Conv5 blocks. As can be seen from Table 2, on the subJHMDB dataset, when LDNet uses the average fusion strategy to aggregate the local decision results obtained after the Conv5 block, the recognition accuracy is 83.9%. After the network is further combined with the SI1 encoding strategy, the accuracy is increased to 84.5%. When LDNet uses the weight-based fusion strategy to aggregate the local decision obtained after the Conv5 block, the accuracy is 84.3%. After the network is further combined with the SI1 encoding strategy, the accuracy is increased to 84.8%. When LDNet uses the average fusion strategy to aggregate the local decision structure obtained after Conv4 and Conv5 blocks, the accuracy is 84.7%. After the network is combined with the SI1 module, the accuracy is increased by 0.4% to 85.1%. When LDNet uses the weight-based fusion strategy to aggregate the local decision results obtained after Conv4 and Conv5 blocks, the accuracy is 85.0%. After the network is combined with the SI1 encoding strategy, the accuracy is increased to 85.4%.
[0120] It can be seen that whether in the subJHMDB dataset or the Penn Action dataset, when LDNet is combined with the SI1 module, the recognition accuracy is improved to a certain extent, which further proves the effectiveness of the SI1 encoding strategy.
[0121] The present invention provides a method for human behavior recognition, which obtains a preset number of video frames in a video to be processed and the position data of human joints in each video frame, and inputs the preset number of video frames into a pre-trained behavior recognition model, so that the behavior recognition model extracts the spatiotemporal features of the video clips, and regards the human skeleton area as the attention area to generate an attention heat map. Since the skeleton point information reflects the posture characteristics of the human body, it can eliminate useless information of the target to a great extent, and then extract more discriminative spatiotemporal features; in addition, the present invention further combines the semantic information of the joint points, effectively enhances the representation ability of the joint point features, and thus improves the accuracy of the human behavior recognition model.
[0122] The embodiment of the present invention further provides an electronic device, such as Figure 4As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.
[0123] Memory 403, used for storing computer programs;
[0124] The processor 401 is used to execute the program stored in the memory 403 to implement the following steps:
[0125] Obtaining a preset number of video frames in the video to be processed and the position data of human joints in each video frame;
[0126] The preset number of video frames are input into a pre-trained behavior recognition model, so that after the behavior recognition model generates the spatiotemporal feature maps and attention heat maps corresponding to the preset number of video frames, it obtains the human joint point features in the video frames to determine the human joint point features containing semantic information, and after obtaining the local decision results of each human joint point feature containing semantic information, the local decision results are fused to obtain the human behavior recognition results of the video frames.
[0127] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0128] The communication interface is used for communication between the above electronic device and other devices.
[0129] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0130] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0131] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.
[0132] As for the electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0133] It should be noted that the electronic device and storage medium of the embodiments of the present invention are respectively the electronic device and storage medium to which the above-mentioned human behavior recognition method is applied. All embodiments of the above-mentioned human behavior recognition method are applicable to the electronic device and storage medium, and can achieve the same or similar beneficial effects.
[0134] By using the terminal device provided by the embodiment of the present invention, proper nouns and / or fixed phrases can be displayed for user selection, thereby reducing user input time and improving user experience.
[0135] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0136] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.
[0137] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in a claim. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.
[0138] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware, which are collectively referred to as "modules" or "systems" herein. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. The computer program is stored / distributed in a suitable medium, provided together with other hardware or as a part of hardware, or may adopt other distribution forms, such as by Internet or other wired or wireless telecommunication systems.
[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.
[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0142] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A human behavior recognition method, characterized in that: include: Obtain a preset number of video frames in the video to be processed and the position data of human joints in each video frame; The preset number of video frames are input into a pre-trained behavior recognition model, and the behavior recognition model includes a feature stream, an attention stream, a local decision module and a decision fusion module, so that after the behavior recognition model generates the spatiotemporal feature graphs and attention heat maps corresponding to the preset number of video frames, the human joint features in the video frames are obtained to determine the human joint features containing semantic information, and after obtaining the local decision results of each human joint feature containing semantic information, the local decision results are fused to obtain the human behavior recognition results of the video frames; this step includes: inputting the preset number of video frames into the pre-trained behavior recognition model, so that the feature The feature flow extracts the spatiotemporal feature map of the video frame, and enables the attention flow to generate a corresponding attention heat map according to the human joint position data; bilinear operations are performed on the spatiotemporal feature map and the attention heat map to obtain all human joint features in the video frame; human joint features containing semantic information are obtained according to the human joint position data and the human joint features in each video frame; the human joint features containing semantic information are input into the local decision module to obtain local decision results of each human joint feature containing semantic information; each local decision result is input into the decision fusion module to obtain a human behavior recognition result of the video frame; Wherein, the step of obtaining the human joint point features containing semantic information according to the human joint point position data and the human joint point features in each video frame comprises: The joint point type in each video frame is encoded according to the following formula to obtain the joint point type code: in, Indicates the type The joint point encoding vector of , R represents a real number; Perform frame index encoding on the joint points in each of the video frames according to the following formula to obtain the frame index encoding of the joint points: in, Indicates The frame index encoding vector of the human joint points in the video frames has a dimension of ; The joint point type code and the joint point frame index code are fused according to the following formula to obtain the human joint point features containing semantic information: Among them, ⊕ represents the splicing operation, Indicates In the video frames, the type is The semantic information of the joint points, and the dimension is .
2. The human behavior recognition method according to claim 1, characterized in that: The characteristic flow includes a multi-fiber network.
3. The human behavior recognition method according to claim 2, characterized in that: The step of inputting each local decision result into the decision fusion module to obtain the human behavior recognition result of the video frame includes: Calculating an average value of all local decision results, and determining a human behavior recognition result of the video frame according to the average value; or, According to preset weights, each local decision result is weighted and summed, and the human behavior recognition result of the video frame is determined according to the weighted summation result.
4. The human behavior recognition method according to claim 3, characterized in that: The step of performing bilinear operation on the spatiotemporal feature map and the attention heat map to obtain features of all human joint points in the video frame includes: The attention heat map is converted into a first matrix, and the spatiotemporal feature map is converted into a second matrix; wherein the size of the attention heat map is , the size of the spatiotemporal feature map is , the first matrix includes Line and The second matrix includes Line and List, , Indicates the number of human joints in each video frame, Represents the temporal dimension of the spatiotemporal feature map, represents the height of the spatiotemporal feature map, Indicates the width of the spatiotemporal feature map; A bilinear operation is performed on the first matrix and the second matrix according to the following formula: In the formula, represents the first matrix, represents the transpose of the second matrix, For all human joint features.
5. The human behavior recognition method according to claim 1, characterized in that: The human behavior recognition model is trained by the following steps: Acquire training samples, each of which includes a preset number of video frames to be tested, first human joint position data corresponding to the video frames to be tested, and a real behavior category corresponding to the video frames to be tested; Inputting the training samples into a neural network model to be trained, wherein the neural network model to be trained is a preset initial neural network model; Determine the predicted real behavior category of each training sample by using the output result of the neural network model to be trained; Determining a first loss value according to the predicted real behavior category, the real behavior category and a preset loss function of each training sample; Judging whether the neural network model to be trained has converged according to the first loss value; if converged, the neural network model to be trained is a trained human behavior recognition model; If convergence has not occurred, the network parameters of the neural network model to be trained are adjusted, and the process returns to the step of inputting the training samples into the neural network model to be trained.
6. The human behavior recognition method according to claim 5, characterized in that: The preset loss function is a cross entropy loss function, and the cross entropy loss function includes a global cross entropy loss function and a local cross entropy loss function.
7. The human behavior recognition method according to claim 6, characterized in that: The joint point type includes at least one of a head, a neck, a left shoulder, and a right shoulder.
