Human body behavior recognition method based on long time sequence motion semantic coding

Abstract

The invention provides a human body behavior recognition method based on long-time-sequence motion semantic coding, and relates to the technical field of video analysis, and the method comprises the steps: S1, inputting a video image, detecting a pedestrian target in the image, and calculating the intersection-union ratio of pedestrian detection frames in a front frame image and a rear frame image to track the human body target; s2, extracting skeleton key points of each human body target by adopting a posture estimation algorithm, and defining the skeleton key points of the human body targets as middle-layer motion semantic features; s3, establishing a 3D CNN network model based on the middle-layer motion semantic features to extract short-time-sequence motion semantic codes; s4, constructing a long-time-sequence motion semantic coding model according to the short-time-sequence motion semantic coding so as to output a one-dimensional feature vector; s5, constructing a human body behavior classifier based on a full-connection network to recognize the one-dimensional feature vector, and outputting a recognition result; the accuracy of behavior recognition can be improved, and the problems that the error recognition rate is high, and application of a real scene is difficult to meet are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of video analysis, and in particular to a method for human behavior recognition based on long-term motion semantic coding. Background Art

[0002] Human behavior recognition refers to extracting pedestrian motion features from videos and images and identifying pedestrian behaviors. This technology has a wide range of applications in both industrial production and leisure and entertainment industries. For example, in the field of industrial production, surveillance cameras installed in factory areas can be used to obtain real-time surveillance videos and identify workers' behaviors, detect whether there are unsafe behaviors, and thus avoid dangerous accidents. In game development, it is also possible to capture the movements of game players and interact with game characters, thereby enhancing the gaming experience.

[0003] At present, human recognition methods based on deep convolutional neural networks have achieved certain results. For example, deep learning models based on two-stream networks and 3D convolutional neural networks have achieved good human behavior recognition results on many open source data sets. However, these methods tend to extract and recognize from static environments. Take the two-stream convolutional neural network model as an example. The model has two networks, spatial and temporal. Among them, the spatial network mainly extracts image features from video frames, but the temporal network mainly extracts image features from optical flow and performs human behavior recognition, rather than focusing on extracting motion features from the characteristics of human motion itself, which cannot meet the application needs of real scenes. Summary of the invention

[0004] The purpose of the present invention is to provide a human behavior recognition method based on long-term motion semantic coding, which can improve the accuracy of behavior recognition and effectively solve the problem that the existing human behavior recognition technology tends to extract static environment features, resulting in a high misrecognition rate and difficulty in meeting the application of real scenes.

[0005] The technical solution of the present invention is:

[0006] In a first aspect, the present application provides a method for human behavior recognition based on long-term motion semantic coding, which comprises the following steps:

[0007] S1, input video images, detect pedestrian targets in the image, calculate the intersection and union ratio of pedestrian detection frames in the previous and next two frames of images to track human targets;

[0008] S2, using the posture estimation algorithm to extract the skeleton key points of each human target, and define the skeleton key points of the human target as the middle-level motion semantic features;

[0009] S3, based on the mid-level motion semantic features, a 3D CNN network model is established to extract short-term motion semantic coding;

[0010] S4, constructing a long-term motion semantic coding model according to the short-term motion semantic coding to output a one-dimensional feature vector;

[0011] S5. Build a human behavior classifier based on the fully connected network to identify the one-dimensional feature vector and output the recognition result.

[0012] Furthermore, in the above step S1, the calculation formula for calculating the intersection-over-union ratio of the pedestrian detection frames in the two frames of images includes:

[0013]

[0014] Where IoU is the intersection-over-union ratio of the pedestrian detection boxes in the previous and next two frames, A is the area of ​​the detection box in the previous frame, and B is the area of ​​the detection box in the next frame.

[0015] Furthermore, in step S2, the calculation formula of the above posture estimation algorithm is:

[0016] M=P(I;WP),M∈R K×H×W

[0017] MS={M1,M 2, ...,M T},MS∈R K×T×H×W

[0018] Where M is the heatmap corresponding to a single human posture, P is the posture estimation algorithm, I is the input image, WP is the network parameter of posture estimation, K is the number of skeleton key points, H is the height of the input image, W is the width of the input image, MS represents a 3D heatmap group of a video clip, which consists of T consecutive M, and T represents the number of consecutive video frames contained in the video clip.

[0019] Furthermore, in step S3, the expression of the above 3D CNN network model is:

[0020] x=F(MS;W)

[0021] Where F represents the 3D CNN network model, MS represents the 3D heatmap group of a video clip, W is the learning parameter of the network, and x is the output short-term motion semantic encoding.

[0022] Furthermore, in step S4, the expression of the long-term motion semantic coding model is:

[0023] X={x1,x2,...,x G}={F(MS1;W);F(MS2;W);...;F(MS G ; W)},

[0024] X∈R G×L

[0025] Y v =Q(X)

[0026] In the formula, F represents the 3D CNN network model, W is the learning parameter of the network, and x G is the output short-term motion semantic coding, X is the entire short-term motion semantic coding of a video action, MS G represents the 3D heatmap group of the Gth feature group video clip, L represents the length of the short time sequence coding feature, G represents the number of feature groups, Y v is the long-term motion semantic coding of a video action, and Q(X) represents the long-term motion semantic coding model.

[0027] Furthermore, step S4 also includes: according to the extracted short-time sequence motion semantic coding, and using the basic principle of TCN, a sequence model based on the time convolutional neural network is constructed by using causal loop convolution and dilated convolution at the same time, so as to expand the one-dimensional long-time sequence motion semantic coding model into a two-dimensional long-time sequence motion semantic coding model, and realize the long-time high-level behavior semantic coding; wherein, the calculation formula for constructing the sequence model based on the time convolutional neural network includes:

[0028]

[0029] Where F(i, j) is a sequence model based on a temporal convolutional neural network, X is the entire short-term motion semantic encoding of a video action, d represents the expansion factor, K is a convolution kernel of size N×k, i is the X-axis coordinate of the input and output data, j is the Y-axis coordinate of the input and output data, f is the extended convolution operation, k is the width of the convolution kernel, N is the height of the convolution kernel, u is the X-axis coordinate of the convolution kernel, and v is the Y-axis coordinate of the convolution kernel.

[0030] Furthermore, in step S5, the human behavior classifier includes two fully connected layers, one of which is used to use a ReLU activation function; and one fully connected layer is used to connect a softmax layer and output a classification result.

[0031] Furthermore, step S5 also includes training the human behavior classifier, and the training steps include:

[0032] Collect video data of various human behaviors;

[0033] Extract continuous human posture key point feature vectors and long-term motion semantic coding in video data;

[0034] The loss function is calculated using cross entropy based on the feature vector, and the network parameters are updated using the stochastic gradient descent optimization algorithm to build a behavior semantic understanding model based on the labeled data;

[0035] By using a fully connected network and utilizing a behavioral semantic understanding model, video behavioral events are identified to output recognition results.

[0036] In a second aspect, the present application provides an electronic device, including:

[0037] A memory for storing one or more programs;

[0038] processor;

[0039] When the one or more programs are executed by the processor, a method for human behavior recognition based on long-term motion semantic coding as described in any one of the first aspects is implemented.

[0040] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a human behavior recognition method based on long-term motion semantic coding as described in any one of the first aspects above.

[0041] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0042] (1) The present invention provides a human behavior recognition method based on long-term motion semantic coding. The method tracks human targets by calculating the intersection-and-union ratio of pedestrian detection frames in two frames of images, and then uses a posture estimation algorithm to extract the skeleton key points of each human target, thereby extracting the operation information of the pedestrian in the pedestrian's action itself. The method is not easily affected by the environment and background. Compared with the existing method based on image feature extraction, the method has better feature extraction effect, higher recognition accuracy, and can effectively reduce the false recognition rate. At the same time, it can meet the needs of multiple types of behavior recognition under a single background.

[0043] (2) The 3D CNN network model designed by the present invention can retain information that changes rapidly in the time dimension, and can extract motion features by setting a minimum step size, effectively avoiding the loss of fine-grained temporal information due to the use of video frames; at the same time, the two-dimensional long-term motion semantic coding model extended by the present invention also ensures that long-term motion semantic information is encoded from the complete video behavior, realizing long-term high-level behavior semantic coding, thereby enhancing the ability to understand human behavior;

[0044] (3) The present invention adopts a lightweight fully connected network to construct a human behavior classifier for human behavior recognition. Compared with the existing two-stream network model, the number of model parameters is greatly reduced, and there is no need to calculate additional data such as optical flow. Therefore, the calculation efficiency is higher and can meet the needs of online trial video analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A step diagram of a human behavior recognition method based on long-term motion semantic coding of the present invention;

[0047] Figure 2 Flowchart of the long-term motion semantic encoding model and human behavior classifier;

[0048] Figure 3 This is the architecture diagram of the TCN basic block;

[0049] Figure 4 Schematic diagram of a 3-layer 2D TCN network architecture;

[0050] Figure 5 Schematic diagram of human behavior classifier;

[0051] Figure 6 The figure is a schematic structural block diagram of an electronic device according to an embodiment of the present invention.

[0052] Icon: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0055] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprises..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0056] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0057] Example 1

[0058] See also Figure 1 , Figure 1 The figure shows a step diagram of a method for human behavior recognition based on long-term motion semantic coding provided in an embodiment of the present application.

[0059] In a first aspect, the present application provides a method for human behavior recognition based on long-term motion semantic coding, which comprises the following steps:

[0060] S1, input video images, detect pedestrian targets in the image, calculate the intersection and union ratio of pedestrian detection frames in the previous and next two frames of images to track human targets;

[0061] S2, using the posture estimation algorithm to extract the skeleton key points of each human target, and define the skeleton key points of the human target as the middle-level motion semantic features;

[0062] S3, based on the mid-level motion semantic features, a 3D CNN network model is established to extract short-term motion semantic coding;

[0063] S4, constructing a long-term motion semantic coding model according to the short-term motion semantic coding to output a one-dimensional feature vector;

[0064] S5. Build a human behavior classifier based on the fully connected network to identify the one-dimensional feature vector and output the recognition result.

[0065] It should be noted that in step S2, this embodiment adopts the HRNet posture estimation algorithm. After continuously tracking the continuous pedestrian target, the continuous skeleton key points of a pedestrian can be extracted. Since the heatmap output by the last convolutional layer of the posture estimation algorithm model implicitly contains the spatiotemporal relationship and motion expression of the human body space, key points and human body posture, this embodiment proposes a new motion semantic feature expression, which directly extracts motion information from the heatmap, thereby converting the heatmap into a middle-level motion semantic feature representation. The heatmap output by the last convolutional layer of the human body posture estimation algorithm model is defined as: M∈R K×H×W , where K represents the number of human skeleton key points, H and W represent the height and width of the image frame respectively; this embodiment groups the heatmaps of continuous time T into a group, thereby obtaining a three-dimensional (3D) heatmap group, which is defined as: MS∈R K×T×H×W ; A complete video is divided into multiple 3D heatmap groups MV, defined as: MS∈MV∈R G ×K×T×H×W , where G represents the number of groups, and MS will be used as input data for subsequent high-level semantic coding;

[0066] Through the human posture extraction and tracking processing in step S2, an action behavior video is divided into multiple 3D heatmap groups. Then, based on step S3, short-term spatiotemporal relationships and behavior representation features are extracted for each 3D heatmap group. An efficient 3D CNN network structure is used to maintain information that changes rapidly in the time dimension. The detailed architecture is shown in Table 1:

[0067] Table 1 3D CNN network structure

[0068]

[0069] The network architecture is based on the 3D residual network 3D ResNet, and the following four aspects are optimized: 1) the input data of the model is changed from RGB video frames to 3D heatmap group MS; 2) the minimum time step (τ=1) is used, that is, all MSs will be used to extract features without sampling processing to ensure that the fine-grained motion information of fast actions is not lost; 3) the downsampling layer in the backbone network is removed to maintain the high-resolution feature layer; 4) because the input 3D heatmap is already considered to be a mid-level motion semantic feature expression, this embodiment removes the first residual convolution layer (res2) in the backbone network responsible for extracting low-level visual features, and reduces the number of channels of subsequent residual convolution layers to improve the time efficiency of the network and ensure that the network model can quickly extract short-time sequence motion information features;

[0070] like Figure 2 As shown, the above scheme realizes extracting short-time motion information features from a 3D heatmap group, and a complete video action has multiple 3D heatmaps. In this embodiment, this series of 3D heatmap groups are sorted by time, which is expressed as: MV = {MS1, MS2, MS3, ..., MS G}, where MS G Represents a 3D heatmap group; then, each MS G They all need to be input into the above-mentioned 3D CNN network to extract short-term motion semantic coding.

[0071] As a preferred implementation, in step S1, the calculation formula for calculating the intersection-over-union ratio of pedestrian detection frames in the two previous and next frames of images includes:

[0072]

[0073] Where IoU is the intersection-over-union ratio of the pedestrian detection boxes in the previous and next frames, A is the area of ​​the detection box in the previous frame, and B is the area of ​​the detection box in the next frame. When IoU>0.3, it can be determined that the human targets corresponding to A and B in the previous and next frames are the same pedestrian target.

[0074] It should be noted that the present invention uses any target detection algorithm (such as Fast R-CNN or YOLO) to detect pedestrian targets in each frame, and then calculates the intersection-over-union ratio of the detection frames of the pedestrians in the previous and next frames.

[0075] As a preferred implementation, in step S2, the calculation formula of the posture estimation algorithm is:

[0076] M=P(I;WP),M∈R K×H×W

[0077] MS={M1,M2,...,M T},MS∈R K×T×H×W

[0078] Where M is the heatmap corresponding to a single human posture, P is the posture estimation algorithm, I is the input image, WP is the network parameter of posture estimation, K is the number of skeleton key points, H is the height of the input image, W is the width of the input image, MS represents a 3D heatmap group of a video clip, which consists of T consecutive M, and T represents the number of consecutive video frames contained in the video clip.

[0079] As a preferred implementation, in step S3, the expression of the 3D CNN network model is:

[0080] x=F(MS;W)

[0081] Where F represents the 3D CNN network model, MS represents the 3D heatmap group of a video clip, W is the learning parameter of the network, and x is the output short-term motion semantic encoding.

[0082] As a preferred implementation, in step S4, the expression of the long-term motion semantic coding model is:

[0083] X={x1,x2,...,x G}={F(MS1;W);F(MS2;W);...;F(MS G ; W)},

[0084] X∈R G×L

[0085] Y v =Q(X)

[0086] In the formula, F represents the 3D CNN network model, W is the learning parameter of the network, and x G is the output short-term motion semantic coding, X is the entire short-term motion semantic coding of a video action, MS G represents the 3D heatmap group of the Gth feature group video clip, L represents the length of the short time sequence coding feature, G represents the number of feature groups, Y v is the long-term motion semantic coding of a video action, and Q(X) represents the long-term motion semantic coding model.

[0087] It should be noted that the short-term motion semantic coding model groups a video action to extract semantic coding. These short-term spatiotemporal relationship motion feature codings also need to further use the long-term model coding model to encode a completed action behavior into video-level human behavior semantic features. Therefore, a long-term motion semantic coding model is constructed based on the short-term motion semantic coding to output a one-dimensional feature vector.

[0088] As a preferred implementation, step S4 also includes: according to the extracted short-time sequence motion semantic coding, and using the basic principle of TCN, a sequence model based on the time convolutional neural network is constructed by using causal loop convolution and dilated convolution at the same time, so as to expand the one-dimensional long-time sequence motion semantic coding model into a two-dimensional long-time sequence motion semantic coding model, and realize the long-time high-level behavior semantic coding; wherein, the calculation formula for constructing the sequence model based on the time convolutional neural network includes:

[0089]

[0090] Where F(i, j) is a sequence model based on a temporal convolutional neural network, X is the entire short-term motion semantic encoding of a video action, d represents the expansion factor, K is a convolution kernel of size N×k, i is the X-axis coordinate of the input and output data, j is the Y-axis coordinate of the input and output data, f is the extended convolution operation, k is the width of the convolution kernel, N is the height of the convolution kernel, u is the X-axis coordinate of the convolution kernel, and v is the Y-axis coordinate of the convolution kernel.

[0091] It should be noted that the architecture of the 2D TCN basic block of the extended convolution is as follows Figure 3 In the above calculation formula, the entire short-term motion semantic encoding of a video action is X = {x1, x2, ..., x G}, the extended convolution achieves a very large perception field of view when the number of convolution layers is not large by skipping the step size id×u, thereby learning the data in the sequence and realizing high-level behavioral semantic encoding of long time series. The three-layer 2D TCN network architecture designed in this embodiment is as follows Figure 4 As shown in the figure, d is set to 1, 2, and 4 respectively. The feature vector output by the last layer covers the historical information of the entire sequence. Therefore, the 2D TCN network model can encode long-term behavioral semantic information.

[0092] As a preferred implementation, in step S5, the human behavior classifier includes two fully connected layers, one of which is used to use the ReLU activation function; and one fully connected layer is used to connect a softmax layer and output the classification result.

[0093] As a preferred implementation, step S5 further includes training a human behavior classifier, wherein the training step includes:

[0094] Collect video data of various human behaviors;

[0095] Extract continuous human posture key point feature vectors and long-term motion semantic coding in video data;

[0096] The loss function is calculated using cross entropy based on the feature vector, and the network parameters are updated using the stochastic gradient descent optimization algorithm to build a behavior semantic understanding model based on the labeled data;

[0097] By using a fully connected network and utilizing a behavioral semantic understanding model, video behavioral events are identified to output recognition results.

[0098] It should be noted that this embodiment builds a human behavior classifier based on a fully connected network, which can identify the one-dimensional feature vector output by the long-term behavior coding model and understand the behavior occurring in the video; in order to maintain the high efficiency of the model operation, the present invention designs a lightweight fully connected network architecture, such as Figure 5As shown, the network architecture includes 2 fully connected layers, of which the first fully connected layer uses the ReLU activation function, and the last fully connected layer is connected to a softmax layer, and outputs the classification result. In order to train the human behavior classifier, the present invention collects video data including a variety of human behaviors offline, and then uses the above method to extract continuous human posture key point feature vectors and long-term behavior semantic feature encoding to keep the data of various behaviors evenly distributed. Then, the cross entropy is used to calculate the loss, and the stochastic gradient descent optimization algorithm (SGD) is used to update the network parameters, so as to learn a behavior semantic understanding model from the labeled data. This embodiment connects the above-mentioned processing steps to build an end-to-end behavior recognition and semantic understanding system, realizes the detection and tracking of pedestrians from surveillance videos, and constructs short-term and long-term human behavior semantic descriptions and feature encoding modes; finally, a fully connected network is used to identify video behavior events and output recognition results.

[0099] Example 2

[0100] See also Figure 6 , Figure 6 A schematic structural block diagram of an electronic device provided in an embodiment of the present application.

[0101] An electronic device includes a memory 101, a processor 102 and a communication interface 103, wherein the memory 101, the processor 102 and the communication interface 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0102] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0103] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0104] It is understood that the structure shown in the figure is only for illustration, and a method for human behavior recognition based on long-term motion semantic coding may also include more or fewer components than those shown in the figure, or have a different configuration than those shown in the figure. Each component shown in the figure may be implemented by hardware, software or a combination thereof.

[0105] In the embodiments provided in the present application, it should be understood that the disclosed method can also be implemented in other ways. The embodiments described above are merely schematic, for example, the flowchart or block diagram in the accompanying drawings shows the possible implementation architecture, functions and operations of the method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0107] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0108] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0109] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A human behavior recognition method based on long-term motion semantic coding, characterized in that: The following steps are involved: S1, input video images, detect pedestrian targets in the image, calculate the intersection and union ratio of pedestrian detection frames in the previous and next two frames of images to track human targets; S2, using the posture estimation algorithm to extract the skeleton key points of each human target, and define the skeleton key points of the human target as the middle-level motion semantic features; S3, based on the mid-level motion semantic features, a 3D CNN network model is established to extract short-term motion semantic coding; S4, constructing a long-term motion semantic coding model according to the short-term motion semantic coding to output a one-dimensional feature vector; S5. Build a human behavior classifier based on the fully connected network to identify the one-dimensional feature vector and output the recognition result.

2. A human behavior recognition method based on long-term motion semantic coding as claimed in claim 1, characterized in that: In step S1, the calculation formula for calculating the intersection-over-union ratio of pedestrian detection frames in the two frames of images includes: Where IoU is the intersection-over-union ratio of the pedestrian detection boxes in the previous and next two frames, A is the area of ​​the detection box in the previous frame, and B is the area of ​​the detection box in the next frame.

3. A human behavior recognition method based on long-term motion semantic coding as claimed in claim 1, characterized in that: In step S2, the calculation formula of the posture estimation algorithm is: M=P(I;WP),M∈R K×H×W MS={M1,M2,...,M T },MS∈R K×T×H×W Where M is the heatmap corresponding to a single human posture, P is the posture estimation algorithm, I is the input image, WP is the network parameter of posture estimation, K is the number of skeleton key points, H is the height of the input image, W is the width of the input image, MS represents a 3D heatmap group of a video clip, which consists of T consecutive M, and T represents the number of consecutive video frames contained in the video clip.

4. A human behavior recognition method based on long-term motion semantic coding as claimed in claim 1, characterized in that: In step S3, the expression of the 3D CNN network model is: x=F(MS;W) Where F represents the 3D CNN network model, MS represents the 3D heatmap group of a video clip, W is the learning parameter of the network, and x is the output short-term motion semantic encoding.

5. The human behavior recognition method based on long-term motion semantic coding as claimed in claim 1, characterized in that: In step S4, the expression of the long-term motion semantic coding model is: X={x1,x2,...,x G }={F(MS1;W);F(MS2;W);...;F(MS G ;W)}, X∈R G×L Y v =Q(X) In the formula, F represents the 3D CNN network model, W is the learning parameter of the network, and x G is the output short-term motion semantic coding, X is the entire short-term motion semantic coding of a video action, MS G represents the 3D heatmap group of the Gth feature group video clip, L represents the length of the short temporal coding feature, G represents the number of feature groups, and Y v is the long-term motion semantic coding of a video action, and Q(X) represents the long-term motion semantic coding model.

6. A human behavior recognition method based on long-term motion semantic coding as claimed in claim 1, characterized in that: Step S4 also includes: according to the extracted short-time sequence motion semantic coding, and using the basic principle of TCN, a sequence model based on the time convolutional neural network is constructed by using causal loop convolution and dilated convolution at the same time, so as to expand the one-dimensional long-time sequence motion semantic coding model into a two-dimensional long-time sequence motion semantic coding model, and realize the long-time high-level behavior semantic coding; wherein, the calculation formula for constructing the sequence model based on the time convolutional neural network includes: Where F(i, j) is a sequence model based on a temporal convolutional neural network, X is the entire short-term motion semantic encoding of a video action, d represents the expansion factor, K is a convolution kernel of size N×k, i is the X-axis coordinate of the input and output data, j is the Y-axis coordinate of the input and output data, f is the extended convolution operation, k is the width of the convolution kernel, N is the height of the convolution kernel, u is the X-axis coordinate of the convolution kernel, and v is the Y-axis coordinate of the convolution kernel.

7. A human behavior recognition method based on long-term motion semantic coding as claimed in claim 1, characterized in that: In step S5, the human behavior classifier includes two fully connected layers, one of which is used to use the ReLU activation function; and one fully connected layer is used to connect a softmax layer and output the classification result.

8. The method for human behavior recognition based on long-term motion semantic coding as claimed in claim 5, characterized in that: Step S5 also includes training the human behavior classifier, and the training steps include: Collect video data of various human behaviors; Extract continuous human posture key point feature vectors and long-term motion semantic coding in video data; The loss function is calculated using cross entropy based on the feature vector, and the network parameters are updated using the stochastic gradient descent optimization algorithm to build a behavior semantic understanding model based on the labeled data; By using a fully connected network and utilizing a behavioral semantic understanding model, video behavioral events are identified to output recognition results.

9. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, a human behavior recognition method based on long-term motion semantic coding as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a human behavior recognition method based on long-term motion semantic coding as described in any one of claims 1 to 8 is implemented.

Images