Continuous video human body behavior positioning method based on context information

Abstract

The invention belongs to the technical field of action recognition, and provides a contextual information-based continuous video human body behavior positioning method. The candidate actions are divided into a starting stage, an executing stage and an ending stage, and correspond to candidate box preceding text information, candidate box information and candidate box following text information in sequence; using the convolutional network model to generate frame-level features of the continuous video; and respectively defining a starting frame and an ending frame by taking the starting point and the ending point of the candidate frame as centers, taking the candidate frame as a node, and aggregating node neighborhood information through an attention mechanism network to generate a context information map of the candidate frame. And forming a behavior positioning network model based on the context information atlas of the candidate box to realize effective segmentation of the video. According to the method, action stage division, feature extraction, context graph generation and model construction are fused, the quality of the candidate box is improved, adaptive aggregation of context information is realized, the boundary is refined, the defect of fuzzy boundary of a traditional method is overcome, and the precision of the candidate box is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition, and in particular to a method for locating human behavior in continuous video based on context information. Background Art

[0002] In recent years, under the call of the "New Generation Artificial Intelligence Development Plan", intelligent video analysis technology has flourished. Among them, human behavior recognition is one of the important research contents of video analysis, and it is also one of the technologies with great application prospects in the era of artificial intelligence. It has been initially applied in scenarios such as human-computer interaction, virtual reality, video retrieval, video surveillance and motion analysis.

[0003] However, the recognition accuracy of human behavior continuous video analysis technology in the existing technology is low. This is mainly due to the following reasons: the timing information of continuous videos is difficult to process, because the video sequence is not trimmed and contains multiple behavior categories and a large amount of background information, which increases the difficulty of capturing the timing information and correlation of behavioral actions; the action boundaries are fuzzy, and behavior positioning requires accurate detection of action intervals, but the action boundaries in life are highly subjective, such as the difference between eating and drinking water, which leads to a decrease in recognition accuracy; the span of behavioral actions is large, and the ratio of the duration of different actions to the video duration varies greatly, making it difficult to generate high-quality candidate frames, further affecting the recognition accuracy.

[0004] The present invention provides a method for locating human behavior in continuous video based on context information, which solves the technical problem that the boundaries of human behavior actions cannot be accurately located in the prior art, and achieves the technical effect of improving the quality of candidate frames. Summary of the invention

[0005] The present invention provides a method for locating human behavior in continuous video based on context information, so as to solve the defect of low recognition accuracy of human behavior continuous video analysis technology in the prior art.

[0006] On the one hand, the present invention provides a method for locating human behavior in continuous video based on context information, comprising:

[0007] The candidate actions are divided into action start, action execution and action end, wherein the action start corresponds to the context information of the candidate box, the action execution corresponds to the candidate box information, and the action end corresponds to the context information of the candidate box;

[0008] Based on the convolutional network model, as well as the context information of the candidate box, the candidate box information, and the context information of the candidate box, corresponding features are extracted, and frame-level features of the continuous video are generated;

[0009] Based on the frame-level features of the continuous video, a start box and an end box are defined with the start point and the end point of the candidate box as the center points, the candidate box is used as a node, and the node neighborhood is aggregated through an attention mechanism network to obtain aggregate information, and a candidate box context information map is generated according to the aggregate information;

[0010] Repeatedly generating a behavior positioning candidate frame based on the candidate frame context information graph until the behavior positioning candidate frame stops changing, and generating a behavior positioning network model;

[0011] The video to be segmented is segmented based on the behavior localization network model.

[0012] According to a method for localizing human behavior in continuous videos based on contextual information provided by the present invention, the convolutional network model is a UNet network based on 3D full convolution, and the convolutional network model consists of a symmetrical encoder and decoder, which are connected to each other through a 3D convolutional neural network.

[0013] According to a method for locating human behavior in continuous video based on context information provided by the present invention, the encoder is used to extract and compress feature information, and the decoder is used to gradually sample and restore the compressed feature information.

[0014] According to a method for localizing human behavior in continuous videos based on contextual information provided by the present invention, the encoder is also used to perform downsampling based on 3D full convolution, and reduce the spatial dimension of the feature information through several convolutional layers and pooling layers, and generate a feature map.

[0015] According to a method for locating human behavior in continuous video based on context information provided by the present invention, the encoder is also used to process the input feature map through a plurality of convolutional layers, and after each convolution operation, the nonlinear ability of the network is increased through a ReLU activation function;

[0016] According to a method for localizing human behavior in continuous video based on contextual information provided by the present invention, the encoder is also used to reduce the spatial size of the feature map based on a maximum pooling operation after processing the input feature map.

[0017] According to a method for positioning human behavior in continuous videos based on contextual information provided by the present invention, the 3D full convolution is performed based on an I3D model, and the I3D model is composed of a number of 3D convolutional layers, a number of Inc modules, a number of 3D maximum pooling layers, and a number of 3D average pooling layers; wherein, among the several 3D convolutional layers, except for the last convolutional layer, a ReLU activation function is used after the other convolutional layers.

[0018] According to a method for localizing human behavior in continuous video based on contextual information provided by the present invention, the attention mechanism network is a mamba model.

[0019] The method for localizing human behavior in continuous video based on context information provided by the present invention captures the context information of each candidate box by constructing a candidate box boundary graph network, further refines the boundary of the candidate action by using the contextual local detail information of the candidate box, and improves the quality of the candidate box; optimizes the information of adjacent nodes of the candidate box by using the Mamba model, realizes adaptive aggregation of the context information of the candidate box, refines the boundary of the candidate box, makes up for the defect of blurred boundaries caused by the subjectivity of the action boundary in the traditional method, further improves the quality of the candidate box, solves the technical problem of low recognition accuracy of the human behavior continuous video analysis technology in the prior art, and achieves the beneficial effect of improving the quality of the candidate box. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 1 is a flow chart of a method for locating human behavior in continuous video based on context information provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of frame-level feature extraction provided by an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of the I3D model structure provided by an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of the Inc module structure provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of constructing a context information graph provided by an embodiment of the present invention;

[0026] Figure 6 It is a schematic diagram of the Mamba architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Combine the following Figure 1-Figure 6 The present invention describes a method for localizing human behavior in continuous video based on context information.

[0029] Figure 1 It is a flow chart of a method for locating human behavior in continuous video based on context information provided by an embodiment of the present invention.

[0030] like Figure 1 As shown, the method for locating human behavior in continuous video based on context information provided by an embodiment of the present invention includes:

[0031] S1 divides the candidate actions into action start, action execution and action end, wherein the action start corresponds to the context information of the candidate box, the action execution corresponds to the candidate box information, and the action end corresponds to the context information of the candidate box;

[0032] S2 extracts corresponding features based on the convolutional network model, the context information of the candidate box, the candidate box information, and the context information of the candidate box, and generates frame-level features of the continuous video;

[0033] S3 defines a start box and an end box based on the frame-level features of the continuous video, with the start point and the end point of the candidate box as the center points, respectively, takes the candidate box as a node, aggregates the node neighborhood through an attention mechanism network to obtain aggregate information, and generates a candidate box context information map according to the aggregate information;

[0034] S4 repeatedly generates a behavior positioning candidate frame based on the candidate frame context information graph until the behavior positioning candidate frame stops changing, and generates a behavior positioning network model;

[0035] S5 performs segmentation processing on the video to be segmented based on the behavior localization network model.

[0036] Figure 2 It is a schematic diagram of frame-level feature extraction provided by an embodiment of the present invention.

[0037] like Figure 2As shown, the present invention adopts a UNet network based on 3D full convolution, which is composed of a symmetrical encoder and decoder, and is connected to each other through a 3D convolutional neural network. This structure not only promotes the effective fusion of low-level detail features and high-level abstract features, but also significantly improves the network's ability to capture details.

[0038] The encoder is responsible for extracting rich feature information from the input. The present invention uses 3D full convolution for downsampling, and gradually reduces the spatial dimension of the feature through a series of convolutional layers and pooling layers, so as to achieve efficient encoding of the video content and increase the depth of the feature map to capture features from local to global. In each stage of the encoder, the input feature map is first processed by a 3*3 convolutional layer, and more complex features are extracted by increasing the depth of the convolutional layer without changing the spatial size of the feature map. After each convolution operation, the nonlinear ability of the network is increased by the ReLU activation function, so that the model can learn more complex feature representations. After the convolution layer, the encoder uses a 2*2 filter and a maximum pooling operation with a step size of 2 at each stage to reduce the spatial size of the feature map, so that the encoder can gradually abstract higher-level feature representations from the input as the network depth increases. As the feature map is passed down along the encoder, its spatial size gradually decreases, and the number of channels increases accordingly, gradually building a deep, multi-scale understanding of the input image. The feature map at the bottom of the encoder contains the most comprehensive semantic information about the input image, providing rich contextual information for the subsequent decoding process.

[0039] The decoder is responsible for gradually upsampling and restoring the features compressed by the encoder, and finally generating a segmentation map of the same size as the original video, thereby achieving accurate pixel-level segmentation. The present invention uses 3D convolution and 3D deconvolution to form a decoder, which captures global feature information while restoring local detail information to ensure the accuracy and delicacy of the segmentation result. The decoder is composed of multiple upsampling modules, each of which includes an upsampling operation and a convolution operation. The spatial size of the feature map is gradually restored through the upsampling operation, reversing the spatial compression caused by the pooling operation in the encoder. After each upsampling, the decoder processes the upsampled feature map through a convolution layer to refine and optimize the feature representation. These convolution layers also use a 3*3 convolution kernel, introduce nonlinearity through the ReLU activation function, and enhance the expression ability of the model. The present invention uses a 3D deconvolution layer to upsample the feature map of the last convolution layer of the encoder to restore it to the same length as the input video. 3D convolution downsampling is performed simultaneously in time and space, while 3D deconvolution only samples in the time dimension to maintain the generalization of features. At the same time, the output of the convolution layer is compressed in space through a 3D convolution as the input of the corresponding 3D deconvolution layer, enhancing the effect of deconvolution and obtaining more detailed information.

[0040] Figure 3 It is a schematic diagram of the I3D model structure provided by an embodiment of the present invention.

[0041] like Figure 3 As shown, the 3D convolution of the present invention adopts the I3D model structure, and the existing I3D model can use the pre-training mode to improve the effectiveness of the features. The I3D model is obtained after the expansion of the Inception-V1 architecture, that is, the original 2D convolution kernel and pooling operation for the two-dimensional matrix are converted into 3D convolution by adding a time dimension. It is composed of a 3D convolution layer, an Inc module, a 3D maximum pooling layer, and a 3D average pooling layer. In the entire network model, except for the last convolution layer that generates the score of each category, a ReLU activation function is used after other convolution layers.

[0042] Figure 4 It is a schematic diagram of the Inc module structure provided by an embodiment of the present invention.

[0043] like Figure 4 As shown, the Inc module structure of the present invention, after obtaining the input features from the previous layer, performs 1*1*1, 3*3*3, 3*3*3 convolution operations and a 3*3*3 maximum pooling operation in parallel, and splices the four parts of the calculation result channels into the next layer. In order to reduce the number of parameters in the network, in addition to the 1*1*1 convolution operation channel, a 1*1*1 convolution module is added to the remaining calculation channels.

[0044] Figure 5 It is a schematic diagram of constructing a context information graph provided by an embodiment of the present invention.

[0045] like Figure 5 As shown, the present invention constructs multi-scale start boxes and end boxes at both ends of the candidate action to provide context information for the candidate box. First, according to the length of the candidate box, the start box and end box of different sizes are designed with the start point and end point of the candidate box as the center point, and each box is regarded as a node of the graph structure, and the node of each candidate box is connected to the start box and end box of different sizes. At the same time, in order to capture more context information, the quality of the candidate box is measured by the area intersection and union ratio (IoU). When the area of ​​the generated candidate box has a higher overlap rate with the area of ​​the real candidate box, the higher the IoU, the higher the quality of the generated candidate box, and the higher the accuracy of the behavior positioning. If the IoU of different start boxes and end boxes is greater than a certain threshold, their nodes are also connected. Then the attention mechanism network is used to complete the node neighborhood aggregation information, thereby realizing the refinement of the candidate box by context information. At all adjacent nodes of the node, the weights of the adjacent nodes are first learned through a single-layer feedforward neural network, and the Softmax function is introduced to regularize all adjacent nodes.

[0046] Figure 6 It is a schematic diagram of the Mamba architecture provided by an embodiment of the present invention.

[0047] The present invention adopts the Mamba architecture to capture more contextual information through the memory initialization and selection mechanism of the high-order polynomial projection operator, and improves the long-range linear time series modeling capability of the state space model (SSM). Among them, the initialization strategy can construct a coherent hidden state matrix to effectively promote long-range memory. Then, the selection mechanism enables the SSM to obtain the representation of the perceptible content.

[0048] like Figure 6 As shown, the Mamba architecture provided by the embodiment of the present invention is centered on SSM, in which the task of the selective SSM layer is to perform the mapping from the input sequence X to Y. By treating the state space model as a semi-separable matrix through matrix transformation, Mamba can decompose the calculation into matrix blocks, where the diagonal blocks represent intra-block calculations. The non-diagonal blocks represent inter-block calculations decomposed by the hidden state of SSM. By creating a mapping from [X, A, B, C] to Y, it is implemented by using a single projection at the starting point of the block to process [X, A, B, C] simultaneously. Then, parallel association scanning is used to obtain the output Y. After that, a jump connection is used to encourage feature reuse and alleviate the performance degradation problem that often occurs during model training. Finally, the Mamba model can be constructed by stacking this module with standard normalization layers and residual connections in an interlaced manner.

[0049] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for locating human behavior in continuous video based on context information, characterized in that: include: The candidate actions are divided into action start, action execution and action end, wherein the action start corresponds to the context information of the candidate box, the action execution corresponds to the candidate box information, and the action end corresponds to the context information of the candidate box; Based on the convolutional network model, as well as the context information of the candidate box, the candidate box information and the context information of the candidate box, corresponding features are extracted, and frame-level features of the continuous video are generated; Based on the frame-level features of the continuous video, a start box and an end box are defined with the start point and the end point of the candidate box as the center points, the candidate box is used as a node, and the node neighborhood is aggregated through an attention mechanism network to obtain aggregate information, and a candidate box context information map is generated according to the aggregate information; Repeatedly generate a behavior positioning candidate frame based on the candidate frame context information graph until the behavior positioning candidate frame stops changing, and generate a behavior positioning network model; The video to be segmented is segmented based on the behavior localization network model.

2. The method for locating human behavior in continuous video based on context information according to claim 1, characterized in that: The convolutional network model is a UNet network based on 3D full convolution, and the convolutional network model consists of a symmetrical encoder and decoder, which are connected to each other through a 3D convolutional neural network; The encoder is used to extract and compress feature information, and the decoder is used to gradually sample and restore the compressed feature information.

3. The method for locating human behavior in continuous video based on context information according to claim 2, characterized in that: The encoder is also used to perform downsampling based on 3D full convolution, reduce the spatial dimension of the feature information through a number of convolutional layers and pooling layers, and generate a feature map.

4. The method for locating human behavior in continuous video based on context information according to claim 3, characterized in that: The encoder is also used to process the input feature map through a plurality of convolutional layers, and after each convolution operation, increase the nonlinear ability of the network through a ReLU activation function; The encoder is also used to reduce the spatial size of the feature map based on a maximum pooling operation after processing the input feature map.

5. The method for locating human behavior in continuous video based on context information according to claim 2, characterized in that: The 3D full convolution is performed based on the I3D model, which consists of several 3D convolutional layers, several Inc modules, several 3D maximum pooling layers and several 3D average pooling layers; wherein, In the three-dimensional convolutional layers, except the last convolutional layer, a ReLU activation function is used after the other convolutional layers.

6. The method for locating human behavior in continuous video based on context information according to claim 2, characterized in that: The attention mechanism network is a mamba model.

Images