A human behavior recognition method and device based on image feature mapping

Abstract

The application relates to the technical field of computer vision, and provides a human behavior recognition method and device based on image feature mapping, wherein a video block is obtained by pre-processing an input video, the video block is encoded through a convolutional neural network, feature conversion of the video block coding is carried out, the video block coding is mapped from the image field to the natural language field, the video block coding is input into a language model, the language model is used as a decoder to complete human behavior recognition, the sequence reasoning capability of the language model is used to understand the content and sequence in the input video, the language model outputs a processing result, the processing result is matched with a recognition category, and the recognition result of the input video is obtained, the language model is applied to video image recognition, the processing capability of the language model is utilized, and the accuracy of video image recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a human behavior recognition method and device based on image feature mapping. BACKGROUND

[0002] In the prior art, behavior recognition methods are mainly divided into two categories, namely traditional methods based on artificial design features and deep learning methods. Compared with traditional methods, deep learning methods can extract higher-level features in images to express behavior characteristics through learning, and have become the mainstream method in the field of behavior recognition. At present, the mainstream network for solving behavior recognition is the convolutional neural network and the long short-term memory network. The convolutional neural network can well learn the spatial information of behavior actions through convolution and other operations, but does not fully consider the time sequence of the actions. The long short-term memory network mainly extracts time sequence features, and does not sufficiently mine spatial information.

[0003] At present, large language models appear in the field of natural language processing. These models usually contain tens of billions to hundreds of billions of parameters and are trained on large text corpora. Large language models have successfully proven their amazing ability in sequential causal reasoning. Although large language models have achieved excellent results on natural language tasks, if language models can be applied to video image recognition, the processing power of language models can be utilized to improve the accuracy of video image recognition.

[0004] Therefore, it is urgent to overcome the defects of the prior art in the technical field. SUMMARY

[0005] The technical problem to be solved by the present application is how to apply language models to video image recognition to improve the effect of video image recognition.

[0006] In a first aspect, a human behavior recognition method based on image feature mapping is provided, comprising:

[0007] preprocessing an input video to obtain a video tile;

[0008] encoding the video tile through a convolutional neural network to obtain video encoding information;

[0009] calculating position encoding information according to the video encoding information;

[0010] adding the video encoding information and the position encoding information to obtain complete encoding of the video tile;

[0011] performing feature conversion on the complete encoding of the video tile through an input layer of a language model to obtain language form encoding of the video tile;

[0012] encoding the language form into a language model, an output layer of the language model outputs a result encoding, and the result encoding corresponds to a recognition result of the input video.

[0013] Preferably, the pre-processing of the input video obtains video tiles, specifically including:

[0014] extracting E+1 continuous video pictures in time sequence from the input video;

[0015] The input dimension of the convolutional neural network is (a, n, m), each video picture is scaled according to the input dimension of the convolutional neural network, and the scaled video picture is cut into n*m video tiles.

[0016] Preferably, the video tiles are encoded by the convolutional neural network to obtain video encoding information, specifically including:

[0017] The n*m video tiles are respectively encoded according to the input dimension of the convolutional neural network to obtain three-dimensional encoding information of each video tile;

[0018] The three-dimensional encoding information of each video tile is extracted and unfolded after feature extraction to obtain one-dimensional encoding information of each video tile;

[0019] According to the one-dimensional encoding information of each video tile and the corresponding video picture of each video tile, video encoding information of each video tile is obtained.

[0020] Preferably, the three-dimensional encoding information of each video tile is extracted and unfolded after feature extraction to obtain one-dimensional encoding information of each video tile, specifically including:

[0021] The three-dimensional encoding information of each video tile is processed and encoded through multiple convolutional layers and pooling layers to obtain first-type three-dimensional encoding information of each video tile;

[0022] The first-type three-dimensional encoding information of each video tile is added with a maxpooling layer and encoded to obtain second-type three-dimensional encoding information of each video tile, and the dimension of the second-type three-dimensional encoding information is (b, c, d);

[0023] The second-type three-dimensional encoding information is unfolded into a b*c*d length one-dimensional vector, and the b*c*d length one-dimensional vector is encoded to obtain the one-dimensional encoding information.

[0024] Preferably, the dimension of the video encoding information is (E, b*c*d).

[0025] Preferably, the position encoding information is calculated according to the video encoding information, and specifically includes:

[0026] The video encoding information is calculated according to the following formula to obtain the position encoding information:

[0027] When the sequence position number of the video tile in the respective video picture is even:

[0028] ;

[0029] When the sequence position number of the video tile in the respective video picture is odd:

[0030] ;

[0031] Wherein, PE (pos,i) is the position encoding information, pos is the encoding information of each video picture corresponding to each video tile, pos∈[0,1,2…E-1], i is the sequence position of each video tile in the respective video picture, i∈[0,1,2…b*c*d-1].

[0032] Preferably, the complete encoding of the video tile is subjected to feature conversion through an input layer of the language model to obtain language form encoding of the video tile, and specifically includes:

[0033] According to the input vector length f of the language model, the video encoding information is divided into b*c*d / f one-dimensional vector encoding information, and the language form encoding is obtained according to the sequence number of the corresponding video picture;

[0034] The dimension of the language form encoding of each video tile is (E*b*c*d / f, f).

[0035] Preferably, the language form encoding is input into the language model, the output layer of the language model outputs result encoding, and the result encoding corresponds to the recognition result of the input video, and specifically includes:

[0036] In the one-dimensional vector encoding output by the language model, the first preset number of values are taken as the result encoding;

[0037] K+1 neurons in the language model are respectively corresponding to K+1 recognition categories;

[0038] The result encoding and the parameters of K+1 neurons in the output layer of the language model are respectively subjected to point multiplication to obtain a point multiplication result corresponding to each neuron;

[0039] The maximum dot product result in the dot product results corresponding to the K+1 neurons is obtained, and the identification category corresponding to the maximum dot product result is taken as the identification result.

[0040] Preferably, when training the language model, the parameters of the input layer and the parameters of the output layer are trained by a loss function, and the loss function is as follows:

[0041] Loss = -(y*log(s) + (1-y)*log(1-y));

[0042] Wherein, Loss is a loss value; y is a true value of a training sample, when the training sample is correct, y = 1, when the training sample is wrong, y = 0; s is a probability of the input video predicted by the language model as the corresponding identification category.

[0043] In a second aspect, a human behavior recognition device based on image feature mapping is provided, comprising at least one processor, and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the human behavior recognition method based on image feature mapping.

[0044] The present application relates to the technical field of computer vision, and provides a human behavior recognition method and device based on image feature mapping, wherein a video block is obtained by preprocessing an input video, the video block is encoded by a convolutional neural network, and then the encoding of the video block is converted into features, so that the encoding of the video block is mapped from the image field to the natural language field, and then the encoding of the video block is input into a language model, the language model is used as a decoder to complete the recognition of human behavior, the sequence reasoning capability of the language model is used to understand the content and sequence in the input video, the language model outputs a processing result, the processing result is compared with an identification category, and the identification result of the input video is obtained, the language model is applied to the recognition of video images, the processing capability of the language model is used, and the accuracy of video image recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0046] Figure 1 It is a method flowchart of a human behavior recognition method based on image feature mapping provided by the embodiments of the present application;

[0047] Figure 2 is a method flowchart of encoding of a video tile of a human behavior recognition method based on image feature mapping provided by an embodiment of the present application;

[0048] Figure 3 is another method flowchart of encoding of a video tile of a human behavior recognition method based on image feature mapping provided by an embodiment of the present application;

[0049] Figure 4 is a method flowchart of generation of a language model recognition result of a video tile of a human behavior recognition method based on image feature mapping provided by an embodiment of the present application;

[0050] Figure 5 is a device schematic diagram of a human behavior recognition device based on image feature mapping provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0052] In the description of the present application, the terms "inner", "outer", "longitudinal", "transverse", "upper", "lower", "top", "bottom", and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and do not require the present application to be constructed and operated in a particular orientation, therefore should not be understood as limiting the present application.

[0053] In the present application, the terms "first", "second", and the like are only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified and limited, the meaning of "multiple" is two or more.

[0054] In the present application, unless otherwise specified and limited, the term "connection" should be understood broadly, for example, "connection" can be fixed connection, or detachable connection, or integral; can be directly connected, or indirectly connected through intermediate medium. In addition, the term "coupling" can be an electrically connected mode for realizing signal transmission.

[0055] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0056] Embodiment 1:

[0057] Embodiment 1 of the present application provides a human behavior recognition method based on image feature mapping, as shown in the figure, the method flow includes: Figure 1

[0058] In step 101, the input video is preprocessed to obtain video tiles.

[0059] In this embodiment, the input video refers to the video that needs to be recognized, and the content in the input video can be a video of human behavior action, etc. Before converting the input video into corresponding encoding, the video needs to be converted into multiple video pictures, and then the video pictures are segmented to obtain the video tiles. Each small part of the video tiles in the video pictures is encoded and recognized separately, which conforms to the logic of image recognition of the model. It should be noted that in this embodiment, the single behavior action of the human body in the input video usually lasts for a short time, so the number of video pictures is usually not too many.

[0060] In step 102, the video tiles are encoded by a convolutional neural network to obtain video encoding information.

[0061] In this embodiment, the video tiles are first encoded and feature-extracted by a convolutional neural network. The convolutional neural network can learn the spatial information of the behavior action well through convolution and other operations, which facilitates feature extraction of each video tile. After feature extraction and encoding by the convolutional neural network, a one-dimensional vector is generated for each video tile, and a two-dimensional vector is generated by combining the corresponding video picture of each video tile, which is used as the video encoding information. The video encoding information represents the content information of the corresponding video tile, which is used for subsequent recognition.

[0062] In step 103, the position encoding information is calculated according to the video encoding information.

[0063] In step 104, the video encoding information and the position encoding information are added to obtain the complete encoding of the video tiles.

[0064] After encoding the video tiles, considering that the video pictures in the input video have time sequence, a position vector is added to represent the position of each video tile or the distance between the video tiles, so that the model can ensure the time sequence in the recognition process when recognizing the encoding information of all video tiles of the entire input video, thereby forming a complete video picture and video according to the time sequence of all video tiles. The complete encoding is obtained by combining the video encoding information of the video tiles and the position encoding information of the video tiles, which ensures that the complete encoding of each video tile contains both the feature content of the video tile and the time sequence information of the video tile.​

[0065] In step 105, the complete encoding of the video block is converted into a language form encoding through the input layer of the language model.

[0066] After the complete encoding of the video block is input into the language model, the complete encoding of the video block needs to be converted into an encoding form that can be read and recognized by the language model, that is, the language form encoding. Therefore, in this embodiment, after the complete encoding is input into the language model, a trained input layer is arranged in the language model, which is used to convert the input complete encoding of the video block. The mapping of the complete encoding of the video block from the image field to the natural language field is completed through a linear function.

[0067] In this embodiment, the language model can be a generative pre-trained model (Generative Pre-trained Transformer, GPT for short) series language model.

[0068] In step 106, the language form encoding is input into the language model, and the output layer of the language model outputs a result encoding, and the result encoding corresponds to the recognition result of the input video.

[0069] The existing large language model contains tens of billions to hundreds of billions of parameters, and has excellent effect in sequence causal reasoning. In this embodiment, the input video is preprocessed and encoded, and the features of the video image are mapped into natural language corpus. The language model is used as a decoder to complete the recognition of human behavior, and the sequence reasoning ability of the language model is used to understand the sequence in the input video. In this embodiment, the input and output of the language model are usually trained in advance, that is, how to convert the encoding information of the input video into features, and how to decode the output result processed by the language model and map it into the corresponding recognition result. In this embodiment, the result encoding is the output result obtained after the language form encoding processed by the language model, and then the result encoding and the pre-set recognition categories are judged, and the recognition category of the input video is obtained by combining the result encoding of all video blocks. In this embodiment, the language model is usually set in advance to set all possible recognition categories corresponding to the input video. Through the network structure of the full connection layer, the result encoding output by the language model is mapped to the recognition category of the video behavior to obtain the final recognition result.

[0070] In this embodiment, the video blocks are obtained by preprocessing the input video, and the video blocks are encoded by the convolutional neural network, and then the encoding of the video blocks is converted into features, which maps the encoding of the video blocks from the image field to the natural language field, so as to input the encoding of the video blocks into the language model, and the language model as a decoder completes the recognition of human behaviors, and the sequence reasoning ability of the language model is used to understand the content and sequence in the input video, and the language model outputs the processing result and corresponds the processing result to the recognition categories to obtain the recognition result of the input video, and the language model is applied to the recognition of video images, the processing ability of the language model is used to improve the accuracy of video image recognition.

[0071] In this embodiment, the input video cannot be directly encoded, and the model usually needs to perform corresponding recognition processing on each frame in the video and each block in each frame when performing corresponding processing and recognition on the video, so this embodiment relates to the following design:

[0072] The preprocessing of the input video to obtain the video blocks specifically includes:

[0073] The continuous E+1 video pictures are extracted from the input video in chronological order, and the input dimension of the convolutional neural network is (a, n, m), and the scaled video pictures are scaled and cut into n*m video blocks according to the input dimension of the convolutional neural network.

[0074] In this embodiment, since the input dimension of the convolutional neural network is a three-dimensional vector of (a, n, m), the input video is first sliced in chronological order to obtain a plurality of video pictures, and then each video picture is scaled and cut according to the input dimension of the convolutional neural network, wherein n is used as the cutting number of the length of the video picture, and m is used as the cutting number of the width of the video picture, so as to facilitate subsequent encoding of each video block according to n*m and inputting into the convolutional neural network, and a is used as the channel number of each video block; wherein E, a, n and m are positive integers.

[0075] In this embodiment, after preprocessing the input video, the video blocks need to be encoded for identifying the content of the video block part, so this embodiment also relates to the following design:

[0076] The video block is encoded by the convolutional neural network to obtain video encoding information, as shown in Figure 2 The method flow includes:

[0077] In step 201, the n*m video blocks are respectively encoded according to the input dimension of the convolutional neural network to obtain three-dimensional encoding information of each video block.

[0078] In this embodiment, the input dimension of the convolutional neural network is (a, n, m), so each video tile is encoded according to three dimensions of channel number, length and width to obtain three-dimensional vector three-bit encoding information with a dimension of (a, n, m).

[0079] In this embodiment, the convolutional neural network can be ResNet18.

[0080] In step 202, the three-dimensional encoding information of each video tile is unfolded after feature extraction to obtain one-dimensional encoding information of each video tile.

[0081] The convolutional neural network continues to process the three-dimensional encoding information through multiple convolutional layers and pooling layers, and adaptively converts the convolutional neural network. Since the existing language model mostly uses two-dimensional vector encoding for input and output, the overall content of the video tile needs to be encoded into a one-dimensional vector in addition to the video picture as one of the vectors. Therefore, the converted three-dimensional vector encoding is finally unfolded into a corresponding one-dimensional vector and encoded to obtain one-dimensional encoding information.

[0082] In step 203, the video encoding information of each video tile is obtained according to the one-dimensional encoding information of each video tile and the serial number of the video picture corresponding to each video tile.

[0083] After obtaining the one-dimensional encoding information of each video tile, the corresponding two-dimensional vector encoding information is obtained according to the corresponding video picture, so that the content information of the video tile and the video picture corresponding to the video tile are obtained simultaneously according to the video encoding information.

[0084] In this embodiment, during the feature extraction and encoding of the video tile by the convolutional neural network, the encoding needs to be converted multiple times for optimization and form adjustment to ensure that the subsequent model can perform corresponding feature recognition and feature conversion. Therefore, this embodiment involves the following design:

[0085] The three-dimensional encoding information of each video tile is unfolded after feature extraction to obtain one-dimensional encoding information of each video tile, as shown in Figure 3 The method flow includes:

[0086] In step 301, the three-dimensional encoding information of each video tile is processed through multiple convolutional layers and pooling layers and encoded to obtain first-type three-dimensional encoding information of each video tile.

[0087] In step 302, a maxpooling layer is added to the first type of three-dimensional coding information of each video tile, and the first type of three-dimensional coding information is coded to obtain the second type of three-dimensional coding information of each video tile, the dimension of the second type of three-dimensional coding information being (b, c, d).

[0088] In this embodiment, in the process of using the convolutional neural network ResNet18 to extract and code the features of the video tile, the first type of three-dimensional coding information is usually obtained by coding the three-dimensional vector with a new dimension after at least 17 convolutional layers and pooling layers are processed. In order to reduce the number of 0 outputs of the ReLU activation function and reduce the parameter amount of the overall coding, a maxpooling layer is added after the features extracted by the convolutional neural network ResNet18, and the second type of three-dimensional coding information is coded according to the three-dimensional vector with a new dimension (b, c, d). The dimension is determined according to the output form of the convolutional neural network ResNet18, and the (b, c, d) dimension can be (512, 3, 3). Wherein, b, c and d are positive integers.

[0089] In step 303, the second type of three-dimensional coding information is vector expanded to a one-dimensional vector with a length of b*c*d, and the one-dimensional vector with a length of b*c*d is coded to obtain the one-dimensional coding information.

[0090] Since the existing language model mostly uses two-dimensional vector coding for input and output, the overall content of the video tile needs to be coded into a one-dimensional vector in addition to the video picture as one of the dimensions of the vector. Therefore, the coding of the converted three-dimensional vector is expanded into a corresponding one-dimensional vector, that is, the three dimensions b, c and d of the second type of three-dimensional coding information are multiplied to obtain a one-dimensional vector with a length of b*c*d, and then the one-dimensional vector with a length of b*c*d is combined with the dimension of the video picture. The total number of video pictures is E+1, so the one-dimensional vector with a length of E+1 is combined with the one-dimensional vector with a length of b*c*d to obtain a two-dimensional vector with a dimension of (E, b*c*d), and the two-dimensional vector is coded to obtain the video coding information.

[0091] In this embodiment, after the video tile is coded, the order of the video tile in the video also needs to be considered, so that the subsequent language model can process the coding information of each video tile in time sequence when processing and identifying the coding information, so that each video tile can be identified in the original video playing order. Therefore, a position vector needs to be added to identify the position of each video tile or the distance between different video tiles. Therefore, this embodiment relates to the following design:

[0092] The position coding information is calculated according to the video coding information, and specifically includes:

[0093] The position encoding information is obtained by the video encoding information according to the following formula:

[0094] When the sequence position number of the video tile in the respective video picture is even:

[0095]

[0096] When the sequence position number of the video tile in the respective video picture is odd:

[0097]

[0098] where PE is the position encoding information, pos is the encoding information of the video picture corresponding to each video tile, pos ∈ [0, 1, 2…E], i is the sequence position of each video tile in the respective video picture, i ∈ [0, 1, 2…b*c*d-1]. (pos,i)

[0099] It should be noted that the dimension of the position encoding information and the dimension of the video encoding information are the same, both are two-dimensional vectors with a dimension of (E, b*c*d).

[0100] After obtaining the position encoding of each video tile, the complete encoding of the video tile is obtained by adding the video encoding information and the position encoding information, and the complete encoding is a two-dimensional vector with a dimension of (E, b*c*d).

[0101] In this embodiment, after the complete encoding is calculated, the complete encoding needs to be converted according to the input layer of the language model, and the mapping of the complete encoding of the video tile from the image field to the natural language field is completed by a linear function, so the following design is designed in this embodiment:

[0102] The complete encoding of the video tile is converted by the input layer of the language model to obtain the language form encoding of the video tile, specifically including:

[0103] According to the input vector length f of the language model, the video encoding information is divided into b*c*d / f one-dimensional vector encoding information; the dimension of each language form encoding is (E*b*c*d / f, f).

[0104] ​​​In the embodiment, the encoding information of the b*c*d-length one-dimensional vector used to represent the content of each video tile in the complete encoding of the video tile is grouped into a group according to every f-length one-dimensional vector, to obtain b*c*d / f one-dimensional vectors of encoding information, and the number of encoded video pictures is also changed to E*b*c*d / f, so that the dimension of each language form encoding is (E*b*c*d / f, f); where f is a positive integer.

[0105] In the embodiment, after the feature conversion, the language form encoding needs to be input into the language model, and the language model outputs the corresponding result encoding after recognizing the language form encoding. At this time, the result encoding also needs to pass through the output layer of the language model, and the result encoding and the pre-set recognition categories are judged to obtain the recognition category of the input video by combining the result encodings of all video tiles. In the embodiment, the language model is usually set in advance to set all possible recognition categories corresponding to the input video. By using the network structure of the full connection layer, the result encoding output by the language model is mapped to the recognition category of the video behavior to obtain the final recognition result. Therefore, the embodiment also relates to the following design:

[0106] The language form encoding is input into the language model, the output layer of the language model outputs the result encoding, and the result encoding is corresponded to the recognition result of the input video, as shown in Figure 4 The method flow includes:

[0107] In step 401, the first preset number of values in the one-dimensional vector encoding output by the language model are taken as the result encoding.

[0108] In the embodiment, after the language form encoding is input into the language model, the form of the encoding information output by the existing GTP series language model is usually a one-dimensional vector form, and usually only the first preset number of encoding values in the entire encoding information are taken as the vector for output, which is used for the subsequent output layer to be mapped to the corresponding neurons, so as to be corresponding to the corresponding recognition categories. The preset value is set by the person skilled in the art, and in the embodiment, the preset value is U, and U is a positive integer.

[0109] In step 402, K+1 neurons in the language model are respectively corresponding to K+1 recognition categories.

[0110] In this embodiment, the corresponding recognition categories need to be set in the language model, and then the output result is encoded and corresponding judgment is made on each recognition category to obtain the category corresponding to the input video. In this embodiment, the output layer of the language model adopts a network structure of a full connection layer, K fixed categories are preset, and one category is additionally set as other results other than the K fixed categories, so K+1 recognition categories need to be set, and K+1 neurons are set in the full connection layer of the language model, each neuron corresponding to one recognition category; wherein K is a positive integer.

[0111] For example, the above description is as follows:

[0112] For recognizing the double-hand action, the categories of the double-hand action include: clapping hands and shaking hands, so K is 2, 3 neurons need to be set in the full connection layer of the language model, and the 3 neurons correspond to the categories: clapping hands, shaking hands, and actions that are neither clapping hands nor shaking hands.

[0113] In step 403, the result code and the parameters of the K+1 neurons in the output layer of the language model are respectively multiplied to obtain the dot product result corresponding to each neuron.

[0114] In this embodiment, the result code is a 1XU matrix, and the parameters of each corresponding neuron are a UX1 matrix. The result code and the parameters of each neuron are respectively multiplied to obtain a corresponding binary number as the dot product result.

[0115] In step 404, the maximum dot product result of the K+1 neurons is obtained, and the recognition category corresponding to the maximum dot product result is taken as the recognition result.

[0116] It is judged whether the corresponding dot product result is 0 or 1. When the dot product result is 0, the recognition category corresponding to the dot product result is not the category of the current input video. When the dot product result is 1, the recognition category corresponding to the dot product result is the category of the current input video. Since only one category result is 1 and the rest are 0 in all categories, the maximum dot product result is selected, and the recognition category corresponding to the maximum dot product result is taken as the recognition result.

[0117] In this embodiment, for the language model, the input layer and the output layer of the language model need to be trained to ensure the feature conversion and form conversion of the language model for the encoding information of the video, ensure the correctness of the input and output, and the correspondence with each recognition category.

[0118] When the language model is trained, the parameters of the input layer and the parameters of the output layer are trained by a loss function, and the loss function is as follows:

[0119] Loss = -(y * log(s) + (1 - y) * log(1 - y));

[0120] wherein, Loss is a loss value; y is a true value of a training sample, when the training sample is correct, y = 1, when the training sample is incorrect, y = 0; s is a probability of a language model predicting that the input video is a corresponding recognition category.

[0121] In this embodiment, when training, an open source ResNet18 model and a GPT-2 model that have been optimized can be used, and the parameters of the ResNet18 model and the GPT-2 model part can not be updated, and mainly the output layer and the input layer are trained.

[0122] Embodiment 2:

[0123] This embodiment 2 based on embodiment 1, in actual situation shows the human behavior recognition method based on image feature mapping as follows:

[0124] 100 short videos about handshaking are collected, 6 video pictures are extracted from each short video, each video picture is converted into 224*224 size through scaling and cutting means, and a total of 224*224 video blocks are obtained.

[0125] In this embodiment, a convolutional neural network ResNet18 is used for feature extraction to complete the encoding of each video block.

[0126] The input dimension of the convolutional neural network ResNet18 network is (3, 224, 224), so each video block is first encoded according to the dimension of (3, 224, 224) to obtain the three-dimensional encoding information, and the three dimensions in (3, 224, 224) represent the channel number, length and width of the video block, respectively.

[0127] After the three-dimensional encoding information is processed through 17 convolutional layers and pooling layers, the output dimension is (512, 7, 7), and the first type of three-dimensional encoding information is obtained according to the dimension of (512, 7, 7).

[0128] In order to reduce the number of 0 outputs due to the ReLU activation function and reduce the parameter amount, a maxpooling layer is added after the ResNet18 extracted features, and the final output dimension is (512, 3, 3). The second type of three-dimensional encoding information is obtained according to the dimension of (512, 3, 3).

[0129] After the feature extraction of the image is completed, the second type of three-dimensional encoding information is unfolded into a three-dimensional vector to be converted into a 3x3x512=4608 one-dimensional vector, that is, a video tile uses a one-dimensional vector with a length of 4608 to be encoded to obtain one-dimensional encoding information, and the one-dimensional encoding information of each video tile is obtained with the respective video picture to obtain video encoding information, and the encoding dimension of the video encoding information is (6, 4608).

[0130] According to the video encoding information, a position encoding vector with a dimension of (6, 4608) is generated by a trigonometric function, and the complete encoding of the video tile is obtained by adding the video encoding information and the position encoding information.

[0131] In this embodiment, the GPT-2 model of OpenAI is used as a language model, and in the GPT-2 model, an embedding vector with a length of 768 is used to represent a word. In this embodiment, the one-dimensional vector with a length of 4608 in the video encoding information with a dimension of (6, 4608) output in the previous step needs to be divided into six one-dimensional vectors with a length of 768, so as to convert the video encoding information with a dimension of (6, 4608) into six language form encodings with a dimension of (36, 768).

[0132] The language form encoding is input into the GPT-2 model, and the output of the GPT-2 model is a one-dimensional vector with a length of 50257. In this embodiment, the preset value is set to 100, that is, the first 100 values of the one-dimensional vector are connected to obtain the result encoding.

[0133] In this embodiment, the categories of the handshaking action include handshaking and not handshaking, K=1, and a total of two neurons are defined, which correspond to handshaking and not handshaking, respectively.

[0134] The result encoding and the two neurons are respectively multiplied to obtain the result encoding of 61 videos in 100 videos and the neuron of handshaking, which is multiplied by 1, that is, the video of the handshaking category, and the result encoding of 39 videos and the neuron of not handshaking, which is multiplied by 1, that is, the video of the not handshaking category.

[0135] Embodiment 3:

[0136] As shown in Figure 5 Fig. 1 is a schematic diagram of a human behavior recognition device based on image feature mapping according to an embodiment of the present application. The human behavior recognition device based on image feature mapping according to the embodiment includes one or more processors 51 and a memory 52. Among them, Figure 5 The processor 51 is taken as an example in this embodiment.

[0137] The processor 51 and the memory 52 can be connected through a bus or other means, Figure 5The bus connection is taken as an example.

[0138] The memory 52 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs and non-volatile computer executable programs, such as the human body behavior recognition method based on image feature mapping in Embodiment 1. The processor 51 executes the human body behavior recognition method based on image feature mapping by running the non-volatile software programs and instructions stored in the memory 52.

[0139] The memory 52 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 52 can optionally include a memory remotely arranged relative to the processor 51, and these remote memories can be connected to the processor 51 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0140] The program instructions / modules are stored in the memory 52, and when executed by the one or more processors 51, the human body behavior recognition method based on image feature mapping in Embodiment 1 described above is executed, for example, the above-described Figures 1-4 Each of the steps shown.

[0141] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for human behavior recognition based on image feature mapping, characterized in that, include: The input video is preprocessed to obtain video tiles; The video tiles are encoded using a convolutional neural network to obtain video encoding information; Extract E+1 consecutive video images from the input video in chronological order; the input dimension of the convolutional neural network is (a,n,m), and each video image is scaled according to the input dimension of the convolutional neural network, and the scaled video image is cut into n*m video blocks; The n*m ​​video blocks are encoded according to the input dimension of the convolutional neural network to obtain the three-dimensional encoding information of each video block; The three-dimensional encoded information of each video block is expanded after feature extraction to obtain the one-dimensional encoded information of each video block; Based on the one-dimensional encoding information of each video block and the sequence number of the video image corresponding to each video block, the video encoding information of each video block is obtained; The location encoding information is calculated based on the video encoding information; The complete encoding of the video tile is obtained by adding the video encoding information and the location encoding information; The complete encoding of the video patch is transformed through the input layer of the language model to obtain the linguistic encoding of the video patch; The language form encoding is input into the language model, the output layer of the language model outputs the result encoding, and the result encoding is mapped to the recognition result of the input video; In the encoding of the one-dimensional vector output by the language model, the first preset number of values ​​are taken as the result encoding; The K+1 neurons in the language model are each associated with a K+1 recognition category; the encoded result is multiplied by the parameters of the K+1 neurons in the output layer of the language model to obtain the multiplication result for each neuron; the largest multiplication result among the K+1 neurons is obtained, and the recognition category corresponding to the largest multiplication result is taken as the recognition result.

2. The human behavior recognition method based on image feature mapping according to claim 1, characterized in that, The step of unfolding the three-dimensional encoded information of each video patch after feature extraction to obtain the one-dimensional encoded information of each video patch specifically includes: The 3D encoding information of each video block is processed and encoded through multiple convolutional and pooling layers to obtain the first type of 3D encoding information of each video block; Add a maxpooling layer to the first type of 3D encoded information of each video patch and encode it to obtain the second type of 3D encoded information of each video patch. The dimensions of the second type of 3D encoded information are (b,c,d). The second type of three-dimensional encoded information is expanded into a vector and transformed into a one-dimensional vector of length b*c*d. The one-dimensional encoded information is then obtained by encoding the one-dimensional vector of length b*c*d.

3. The human behavior recognition method based on image feature mapping according to claim 2, characterized in that, The dimensions of the video encoding information are (E, b*c*d).

4. The human behavior recognition method based on image feature mapping according to claim 3, characterized in that, The step of calculating the location encoding information based on the video encoding information specifically includes: The location encoding information is obtained by calculating the video encoding information according to the following formula: When the sequence position number of the video tile in its respective video image is even: ； When the sequence position number of the video tile in its respective video image is odd: ； Among them, PE (pos,i) For positional encoding information, pos is the encoding information of the video image corresponding to each video tile, pos∈[0,1,2…E-1], and i is the sequence position of each video tile in its respective video image, i∈[0,1,2…b*c*d-1].

5. The human behavior recognition method based on image feature mapping according to claim 3, characterized in that, The step of performing feature transformation on the complete encoding of the video patch through the input layer of the language model to obtain the linguistic encoding of the video patch specifically includes: Based on the length f of the input vector of the language model, the video encoding information is divided into b*c*d / f one-dimensional vector encoding information, and the language form encoding is obtained according to the sequence number of the corresponding video image. The dimension of the language form encoding for each video tile is (E*b*c*d / f, f).

6. The human behavior recognition method based on image feature mapping according to claim 1, characterized in that, When training the language model, the parameters of the input layer and the parameters of the output layer are trained using a loss function, which is as follows: Loss=-(y*log(s)+(1-y)*log(1-y)); Where Loss is the loss value; y is the true value of the training sample, where y=1 when the training sample is correct and y=0 when the training sample is incorrect; and s is the probability that the language model predicts the input video to be of the corresponding recognition category.

7. A human behavior recognition device based on image feature mapping, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the human behavior recognition method based on image feature mapping as described in any one of claims 1-6.

Images