Video recognition method, device, electronic device and storage medium

Abstract

The present application discloses a video recognition method, apparatus, electronic device, and storage medium. The method includes: determining n first feature vectors corresponding to each of m first image frames of a first video; the first feature vector represents a spatial feature vector of the corresponding first image frame; extracting a second feature vector from the first feature vectors corresponding to the m first image frames, and processing the second feature vector through a fully connected layer to obtain a third feature vector; the second feature vector represents a temporal feature vector corresponding to the m first image frames; determining a first behavior type between a first object and a second object corresponding to the first video based on the third feature vector; each element in the third feature vector represents a probability of representing a behavior type; and determining a video recognition result for the first video based on the first behavior type and the type of the second object when the first behavior type is a set behavior type; wherein m and n are both positive integers.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a video recognition method, device, electronic device and storage medium. Background Art

[0002] With the increasing popularity of video ads, people-centered videos can showcase interactions between people and products, achieving better product display. Currently, identifying interactions between people and products in videos requires a massive amount of labeled samples, which is very expensive to obtain for model training. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a video recognition method, device, electronic device, and storage medium to at least solve the problem of high cost of obtaining model training samples in related technologies.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a method for video recognition, which includes:

[0006] Determining n first feature vectors corresponding to each of m first image frames of a first video, wherein the first feature vector represents a spatial feature vector of the corresponding first image frame; and image content of the first image frame includes a first object and a second object;

[0007] Extracting a second eigenvector from the first eigenvectors corresponding to the m first image frames, and processing the second eigenvector through a fully connected layer to obtain a third eigenvector; the second eigenvector represents the temporal eigenvectors corresponding to the m first image frames;

[0008] determining a first behavior type between a first object and a second object corresponding to the first video based on the third feature vector; wherein each element in the third feature vector corresponds to a probability of representing a behavior type;

[0009] In the case where the first behavior type is a set behavior type, a video recognition result of the first video is determined based on the first behavior type and the type of the second object; wherein,

[0010] Both m and n are positive integers.

[0011] In the above solution, determining n first feature vectors corresponding to each of the m first image frames of the first video includes:

[0012] Inputting each of the m first image frames into a first feature extraction model to obtain a first feature map of each first image frame output by the first feature extraction model;

[0013] Obtain n second feature maps corresponding to the first feature map of each first image frame by using a convolution kernel of a set size;

[0014] Feature extraction is performed on each of the n second feature maps corresponding to each first feature map to obtain n first feature vectors corresponding to each first image frame.

[0015] In the above solution, inputting each of the m first image frames into the first feature extraction model includes:

[0016] scaling each of the m first image frames of the first video according to a set ratio, and cropping the image according to a cropping frame of a set size to obtain m processed first image frames;

[0017] Each of the processed m first image frames is input into a first feature extraction model.

[0018] In the above solution, the step of extracting the second feature vector from the first feature vectors corresponding to the m first image frames includes:

[0019] The first feature vectors corresponding to the m first image frames are input into a second feature extraction model to obtain a second feature vector output by the second feature extraction model; the second feature extraction model is used to perform time series feature extraction on the input first feature vector to obtain a corresponding second feature vector.

[0020] In the above scheme, the second feature extraction model includes at least two hidden layer combinations connected in series; each hidden layer combination includes a first hidden layer and a second hidden layer connected in series; the first hidden layer is used to extract the spatial features of each first image frame based on the input feature vector; the second hidden layer is used to output the temporal features between m first image frames based on the spatial features of each input first image frame.

[0021] In the above solution, before inputting the first feature vectors corresponding to the m first image frames into the second feature extraction model, the method further includes:

[0022] When the behavior type of the sample is the set behavior type, deleting the type of the second object in the corresponding annotation to obtain a processed sample;

[0023] Based on the processed samples, a second feature extraction model is trained.

[0024] In the above solution, before determining the n first feature vectors corresponding to each of the m first image frames of the first video, the method further includes:

[0025] Inputting a plurality of second image frames of the second video into the recognition model to obtain an image recognition result output by the recognition model;

[0026] At least two second image frames whose corresponding image recognition results meet the set splicing conditions are spliced ​​together to obtain a first video; wherein,

[0027] The recognition model is used to recognize the first object in the input second image frame and output a corresponding image recognition result; the image recognition result represents the confidence level that the corresponding second image frame contains the first object.

[0028] In the above solution, the first object represents a set part of a person; and the second object represents an object.

[0029] In the above solution, the method further includes:

[0030] In a case where the first behavior type is not a set behavior type, a video recognition result of the first video is determined based on the first behavior type.

[0031] The embodiment of the present application further provides a video recognition device, comprising:

[0032] A first processing unit is configured to determine n first feature vectors corresponding to each of m first image frames of a first video, wherein the first feature vector represents a spatial feature vector of the corresponding first image frame, and image content of the first image frame includes a first object and a second object;

[0033] a second processing unit, configured to extract a second eigenvector from the first eigenvectors corresponding to the m first image frames, and process the second eigenvector through a fully connected layer to obtain a third eigenvector; the second eigenvector represents a temporal eigenvector corresponding to the m first image frames;

[0034] A classification unit, configured to determine a first behavior type between a first object and a second object corresponding to the first video based on the third feature vector; each element in the third feature vector corresponds to a probability of representing a behavior type;

[0035] The third processing unit is configured to determine a video recognition result of the first video based on the first behavior type and the type of the second object when the first behavior type is a set behavior type; wherein,

[0036] Both m and n are positive integers.

[0037] An embodiment of the present application further provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor,

[0038] Wherein, the processor is used to execute the steps of the above-mentioned video recognition method when running the computer program.

[0039] An embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video recognition method are implemented.

[0040] The embodiments of the present application provide a video recognition method, device, electronic device, and storage medium, which determine n first feature vectors corresponding to each of m first image frames of a first video; the first feature vector represents a spatial feature vector of the corresponding first image frame; the image content of the first image frame includes a first object and a second object; a second feature vector is extracted from the first feature vectors corresponding to the m first image frames, and the second feature vector is processed through a fully connected layer to obtain a third feature vector; the second feature vector represents a temporal feature vector corresponding to the m first image frames; based on the third feature vector, a first behavior type between the first object and the second object corresponding to the first video is determined; each element in the third feature vector represents a probability of a behavior type; when the first behavior type is a set behavior type, a video recognition result of the first video is determined based on the first behavior type and the type of the second object; wherein m and n are both positive integers. In the above scheme, the video recognition result is determined by detecting the behavior type and object type of the video respectively. In this way, it is not necessary to label the sample with a combination of the behavior type and the object type, which reduces the number of samples required for video recognition and reduces the cost of obtaining a video recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the video recognition method implementation process provided in the embodiment of the present application;

[0042] Figure 2 A schematic diagram of the video recognition method implementation process provided in the application embodiment of the present application;

[0043] Figure 3 A schematic diagram of the detection process provided in the application embodiment of the present application;

[0044] Figure 4 A schematic diagram of feature vector embedding provided in an application embodiment of this application;

[0045] Figure 5 A schematic diagram of the structure of a video recognition device provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] With the increasing popularity of video ads, they are gradually replacing print ads as the new mainstream form of product advertising. Human-centered video ads can showcase the interaction between people and products, achieving better product display results. In video ads, people and products frequently interact with each other. Identifying this interaction allows for precise product recommendations and improves the quality of video ads.

[0048] Currently, in order to identify the interactive behaviors between people and products in videos, for each behavior type, a combination with various product types must be prepared as model training samples. In other words, a large number of combinations of behavior types and product types are required as labeled samples, and the cost of obtaining model training samples is high.

[0049] Based on this, in various embodiments of the present application, n first feature vectors corresponding to each of the m first image frames of the first video are determined; the first feature vector represents the spatial feature vector of the corresponding first image frame; the image content of the first image frame includes a first object and a second object; a second feature vector is extracted from the first feature vectors corresponding to the m first image frames, and the second feature vector is processed through a fully connected layer to obtain a third feature vector; the second feature vector represents the temporal feature vector corresponding to the m first image frames; the first behavior type between the first object and the second object corresponding to the first video is determined based on the third feature vector; each element in the third feature vector represents the probability of a behavior type; when the first behavior type is a set behavior type, the video recognition result of the first video is determined based on the first behavior type and the type of the second object; wherein m and n are both positive integers. In the above scheme, the video recognition result is determined by detecting the behavior type and object type of the video respectively. In this way, it is not necessary to label the sample with a combination of the behavior type and the object type, which reduces the number of samples required for video recognition and reduces the cost of obtaining a video recognition model.

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] Figure 1 This is a schematic diagram of the implementation process of the video recognition method provided in the embodiment of the present application. The embodiment of the present application provides a video recognition method applied to electronic devices, where the electronic devices include but are not limited to servers, terminals and other electronic devices. It includes:

[0052] Step 101: Determine n first feature vectors corresponding to each of m first image frames of a first video.

[0053] The first feature vector represents a spatial feature vector of the corresponding first image frame; the image content of the first image frame includes a first object and a second object; m and n are both positive integers.

[0054] m first image frames are determined from the first video, and feature extraction is performed on each first image frame to obtain corresponding n first feature vectors. Here, when determining the corresponding first feature vectors based on the first image frames, different image frame feature extraction methods can be adopted, including but not limited to: performing spatial feature extraction on each first image frame to obtain a corresponding feature map, then processing the corresponding feature map with a convolution kernel of a set size, and obtaining the first feature vector based on the processed feature map; dividing the first image frame into a set number of image blocks, and performing feature extraction on the image blocks to obtain the first feature vector.

[0055] In one embodiment, the first object represents a set part of a person; and the second object represents an article.

[0056] The image content of each first image frame includes a set part of a person and an object. The set part of the person may be a face, hand, limb, and / or torso, etc. The first behavior type between the first object and the second object may be an behavior between the set part of the person and the object, such as an interactive behavior between the set part of the person and the object. The object included in the image content of the image frame may be a commodity.

[0057] Step 102: extracting a second eigenvector from the first eigenvectors corresponding to the m first image frames, and processing the second eigenvector through a fully connected layer to obtain a third eigenvector.

[0058] The second feature vector represents a temporal feature vector corresponding to the m first image frames; and each element in the third feature vector represents a probability of a behavior type.

[0059] Feature extraction is performed on the first feature vectors corresponding to the m first image frames to obtain a second feature vector. The second feature vector is processed by a predetermined fully connected layer, and the elements of the output feature vector are processed by a softmax function to obtain a third feature vector of a predetermined dimension. Each element in the third feature vector corresponds to the probability that the behavior in the first video is a certain behavior type.

[0060] Step 103: Determine a first behavior type between a first object and a second object corresponding to the first video based on the third feature vector.

[0061] Based on the elements of the third eigenvector, determining that the behavior type corresponding to at least one element is a first behavior type between the first object and the second object corresponding to the first video. Here, determining the first behavior type includes, but is not limited to: selecting the behavior type corresponding to one or more largest elements of the third eigenvector; and selecting the behavior type corresponding to one or more elements of the third eigenvector whose number of elements is greater than a set threshold.

[0062] Step 104: When the first behavior type is a set behavior type, determine a video recognition result of the first video based on the first behavior type and the type of the second object.

[0063] At least one behavior type is set as a predetermined behavior type, and a determination is made as to whether the first behavior type is the predetermined behavior type. Based on the determination, a determination is made as to how to determine the video recognition result for the first video. If the first behavior type is the predetermined behavior type, a video recognition result for the first video is determined based on the first behavior type and the type of the second object identified in the first video.

[0064] Here, a Yolov5-based detection network can be used to identify the type of the second object in at least one first image frame of the first video, identifying the type of the second object corresponding to each first image frame, and weighting the type recognition results for each of the at least one first image frame to determine the type of the second object in the first video. The detection network can be configured as needed. If the second object represents an object, object type recognition can include identifying the category name to which the object belongs.

[0065] In practical applications, let's take an e-commerce scenario as an example. In this e-commerce scenario, the item represented by the second object is a commodity. Here, some or all of the behavior types are determined to be behavior types with commodities (the second object) as the main subject. These behavior types are the set behavior types; the remaining behavior types are behavior types with non-commodities (the first object) as the main subject. Typically, the behavior types with non-commodities as the main subject are people. If the determined first behavior type is the set behavior type, the video recognition result of the first video is determined based on the first behavior type and the identified commodity type of the second object.

[0066] In an embodiment of the present application, the video recognition result is determined by separately detecting the behavior type and object type of the video. In this way, there is no need to label samples with a combination of behavior type and object type, which reduces the number of samples required for video recognition and reduces the cost of obtaining a video recognition model.

[0067] At the same time, different strategies for determining video recognition results are implemented based on whether the first behavior type is a set behavior type. By determining whether it is a set behavior type, the type of the second object in the first video is identified through the detection network for the set behavior type. In this way, based on the type of behavior between the first object and the second object in the first video and the type of the second object, the type of interaction (behavior, object) in the video is determined, thereby more accurately determining the video recognition result.

[0068] In one embodiment, the method further includes:

[0069] In a case where the first behavior type is not a set behavior type, a video recognition result of the first video is determined based on the first behavior type.

[0070] In a case where the first behavior type is not the set behavior type, the first behavior type is used as the video recognition result.

[0071] As mentioned above, using the e-commerce scenario as an example, some or all behavior types are determined to be behavior types centered around goods (second objects), and these behavior types are the set behavior types; the remaining behavior types are behavior types centered around non-goods (first objects). Here, if the determined first behavior type is not a set behavior type, that is, a behavior type centered around non-goods, the video recognition result for the first video is determined based on the first behavior type.

[0072] A multi-strategy approach is set up, and different strategies for determining video recognition results are executed by using the set behavior type as a branch judgment condition. In actual applications, by setting the set behavior type, for the behavior type with the second object as the main body, the object type is identified by detecting the network, and the video recognition result is accurately determined based on the behavior type and object type between the first object and the second object in the first video; for the behavior type with the first object as the main body, the behavior type in the first video is used as the video recognition result. In this way, the behavior type is used to determine whether to use the object type as the video recognition result. For the behavior type with the second object (item) as the main body, these behavior types usually have a high degree of interaction with the second object, such as cutting an apple, lifting a cup, etc., the video recognition result is further determined in combination with the object type, thereby improving the recognition accuracy of the video recognition result.

[0073] Preferably, in e-commerce scenarios, behavior types centered around commodities usually have a high degree of interaction with commodities. The video recognition results are determined in combination with commodity types, thereby improving the recognition accuracy of video advertisement recognition results. Based on the recognition results, accurate recommendations can be made, thereby improving the quality of video advertisements.

[0074] In one embodiment, determining n first feature vectors corresponding to each of the m first image frames of the first video includes:

[0075] Inputting each of the m first image frames into a first feature extraction model to obtain a first feature map of each first image frame output by the first feature extraction model;

[0076] Obtain n second feature maps corresponding to the first feature map of each first image frame by using a convolution kernel of a set size;

[0077] Feature extraction is performed on each of the n second feature maps corresponding to each first feature map to obtain n first feature vectors corresponding to each first image frame.

[0078] Feature extraction is performed on each of the m first image frames through the first feature extraction model to obtain a first feature map corresponding to each first image frame, and then convolution is performed through a convolution kernel of a set size (for example, a 1*1 convolution kernel) to compress the channel features to obtain n second feature maps. Based on each second feature map in the n second feature maps, a corresponding first feature vector is obtained, thereby obtaining n first feature vectors corresponding to each first image frame.

[0079] Here, the first feature extraction model can be a set ResNet model. Preferably, the first feature extraction model is a ResNet50 pre-trained on ImageNet.

[0080] In an embodiment of the present application, for the first feature map extracted from each first image frame feature, a second feature map is obtained by compressing the channel features through a convolution kernel of a set size, and a first feature vector is obtained based on the second feature map. In this way, the feature information of the image is extracted without separating the spatial features of the image itself, and the first feature vector is used as the input of the recognition network, which can improve the accuracy of the network's behavior type recognition.

[0081] In one embodiment, inputting each of the m first image frames into the first feature extraction model includes:

[0082] scaling each of the m first image frames of the first video according to a set ratio, and cropping the image according to a cropping frame of a set size to obtain m processed first image frames;

[0083] Each of the processed m first image frames is input into a first feature extraction model.

[0084] Because videos come from a wide variety of sources and have varying aspect ratios, resolutions, and other specifications, m first image frames of a first video can be processed, each of which has a set size. Here, the first image frames are first scaled according to a set ratio, then the scaled images are cropped using a cropping frame of the set size, and the cropped images are used as input to the first feature extraction model.

[0085] In practical applications, when determining the set ratio, bilinear / bicubic sampling can be used to randomly determine any value in (256, 320) as the length of the short side of the scaled image, and the determined scaling ratio is used as the set ratio.

[0086] After scaling the video image to a set ratio, the first image frame is cropped within a cropping frame of a set size, ensuring that the size of the cropped first image frame is within the optimal range for the first feature extraction model. This optimal range is determined by the sample size used to train the first feature extraction model. This allows the extracted feature vector to better represent the image content, thereby improving the accuracy of the network's behavior type recognition.

[0087] In one embodiment, extracting the second feature vector from the first feature vectors corresponding to the m first image frames includes:

[0088] The first feature vectors corresponding to the m first image frames are input into a second feature extraction model to obtain a second feature vector output by the second feature extraction model; the second feature extraction model is used to perform time series feature extraction on the input first feature vector to obtain a corresponding second feature vector.

[0089] Here, the second feature vector can be extracted from the first feature vectors corresponding to the m first image frames using a second feature extraction model.

[0090] The second feature extraction model can be a predefined Transformer model. The predefined Transformer model uses an attention mechanism. Compared to the related art Long Short-Term Memory (LSTM) model, which suffers from gradient vanishing issues when processing long-distance sequences, the attention-based Transformer model is more closely associated with the input feature vector and performs better on long-distance sequences. The second feature vector extracted based on the predefined Transformer model can improve the accuracy of determining the first behavior type.

[0091] In one embodiment, the second feature extraction model includes at least two hidden layer combinations connected in series; each hidden layer combination includes a first hidden layer and a second hidden layer connected in series; the first hidden layer is used to extract the spatial features of each first image frame based on the input feature vector; the second hidden layer is used to output the temporal features between m first image frames based on the spatial features of each first image frame input.

[0092] Here, the second feature extraction model includes at least two hidden layer combinations connected in series, and each hidden layer combination includes a first hidden layer and a second hidden layer connected in series: the first hidden layer is used to extract the spatial features of each first image frame based on the n feature vectors corresponding to each first image frame of the input; the second hidden layer is used to output the temporal features between m first image frames based on the spatial features of each first image frame of the input.

[0093] After the first feature vectors corresponding to the m first image frames are input into the first hidden layer of the first hidden layer combination of the second feature extraction model, the n first feature vectors corresponding to each first image frame are first processed by the first hidden layer of the first hidden layer combination to extract the spatial features of each first image frame; then the spatial features of each first image frame in the m first image frames are processed by the second hidden layer of the first hidden layer combination to extract the temporal features between the m first image frames, and the determined feature vectors are input into the first hidden layer of the next hidden layer combination (the second hidden layer combination). The above process is repeated until the set termination condition is met and the temporal features between the m first image frames are output.

[0094] Compared with single hidden layer processing, each hidden layer combination of the second feature extraction model is set to two layers, which are used to extract the spatial features of each image frame and the temporal features between m image frames respectively. In this way, spatial features and temporal features are extracted respectively through two hidden layers, thereby realizing the separation of spatial features and temporal features in feature extraction. The hidden layer requires fewer parameters and the required training cost is lower.

[0095] In one embodiment, before inputting the first feature vectors corresponding to the m first image frames into the second feature extraction model, the method further includes:

[0096] When the behavior type of the sample is the set behavior type, deleting the type of the second object in the corresponding annotation to obtain a processed sample;

[0097] Based on the processed samples, a second feature extraction model is trained.

[0098] Before using the second feature extraction model, the second feature extraction model is trained. By preprocessing data samples from the Kinectics700 dataset or other datasets, the behavior type labels of the data based on the first object remain unchanged, while the behavior type labels of the data based on the second object are removed from the specific object type names, leaving only the action verbs as the data labels.

[0099] Here, when preprocessing the data samples of the data set, the same video processing method as that used by the second feature extraction model is adopted.

[0100] Based on the processed sample training model, the model output result obtained by training can be used to determine whether the subject corresponding to the behavior type is the second object. In this way, the model output result can be used as a condition for whether to further combine the object type of the identified second object. When the model output result represents that the corresponding first behavior type is the set behavior type, the video recognition result of the first video is determined in combination with the first behavior type and the object type of the second object.

[0101] In practical applications, still using the e-commerce scenario as an example, when processing the annotation of data in a dataset, it is necessary to determine whether the subject corresponding to the behavior type is a commodity (the second object). Here, the subject of the behavior type can be used as the basis for classifying whether the subject corresponding to the behavior type is a commodity. In other words, the judgment is based on whether the object of the behavior type (verb) is a commodity. For example, if the sample is labeled "blowing hair", and hair is a non-commodity, then the behavior type annotation of the data remains unchanged and remains "blowing hair". For another example, if the sample is labeled "cutting apples", and apples are a commodity, then the specific commodity name "apple" is removed from the behavior type annotation of the data, and only the verb "cut" of the behavior type is retained as the data annotation. Then, if the first behavior type determined by the second feature vector output by the second feature extraction model is a single verb (for example, "cut"), then the corresponding first video is based on the commodity. If the first behavior type determined by the second feature vector output by the second feature extraction model is a verb + object (for example, "blowing hair"), then the corresponding first video is based on the non-commodity. In this way, the video recognition result of the first video can be further determined based on the identified behavior type.

[0102] In one embodiment, before determining the n first feature vectors corresponding to each of the m first image frames of the first video, the method further includes:

[0103] Inputting a plurality of second image frames of the second video into the recognition model to obtain an image recognition result output by the recognition model;

[0104] At least two second image frames whose corresponding image recognition results meet the set splicing conditions are spliced ​​together to obtain a first video; wherein,

[0105] The recognition model is used to recognize the first object in the input second image frame and output a corresponding image recognition result; the image recognition result represents the confidence level that the corresponding second image frame contains the first object.

[0106] Videos come from a wide range of sources. For example, videos obtained through live screen recording may not include people due to the host's movement, and therefore do not include the set parts of the person (the first object). If these image frames of video clips that do not include the first object are used for recognition, the accuracy of video recognition will be affected.

[0107] In an embodiment of the present application, a recognition model is used to identify a first object in image frames of a video, and at least two second image frames whose corresponding recognition results meet a set splicing condition are sorted in chronological order, and the at least two second image frames are spliced ​​in sequence to obtain a first video. The set splicing condition is set according to the type of image output result, and may include but is not limited to: if the image recognition result is a binary classification result, determining that the second image frame contains the first object; if the image recognition result is a confidence level, determining that the confidence level of the second image frame containing the first object is greater than a set threshold.

[0108] Here, the recognition model can be a set MTCNN model. Preferably, the recognition model is trained by a large-scale dataset of set parts of a person, such as a Winderface dataset.

[0109] In this way, the first video is obtained by screening at least two second image frames in the second video that meet the set splicing condition, ensuring that each first image frame in the first video includes the first object, thereby improving the accuracy of video recognition.

[0110] The present application will be described in further detail below in conjunction with application examples.

[0111] Recognizing interactions between people and products in e-commerce videos presents the following challenges:

[0112] 1) In e-commerce scenarios, there are many types of goods displayed through videos. Manual annotation is used to establish a behavior recognition dataset for e-commerce scenarios. Due to the large number of product types, the labeling cost of samples is huge. Existing behavior recognition datasets are classified by action. Even if different objects interact with people, they will be classified as the same category.

[0113] 2) The video content in e-commerce scenarios is complex, and there are video clips such as brand promotion and product special effects that are not related to interactive recognition. When the video is framed at equal intervals, the sampling of the obtained video clips will affect the accuracy of product recognition.

[0114] 3) When performing video recognition, each image frame needs to be divided into multiple patches. Each patch represents a small image region obtained by segmenting the image. Each patch corresponds to a feature vector. Performing video recognition based on such feature vectors will disrupt the spatial characteristics of the image itself. Patches refer to small image regions obtained by segmenting the image. For example, a 256*256 image can be divided into 16 16*16 patches.

[0115] Based on this, this application embodiment proposes a video recognition solution based on spatiotemporal features, which improves the accuracy of video recognition and the richness of interactive recognition types by segmenting video clips and detecting product types through face and / or body recognition.

[0116] 1) To address the problem of a large number of product categories in e-commerce scenarios and a relatively small number of categories in behavior recognition datasets, a behavior recognition + product detection solution is adopted to improve the richness of interaction (behavior, object) types in video recognition. In addition, the sample annotations in the behavior recognition dataset are preprocessed: for samples with products as the main subject (i.e., the behavior recipient is a product), the nouns in the sample annotations are removed and the behavior verbs are retained; for samples with non-products as the main subject (i.e., the behavior recipient is not a product, such as human hair), the sample annotations are not processed. In this way, during recognition, videos with products as the main subject return behavior verbs, which are further combined with the nouns in the product type recognition results to obtain video recognition results (e.g., "cutting cake", "cutting fruit"); while videos with non-products as the main subject return behavior verbs as video recognition results (e.g., "blowing hair").

[0117] 2) To address the complex content of videos in e-commerce scenarios, face and / or body recognition is used on videos to extract segments containing faces and / or bodies, and the extracted video segments are frame-sampled for interactive recognition.

[0118] 3) To address the problem that dividing image frames into multiple patches will destroy spatial features, a convolutional neural network is used to extract features, and the multi-channel features (feature maps) are vectorized as input to the behavior type recognition network.

[0119] Figure 2 The following is a schematic diagram of a video recognition method implementation process provided by an application embodiment of the present application, which at least includes:

[0120] 1) Face / body detection.

[0121] The purpose of face and / or body detection is to obtain video clips with people involved. These video clips are highly relevant to the video recognition task, while irrelevant video clips will affect the accuracy of video recognition detection. Here, face recognition is used as an example. In actual applications, it can be face recognition, body recognition, or a combination of face recognition and body recognition. The specific process is as follows:

[0122] First, the input video is a product video for a video ad, including but not limited to the main product video and recommended videos. There are no restrictions on the size or frame rate of the input video. The input video's aspect ratio is r, and the input video's duration t is no more than 2 minutes. If the video is longer than 2 minutes, it is split into several segments of no more than 2 minutes, and each segment is tested.

[0123] Second, to speed up face detection, the input video is frame-decimated and scaled. In practice, a sampling frequency of 2 frames per second is used to obtain 2*t image frames for face detection. Each image frame is then scaled to a standard 224*224r image frame using bilinear / bicubic sampling.

[0124] Third, face recognition uses the MTCNN network, which is trained with a large human face dataset (such as the Winderface dataset) and has the ability to recognize faces. Figure 3 As shown, each image frame extracts features through the network to obtain a feature vector, which is compressed in the output layer to obtain a binary classification result, indicating whether the image frame contains a face or not.

[0125] Fourth, video images that do not contain faces are discarded. For segments that do contain faces, they are spliced ​​in chronological order, resulting in frame t1 as input for the video preprocessing stage. Note that if all frames in the video are detected as not containing faces, the video is directly returned as having no action, i.e., it is returned as empty, and the next video preprocessing stage is not executed.

[0126] 2) Video preprocessing.

[0127] In the video preprocessing stage, the video is standardized and the feature vector of the spatial features of the image frame is extracted. The obtained standardized video feature vector is used as the input of the temporal classification model.

[0128] This step mainly includes three parts: video frame extraction, random scaling and cropping by frame, and feature vector embedding.

[0129] Video frame extraction: The t1 frame video obtained in the human body / face detection stage is sampled at a sampling frequency of t1 / 16, so that 16 frames of image frames containing faces and / or human bodies are obtained.

[0130] Random Frame Scaling and Cropping: Using bilinear / bicubic sampling, the short sides of 16 image frames are randomly scaled to any value between (256, 320). The long sides are scaled to the corresponding values ​​between (256*r, 320*r). After scaling, the images are cropped within a set cropping frame (256×256).

[0131] Feature vector embedding: First, image spatial features are extracted using ResNet50 pre-trained on ImageNet. Then, 1*1 convolution is used to compress channel features (feature maps) to 256. Finally, each feature map is vectorized to obtain a 1*512-dimensional feature vector. Figure 4 The feature vector embedding diagram shown in FIG. 3 ultimately yields 256 1*512-dimensional feature vectors corresponding to each image frame. These 1*512-dimensional feature vectors correspond to the first feature vectors in each embodiment and serve as input to the temporal behavior classification stage.

[0132] 3) Temporal behavior classification.

[0133] The temporal behavior classification stage takes the image frame vector of the preprocessed video as input, extracts the temporal features between the image frames and performs behavior type classification.

[0134] First, the Kinectics700 dataset is preprocessed. The behavior type labels of data that are not based on products remain unchanged, while the behavior type labels of data based on products remove the specific product names and only retain the behavior verbs as data labels.

[0135] Second, the processed Kinectics700 dataset is used to train the Transformer model to extract temporal features and perform behavior type recognition. The video preprocessing method in the dataset is the same as the video preprocessing in the video preprocessing step, and multi-classification cross entropy loss is used as the loss function for training.

[0136] The 256 1*512 feature vectors corresponding to each image frame obtained in the video preprocessing step are used as input, and the trained Transformer model is used to extract the temporal features. The obtained temporal feature vector (i.e., the second feature vector in each embodiment) is processed by the fully connected layer to output a 1*700 dimensional vector (i.e., the third feature vector in each embodiment). The classification probability of different behavior types is obtained through softmax, and the behavior type corresponding to the maximum probability is taken as the output of the temporal behavior classification stage.

[0137] Among them, for data sets with existing numbers, such as the Kinectics700 data set, some numbers are marked as behavior types with commodities as the main body during annotation. In this way, when using the trained Transformer model to extract time series features, based on the model output results, the corresponding number of the behavior type in the data set can be determined, and based on the corresponding annotation information, it can be determined whether the behavior type is mainly based on commodities.

[0138] Here, the Transformer model extracts both spatial features from the 256 feature vectors corresponding to each of the 16 image frames and temporal features between the 16 frames. In this application embodiment, two hidden layers are used to extract these features separately, achieving spatial and temporal separation. This reduces the number of model parameters and improves the model's feature extraction performance.

[0139] 4) Product testing.

[0140] The goal of the product detection phase is to obtain product category names in preparation for the multi-strategy behavior recognition phase. This phase detects products based on 16 randomly scaled image frames. A Yolov5-based detection network is used to detect product categories frame by frame and return the product type. Finally, the product category recognition results for the 16 frames are weighted to determine the video's product category, which serves as the final output of the product detection phase.

[0141] 5) Multi-strategy behavior recognition.

[0142] The multi-strategy behavior recognition stage outputs the recognition and detection results of the video. This stage uses the category names of the product types obtained in the product detection stage and the behavior type classification results in the temporal behavior classification stage to execute different output strategies according to the behavior type.

[0143] If the behavior type detection result is based on non-products, the behavior type classification result is directly returned as the video recognition result, and the process ends.

[0144] If the result of the behavior type detection is based on products, the behavior type classification result is used as a verb, and the product type classification result of the product detection phase is used as a noun as the output. That is, the verb + noun form is returned as the result of video interaction recognition, and the process ends.

[0145] In this application embodiment, for video recognition in e-commerce scenarios, video clips are segmented based on face and / or body recognition, and video clips irrelevant to video recognition are removed. Frame extraction is performed on the processed video to extract spatial features. A pre-defined Transformer model is used to extract temporal features, which are then combined with the product type recognition results to return the corresponding video recognition results. At least the following technical means are employed to achieve the corresponding effects:

[0146] 1) Using face / body detection to extract video clips as input to the behavior classification network can filter out video clips that are not related to human-product interaction and improve the accuracy of subsequent behavior classification network recognition.

[0147] 2) By combining the product detection strategy (product type recognition) with the behavior recognition strategy (behavior type recognition), behavior type detection and product type detection are separated. The model is trained using the corresponding labeled samples, and then combined according to the set strategy after detection. In this way, the richness of video recognition types in e-commerce scenarios can be improved without relying on a large number of labeled samples.

[0148] 3) The combination of spatial features and temporal features: extracting spatial features through a convolutional network and embedding image frame feature vectors, and then using the Transformer model to extract temporal features, can achieve time and space separation. In this way, the embedded feature vector is obtained without separating the spatial features of the image, thereby improving the network's behavior recognition detection accuracy.

[0149] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a video recognition device, such as Figure 5 As shown, the device includes:

[0150] A first processing unit 501 is configured to determine n first feature vectors corresponding to each of m first image frames of a first video, wherein the first feature vector represents a spatial feature vector of the corresponding first image frame; and image content of the first image frame includes a first object and a second object.

[0151] A second processing unit 502 is configured to extract a second feature vector from the first feature vectors corresponding to the m first image frames, and process the second feature vector through a fully connected layer to obtain a third feature vector; the second feature vector represents the temporal feature vectors corresponding to the m first image frames;

[0152] A classification unit 503 is configured to determine a first behavior type between a first object and a second object corresponding to the first video based on the third feature vector; each element in the third feature vector corresponds to a probability of representing a behavior type;

[0153] The third processing unit 504 is configured to determine a video recognition result of the first video based on the first behavior type and the type of the second object when the first behavior type is a set behavior type; wherein,

[0154] Both m and n are positive integers.

[0155] In one embodiment, the first processing unit 501 is configured to:

[0156] Inputting each of the m first image frames into a first feature extraction model to obtain a first feature map of each first image frame output by the first feature extraction model;

[0157] Obtain n second feature maps corresponding to the first feature map of each first image frame by using a convolution kernel of a set size;

[0158] Feature extraction is performed on each of the n second feature maps corresponding to each first feature map to obtain n first feature vectors corresponding to each first image frame.

[0159] In one embodiment, the first processing unit 501 is configured to:

[0160] scaling each of the m first image frames of the first video according to a set ratio, and cropping the image according to a cropping frame of a set size to obtain m processed first image frames;

[0161] Each of the processed m first image frames is input into a first feature extraction model.

[0162] In one embodiment, the second processing unit 502 is configured to:

[0163] The first feature vectors corresponding to the m first image frames are input into a second feature extraction model to obtain a second feature vector output by the second feature extraction model; the second feature extraction model is used to perform time series feature extraction on the input first feature vector to obtain a corresponding second feature vector.

[0164] In one embodiment, the second feature extraction model includes at least two hidden layer combinations connected in series; each hidden layer combination includes a first hidden layer and a second hidden layer connected in series; the first hidden layer is used to extract the spatial features of each first image frame based on the input feature vector; the second hidden layer is used to output the temporal features between m first image frames based on the spatial features of each first image frame input.

[0165] In one embodiment, the apparatus further comprises:

[0166] The training unit is used to delete the type of the second object in the corresponding annotation before the second processing unit 502 inputs the first feature vectors corresponding to the m first image frames into the second feature extraction, when the behavior type of the sample is the set behavior type, to obtain a processed sample; and train the second feature extraction model based on the processed sample.

[0167] In one embodiment, the apparatus further comprises:

[0168] The recognition unit is used to input multiple second image frames of the second video into the recognition model before the first processing unit 501 determines the n first feature vectors corresponding to each first image frame in the m first image frames of the first video, so as to obtain the image recognition result output by the recognition model; stitching at least two second image frames whose corresponding image recognition results meet the set splicing conditions to obtain the first video; wherein the recognition model is used to recognize the first object in the input second image frame and output the corresponding image recognition result; the image recognition result represents the confidence level that the first object is contained in the corresponding second image frame.

[0169] In one embodiment, the first object represents a set part of a person; and the second object represents an article.

[0170] In one embodiment, the apparatus further comprises:

[0171] The fourth processing unit is configured to determine a video recognition result of the first video based on the first behavior type when the first behavior type is not a set behavior type.

[0172] In actual application, the first processing unit 501, the second processing unit 502, the classification unit 503, the third processing unit 504, the training unit, the recognition unit, and the fourth processing unit can be implemented by a processor in a video recognition device, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU), or a programmable gate array (FPGA).

[0173] It should be noted that the video recognition device provided in the above embodiment only uses the division of the above program modules as an example to illustrate video recognition. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the above-described processing. In addition, the video recognition device provided in the above embodiment and the video recognition method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0174] Based on the hardware implementation of the above program modules, and in order to implement the video recognition method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Figure 6 This is a schematic diagram of the hardware structure of the electronic device according to the embodiment of the present application. Figure 6 As shown, the electronic equipment includes:

[0175] Communication interface 1, capable of exchanging information with other devices such as network devices;

[0176] The processor 2 is connected to the communication interface 1 to implement information exchange with other devices and is used to execute the method provided by one or more of the above technical solutions when running a computer program. The computer program is stored in the memory 3.

[0177] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 4.

[0178] The memory 3 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0179] It is understood that the memory 3 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 2 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0180] The method disclosed in the above embodiment of the present invention can be applied to processor 2 or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the hardware integrated logic circuit in processor 2 or by instructions in software form. The above processor 2 can be a general-purpose processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. Processor 2 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the above method in combination with its hardware.

[0181] When the processor 2 executes the program, the corresponding processes in the various methods of the embodiments of the present invention are implemented, which will not be described here for the sake of brevity.

[0182] In an exemplary embodiment, the present invention further provides a storage medium, namely, a computer storage medium, specifically, a computer-readable storage medium, such as a memory 3 storing a computer program. The computer program can be executed by a processor 2 to perform the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface mount storage, optical disk, or CD-ROM.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0186] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, ROM, RAM, disks or optical disks, etc. Various media that can store program codes.

[0187] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0188] It is understandable that in the embodiments of the present application, when user information is involved, such as facial information of image content and other related data, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0189] It should be noted that the technical solutions described in the embodiments of this application can be arbitrarily combined without conflict. Unless otherwise specified or limited, the term "connection" should be understood in a broad sense. For example, it can be an electrical connection or internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0190] Additionally, in the examples of this application, "first," "second," etc., are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. It should be understood that the objects distinguished by "first," "second," and "third" can be interchanged where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0191] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0192] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0193] The various specific technical features in the various embodiments described in the specific implementation methods can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this application will not be described separately.

Claims

1. A video recognition method, characterized in that: The method comprises: Determining n first feature vectors corresponding to each of m first image frames of a first video; the first feature vector represents a spatial feature vector of the corresponding first image frame; image content of the first image frame includes a first object and a second object; the first object represents a set part of a person; and the second object represents an object; Extracting a second eigenvector from the first eigenvectors corresponding to the m first image frames, and processing the second eigenvector through a fully connected layer to obtain a third eigenvector; the second eigenvector represents the temporal eigenvectors corresponding to the m first image frames; determining a first behavior type between a first object and a second object corresponding to the first video based on the third feature vector; wherein each element in the third feature vector corresponds to a probability of representing a behavior type; In the case where the first behavior type is a set behavior type, a video recognition result of the first video is determined based on the first behavior type and the type of the second object; wherein, Both m and n are positive integers.

2. The method according to claim 1, characterized in that The determining of n first feature vectors corresponding to each of the m first image frames of the first video includes: Inputting each of the m first image frames into a first feature extraction model to obtain a first feature map of each first image frame output by the first feature extraction model; Obtain n second feature maps corresponding to the first feature map of each first image frame by using a convolution kernel of a set size; Feature extraction is performed on each of the n second feature maps corresponding to each first feature map to obtain n first feature vectors corresponding to each first image frame.

3. The method according to claim 2, characterized in that Inputting each of the m first image frames into the first feature extraction model includes: scaling each of the m first image frames of the first video according to a set ratio, and cropping the image according to a cropping frame of a set size to obtain m processed first image frames; Each of the processed m first image frames is input into a first feature extraction model.

4. The method according to claim 1, wherein The extracting the second feature vector from the first feature vectors corresponding to the m first image frames includes: The first feature vectors corresponding to the m first image frames are input into a second feature extraction model to obtain a second feature vector output by the second feature extraction model; the second feature extraction model is used to perform time series feature extraction on the input first feature vector to obtain a corresponding second feature vector.

5. The method according to claim 4, characterized in that The second feature extraction model includes at least two hidden layer combinations connected in series; each hidden layer combination includes a first hidden layer and a second hidden layer connected in series; the first hidden layer is used to extract spatial features of each first image frame based on an input feature vector; The second hidden layer is used to output temporal features between m first image frames based on the spatial features of each input first image frame.

6. The method according to claim 4, characterized in that Before inputting the first feature vectors corresponding to the m first image frames into the second feature extraction model, the method further includes: When the behavior type of the sample is the set behavior type, deleting the type of the second object in the corresponding annotation to obtain a processed sample; Based on the processed samples, a second feature extraction model is trained.

7. The method according to claim 1, characterized in that Before determining the n first feature vectors corresponding to each of the m first image frames of the first video, the method further includes: Inputting a plurality of second image frames of the second video into the recognition model to obtain an image recognition result output by the recognition model; At least two second image frames whose corresponding image recognition results meet the set splicing conditions are spliced ​​together to obtain a first video; wherein, The recognition model is used to recognize the first object in the input second image frame and output a corresponding image recognition result; the image recognition result represents the confidence level that the corresponding second image frame contains the first object.

8. The method according to claim 1, characterized in that The first object represents a set part of a person; the second object represents an object.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: In a case where the first behavior type is not a set behavior type, a video recognition result of the first video is determined based on the first behavior type.

10. A video recognition device, characterized in that: include: A first processing unit is configured to determine n first feature vectors corresponding to each of the m first image frames of the first video; The first feature vector represents a spatial feature vector of a corresponding first image frame; the image content of the first image frame includes a first object and a second object; the first object represents a set part of a person; and the second object represents an object; a second processing unit, configured to extract a second eigenvector from the first eigenvectors corresponding to the m first image frames, and process the second eigenvector through a fully connected layer to obtain a third eigenvector; the second eigenvector represents a temporal eigenvector corresponding to the m first image frames; A classification unit, configured to determine a first behavior type between a first object and a second object corresponding to the first video based on the third feature vector; each element in the third feature vector corresponds to a probability of representing a behavior type; The third processing unit is configured to determine a video recognition result of the first video based on the first behavior type and the type of the second object when the first behavior type is a set behavior type; wherein, Both m and n are positive integers.

11. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method according to any one of claims 1 to 9.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images