Small sample video classification and classification model training method and device
By using the compressed domain length and short time Cross-Transformer model and lateral connection interaction technology in small sample video classification, important information of video frames is extracted, and the problem of insufficient utilization of timing information is solved, and high-precision and efficient small sample video classification is achieved. .
Patent Information
- Application Number
- CN202210557333.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-05-20
AI Technical Summary
The existing small sample video classification method only processes RGB images of video frames, and does not utilize the timing information sufficiently, resulting in low classification accuracy and low computing efficiency, and it is impossible to effectively utilize uneven information between video frames.
Using the Cross-Transformer model for compressed domain length and short time, the predicted frame importance value and image group importance value are obtained, combined with RGB images and motion vectors, the compressed domain information is obtained, and the I branch and MV branch are shortened by the side-connected I branch and MV branch. time interaction, extract short-term fused frame features and long-term frame features, query the global matching of video and support video, and output classification scores.
High-precision classification of videos under a small number of examples is achieved, which improves computing efficiency, and reduces the additional collection and labeling of video data, and expands the scope of application.
Smart Images

Figure CN115019087B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a small sample video classification and classification model training method and device thereof. Background Art
[0002] As an important research field in computer vision, small sample video classification methods classify videos of unknown categories into one of the example categories given a small number of examples in each category. The categories for which examples are given are usually called support categories, the given examples are called support videos, and the videos to be classified are called query videos. Small sample video classification technology can significantly reduce the workload of video collection and annotation, and can also be quickly deployed to new category classification problems without re-training on a large scale. In order to correctly classify new videos based on extremely limited examples, small sample video classification usually inputs one task at a time, which consists of a support video and several query videos; and outputs the support categories to which all query videos in the task belong. Episodic training is usually used in small sample video classification to estimate model parameters. This training technology is based on a large-scale video classification dataset and divides the dataset into three subsets by category. Different subsets contain completely different categories. Several tasks are sampled from each subset to form a meta-training set, a meta-validation set, and a meta-test set. This training method ensures that the categories of training tasks and test tasks are completely non-overlapping, so the generalization ability of small sample classification methods on new categories can be measured. In terms of the model, small sample video classification methods classify query videos according to supporting videos. Existing methods can be incorporated into the matching framework, that is, first extract the features of the supporting videos and query videos, and then match the features of the query videos and supporting videos. The differences between the methods mainly lie in the differences in feature extraction methods and matching methods.
[0003] Conventional video classification methods require thousands of examples for each category, and use these examples to train the classification model and obtain model parameters, and then use the classification model to determine the category to which the sample to be classified belongs. When there are too few examples, overfitting will occur during classification model training, causing the classification accuracy of the classification model to drop significantly on videos other than the examples. Since there are only a few examples per category in small sample video classification, conventional video classification methods cannot be directly used for small sample video classification. Therefore, small sample video classification methods are quite different from conventional video classification methods in terms of input and output, training methods, and model design.
[0004] In the prior art, existing small sample video classification methods only process the RGB images of video frames, and do not make full use of temporal information. On the one hand, the existing methods uniformly extract video frames, ignoring the uneven distribution of information between frames, and therefore cannot extract the most discriminative video features, hindering the improvement of classification accuracy; on the other hand, the existing methods can only implicitly infer temporal information from RGB images, lacking explicit temporal information to guide classification, so the computational efficiency is not high, limiting the practical application of small sample video classification methods. Summary of the invention
[0005] The present invention provides a small sample video classification and classification model training method and device thereof, so as to solve the defect that only RGB images of video frames are processed in the prior art and time series information is not fully utilized, and realize fast, high-precision and high-efficiency small sample video classification based on a small number of example videos.
[0006] The present invention provides a small sample video classification method, comprising:
[0007] Input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information;
[0008] Based on the compressed domain information, obtaining short-time fused frame features;
[0009] Based on the short-time fused frame features, query features are obtained, and classification scores of the support categories to which the query video belongs to each query category prototype in the small sample classification task obtained based on the query features are output, wherein the support category with the largest classification score is used to represent the classification result of the query video.
[0010] According to the small sample video classification method provided by the present invention, the small sample classification task is input into a pre-built convergent compressed domain long-short time Cross-Transformer model to obtain compressed domain information, including:
[0011] Based on the small sample classification task, the predicted frame importance value and image group importance value are obtained;
[0012] Acquire an RGB image of an information frame based on the image group importance value, and acquire a motion vector of a prediction frame based on the prediction frame importance value;
[0013] Based on the preprocessed RGB image of the information frame and the motion vector of the prediction frame, compressed domain information is obtained.
[0014] According to the small sample video classification method provided by the present invention, the RGB image of the information frame is obtained based on the image group importance value, and the motion vector of the predicted frame is obtained based on the predicted frame importance value, including:
[0015] Based on the image group importance value, a stage image group is obtained, wherein the stage image group includes a training image group and a test image group, the training image group is randomly determined based on the probability distribution of the image group obtained based on the normalized image group importance value, and the test image group is determined based on the highest value of the image group importance value;
[0016] Decoding the RGB image of the information frame of the stage image group;
[0017] Based on the stage image group, a stage prediction frame is obtained, wherein the stage prediction frame includes a training prediction frame and a test prediction frame, the training prediction frame is randomly determined based on the probability distribution of the prediction frame obtained based on the normalized prediction frame importance value, and the test prediction frame is determined based on the highest value of the prediction frame importance value;
[0018] The motion vector of the prediction frame of the prediction frame of the stage prediction frame is decoded, and the compression domain information is determined based on the acquisition of the RGB image of the preprocessed information frame and the motion vector of the prediction frame.
[0019] According to the small sample video classification method provided by the present invention, the short-time fused frame features are obtained based on the compressed domain information, including: based on the compressed domain information, an I branch and an MV branch are respectively constructed, and based on the short-time interaction of the laterally connected I branch and the MV branch, each branch outputs the short-time fused frame features, wherein the short-time interaction of the laterally connected I branch and the MV branch occurs within the same image group.
[0020] According to the small sample video classification method provided by the present invention, the query feature is obtained based on the short-time fused frame feature, and the classification score of the query video for the support category to which each query category prototype belongs in the small sample classification task obtained based on the query feature is output, wherein the support category with the largest classification score is used to represent the classification result of the query video, including:
[0021] Based on the processing of short-term fused frame features by the self-attention layer, long-term and short-term frame features are obtained;
[0022] When the long and short time frame features are projected onto the feature space where the prototype of the query category is located, the query features are obtained;
[0023] Output the classification score of the support category to which each query category prototype belongs based on the query video obtained based on the query feature, wherein the query category prototype is obtained by constructing a tuple representation for the query video and the support category respectively based on the mutual attention layer.
[0024] According to the small sample video classification method provided by the present invention, the query category prototype is obtained by constructing a tuple representation for the query video and the support category based on the mutual attention layer, including:
[0025] Based on the vector obtained by concatenating the multiple long and short time frame features, an n-tuple of the video is obtained, where n represents a hyperparameter and is a positive integer;
[0026] Based on the n-tuple set of the query video, obtain an n-tuple representation of the query video;
[0027] Based on the n-tuple set of all supporting videos under the supporting category, an n-tuple representation of the supporting category is obtained.
[0028] The present invention also provides a small sample video classification model training method, comprising:
[0029] Construct a compressed domain long-short time Cross-Transformer model;
[0030] Based on a pre-constructed small sample video classification training data set, the compressed domain long-short time Cross-Transformer model is trained to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the above-mentioned small sample video classification method.
[0031] According to the small sample video classification model training method provided by the present invention, the compressed domain long-short time Cross-Transformer model is trained based on a pre-constructed small sample video classification training data set to obtain a converged compressed domain long-short time Cross-Transformer model, including:
[0032] Based on the classification score of the query video and the query sample label output by the compressed domain long-short time Cross-Transformer model, an objective function value is determined, and based on the objective function value, the parameters of the compressed domain long-short time Cross-Transformer model are optimized using a gradient descent optimization method to obtain a converged compressed domain long-short time Cross-Transformer model.
[0033] The present invention also provides a small sample video classification device, comprising:
[0034] The compressed domain acquisition module is used to input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information;
[0035] A feature extraction module, used for obtaining short-time fused frame features based on the compressed domain information;
[0036] The output module is used to obtain query features based on the short-time fused frame features, and output the classification scores of the support categories to which the query video belongs to each query category prototype in the small sample classification task obtained based on the query features, wherein the support category with the largest classification score is used to represent the classification result of the query video.
[0037] The present invention also provides a small sample video classification model training device, comprising:
[0038] Construction module, used to construct the compressed domain long-short time Cross-Transformer model;
[0039] A training module is used to perform plot training on the compressed domain long-short time Cross-Transformer model based on a pre-constructed small sample video classification training data set to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the small sample video classification method as described in any of the above items.
[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any one of the small sample video classification methods and the small sample video classification model training methods described above.
[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the small sample video classification methods and the small sample video classification model training methods described above.
[0042] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the small sample video classification methods and the small sample video classification model training methods described above.
[0043] The small sample video classification and classification model training method and device provided by the present invention select image groups with higher importance values by utilizing compressed domain information during classification to obtain important frames containing more information, obtain short-time fused frame features and long-short-time frame features through short-time interaction of RGB image appearance information of information frames and predicted frame motion vectors, and globally match query videos and supporting videos, and use the supporting category with the largest predicted classification score as the classification category of the query video, so as to achieve high-precision classification of the query video under the condition of only a small amount of supporting videos, improve computing efficiency, have a wide range of applications, can be trained based on existing large-scale video classification data, do not need to additionally collect and annotate video data, and reduce workload. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 It is a flowchart of a small sample video classification method provided by the present invention;
[0046] Figure 2 It is a schematic diagram of the process of obtaining information based on a single video compression domain provided by the present invention;
[0047] Figure 3 It is a schematic diagram of the process of obtaining the frame features of short-term fusion provided by the present invention;
[0048] Figure 4 It is one of the schematic diagrams provided by the present invention of two branches interacting for a short time through a lateral connection;
[0049] Figure 5 This is the second schematic diagram of the two branches provided by the present invention performing short-term interaction through lateral connection;
[0050] Figure 6 It is a schematic diagram of the process of obtaining query features and classification scores in a single branch provided by the present invention;
[0051] Figure 7 It is a structural schematic diagram of a small sample video classification device provided by the present invention;
[0052] Figure 8 It is a structural schematic diagram of a small sample video classification model training device provided by the present invention;
[0053] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] Combine the following Figure 1-Figure 6 The small sample video classification method of the present invention is described.
[0056] The present invention provides a small sample video classification method. Figure 1 FIG. 1 is a flow chart of a small sample video classification method provided by the present invention. Figure 1 As shown, the method includes:
[0057] Step 110: Input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information.
[0058] Optionally, the method for obtaining compressed domain information includes:
[0059] Based on the small sample classification task, the predicted frame importance value and image group importance value are obtained;
[0060] Acquire the RGB image of the information frame based on the image group importance value, and acquire the motion vector of the prediction frame based on the prediction frame importance value;
[0061] Based on the RGB image of the preprocessed information frame and the motion vector of the predicted frame, the compressed domain information is obtained.
[0062] Optionally, the classification of the compressed domain long-short time Cross-Transformer model is based on a task, where the task is a collection of support categories and query videos, wherein each support category provides multiple support videos as examples, i.e., the support videos are videos of known classification categories, the support categories are classification categories of the support videos, the support categories to which the support videos belong are given in the task, and the categories to which the query videos belong are unknown, and the goal of the task is to classify all query videos into one of the support categories. For example, the task input to the compressed domain long-short time Cross-Transformer model includes K support videos and N query videos, and the output of the compressed domain long-short time Cross-Transformer model is the classification scores of all N query videos for each support category. The compressed domain long-short time Cross-Transformer model of the present invention does not restrict the structure of the task, that is, the model does not restrict the number of support categories in the task, the number of support videos contained in each support category, and the number of query videos. It does not require that all tasks must have the same structure. For convenience, the task structure can be restricted to be the same during implementation, and missing videos can be replaced with placeholder inputs. Attention masks are added to the mutual attention layer when constructing tuple representations to avoid placeholder inputs from participating in prototype calculations.
[0063] Optionally, Figure 2 is a schematic diagram of a process for obtaining information based on a single video compression domain provided by the present invention, such as Figure 2 As shown, the method for obtaining compressed domain information of a single video includes:
[0064] A compressed video stored in a compressed format is used as an input video; the motion vectors of all predicted frames in the compressed video are extracted; the predicted frame importance value and the image group importance value are calculated based on the motion vector of the predicted frame; the RGB image of the information frame and the motion vector of the predicted frame are extracted from the compressed video based on the predicted frame importance value and the image group importance value, and the extracted RGB image of the information frame and the motion vector of the predicted frame are preprocessed to obtain compressed domain information. The predicted frame importance value I(M) is calculated based on the motion vector of the predicted frame. t,l ) is calculated as shown in formula (1):
[0065]
[0066] Where t represents the image group number, l represents the prediction frame number, |G t,l | represents the number of elements in the image coordinate set of the lth prediction frame of the tth image group, (x, y) represents the image coordinates, ||M t,l (x,y)||1 represents the L1 norm of the motion vector at the image coordinate (x,y), and α represents a hyperparameter.
[0067] Based on the predicted frame importance value I(M t,l ) Calculate the importance value of the picture group I(GOP t ) is calculated as shown in formula (2):
[0068]
[0069] Where t represents the image group number, L t represents the number of predicted frames of the t-th image group, I(M t,l ) represents the importance value of the lth predicted frame of the tth image group.
[0070] Alternatively, if Figure 2 As shown in the figure, the input is a video stored in a compressed format, and the output is the compressed domain information of the video. In practical applications, the video is almost stored in a compressed format. Common formats such as MPEG and H.264 use motion prediction to encode the video. The video is divided into multiple groups of pictures by frame. Each group of pictures consists of an information frame (I-frame) and a number of consecutive prediction frames (P-frames) following it. The information frame is used to encode the RGB image, and the prediction frame is used to encode the motion vector and residual calculated by block with the previous frame as a reference. Starting from the first prediction frame after the information frame, the RGB image of the current frame is calculated according to the current prediction frame and the RGB image of the previous frame. Therefore, only part of the decoding process is needed to obtain the RGB image of the information frame and the motion vector of the prediction frame, which greatly improves the decoding efficiency.
[0071] Optionally, the method for acquiring the RGB image of the information frame and the motion vector of the prediction frame includes:
[0072] Based on the importance values of the image groups, a stage image group is obtained, wherein the stage image group includes a training image group and a test image group, the training image group is randomly determined based on the probability distribution of the image group obtained based on the normalized image group importance values, and the test image group is determined based on the highest image group importance value, that is, during training, the image group importance values are normalized to obtain the probability distribution of the image groups, and g image groups are randomly selected based on the probability distribution, namely the training image groups, and g image groups with the highest image group importance values are selected during testing, namely the test image groups, where g represents a hyperparameter and is a positive integer;
[0073] RGB image of the information frame of the image group in the decoding stage;
[0074] Based on the stage image group, a stage prediction frame is obtained, wherein the stage prediction frame includes a training prediction frame and a test prediction frame, the training prediction frame is randomly determined based on the probability distribution of the prediction frame obtained based on the normalized prediction frame importance value, and the test prediction frame is determined based on the highest prediction frame importance value, that is, based on the selected stage image group, during training, the prediction frame importance values of all prediction frames in the training image group are normalized to obtain the probability distribution of the prediction frame, and m prediction frames are randomly selected based on the probability distribution of the prediction frame, namely, the training prediction frame, and during testing, the m prediction frames with the highest prediction frame importance values are selected, where m represents a hyperparameter and is a positive integer. Considering the stability of the test, the extraction process during the test eliminates the randomness of the extraction process during the training;
[0075] The motion vector of the prediction frame of the prediction frame of the stage prediction frame is decoded, and the compression domain information is determined based on the acquisition of the RGB image of the preprocessed information frame and the motion vector of the prediction frame.
[0076] Optionally, the method for extracting the RGB image of the information frame and the motion vector of the prediction frame also includes: the extraction process during testing is the same as the extraction process during training, but multiple predictions are performed on the same task, the classification scores of the query video are averaged as the final predicted classification score, and then the supporting category with the largest score is taken as the predicted category of the query video.
[0077] Optionally, the preprocessing method of the RGB image of the information frame and the motion vector of the prediction frame includes: accumulating and aligning the motion vector of the prediction frame; dividing the motion vector of the prediction frame by the height and width of the video respectively; scaling the video to a fixed size; randomly cropping the video according to random size and aspect ratio during training, and center cropping the video according to a fixed size during testing; performing random color perturbations on the RGB image of the information frame during training, and not operating the color of the RGB image of the information frame during testing; and normalizing the RGB image of the RGB image of the information frame. Among them, the preprocessing during testing is different from the preprocessing during training. Its purpose is to eliminate randomness during testing and maintain the stability of the test results. To achieve this goal, the same preprocessing operation as that during training can also be used during testing, but multiple predictions are made for the same task, and then the classification scores of the query video are averaged as the final predicted classification score, and the support category with the largest classification score is removed as the predicted category of the query video.
[0078] Optionally, in the preprocessing, the motion vectors of the predicted frames may be accumulated and aligned using an iterative algorithm, and the steps of the iterative algorithm are as follows:
[0079] 1) Initialize the current frame as the frame for which the cumulative motion vector is to be calculated, and initialize the cumulative motion vector as the motion vector of the frame;
[0080] 2) Find the reference position of the cumulative motion vector in the previous frame, connect the motion vector of the reference position of the previous frame with the cumulative motion vector, and move the current frame forward by one frame;
[0081] 3) Loop through step 2) until the current frame is an information frame;
[0082] 4) Store the accumulated motion vector to the image coordinates to which it refers.
[0083] Step 120: Obtain short-time fused frame features based on the compressed domain information.
[0084] Optionally, Figure 3 FIG. 1 is a schematic diagram of the process of obtaining the short-term fusion frame features provided by the present invention, such as Figure 3 As shown, the method includes:
[0085] Based on the compressed domain information, the I branch and the MV branch are constructed respectively, and based on the short-time interaction of the laterally connected I branch and the MV branch, each branch outputs short-time fused frame features, wherein the short-time interaction of the laterally connected I branch and the MV branch occurs within the same image group.
[0086] Alternatively, if Figure 3 As shown, both the I branch and the MV branch include but are not limited to a backbone neural network, the I branch takes the RGB image of the information frame as input, the MV branch takes the motion vector of the predicted frame as input, the two branches interact for a short time through lateral connections, and the two branches respectively output short-time fused frame features.
[0087] For example, both branches use ResNet as the backbone neural network. The I branch uses the ResNet18 structure, and the MV branch uses the ResNet50 structure, and they interact at each level. ResNet is a multi-level convolutional neural network, which contains five levels of convolutional layers, namely Conv1, Conv2, Conv3, Conv4 and Conv5. In particular, an instance normalization layer is added before Conv1 of the MV branch. The first convolutional layer of the MV branch is replaced by a 7×7 convolutional layer with 2 input channels. The reason for the replacement is that the motion vector of the predicted frame has only two channels, represented as the x component and the y component. The last fully connected layer of both branches is removed to obtain the feature vector as the frame feature. Except for Conv1, each level contains multiple residual blocks.
[0088] Optionally, Figure 4 is one of the schematic diagrams of the present invention providing two branches that interact for a short time through lateral connection, such as Figure 4As shown in , the interaction only occurs between the I branch features and the MV branch features in the same image group. The channels of the p part of the I branch features and all the channels of the MV branch participate in the lateral connection. p represents a hyperparameter and p∈[0,1]. The p part of the RGB image features of the information frame that participates in the lateral connection is divided into r parts, and then the lateral connection is established with the motion vector features of the selected prediction frame in the image group where the frame belongs. For example, Figure 4 As shown, the number of prediction frames in each image group is 2, so the p part of the information frame I branch feature is divided into 2 equal parts, that is, r=2, and interacts with the MV branch feature keys of the two prediction frames in the same image group respectively. Figure 4 The I branch and MV branch shown both use ResNet neural networks, and lateral connections are only performed between the first residual blocks of Conv2, Conv3, Conv4 and Conv5 of the I branch and the first residual block of the corresponding level of the MV branch. The residual blocks participating in the interaction are called interactive residual blocks.
[0089] Optionally, Figure 5 This is the second schematic diagram of the two branches provided by the present invention for short-term interaction through lateral connection, such as Figure 5 As shown, the interactive method of the interactive residual block includes: using 1×1 group convolution to realize lateral connection, and the I branch features and MV branch features participating in the lateral connection are respectively convolved and superimposed on the residual path of the corresponding interactive residual block of the relative branch.
[0090] Optionally, for each video, after the RGB images of g information frames are processed in step 220, g I-branch frame feature vectors are obtained, and the g I-branch frame feature vectors are stacked row by row to obtain an I-branch feature matrix Among them, d I is the dimension of the I branch frame feature vector; the motion vectors of the gr predicted frames are processed in step 220 to obtain gr MV branch frame feature vectors, and the gr MV branch frame feature vectors are stacked row by row to obtain the MV branch feature matrix Among them, d MV is the dimension of the MV branch frame feature vector. I and Z MV It is the short-term fused frame feature of the final output, which is used as the input of the subsequent two long-term units 4013 respectively.
[0091] Step 130: Based on the short-time fused frame features, query features are obtained, and classification scores of the support categories to which the query video belongs for each query category prototype in the small sample classification task obtained based on the query features are output, wherein the support category with the largest classification score is used to represent the classification result of the query video.
[0092] Optionally, the method for obtaining the query features and classification scores includes:
[0093] Based on the processing of short-term fused frame features by the self-attention layer, long-term and short-term frame features are obtained;
[0094] When the long and short time frame features are projected onto the feature space where the prototype of the query category is located, the query features are obtained;
[0095] Output the classification scores of the support categories to which each query category prototype belongs based on the query video obtained based on the query features, where the query category prototype is obtained by constructing tuple representations for the query video and the support category respectively based on the mutual attention layer.
[0096] Optionally, Figure 6 is a schematic diagram of the process of obtaining query features and classification scores in a single branch provided by the present invention, such as Figure 6 As shown, the method includes:
[0097] 1) The short-term fused frame features of each video are processed through the self-attention layer to obtain the long-term and short-term frame features. The calculation formula of the self-attention layer is shown in formula (3):
[0098]
[0099] Among them, H represents the output of self-attention, softmax() represents the softmax function, Represents three learnable linear transformations.
[0100] 2) Through the mutual attention layer, tuple representations are constructed for the query video and each supporting category respectively. The query category prototype is calculated based on the tuple representation, and the long and short time frame features of the query video are projected into the feature space where the query category prototype is located to obtain the query features. Based on the n-tuple representation of the supporting category and the query video, the calculation formula for calculating each supporting category prototype using the mutual attention layer is shown in formula (4):
[0101]
[0102] Among them, u b,c Represents the n-tuple q for the query video b The calculated prototype of the support category c, q b represents an n-tuple with b as the subscript, and represents three learnable linear transformations, d k and d v represents a hyperparameter and is a positive integer, S c It represents the class tuple representation obtained by stacking all n-tuples of all supporting videos of supporting category c in rows.
[0103] 3) Calculate the negative distance between the query feature and each query category prototype in the feature space where the query category prototype is located, and obtain the classification score of the query video for the supporting category to which each query category prototype belongs as the model output. The calculation formula of the negative distance is shown in formula (5):
[0104] δ b,c =-||q b W V -u b,c || (5),
[0105] Among them, δ b,c is the n-tuple q of the query video b To support the prototype u corresponding to category c b,c The distance between the query video and the support category c is calculated as follows: ||·|| represents any distance metric, including but not limited to the squared Euclidean distance. After calculating the negative distance of each n-tuple of the query video as shown in formula (5), the mean of the negative distances is taken as the classification score of the query video to the support category c, which is recorded as For each input task, the compressed domain long-short time Cross-Transformer model outputs the corresponding classification score for each query video in it.
[0106] Optionally, the processing methods for the I branch and the MV branch in step 130 are exactly the same, except that the parameter dimension Z I and Z MV The dimensions of Z∈{Z I ,Z MV} as the feature matrix of the corresponding branch output, d∈{d I ,d MV} is the dimension of the corresponding matrix.
[0107] Alternatively, if Figure 6 As shown, Z is layer-normalized between self-attention layers. Multi-head self-attention layers can be used to replace self-attention layers, and position encoding can be superimposed on the row vectors of Z. The tuple representation is layer-normalized before the mutual attention layer.
[0108] Optionally, an n-tuple of a vector set is defined as a vector obtained by concatenating all vectors in an n-element subset of the set head to tail. Based on the vectors obtained by concatenating multiple long and short time frame features head to tail, an n-tuple of the video is obtained, where n represents a hyperparameter and is a positive integer;
[0109] Based on the n-tuple set of the query video, obtaining an n-tuple representation of the query video, wherein the n-tuple set of the query video includes n-tuples of multiple query videos;
[0110] Based on the n-tuple set of all supporting videos under the supporting category, an n-tuple representation of the supporting category is obtained.
[0111] Optionally, the calculation method of the mutual attention layer includes: constructing an n-tuple representation for the query video and the supporting category; calculating the prototype of each supporting category using the Hu-attention layer based on the n-tuple representation of the query video and the supporting category; projecting the n-tuple representation of the query video to the feature space where the supporting category prototype is located, and then calculating the negative distance in the feature space where the supporting category prototype is located as the classification score.
[0112] The n-tuple representation of the video consists of all possible n-tuples of the row vector of the output H of the self-attention layer. The calculation formula of each n-tuple is shown in formula (6):
[0113]
[0114] Where b = {j1, j2, …, j n}, which represents a non-repeating combination of n of the row vector numbers of the output H. For example, when n=3, H contains 8 rows and the row numbers are {1,2,…,8}, b can be {1,3,6}. Represents a vector concatenation operation. The output H is superimposed with a positional encoding before computing the n-tuple.
[0115] Optionally, the small sample classification task includes: supporting categories, supporting videos and query videos. The supporting videos in the small sample classification task include but are not limited to videos in a meta-training set and a meta-validation set in a small sample video classification training dataset. The supporting categories in the small sample classification task include but are not limited to categories in a meta-training set and a meta-validation set. The classification result calculation formula is shown in formula (7):
[0116]
[0117] in, represents the classification score of the query video for the support category c in step 130.
[0118] The small sample video classification method provided by the present invention selects image groups with higher importance values by utilizing compressed domain information during classification to obtain important frames containing more information, obtains short-time fused frame features and long-short-time frame features through short-time interaction of RGB image appearance information of information frames and predicted frame motion vectors, and globally matches query videos and supporting videos, and uses the supporting category with the largest predicted classification score as the classification category of the query video, thereby achieving high-precision classification of the query video under the condition of only a small amount of supporting videos, improving computational efficiency, and having a wide range of applications. At the same time, it can be trained based on existing large-scale video classification data, without the need to additionally collect and annotate video data, thereby reducing workload.
[0119] The present invention also provides a small sample video classification model training method, the method comprising:
[0120] Construct a compressed domain long-short time Cross-Transformer model;
[0121] Based on a pre-constructed small sample video classification training dataset, a compressed domain long-short time Cross-Transformer model is trained on a plot to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the above-mentioned small sample video classification method. Optionally, the method for constructing a small sample video classification training dataset includes: dividing a large-scale video classification dataset into three subsets by category, and the categories contained in different subsets do not overlap; sampling multiple tasks from each of the three subsets to form a meta-training set, a meta-validation set, and a meta-test set, respectively, the meta-training set and the meta-validation set are used for model training and hyperparameter selection, and the meta-test set is used for model evaluation and comparison between different models.
[0122] Optionally, a complete forward computation of the model starts with inputting a task and ends with outputting a classification score or classification result. The label of the task, that is, the actual category of the query video, is used for model training and evaluation and should not be used as the input of the model.
[0123] Optionally, the episode training method includes:
[0124] Based on the classification score of the query video and the query sample label output by the compressed domain long-short time Cross-Transformer model, the objective function value is determined, and based on the objective function value, the parameters of the compressed domain long-short time Cross-Transformer model are optimized using the gradient descent optimization method to obtain a converged compressed domain long-short time Cross-Transformer model.
[0125] Optionally, the objective function adopts a cross entropy loss function, and the calculation formula is shown in formula (8):
[0126]
[0127] Where N is the number of query videos in the task, C is the number of support categories in the task, and h c,i is the 0-1 label of the i-th query video for the c-th support category, is the classification score of the i-th query video for the c-th support category.
[0128] Optionally, methods to speed up the convergence of the plot training process include but are not limited to: using ImageNet to preprocess weights, initializing the ResNet network, reducing the learning rate by 0.1 according to a certain training step, and using the Warmup technique to linearly increase from a lower learning rate to a higher initial learning rate.
[0129] The small sample video classification model training method provided by the present invention constructs a compressed domain long-short time Cross-Transformer model, and optimizes the parameters in the converged compressed domain long-short time Cross-Transformer model through plot training, thereby improving the accuracy of compressed domain information, short-time fused frame features and query features, and further improving classification accuracy.
[0130] The small sample video classification device provided by the present invention is described below. The small sample video classification device described below and the small sample video classification method described above can be referenced to each other.
[0131] The present invention also provides a small sample video classification device. Figure 7 is a schematic diagram of the structure of the small sample video classification device provided by the present invention, such as Figure 7 As shown, the small sample video classification device 400 includes: a compression domain acquisition module 401, a feature extraction module 402, and an output module 403, wherein:
[0132] A compressed domain acquisition module 401 is used to input a small sample classification task into a pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information;
[0133] A feature extraction module 402, used to obtain short-time fused frame features based on compressed domain information;
[0134] The output module 403 is used to obtain query features based on short-time fused frame features, and output the classification scores of the support categories to which the query video belongs to each query category prototype in the small sample classification task obtained based on the query features, wherein the support category with the largest classification score is used to represent the classification result of the query video.
[0135] The small sample video classification device provided by the present invention selects image groups with higher importance values by utilizing compressed domain information during classification to obtain important frames containing more information, obtains short-time fused frame features and long-short-time frame features through short-time interaction of RGB image appearance information of information frames and predicted frame motion vectors, and globally matches query videos and supporting videos, and uses the supporting category with the largest predicted classification score as the classification category of the query video, thereby achieving high-precision classification of the query video under the condition of only a small amount of supporting videos, and improving computing efficiency, with a wide range of applications, and can be trained based on existing large-scale video classification data, without the need to additionally collect and annotate video data, thereby reducing workload.
[0136] Optionally, the compression domain acquisition module 401 is specifically configured to:
[0137] Based on the small sample classification task, the predicted frame importance value and image group importance value are obtained;
[0138] Acquire the RGB image of the information frame based on the image group importance value, and acquire the motion vector of the prediction frame based on the prediction frame importance value;
[0139] Based on the RGB image of the preprocessed information frame and the motion vector of the predicted frame, the compressed domain information is obtained.
[0140] Optionally, the compression domain acquisition module 401 is specifically configured to:
[0141] Based on the importance value of the image group, a stage image group is obtained, wherein the stage image group includes a training image group and a test image group, the training image group is randomly determined based on the probability distribution of the image group obtained based on the normalized image group importance value, and the test image group is determined based on the highest value of the image group importance value;
[0142] RGB image of the information frame of the image group in the decoding stage;
[0143] Based on the stage image group, a stage prediction frame is obtained, wherein the stage prediction frame includes a training prediction frame and a test prediction frame, the training prediction frame is randomly determined based on the probability distribution of the prediction frame obtained based on the normalized prediction frame importance value, and the test prediction frame is determined based on the highest value of the prediction frame importance value;
[0144] The motion vector of the predicted frame of the predicted frame in the decoding stage is obtained based on the RGB image of the preprocessed information frame and the motion vector of the predicted frame to determine the compression domain information.
[0145] Optionally, the feature extraction module 402 is specifically used for:
[0146] Based on the compressed domain information, the I branch and the MV branch are constructed respectively, and based on the short-time interaction of the laterally connected I branch and the MV branch, each branch outputs short-time fused frame features, wherein the short-time interaction of the laterally connected I branch and the MV branch occurs within the same image group.
[0147] Optionally, the output module 403 is specifically configured to:
[0148] Based on the processing of short-term fused frame features by the self-attention layer, long-term and short-term frame features are obtained;
[0149] When the long and short time frame features are projected onto the feature space where the prototype of the query category is located, the query features are obtained;
[0150] Output the classification scores of the support categories to which each query category prototype belongs based on the query video obtained based on the query features, where the query category prototype is obtained by constructing tuple representations for the query video and the support category respectively based on the mutual attention layer.
[0151] Optionally, the output module 403 is specifically configured to:
[0152] Based on the vector obtained by concatenating multiple long and short time frame features, an n-tuple of the video is obtained, where n represents a hyperparameter and is a positive integer;
[0153] Based on the n-tuple set of the query video, obtain an n-tuple representation of the query video;
[0154] Based on the n-tuple set of all supporting videos under the supporting category, an n-tuple representation of the supporting category is obtained.
[0155] The present invention also provides a small sample video classification model training device, Figure 8 is a structural diagram of a small sample video classification model training device provided by the present invention, such as Figure 8 As shown, the small sample video classification model training device 500 includes a construction module 501 and a training module 502, wherein:
[0156] A construction module 501 is used to construct a compressed domain long-short time Cross-Transformer model;
[0157] The training module 502 is used to perform plot training on the compressed domain long-short time Cross-Transformer model based on a pre-constructed small sample video classification training data set to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the small sample video classification method as described in any of the above items.
[0158] The small sample video classification model training device provided by the present invention constructs a compressed domain long-short time Cross-Transformer model, and optimizes the parameters in the converged compressed domain long-short time Cross-Transformer model through plot training, thereby improving the accuracy of compressed domain information, short-time fused frame features and query features, and further improving classification accuracy.
[0159] Optionally, the training module 502 is specifically configured to:
[0160] Based on the classification score of the query video and the query sample label output by the compressed domain long-short time Cross-Transformer model, an objective function value is determined, and based on the objective function value, the parameters of the compressed domain long-short time Cross-Transformer model are optimized using a gradient descent optimization method to obtain a converged compressed domain long-short time Cross-Transformer model.
[0161] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device 600 may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the small sample video classification method and the small sample video classification model training method, wherein:
[0162] Small sample video classification methods include:
[0163] Input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information;
[0164] Based on the compressed domain information, the short-term fused frame features are obtained;
[0165] Based on the short-time fused frame features, the query features are obtained, and the classification scores of the support categories to which the query video belongs to each query category prototype in the small sample classification task obtained based on the query features are output, where the support category with the largest classification score is used to represent the classification result of the query video.
[0166] The small sample video classification model training method includes:
[0167] Construct a compressed domain long-short time Cross-Transformer model;
[0168] Based on a pre-constructed small sample video classification training dataset, a compressed domain long-short time Cross-Transformer model is trained to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the above-mentioned small sample video classification method.
[0169] Optionally, the specific hardware and programming language of the present invention include but are not limited to: a computer with a 2.8 GHz central processor, 16 GB of memory and 4 NVIDIA RTX2060 image processors, and a small sample video classification working program is compiled using Python and C languages based on the PyTorch deep learning framework and the FFmpeg video encoding program to implement a small sample video classification method and a small sample video classification model training method.
[0170] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0171] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the small sample video classification method and the small sample video classification model training method provided by the above methods, wherein:
[0172] Small sample video classification methods include:
[0173] Input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information;
[0174] Based on the compressed domain information, the short-term fused frame features are obtained;
[0175] Based on the short-time fused frame features, the query features are obtained, and the classification scores of the support categories to which the query video belongs to each query category prototype in the small sample classification task obtained based on the query features are output, where the support category with the largest classification score is used to represent the classification result of the query video.
[0176] The small sample video classification model training method includes:
[0177] Construct a compressed domain long-short time Cross-Transformer model;
[0178] Based on a pre-constructed small sample video classification training dataset, a compressed domain long-short time Cross-Transformer model is trained to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the above-mentioned small sample video classification method.
[0179] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the small sample video classification method and the small sample video classification model training method provided by the above methods, wherein:
[0180] Small sample video classification methods include:
[0181] Input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information;
[0182] Based on the compressed domain information, the short-term fused frame features are obtained;
[0183] Based on the short-time fused frame features, the query features are obtained, and the classification scores of the support categories to which the query video belongs to each query category prototype in the small sample classification task obtained based on the query features are output, where the support category with the largest classification score is used to represent the classification result of the query video.
[0184] The small sample video classification model training method includes:
[0185] Construct a compressed domain long-short time Cross-Transformer model;
[0186] Based on a pre-constructed small sample video classification training dataset, a compressed domain long-short time Cross-Transformer model is trained to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the above-mentioned small sample video classification method.
[0187] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.
[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A small sample video classification method, characterized in that: include: Input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information; Based on the compressed domain information, obtaining short-time fused frame features; Based on the short-time fused frame features, query features are obtained, and classification scores of the query video for the support categories to which each query category prototype belongs in the small sample classification task obtained based on the query features are output, wherein the support category with the largest classification score is used to represent the classification result of the query video; The small sample classification task is input into a pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information, including: Based on the small sample classification task, the predicted frame importance value and image group importance value are obtained; Acquire an RGB image of an information frame based on the image group importance value, and acquire a motion vector of a prediction frame based on the prediction frame importance value; Based on the preprocessed RGB image of the information frame and the motion vector of the prediction frame, obtaining compressed domain information; The step of acquiring the RGB image of the information frame based on the image group importance value, and acquiring the motion vector of the prediction frame based on the prediction frame importance value, comprises: Based on the image group importance value, a stage image group is obtained, wherein the stage image group includes a training image group and a test image group, the training image group is randomly determined based on the probability distribution of the image group obtained based on the normalized image group importance value, and the test image group is determined based on the highest value of the image group importance value; Decoding the RGB image of the information frame of the stage image group; Based on the stage image group, a stage prediction frame is obtained, wherein the stage prediction frame includes a training prediction frame and a test prediction frame, the training prediction frame is randomly determined based on the probability distribution of the prediction frame obtained based on the normalized prediction frame importance value, and the test prediction frame is determined based on the highest value of the prediction frame importance value; The motion vector of the prediction frame of the prediction frame of the stage prediction frame is decoded, and the compression domain information is determined based on the acquisition of the RGB image of the preprocessed information frame and the motion vector of the prediction frame.
2. The small sample video classification method according to claim 1, characterized in that: The method of obtaining short-time fused frame features based on the compressed domain information includes: constructing an I branch and an MV branch respectively based on the compressed domain information, and based on the short-time interaction of the laterally connected I branch and the MV branch, each branch outputs short-time fused frame features, wherein the short-time interaction of the laterally connected I branch and the MV branch occurs in the same image group.
3. The small sample video classification method according to claim 1 or 2, characterized in that: The query feature is obtained based on the frame feature of the short-time fusion, and the classification score of the query video for the support category to which each query category prototype belongs in the small sample classification task obtained based on the query feature is output, wherein the support category with the largest classification score is used to represent the classification result of the query video, including: Based on the processing of short-term fused frame features by the self-attention layer, long-term and short-term frame features are obtained; When the long and short time frame features are projected onto the feature space where the prototype of the query category is located, the query features are obtained; Output the classification score of the support category to which each query category prototype belongs based on the query video obtained based on the query feature, wherein the query category prototype is obtained by constructing a tuple representation for the query video and the support category respectively based on the mutual attention layer.
4. The small sample video classification method according to claim 3, characterized in that: The query category prototype is obtained by constructing tuple representations for query videos and support categories based on the mutual attention layer, including: Based on the vector obtained by concatenating the multiple long and short time frame features, an n-tuple of the video is obtained, where n represents a hyperparameter and is a positive integer; Based on the n-tuple set of the query video, obtain an n-tuple representation of the query video; Based on the n-tuple set of all supporting videos under the supporting category, an n-tuple representation of the supporting category is obtained.
5. A small sample video classification model training method, characterized in that: include: Construct a compressed domain long-short time Cross-Transformer model; Based on a pre-constructed small sample video classification training data set, the compressed domain long-short time Cross-Transformer model is trained to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the small sample video classification method as described in any one of claims 1-4.
6. The small sample video classification model training method according to claim 5, characterized in that: The method of performing plot training on the compressed domain long-short time Cross-Transformer model based on the pre-constructed small sample video classification training data set to obtain a converged compressed domain long-short time Cross-Transformer model includes: Based on the classification score of the query video and the query sample label output by the compressed domain long-short time Cross-Transformer model, an objective function value is determined, and based on the objective function value, the parameters of the compressed domain long-short time Cross-Transformer model are optimized using a gradient descent optimization method to obtain a converged compressed domain long-short time Cross-Transformer model.
7. A small sample video classification device, characterized in that: include: The compressed domain acquisition module is used to input the small sample classification task into the pre-built converged compressed domain long-short time Cross-Transformer model to obtain compressed domain information; A feature extraction module, used for obtaining short-time fused frame features based on the compressed domain information; An output module is used to obtain query features based on the short-time fused frame features, and output classification scores of the query video for the support categories to which each query category prototype belongs in the small sample classification task obtained based on the query features, wherein the support category with the largest classification score is used to represent the classification result of the query video; The compression domain acquisition module is specifically used for: Based on the small sample classification task, the predicted frame importance value and image group importance value are obtained; Acquire the RGB image of the information frame based on the image group importance value, and acquire the motion vector of the prediction frame based on the prediction frame importance value; Based on the RGB image of the preprocessed information frame and the motion vector of the predicted frame, the compressed domain information is obtained; The compression domain acquisition module is also used to: Based on the importance value of the image group, a stage image group is obtained, wherein the stage image group includes a training image group and a test image group, the training image group is randomly determined based on the probability distribution of the image group obtained based on the normalized image group importance value, and the test image group is determined based on the highest value of the image group importance value; RGB image of the information frame of the image group in the decoding stage; Based on the stage image group, a stage prediction frame is obtained, wherein the stage prediction frame includes a training prediction frame and a test prediction frame, the training prediction frame is randomly determined based on the probability distribution of the prediction frame obtained based on the normalized prediction frame importance value, and the test prediction frame is determined based on the highest value of the prediction frame importance value; The motion vector of the predicted frame of the predicted frame in the decoding stage is obtained based on the RGB image of the preprocessed information frame and the motion vector of the predicted frame to determine the compression domain information.
8. A small sample video classification model training device, characterized in that: include: Construction module, used to construct the compressed domain long-short time Cross-Transformer model; A training module is used to perform plot training on the compressed domain long-short time Cross-Transformer model based on a pre-constructed small sample video classification training data set to obtain a converged compressed domain long-short time Cross-Transformer model, wherein the converged compressed domain long-short time Cross-Transformer model executes the small sample video classification method as described in any one of claims 1-4.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the small sample video classification method as described in any one of claims 1 to 4 and the small sample video classification model training method as described in claim 5 or 6 are implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the small sample video classification method as described in any one of claims 1 to 4 and the small sample video classification model training method as described in claim 5 or 6.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the small sample video classification method as described in any one of claims 1 to 4 and the small sample video classification model training method as described in claim 5 or 6.
Citation Information
Patent Citations
Covariance measure driven small sample GIF short video emotion recognition method and system
CN110532911A
Rapid video classification method, system and device based on H264 compressed domain
CN112235569A
Method and system for generating a vector representation of an image
US20210224612A1
