Model training and video processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a model training and video processing method and device, electronic equipment and storage medium. The model training method comprises: obtaining a training video; performing spatio-temporal transformation on the training video according to different spatio-temporal transformation modes to obtain at least two different spatio-temporal transformation videos; and training the video feature extraction model using video continuity information in the spatio-temporal transformation videos as self-supervised information. This technical solution can train the video feature extraction model without manual label annotation, and use the video continuity information in the spatio-temporal transformation videos as self-supervised information, so that the video feature extraction model can learn the video features sufficiently, which can not only reduce the training cost of the video feature extraction model, but also improve the recognition accuracy of the video feature extraction model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to a model training and video processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of technology, machine learning is widely applied in various fields. Machine learning types can include a variety of, for example, self-supervised learning. Self-supervised learning is a supervised learning without manual labeling of sample data, that is, there is no human involvement in the self-supervised learning process; but the label still exists in self-supervised learning, and the label is automatically generated from the input data, i.e. sample data. Self-supervised learning is less used in the field of video, but due to the difficulty of manual labeling of video samples and high cost, how to use self-supervised learning method to learn good video representation from video samples is one of the technical problems to be solved in the field. SUMMARY

[0003] The embodiments of the present disclosure provide a model training and video processing method, device, electronic equipment and computer readable storage medium.

[0004] In a first aspect, the embodiments of the present disclosure provide a model training method, comprising:

[0005] obtaining a training video;

[0006] spatial-temporal transforming the training video according to different spatial-temporal transformation modes to obtain at least two different spatial-temporal transformation videos;

[0007] training the video feature extraction model with video continuity information in the spatial-temporal transformation video as self-supervised information.

[0008] Further, the spatial-temporal transforming the training video according to different spatial-temporal transformation modes to obtain at least two different spatial-temporal transformation videos comprises:

[0009] obtaining a preset transformation order between time dimension and space dimension;

[0010] transforming the training video in time dimension and space dimension according to the preset transformation order to obtain the spatial-temporal transformation video.

[0011] Further, the spatial-temporal transforming the training video according to different spatial-temporal transformation modes to obtain at least two different spatial-temporal transformation videos comprises:

[0012] dividing video frames in the training video into pixel blocks;

[0013] In the spatio-temporal transformation mode, spatio-temporal transformation is performed on the pixel blocks to obtain the spatio-temporal transformation video.

[0014] Further, the video feature extraction model is trained by using the video continuity information in the spatio-temporal transformation video as self-supervised information, including:

[0015] The video feature extraction model is used to obtain the feature vectors corresponding to the training video and the spatio-temporal transformation video;

[0016] The prediction result of the video continuity information is obtained from the feature vectors by using the at least one training task;

[0017] The video feature extraction model is trained according to the prediction result and the true result of the video continuity information, wherein the true result is determined by using the spatio-temporal transformation mode corresponding to the spatio-temporal transformation video.

[0018] Further, the at least one training task includes at least one of the following:

[0019] The first training task is to predict the continuous pixel block information in the spatio-temporal transformation video based on the feature vectors;

[0020] The second training task is to predict the volume ratio of the continuous pixel block in the spatio-temporal transformation video relative to the spatio-temporal transformation video, and the Hamming distance of the continuous pixel block in the time dimension and the space dimension relative to the training video based on the feature vectors;

[0021] The third training task is to predict the spatio-temporal transformation mode corresponding to the spatio-temporal transformation video based on the feature vectors;

[0022] The fourth training task is to predict the most similar one of the plurality of spatio-temporal transformation videos to the training video based on the feature vectors.

[0023] Further, the method further includes obtaining the true result by at least one of the following ways:

[0024] The position information of the largest continuous image block pixel block and the second largest continuous pixel block in the spatio-temporal transformation video is determined, and the position information is determined as the true result;

[0025] The volume ratio between the sum of the volumes of the largest continuous pixel block and the second largest continuous pixel block in the spatio-temporal transformation video and the total volume of the spatio-temporal transformation video is determined, and the standard Hamming distance of the largest continuous pixel block in the time dimension and the space dimension is determined, and the volume ratio and the standard Hamming distance are determined as the true result;

[0026] determine the spatio-temporal transformation video corresponding to the preset spatio-temporal transformation mode as the real result.

[0027] determine the spatio-temporal transformation video corresponding to the preset spatio-temporal transformation mode as the real result.

[0028] In a second aspect, the embodiments of the present disclosure provide a video processing method, which includes:

[0029] obtaining an input video;

[0030] representing the input video as a feature vector by using a video feature extraction model; the video feature extraction model is trained by using the method of the first aspect.

[0031] In a third aspect, the embodiments of the present disclosure provide a model training method, which includes:

[0032] obtaining a training video;

[0033] calling a preset service interface, so as to perform spatio-temporal transformation on the training video according to different spatio-temporal transformation modes by using the preset service interface, obtain at least two different spatio-temporal transformation videos, and train the video feature extraction model by using video continuity information in the spatio-temporal transformation video as self-supervised information;

[0034] outputting the video feature extraction model.

[0035] In a fourth aspect, the embodiments of the present disclosure provide a model training device, which includes:

[0036] a first obtaining module configured to obtain a training video;

[0037] a transformation module configured to perform spatio-temporal transformation on the training video according to different spatio-temporal transformation modes, and obtain at least two different spatio-temporal transformation videos;

[0038] a training module configured to train the video feature extraction model by using video continuity information in the spatio-temporal transformation video as self-supervised information.

[0039] In a fifth aspect, the embodiments of the present disclosure provide a video processing device, which includes:

[0040] a second obtaining module configured to obtain an input video;

[0041] a feature representation module configured to represent the input video as a feature vector by using a video feature extraction model; the video feature extraction model is trained by using the device of the fourth aspect.

[0042] In a sixth aspect, the embodiments of the present disclosure provide a model training device, which includes:

[0043] a third obtaining module, configured to obtain a training video;

[0044] a calling module, configured to call a preset service interface, so as to perform spatio-temporal transformation on the training video according to different spatio-temporal transformation modes to obtain at least two different spatio-temporally transformed videos, and train the video feature extraction model by taking video continuity information in the spatio-temporally transformed videos as self-supervised information;

[0045] an output module, configured to output the video feature extraction model.

[0046] The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above.

[0047] In one possible design, the apparatus includes a memory and a processor. The memory is configured to store one or more computer instructions for supporting the apparatus to perform the corresponding method described above. The processor is configured to execute the computer instructions stored in the memory. The apparatus can further include a communication interface configured to enable the apparatus to communicate with other devices or communication networks.

[0048] In a third aspect, an electronic device is provided, including a memory and a processor. The memory is configured to store one or more computer instructions. The one or more computer instructions are executed by the processor to implement the method in any of the aspects described above.

[0049] In a fourth aspect, a computer-readable storage medium is provided, configured to store computer instructions for the apparatus described above, which includes computer instructions for implementing the method in any of the aspects described above.

[0050] The technical solutions provided by the embodiments of the present disclosure can have the following beneficial effects:

[0051] In the process of training the video feature extraction model, the embodiments of the present disclosure first perform spatio-temporal transformation on the training video in different modes to obtain at least two different spatio-temporally transformed videos, and then take the video continuity information in the spatio-temporally transformed videos as self-supervised learning information to perform self-supervised training on the video feature extraction model. This way can train the video feature extraction model without manual label annotation, and use the video continuity information in the spatio-temporally transformed videos as self-supervised information, so that the video feature extraction model can learn the video features sufficiently, which can reduce the training cost of the video feature extraction model and improve the recognition accuracy of the video feature extraction model.

[0052] It should be understood that the general description above and the detailed description below are only exemplary and explanatory, and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0053] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. In the drawings:

[0054] Figure 1 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown;

[0055] Figure 2 An implementation flowchart illustrating training of a video feature extraction model according to an embodiment of the present disclosure is shown;

[0056] Figure 3 An implementation flowchart illustrating training of a video feature extraction model using the four training tasks described above according to an embodiment of the present disclosure is shown;

[0057] Figure 4 A flowchart illustrating a video processing method according to another embodiment of the present disclosure is shown;

[0058] Figure 5 A flowchart illustrating a model training method according to another embodiment of the present disclosure is shown;

[0059] Figure 6 is a structural schematic diagram of an electronic device suitable for implementing a model training method and / or a video processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so as to be easily implemented by those skilled in the art. Also, portions unrelated to describing the exemplary embodiments are omitted in the accompanying drawings for the sake of clarity.

[0061] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there are features, numbers, steps, actions, components, parts or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, components, parts or combinations thereof exist or are added.

[0062] It should also be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0063] The present inventors find that self-supervised representation learning is widely used in the image field, and the learning effect even exceeds that of supervised learning in the image field. In the video field, self-supervised representation learning has not been widely used, because how to use a large amount of unlabeled video data to enable the video processing model to learn good representation, and which training task can better depict the video representation, have not been explored in the video field. The present inventors also find that the most important feature of a video is the continuity in the time dimension and the space dimension, and how to identify whether the video is continuous in the time dimension and the space dimension is an important task in the self-supervised representation learning process. Therefore, the present disclosure provides a model training method, which quickly obtains different transformation forms of a video by constraining the spatio-temporal transformation, and trains a video feature extraction model based on the continuity information in the transformed video, so that the video feature extraction model can learn model parameters conducive to understanding the video from different spatio-temporal continuity angles.

[0064] The details of the embodiments of the present disclosure will be described in detail below with specific embodiments.

[0065] Figure 1 A flowchart of a model training method according to an embodiment of the present disclosure is shown. As shown in the flowchart, the model training method includes the following steps: Figure 1

[0066] In step S101, a training video is obtained.

[0067] In step S102, the training video is spatio-temporally transformed according to different spatio-temporal transformation modes to obtain at least two different spatio-temporally transformed videos.

[0068] In step S103, the video feature extraction model is trained using the video continuity information in the spatio-temporally transformed video as self-supervised learning information.

[0069] In this embodiment, the training video can be any video collected, and the plurality of training videos can have certain semantic correlation, and the training video does not need to be manually labeled with a label, that is, the training video can not include label information.

[0070] For any training video, different spatio-temporal transformation modes can be used for spatio-temporal transformation. Spatio-temporal transformation can be understood as randomly transforming the pixel points or pixel blocks in the video data in the time dimension and the space dimension. The transformed video obtained after spatio-temporal transformation is different from the original video in that the order of the pixel points or pixel blocks in the video in the time dimension and the space dimension has been randomly changed. The spatio-temporal transformation mode can include multiple modes, such as spatio-temporal transformation modes in the time dimension and spatio-temporal transformation modes in the space dimension. Different transformation modes can obtain different spatio-temporally transformed videos. ​

[0071] In some embodiments, the spatio-temporal transformation mode can be a constrained spatio-temporal transformation mode. The constrained spatio-temporal transformation mode can be understood as a spatio-temporal transformation mode that is constrained on the basis of the spatio-temporal transformation mode, reducing the transformation space. For example, the constrained spatio-temporal transformation mode can be a spatio-temporal transformation mode that specifies the transformation order of the time dimension and the space dimension in advance, such as specifying transforming in the time dimension first and then transforming in the space dimension. In this way, the transformation space of the video can be reduced.

[0072] For each of the plurality of training videos, a spatio-temporal transformation video can be obtained by transforming the training video through a different spatio-temporal transformation mode. Assuming that the spatio-temporal transformation mode includes 8 types, each training video can obtain 8 spatio-temporal transformation videos.

[0073] The video feature extraction model can be a machine self-learning model that can be trained using a self-supervised learning method, for example, a neural network model. The function of the video feature extraction model is to extract a feature vector that can represent the corresponding information in the input video from the input video. In some embodiments, the original training video and the spatio-temporal transformation video (for example, 8 spatio-temporal transformation videos) corresponding to the original training video can be used as the input of the video feature extraction model, and then the first feature vector corresponding to the original training video and the second feature vector corresponding to the spatio-temporal transformation video can be obtained after the video feature extraction model is processed.

[0074] In the training process, since the model parameters of the video feature extraction model have not yet fully learned the continuity information in the video, the first feature vector and the second feature vector extracted from the input video cannot accurately represent the corresponding information in the input video. In order to enable the video feature extraction model to accurately understand the video content and then use an accurate feature vector to represent the corresponding information in the video, the video feature extraction model is trained in a self-supervised learning manner. In the training process, the video continuity information in the spatio-temporal transformation video is used as the self-supervised information to train the video feature extraction model.

[0075] In some embodiments, real video continuity information of different aspects in the spatio-temporal transformed video can be obtained, and then different training tasks can be utilized to train the video feature extraction model by taking the obtained real video continuity information as self-supervised information. The video continuity information can be understood as video information that remains continuous in the spatio-temporal transformed video after spatio-temporal transformation of the original training video. The video continuity information can include, but is not limited to, related information of a pixel block (e.g., a continuous pixel block) that remains consistent between the spatio-temporal transformed video and the original training video. The related information can include, but is not limited to, the position, volume, distance (e.g., Hamming distance) of the pixel block, and can also include information such as the spatio-temporal transformation mode corresponding to the pixel block, whether it is the most similar to the original pixel block in the original training video, and the like. Different angles can include, but are not limited to, related information of a pixel block in the spatio-temporal transformed video, a volume ratio of a pixel block in the spatio-temporal transformed video, Hamming distance of a pixel block in the time dimension and the space dimension, a spatio-temporal transformation mode corresponding to the pixel block, whether it is the most similar to the original pixel block in the original training video, and the like. One training task corresponds to one angle, and the video feature extraction model is trained from the angle. In each training task, the first feature vector and the second feature vector extracted from the video feature extraction model are used to predict the video continuity information corresponding to the angle, and then the model parameters of the video feature extraction model are adjusted by comparing the difference between the predicted video continuity information and the real video continuity information.

[0076] In the process of training the video feature extraction model, the embodiments of the present disclosure first perform spatio-temporal transformation on the training video in different modes to obtain at least two different spatio-temporal transformed videos, and then perform self-supervised training on the video feature extraction model by taking the video continuity information in the spatio-temporal transformed video as self-supervised learning information. This way can train the video feature extraction model without manual label annotation, and use the video continuity information in the spatio-temporal transformed video as self-supervised information, so that the video feature extraction model can learn sufficient video features, which can reduce the training cost of the video feature extraction model and improve the recognition accuracy of the video feature extraction model.

[0077] In an optional implementation of the present embodiment, step S102, i.e., the step of performing spatio-temporal transformation on the training video according to different spatio-temporal transformation modes to obtain at least two different spatio-temporal transformed videos, further includes the following steps:

[0078] Obtaining a preset transformation order between the time dimension and the space dimension;

[0079] Transforming the training video in the time dimension and the space dimension according to the preset transformation order to obtain the spatio-temporal transformed video.

[0080] In the optional implementation, the spatio-temporal transformation mode can correspond to a constraint spatio-temporal transformation mode, that is, the original training video is spatio-temporally transformed by using different constraint spatio-temporal transformation modes. Different constraint spatio-temporal transformation modes correspond to different spatio-temporal transformation modes under the same constraint. In some embodiments, the constraint spatio-temporal transformation mode can be a preset transformation order between the time dimension and the space dimension as a constraint. When spatio-temporally transforming the original training video, the constraint to be followed is the preset transformation order. For example, the preset transformation order is that the time dimension precedes the space dimension. When spatio-temporally transforming, all spatio-temporal transformation modes follow the principle of first transforming in the time dimension and then transforming in the space dimension, that is, after transforming the original training video in the time dimension, a corresponding spatio-temporally transformed video is obtained by further transforming in the space dimension on the basis of the transformation. It should be noted that in the preset transformation order, the space dimension includes two dimensions of H and W, and thus the order between the two dimensions of H and W can also be defined in the preset transformation order.

[0081] It should be noted that the preset transformation order can be a pre-defined order range, and a plurality of different transformation sub-orders can be enumerated in the order range. Therefore, in the process of spatio-temporally transforming, the preset transformation order can be obtained first, and then a plurality of different transformation sub-orders in the preset transformation order range can be determined according to the preset transformation order. The original training video can be spatio-temporally transformed according to all transformation sub-orders or a part of the transformation sub-orders, and a plurality of different spatio-temporally transformed videos can be obtained. It should be noted that one transformation sub-order can correspond to one spatio-temporal transformation mode. Of course, it can also be understood that in other embodiments, the preset transformation order can also be a fixed transformation sub-order. A plurality of different preset transformation orders can be pre-defined, and each preset transformation order corresponds to one spatio-temporal transformation mode. After spatio-temporally transforming the original training video by using one preset transformation order, a corresponding spatio-temporally transformed video can be obtained.

[0082] In an optional implementation of the present embodiment, step S102, that is, the step of spatio-temporally transforming the training video according to different spatio-temporal transformation modes to obtain at least two different spatio-temporally transformed videos, further includes the following steps:

[0083] dividing the video frames in the training video into pixel blocks;

[0084] spatio-temporally transforming the pixel blocks to obtain the spatio-temporally transformed video under the spatio-temporal transformation mode.

[0085] In the optional implementation, before the spatio-temporal transformation is performed on the training video, the video frames can be divided into pixel blocks, each pixel block can have a fixed size, and the spatio-temporal transformation is performed in units of pixel blocks. That is, in this embodiment, when the training video is transformed in space, the transformation is performed in units of pixel blocks rather than in units of pixels. In this way, the transformation space of the spatio-temporal transformation can be further limited.

[0086] The following illustrates the constrained spatio-temporal transformation mode.

[0087] Suppose that an input video is represented as V, has T frames, and each frame has a size of H*W pixels. After the constrained spatio-temporal transformation is performed on the video, a spatio-temporal transformed video V' is obtained, which has the same size as the input video but has different pixel positions. represents a set of 8 spatio-temporal transformed videos, which are obtained by using 8 different spatio-temporal transformation modes. The following illustrates the 8 spatio-temporal transformation modes in detail.

[0088] The input video V and the spatio-temporal transformed video both contain C*T*H*W pixels (C is the RGB three channels). Suppose that a preset transformation order of the constrained spatio-temporal transformation in three dimensions is T>H>W.

[0089] Suppose that T*H*W=16*112*112, and the constrained spatio-temporal transformation is performed in units of pixel blocks rather than in units of pixels, and the size of the pixel block is set to 1*4*4. Then the actual transformation space is:

[0090]

[0091] In subsequent operations, the transformation space is also 16*28*28, which can be represented as B T *B H *B W = 16*28*28. For example, the original order of the pixel blocks in the T dimension is 1, 2, 3,..., 16, and after the constrained spatio-temporal transformation, it becomes 4, 5, 6, 7, 8, 1, 2, 3, 15, 16, 9, 10, 11, 12, 13, 14. Then the last six pixel blocks (i.e., the pixel blocks with subscripts 9, 10, 11, 12, 13, and 14) in the T dimension are the pixel blocks corresponding to the maximum continuous subscripts, i.e., the maximum continuous pixel blocks, and the first five pixel blocks (i.e., the pixel blocks with subscripts 4, 5, 6, 7, and 8) in the T dimension are the pixel blocks corresponding to the subscripts with the second largest continuous subscripts, i.e., the second largest continuous pixel blocks. According to the relative order of the maximum continuous subscripts and the second largest continuous subscripts being positive and negative, two groups of spatio-temporal transformed videos in the T dimension can be generated, and the order of the pixel block subscripts after the transformation can be represented as:

[0092] T P= 4, 5, 6, 7, 8, 1, 2, 3, 15, 16, 9, 10, 11, 12, 13, 14

[0093] T N = 9, 10, 11, 12, 13, 14, 1, 2, 3, 15, 16, 4, 5, 6, 7, 8

[0094] Similarly, H P ,H N ,W P ,W N , 8 groups of transformations are generated by combining the transformations in the three dimensions respectively, and the 8 groups of transformations correspond to 8 different spatio-temporal transformation modes, which can be represented as:

[0095]

[0096] The output video, i.e., the spatio-temporal transformation video, can be represented as:

[0097]

[0098] In an optional implementation of the embodiment, the step S102, i.e., the step of performing spatio-temporal transformation on the training video according to different spatio-temporal transformation modes to obtain at least two different spatio-temporal transformation videos, further includes the following steps:

[0099] obtaining feature vectors corresponding to the training video and the spatio-temporal transformation video by using the video feature extraction model;

[0100] obtaining a prediction result of the video continuity information from the feature vectors by using the at least one training task;

[0101] training the video feature extraction model according to the prediction result and a real result of the video continuity information, wherein the real result is determined by using the spatio-temporal transformation mode corresponding to the spatio-temporal transformation video.

[0102] In the optional implementation, the training video and the spatio-temporal transformation video are processed by using the video feature extraction model to obtain a first feature vector for representing the training video and a second feature vector for representing the spatio-temporal transformation video.

[0103] In some embodiments, the video feature extraction model can be a 3D convolutional neural network (ResNet3D), and the input is the training video and the spatio-temporal transformation video, and the output is the first feature vector and the second feature vector. In some embodiments, the features can be extracted by selecting one frame of image at equal intervals, for example, one frame of image can be selected every 4 frames for feature extraction.

[0104] It can be understood that the first feature vector and the second feature vector extracted from the training video and the spatio-temporal transformed video by the video feature extraction model before the training of the video feature extraction model is completed may not be able to correctly describe the information in the training video and the spatio-temporal transformed video, such as video continuity information. Therefore, the video continuity information in the spatio-temporal transformed video can be predicted based on the first feature vector and the second feature vector through one or more training tasks, and the corresponding prediction result may have a large difference from the true result. Therefore, the role of the training task is to train the video feature extraction model through the difference between the true result and the prediction result, that is, to adjust the model parameters in the video feature extraction model according to the difference between the true result and the prediction result in each training process. After multiple rounds of training, the first feature vector and the second feature vector obtained by the video feature extraction model after parameter adjustment can correctly represent the information in the input video such as video continuity information, so that the video feature extraction model can correctly understand the input video.

[0105] Different training tasks predict video continuity information from different angles, that is, predict different video continuity information. The true result of the video continuity information can be determined based on the spatio-temporal transformation mode corresponding to the spatio-temporal transformed video. For example, when the video continuity information is the position information of the largest continuous block in the spatio-temporal transformed video, the true result corresponding to the video continuity information can be recorded according to the spatio-temporal transformation mode used when the spatio-temporal transformation is performed. That is, when the training video is spatio-temporally transformed, the true result corresponding to the video continuity information can be directly obtained when the video feature extraction model is trained using different training tasks. During the entire training process, there is no need for manual participation in labeling, which saves the labor cost and improves the model training efficiency.

[0106] Figure 2 An implementation flowchart for training a video feature extraction model according to an embodiment of the present disclosure is shown. As shown in Figure 2 The training process of the video feature extraction model can be divided into the following stages:

[0107] (I) Spatio-temporal transformation stage

[0108] The spatio-temporal transformation of the training video V can obtain a plurality of spatio-temporally transformed videos and the video continuity information in each spatio-temporally transformed video. The spatio-temporal transformation mode corresponding to each spatio-temporally transformed video is different, so different spatio-temporally transformed videos are different. When the training spatio-temporal transformation is performed, the video continuity information corresponding to each spatio-temporally transformed video can also be generated according to the spatio-temporal transformation mode, such as the position and size of the continuous pixel block in the spatio-temporally transformed video.

[0109] (II) Video feature extraction stage

[0110] training videos V and a plurality of spatio-temporal transformed videos As input videos into the video feature extraction model, the video feature extraction model respectively for training videos V and a plurality of spatio-temporal transformed videos output the first feature vector f(V) and the second feature vector

[0111] (Three) training phase

[0112] The video continuity information obtained in the above spatio-temporal transformation phase is used as a real result by at least one training task, and the first feature vector f(V) and the second feature vector The prediction result of the video continuity information is obtained, and by comparing the difference between the real result and the prediction result, the model parameters of the video feature extraction model are adjusted by using back propagation.

[0113] After multiple rounds of training of multiple training videos, the video feature extraction model with converged model parameters can be finally obtained.

[0114] In an optional implementation of the embodiment, the at least one training task includes at least one of the following:

[0115] The first training task is to predict the continuous pixel block information in the spatio-temporal transformed video based on the feature vector;

[0116] The second training task is to predict the volume ratio of the continuous pixel block in the spatio-temporal transformed video relative to the spatio-temporal transformed video, and the Hamming distance of the continuous pixel block in the time dimension and the space dimension relative to the training video based on the feature vector;

[0117] The third training task is to predict the spatio-temporal transformation mode corresponding to the spatio-temporal transformed video based on the feature vector;

[0118] The fourth training task is to predict one of the plurality of spatio-temporal transformed videos that is most similar to the training video based on the feature vector.

[0119] In the optional implementation, the training task can include one or a combination of the first training task, the second training task, the third training task and the fourth training task.

[0120] The above training tasks are described in detail as follows.

[0121] (One) first training task

[0122] The first training task can be used to detect the information of the continuous pixel blocks in the spatio-temporal transformed video. For example, the first training task can be used to detect the information of the maximum continuous pixel block in the spatio-temporal transformed video (such as the position information of the maximum continuous pixel block in the spatio-temporal transformed video), the continuous pixel block being a pixel block that, after spatio-temporal transformation, still has the same semantic information as the training video before transformation in the time dimension and the space dimension, without any relative transformation. The maximum continuous pixel block can be the continuous pixel block with the largest number of pixel blocks in the spatio-temporal transformed video.

[0123] When the training video is spatio-temporally transformed, the positions of the maximum continuous pixel block and the second maximum continuous pixel block in each spatio-temporally transformed video can be obtained at the same time, and a mask matrix of the positions can also be obtained according to the two positions, the mask matrix having the same size as the size of the pixel block, for example, being a matrix with a size of 16*28*28. In the mask matrix, the elements at positions other than the positions of the maximum continuous pixel block and the second maximum continuous pixel block can be set to 0. The element values at the positions of the maximum continuous pixel block and the second maximum continuous pixel block can be in a Gaussian distribution with the center of the pixel block as the origin. The spatio-temporally transformed video corresponding to each spatio-temporal transformation mode can obtain a corresponding mask matrix, which can be represented as:

[0124]

[0125] For each spatio-temporally transformed video, the first feature vector f(V) and the second feature vector f(V) can be fused by using a non-local function, for example. So that the spatio-temporally transformed video obtains the attention of the original training video:

[0126]

[0127] where NL(·) represents the non-local function, and the fused feature vector is up-sampled to obtain a feature map with the same size as the mask matrix Regression of the feature map can obtain the prediction result of the mask matrix .

[0128] (II) Second training task

[0129] The second training task can be used to calculate the volume ratio of the continuous pixel block in the spatio-temporal transformed video, and the Hamming distance of the continuous pixel block in the time dimension and the space dimension relative to the original training video. For example, the second training task can calculate the ratio of the sum of the volumes of the largest continuous pixel block and the second largest continuous pixel block to the volume in the spatio-temporal transformed video where the largest continuous pixel block is located, and can also calculate the Hamming distance of the largest continuous pixel block and / or the second largest continuous pixel block in the time dimension and the space dimension relative to the original training video. In some embodiments, the three Hamming distances calculated in the time dimension and the space dimension (corresponding to the T, H and W dimensions respectively) can also be converted into a standard Hamming distance, that is, the ratio of the Hamming distance to the maximum possible Hamming distance.

[0130] When performing spatio-temporal transformation on the training video, the ratio l of the volume of the large continuous pixel block to the total volume of the spatio-temporal transformed video can be obtained according to the volumes of the largest continuous pixel block and the second largest continuous pixel block in each spatio-temporal transformed video (such as 6*8*8, 5*8*8). ld For example, it can be expressed as follows:

[0131]

[0132] Meanwhile, the second training task can also calculate the standard Hamming distance in the three dimensions according to the Hamming distance of the largest continuous pixel block in each dimension (T, H, W).

[0133] For example, the pixel block transformation order of the spatio-temporal transformed video in the T dimension is: 9, 10, 11, 12, 13, 14, 1, 2, 3, 15, 16, 4, 5, 6, 7, 8. The pixel block order obtained by simplifying the continuous pixel blocks in it as a whole can be expressed as: 3, 1, 4, 2 (where 3 represents the continuous pixel block "9, 10, 11, 12, 13, 14", 1 represents the continuous pixel block "1, 2, 3", 4 represents "15, 16", and 2 represents "4, 5, 6, 7, 8"). According to the calculation method of the Hamming distance, it can be determined that the Hamming distance of the above transformation in the time dimension T is 4, and the standard Hamming distance is (the maximum possible Hamming distance is 6).

[0134] The second training task can use a fully connected layer to process the second feature vector , and then can predict the above volume ratio l ld and the prediction result of the standard Hamming distance .

[0135] (III) Third training task

[0136] The third training task can be used to predict the spatio-temporal transformation mode corresponding to the spatio-temporal transformation video. The third training task can be understood as a classification task for the spatio-temporal transformation video, which is used to classify the spatio-temporal transformation video into the corresponding spatio-temporal transformation mode. For example, in the case of using the eight spatio-temporal transformation modes described in the foregoing example, the third training task can predict the classification result corresponding to the spatio-temporal transformation video based on the second feature vector corresponding to the spatio-temporal transformation video, that is, predict the probability that the spatio-temporal transformation video belongs to the eight spatio-temporal transformation modes, and the maximum probability can be determined as the spatio-temporal transformation mode corresponding to the spatio-temporal transformation video.

[0137] When performing spatio-temporal transformation on the training video, different spatio-temporal transformation videos are generated according to different spatio-temporal transformation modes. Since different spatio-temporal transformation modes follow different spatio-temporal transformation ways based on the same constraint condition, the generated spatio-temporal transformation videos are completely different but related. Taking the eight spatio-temporal transformation modes in the foregoing example as an example, for the spatio-temporal transformation modes {T P ,H P ,W P} and {T P ,H P ,W N}, they only differ in the order of the dimension W, and the transformation of the other two dimensions T and H is completely consistent, especially {T P ,H P ,W P}, since the relative order of the two largest continuous pixel blocks in the three dimensions is consistent with the relative order in the original training video, that is, most of the pixel points in the spatio-temporal transformation video can be normally understood. Therefore, the degree to which the spatio-temporal transformation videos obtained by different spatio-temporal transformation modes can be understood by the video feature extraction mode is inconsistent. Therefore, the video feature extraction model can be trained through the third training task of the spatio-temporal transformation mode.

[0138] The third training task can use a fully connected layer to process the second feature vector , and then can predict the prediction result of the spatio-temporal transformation mode corresponding to the current spatio-temporal transformation video according to the processing result, that is, the classification result.

[0139] (Four) Fourth training task

[0140] The fourth training task can predict one of the plurality of spatio-temporal transformation videos that is most similar to the original training video. For example, the fourth training task can be used to predict whether the current spatio-temporal transformation video is the transformation video that is most similar to the original training video among the plurality of different spatio-temporal transformation videos. The fourth training task can be understood as a similarity prediction task, and the video feature extraction model is trained through comparison between the spatio-temporal transformation video and the original training video.

[0141] Taking the eight different spatiotemporal transformation modes mentioned above as examples, when performing spatiotemporal transformation on the training video, {T P H P W P This spatiotemporal transformation mode is the most special because it preserves the understanding of the vast majority of pixels (i.e., it preserves the consistency of the vast majority of pixels with the original training video). Therefore, the spatiotemporal transformation video corresponding to this mode is most similar to the original training video. The fourth training task can predict the transformation video that is most similar to the original training video from the eight spatiotemporal transformation videos, that is, using the spatiotemporal transformation mode {T}. P H P W P The resulting spatiotemporal transformation video.

[0142] In the fourth training task, it is not suitable to use a non-local function to fuse the first feature vector corresponding to the original training video and the second feature vector corresponding to the spatiotemporal transformation video. However, the second feature vector of each spatiotemporal transformation video can be fused. The second feature vector, which is most similar to the first feature vector f(V) of the original training video, is selected by comparing its similarity. The spatiotemporal transformation video corresponding to the most similar second feature vector is then predicted as {T}. P H P W P Transformed video in time-space transformation mode.

[0143] The fourth training task can use the InfoNCE Loss method to train the video feature extraction model.

[0144] As described above, the first training task can be understood as the task of detecting the positional information of continuous pixel blocks in the spatiotemporal transformed video; the second training task can be understood as the task of detecting the transformation amplitude of the spatiotemporal transformed video relative to the original training video. The transformation amplitude can be represented by the volume ratio of continuous pixel blocks in the spatiotemporal transformed video and the transformed Hamming distance. That is, the third training task is the task of measuring the continuity information in the spatiotemporal transformed video; the fourth training task can be understood as the task of classifying spatiotemporal transformation patterns; and the fifth training task can be understood as the task of comparative learning with the original training video.

[0145] In an optional implementation of this embodiment, the method further includes obtaining the true result by at least one of the following methods:

[0146] Determine the position information of the largest and second largest consecutive image block pixels in the spatiotemporal transformed video, and determine the position information as the true result;

[0147] determining a volume ratio between a sum of volumes of a largest continuous pixel block and a second largest continuous pixel block in the spatio-temporal transformed video and a total volume of the spatio-temporal transformed video, and determining a standard Hamming distance of the largest continuous pixel block in a time dimension and a space dimension, and determining the volume ratio and the standard Hamming distance as the ground truth;

[0148] determining the spatio-temporal transformation mode corresponding to the spatio-temporal transformed video as the ground truth;

[0149] determining the spatio-temporal transformed video corresponding to a preset spatio-temporal transformation mode as the ground truth.

[0150] In the optional implementation, when performing spatio-temporal transformation, the ground truth required in at least one training task can be determined in advance according to the spatio-temporal transformation mode. For example, when the training task adopts the first training task described above, the position information of the largest continuous pixel block in the spatio-temporal transformed video can be determined through the spatio-temporal transformation mode, and the position information is determined as the ground truth, and the model parameters of the video feature extraction model are adjusted by comparing the difference between the ground truth and the predicted result; when the training task adopts the second training task described above, the volume of the largest continuous pixel block and the sum of the volumes of the second largest continuous pixel blocks can be determined through the spatio-temporal transformation mode, and then the volume ratio can be obtained according to the sum of the volumes and the total volume of the spatio-temporal transformed video, and the standard Hamming distance of the largest continuous pixel block in the time dimension and the space dimension can also be determined through the spatio-temporal transformation mode, and the volume ratio and the standard Hamming distance are determined as the ground truth required by the second training task; when the training task adopts the third training task, the spatio-temporal transformation mode corresponding to the spatio-temporal transformed video can be directly determined as the ground truth of the third training task; when the training task adopts the fourth training task, the spatio-temporal transformed video corresponding to a preset spatio-temporal transformation mode can be determined as the transformed video most similar to the original training video, and the transformed video is used as the ground truth required by the fourth training task. The preset spatio-temporal transformation mode can be a mode in multiple spatio-temporal transformation modes, which can keep the original pixel information unchanged to the greatest extent compared with the original training video, for example, in the eight spatio-temporal transformation modes described above, {T P ,H P ,W P} can be set as the preset spatio-temporal transformation mode.

[0151] Figure 3 An implementation flowchart for training the video feature extraction model by using the four training tasks described above according to an embodiment of the present disclosure is shown. As shown in FIG. 2, the implementation flowchart includes the following steps: Figure 3As shown, the rules for constraining the spatio-temporal transformation mode are set as follows: the preset transformation order is T > H > W, and for each dimension, the manner of random spatio-temporal transformation can be referred to as the first spatio-temporal transformation in the dimension, and the manner of exchanging the positions of the largest continuous pixel block and the second largest continuous pixel block in the spatio-temporal transformation video obtained by the first spatio-temporal transformation can be referred to as the second spatio-temporal transformation in the dimension. By combining the above-mentioned first spatio-temporal transformation and the second spatio-temporal transformation obtained in each dimension in the above-mentioned preset transformation order, 8 spatio-temporal transformation modes can be determined, which are represented as follows:

[0152]

[0153] wherein T P , H P , and W P represent the first spatio-temporal transformation in the three dimensions, respectively, T N , H N , and W N represent the second spatio-temporal transformation in the three dimensions, respectively.

[0154] After the training video is subjected to the spatio-temporal transformation of the above-mentioned 8 spatio-temporal transformation modes, 8 spatio-temporal transformation videos can be obtained, and for each spatio-temporal transformation video, the true results corresponding to the four training tasks can be recorded, which are as follows: (1) the position information of the largest continuous pixel block and the second largest continuous pixel block, which is represented by a Mask matrix, wherein the element values at the positions of the largest continuous pixel block and the second largest continuous pixel block in the Mask matrix are Gaussian distributed with the center of the pixel block as the origin, and the element values at other positions are 0, and the Mask matrix is taken as the first true result; (2) the ratio of the sum of the volumes of the largest continuous pixel block and the second largest continuous pixel block to the volume of the spatio-temporal transformation video, and the standard Hamming distance of the largest continuous pixel block, and the volume ratio and the standard Hamming distance are determined as the second true result; (3) the spatio-temporal transformation mode corresponding to the spatio-temporal transformation video, and the correspondence between each spatio-temporal transformation video and the spatio-temporal transformation mode is determined as the third true result; (4) the spatio-temporal transformation video corresponding to {T P , H P , and W P is determined as the spatio-temporal transformation video most similar to the training video, that is, the fourth true result.

[0155] The training video and the 8 spatio-temporal transformation videos are input into a video feature extraction model to obtain a first feature vector f(V) and a second feature vector

[0156] The first training task uses the non-local function to combine the first feature vector f(V) and the second feature vector After fusion, upsampling is performed to obtain a feature map with the same size as the Mask matrix, and the feature map is regressed to obtain a first prediction result of the Mask matrix.

[0157] The second training task processes the second feature vectors through a fully connected layer , and obtains a second prediction result corresponding to the ratio and the standard Hamming distance of the volume according to the obtained processing result.

[0158] The third training task processes the second feature vectors through a fully connected layer , and obtains a classification result of the spatio-temporal transformation mode corresponding to each spatio-temporal transformation video, that is, a third prediction result, according to the obtained processing result.

[0159] The fourth training task compares each second feature vector with the first feature vector respectively, determines the second feature vector most similar to the first feature vector, and determines the spatio-temporal transformation video corresponding to the second feature vector as the third prediction result.

[0160] The first loss between the first real result and the first prediction result, the second loss between the second real result and the second prediction result, the third loss between the third real result and the third prediction result, and the fourth loss between the fourth real result and the fourth prediction result are fitted respectively, and the model parameters of the video feature extraction model are adjusted through the back propagation mode by combining the first loss, the second loss, the third loss and the fourth loss.

[0161] After the training end condition is reached, a video feature extraction model capable of correctly understanding semantic information in a video can be obtained, and the video feature extraction model can be used for feature representation of an input video.

[0162] Figure 4 A flowchart of a video processing method according to another embodiment of the present disclosure is shown. As shown in Figure 4 , the video processing method includes the following steps:

[0163] In step S401, an input video is obtained.

[0164] In step S402, the input video is represented as a feature vector by using a video feature extraction model; the video feature extraction model is trained by using the above method.

[0165] In this embodiment, the input video can be any video, and the video feature extraction model can be a model trained by using the above method. Figure 1The model trained by the method in the embodiment and related embodiments can correctly understand semantic information in a video and can represent an input video in the form of a feature vector, which can be used for further processing of the input video, for example, comparing the similarity between two videos by using the feature vector obtained by the video feature extraction model. Details of training of the video feature extraction model can be found in the description of the video feature extraction model above. Figure 1 The video feature extraction model trained by the method described in the embodiment and related embodiments can correctly understand an input video semantically and represent the input video in the form of a feature vector, which can achieve more accurate results in subsequent processing of the video.

[0166] Figure 5 A flowchart of a model training method according to an embodiment of the present disclosure is shown. As shown in the figure, Figure 5 The model training method includes the following steps:

[0167] In step S501, a training video is obtained.

[0168] In step S502, a preset service interface is called to perform spatiotemporal transformation on the training video according to different spatiotemporal transformation modes, to obtain at least two different spatiotemporal transformation videos, and to train the video feature extraction model by using video continuity information in the spatiotemporal transformation videos as self-supervised information.

[0169] In step S503, the video feature extraction model is output.

[0170] In this embodiment, the video processing method can be executed in the cloud. The preset service interface can be pre-deployed in the cloud. The preset service interface can be a Saas (Software-as-a-service) interface. The demander can obtain the right to use the preset service interface in advance, and can train the video feature extraction model by calling the preset service interface when needed.

[0171] In this embodiment, the demander can obtain a set of training videos related semantically in advance and provide the set of training videos to the preset service interface.

[0172] After the preset service interface receives the training video, for any training video, different spatio-temporal transformation modes can be used for spatio-temporal transformation. The spatio-temporal transformation can be understood as random transformation of pixel points or pixel blocks in the video data in the time dimension and the space dimension. The transformed video obtained after spatio-temporal transformation is different from the original video in that the order of pixel points or pixel blocks in the video in the time dimension and the space dimension has been randomly changed. The spatio-temporal transformation mode can include multiple modes, for example, spatio-temporal transformation modes in the time dimension and spatio-temporal transformation modes in the space dimension. Different transformation modes can obtain different spatio-temporal transformation videos.

[0173] In some embodiments, the spatio-temporal transformation mode can be a constrained spatio-temporal transformation mode. The constrained spatio-temporal transformation mode can be understood as a spatio-temporal transformation mode that is constrained on the basis of the spatio-temporal transformation mode and reduces the transformation space. For example, the constrained spatio-temporal transformation mode can be a spatio-temporal transformation mode that specifies the transformation order of the time dimension and the space dimension in advance, such as specifying transformation in the time dimension first and then transformation in the space dimension. In this way, the transformation space of the video can be reduced.

[0174] For each of the plurality of training videos, a spatio-temporal transformation video can be obtained by using different spatio-temporal transformation modes. If there are 8 spatio-temporal transformation modes, 8 spatio-temporal transformation videos can be obtained for each training video.

[0175] The video feature extraction model can be a machine self-learning model that can be trained using a self-supervised learning method, for example, a neural network model. The function of the video feature extraction model is to extract a feature vector that can represent the corresponding information in the input video from the input video. In some embodiments, the original training video and the spatio-temporal transformation video (for example, 8 spatio-temporal transformation videos) corresponding to the original training video can be used as the input of the video feature extraction model, and then the first feature vector corresponding to the original training video and the second feature vector corresponding to the spatio-temporal transformation video can be obtained after the video feature extraction model is processed.

[0176] During the training process, the model parameters of the video feature extraction model have not yet completely learned the continuity information in the video, so the first feature vector and the second feature vector extracted from the input video cannot accurately represent the corresponding information in the input video. In order to enable the video feature extraction model to accurately understand the video content and then use the accurate feature vector to represent the corresponding information in the video, the video feature extraction model is trained in a self-supervised learning manner. During the training process, the video continuity information in the spatio-temporal transformation video is used as the self-supervised information to train the video feature extraction model.

[0177] In some embodiments, real video continuity information of different aspects in the spatio-temporal transformed video can be obtained, and then different training tasks can be utilized to train the video feature extraction model by taking the obtained real video continuity information as self-supervised information. The video continuity information can be understood as video information that remains continuous in the spatio-temporal transformed video after spatio-temporal transformation is performed on the original training video. The video continuity information can include, but is not limited to, related information of a pixel block (for example, a continuous pixel block) that remains consistent between the spatio-temporal transformed video and the original training video. The related information can include, but is not limited to, the position, volume, distance (for example, Hamming distance) of the pixel block, and can also include information such as the spatio-temporal transformation mode corresponding to the pixel block and whether the pixel block is most similar to the original pixel block in the original training video. Different angles can include, but are not limited to, the related information of the pixel block in the spatio-temporal transformed video, the volume ratio of the pixel block in the spatio-temporal transformed video, the Hamming distance of the pixel block in the time dimension and the space dimension, the spatio-temporal transformation mode corresponding to the pixel block, and whether the pixel block is most similar to the original pixel block in the original training video. One training task corresponds to one angle, and the video feature extraction model is trained from the angle. In each training task, the first feature vector and the second feature vector extracted from the video feature extraction model are used to predict the video continuity information corresponding to the angle, and then the model parameters of the video feature extraction model are adjusted by comparing the difference between the predicted video continuity information and the real video continuity information.

[0178] In the process of training the video feature extraction model, the embodiments of the present disclosure first perform spatio-temporal transformation on the training video in different modes to obtain multiple different spatio-temporal transformed videos, and then perform self-supervised training on the video feature extraction model by taking the video continuity information in the spatio-temporal transformed video as self-supervised learning information. This way can train the video feature extraction model without manual label annotation, and use the video continuity information in the spatio-temporal transformed video as self-supervised information, so that the video feature extraction model can learn sufficient video features, which can reduce the training cost of the video feature extraction model and improve the recognition accuracy of the video feature extraction model.

[0179] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure.

[0180] The model training apparatus according to an embodiment of the present disclosure can be realized as part or all of an electronic device through software, hardware, or a combination of both. The model training apparatus includes:

[0181] The first obtaining module is configured to obtain a training video.

[0182] a transformation module configured to perform spatio-temporal transformation on the training video according to different spatio-temporal transformation modes to obtain at least two different spatio-temporal transformation videos;

[0183] a training module configured to train the video feature extraction model by taking video continuity information in the spatio-temporal transformation video as self-supervised information.

[0184] In an optional implementation of the embodiment, the transformation module comprises:

[0185] a first obtaining sub-module configured to obtain a preset transformation sequence between a time dimension and a space dimension;

[0186] a first transformation sub-module configured to transform the training video in the time dimension and the space dimension according to the preset transformation sequence to obtain the spatio-temporal transformation video.

[0187] In an optional implementation of the embodiment, the transformation module comprises:

[0188] a division sub-module configured to divide video frames in the training video into pixel blocks;

[0189] a second transformation sub-module configured to perform spatio-temporal transformation on the pixel blocks in the spatio-temporal transformation mode to obtain the spatio-temporal transformation video.

[0190] In an optional implementation of the embodiment, the training module comprises:

[0191] a second obtaining sub-module configured to obtain feature vectors corresponding to the training video and the spatio-temporal transformation video by using the video feature extraction model;

[0192] a third obtaining sub-module configured to obtain a prediction result of the video continuity information from the feature vectors by using the at least one training task;

[0193] a training sub-module configured to train the video feature extraction model according to the prediction result and a real result of the video continuity information, wherein the real result is determined by using the spatio-temporal transformation mode corresponding to the spatio-temporal transformation video.

[0194] In an optional implementation of the embodiment, the at least one training task comprises at least one of the following:

[0195] a first training task for predicting continuous pixel block information in the spatio-temporal transformation video based on the feature vectors;

[0196] a second training task of predicting, based on the feature vector, a volume ratio of a continuous pixel block in the spatio-temporal transformed video relative to a volume of the spatio-temporal transformed video, and a Hamming distance of the continuous pixel block in the spatio-temporal transformed video relative to the training video in a time dimension and a space dimension;

[0197] a third training task of predicting, based on the feature vector, a spatio-temporal transformation mode corresponding to the spatio-temporal transformed video;

[0198] a fourth training task of predicting, based on the feature vector, one of the plurality of spatio-temporal transformed videos that is most similar to the training video.

[0199] In an optional implementation of the embodiment, the device further comprises obtaining the real result by at least one of the following modules:

[0200] a first determining sub-module configured to determine position information of a largest continuous pixel block and a second largest continuous pixel block in the spatio-temporal transformed video, and determine the position information as the real result;

[0201] a second determining sub-module configured to determine a volume ratio between a sum of volumes of the largest continuous pixel block and the second largest continuous pixel block in the spatio-temporal transformed video and a total volume of the spatio-temporal transformed video, and determine a standard Hamming distance of the largest continuous pixel block in a time dimension and a space dimension, and determine the volume ratio and the standard Hamming distance as the real result;

[0202] a third determining sub-module configured to determine the spatio-temporal transformation mode corresponding to the spatio-temporal transformed video as the real result;

[0203] a fourth determining sub-module configured to determine the spatio-temporal transformed video corresponding to a preset spatio-temporal transformation mode as the real result.

[0204] The model training device in the embodiment corresponds to the model training method in the embodiment shown in Figure 1 and related embodiments, and specific details can be referred to the description of the model training method in the embodiment shown in Figure 1 and related embodiments, which will not be described herein again.

[0205] According to the video processing device of another embodiment of the present disclosure, the device can be realized by software, hardware or a combination of both as part or all of an electronic device. The video processing device comprises:

[0206] a second obtaining module configured to obtain an input video;

[0207] a feature representation module configured to represent the input video as a feature vector by using a video feature extraction model; the video feature extraction model is trained by using the video processing device.

[0208] The video processing device in this embodiment corresponds to the video processing method in the embodiments shown in the drawings and related embodiments, and specific details can be referred to the description of the embodiments shown in the drawings and related embodiments above, which will not be repeated here. Figure 4 The video processing method in this embodiment corresponds to the video processing device in the embodiments shown in the drawings and related embodiments, and specific details can be referred to the description of the embodiments shown in the drawings and related embodiments above, which will not be repeated here. Figure 4 The video processing method in this embodiment corresponds to the video processing device in the embodiments shown in the drawings and related embodiments, and specific details can be referred to the description of the embodiments shown in the drawings and related embodiments above, which will not be repeated here.

[0209] According to another embodiment of the present disclosure, the model training device can be realized by software, hardware or a combination of both as part of or all of the electronic device. The model training device comprises:

[0210] The third acquisition module is configured to acquire a training video;

[0211] The calling module is configured to call a preset service interface, so as to perform spatio-temporal transformation on the training video according to different spatio-temporal transformation modes to obtain at least two different spatio-temporal transformation videos, and train the video feature extraction model by taking video continuity information in the spatio-temporal transformation video as self-supervised information;

[0212] The output module is configured to output the video feature extraction model.

[0213] The video processing device in this embodiment corresponds to the video processing method in the embodiments shown in the drawings and related embodiments, and specific details can be referred to the description of the embodiments shown in the drawings and related embodiments above, which will not be repeated here. Figure 5 The video processing method in this embodiment corresponds to the video processing device in the embodiments shown in the drawings and related embodiments, and specific details can be referred to the description of the embodiments shown in the drawings and related embodiments above, which will not be repeated here. Figure 5 The video processing method in this embodiment corresponds to the video processing device in the embodiments shown in the drawings and related embodiments, and specific details can be referred to the description of the embodiments shown in the drawings and related embodiments above, which will not be repeated here.

[0214] Figure 6 is a structural schematic diagram of an electronic device suitable for implementing the model training method and / or video processing method according to the embodiments of the present disclosure.

[0215] As shown in Figure 6 The electronic device 600 includes a processing unit 601, which can be implemented as a CPU, GPU, FPGA, NPU, etc. The processing unit 601 can perform various processes in the embodiments of any method of the present disclosure according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0216] The following components are connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage part 608 as necessary.

[0217] In particular, according to embodiments of the present disclosure, the above-mentioned methods with reference to any of the embodiments of the present disclosure can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a non-transitory computer readable medium, the computer program comprising program code for executing any of the methods of the embodiments of the present disclosure. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 609, and / or installed from the removable medium 611.

[0218] The flow charts and block diagrams in the drawings are illustrations of possible architectural, functional, and operational architectures of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow charts and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0219] The units or modules described in the embodiments of the present disclosure can be implemented by means of software, or by means of hardware. The described units or modules can also be provided in a processor, and the names of the units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0220] As another aspect, the disclosure also provides a computer readable storage medium, which can be the computer readable storage medium included in the apparatus described in the above embodiments; or can exist separately and not be assembled into the apparatus. The computer readable storage medium stores one or more programs for being executed by one or more processors to perform the method described in the disclosure.

[0221] The above description is merely the preferred embodiments of the disclosure and the explanation of the principles of the applied technologies. It should be understood by those skilled in the art that the inventive scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the disclosure (but not limited to) with similar functions.

Claims

1. A model training method, wherein, include: Obtain training videos; The training video is spatiotemporally transformed according to different spatiotemporal transformation modes to obtain at least two different spatiotemporally transformed videos. The video feature extraction model is trained using the video continuity information in the spatiotemporal transformed video as self-supervised information; The training of the video feature extraction model using video continuity information in the spatiotemporal transformed video as self-supervised information includes: obtaining feature vectors corresponding to the training video and the spatiotemporal transformed video using the video feature extraction model; obtaining prediction results of the video continuity information from the feature vectors using the at least one training task; and training the video feature extraction model based on the prediction results and the actual results of the video continuity information. The actual results are determined using the spatiotemporal transformation mode corresponding to the spatiotemporal transformed video. The at least one training task includes at least: a second training task, predicting the volume ratio of consecutive pixel blocks in the spatiotemporal transformed video relative to the spatiotemporal transformed video, and the Hamming distance of the consecutive pixel blocks relative to the training video in both the temporal and spatial dimensions, based on the feature vectors.

2. The method according to claim 1, wherein, The training video is spatiotemporally transformed according to different spatiotemporal transformation modes to obtain at least two different spatiotemporally transformed videos, including: Obtain the preset transformation order between the time dimension and the space dimension; The training video is transformed in both time and space dimensions according to the preset transformation order to obtain the spatiotemporal transformed video.

3. The method according to claim 1 or 2, wherein, The training video is spatiotemporally transformed according to different spatiotemporal transformation modes to obtain at least two different spatiotemporally transformed videos, including: The video frames in the training video are divided into pixel blocks; In the spatiotemporal transformation mode, the spatiotemporal transformation is performed on the pixel blocks to obtain the spatiotemporal transformation video.

4. The method according to claim 1, wherein, The at least one training task also includes at least one of the following: The first training task is to predict continuous pixel block information in the spatiotemporal transformed video based on the feature vector. The third training task is to predict the spatiotemporal transformation pattern corresponding to the spatiotemporal transformation video based on the feature vector. The fourth training task is to predict, based on the feature vector, the video that is most similar to the training video among the multiple spatiotemporal transformation videos.

5. The method according to claim 1, wherein, The method further includes obtaining the true result by at least one of the following methods: Determine the position information of the largest and second largest consecutive image block pixels in the spatiotemporal transformed video, and determine the position information as the true result; Determine the volume ratio between the sum of the volumes of the largest and second-largest consecutive pixel blocks in the spatiotemporal transformed video and the total volume of the spatiotemporal transformed video, and determine the standard Hamming distance of the largest consecutive pixel block in the temporal and spatial dimensions, and determine the volume ratio and the standard Hamming distance as the true result; The spatiotemporal transformation mode corresponding to the spatiotemporal transformation video is determined as the true result; The spatiotemporal transformation video corresponding to the preset spatiotemporal transformation mode is determined as the real result.

6. A video processing method, wherein, include: Get the input video; The input video is represented as a feature vector using a video feature extraction model; The video feature extraction model is trained using the method described in any one of claims 1-5.

7. A model training method, wherein, include: Obtain training videos; Call a preset service interface so that the preset service interface can perform spatiotemporal transformation on the training video according to different spatiotemporal transformation modes to obtain at least two different spatiotemporal transformation videos, and use at least one training task to train the video feature extraction model with the video continuity information in the spatiotemporal transformation video as self-supervised information. Output the video feature extraction model; The training of the video feature extraction model using video continuity information in the spatiotemporal transformed video as self-supervised information includes: obtaining feature vectors corresponding to the training video and the spatiotemporal transformed video using the video feature extraction model; obtaining prediction results of the video continuity information from the feature vectors using the at least one training task; and training the video feature extraction model based on the prediction results and the actual results of the video continuity information. The actual results are determined using the spatiotemporal transformation mode corresponding to the spatiotemporal transformed video. The at least one training task includes at least: a second training task, predicting the volume ratio of consecutive pixel blocks in the spatiotemporal transformed video relative to the spatiotemporal transformed video, and the Hamming distance of the consecutive pixel blocks relative to the training video in both the temporal and spatial dimensions, based on the feature vectors.

8. A model training device, wherein, include: The first acquisition module is configured to acquire training videos; The transformation module is configured to perform spatiotemporal transformation on the training video according to different spatiotemporal transformation modes to obtain at least two different spatiotemporal transformed videos. The training module is configured to train the video feature extraction model using at least one training task and with video continuity information in the spatiotemporal transformed video as self-supervised information. The training module is further configured to: acquire feature vectors corresponding to the training video and the spatiotemporal transformation video using the video feature extraction model; acquire prediction results of the video continuity information from the feature vectors using the at least one training task; and train the video feature extraction model based on the prediction results and the actual results of the video continuity information. The actual results are determined using the spatiotemporal transformation mode corresponding to the spatiotemporal transformation video. The at least one training task includes at least: a second training task, which predicts the volume ratio of consecutive pixel blocks in the spatiotemporal transformation video relative to the spatiotemporal transformation video, and the Hamming distance of the consecutive pixel blocks relative to the training video in both the temporal and spatial dimensions, based on the feature vectors.

9. A video processing apparatus, wherein, include: The second acquisition module is configured to acquire the input video; The feature representation module is configured to represent the input video as a feature vector using a video feature extraction model; the video feature extraction model is trained using the apparatus of claim 8.

10. A model training device, wherein, include: The third acquisition module is configured to acquire training videos; The calling module is configured to call a preset service interface so that the preset service interface can perform spatiotemporal transformation on the training video according to different spatiotemporal transformation modes to obtain at least two different spatiotemporal transformation videos, and use at least one training task to train the video feature extraction model with the video continuity information in the spatiotemporal transformation video as self-supervised information. The output module is configured to output the video feature extraction model; The preset service interface further utilizes the video feature extraction model to obtain feature vectors corresponding to the training video and the spatiotemporal transformation video; utilizes the at least one training task to obtain prediction results of the video continuity information from the feature vectors; and trains the video feature extraction model based on the prediction results and the actual results of the video continuity information. The actual results are determined using the spatiotemporal transformation mode corresponding to the spatiotemporal transformation video. The at least one training task includes at least: a second training task, which predicts the volume ratio of consecutive pixel blocks in the spatiotemporal transformation video relative to the spatiotemporal transformation video, and the Hamming distance of the consecutive pixel blocks relative to the training video in both the temporal and spatial dimensions, based on the feature vectors.

11. An electronic device, wherein, Includes memory and processor; among which, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1-7.

12. A computer-readable storage medium having stored thereon computer instructions, wherein, When executed by a processor, the computer instructions implement the method described in any one of claims 1-7.

Images