Video processing method and device, training method and device, processing equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a video processing method, a model training method and device for video processing, a processing device and a storage medium. The method comprises: obtaining predetermined features of a video, the predetermined features comprising video features and audio features; inputting the predetermined features into a trained network model to extract an Embedding expression of the video, to obtain embedding vectors of different granularities of the video; wherein the different granularities comprise at least one of the following: video frame level granularity; video segment level granularity; and video level granularity. In the embodiments of the present disclosure, after the video features and the audio features are input into the trained network model, embedding vectors of different granularities of the video that can adapt to different application scenarios can be obtained, and the ability to adapt to application scenarios will be stronger. In this way, the expression ability and adaptability of video processing based on the Embedding expression can be improved, and the use requirements in different application scenarios can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video processing, and in particular, to a video processing method, a model training method for video processing, a device, a processing apparatus and a storage medium. BACKGROUND

[0002] The Internet is developing rapidly, and short video applications are growing explosively, which brings a large amount of video data to be processed. In application scenarios such as video recommendation, video retrieval and / or video infringement detection, the Embedding expression of a video is a core algorithm. The Embedding expression of a video is to find a low-dimensional vector to uniquely represent the video to be expressed. In this way, the distance between vectors of different videos can be calculated to describe the potential relationship between two videos. In the related art, the processing of the Embedding expression of a video has poor adaptability to application scenarios and limited expression ability, which cannot meet the use requirements in different application scenarios. SUMMARY

[0003] Therefore, the embodiments of the present disclosure disclose a video processing method, a model training method, a device, a processing apparatus and a storage medium to at least improve the expression ability and adaptability of video processing and improve the user experience of processing videos in different application scenarios.

[0004] According to a first aspect of the embodiments of the present disclosure, a video processing method is provided, comprising:

[0005] obtaining predetermined features of a video, the predetermined features comprising video features and audio features;

[0006] inputting the predetermined features into a trained network model to extract an Embedding expression of the video, to obtain an Embedding vector of the video at different granularities; wherein the different granularities comprise at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity.

[0007] In one embodiment, the method further comprises:

[0008] based on the type of a predetermined service, selecting at least one video processing for performing the predetermined service from the Embedding vectors at different granularities.

[0009] In one embodiment, the network model comprises a first Inception network, a second Inception network and a fusion network.

[0010] The inputting the predetermined features into the trained network model to extract the Embedding expression of the video, to obtain the Embedding vector of the video at different granularities comprises:

[0011] inputting the video feature into the first created Inception network for video feature vector extraction to obtain a first feature vector;

[0012] inputting the audio feature into the second created Inception network for audio feature vector extraction to obtain a second feature vector;

[0013] inputting the first feature vector and the second feature vector into the fusion network for feature vector fusion to obtain the video frame-level granularity embedding vector; wherein the first feature vector and the second feature vector correspond to the same time position of the video.

[0014] In one embodiment, the fusion network comprises a merging network and a multi-layer perception (MLP) network.

[0015] The inputting the first feature vector and the second feature vector into the fusion network for feature vector fusion to obtain the video frame-level granularity embedding vector comprises:

[0016] inputting the first feature vector and the second feature vector into the merging network for feature vector merging to obtain a third feature vector;

[0017] inputting the third feature vector into the MPL network for feature vector fusion to obtain the video frame-level granularity embedding vector.

[0018] In one embodiment, the network model further comprises a first conversion coding network.

[0019] The inputting the predetermined feature into the trained network model for extracting the embedding Embedding expression of the video to obtain the embedding vector of the video at different granularities comprises:

[0020] obtaining N video frame-level granularity embedding vectors of the video, wherein N is an integer greater than 1.

[0021] inputting the N video frame-level granularity embedding vectors of the video into the first conversion coding network for conversion between video frame granularity and video segment granularity to obtain a video segment-level granularity embedding vector.

[0022] In one embodiment, the network model further comprises a second conversion coding network.

[0023] The inputting the predetermined feature into the trained network model for extracting the embedding Embedding expression of the video to obtain the embedding vector of the video at different granularities comprises:

[0024] obtaining embedding vectors of the video frame-level granularity of M videos, wherein M is an integer greater than 1;

[0025] inputting the embedding vectors of the video frame-level granularity of the M videos into the second conversion coding network to perform conversion between the video frame-level granularity and the video-level granularity, to obtain embedding vectors of the video-level granularity.

[0026] According to a second aspect of the embodiments of the present disclosure, a model training method for video processing is provided, comprising:

[0027] obtaining training samples, wherein the training samples comprise predetermined features of video samples; wherein the predetermined features comprise video features and audio features;

[0028] inputting the training samples into an initial network model to be trained to perform iterative training until a convergence condition is met, to obtain a trained network model; wherein the network model is used to extract embedding Embedding representation of a video, to obtain embedding vectors of different granularities of the video; wherein the different granularities comprise at least one of the following: video frame-level granularity; video segment-level granularity; and video-level granularity.

[0029] According to a third aspect of the embodiments of the present disclosure, a video processing apparatus is provided, comprising:

[0030] an obtaining module configured to obtain predetermined features of a video, wherein the predetermined features comprise video features and audio features;

[0031] an input module configured to input the predetermined features into a trained network model to extract embedding Embedding representation of the video, to obtain embedding vectors of different granularities of the video; wherein the different granularities comprise at least one of the following: video frame-level granularity; video segment-level granularity; and video-level granularity.

[0032] According to a fourth aspect of the embodiments of the present disclosure, a model training apparatus for video processing is provided, comprising:

[0033] an obtaining module configured to obtain training samples, wherein the training samples comprise predetermined features of video samples; wherein the predetermined features comprise video features and audio features;

[0034] a training module configured to input the training samples into an initial network model to be trained to perform iterative training until a convergence condition is met, to obtain a trained network model; wherein the network model is used to extract embedding Embedding representation of a video, to obtain embedding vectors of different granularities of the video; wherein the different granularities comprise at least one of the following: video frame-level granularity; video segment-level granularity; and video-level granularity.

[0035] According to a fifth aspect of the embodiments of the present disclosure, a processing device is provided, which comprises:

[0036] a memory for storing an executable program;

[0037] a processor for implementing the method according to any of the embodiments of the present disclosure when executing the executable program stored in the memory.

[0038] According to a sixth aspect of the embodiments of the present disclosure, a computer storage medium is provided, which stores an executable program, and the executable program, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0039] In the embodiments of the present disclosure, a predetermined feature of a video is acquired, the predetermined feature comprising a video feature and an audio feature; the predetermined feature is input into a trained network model to extract an Embedding expression of the video, to obtain an Embedding vector of the video at different granularities; wherein the different granularities comprise at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity. Here, since the feature input into the network model comprises both the video feature of the video and the audio feature of the video, compared with the way of inputting only the video feature of the video, the feature description of the video will be more comprehensive, and the Embedding expression result will be more accurate. After the video feature and the audio feature are input into the trained network model, the Embedding vector of the video at multiple different granularities can be obtained, compared with the way of obtaining only the Embedding vector at a single granularity, the Embedding vector of the video at different granularities which can adapt to different application scenarios can be obtained, and the adaptability to application scenarios will be stronger. In this way, the expression ability and adaptability of the video processing based on the Embedding expression can be improved, and the use requirements in different application scenarios can be met. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 a schematic diagram of an Embedding expression model according to an exemplary embodiment;

[0041] Figure 2 a flowchart of a video processing method according to an exemplary embodiment;

[0042] Figure 3 a flowchart of a video processing method according to an exemplary embodiment;

[0043] Figure 4 a flowchart of a model training method of video processing according to an exemplary embodiment;

[0044] Figure 5A flowchart of a video processing method according to an exemplary embodiment is shown.

[0045] Figure 6 A flowchart of a video processing method according to an exemplary embodiment is shown.

[0046] Figure 7 A schematic diagram of an Inception V1 network model according to an exemplary embodiment is shown.

[0047] Figure 8 A schematic diagram of a video frame and audio frame fusion according to an exemplary embodiment is shown.

[0048] Figure 9 A schematic diagram of a transform encoder network model according to an exemplary embodiment is shown.

[0049] Figure 10 A schematic diagram of a video processing method according to an exemplary embodiment is shown.

[0050] Figure 11 A schematic diagram of a video processing device according to an exemplary embodiment is shown.

[0051] Figure 12 A schematic diagram of a model training device for video processing according to an exemplary embodiment is shown. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0053] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0054] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0056] In order to better understand the embodiments of the present disclosure, the following describes the scenario in the related art through exemplary embodiments:

[0057] In one embodiment, the Embedding expression can find a low-dimensional vector that can uniquely represent a video. By calculating the distance between the vectors of different videos, such as Euclidean distance and cosine distance, etc., the potential relationship between the two videos, such as the degree of similarity, can be described.

[0058] In one embodiment, using Embedding expression of a predetermined dimension to express video content can structure the unstructured video content. According to the learning mode, the video Embedding algorithm can be divided into supervised video Embedding algorithm and unsupervised video Embedding algorithm.

[0059] Among them, the supervised learning video Embedding algorithm can use deep learning technology to classify videos according to a large number of manually annotated videos, and extract fully connected layer (FC, Fully Connected) layer features as video Embedding. For example, the classification methods such as Two-Stream CNN, TSN, P3D, I3D, SlowFast, etc. can be used for video Embedding extraction. Such methods have the following disadvantages: 1. A large amount of manually annotated data is required, and the labor cost is extremely high. 2. The label system needs to be defined in advance, and if the test video is not within this label system, the extracted Embedding is not good.

[0060] Among them, the unsupervised learning video Embedding can be trained on a large number of unannotated video data. If there is another task, such as a video classification task, a small amount of data can be labeled on the basis of pre-training for fine-tuning. This method can be divided into two types according to the information used: single mode and multi-mode. Single mode is to extract features only according to video pictures; multi-mode is to learn by combining video with text, audio and other related information. In the single mode method, most of them are a combination of temporal information and spatial image information.

[0061] Please refer to Figure 1, which adopts R3D-18 or S3D as the basis. A 32-frame-long segment is extracted from the video, divided into two 16-frame-long segments, and the two segments are positive sample pairs, and 16-frame-long segments from other videos are used as negative samples. The samples are subjected to time and space enhancement transformation, and the positive samples are expanded. This method combines semantic-level contrast learning and instance-level contrast learning to supervise the extraction of low-level features and middle-level features, thereby obtaining better high-level features.

[0062] In the related art, although the learning of low-level and middle-level features is supervised by contrast learning, no fine-grained Embedding features are generated, and only video-level semantic Embedding features are finally generated. For some application scenarios, what is needed is the overall expression of a video, such as video classification, which needs to start from the overall content of the video, and some scenarios; such as video copyright protection, only a small segment of the video may infringe the original video copyright, in which case the Embedding of the video segment needs to be compared, rather than the overall Embedding of the video. In addition, a convolutional neural network (CNN) is used as the basis in the algorithm architecture, so the ability to capture the temporal relationship is limited, especially for long videos, and the longest video image processed is only 32 frames. Only the features of the video image are considered, and the voice feature is not considered, but the voice information is very important for video understanding.

[0063] As shown in Figure 2 , the present disclosure provides a video processing method, which comprises:

[0064] Step S21, obtaining predetermined features of a video, wherein the predetermined features comprise video features and audio features;

[0065] Step S22, inputting the predetermined features into a trained network model to extract an Embedding expression of the video, to obtain an Embedding vector of different granularities of the video; wherein the different granularities comprise at least one of the following: video frame-level granularity; video segment-level granularity; and video-level granularity.

[0066] The video processing method of the present disclosure can be applied to an electronic device, and the electronic device related to the present disclosure can be, but is not limited to, a computer, a mobile phone, a wearable device, a vehicle-mounted terminal, a roadside unit (RSU), a smart home terminal, an industrial sensing device, and / or a medical device, etc.

[0067] In an embodiment, the video frame level granularity can be a granularity capable of representing features of a video frame of the video, and the embedding vector of the video frame level granularity can be used in a scenario requiring features at the video frame level, for example, in a video copyright protection application scenario, in order to accurately determine the time point at which infringement occurs, the embedding vector of the video frame level granularity is required.

[0068] In an embodiment, the video segment level granularity can be a granularity capable of representing features of a video segment, and the embedding vector of the video segment level granularity can be used in a scenario requiring features at the video segment level, for example, in a video duplication detection application scenario, in order to determine whether two videos have partial duplication, the embedding vector of the video segment level granularity is required.

[0069] In an embodiment, the video level granularity can be a granularity capable of representing features of a video segment, and the embedding vector of the video level granularity can be used in a scenario requiring features at the video level, for example, in a video classification scenario, in order to express the video as a whole, the embedding vector of the video level granularity is required.

[0070] It should be noted that the above method corresponds to the application process of the network model of the Embedding expression of the video, and the application process can be executed on an electronic device requiring the method. Before applying the network model, the network model also needs to be trained, and the training process of the network model can be executed on a special device dedicated to model training, or on the electronic device applying the method, which is not limited here. When the training process of the network model is executed on the special device, after the network model training is completed, the trained network model can be transplanted from the special device to the electronic device applying the method for execution.

[0071] In an embodiment, a training sample is obtained, and the training sample includes predetermined features of a video sample; wherein the predetermined features include video features and audio features; the training sample is input into an initial network model to be trained for iterative training until a convergence condition is met, and a trained network model is obtained. The predetermined features of a video are obtained, and the predetermined features include video features and audio features; the predetermined features are input into the trained network model to extract an Embedding expression of the video, and embedding vectors of different granularities of the video are obtained; wherein the different granularities include at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity.

[0072] It should be noted that the above embodiments include the training process and the application process of the network model for the Embedding representation of the video, the training process can be executed on a dedicated device, and the application process can be executed on an electronic device applying the method. Alternatively, the training process and the application process can be executed on an electronic device applying the method.

[0073] In some embodiments, the convergence condition can be that the convergence function of the initial network model satisfies the convergence condition, or the convergence condition can be determined when the number of training reaches a predetermined number of times, or other situations, which are not limited herein.

[0074] In one embodiment, a predetermined feature of a video is obtained, the predetermined feature including a video feature and an audio feature; the predetermined feature is input into the trained network model to extract an Embedding representation of the video, to obtain an Embedding vector of different granularities of the video; wherein the different granularities include at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity. Based on the type of a predetermined service, at least one video processing of the predetermined service is selected from the Embedding vector of different granularities. It should be noted that the predetermined service can be at least one of the following: a video classification service, a video copyright protection service, and a video duplicate detection service.

[0075] Exemplarily, in response to the predetermined service being the video classification service, the Embedding vector of the video level granularity is selected to perform the video processing of the predetermined service; in response to the predetermined service being the video copyright protection service, the Embedding vector of the video segment level granularity is selected to perform the video processing of the predetermined service; and in response to the predetermined service being the video duplicate detection service, the Embedding vector of the video frame level granularity is selected to perform the video processing of the predetermined service.

[0076] In an embodiment, predetermined features of a video are acquired, the predetermined features comprising video features and audio features; the predetermined features are input into a trained network model to extract an Embedding representation of the video, to obtain an Embedding vector of different granularities of the video; wherein the different granularities comprise at least one of: a video frame level granularity; a video segment level granularity; and a video level granularity. The network model comprises: a first Inception network, wherein the first Inception network is configured to extract a video feature vector; a second Inception network, wherein the second Inception network is configured to extract an audio feature vector; a fusion network, wherein the fusion network is configured to fuse the audio feature vector and the video feature vector; a first conversion and encoding network, wherein the first conversion and encoding network is configured to convert between the video frame level granularity and the video segment level granularity; and a second conversion and encoding network, wherein the second conversion and encoding network is configured to convert between the video frame level granularity and the video level granularity.

[0077] In an embodiment, the network model comprises a first Inception network, a second Inception network and a fusion network. Predetermined features of a video are acquired, the predetermined features comprising video features and audio features. Predetermined features of a video are acquired, the predetermined features comprising video features and audio features. The video features are input into the first Inception network for video feature vector extraction, to obtain a first feature vector. The audio features are input into the second Inception network for audio feature vector extraction, to obtain a second feature vector. The first feature vector and the second feature vector are input into the fusion network for feature vector fusion, to obtain an Embedding vector of the video frame level granularity; wherein the first feature vector and the second feature vector correspond to a same time position of the video.

[0078] In one embodiment, the network model comprises a first inception network, a second inception network, and a fusion network; the fusion network comprises a merging network and a multi-layer perception (MLP) network. Predetermined features of a video are obtained, including video features and audio features. Predetermined features of a video are obtained, including video features and audio features. The video features are input into the first inception network for video feature vector extraction, obtaining a first feature vector. The audio features are input into the second inception network for audio feature vector extraction, obtaining a second feature vector. The first feature vector and the second feature vector are input into the merging network for feature vector merging, obtaining a third feature vector; the third feature vector is input into the MPL network for feature vector fusion, obtaining an embedding vector at a frame-level granularity of the video; wherein the first feature vector and the second feature vector correspond to the same time position of the video.

[0079] In one embodiment, the network model comprises a first inception network, a second inception network, a fusion network, and a first conversion encoding network. Predetermined features of a video are obtained, including video features and audio features. Predetermined features of a video are obtained, including video features and audio features. The video features are input into the first inception network for video feature vector extraction, obtaining a first feature vector. The audio features are input into the second inception network for audio feature vector extraction, obtaining a second feature vector. The first feature vector and the second feature vector are input into the fusion network for feature vector fusion, obtaining an embedding vector at a frame-level granularity of the video; wherein the first feature vector and the second feature vector correspond to the same time position of the video. The embedding vector at the frame-level granularity of the N videos is obtained, wherein N is an integer greater than 1; the embedding vector at the frame-level granularity of the N videos is input into the first conversion encoding network for conversion between video frame granularity and video segment granularity, obtaining an embedding vector at a video segment-level granularity.

[0080] In an embodiment, the network model comprises a first created Inception network, a second created Inception network, a fusion network, and a second conversion coding network. A predetermined feature of a video is obtained, the predetermined feature comprising a video feature and an audio feature. A predetermined feature of a video is obtained, the predetermined feature comprising a video feature and an audio feature. The video feature is input into the first created Inception network for video feature vector extraction, to obtain a first feature vector. The audio feature is input into the second created Inception network for audio feature vector extraction, to obtain a second feature vector. The first feature vector and the second feature vector are input into the fusion network for feature vector fusion, to obtain an embedding vector of the video at a frame-level granularity; wherein the first feature vector and the second feature vector correspond to a same time position of the video. M embedding vectors of the video at the frame-level granularity of the video are obtained, wherein M is an integer greater than 1; the M embedding vectors of the video at the frame-level granularity of the video are input into the second conversion coding network for conversion between the frame-level granularity and a video-level granularity, to obtain an embedding vector of the video at the video-level granularity.

[0081] In an embodiment, the video in the present disclosure can be a video with a set length, for example, the video can contain X frames, or the video corresponds to a time length of T, wherein X is an integer greater than 1, and T is greater than zero.

[0082] In an embodiment, the video can be preprocessed. A predetermined feature of a video is obtained, the predetermined feature comprising a video feature and an audio feature; the predetermined feature is input into a trained network model to extract an embedding expression of the video, to obtain an embedding vector of the video at different granularities; wherein the different granularities comprise at least one of the following: a video frame-level granularity; a video segment-level granularity; and a video-level granularity. The preprocessing process can comprise at least one of the following: obtaining an image frame, enhancing the image frame, and adjusting the size of the image frame.

[0083] Illustratively, the process of obtaining an image frame can comprise: using an Ffmpeg tool to obtain a sequence of image frames at a predetermined period, wherein the Ffmpeg tool is used for video coding, format conversion, and video stream content processing. It should be noted that if the video is T seconds and the image frames are obtained at a speed of 1 frame per second, N image frames can be obtained.

[0084] Exemplarily, the process of performing enhancement processing on the image frame can include: selecting the image frame; and performing enhancement processing on the image frame to obtain an enhanced image frame; wherein the enhancement processing includes at least one of: flipping processing, rotation processing, cropping processing, deformation processing, scaling processing, and image parameter adjustment. It should be noted that the original image frame and the enhanced image frame are similar image frames.

[0085] Exemplarily, the process of adjusting the size of the image frame can include: adjusting the image frame to a fixed size, for example, 299x299.

[0086] In an embodiment, the process of preprocessing can include at least one of: obtaining an audio frame and performing enhancement processing on the audio frame.

[0087] Exemplarily, the process of obtaining the audio frame includes: obtaining a sequence of audio frames according to a predetermined period using an Ffmpeg tool, where the Ffmpeg tool is used for audio coding, format conversion, and audio stream content processing. It should be noted that if the video is T seconds and the audio frame is obtained at a speed of 1 frame per second, N audio frames can be obtained.

[0088] Exemplarily, the process of processing the audio frame includes: selecting the audio frame; and performing enhancement processing on the audio frame to obtain an enhanced audio frame; wherein the enhancement processing includes at least one of: noise processing, reverberation processing, time shift processing, tone processing, pull-up processing, and audio parameter adjustment. It should be noted that the original audio frame and the enhanced audio frame are similar audio frames.

[0089] It should be noted that the image data and the audio data obtained through preprocessing can correspond to video features and audio features input into a network model.

[0090] In the embodiments of the present disclosure, a predetermined feature of a video is acquired, the predetermined feature including a video feature and an audio feature; the predetermined feature is input into a trained network model to extract an Embedding expression of the video, to obtain an Embedding vector of different granularities of the video; wherein the different granularities include at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity. Here, since the feature input into the network model includes both the video feature of the video and the audio feature of the video, compared with a manner of inputting only the video feature of the video, the feature description of the video will be more comprehensive, and the Embedding expression result will be more accurate. After the video feature and the audio feature are input into the trained network model, the Embedding vector of multiple different granularities of the video can be obtained, compared with a manner of obtaining only the Embedding vector of a single granularity, the Embedding vector of different granularities of the video that can adapt to different application scenarios can be obtained, and the adaptability to application scenarios will be stronger. In this way, the expression ability and adaptability of the video processing based on the Embedding expression can be improved, and the use demand in different application scenarios can be met.

[0091] It should be noted that those skilled in the art can understand that the method provided by the embodiments of the present disclosure can be executed alone or together with some methods in some methods or related technologies in the embodiments of the present disclosure.

[0092] As shown in Figure 3 A model training method for video processing is provided in the embodiments of the present disclosure, and the method includes:

[0093] In step S31, a predetermined feature of a video is acquired, the predetermined feature including a video feature and an audio feature;

[0094] In step S32, the predetermined feature is input into a trained network model to extract an Embedding expression of the video, to obtain an Embedding vector of different granularities of the video; wherein the different granularities include at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity.

[0095] In step S33, based on a type of a predetermined business, at least one video processing for performing the predetermined business is selected from the Embedding vector of different granularities.

[0096] In an embodiment, a training sample is obtained, the training sample comprising predetermined features of a video sample; wherein the predetermined features comprise video features and audio features; the training sample is input into an initial network model to be trained for iterative training until a convergence condition is met, to obtain a trained network model. Predetermined features of a video are obtained, the predetermined features comprising video features and audio features; the predetermined features are input into the trained network model to extract an Embedding representation of the video, to obtain an Embedding vector of the video at different granularities; wherein the different granularities comprise at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity. Based on a type of a predetermined service, at least one video processing for the predetermined service is selected from the Embedding vector at the different granularities. It should be noted that the predetermined service can be at least one of the following: a video classification service, a video copyright protection service, and a video duplication detection service.

[0097] It should be noted that those skilled in the art can understand that the method provided by the embodiments of the present disclosure can be executed alone or together with some methods in some methods or related technologies in the embodiments of the present disclosure.

[0098] As shown in Figure 4 A model training method for video processing is provided in the embodiments of the present disclosure, and the method comprises:

[0099] In step S41, a training sample is obtained, the training sample comprising predetermined features of a video sample; wherein the predetermined features comprise video features and audio features.

[0100] In step S42, the training sample is input into an initial network model to be trained for iterative training until a convergence condition is met, to obtain a trained network model; wherein the network model is used to extract an Embedding representation of a video, to obtain an Embedding vector of the video at different granularities; wherein the different granularities comprise at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity.

[0101] The model training method for video processing of the present disclosure can be applied to an electronic device. The electronic device related to the present disclosure can be, but is not limited to, a computer, a mobile phone, a wearable device, a vehicle-mounted terminal, a roadside unit (RSU), a smart home terminal, an industrial sensing device, and / or a medical device, etc.

[0102] It should be noted that the training process of the initial network model can be performed on a dedicated device for model training, or can be performed on the electronic device applying the video processing of the present disclosure, which is not limited here. When the training process of the initial network model is performed on the dedicated device, after the initial network model training is completed, the trained network model can be transplanted from the dedicated device to the electronic device applying the video processing of the present disclosure for execution.

[0103] In one embodiment, a training sample is obtained, which includes a predetermined feature of a video sample; wherein the predetermined feature includes a video feature and an audio feature; the training sample is input into an initial network model to be trained for iterative training until a convergence condition is met, to obtain a trained network model. The predetermined feature is input into the trained network model to extract an Embedding expression of the video, to obtain an Embedding vector of different granularities of the video; wherein the different granularities include at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity.

[0104] In one embodiment, the video used for training in the present disclosure can be a video with a set length, for example, the video can contain X frames, or the time length corresponding to the video can be T, wherein X is an integer greater than 1, and T is greater than zero. The length distribution of the video used for training is uniform. The number of videos used for training can be greater than a number threshold, which can be one hundred thousand. The samples of the training process can be preprocessed, and the preprocessing process can refer to the description of the network model application process, which will not be repeated here.

[0105] It should be noted that the above embodiment includes the training process and application process of the network model for the Embedding expression of the video, the training process can be performed on a dedicated device, and the application process can be performed on an electronic device applying the method. Alternatively, the training process and the application process can both be performed on an electronic device applying the method.

[0106] In some embodiments, meeting the convergence condition can be that the convergence function of training the initial network model meets the convergence condition, or can be that the convergence condition is determined to be met when the number of training times reaches a predetermined number of times, or other situations, which are not limited here.

[0107] In one embodiment, the convergence function (or loss function) can be:

[0108]

[0109] wherein d = ||a-b||, d represents the Euclidean distance of two feature vectors (a and b), y is the label of whether two samples match, y = 1 represents that two samples are similar or match (for example, a source video frame and an enhanced video frame), y = 0 represents that two samples do not match (for example, a source video frame and another video frame), and margin is a hyperparameter used to set an optimization effect threshold.

[0110] In one embodiment, the loss value corresponding to the convergence function is determined by weighting according to the loss values corresponding to the video, the video frame and the video segment respectively.

[0111] It should be noted that the method provided by the embodiments of the present disclosure can be executed alone or together with some methods in some methods or related technologies in the embodiments of the present disclosure.

[0112] In order to better understand the technical solutions of the present disclosure, the technical solutions of the present disclosure are further described below through an exemplary embodiment:

[0113] As shown in Figure 5 , a schematic diagram of a video processing flow is shown, which shows the ffmeg video frame extraction flow, the image Resize flow, the video preprocessing flow, the creation of the inception V1 network processing flow, the merging (concat) flow of the video vector and the audio vector, the fusion processing flow and the transform encoder flow of the video processing.

[0114] Based on Figure 5 , please refer to Figure 6 , a video processing method is provided in the embodiments of the present disclosure, and the method comprises:

[0115] In step S601, a training sample is obtained.

[0116] Exemplarily, the number of videos in the training sample is greater than a number threshold, for example, the number of videos in the training sample is greater than 100,000. In one embodiment, the time length of the videos in the training sample is uniformly distributed, for example, the image frames included in each video can be the same.

[0117] In step S602, a video frame is obtained from the video.

[0118] Exemplarily, the Ffmpeg tool (an open source computer program that can be used to record, convert digital audio, video and convert it into a stream) is used to obtain the video frame. For example, the video frame is extracted from the video in a manner of extracting 1 frame per second to obtain an image sequence of the video frame. If the time length of the video is N seconds, N video frames can be obtained.

[0119] Step S603, pre-processing the video frame.

[0120] Exemplarily, a video file is randomly obtained, a frame image (corresponding to a video frame) is selected from the video file, the frame image is data-augmented, the operation of data-augmentation exemplarily includes at least one of flipping, rotating, cropping, deforming and scaling, and the strength of the parameters corresponding to the above operations is adjusted, to obtain the image of the augmented video frame. In an embodiment, the original video frame and the augmented video frame are defined as similar video frames.

[0121] Step S604, obtaining an audio frame from the video and performing enhancement processing on the audio frame.

[0122] Exemplarily, the Ffmpge tool is used to obtain the audio frame. For example, the audio frame is extracted from the video at a rate of 1 frame per second to obtain an audio frame sequence. For example, if the audio duration is N seconds, N audio frames can be obtained. In an embodiment, the audio frame is enhanced. Exemplarily, the operation of enhancing the audio frame includes at least one of noise processing, reverberation processing, time shift processing, tone processing and pull-up processing, and the strength of the parameters corresponding to the above operations is adjusted, and the original audio frame and the enhanced audio frame are defined as similar audio frames.

[0123] Step S605, inputting the video frame into the Inception V1 network (corresponding to the first created Inception network in the present disclosure).

[0124] Exemplarily, the augmented video frame obtained in step S603 is adjusted to a fixed size of 299x299, and the video frame is input into the Inception V1 network. The Inception V1 network model is as shown in Figure 7 .

[0125] Exemplarily, since the dimension of the finally obtained feature vector is 128, the average pooling parameter of the network can be set to 128 to obtain the feature vector to be optimized (corresponding to the first feature vector in the present disclosure) of the video frame.

[0126] Step S606, inputting the audio frame into the Inception V1 network (corresponding to the second created Inception network in the present disclosure).

[0127] Exemplarily, the enhanced audio frame obtained in step S604 is input into the Inception V1 network (based on the open source software Librosa), and the Average Pooling parameter of the network is set to 128. Thus, if the audio length is N, the feature vector N*128 to be optimized of the audio frame (corresponding to the second feature vector in the present disclosure) is obtained.

[0128] Step S607, feature vector merging.

[0129] Exemplarily, the feature vector to be optimized of the video frame obtained in step S605 and the feature vector of the audio frame at the same position of the same video frame obtained in step S606 are merged to obtain a 256-dimensional feature vector (corresponding to the third feature vector in the present disclosure).

[0130] Step S608, feature vector fusion.

[0131] Exemplarily, the 256-dimensional feature vector obtained in step S607 is fused by a multilayer perceptron (MLP, Multilayer Perceptron), a 256*128 full connection layer is used, and a 128-dimensional feature vector after fusion of the video frame and the audio frame (corresponding to the video frame level granularity feature vector in the present disclosure) is obtained, as shown in Figure 8 .

[0132] Step S609, determining a contrast loss function.

[0133] Exemplarily, the contrast loss function is as follows:

[0134]

[0135] wherein d=||a-b||, d represents the Euclidean distance of two feature vectors (a and b), y is the label of whether two samples match, y=1 represents that two samples are similar or match (for example, the source video frame and the enhanced video frame), y=0 represents that they do not match (for example, the source video frame and other video frames), and margin is a hyperparameter used to set the optimization effect threshold.

[0136] Step S610, obtaining the loss value of the video frame positive sample.

[0137] Exemplarily, 128 video frame images are randomly selected, 128 feature vectors are generated by using step S608, 128 feature vectors are generated by using step S608 after the video frame image data is enhanced, and the Euclidean distance of each pair of feature vectors is calculated by using step S609 formula. Since the feature vectors are generated for the source video frame and the enhanced video frame, y=1 is taken, and the loss value of the video frame positive sample is obtained.

[0138] Step S611, obtain the loss value of the video frame negative sample.

[0139] Exemplarily, the video frame selected in step S610 is selected randomly from videos other than the video, the positive-negative sample ratio can be 1:4, 512 feature vectors are generated by using step S608, and the Euclidean distance of each pair of feature vectors is calculated by using step S609 formula. Since the source video frame and the video frame of the remaining video are used, y=0, and the loss value of the video frame negative sample is obtained.

[0140] Step S612, a method for determining a network model (corresponding to the initial network model in the present disclosure) used for optimizing (training) the Embedding representation of the video.

[0141] Exemplarily, the optimization goal is that the smaller the distance between similar video frame feature vectors is, the better (for example, in the formula of step S609, y=1, and the loss function only has the first part), and the larger the distance between dissimilar video frame feature vectors is, the better (for example, in the formula of step S609, y=0, and the loss function only has the second part). When the Euclidean distance is greater than the hyperparameter margin, it is determined that the network model optimization is completed.

[0142] Step S613, input the feature vector into a transform encoder network model included in the network model.

[0143] Exemplarily, the length of the video segment is set to M, 128 videos are selected, and M consecutive video frames are randomly selected from each video. M*128 feature vectors are obtained by using step S608, and the M*128 feature vectors are input into a transform encoder network model included in the network model. The network structure is as shown in Figure 9 The video segment 128-dimensional feature vector is obtained by connecting an average pooling layer (Average Poll) at the back. Figure 9 In the figure, 1 is a position encoding module; 2 is a self-attention mechanism module; 3 is a normalization and residual connection module; and 4 is a transform encoding module.

[0144] Step S614, obtain the loss value of the video segment positive sample and the loss value of the video segment negative sample.

[0145] Exemplarily, the loss value of the video segment positive sample is obtained based on step S610, and the loss value of the video segment negative sample is obtained based on step S611.

[0146] Step S615, input the feature vector of the video into the transformer encoder network.

[0147] Exemplarily, 128 videos are selected, all video frames of each video are grouped into a frame sequence, and the feature vector of each video is obtained by using step S608. If the video length is P, P*128 feature vectors are obtained, and the P*128 feature vectors are input into a transformer encoder network, the network structure is as shown in Figure 9 The Average Pool layer is connected at the back end to obtain a 128-dimensional feature vector of the video.

[0148] Step S616, obtaining the loss value of the video-level positive sample and the loss value of the video-level negative sample.

[0149] Exemplarily, the loss value of the video-level positive sample is obtained by using step S610, and the loss value of the video-level negative sample is obtained by using step S611.

[0150] Step S617, obtaining the number of video segments.

[0151] Exemplarily, the video length is set to X, the video segment length is set to Y, and a step parameter K is set to take into account the robustness of the segment feature vector. Each segment contains K frames of information of the previous segment. Thus, the number S of segments with a video length of X can be obtained, where S=(X-M) / (Y-K)+1.

[0152] Step S618, obtaining the fusion loss value.

[0153] Exemplarily, the video frame loss value (obtained by steps S610 and S611), the video segment loss value (obtained by step S614), and the video-level loss value (obtained by step S616) are weighted and added by using the video frame length X and the number S of video segments (obtained by step S617). The weight value Q is obtained, where Q=X / S. The video frame loss value is divided by Q, the video-level loss value is multiplied by Q, and the video segment loss value remains unchanged.

[0154] Step S619, training the network model for the Embedding expression of the video.

[0155] Exemplarily, according to the principle of step S612, the model is iteratively optimized and trained to obtain the trained network model for the Embedding expression of the video.

[0156] Step S620, obtaining the embedding vectors of different granularities.

[0157] Exemplarily, one video is input into the network by extracting video frames and audio, obtains the embedding vector of the video frame level granularity in step 8, obtains the embedding vector of the video segment level granularity in step 13, and obtains the embedding vector of the video level granularity in step 15. Different granularity embedding vectors can be selected for different downstream tasks.

[0158] In one embodiment, referring to Figure 10 , in the video classification scene, the video needs to be expressed as a whole, and the embedding vector of the video level granularity in the network is used. Figure 10 If only video images are used for training, the audio features are missing, the model has poor distinguishability in the field where the video image distinguishability is not high, for example, games, sports and live videos, and the like, and the model has poor robustness due to the reasons such as the fixed video length, the large difference between the front and back pictures of the video, and the like. In order to solve these problems, the embodiment of the present disclosure first introduces the audio features, the audio features have good distinguishability in the field where the video image distinguishability is not high, and the training of the network model can combine the loss calculated by the video frames, the video semantic information is more rich, and the effect of the embedding expression of the video level granularity is improved.

[0159] In one embodiment, in the video repetition detection scene, it is needed to judge whether two videos have partial repetition, and the embedding vector of the video level granularity in the network is selected. Figure 10 If only the embedding of the video frame level granularity or the embedding of the video level granularity is extracted, the embedding of the video segment level granularity is not used. The embodiment of the present disclosure provides an embedding of the video segment level granularity, which has better effect than the embedding of the video level granularity and has less resource consumption than the embedding of the video frame level granularity. The suitable video segment length can be found according to the needs of the business, and the balance between the model effect and the resource consumption is good.

[0160] In one embodiment, in the video copyright protection application, it is needed to accurately judge the time point of the infringement, and the embedding vector of the video frame level granularity in the network is selected. The embedding vector of the video frame level granularity has better representation effect. Figure 10

[0161] As shown in Figure 11 , the embodiment of the present disclosure provides a video processing apparatus, which comprises:

[0162] The acquisition module 111 is configured to acquire predetermined features of a video, wherein the predetermined features comprise video features and audio features.

[0163] ​The processing module 112 is configured to input the predetermined features into the trained network model to extract an embedding expression of the video, to obtain embedding vectors of different granularities of the video; wherein the different granularities include at least one of the following: a video frame level granularity; a video segment level granularity; and a video level granularity.

[0164] In one embodiment, the apparatus further includes:

[0165] The selection module 113 is configured to select at least one video processing for performing the predetermined service from the embedding vectors of different granularities based on a type of the predetermined service.

[0166] In one embodiment, the network model includes a first Inception network, a second Inception network, and a fusion network; and the processing module 112 is further configured to:

[0167] input the video features into the first Inception network to extract a video feature vector, to obtain a first feature vector;

[0168] input the audio features into the second Inception network to extract an audio feature vector, to obtain a second feature vector;

[0169] input the first feature vector and the second feature vector into the fusion network to fuse the feature vectors, to obtain the embedding vector of the video frame level granularity; wherein the first feature vector and the second feature vector correspond to a same time position of the video.

[0170] In one embodiment, the fusion network includes a merging network and a multi-layer perception (MLP) network; and the processing module 112 is further configured to:

[0171] The inputting the first feature vector and the second feature vector into the fusion network to fuse the feature vectors, to obtain the embedding vector of the video frame level granularity, includes:

[0172] input the first feature vector and the second feature vector into the merging network to merge the feature vectors, to obtain a third feature vector;

[0173] input the third feature vector into the MPL network to fuse the feature vectors, to obtain the embedding vector of the video frame level granularity.

[0174] In one embodiment, the network model further includes a first conversion encoding network; and the processing module 112 is further configured to:

[0175] obtaining embedding vectors of the video frame level granularity of N videos, wherein N is an integer greater than 1;

[0176] inputting the embedding vectors of the video frame level granularity of the N videos into the first conversion coding network for conversion between the video frame level granularity and the video segment level granularity, to obtain embedding vectors of the video segment level granularity.

[0177] In one embodiment, the network model further comprises a second conversion coding network; the processing module 112 is further configured to:

[0178] obtaining embedding vectors of the video frame level granularity of M videos, wherein M is an integer greater than 1;

[0179] inputting the embedding vectors of the video frame level granularity of the M videos into the second conversion coding network for conversion between the video frame level granularity and the video level granularity, to obtain embedding vectors of the video level granularity.

[0180] As shown in Figure 12 The embodiment of the present disclosure provides a model training device for video processing, comprising:

[0181] an obtaining module 121 configured to obtain training samples, wherein the training samples comprise predetermined features of video samples; wherein the predetermined features comprise video features and audio features;

[0182] a processing module 122 configured to input the training samples into an initial network model to be trained for iterative training until a convergence condition is met, to obtain a trained network model; wherein the network model is used to extract embedding Embedding expression of a video, to obtain embedding vectors of different granularities of the video; wherein the different granularities comprise at least one of the following: video frame level granularity; video segment level granularity; and video level granularity.

[0183] The embodiment of the present disclosure provides a processing device, comprising:

[0184] a memory for storing executable programs;

[0185] a processor for executing the executable programs stored in the memory, to implement the method according to any one of the embodiments of the present disclosure.

[0186] It can be understood that the memory can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0187] The method disclosed in the present application can be applied to or implemented by the processor. The processor can be an integrated circuit chip with processing capability. In the implementation process, the steps of the method for speech conversion can be completed by the integrated logic circuit or the instruction of software form in the processor. The processor mentioned above can be a general processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor can implement or execute the disclosed methods, steps and logic block diagrams in the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the present application, the method can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines the hardware to complete the steps of the method for speech conversion provided in the embodiments of the present application.

[0188] The present application also provides a computer storage medium, which stores an executable program. When the executable program is executed by a processor, the method according to any one of the embodiments of the present application is implemented. Specifically, the computer storage medium can be a computer readable storage medium, such as a memory for storing a computer program, which can be executed by a processor of a processing device to complete the steps of the method according to the embodiments of the present application. The computer readable storage medium can be a ROM, a PROM, an EPROM, an EEPROM, a flash memory, a magnetic surface memory, an optical disc, or a CD-ROM, etc.

[0189] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of video processing, the method comprising: The method comprises: acquiring predetermined features of a video, the predetermined features comprising video features and audio features; inputting the predetermined features into a trained network model to extract an Embedding representation of the video, to obtain an Embedding vector of the video at different granularities; wherein the different granularities comprise at least two of the following: video frame level granularity; video segment level granularity; and video level granularity; the video frame level granularity is a granularity representing features of a video frame extracted from the video, the video segment level granularity is a granularity representing features of a video segment extracted from the video, the video segment comprising at least two video frames; and the video level granularity is a granularity representing features of the video, the video comprising video frames; and the network model comprises: a first Inception network, wherein the first Inception network is configured to extract a video feature vector; a second Inception network, wherein the second Inception network is configured to extract an audio feature vector; a fusion network, wherein the fusion network is configured to fuse the audio feature vector and the video feature vector; a first conversion and encoding network, wherein the first conversion and encoding network is configured to convert between the video frame level granularity and the video segment level granularity; and a second conversion and encoding network, wherein the second conversion and encoding network is configured to convert between the video frame level granularity and the video level granularity.

2. The method of claim 1, wherein, The method further comprises: based on a type of a predetermined service, selecting at least one video processing for the predetermined service from the Embedding vectors at the different granularities.

3. The method of claim 1, wherein, The method further comprises: inputting the video features into the first Inception network to extract a video feature vector, to obtain a first feature vector; inputting the audio features into the second Inception network to extract an audio feature vector, to obtain a second feature vector; inputting the first feature vector and the second feature vector into the fusion network to fuse the feature vectors, to obtain an Embedding vector of the video at the video frame level granularity; wherein the first feature vector and the second feature vector correspond to a same time position of the video.

4. The method of claim 3, wherein, The fusion network comprises a merging network and a multi-layer perception (MLP) network. The method further comprises: inputting the first feature vector and the second feature vector into the merging network to merge the feature vectors, to obtain a third feature vector; inputting the third feature vector into the MLP network to fuse the feature vectors, to obtain the Embedding vector of the video at the video frame level granularity.

5. The method of claim 3, wherein: the inputting the predetermined features into the trained network model to extract the Embedding representation of the video, to obtain the Embedding vector of the video at the different granularities comprises: obtaining embedding vectors of the video frame level granularity of N videos, wherein N is an integer greater than 1; inputting the embedding vectors of the video frame level granularity of the N videos into the first conversion coding network for conversion between the video frame level granularity and the video segment level granularity, to obtain embedding vectors of the video segment level granularity of the videos.

6. The method of claim 3, wherein, the inputting the predetermined features into the trained network model to extract an embedding Embedding representation of the video, to obtain embedding vectors of different granularities of the video, comprises: obtaining embedding vectors of the video frame level granularity of M videos, wherein M is an integer greater than 1; inputting the embedding vectors of the video frame level granularity of the M videos into the second conversion coding network for conversion between the video frame level granularity and the video level granularity, to obtain embedding vectors of the video level granularity of the videos.

7. A method for model training of video processing, characterized in that, comprises: obtaining training samples, the training samples comprising predetermined features of video samples; wherein the predetermined features comprise video features and audio features; inputting the training samples into an initial network model to be trained for iterative training until a convergence condition is met, to obtain a trained network model; wherein the network model is used to extract an embedding Embedding representation of a video, to obtain embedding vectors of different granularities of the video; wherein the different granularities comprise at least two of the following: video frame level granularity; video segment level granularity; and video level granularity; the video frame level granularity is a granularity representing features of a video frame extracted from a video; the video segment level granularity is a granularity representing features of a video segment extracted from a video, the video segment comprising at least two video frames; the video level granularity is a granularity representing features of a video, the video comprising video frames; and the network model comprises: a first created Inception network, wherein the first created Inception network is used to extract a video feature vector; a second created Inception network, wherein the second created Inception network is used to extract an audio feature vector; a fusion network, wherein the fusion network is used for fusion of the audio feature vector and the video feature vector; a first conversion coding network, wherein the first conversion coding network is used for conversion between the video frame level granularity and the video segment level granularity; and a second conversion coding network, wherein the second conversion coding network is used for conversion between the video frame level granularity and the video level granularity.

8. A video processing apparatus, comprising: comprises: an obtaining module configured to obtain predetermined features of a video, the predetermined features comprising video features and audio features; The input module is configured to input the predetermined features into the trained network model to extract an Embedding expression of the video, to obtain an Embedding vector of different granularities of the video; wherein the different granularities include at least two of the following: a video frame level granularity; a video segment level granularity; and a video level granularity; the video frame level granularity is a granularity representing features of a video frame extracted from the video; the video segment level granularity is a granularity representing features of a video segment extracted from the video, the video segment including at least two video frames; and the video level granularity is a granularity representing features of the video, the video including video frames; The network model includes: a first Inception network, wherein the first Inception network is used to extract a video feature vector; a second Inception network, wherein the second Inception network is used to extract an audio feature vector; a fusion network, wherein the fusion network is used for fusion of the audio feature vector and the video feature vector; a first conversion and encoding network, wherein the first conversion and encoding network is used for conversion between the video frame level granularity and the video segment level granularity; and a second conversion and encoding network, wherein the second conversion and encoding network is used for conversion between the video frame level granularity and the video level granularity.

9. A processing device, characterized by The processing device includes: a memory for storing an executable program; a processor for executing the executable program stored in the memory, to implement the method of any one of claims 1 to 6 or claim 7.

10. A computer storage medium, characterized in that, The computer storage medium stores an executable program, and the executable program is executed by the processor to implement the method of any one of claims 1 to 6 or claim 7.

Images