Method and device for processing video data, equipment and medium

Abstract

The invention provides a method, a device, equipment and a medium for processing video data. In one method, in response to receiving video data, language data is obtained from the video data. And obtaining visual data from the video data. A video representation indicative of the video data is determined based on the language data and the visual data in accordance with a video processing model. According to the embodiment of the invention, in the process of realizing video understanding, the language aspect and the visual aspect in the video data can be fully considered, so that the video representation capable of more accurately describing multi-aspect information of the video data can be obtained.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure relate generally to video processing, and more particularly to methods, devices, apparatuses, and computer-readable storage media for processing video data. Background Art

[0002] With the development of Large Language Model (LLM), media content understanding tasks have been widely used. At present, it has been proposed to process media data based on LLM. However, the training and adjustment of LLM requires a lot of resources, and the effect for a specific video processing task is not ideal. At this time, it is expected to use pre-trained language networks and pre-trained visual networks in a more convenient and effective way to achieve lightweight media content understanding. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for processing video data is provided. In the method, in response to receiving the video data, language data is obtained from the video data. Visual data is obtained from the video data. According to a video processing model, a video representation for indicating the video data is determined based on the language data and the visual data.

[0004] In a second aspect of the present disclosure, a device for processing video data is provided. The device comprises: a first acquisition module configured to acquire language data from the video data in response to receiving the video data; a second acquisition module configured to acquire visual data from the video data; and a determination module configured to determine a video representation indicating the video data based on the language data and the visual data according to a video processing model.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar figure labels represent the same or similar elements, wherein:

[0009] Figure 1 A block diagram showing an application environment according to an exemplary implementation of the present disclosure is shown;

[0010] Figure 2 A block diagram for processing video data according to some implementations of the present disclosure is shown;

[0011] Figure 3A A block diagram illustrating a video processing model architecture according to some implementations of the present disclosure;

[0012] Figure 3B A block diagram showing an overview process for training a video processing model according to some implementations of the present disclosure;

[0013] Figure 4 A block diagram showing a detailed process for training a video processing model according to some implementations of the present disclosure;

[0014] Figure 5A A block diagram showing pseudo code of a first stage training algorithm according to some implementations of the present disclosure;

[0015] Figure 5B A block diagram showing pseudo code of a second stage training algorithm according to some implementations of the present disclosure;

[0016] Figure 6 A block diagram illustrating utilizing video representations to perform downstream processing tasks according to some implementations of the present disclosure is shown;

[0017] Figure 7 A block diagram showing the effects of performing downstream processing tasks according to some implementations of the present disclosure;

[0018] Figure 8 A block diagram showing the effects of performing downstream processing tasks according to some implementations of the present disclosure;

[0019] Fig. 9 A flowchart of a method for processing video data according to some implementations of the present disclosure is shown;

[0020] Fig.10 A block diagram showing an apparatus for processing video data according to some implementations of the present disclosure; and

[0021] Fig.11A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0024] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0025] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0026] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0027] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0029] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0030] Example Environment

[0031] Determining the summary of a video is one of the fundamental tasks in computer vision, and it has many downstream applications in video analysis, content understanding, etc. At present, LLM-based methods have been proposed to process media content. Figure 1 Describe an application environment according to an example implementation of the present disclosure, Figure 1 A block diagram 100 of an application environment according to an exemplary implementation of the present disclosure is shown. Figure 1 As shown, a machine learning model 120 can be used to process a video 110. For example, the machine learning model 120 can be implemented using an LLM to perform a variety of processing tasks. For example, where the processing task involves extracting a summary description of a video, the machine learning model 120 can output a text description 130 of the video data 110.

[0032] However, the training and fine-tuning of LLM requires a lot of resources, and the effect is not ideal for a specific video processing task. At this time, it is expected that a lightweight media content understanding model can be implemented in a more convenient and effective way by using pre-trained language networks and pre-trained visual networks.

[0033] Overview of visual processing tasks

[0034] Existing video understanding models focus more on visual modalities and ignore information such as subtitles. In addition, the training of end-to-end video understanding models usually incurs large energy costs, and the accuracy of video description models is not satisfactory. In some application scenarios, users expect to control the granularity of generated descriptions, which is difficult to achieve in existing models.

[0035] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a method for processing video data is proposed. It should be understood that the content understanding of video data is the basis of downstream specific processing tasks. Therefore, the video data can be analyzed from both language and vision aspects, and then the corresponding feature representations can be extracted. Starting from the existing graphic and text understanding model, the present disclosure uses the graphic and text understanding model to process a single frame of the input video. On this basis, by adding a lightweight module, it realizes the integration of multi-frame image information and can simultaneously integrate other modal information (such as automatic speech recognition information and OCR information), thereby reducing the overhead of the training process and reducing energy consumption.

[0036] See also Figure 2 An overview of an exemplary implementation of the present disclosure is described. Figure 2 A block diagram 200 for processing video data is shown in accordance with some implementations of the present disclosure.

[0037] like Figure 2 As shown, multimodal data can be extracted from the video data 110, for example, language data 210 and visual data 220 can be extracted. Here, the language data 210 can include, for example, subtitle data in the video and language data obtained via automatic speech recognition (Automatic Speech Recognition, abbreviated ASR) technology. The visual data 220 can include, for example, image data of each data frame in the video. Further, according to the video processing model 230, a video representation 240 for indicating the video data 110 can be determined based on the language data 210 and the visual data 220. It should be understood that the visual representation 240 here can include, for example, features of the video data 110, and can be represented by a multidimensional vector (e.g., embedding).

[0038] By using the example implementation of the present disclosure, the image and text understanding model can be extended to the field of video understanding, and the language and visual aspects of the video data can be fully considered, so as to obtain a video representation that can more accurately describe the multi-faceted information of the video data. Further, the video representation can be used to process various types of downstream processing tasks, thereby improving the accuracy of downstream processing tasks.

[0039] Detailed process of handling visual tasks

[0040] An overview of an example implementation according to the present disclosure has been described, and more details for video processing will be described below. According to an example implementation of the present disclosure, language data 210 and visual data 220 can be extracted from video data 110 in a variety of ways. For example, language data 210 can be obtained based on voice data in the video data. Specifically, voice data can be recognized based on a variety of ASR technologies currently known and / or to be developed in the future to obtain language data 210. Alternatively and / or additionally, language data 210 can be obtained based on subtitle data in the video data based on a variety of optical character recognition (OCR) currently known and / or to be developed in the future. In this way, language data 210 can be extracted in a variety of ways, thereby improving the accuracy of video representation 240.

[0041] According to an example implementation of the present disclosure, in the process of acquiring visual data 220, one or more video frames can be selected from video data 110 as visual data 220. For example, multiple video frames can be parsed from video data 110, and a group of video frames can be selected from the multiple video frames according to a predetermined condition to serve as visual data 220. Here, the predetermined condition can be defined in a variety of ways. For example, the predetermined condition can indicate: select each video frame in a group of video frames according to a predetermined time interval (for example, 5 seconds or other values). For another example, the predetermined condition can indicate: extract a predetermined number (for example, 16 frames or other values) of video frames from video data 110. In this way, the specific extraction method can be determined according to various parameters of video data 110 (for example, time length, frame rate, etc.), so as to balance the richness and data volume of visual data 220.

[0042] According to an exemplary implementation of the present disclosure, the video processing model 230 according to the present disclosure may be implemented using LLM. Figure 3A Describes the architecture of the video processing model 230 according to an example implementation of the present disclosure. Figure 3A A block diagram 300A of a video processing model architecture according to some implementations of the present disclosure is shown. As shown in FIG3 , based on the LLM, a language network 320 and a visual network 330 can be added to the video processing model 230. At this time, during the training process of the video processing model 230, the weights of the pre-trained model can be frozen, and only a small number of parameters need to be trained to achieve better video understanding. Furthermore, since the language network can be fine-tuned by prompt words, a dialogue with the image understanding model can be achieved by constructing supervised fine-tuning data, and more diverse answers can be obtained by giving instructions.

[0043] According to an example implementation of the present disclosure, the video processing model 230 can be used to complete the reasoning process. During the training process, the text network 340 can be constructed, and the intermediate latent variables of the video processing model 230 and the text network 340 can be matched through an "alignment" operation, thereby improving the performance of the video processing model. It should be understood that the text network 340 is only used to assist the training process. In the reasoning stage after the training process is completed, only the video processing model 230 is used.

[0044] First, the training process is described. During the training process, reference data (e.g., labeled training data) can be used to determine the video processing model. Figure 3B Show more details. Figure 3B A block diagram 300B is shown that summarizes the process for training a video processing model according to some implementations of the present disclosure. Figure 3B As shown, the reference data 310 may include, for example, reference video data 340 and reference text data 316, that is, a true value label describing the reference video data 340. Further, reference language data 312 and reference visual data 314 may be extracted from the reference video data 340 according to the method described above.

[0045] It should be understood that in the context of the present disclosure, the language used by the reference video data 340 and the reference text data 316 is not limited, for example, the reference data 310 may be in Chinese, English, French, etc. Further, the multi-language conversion capability of LLM may be used to support reference data 310 involving different languages.

[0046] According to an example implementation of the present disclosure, during the training process, the language network 320 in the video processing model 230 can be used to extract the reference language representation of the reference language data 312, and the visual network 330 in the video processing model 230 can be used to extract the reference visual representation of the reference visual data. Further, a reference video representation can be generated based on the reference language representation and the reference visual representation. Using the example implementation of the present disclosure, a reference video representation can be generated based on various information, thereby helping the video processing model 230 to grasp more knowledge about the association relationship between the reference video data 240 and the reference text data 316.

[0047] See also Figure 4 Describes the training process in more detail. Figure 4 A block diagram 400 is shown of a detailed process for training a video processing model according to some implementations of the present disclosure. Figure 4 As shown, reference language data 312 may be input into a language network 320 , reference visual data 314 may be input into a visual network 330 , and reference language data 316 may be input into a text network 340 .

[0048] According to an example implementation of the present disclosure, a reference text representation of reference text data 316 can be extracted using a text network 340. The text network 340 may include, for example, an extraction network 430. Specifically, the extraction network 430 may be implemented using a text autoencoder, the goal of which is to decouple the text of the reference language data, and is mainly composed of a bidirectional self-encoder (Bidirectional Encoder Representations from Transformers, abbreviated BERT) from a converter and a text query former (Q-Former). The extraction network 430 can process the reference text data 316 to generate an intermediate representation. The intermediate representation can be further input to a processing network 432, and further processing is performed via a Q-Former 434.

[0049] In the training phase, the reference text data 312 may be input into the text autoencoder, and the text content may be encoded into a fixed-length reference text representation R by the text Q-Former. text Specifically, the above conversion operation can be performed based on the following formula 1:

[0050] R text =QF text (BERT(text))

[0051] Formula 1

[0052] In the above formula, R text Indicates the textual representation of the reference text data, QF text indicates the operation performed using the text Q-Former, and BERT indicates the operation performed using the BERT architecture, and text indicates the reference text data.

[0053] According to an example implementation of the present disclosure, in the process of extracting the reference visual representation, the visual network 330 can be used to extract a set of frame representations of a set of video frames from the reference video data, and then a flattening operation is performed on the set of frame representations to form the reference visual representation. In this way, when generating the reference video representation, the content of the video frames distributed at different positions of the reference video data can be comprehensively considered, thereby improving the accuracy of the video representation.

[0054] According to an example implementation of the present disclosure, a video base model can be constructed based on an existing image-text understanding model (e.g., BLIP-2). In order to facilitate video understanding, a cascaded Q-Former is proposed to integrate information from different video frames and automatic speech recognition. Assume that the input video frame is Where F, C, H, and W indicate the number of video frames, the number of image channels, the image height, and the image width, respectively. The BLIP-2 visual encoder and Q-Former can be used to extract the representation of each frame. Where Q i , E i Indicates the number of query tokens and embedding dimensions of the image Q-Former respectively. Then, a flattening operation can be performed on the representations from each video frame, and the visual representation portion of the reference video (ie, the reference visual representation) is obtained.

[0055] Specifically, Figure 4 As shown, the reference visual data 314 can be input to an extraction network 420 (e.g., based on VIT and image Q-Former implementation) to generate an intermediate representation. Further, the intermediate representation can be input to a processing network 422, and then input to a Q-Former. The reference visual representation can be determined based on the following formula:

[0056] R vision =QF img (VIT(frames))

[0057] Formula 2

[0058] In the above formula, R vision Indication reference visual representation, QF img indicates the image Q-Former, VIT indicates the integration operation, and frames indicates the visual data of each video frame.

[0059] For the reference language data 312, an extraction network 410 (e.g., based on the BERT architecture) may be used for processing. Specifically, the reference language representation of the reference language data may be determined using the following formula:

[0060] R asr =BERT(asr)

[0061] Formula 3

[0062] In the above formula, R asr indicates the reference language representation, BERT indicates the operations performed using the BERT architecture, and asr indicates the reference language data.

[0063] Then, the output reference language representation can be compared with R visionA join is performed to obtain information of the two modalities. According to an example implementation of the present disclosure, a video representation can be generated based on a language representation and a visual representation according to a cross-attention operation. Specifically, the video Q-Former 424 can be used and the information from the two modalities can be further extracted and fused through a cross-attention operation. In this way, information including two different modalities can be obtained, thereby describing the reference video data in a more accurate manner.

[0064] Finally, the reference video representation of the entire reference video data can be obtained Where Q v , E v Indicates the query number of the video Q-Former and the embedding dimension of the video Q-Former respectively. Specifically, the reference video representation can be determined based on the following formula 4:

[0065] R video =QF video (Concat(R vision , R asr ))

[0066] Formula 4

[0067] In the above formula, R video Instructions Reference Video Shows, QF video Instruction video Q-Former, Concat instruction connection operation, R vision Indicates reference visual representation, R asr Indicates reference language representation. According to an exemplary implementation of the present disclosure, R video It is divided into two parts, where the first part represents R for video-text contrastive learning video and the second part represents the identification for fine-grained modality alignment and video-based text reconstruction.

[0068] Having described the architecture of the video processing model 230, more details on determining the loss function in the training phase will be provided below. According to an example implementation of the present disclosure, in the process of determining the video processing model using reference data, a reference language representation of reference language data associated with the reference video data can be determined. The loss between the text reconstructed using the reference language representation and the reference text data is determined (for ease of distinction, the loss can be referred to as the first loss). Further, the language network in the video processing model can be updated based on the first loss. Specifically, the first loss can be determined based on the following formula:

[0069]

[0070] In the above formula, indicates the loss between the text reconstructed using the reference language representation and the reference text data, LLM indicates the large language model 450, and Proj indicates the projection operation used for reconstruction (for example, Figure 4 The projection 4442 shown), R text indicates the reference text representation, text indicates the reference text, and LLM().Loss indicates the use of a large language model to determine Proj(R text ) and the correlation loss between the text. Using the example implementation of the present disclosure, when updating the model, the correlation loss of the reconstructed text data can be considered, thereby improving the model optimization efficiency.

[0071] According to an example implementation of the present disclosure, in the process of determining the video processing model using reference data, a reference video representation of the reference video data can be determined, and then the loss between the text reconstructed using the reference video representation and the reference text data can be determined (for ease of distinction, the loss can be referred to as the second loss). Further, the language network and the visual network in the video processing model can be updated based on the second loss. Specifically, the second loss can be determined based on the following formula:

[0072]

[0073] In the above formula, indicates the loss between the text reconstructed using the reference video representation and the reference text data, LLM indicates the large language model 452, and Proj indicates the projection operation used for reconstruction (for example, Figure 4 The projection 440 shown, R video indicates the reference video representation, text indicates the reference text, and LLM().Loss indicates the use of a large language model to determine Proj(R video ) and text. Using the example implementation of the present disclosure, when updating the model, the correlation loss of the reconstructed video data can be considered, thereby improving the model optimization efficiency.

[0074] According to an example implementation of the present disclosure, a text network 340 for assisting the training process may be constructed. In the process of determining the text network, a reference text representation of the reference text data is determined; a loss between the reference video representation and the reference text representation is determined (for ease of distinction, the loss may be referred to as a third loss). Further, the video processing model in the text network and the video processing model may be updated based on the third loss. It should be understood that the text network and the language network share network parameters. Specifically, the third loss may be determined based on the following formula:

[0075]

[0076] In the above formula, indicates the loss between the reference video representation and the reference text representation, MSE indicates the mean square error operation, R video Indicates the reference video representation, R text Indicates the reference text representation. As mentioned earlier, the text autoencoder can convert the input text into a fixed-length intermediate representation, which contains key information for text reconstruction. For the video-to-text task, the goal is to generate text with an intermediate video representation. Therefore, the fixed-length intermediate representation after decoupling the text autoencoder can be used as an intermediate target for the video-based model. In this way, the video can be aligned with the corresponding text not only through video-based text reconstruction, but also through the consistency of hidden features, that is, fine-grained modality alignment. This representation is applied to text reconstruction and fine-grained modality alignment, which can be regarded as a soft hint that enables large language models to generate target text.

[0077] According to an example implementation of the present disclosure, a loss between a projection of a reference language representation and a projection of a reference text representation may be determined (for ease of distinction, the loss may be referred to as a fourth loss). Further, the video processing model in the text network and the video processing model may be updated based on the fourth loss. Specifically, the fourth loss may be determined based on the following formula:

[0078]

[0079] In the above formula, indicates the loss between the projection of the reference language representation and the projection of the reference text representation, and Proj indicates the projection operation used for reconstruction (e.g., Figure 4 The projections 416 and 436 shown), R video Indicates the reference video representation, R text Indicates the reference text representation, Contra represents determining the contrast loss (i.e., shortening the distance between representations with similar projections and pushing away the distance between representations with dissimilar projections). Using the example implementation of the present disclosure, the distance between the reconstruction using the reference language representation and the reconstruction using the reference text representation can be fully considered, thereby improving the accuracy of the video processing model.

[0080] According to an exemplary implementation of the present disclosure, the multiple losses described above may be used in combination to perform the training process. For example, the training may be performed based on the following formula 9 or 10.

[0081]

[0082]

[0083] In the above formula, Represents the overall loss function, λ1 and λ2 represent weights, and the meanings of other symbols are the same as described above. In this way, multiple losses can be fully considered during the training process, thereby improving the accuracy of the video processing model.

[0084] According to an example implementation of the present disclosure, the training process may be performed in three stages. Figure 5A A block diagram 500A shows pseudo code of the first stage training algorithm according to some implementations of the present disclosure. As shown in the first paragraph of Algorithm 1, the training data set may include 1.9M sentences, covering 0.5M abstracts (long sentences) and 1.5M titles (short sentences). It should be understood that the sentences of the test set are not included here. Figure 5A The algorithm shown is used to train networks involving text (e.g., text networks and language networks).

[0085] As shown in paragraph 2, the trained model may include a text autoencoder (e.g., the language network and text network described above), mainly covering BERT and text Q-Former (e.g., 5 layers). The model involves approximately 150M trainable parameters, of which 110M parameters are shared with the visual part. As shown in paragraph 3, the input data is text. As shown in paragraphs 4 to 8, the training process may be performed in a loop until the convergence condition defined in paragraph 8 is reached.

[0086] As shown in Section 5, the characteristic decoupling process can be performed, in which case R text =QF text (BERT(text)), where QF text represents the operation performed using the text Q-Former. As shown in paragraph 6, the loss can be determined based on the following formula: Where Proj represents a linear projection operation, LLM represents a large language model, and LLM().Loss represents an operation of determining a loss performed using a large language model. As shown in paragraph 7, back propagation can be performed based on the loss determined in the above manner and the weights of the model can be updated.

[0087] Figure 5B A block diagram 500B shows pseudo code of the second stage training algorithm according to some implementations of the present disclosure. As shown in the first paragraph of Algorithm 2, the training data set can use the WebVid2.5M data set, covering 2M (video, text) pairs; CC3M data set, covering 2.M (image, text) pairs, and at this time, the image can be considered as a static video.

[0088] As shown in paragraph 2, the trained models can include text autoencoders (e.g., the language network and text network described above), mainly covering BERT and Q-Former (e.g., 5 layers). Video base models, mainly covering image Q-Former, video Q-Former (5 layers) and ASR BERT, where ASR BERT is shared by both parts. There are about 307M trainable parameters, of which only 40M parameters belong uniquely to the text autoencoder.

[0089] As shown in paragraph 3, the input data is (language data asr, visual data frames, text data text), where the language data is, for example, data from ASR processing, the visual data is, for example, individual video frames from video data, and the text data is, for example, a true value label. Here, "none" can be used to represent the asr of the video in the training data set. As shown in paragraphs 4 to 17, the training process can be performed in a loop until the convergence condition defined in paragraph 17 is reached. Specifically, the pipeline executed for the text autoencoder is shown in paragraphs 5 to 7, which is similar to Algorithm 1 above and is not repeated here.

[0090] The processing pipeline for video is shown in sections 9 to 15. In section 10, the data for asr may include: asr = BERT(asr), that is, using the BERT network to process the asr data to obtain the corresponding representation R asr In paragraph 11, the processing of the video frame may include: R vision =QF img (VIT(frames)), where QF img represents the image Q-Former implemented using the BLIP-2 algorithm. The video feature decoupling operation is shown in section 12: R video =QF video (Concat(R vision , R asr )). The process of determining loss is set out in paragraph 13: Indicates that from R video The first part and R text Determined contrast loss,

[0091] The process of determining the overall training loss is shown in paragraphs 14-15: That is, the overall training loss can be determined based on the sum of the losses determined in the two pipelines above. At paragraph 16, back propagation can be performed based on the loss determined in the above manner and the weights of the model can be updated.

[0092] According to an example implementation of the present disclosure, the training process may include a third stage, the algorithm of which is similar to the second stage, except that a different data set is used. At this point, the MSR-VTT and VIDEO-CSR data sets may be used to perform training.

[0093] According to an example implementation of the present disclosure, after the training process has been completed, the trained video processing model 230 can be used to determine the video representation of the video data to be processed. At this point, the data flow is similar to the training process described above. Figure 4 ,at this time Figure 4 The reference language data 312 in will be replaced by the language data extracted from the video data to be processed, and the reference visual data 314 will be replaced by the visual data extracted from the video data to be processed. According to an example implementation of the present disclosure, the text network 340 is only used for auxiliary training purposes and is not involved in the reasoning process.

[0094] According to an example implementation of the present disclosure, in the process of determining the video representation, the language network in the video processing model can be used to extract the language representation of the language data. The visual network in the video processing model can be used to extract the visual representation of the visual data. Further, the video representation can be generated based on the language representation and the visual representation. According to an example implementation of the present disclosure, in the process of extracting the visual representation, the visual network can be used to respectively extract a set of frame representations of a set of video frames; then a flattening operation is performed on the set of frame representations to form a visual representation. According to an example implementation of the present disclosure, in the process of generating the video representation, the video representation can be generated based on the language representation and the visual representation according to the cross-attention operation.

[0095] Using the example implementation of the present disclosure, the reasoning process can use the association between the relevant video data and its corresponding text description grasped by the video processing model 230 to obtain a video representation that is more conducive to the purpose of video understanding.

[0096] According to an example implementation of the present disclosure, the reference data further includes a precision level of the reference text data, and determining the video processing model further includes: using the reference data to determine the video processing model, so that the video processing model further describes the association relationship between the reference text data and the precision level.

[0097] Specifically, the reference text data in the reference data may have different precision levels. For example, a lower precision level may correspond to shorter reference text data, i.e., shorter sentences; a medium precision level may correspond to medium-length reference text data; and a higher precision level may correspond to longer reference text data, i.e., long sentences. In this way, the video processing model may further learn about different precision levels, and then output a result that meets the specified precision level in the inference stage.

[0098] According to an example implementation of the present disclosure, a video processing model can be used to generate a video representation that matches a predetermined level of accuracy. Specifically, the video representation can be input to a downstream processing model downstream of the video processing model, and then the video representation can be used to perform the processing tasks specified by the downstream processing model. In this way, a visual representation including rich information can be used to perform downstream processing tasks related to media content understanding, thereby improving the performance of downstream processing tasks.

[0099] See also Figure 6 Describe in more detail the Figure 6 A block diagram 600 is shown for performing downstream processing tasks using video representations according to some implementations of the present disclosure. Figure 6 As shown, the downstream model 620 can perform a variety of tasks, for example, including but not limited to: a task of extracting a video title, a task of generating a video summary, a video retrieval task, etc. The video data 610 to be processed and the desired accuracy level 612 can be input.

[0100] When the downstream task is to extract the video title, the precision level 612 may indicate a lower precision, and the video processing model 230 may generate a video representation 240 that matches the precision level 612. Then, the downstream model 620 may output a corresponding video title, such as "a rugby match". When the downstream task is to generate a video summary, the precision level 612 may indicate a medium precision, and the video processing model 230 may generate a video representation 240 that matches the precision level 612. Then, the downstream model 620 may output a corresponding video summary, such as "two groups of men...", etc.

[0101] According to an example implementation of the present disclosure, the technical solution described above can be used to perform various downstream tasks. In the following, performance parameters related to the execution of downstream tasks are provided. Regarding data sets, the video understanding capabilities of the proposed technical solution can be tested using MSR-VTT and Video-CSR data sets. MSR-VTT is a large-scale video understanding benchmark, particularly for mapping videos to text. It covers 10K videos, each with approximately 20 natural semantic annotations. There are 6513 videos for training and 2990 videos for testing.

[0102] It should be understood that since the MSR-VTT dataset does not provide audio recognition data, "empty" can be used to replace the asr data. Video-CSR is a newly released large-scale video understanding benchmark that covers approximately 5,000 videos ranging in length from 15 seconds to 1 minute, each with manual annotations. Each video includes 5 titles and 5 summaries. Video-CSR also provides audio recognition data for the video, and if the ASR of the video involves only less information, the asr data can be replaced with "empty". The videos in this dataset can be divided into two parts, one with rich audio recognition data information and the other with very little audio recognition data information.

[0103] For the MSR-VTT experiment pre-training, WebVid-2.5M and CC3M can be used. In the Video-CSR experiment, a dataset consisting of 100k videos can be used, where each video has 5 subtitles and 5 summaries generated by LLM using the video’s metadata.

[0104] Regarding model configuration, the vision model can be built directly from the pre-trained BLIP-2, thereby leveraging its extensive image prior knowledge. For text autoencoders, BERT can be used. base to process the original input. Then, from BERT base Initialized 5-layer text Q-Former (only first 5 layers are used) using 33 learnable (first for video-text contrastive learning) queries to project text into fixed-length representations with cross-attention. For video Q-Former, we can start with pre-trained BERT with 33 learnable query tokens. base For audio recognition data, the BERT model can be shared with the text autoencoder, with approximately 307M trainable parameters.

[0105] Regarding training and evaluation, for the first stage, in the text autoencoder pre-training stage, 32 Tesla-V100-SXM2-32GB GPUs can be used with a batch size of 8 on each GPU to perform two epochs of training. For the second stage, 64 Tesla-V100-SXM2-32GB GPUs can be used with a batch size of 8 on each GPU (6 for video summarization) to perform two epochs of training. For the third stage, 32 Tesla-V100-SXM2-32GB GPUs can be used to perform three epochs of fine-tuning (10 epochs of fine-tuning in the Video-CSR experiment).

[0106] See also Figure 7 Describes the experimental effect of an example implementation method according to the present disclosure, Figure 7 A block diagram 700 is shown for performing the effects of downstream processing tasks according to some implementations of the present disclosure. Figure 7 As shown in FIG. 1 , “Video-Teller” represents a video processing method according to an example implementation of the present disclosure. In the video title extraction experiment performed on the MSR-VTT dataset, “align” represents fine-grained alignment in the training phase, and SCST represents self-key sequence training. Figure 7 It can be seen that the proposed video processing method can achieve better performance.

[0107] further, Figure 8 Block diagram 800 is shown for performing the effects of downstream processing tasks according to some implementations of the present disclosure. Figure 8 The video title extraction results performed on the Video-CSR dataset are shown. Figure 8 It can be seen that the proposed video processing method can achieve better performance.

[0108] In summary, according to an example implementation of the present disclosure, a video understanding model with fine-grained modality alignment is used. Specifically, the image-text understanding model is extended to video understanding by proposing a hierarchical Q-Former. At the same time, ASR information is introduced, and the ASR information and visual information are integrated into a whole through the hierarchical Q-Former, which are used together for video understanding. Furthermore, the text representation is decoupled using a text autoencoder, and the resulting representation can be fine-grained aligned with the video representation to assist model training, thereby improving the text generation accuracy of the model. In addition, a controllable video description generation method is proposed, which can achieve video descriptions of different granularities.

[0109] According to an example implementation of the present disclosure, the proposed fine-grained modality alignment training method can train a video understanding model, which can improve the accuracy of the model's description of video content while increasing training overhead. There are obvious improvements in various test indicators. Using the example implementation of the present disclosure, a rough description of the video or a detailed description of the video can be achieved by simply using different video representations, thereby adapting to more business scenarios.

[0110] Example Process

[0111] Fig. 9 A flow chart of a method 900 for processing video data according to some implementations of the present disclosure is shown. At block 910, in response to receiving the video data, language data is obtained from the video data. At block 920, visual data is obtained from the video data. At block 930, a video representation for indicating the video data is determined based on the language data and the visual data according to a video processing model.

[0112] According to an example implementation of the present disclosure, acquiring language data includes at least any one of the following: acquiring language data based on voice data in video data; and acquiring language data based on subtitle data in video data.

[0113] According to an example implementation of the present disclosure, acquiring visual data includes: selecting a group of video frames from a plurality of video frames in the video data according to a predetermined condition; and determining visual data based on the group of video frames.

[0114] According to an example implementation of the present disclosure, determining a video representation includes: extracting a language representation of language data using a language network in a video processing model; extracting a visual representation of visual data using a visual network in the video processing model; and generating a video representation based on the language representation and the visual representation.

[0115] According to an example implementation of the present disclosure, extracting a visual representation includes: utilizing a visual network to respectively extract a set of frame representations of a set of video frames; and performing a flattening operation on the set of frame representations to form a visual representation.

[0116] According to an example implementation of the present disclosure, generating a video representation includes generating the video representation based on a language representation and a visual representation according to a cross-attention operation.

[0117] According to an example implementation of the present disclosure, the method further includes acquiring the video processing model by: determining the video processing model using reference data, the reference data including reference video data and reference text data.

[0118] According to an example implementation of the present disclosure, determining a video processing model using reference data includes: determining a reference language representation of reference language data associated with reference video data; determining a first loss between text reconstructed using the reference language representation and the reference text data; and updating the video processing model based on the first loss.

[0119] According to an example implementation of the present disclosure, determining a video processing model using reference data includes: determining a reference video representation of the reference video data; determining a second loss between text reconstructed using the reference video representation and the reference text data; and updating the video processing model based on the second loss.

[0120] According to an example implementation of the present disclosure, the method further includes: determining a text network by the following, determining a reference text representation of reference text data; determining a third loss between a reference video representation and a reference text representation; and updating the text network and the video processing model based on the third loss, the text network and the language network sharing network parameters.

[0121] According to an example implementation of the present disclosure, the method further includes: determining a fourth loss between the projection of the reference language representation and the projection of the reference text representation; and updating the video processing model in the text network and the video processing model based on the fourth loss.

[0122] According to an example implementation of the present disclosure, the reference data further includes a precision level of the reference text data, and determining the video processing model further includes: using the reference data to determine the video processing model, so that the video processing model further describes the association relationship between the reference text data and the precision level.

[0123] According to an example implementation of the present disclosure, the method further includes: utilizing a video processing model to generate a video representation matching a predetermined level of accuracy.

[0124] According to an example implementation of the present disclosure, the method further includes: inputting the video representation to a downstream processing model downstream of the video processing model; and using the video representation to perform a processing task specified by the downstream processing model.

[0125] Example devices and equipment

[0126] Fig.10A block diagram of an apparatus 1000 for processing video data according to some implementations of the present disclosure is shown. The apparatus 1000 includes: a first acquisition module 1010, configured to acquire language data from the video data in response to receiving the video data; a second acquisition module 1020, configured to acquire visual data from the video data; and a determination module 1030, configured to determine a video representation for indicating the video data based on the language data and the visual data according to a video processing model.

[0127] According to an example implementation of the present disclosure, the first acquisition module includes at least any one of the following: a voice acquisition module configured to acquire language data based on voice data in video data; and a subtitle acquisition module configured to acquire language data based on subtitle data in video data.

[0128] According to an example implementation of the present disclosure, the second acquisition module includes: a selection module configured to select a group of video frames from multiple video frames in the video data according to predetermined conditions; and a frame-based determination module configured to determine visual data based on a group of video frames.

[0129] According to an example implementation of the present disclosure, the determination module includes: a first calling module, configured to utilize a language network in a video processing model to extract a language representation of language data; a second calling module, configured to utilize a visual network in the video processing model to extract a visual representation of visual data; and a generation module, configured to generate a video representation based on the language representation and the visual representation.

[0130] According to an example implementation of the present disclosure, the second calling module includes: a frame representation determination module, configured to utilize a visual network to respectively extract a set of frame representations of a set of video frames; and a formation module, configured to perform a flattening operation on a set of frame representations to form a visual representation.

[0131] According to an example implementation of the present disclosure, the generation module includes: an attention module configured to generate a video representation based on a language representation and a visual representation according to a cross-attention operation.

[0132] According to an example implementation of the present disclosure, it further includes: a training module configured to obtain a video processing model by: using reference data to determine the video processing model, the reference data includes reference video data, and reference text data.

[0133] According to an example implementation of the present disclosure, the training module includes: a reference language representation determination module configured to determine a reference language representation of reference language data associated with reference video data; a first loss determination module configured to determine a first loss between text reconstructed using the reference language representation and the reference text data; and an update module configured to update the video processing model based on the first loss.

[0134] According to an example implementation of the present disclosure, the training module includes: a reference video representation determination module configured to determine a reference video representation of reference video data; a second loss determination module configured to determine a second loss between text reconstructed using the reference video representation and reference text data; and an update module configured to update the video processing model based on the second loss.

[0135] According to an example implementation of the present disclosure, the apparatus further includes: a text network training module configured to determine a text network, the text network training module including: a reference text representation determination module configured to determine a reference text representation of reference text data; a third loss determination module configured to determine a third loss between a reference video representation and a reference text representation; and an update module configured to update the text network and the video processing model based on the third loss, the text network and the language network sharing network parameters.

[0136] According to an example implementation of the present disclosure, the apparatus further includes: a fourth loss determination module configured to determine a fourth loss between a projection of a reference language representation and a projection of a reference text representation; and an update module configured to update a text network and a video processing model in the video processing model based on the fourth loss.

[0137] According to an example implementation of the present disclosure, the reference data further includes the accuracy level of the reference text data, and the training module further includes: an accuracy-based training module configured to use the reference data to determine the video processing model, so that the video processing model further describes the association relationship between the reference text data and the accuracy level.

[0138] According to an example implementation of the present disclosure, the apparatus further includes: a precision-based generation module configured to generate a video representation matching a predetermined precision level using a video processing model.

[0139] According to an example implementation of the present disclosure, the apparatus further includes: an input module configured to input a video representation to a downstream processing model downstream of the video processing model; and an execution module configured to utilize the video representation to execute a processing task specified by the downstream processing model.

[0140] Fig.11 1 shows a block diagram of a device 1100 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig.11 The computing device 1100 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig.11 The computing device 1100 shown may be used to implement the methods described above.

[0141] like Fig.11 As shown, computing device 1100 is in the form of a general-purpose computing device. Components of computing device 1100 may include, but are not limited to, one or more processors or processing units 1110, memory 1120, storage device 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processing unit 1110 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 1120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 1100.

[0142] The computing device 1100 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1120 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1130 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 1100.

[0143] The computing device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.11 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 1120 may include a computer program product 1125 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0144] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 1100 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1100 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0145] Input device 1150 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1160 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 1100 may also communicate with one or more external devices (not shown) through communication unit 1140 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 1100, or communicate with any device that allows computing device 1100 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0146] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0147] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0148] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0149] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0150] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0151] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing video data, comprising: In response to receiving the video data, acquiring language data from the video data; Acquiring visual data from the video data; as well as According to a video processing model, a video representation for indicating the video data is determined based on the language data and the visual data.

2. The method according to claim 1, wherein obtaining the language data comprises at least one of the following: Based on the voice data in the video data, acquiring the language data; and The language data is acquired based on subtitle data in the video data.

3. The method according to claim 1, wherein acquiring the visual data comprises: Selecting a group of video frames from a plurality of video frames in the video data according to a predetermined condition; as well as Based on the set of video frames, the visual data is determined.

4. The method of claim 3, wherein determining the video representation comprises: extracting a language representation of the language data using a language network in the video processing model; extracting a visual representation of the visual data using a visual network in the video processing model; as well as Based on the language representation and the visual representation, the video representation is generated.

5. The method of claim 4, wherein extracting the visual representation comprises: Using the visual network, extracting a set of frame representations of the set of video frames respectively; as well as A flattening operation is performed on the set of frame representations to form the visual representation.

6. The method of claim 1 , wherein generating the video representation comprises: The video representation is generated based on the language representation and the visual representation according to a cross-attention operation.

7. The method according to claim 1 further comprises obtaining the video processing model by: determining the video processing model using reference data, wherein the reference data comprises reference video data and reference text data.

8. The method of claim 7, wherein using the reference data to determine the video processing model comprises: determining a reference language representation of reference language data associated with the reference video data; determining a first loss between text reconstructed using the reference language representation and the reference text data; as well as The video processing model is updated based on the first loss.

9. The method of claim 8, wherein using the reference data to determine the video processing model comprises: determining a reference video representation of the reference video data; determining a second loss between text reconstructed using the reference video representation and the reference text data; as well as The video processing model is updated based on the second loss.

10. The method according to claim 8, further comprising: The text network is determined by, determining a reference text representation of the reference text data; determining a third loss between the reference video representation and the reference text representation; as well as The text network and the video processing model are updated based on the third loss, and the text network and the language network share network parameters.

11. The method according to claim 10, further comprising: determining a fourth loss between the projection of the reference language representation and the projection of the reference textual representation; as well as The video processing model in the text network and the video processing model is updated based on the fourth loss.

12. The method of claim 7, wherein the reference data further comprises a level of accuracy of the reference text data, and determining the video processing model further comprises: The video processing model is determined using the reference data, so that the video processing model further describes the association relationship between the reference text data and the accuracy level.

13. The method according to claim 12, further comprising: The video processing model is utilized to generate the video representation that matches a predetermined level of accuracy.

14. The method according to claim 1, further comprising: inputting the video representation into a downstream processing model downstream of the video processing model; as well as The video representation is utilized to perform processing tasks specified by the downstream processing model.

15. An apparatus for processing video data, comprising: a first acquisition module, configured to acquire language data from the video data in response to receiving the video data; A second acquisition module is configured to acquire visual data from the video data; as well as A determination module is configured to determine a video representation indicating the video data based on the language data and the visual data according to a video processing model.

16. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processing unit.

17. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 14.

Images