Method, apparatus, electronic device and computer program product for video processing

Abstract

Embodiments of the present disclosure relate to a method, apparatus, electronic device and computer program product for video processing. The method comprises determining a temporal feature and a spatial feature of a video. In addition, the method further comprises generating a text output corresponding to the video based on the temporal feature and the spatial feature using a spatio-temporal query conversion module, the spatio-temporal query conversion module comprising a temporal query and a corresponding temporal expert network and a spatial query and a corresponding spatial expert network. Thus, by utilizing the spatio-temporal query conversion module, embodiments of the present disclosure can effectively generate a spatio-temporal semantic alignment representation of an input video, enhance the visual language alignment capability of a video processing model, make it better understand the video content, and thus improve the effect of the video processing model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to methods, apparatus, electronic devices, and computer program products for video processing. Background Technology

[0002] With the continuous development of artificial intelligence technology, generative language models are becoming increasingly important. These models can automatically generate high-quality, natural, and fluent text, bringing new possibilities to various application fields. From natural language processing to creative content generation, generative language models play a key role in dialogue systems, text summarization, translation, and creative copywriting.

[0003] As the demand for multimodal intelligence continues to grow, multimodal models that integrate generative language models and image processing models are becoming increasingly important. These models can simultaneously process multiple data types, such as text and images, thereby more accurately understanding and expressing complex semantic information. In fields such as natural language processing, computer vision, and intelligent dialogue, multimodal models offer new possibilities for achieving more comprehensive and intelligent human-computer interaction. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, electronic device, computer program product, and medium for video processing.

[0005] According to a first aspect of this disclosure, a video processing method is provided. The method includes determining the temporal and spatial features of the video. Furthermore, the method includes generating text output corresponding to the video using a spatiotemporal query transformation module based on the temporal and spatial features. The spatiotemporal query transformation module includes a temporal query and a corresponding temporal expert network, as well as a spatial query and a corresponding spatial expert network.

[0006] According to a second aspect of this disclosure, a video processing apparatus is provided. The apparatus includes a spatiotemporal feature determination module configured to determine temporal and spatial features of the video. Furthermore, the apparatus includes a text output generation module configured to generate text output corresponding to the video based on the temporal and spatial features using a spatiotemporal query transformation module, the spatiotemporal query transformation module including a time query and a corresponding time expert network, and a spatial query and a corresponding spatial expert network.

[0007] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to the first aspect.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer program product is tangibly stored on the non-transitory computer-readable medium and includes computer-executable instructions that, when executed, cause a computer to perform the steps of the method of the first aspect of this disclosure.

[0009] In a fifth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.

[0010] The summary section is intended to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram of an example environment in which the apparatus and / or methods according to embodiments of the present disclosure may be implemented is shown;

[0013] Figure 2 A flowchart of a video processing method according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram illustrating the process of processing video using a video processing model according to an embodiment of the present disclosure is shown;

[0015] Figure 4A A schematic diagram of the structure of a spatiotemporal query conversion module according to an embodiment of the present disclosure is shown;

[0016] Figure 4B A schematic diagram of a cross-attention mask matrix according to an embodiment of the present disclosure is shown;

[0017] Figure 5 A schematic diagram illustrating the pre-training process of a video processing model according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A schematic diagram illustrating the process of interacting with a video processing system according to an embodiment of the present disclosure is shown;

[0019] Figure 7 A block diagram of a video processing apparatus according to embodiments of the present disclosure is shown; and

[0020] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0021] In all the accompanying figures, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0025] As mentioned earlier, multimodal models play an important role in many fields, and they perform well when processing text and image data. However, when it comes to processing video data, multimodal models perform poorly. Analysis revealed that video data, due to its simultaneous spatial and temporal features, often presents challenges in semantic feature alignment, leading to the poor performance of multimodal models in video processing.

[0026] To this end, embodiments of this disclosure propose a video processing scheme. The scheme first generates the temporal and spatial features of the video, and then uses the temporal and spatial features of the video to generate text output corresponding to the video using a spatiotemporal query transformation module. The spatiotemporal query transformation module includes a time query and a corresponding time expert network, as well as a spatial query and a corresponding spatial expert network.

[0027] The embodiments of this disclosure utilize a spatiotemporal query transformation module, leveraging temporal queries and corresponding temporal expert networks, as well as spatial queries and corresponding spatial expert networks, to effectively generate spatiotemporal semantically aligned representations of the input video. This significantly enhances the visual-language alignment capability of the video processing model, enabling it to better understand video content. As the model's understanding of video input improves, the performance of the video processing model also significantly increases. The spatiotemporal query transformation module not only improves the performance of the processing model but also lays the foundation for subsequent research and applications in multimodal video processing.

[0028] Figure 1 A schematic diagram of an example environment in which the apparatus and / or methods according to embodiments of the present disclosure may be implemented is shown. Figure 1 As shown, example environment 100 may include computing device 110, which may be a user terminal, mobile device, computer, etc., or it may be a computing system, a single server, a distributed server, or a cloud-based server. Computing device 110 can receive video 120. Embodiments of this disclosure can process multimodal data, namely video modal data and text modal data. Multimodal data refers to a collection of data containing multiple types or forms, which may come from different sensors, devices, or sources, and typically include at least two of the following forms: text, images, audio, video, etc.

[0029] The computing device 110 may include a video processing system 130, which can generate temporal features 132 and spatial features 134 of the video 120. Unlike still images, videos not only have spatial information of image frames but also temporal information between image frames. Therefore, it is necessary to capture not only the spatial features 134 of the static scene of the video 120 but also its temporal features 132 that change over time. The spatiotemporal query conversion module 136 can semantically align the temporal features 132 and spatial features of the video 120. The spatiotemporal query conversion module 136 may include a temporal query 138 and a corresponding temporal expert network 140, a spatial query 142 and a corresponding spatial expert network 144. The video processing system 130 can generate text output 150 corresponding to the video 120. The video processing system 130 can generate text output 150 corresponding to the video 120 based on temporal features 132 and spatial features 134 using the spatiotemporal query conversion module 136. The spatiotemporal query conversion module 136 includes a time query 138 and a corresponding time expert network 140, as well as a spatial query 142 and a corresponding spatial expert network 144.

[0030] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functionalities.

[0031] The following will combine Figures 2 to 8 The process according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0032] Figure 2 A flowchart of a video processing method 200 according to an embodiment of the present disclosure is shown. At block 202, temporal and spatial characteristics of the video can be determined. For example, refer to... Figure 1 The video processing system 130 can determine the temporal characteristics 132 and spatial characteristics 134 of the video 120.

[0033] At box 204, a spatiotemporal query transformation module can be used to generate text output corresponding to the video based on temporal and spatial features. This module includes a time query and its corresponding temporal expert network, as well as a spatial query and its corresponding spatial expert network. For example, refer to... Figure 1 The video processing system 130 can generate text output 150 corresponding to the video 120 based on time features 132 and spatial features 134 using the spatiotemporal query conversion module 136. The spatiotemporal query conversion module 136 includes time query 138 and the corresponding time expert network 140, as well as spatial query 142 and the corresponding spatial expert network 144.

[0034] Therefore, according to the method 200 of the embodiments of this disclosure, by utilizing the spatiotemporal query transformation module, a spatiotemporal semantically aligned representation of the input video can be effectively generated, significantly enhancing the visual-language alignment capability of the video processing model and enabling it to better understand the video content. As the model's understanding of the video input improves, the performance of the video processing model also significantly improves. Furthermore, the spatiotemporal query transformation module not only improves the performance of the processing model but also helps lay the foundation for subsequent research and applications in multimodal video processing.

[0035] Figure 3 A schematic diagram of a process 300 for processing video using a video processing model according to an embodiment of the present disclosure is shown. Figure 3As shown, video 302 can be input into the visual encoder. The embodiments of this disclosure do not limit the format of video 302 and can be applied to various video formats, including but not limited to MP4, AVI, MOV, etc. Regardless of the device or software that generated the video data, a corresponding semantically aligned representation can be generated through the spatiotemporal query conversion module. The visual encoder 304 can receive video 302 and generate corresponding visual features or visual representations. A visual encoder is a neural network model used for image processing and computer vision tasks. It can be used as a visual feature extractor, converting input image data into high-dimensional feature vectors containing semantic and visual information of the image. These feature vectors can be used for various tasks, such as image classification, object detection, and image semantic segmentation.

[0036] The attention pooling module 306 can receive visual features from video 302 and generate spatial features 308 and temporal features 310. The attention pooling module 306 can be used to decouple the spatiotemporal features of the video. Explicit modeling of the spatiotemporal features of a video is crucial for a language model to effectively understand video content, allowing the language model to capture rich semantic information, dynamic changes, and contextual cues, thereby enhancing the language model's ability to understand video content. The attention pooling module 306 consists of a cross-attention layer and a feedforward layer, and can acquire the spatial features 308 and temporal features 310 of video 302 through a learnable pooling process. For example, suppose the input video is... Where T is the frame number, and H, W, and C represent the height, width, and number of channels of each image frame. After video 302 passes through visual encoder 304, an initial video embedding is generated. Where N represents the number of image patches in each frame, and D represents the feature dimension.

[0037] To extract spatiotemporal features from video 302, the attention pooling module 306 can introduce two attention pooling queries to learn how to extract the corresponding features. Specifically, a temporal pooling query can be used. To embed video Perform cross-attention operations to generate temporal features 310 This process can be represented by formulas (1) and (2):

[0038]

[0039]

[0040] Where CA(Q) s Let x, x' represent a cross-attention network, and FFN(·) represent a feedforward neural network. Similarly, spatial pooling queries can be utilized. Transpose of video embedding A cross-attention operation is performed to generate spatial features 308, which can be represented by formulas (3) and (4):

[0041]

[0042]

[0043] like Figure 3 As shown, the spatiotemporal query transformation module 312 can utilize time query 314, spatial query 316, and fusion query 318 to process spatial feature 308 and temporal feature 310. For example, time query 314, spatial query 316, and fusion query 318 can be viewed as feature extractors to extract corresponding features from spatial feature 308 and temporal feature 310 to achieve semantic alignment. In some embodiments, the feature dimensions of time query 314 and spatial query 316 can be 32-dimensional, and the dimension of the fusion query can be 1-dimensional. In some embodiments, only time query 314 and spatial query 316 may be included, without fusion query 318.

[0044] The spatiotemporal query transformation module 312 can generate video features 320 of video 302, and align the dimensions of video features 320 with the dimensions of language model 324 through a multilayer perceptron (MLP) layer 322. For example, the video feature dimension can be 1024 dimensions, while the input dimension of language model 324 is 4096 dimensions. Therefore, the MLP layer 322 can be used to convert the dimensions of video features 320 into the input dimensions of language model 324 for further processing to generate output 328. Furthermore, the video feature dimension and the input dimension of the language model can be other numbers, and the embodiments of this disclosure are not limited in this regard. For example, output 328 can be a summary or description of video 302. In some embodiments, prompt content 326 can also be input to language model 324 to instruct language model 324 to generate the output indicated by prompt content 326. For example, prompt content 326 can be "What are the people in the video doing?", then language model 324 can generate the corresponding output 328 based on its understanding of the video content. For example, output 328 could be "A girl is feeding a boy a piece of pizza," and this process can be viewed as a video question-answering task. Furthermore, the video processing model disclosed herein can also perform other tasks, including but not limited to multimodal instruction following, video object localization, video description generation, and video summary generation, etc.

[0045] Figure 4A A schematic diagram of the structure 400A of a spatiotemporal query conversion module according to an embodiment of the present disclosure is shown. Figure 4AAs shown, video 402 can generate visual features through visual encoder 404, and spatial features 408 and temporal features 410 through attention pooling module 406. Spatiotemporal query transformation module 412 can connect the visual encoder and the language model to bridge the gap between visual representation and language patterns. Specifically, spatiotemporal query transformation module 412 can include three expert networks: a temporal expert network 414, a spatial expert network 416, and a fusion expert network 418. In spatiotemporal query transformation module 412, each expert network is an independent sub-model, and each expert network is responsible for processing a specific subspace or subtask. During training, spatiotemporal query transformation module 412 dynamically combines the outputs of various expert networks by selecting and weighting them to effectively process the temporal features, spatial features, and spatiotemporal fusion features in the video data.

[0046] The spatiotemporal query conversion module 412 may further include a learnable spatial query 420, a temporal query 422, and a fusion query 424. In some embodiments, the spatial query 420, temporal query 422, and fusion query 424 can respectively extract the spatial representation, temporal representation, and spatiotemporal fusion representation of the video 402. In some embodiments, a spatial query 420 with a feature dimension of 32 dimensions, a temporal query 422 with a feature dimension of 32 dimensions, and a fusion query 424 with a feature dimension of 1 dimension can be used. It should be understood that query vectors of other dimensions can also be used, and the embodiments of this disclosure do not limit the dimension of the vectors. Furthermore, in some embodiments, the spatiotemporal query conversion module 412 may not include the fusion query 424 and the corresponding fusion expert network 418.

[0047] Spatial query 420, temporal query 422, and fusion query 424 interact through self-attention network 426 and through cross-attention network 428 with spatial features 408 and temporal features 410 of video 402. Furthermore, each query can also interact with video description 430 through the same self-attention network 426. Video description 430 can be descriptive text corresponding to video 402. For example, video description 430 could be "A girl is feeding a boy a piece of pizza." Feedforward neural network 432 can receive the output of self-attention network 426 to further encode video description 430. To ensure that each query can perform cross-attention operations with the corresponding features, embodiments of this disclosure design a cross-attention mask matrix in cross-attention network 428, which will be discussed below. Figure 4B Describe the cross-attention mask.

[0048] Figure 4B A schematic diagram of a cross-attention mask matrix 400B according to an embodiment of the present disclosure is shown. Figure 4BAs shown, gray blocks represent corresponding positions in the matrix that are "1", and white blocks represent corresponding positions that are "0". Cross-attention masks control the visibility of spatial and temporal queries across various spatiotemporal features. Spatial and temporal queries are limited to focusing on their respective features, while fusion queries can focus on all features. For example, the mask corresponding to spatial query 450 and spatial feature 452 is 1, indicating that spatial query 450 is visible to spatial feature 452. Furthermore, the mask corresponding to spatial query 450 and temporal feature 462 is 0, indicating that spatial query 450 is not visible to temporal feature 462. Similarly, the mask corresponding to temporal feature 460 and temporal feature 462 is 1, but the mask corresponding to spatial feature 452 is 0. Moreover, the masks corresponding to both spatial feature 452 and temporal feature 462 for fusion query 470 are 1, because fusion query 470 can focus on both spatial feature 452 and temporal feature 462 simultaneously. The design of cross-attention masks enables precise control over the allocation of attention when processing spatiotemporal features, thereby more effectively capturing important information in video data.

[0049] Return to reference Figure 4A Different expert networks were used to process different query vectors in parallel. Furthermore, L... VTM Loss function 434, L VTC Loss function 436 and L VTG Loss function 438 is used to train the spatiotemporal query transformation module 412. For example, L VTM Loss function 434 is used to represent the video text matching (VTM) loss function, L VTC Loss function 436 is used to represent the video text contrast (VTC) loss function, L VTG Loss function 438 is used to represent the loss function for Video Localization Text Generation (VTG).

[0050] Figure 5 A schematic diagram of a pre-training process 500 for a video processing model according to an embodiment of the present disclosure is shown. Figure 5 As shown, at box 502, the spatiotemporal query transformation module is trained independently, performing the first stage of pre-training. In this first stage, the spatiotemporal query transformation module is trained to extract the spatiotemporal video embeddings most relevant to the video description. Combined with... Figure 4AThe spatiotemporal query transformation module 412 is trained to extract the spatiotemporal video embedding most relevant to the video description 430. In some embodiments, during the first stage of pre-training, the spatiotemporal query transformation module is trained by jointly optimizing the VTM loss function, VTC loss function, and VTG loss function. When calculating the VTM loss function, the average values ​​of the spatial query and the temporal query can be calculated separately, and then this average value can be concatenated with the classification label and input into a binary classification task to predict whether the video and the video description match.

[0051] At box 504, the spatiotemporal query transformation module is connected to the language model for the second stage of pre-training. For example, combining... Figure 3 The spatiotemporal query transformation module 312 can be connected to the language model 324 to train the spatiotemporal query transformation module 312. In some embodiments, the parameters of the language model 324 can remain unchanged, which can speed up the model training process. Furthermore, to connect the spatiotemporal query transformation module and the language model, an MLP layer can be used to project video features from the spatiotemporal query transformation module to the embedding space of the language model. For example, combining... Figure 3 The MLP layer 332 can be used to connect the spatiotemporal query transformation module 312 and the language model 324. The MLP layer 332 can project the video features 320 into the embedding space of the language model 324.

[0052] Figure 6 A schematic diagram illustrating a process 600 of interacting with a video processing system according to an embodiment of the present disclosure is shown. Figure 6 As shown, a user can input video 602 into a video processing system, which may have a video processing model according to embodiments of this disclosure. For example, video 602 may be a video related to a vehicle. In some embodiments, the video processing system may also display image frames below video 602. The user can then interact with the video processing system. For example, the user can send dialogue 604, "What happened in this video?", and the video processing system can then generate a corresponding dialogue 606, "In this video, a car's rearview mirror was damaged after a car accident." Furthermore, the video processing system may also support multi-turn dialogues. For example, the user can input dialogue 608 to further inquire about content related to video 602, and the video processing system can further generate dialogue 610 to respond.

[0053] Figure 7 A block diagram of a video processing apparatus 700 according to an embodiment of the present disclosure is shown. Figure 7As shown, the device 700 includes a spatiotemporal feature determination module 702, configured to determine the temporal and spatial features of the video. Furthermore, the device 700 also includes a text output generation module 704, configured to generate text output corresponding to the video based on the temporal and spatial features using a spatiotemporal query conversion module. The spatiotemporal query conversion module includes a time query and a corresponding time expert network, as well as a spatial query and a corresponding spatial expert network.

[0054] Figure 8 A block diagram of an electronic device 800 according to certain embodiments of the present disclosure is shown. Figure 8 A block diagram of an electronic device 800 according to certain embodiments of the present disclosure is shown. Device 800 may be the device or apparatus described in the embodiments of the present disclosure. Figure 8 As shown, device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of device 800. The CPU / GPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804. Although not shown in... Figure 8 As shown, device 800 may also include a coprocessor.

[0055] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0056] The various methods or processes described above can be executed by CPU / GPU 801. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU / GPU 801, one or more steps or actions in the methods or processes described above can be performed.

[0057] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0058] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0059] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0060] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0061] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0062] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0063] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0064] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0065] The following are some example implementations of this disclosure.

[0066] Example 1. A video processing method, comprising:

[0067] Determine the temporal and spatial characteristics of the video; and

[0068] Based on the time and space features, a spatiotemporal query conversion module is used to generate text output corresponding to the video. The spatiotemporal query conversion module includes a time query and a corresponding time expert network, as well as a space query and a corresponding space expert network.

[0069] Example 2. According to the method of Example 1, wherein the temporal features and the spatial features are generated by an attention pooling module, the method further includes:

[0070] Based on the video, generate the visual features of the video; and

[0071] Based on the visual features, the temporal features and spatial features are generated using the cross-attention network and feedforward neural network in the attention pooling module.

[0072] Example 3. The method according to any one of Examples 1-2, wherein the spatiotemporal query transformation module further includes a fusion query and a corresponding fusion expert network.

[0073] Example 4. The method according to any one of Examples 1-3 further includes:

[0074] Get the prompt text; and

[0075] Based on the time features, the spatial features, and the prompt text, the spatiotemporal query conversion module generates the text output indicated by the prompt text.

[0076] Example 5. The method according to any one of Examples 1-4, wherein generating the text output indicating the prompt text includes:

[0077] Based on the temporal and spatial features, video features are generated using the spatiotemporal query and conversion module; and

[0078] Based on the video features and the prompt text, the text output indicated by the prompt text is generated.

[0079] Example 6. The method according to any one of Examples 1-5, wherein the spatiotemporal query transformation module further includes a cross-attention mask, the cross-attention mask including a temporal mask, a spatial mask and a fusion mask.

[0080] Example 7. The method according to any one of Examples 1-6, wherein the text output is generated by a video processing model, the video processing model including the spatiotemporal query transformation module, the method further comprising:

[0081] The video processing model is trained based on the training video, the video description text of the training video, and the training prompt text.

[0082] Example 8. The method according to any one of Examples 1-7, wherein training the video processing model comprises:

[0083] Based on the training video, a visual encoder generates training visual features;

[0084] Based on the training visual features, the attention pooling module generates training temporal features and training spatial features.

[0085] Based on the training time features, the training space features, and the video description text, the spatiotemporal query and conversion module generates a conversion output; and

[0086] Based on the transformation output, the attention pooling module and the spatiotemporal query transformation module are trained.

[0087] Example 9. The method according to any one of Examples 1-8, wherein training the attention pooling module and the spatiotemporal query transformation module includes:

[0088] The attention pooling module and the spatiotemporal query transformation module are trained based on the video text matching loss function, the video text comparison loss function, and the video localization text generation loss function.

[0089] Example 10. The method according to any one of Examples 1-9 further includes:

[0090] Based on the transformation output and the training prompt text, the language model outputs the target output indicated by the training prompt text; and

[0091] Based on the target output indicated by the training prompt text, the attention pooling module and the spatiotemporal query transformation module are trained, and the parameters of the language model remain unchanged during training.

[0092] Example 11. A video processing apparatus, comprising:

[0093] The spatiotemporal feature determination module is configured to determine the temporal and spatial features of the video; and

[0094] The text output generation module is configured to generate text output corresponding to the video based on the time features and the spatial features using the spatiotemporal query transformation module. The spatiotemporal query transformation module includes a time query and a corresponding time expert network, as well as a spatial query and a corresponding spatial expert network.

[0095] Example 12. The apparatus according to Example 11, wherein the temporal features and the spatial features are generated by an attention pooling module, the apparatus further comprising:

[0096] A visual feature generation module is configured to generate visual features of the video based on the video; and

[0097] The spatiotemporal feature generation module is configured to generate the temporal features and the spatial features based on the visual features, using the cross-attention network and feedforward neural network in the attention pooling module.

[0098] Example 13. The apparatus according to any one of Examples 11-12, wherein the spatiotemporal query transformation module further includes a fusion query and a corresponding fusion expert network.

[0099] Example 14. The apparatus according to any one of Examples 11-13, further comprising:

[0100] The prompt text acquisition module is configured to acquire prompt text; and

[0101] The instruction output generation module is configured to generate text output of the prompt text indication based on the time features, the spatial features, and the prompt text using the spatiotemporal query conversion module.

[0102] Example 15. The apparatus according to any one of Examples 11-14, wherein the instruction output generation module comprises:

[0103] The video feature generation module is configured to generate video features based on the temporal and spatial features using the spatiotemporal query and transformation module; and

[0104] The second instruction output generation module is configured to generate the text output indicated by the prompt text based on the video features and the prompt text.

[0105] Example 16. The apparatus according to any one of Examples 11-15, wherein the spatiotemporal query transformation module further includes a cross-attention mask, the cross-attention mask including a temporal mask, a spatial mask and a fusion mask.

[0106] Example 17. The apparatus according to any one of Examples 11-16, wherein the text output is generated by a video processing model, the video processing model including the spatiotemporal query transformation module, the apparatus further comprising:

[0107] The video processing model training module is configured to train the video processing model based on the training video, the video description text of the training video, and the training prompt text.

[0108] Example 18. The apparatus according to any one of Examples 11-17, wherein the video processing model training module comprises:

[0109] The training visual feature generation module is configured to generate training visual features from the visual encoder based on the training video;

[0110] The training spatiotemporal feature generation module is configured to generate training temporal features and training spatial features based on the training visual features by the attention pooling module;

[0111] The conversion output generation module is configured to generate a conversion output based on the training temporal features, the training spatial features, and the video description text, using the spatiotemporal query conversion module; and

[0112] The second training module of the processing model is configured to train the attention pooling module and the spatiotemporal query transformation module based on the transformation output.

[0113] Example 19. The apparatus according to any one of Examples 11-18, wherein the second training module of the processing model comprises:

[0114] The third training module of the processing model is configured to train the attention pooling module and the spatiotemporal query transformation module based on the video text matching loss function, the video text comparison loss function, and the video localization text generation loss function.

[0115] Example 20. The apparatus according to any one of Examples 11-19, further comprising:

[0116] The target output module is configured to, based on the transformed output and the training prompt text, output the target output indicated by the training prompt text from a language model; and

[0117] The fourth training module of the processing model trains the attention pooling module and the spatiotemporal query transformation module based on the target output indicated by the training prompt text, and the parameters of the language model remain unchanged during training.

[0118] Example 21. An electronic device comprising:

[0119] Processor; and

[0120] A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform actions, the actions including:

[0121] Determine the temporal and spatial characteristics of the video; and

[0122] Based on the time and space features, a spatiotemporal query conversion module is used to generate text output corresponding to the video. The spatiotemporal query conversion module includes a time query and a corresponding time expert network, as well as a space query and a corresponding space expert network.

[0123] Example 22. The electronic device according to Example 21, wherein the temporal features and the spatial features are generated by an attention pooling module, the method further comprising:

[0124] Based on the video, generate the visual features of the video; and

[0125] Based on the visual features, the temporal features and spatial features are generated using the cross-attention network and feedforward neural network in the attention pooling module.

[0126] Example 23. An electronic device according to any one of Examples 21-22, wherein the spatiotemporal query conversion module further includes a fusion query and a corresponding fusion expert network.

[0127] Example 24. The electronic device according to any one of Examples 21-23, wherein the operation further includes:

[0128] Get the prompt text; and

[0129] Based on the time features, the spatial features, and the prompt text, the spatiotemporal query conversion module generates the text output indicated by the prompt text.

[0130] Example 25. An electronic device according to any one of Examples 21-24, wherein generating the text output indicating the prompt text includes:

[0131] Based on the temporal and spatial features, video features are generated using the spatiotemporal query and conversion module; and

[0132] Based on the video features and the prompt text, the text output indicated by the prompt text is generated.

[0133] Example 26. An electronic device according to any one of Examples 21-25, wherein the spatiotemporal query transformation module further includes a cross-attention mask, the cross-attention mask including a temporal mask, a spatial mask and a fusion mask.

[0134] Example 27. An electronic device according to any one of Examples 21-26, wherein the text output is generated by a video processing model, the video processing model including the spatiotemporal query transformation module, and the method further includes:

[0135] The video processing model is trained based on the training video, the video description text of the training video, and the training prompt text.

[0136] Example 28. An electronic device according to any one of Examples 21-27, wherein training the video processing model comprises:

[0137] Based on the training video, a visual encoder generates training visual features;

[0138] Based on the training visual features, the attention pooling module generates training temporal features and training spatial features.

[0139] Based on the training time features, the training space features, and the video description text, the spatiotemporal query and conversion module generates a conversion output; and

[0140] Based on the transformation output, the attention pooling module and the spatiotemporal query transformation module are trained.

[0141] Example 29. An electronic device according to any one of Examples 21-28, wherein training the attention pooling module and the spatiotemporal query transformation module comprises:

[0142] The attention pooling module and the spatiotemporal query transformation module are trained based on the video text matching loss function, the video text comparison loss function, and the video localization text generation loss function.

[0143] Example 30. The electronic device according to any one of Examples 21-29, wherein the operation further includes:

[0144] Based on the transformation output and the training prompt text, the language model outputs the target output indicated by the training prompt text; and

[0145] Based on the target output indicated by the training prompt text, the attention pooling module and the spatiotemporal query transformation module are trained, and the parameters of the language model remain unchanged during training.

[0146] Example 31. A computer-readable storage medium having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of Examples 1 to 10.

[0147] Example 32. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 10.

[0148] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video processing method, comprising: Determine the temporal and spatial characteristics of the video; as well as Based on the temporal and spatial features, a spatiotemporal query transformation module is used to generate text output corresponding to the video. The spatiotemporal query transformation module includes a time query and a corresponding temporal expert network, as well as a spatial query and a corresponding spatial expert network. The temporal and spatial features are generated through an attention pooling module. The method further includes: Based on the video, generate the visual features of the video; as well as Based on the visual features, the temporal features and spatial features are generated using the cross-attention network and feedforward neural network in the attention pooling module.

2. The method according to claim 1, wherein the spatiotemporal query conversion module further includes a fusion query and a corresponding fusion expert network.

3. The method according to claim 1, further comprising: Get the prompt text; as well as Based on the time features, the spatial features, and the prompt text, the spatiotemporal query conversion module generates the text output indicated by the prompt text.

4. The method of claim 3, wherein generating the text output indicating the prompt text comprises: Based on the time features and the spatial features, video features are generated using the spatiotemporal query and conversion module; as well as Based on the video features and the prompt text, the text output indicated by the prompt text is generated.

5. The method according to claim 4, wherein the spatiotemporal query transformation module further includes a cross-attention mask, wherein the cross-attention mask includes a temporal mask, a spatial mask, and a fusion mask.

6. The method according to claim 1, wherein the text output is generated by a video processing model, the video processing model including the spatiotemporal query transformation module, and the method further includes: The video processing model is trained based on the training video, the video description text of the training video, and the training prompt text.

7. The method of claim 6, wherein training the video processing model comprises: Based on the training video, a visual encoder generates training visual features; Based on the training visual features, the attention pooling module generates training temporal features and training spatial features. Based on the training time features, the training space features, and the video description text, the spatiotemporal query and conversion module generates a conversion output; and Based on the transformation output, the attention pooling module and the spatiotemporal query transformation module are trained.

8. The method according to claim 7, wherein training the attention pooling module and the spatiotemporal query transformation module comprises: The attention pooling module and the spatiotemporal query transformation module are trained based on the video text matching loss function, the video text comparison loss function, and the video localization text generation loss function.

9. The method of claim 7, further comprising: Based on the transformation output and the training prompt text, the language model outputs the target output indicated by the training prompt text; as well as Based on the target output indicated by the training prompt text, the attention pooling module and the spatiotemporal query transformation module are trained, and the parameters of the language model remain unchanged during training.

10. A video processing apparatus, comprising: The spatiotemporal feature determination module is configured to determine the temporal and spatial features of the video; as well as A text output generation module is configured to generate text output corresponding to the video based on the temporal and spatial features using a spatiotemporal query transformation module. The spatiotemporal query transformation module includes a time query and a corresponding temporal expert network, as well as a spatial query and a corresponding spatial expert network. The temporal and spatial features are generated by an attention pooling module. The device further includes: A visual feature generation module is configured to generate visual features of the video based on the video. as well as The spatiotemporal feature generation module is configured to generate the temporal features and the spatial features based on the visual features, using the cross-attention network and feedforward neural network in the attention pooling module.

11. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A computer program product tangibly stored on a non-transient computer-readable medium and comprising computer-executable instructions for performing the method according to any one of claims 1 to 9.

Images