Video task processing method, video processing model training method, video processing model-based information processing method, and task platform

Abstract

Embodiments of the present description provide a video task processing method, a video processing model training method, a video processing model-based information processing method, and a task platform. The video task processing method comprises: acquiring task data of a target video task, wherein the task data comprises a video to be processed; determining a spatial fusion feature and a temporal alignment feature of said video, wherein the spatial fusion feature is obtained by performing spatial fusion on the basis of image information and sound information of said video, the spatial fusion feature comprises a fused visual feature and a fused audio feature, and the temporal alignment feature is obtained by performing temporal alignment on the basis of the fused visual feature and the fused audio feature; and on the basis of the spatial fusion feature and the temporal alignment feature, using a target video processing model corresponding to the target video task to obtain a target processing result of the target video task. Fine-grained fusion across both spatial and temporal dimensions is performed on the image information and the sound information of said video, thereby obtaining an accurate target processing result.

Description

Video task processing, training of video processing model, information processing method based on video processing model and task platform

[0001] The present disclosure claims priority to Chinese Patent Application No. 2024110470647, filed on July 31, 2024, with the Chinese Patent Office, entitled "Video task processing, training of video processing model, information processing method based on video processing model and task platform", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present specification relate to the technical field of deep learning, in particular to a video task processing method, a training method of a video processing model, an information processing method based on a video processing model, and a task platform. BACKGROUND

[0003] With the rapid development of computer technology and artificial intelligence technology, deep learning models have shown great potential in video task processing and can perform different downstream video tasks. Currently, when processing video tasks, the audio content in the video cannot be accurately understood, which affects the processing accuracy of the video task. Therefore, there is an urgent need for a video task processing method that can improve task processing accuracy. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a video task processing method. One or more embodiments of the present specification also relate to a training method of a video processing model, an information processing method based on a video processing model, a task platform, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a video task processing method is provided, comprising:

[0006] obtaining task data of a target video task, wherein the task data includes a to-be-processed video;

[0007] determining spatial fusion features and time alignment features of the to-be-processed video, wherein the spatial fusion features are obtained based on spatial fusion of image information and sound information of the to-be-processed video, and the time alignment features are obtained based on time alignment of the spatial fusion features;

[0008] According to the spatial fusion features and the time alignment features, a target processing result of the target video task is obtained by using a target video processing model corresponding to the target video task.

[0009] According to a second aspect of the embodiments of the present specification, a training method of a video processing model is provided, comprising:

[0010] obtaining training data of a training task, wherein the training data comprises a sample video and a result label;

[0011] determining a sample spatial fusion feature and a sample time alignment feature of the sample video, wherein the sample spatial fusion feature is obtained by spatial fusion based on sample image information and sample sound information of the sample video, and the sample time alignment feature is obtained by time alignment based on the sample spatial fusion feature;

[0012] training an initial video model according to the sample spatial fusion feature and the sample time alignment feature, in combination with the result label, to obtain a target video processing model trained.

[0013] According to a third aspect of the embodiments of the present specification, a video processing model-based information processing method is provided, applied to a task platform, comprising:

[0014] receiving a model request sent by a terminal device, wherein the model request comprises at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter;

[0015] determining a corresponding target video processing model from at least one video processing model based on the model request, wherein the at least one video processing model is trained based on the above-mentioned training method of the video processing model.

[0016] According to a fourth aspect of the embodiments of the present specification, a task platform is provided, comprising a request interface and a response unit;

[0017] The request interface is configured to receive a model request sent by a terminal device, wherein the model request comprises at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter;

[0018] The response unit is configured to determine a corresponding target video processing model from at least one video processing model based on the model request, wherein the at least one video processing model is trained based on the above-mentioned training method of the video processing model.

[0019] According to a fifth aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0020] a memory and a processor;

[0021] The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model are implemented.

[0022] According to a sixth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions. When the computer programs / instructions are executed by a processor, the steps of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model are implemented.

[0023] According to a seventh aspect of an embodiment of the present specification, a computer program product is provided, which includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the steps of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model are implemented.

[0024] One embodiment of the present specification implements that, for a to-be-processed video of a target video task, a spatial fusion feature and a time alignment feature can be determined. The spatial fusion feature is obtained based on spatial fusion of image information and sound information of the to-be-processed video, and includes a fusion visual feature and a fusion audio feature. The time alignment feature is obtained based on time alignment of the fusion visual feature and the fusion audio feature. That is, the image information and the sound information of the to-be-processed video are fused in a fine-grained manner in terms of space and time. Then, based on the spatial fusion feature and the time alignment feature, a target processing result of the target video task is obtained by using a target video processing model corresponding to the target video task. Thus, the target video processing model can more accurately understand the fine-grained features of the image information and the sound information of the to-be-processed video in terms of time and space, and obtain an accurate target processing result of the target video task. BRIEF DESCRIPTION OF DRAWINGS

[0025] FIG. 1 is an application architecture diagram of a video task according to one embodiment of the present specification;

[0026] FIG. 2 is a flowchart of a video task processing method according to one embodiment of the present specification;

[0027] FIG. 3 is a schematic diagram of a feature fusion process in a video task processing method according to one embodiment of the present specification;

[0028] FIG. 4 is a schematic diagram of a processing process of a video task processing method according to one embodiment of the present specification;

[0029] FIG. 5 is a schematic diagram of a video task processing interface according to one embodiment of the present specification;

[0030] Figure 6 is a schematic diagram of another video task processing interface provided in one embodiment of this specification;

[0031] Figure 7 is a flowchart of a training method for a video processing model provided in one embodiment of this specification;

[0032] Figure 8 is a flowchart of an information processing method based on a video processing model provided in one embodiment of this specification;

[0033] Figure 9 is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0034] Figure 10 is a schematic diagram of a video task processing device provided in one embodiment of this specification;

[0035] Figure 11 is a schematic diagram of the structure of a training device for a video processing model provided in one embodiment of this specification;

[0036] Figure 12 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0037] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0038] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0040] In addition, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or rejection.

[0041] In one or more embodiments of the present specification, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, thousands of billions or even tens of billions of model parameters. The large model can also be called a foundation model. Through large-scale unlabeled corpus pre-training of the large model, a pre-trained model with hundreds of millions of parameters is output. Such a model can adapt to a wide range of downstream tasks, and the model has good generalization ability. For example, a large-scale language model (LLM, Large Language Model) and a multi-modal pre-training model.

[0042] In actual application, the large model only needs a small amount of samples to fine-tune the pre-trained model and can be applied to different tasks. The large model can be widely applied to natural language processing (NLP, Natural Language Processing) and computer vision fields. Specifically, it can be applied to computer vision field tasks such as visual question answering (VQA, Visual Question Answering), image captioning (IC, Image Caption), image generation, and natural language processing field tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0043] First, the terms involved in one or more embodiments of the present specification are explained.

[0044] Fine-grained: Granularity refers to the degree of division. In general audio-visual pre-training methods, the model often focuses on global features of visual and auditory modalities, ignoring local features. Global features are usually coarse-grained, used for scene-level classification, and local features have local position and object-level semantic information.

[0045] Transformer Model (Self-Attention Model): Introduces an innovation called "self-attention mechanism" that allows the model to calculate in parallel when processing sequences, rather than processing element by element like traditional recurrent neural networks. The self-attention mechanism enables the Transformer model to establish connections between any two positions of given features.

[0046] VIT Model: Full name Vision Transformer, is a visual model based on Transformer architecture, mainly used for computer vision tasks such as image classification, object detection and image segmentation, etc. Transformer is an attention mechanism network proposed in the field of natural language processing (NLP), used to process sequence data, especially long-distance dependency problems. The VIT model migrates this successful sequence processing idea to the field of computer vision, updating the design of visual models based on convolutional neural networks (CNN).

[0047] AST Model: Full name Audio Spectrogram Transformer, is a Transformer model designed specifically for audio processing tasks. Similar to VIT, which applies Transformer architecture to computer vision, the AST model can apply the attention mechanism of Transformer to audio signal processing, analyzing audio spectrograms to solve complex audio understanding tasks such as speech recognition, music generation, and sound event detection.

[0048] Multilayer Perceptron (MLP): is a structure in artificial neural networks, belonging to the category of feedforward neural networks.

[0049] Gaussian Error Linear Unit (GELU): is an activation function widely used in deep learning networks, especially in Transformer models and other neural network architectures. GELU combines linear transformation and Gaussian cumulative distribution function (CDF) to provide nonlinear characteristics while maintaining efficient training and good generalization ability of the model.

[0050] It should be noted that video large models are developing rapidly, but most of them do not contain audio modalities, and the description of audio in audio-visual large models is mostly derived from the association of videos, and in fact cannot accurately perceive and understand the audio modality.

[0051] Currently, in the field of video large models, related models such as video-llama, macaw-llm can realize related functions, but they are simply added audio modalities and cannot make audio modalities play a good role.

[0052] In addition, Video-ChatGPT is a pre-trained encoder and a spatiotemporal two-branch pooling integrated input; VideoChat is to split the video into description and video itself embedding, and the two branches are integrated into the input; LLaMA-VID is to split the video into context token and content token, and the two tokens are superimposed to represent a frame of video; Video-LaVIT is to split the video into key frames and motion vectors (motion vectors) for tokenization, and to integrate understanding and generation; Chat-UniVi is to unify image and video input, object-based adaptive clustering token; Video-LLaVA is to unify image and video input, align before projection; Video-LLaMA is audio and video integrated input through two Q-former branches. In the above large models, the audio does not play a role.

[0053] Therefore, the embodiments of the present specification provide a video task processing method, which can make the target video processing model (video large model) perceive the video modality and the audio modality equally, and simultaneously consider the alignment of the visual modality and the auditory modality in time and space, so that the target video processing model can comprehensively and accurately understand the video to be processed.

[0054] To solve the above technical problems, in the present specification, a video task processing method is provided. One or more embodiments of the present specification simultaneously relate to a video processing model training method, a video processing model-based information processing method, a task platform, a computing device, a computer readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0055] Considering that the model parameter quantity of the large model is large, and the operation resource of the mobile terminal is limited, the video task processing method provided in the embodiments of the present disclosure can be applied to the application scenario as shown in FIG. 1, but is not limited thereto. FIG. 1 is an application architecture diagram of a video task provided by an embodiment of the present specification. In the application scenario as shown in FIG. 1, the large model is deployed in a server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 can include, but is not limited to, a smart phone, a tablet computer, a notebook computer, a palm computer, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface as shown in FIG. 1 to realize the calling of the large model, and thus realize the method provided in the embodiments of the present specification.

[0056] In the embodiments of the present specification, as shown in FIG. 1, the system composed of the client device and the server can perform the following steps: the client device performs receiving the user inputted video to be processed, generating a target video task and uploading to the server. The server performs obtaining the task data of the target video task, wherein the task data includes the video to be processed; determining the spatial fusion feature and the time alignment feature of the video to be processed, wherein the spatial fusion feature is obtained based on the image information and the sound information of the video to be processed, the spatial fusion feature includes the fusion visual feature and the fusion audio feature, and the time alignment feature is obtained based on the fusion visual feature and the fusion audio feature; according to the spatial fusion feature and the time alignment feature, using the target video processing model corresponding to the target video task, obtaining the target processing result of the target video task, and feeding back the task processing result to the client device to display to the user. In this way, the image and sound of the video to be processed are fused in the spatial and temporal two levels, so that the target video processing model can more accurately understand the time and spatial fine-grained features of the image information and sound information of the video to be processed, and obtain the accurate target processing result of the target video task. It should be noted that in the case that the running resource of the client device can meet the deployment and running conditions of the large model, the embodiments of the present disclosure can be performed in the client device.

[0057] Referring to FIG. 2, FIG. 2 shows a flowchart of a video task processing method according to an embodiment of the present specification, which specifically includes the following steps 202-206.

[0058] Step 202: Obtain the task data of the target video task, wherein the task data includes the video to be processed.

[0059] Embodiments of the present specification are applied to an application, a website or a mini-program with a video task processing function. On the application, the website or the mini-program, a video task processing function is implemented, for example, a website deploying a target video processing model, implementing a task processing function of a video task such as video question and answer, video quality evaluation, video description, object positioning in a video, object extraction in a video, etc. For another example, a third-party application can implement a processing function of a corresponding video task by calling a deployed target video processing model through an application programming interface (API).

[0060] The target video task is a task to be processed, which is used to indicate video processing based on task data. For example, the target video task can be a video question and answer task, a video quality evaluation task, a video description task, an audio-video positioning task, etc. The task data can include a video to be processed, and the target processing result of the target video task can be obtained by analyzing and processing the video to be processed.

[0061] In actual implementation, the task data of the target video task can be obtained based on front-end input.

[0062] Step 204: determining spatial fusion features and time alignment features of the video to be processed, wherein the spatial fusion features are obtained by spatial fusion based on image information and sound information of the video to be processed, the spatial fusion features include fusion visual features and fusion audio features, and the time alignment features are obtained by time alignment based on the fusion visual features and the fusion audio features.

[0063] The fusion visual features are features obtained by incorporating sound information into image information, and the fusion audio features are features obtained by incorporating image information into sound information.

[0064] It should be noted that a video can include image information of a visual modality and sound information of an auditory modality. In order to consider the feature information of the visual modality and the auditory modality, the spatial fusion features including the fusion visual features and the fusion audio features can be determined based on the image information and the sound information of the video to be processed, and the time alignment features can be determined based on the fusion visual features and the fusion audio features, so as to provide the target video processing model with time and spatial fine-grained features of the image information and the sound information of the video to be processed, so that the target video processing model obtains an accurate target processing result.

[0065] The spatial fusion refers to fusing information of image information and sound information at corresponding positions, fusing the sound information into the image information to obtain a fusion visual feature, and fusing the image information into the sound information to obtain a fusion audio feature. The time alignment refers to time aligning the fusion visual feature and the fusion audio feature of each frame of video according to a time point of the video.

[0066] In actual implementation, the fusion visual feature and the fusion audio feature of the to-be-processed video can be determined by using the visual model and the audio model, and the fusion visual feature and the fusion audio feature at each time point can be time-aligned to obtain a time-aligned feature.

[0067] In step 206, a target video processing model corresponding to the target video task is used to obtain a target processing result of the target video task according to the spatial fusion feature and the time-aligned feature.

[0068] The target video processing model is a video processing model that is pre-trained and can analyze the spatial fusion feature and the time-aligned feature to output the target processing result of the target video task.

[0069] In addition to the to-be-processed video, the video task can also include task information, which is used to instruct the target video processing model to analyze the spatial fusion feature and the time-aligned feature according to a corresponding processing manner to output the target processing result of the target video task.

[0070] The video task processing method provided in this embodiment can perform spatial and temporal fine-grained fusion on image information and sound information of the to-be-processed video, promote interaction between the visual modality and the auditory modality, and enable the target video processing model to more accurately understand fine-grained features of the image information and the sound information of the to-be-processed video in time and space, and obtain an accurate target processing result of the target video task.

[0071] The following further explains optional implementations of the above embodiments:

[0072] In an optional implementation of this embodiment, the above obtaining of the task data of the target video task includes:

[0073] The target video task is generated in response to the to-be-processed video and the task information input in the front end.

[0074] The task information is used to indicate a processing manner of the to-be-processed video. If the target video task is a video question answering task, the task information is a question related to the to-be-processed video, such as “What does this video describe?”. If the target video task is a video object positioning task, the task information is an object to be positioned and a related description, such as “Please locate the object described by the audio in this video”.

[0075] It should be noted that the to-be-processed video and the task information can be input by the front end, and the target video task is generated in response to the input to-be-processed video and the task information, and the target video task is subsequently processed. Of course, the target video task can also be directly generated by the front end based on the input to-be-processed video and the task information. At this time, the task data of the target video task includes the to-be-processed video and the task information

[0076] In the embodiments of the present specification, the target video task can be generated in response to the to-be-processed video and the task information input by the front end, and the processing method of the to-be-processed video is described through the task information. The user can autonomously select the video to be processed and the corresponding processing method, providing a rich interactive mode, realizing different video task processing, meeting different task requirements, having a low operation threshold, and improving user experience.

[0077] Of course, in actual implementation, in addition to providing the to-be-processed video and the task information by the front end, the front end can also only provide the task information, select a related video from a video library as the to-be-processed video based on a keyword of the task information, and generate the target video task. For example, the task information input by the front end is “please provide a pancake making video and a text explanation of the specific making process”, at this time, a video of making pancakes can be selected from the video library as the to-be-processed video, and the to-be-processed video is subsequently analyzed and processed to obtain the text explanation.

[0078] In an optional implementation of the present embodiment, the above-mentioned determination of the spatial fusion feature and the time alignment feature of the to-be-processed video includes:

[0079] The image information and the sound information corresponding to the to-be-processed video are determined.

[0080] The image information and the sound information are spatially fused to obtain fused visual features and fused audio features, and the fused visual features and the fused audio features are taken as the spatial fusion feature.

[0081] The fused visual features and the fused audio features are time-aligned to obtain the time alignment feature.

[0082] In actual implementation, a software tool or a programming technique can be used to process the video to be processed to extract image information (frame image) and sound information (audio stream) therein. Exemplarily, the image information and the sound information in the video to be processed can be extracted generally using a multimedia processing tool or a programming library, such as FFmpeg. The image information extraction can parse the video to be processed through a command line instruction, and save each frame as an image file as needed. The sound information extraction can export an audio stream to a separate audio file through the same tool by setting parameters. The frame rate, time point and output format of the extraction can be customized in the process to adapt to the needs of different application scenarios. FFmpeg is a powerful cross-platform command line tool for processing audio, video and image files.

[0083] After the image information and the sound information in the video to be processed are extracted, the image information can be fused with the sound information to obtain a fused visual feature, and the sound information can be fused with the image information to obtain a fused audio feature. The fused visual feature and the fused audio feature are spatial fusion features.

[0084] In addition, the fused visual feature and the fused audio feature are obtained by fusing the image information and the sound information in the video to be processed in the spatial dimension. Then, the fused visual feature and the fused audio feature can be time-aligned to obtain a time-aligned feature, thereby fusing the image information and the sound information in the video to be processed in the time dimension, and providing a spatial and time-dimension fine-grained feature reference for the target video processing model.

[0085] In the embodiments of the present specification, the image information and the sound information can be spatially fused to obtain the fused visual feature and the fused audio feature, respectively. Then, the fused visual feature and the fused audio feature can be further time-aligned to obtain the time-aligned feature, thereby performing double fusion of the video to be processed in the spatial and time dimensions, providing a finer-grained feature, and providing a fine-grained feature reference for the target video processing model to process the target video task, thereby ensuring the processing accuracy of the target video task.

[0086] In an optional implementation of the present embodiment, the spatial fusion of the image information and the sound information to obtain the fused visual feature and the fused audio feature includes:

[0087] The multi-scale visual features of the image information are spatially fused with the image information and the sound information by using the visual model and the audio model to obtain multi-scale visual spatial features. The multi-scale visual spatial features are fused with the image information to obtain the fused visual feature.

[0088] The multiscale audio features of the sound information are respectively spatially fused with the image information and the sound information by using the visual model and the audio model to obtain multiscale audio spatial features, and the multiscale audio spatial features are fused with the sound information to obtain fused audio features.

[0089] The multiscale visual spatial features are information obtained by fusing the multiscale visual features with the image information and the sound information, and the multiscale audio spatial features are information obtained by fusing the multiscale audio features with the image information and the sound information.

[0090] The visual model can be any image processing model capable of analyzing and extracting image information to obtain visual features, such as a VIT model, a CNN (Convolutional Neural Networks), a residual network, etc. The audio model can be any audio processing model capable of analyzing and extracting sound information to obtain audio features, such as an AST, MFCCs (Mel Frequency Cepstral Coefficients), a Log-Mel Spectrogram.

[0091] In actual implementation, the multiscale visual features are obtained by sampling the image information at at least one spatial scale, and the multiscale audio features are obtained by sampling the sound information at at least one spatial scale. The spatial scale refers to the degree of detail in the spatial dimension that is collected or represented, which can affect the quality, detail preservation, and subsequent processing and analysis capabilities of the sampling results, such as resolution, pixel size, etc.

[0092] In the embodiments of the present specification, multiscale visual features of the image information can be extracted, and then the multiscale visual features are respectively spatially fused with the image information and the sound information to obtain multiscale visual spatial features, and the multiscale visual spatial features are fused with the image information to obtain fused visual features, thereby integrating the multiscale visual features, the image information, and the sound information into the visual modality. Meanwhile, multiscale audio features of the sound information can be extracted, and then the multiscale audio features are respectively spatially fused with the image information and the sound information to obtain multiscale audio spatial features, and the multiscale audio spatial features are fused with the sound information to obtain fused audio features, thereby integrating the multiscale audio features, the image information, and the sound information into the auditory modality. In this way, the information of the auditory modality is continuously fused into the visual modality, and the information of the visual modality is continuously fused into the auditory modality, and the fusion is performed for the visual modality and the auditory modality respectively to obtain the fused visual features and the fused audio features respectively, thereby realizing fine-grained fusion.

[0093] In an optional implementation of the embodiment, the visual model comprises N visual feature extraction modules, and the audio model comprises N audio feature extraction modules; the multi-scale visual features of the image information are respectively spatially fused with the image information and the sound information by using the visual model and the audio model to obtain multi-scale visual spatial features, and the multi-scale visual spatial features are fused with the image information to obtain fused visual features, comprising:

[0094] The following steps are repeatedly performed until the value of i is N, and the initial value of i is 1:

[0095] The image information is input into the i-th visual feature extraction module to obtain original visual features extracted by the i-th visual feature extraction module, and the sound information is input into the i-th audio feature extraction module to obtain original audio features extracted by the i-th audio feature extraction module, wherein the initial value of i is 1.

[0096] The image information is down-sampled at at least one spatial scale to obtain multi-scale visual features.

[0097] The multi-scale visual features, the original visual features extracted by the i-th visual feature extraction module, and the original audio features extracted by the i-th audio feature extraction module are fused to obtain multi-scale visual spatial features.

[0098] The multi-scale visual spatial features are fused with the original visual features extracted by the i-th visual feature extraction module to obtain initial fused visual features.

[0099] The initial fused visual features are determined as new image information, and the value of i is determined to be increased by 1.

[0100] The multi-scale visual spatial features are fused with the original visual features extracted by the i-th visual feature extraction module to obtain initial fused visual features.

[0101] It should be noted that the visual model comprises N visual feature extraction modules, and the audio model comprises N audio feature extraction modules. The visual feature extraction module can analyze the input image information to extract original visual features. The original visual features extracted by the current visual feature extraction module are fused with the sound information in the visual modality to obtain initial fused visual features, which can be used as the input of the next visual feature extraction module for continuous analysis to extract corresponding original visual features, and the sound information in the visual modality is continuously fused until the original visual features extracted by the last visual feature extraction module are fused with the multi-scale visual spatial features to obtain the fused visual features.

[0102] In actual implementation, the obtained image information can be input into the first visual feature extraction module to obtain the original visual features extracted by the first visual feature extraction module, the obtained sound information can be input into the first audio feature extraction module to obtain the original audio features extracted by the first audio feature extraction module, and the obtained image information is sampled at at least one spatial scale to obtain multi-scale visual features, and the multi-scale visual features, the original visual features extracted by the first visual feature extraction module and the original audio features extracted by the first audio feature extraction module are fused to obtain multi-scale visual spatial features; then, the multi-scale visual spatial features are superimposed on the original visual features extracted by the first visual feature extraction module to obtain initial fusion visual features.

[0103] The initial fusion visual features are taken as new image information, input into the second visual feature extraction module to obtain the original visual features extracted by the second visual feature extraction module, the obtained sound information is input into the second audio feature extraction module to obtain the original audio features extracted by the second audio feature extraction module, and the obtained image information is sampled at at least one spatial scale to obtain multi-scale visual features, and the multi-scale visual features, the original visual features extracted by the second visual feature extraction module and the original audio features extracted by the second audio feature extraction module are fused to obtain multi-scale visual spatial features; then, the multi-scale visual spatial features are superimposed on the original visual features extracted by the second visual feature extraction module to obtain initial fusion visual features.

[0104] By analogy, the above steps are continuously performed until the initial fusion visual features obtained by fusing the multi-scale visual spatial features and the original visual features extracted by the Nth visual feature extraction module, that is, the required fusion visual features.

[0105] The multi-scale visual features include features obtained by sampling at at least one spatial scale, that is, the number of multi-scale visual features corresponds to the number of spatial scales, and each spatial scale corresponds to multi-scale visual features, which are sequentially subjected to cross-attention calculation with the original visual features extracted by the i th visual feature extraction module and the original audio features extracted by the i th audio feature extraction module to obtain fused multi-scale visual spatial features; and the multi-scale visual spatial features are superimposed on the original visual features extracted by the i th visual feature extraction module to obtain initial fusion visual features.

[0106] Cross-attention calculation is a cross-attention mechanism, which is widely used in multi-modal processing, sequence-to-sequence (Seq2Seq) models and other tasks that need to fuse information from different sources in the field of machine learning and deep learning. It is an extension of self-attention mechanism, which not only considers the dependency within the sequence, but also can establish dynamic association between two different sequences or data structures, and realize cross-sequence interaction of information.

[0107] It should be noted that the spatial scale refers to the degree of detail collected or represented in the spatial dimension, which can affect the quality of the sampling result, the ability to retain details and subsequent processing and analysis. For example, the spatial scale can be resolution, pixel size, etc. The image information is sampled at at least one spatial scale to obtain multi-scale visual features. As an example, taking resolution as an example of spatial scale, each image frame in the image information can be sampled at resolutions of 1 / 8, 1 / 16 and 1 / 32 of the original resolution, respectively, to obtain multi-scale visual features at resolutions of 1 / 8, 1 / 16 and 1 / 32, respectively. The multi-scale visual features at resolutions of 1 / 8, 1 / 16 and 1 / 32 are sequentially cross-attention calculated with the original visual features extracted by the i-th visual feature extraction module and the original audio features extracted by the i-th audio feature extraction module, respectively, to obtain fused multi-scale visual spatial features. Then, the multi-scale visual spatial features are superimposed on the original visual features extracted by the i-th visual feature extraction module to obtain the initial fused visual features.

[0108] In the embodiments of the present specification, for the visual modality, sampling at different spatial scales to obtain visual features at different spatial scales, and then sequentially and respectively performing feature fusion with the original visual features and the original audio features, and superimposing the fusion results on the original visual features, thereby continuously fusing the information of the auditory modality and the visual features at different spatial scales into the original visual features to obtain the fused visual features, which strengthens the interaction between the visual modality and the auditory modality, guides and promotes the fine-grained alignment between the visual modality and the auditory modality. The fused visual features can provide more fine-grained features in the spatial dimension, ensuring accurate processing of subsequent video tasks.

[0109] As an example, taking the VIT model as an example, in the VIT model, unlike the CNN, each image frame in the image information is not directly processed by the convolutional layer, but is first cut into a plurality of fixed-size patches (image blocks), which are flattened and linearly mapped into a high-dimensional vector space as an input sequence of the model, each image block is regarded as a "word" (token), and then the "word" sequence is processed through the multi-head self-attention (Multi-Head Self-Attention) mechanism of the Transformer and the feed-forward neural network (Feed-Forward Networks) layer to learn the global context information. In actual implementation, the VIT model can be divided to obtain N VIT layers, and one VIT layer serves as a visual feature extraction module. For example, the VIT model includes 24 layers of parameters, and the VIT model can be divided into 6 VIT layers, and each VIT layer includes 4 layers of parameters. That is, the visual model includes 6 visual feature extraction modules, and the extraction of the visual features is implemented based on the 6 visual feature extraction modules. Similarly, taking the AST model as an example, the AST model can also be divided into 6 AST layers, and each AST layer serves as an audio feature extraction module.

[0110] In the embodiments of the present specification, the next visual feature extraction module extracts visual features based on the fusion result of the current visual feature extraction module, and continues to fuse, so as to continuously fuse the multi-scale visual spatial features and the original audio features into the original visual features extracted by the current visual feature extraction module, gradually realizes information fusion, and provides more fine-grained reference features.

[0111] In an optional implementation of the present embodiment, the visual model includes N visual feature extraction modules, and the audio model includes N audio feature extraction modules; the multi-scale audio spatial features of the sound information are respectively spatially fused with the image information and the sound information by using the visual model and the audio model, the multi-scale audio spatial features are fused with the sound information to obtain the fused audio features, including:

[0112] The following steps are repeatedly executed until the value of i is N, and the initial value of i is 1:

[0113] The image information is input into the i-th visual feature extraction module to obtain the original visual features extracted by the i-th visual feature extraction module, and the sound information is input into the i-th audio feature extraction module to obtain the original audio features extracted by the i-th audio feature extraction module.

[0114] The sound information is down-sampled at at least one spatial scale to obtain the multi-scale audio features.

[0115] The multi-scale audio feature, the original visual feature extracted by the i-th visual feature extraction module, and the original audio feature extracted by the i-th audio feature extraction module are fused to obtain a multi-scale audio spatial feature.

[0116] The multi-scale audio spatial feature is fused with the original audio feature extracted by the i-th audio feature extraction module to obtain an initial fusion audio feature.

[0117] The initial fusion audio feature is determined as new sound information, and the value of i is determined to be increased by 1.

[0118] The multi-scale audio spatial feature is fused with the original audio feature extracted by the i-th audio feature extraction module to obtain an initial fusion audio feature.

[0119] It should be noted that the fusion under the auditory modality is similar to the fusion under the visual modality described above to obtain a fusion audio feature. The audio model can also include at least two audio feature extraction modules, and the fusion result of the current audio feature extraction module can be used as the input of the next audio feature extraction module to continue the fusion until the last audio feature extraction module to obtain the fusion audio feature.

[0120] In actual implementation, the number of audio feature extraction modules is the same as the number of visual feature extraction modules, and the features extracted by the audio feature extraction modules and the visual feature extraction modules at the corresponding positions are fused.

[0121] Exemplarily, the image information is input into the i-th visual feature extraction module to obtain the original visual feature extracted by the i-th visual feature extraction module, and the sound information is input into the i-th audio feature extraction module to obtain the original audio feature extracted by the i-th audio feature extraction module. The multi-scale visual feature, the original visual feature extracted by the i-th visual feature extraction module and the original audio feature extracted by the i-th audio feature extraction module are fused to obtain a multi-scale visual spatial feature, and then the multi-scale visual spatial feature is fused with the original visual feature extracted by the i-th visual feature extraction module to obtain an initial fusion visual feature as the input of the (i+1)-th visual feature extraction module. Meanwhile, the multi-scale audio feature, the original visual feature extracted by the i-th visual feature extraction module and the original audio feature extracted by the i-th audio feature extraction module are fused to obtain a multi-scale audio spatial feature, and then the multi-scale audio spatial feature is fused with the original audio feature extracted by the i-th audio feature extraction module to obtain an initial fusion audio feature as the input of the i-th audio feature extraction module. Until i equals N, the initial fusion visual feature obtained by fusing the multi-scale visual spatial feature with the original visual feature extracted by the N-th visual feature extraction module is the required fusion visual feature, and the initial fusion audio feature obtained by fusing the multi-scale audio spatial feature with the original audio feature extracted by the N-th audio feature extraction module is the required fusion audio feature.

[0122] Exemplarily, FIG. 3 is a schematic diagram of a processing process of feature fusion in a video task processing method provided by an embodiment of the present specification. As shown in FIG. 3, the visual model includes N visual feature extraction modules, the audio model includes N audio feature extraction modules, and the fusion calculation is performed through N fusion modules. The image information is input into the 1st visual feature extraction module to obtain the original visual feature 1, and the image information is down-sampled in at least one spatial scale to obtain the multi-scale visual feature 1. The sound information is input into the 1st audio feature extraction module to obtain the original audio feature 1, and the sound information is down-sampled in at least one spatial scale to obtain the multi-scale audio feature 1.

[0123] The multi-scale visual feature 1, the original visual feature 1 and the original audio feature 1 are input into the fusion module 1 for feature fusion to obtain the multi-scale visual spatial feature 1, and the multi-scale visual spatial feature 1 is superimposed on the original visual feature 1 to obtain the initial fusion visual feature 1. Similarly, the multi-scale audio feature 1, the original visual feature 1 and the original audio feature 1 are input into the fusion module 1 for feature fusion to obtain the multi-scale audio spatial feature 1, and the multi-scale audio spatial feature 1 is superimposed on the original audio feature 1 to obtain the initial fusion audio feature 1.

[0124] The initial fusion visual feature 1 is input into the second visual feature extraction module to obtain an original visual feature 2, the initial fusion audio feature 1 is input into the second audio feature extraction module to obtain an original audio feature 2, and the above feature fusion process is continued until the Nth visual feature extraction module to obtain a fusion visual feature, which includes fusion visual features at each time point of V0-Vt; similarly, until the Nth audio feature extraction module to obtain a fusion audio feature, which includes fusion audio features at each time point of V0-Vt, and t is the time length of the video to be processed.

[0125] In the embodiments of the present specification, for the auditory modality, the sound information is sampled at different spatial scales to obtain audio features at different spatial scales, and then the original visual feature and the original audio feature are sequentially fused respectively, and the fusion result is superimposed on the original audio feature, so that the information of the visual modality and the audio features at different spatial scales are continuously fused into the original audio feature, the fusion audio feature is obtained, the interaction between the visual modality and the auditory modality is strengthened, and the fine-grained alignment between the visual modality and the auditory modality is guided and promoted. The fusion audio feature can provide more fine-grained features in the spatial dimension as a reference to ensure accurate processing of subsequent video tasks.

[0126] In addition, the next visual feature extraction module extracts the original visual feature based on the fusion result of the current visual feature extraction module, and the next audio feature extraction module extracts the original audio feature based on the fusion result of the current audio feature extraction module. The original visual feature extraction and the original audio feature extraction are continuously performed on the current fusion result, and the fusion is continued, so that the information fusion is gradually realized, and more fine-grained reference features are provided.

[0127] In an optional embodiment of the present embodiment, the spatial scale is resolution; the sound information is sampled at at least one spatial scale to obtain multi-scale audio features, including:

[0128] The time domain transformation is performed on the sound information to obtain a time-frequency graph of the sound information.

[0129] The time-frequency graph is sampled at at least one resolution to obtain multi-scale audio features.

[0130] It should be noted that, in order to sample the sound information in the spatial dimension and perform spatial fusion with the image information, the time domain transformation can be performed on the sound information to obtain a time-frequency graph of the sound information. The time-frequency graph is a kind of sound information visualization technology, which is used to show the distribution of sound information in time and frequency dimensions. The time-frequency graph decomposes the signal into different frequency components and records how these components change over time, converts the time domain information (time series data) to the frequency domain (frequency distribution), and obtains a visual image (time-frequency graph) for subsequent pixel analysis and spatial alignment.

[0131] Exemplarily, the sound information can be subjected to Fourier transform to obtain a time-frequency graph of the sound information, and then the time-frequency graph is down-sampled at at least one resolution to obtain multi-scale audio features. The resolution at which the time-frequency graph is sampled is the same as the resolution at which the image information is sampled.

[0132] In the embodiments of the present specification, the sound information can be converted from the time domain to the frequency domain to obtain a visualized time-frequency graph, and the sound information is also converted to the spatial dimension, so as to facilitate alignment of the image information and the sound information in the spatial dimension, and obtain the fusion features of the image information and the sound information of the target video in space, so as to provide the image information and the sound information in space for the target video processing model to process the target video task, and ensure the accuracy of the task processing.

[0133] In an optional implementation of the present embodiment, the fusion visual features and the fusion audio features are subjected to time alignment to obtain time alignment features, including:

[0134] For a target time of the target video, the target fusion visual features of the target time and the target fusion audio features of the target time are determined, and the target fusion visual features and the target fusion audio features are fused to obtain audio-visual fusion features of the target time, wherein the target time is any time of the target video.

[0135] The audio-visual fusion features of each time of the target video are taken as the time alignment features.

[0136] In actual implementation, the target video can be a video with a time length of t, and for a target time, the target fusion visual features of the target time and the target fusion audio features of the target time can be fused to obtain audio-visual fusion features of the target time. Each time of the target video can be taken as a target time to obtain corresponding audio-visual fusion features, so as to realize time alignment.

[0137] In specific implementation, the target fusion visual features of the target time and the target fusion audio features of the target time can be spliced together to obtain spliced features of the target time, and context attention is performed on the target fusion visual features and the target fusion audio features in the spliced features of the target time to obtain audio-visual fusion features of the target time. The context attention is an attention mechanism in the field of deep learning, which is important in processing sequence data, image recognition, natural language processing (NLP) and other tasks, and aims to enable the model to flexibly focus on the relevant part of the input data, dynamically allocate attention weights according to different context environments, and thus improve the expression ability and understanding depth of the model.

[0138] In the embodiments of the present specification, for each time instant of the to-be-processed video, the corresponding fused visual feature and the fused audio feature can be spliced together, and context attention calculation is performed to obtain the audio-visual fusion feature of each time instant as the time alignment feature, thereby further fusing the image information and the sound information in the time dimension, providing the image information and the sound information in the time dimension as the fusion feature for the target video processing model to process the target video task, and ensuring the accuracy of the task processing.

[0139] In an optional implementation of the present embodiment, the video task further includes task information, and the target video processing model includes a projection layer and a video processing layer; according to the spatial fusion feature and the time alignment feature, a target processing result of the target video task is obtained by using the target video processing model corresponding to the target video task, including:

[0140] Based on the projection layer, the spatial fusion feature is projected in space to obtain a spatial projection feature, and the time alignment feature is projected in time to obtain a time projection feature.

[0141] According to the spatial projection feature, the time projection feature, and the text prompt word, the target processing result of the target video task is obtained by using the video processing layer, wherein the text prompt word is generated based on the task information.

[0142] The spatial projection feature is a feature obtained by mapping the spatial fusion feature to a language space, and the time projection feature is a feature obtained by mapping the time alignment feature to a language space. That is, the spatial fusion feature and the time alignment feature are further converted into an expression space that is more biased towards text understanding, so that information from non-language modalities can be "translated" into a form that is easier for a language model to understand and process. Through the conversion of the spatial fusion feature and the time alignment feature to the language domain, the boundaries between vision and language can be crossed, and deeper language understanding can be promoted.

[0143] It should be noted that, in addition to the to-be-processed video, the video task can also include task information, which is used to instruct the target video processing model to analyze the spatial fusion feature and the time alignment feature according to the corresponding processing manner, and output the target processing result of the target video task.

[0144] In actual implementation, the task information can be directly used as the text prompt word, or a corresponding prompt sentence can be generated according to the task information as the text prompt word.

[0145] As an example, the front end uploads a video to be processed and task information "describe the video to be processed", at this time, a text prompt word "what does this video describe?" can be generated according to the task information, and the video processing layer can output the target processing result of the target video task "the video guides the audience on how to make pancakes, starting with the batter in a frying pan, cooking step by step until the pancakes turn golden yellow. The accompanying English audio commentary adds insight, highlighting the speed of the making process and guiding the audience to stack the pancakes, the combination of close-up shots and descriptive audio provides a practical and engaging tutorial that makes pancake preparation look simple and enjoyable, the combination of visual steps and audio cues provides a clear, step-by-step process for making a delicious pancake".

[0146] In addition, the target video processing model can include a projection layer and a video processing layer. The projection layer can be a two-layer MLP and a one-layer GELU, which is used to project the spatial fusion features and the time alignment features into a language space. The video processing layer can be a large model (LLM) used to analyze and process the features projected into the language space based on the text prompt word to give the target processing result of the target video task.

[0147] In actual implementation, the spatial projection feature can be obtained by performing spatial projection on the spatial fusion feature based on the projection layer, and the time projection feature can be obtained by performing time projection on the time alignment feature. Then, the spatial projection feature, the time projection feature, and the text prompt word are sequentially spliced into input data, and the input data is input into the video processing layer. The video processing layer analyzes and processes to obtain the target processing result of the target video task.

[0148] In the embodiments of the present specification, the image and sound of the video to be processed are fused in a fine-grained manner in the spatial and temporal dimensions, and projected into a language space to obtain spatial projection features and time projection features, so as to obtain the target processing result of the target video task in combination with the text prompt word. This can help the target video processing model to better mine complementary audio-visual information and improve the task processing accuracy.

[0149] In an optional implementation of the present embodiment, the spatial fusion feature includes a fusion visual feature and a fusion audio feature, and the projection layer includes a visual projection layer and an audio projection layer. The spatial projection of the spatial fusion feature to obtain the spatial projection feature includes:

[0150] The fusion visual feature is input into the visual projection layer for visual projection to obtain a visual projection feature.

[0151] The fusion audio feature is input into the audio projection layer for audio projection to obtain an audio projection feature.

[0152] The visual projection feature and the audio projection feature are taken as the spatial projection feature.

[0153] The visual projection feature is a feature obtained by mapping the fused visual feature to a language space, and the audio projection feature is a feature obtained by mapping the fused audio feature to the language space. That is, the fused visual feature is converted from the visual modality to an expression space that is more biased towards text understanding, and the fused audio feature is converted from the auditory modality to an expression space that is more biased towards text understanding, so that information of the visual modality, the auditory modality, and the like non-language modalities can be "translated" into a form that is easier for the language model to understand and process.

[0154] It should be noted that the spatial fusion feature can include the fused visual feature and the fused audio feature, the projection layer can include a visual projection layer and an audio projection layer, the visual projection layer can be used to perform visual projection on the fused visual feature to obtain a visual projection feature, and the audio projection layer can be used to perform audio projection on the fused audio feature to obtain an audio projection feature. In this way, the fused visual feature and the fused audio feature can be projected separately to obtain projection features of the visual modality and the auditory modality, respectively, so as to provide the video processing layer with projection features in the visual and auditory dimensions, facilitating the video processing layer to accurately process the target video task.

[0155] In actual implementation, the visual projection feature, the audio projection feature, the time projection feature, and the text feature of the text prompt word can be sequentially spliced as an input of the video processing layer (large model), so that the video processing layer (large model) can more accurately understand the features of the video to be processed in the visual and auditory dimensions, and implement processing of the target video task.

[0156] In an optional implementation of the embodiment, after obtaining the target processing result of the target video task by using the target video processing model corresponding to the target video task according to the spatial fusion feature and the time alignment feature, the method further includes:

[0157] feeding back the target processing result to the front end.

[0158] In an optional implementation of the embodiment, after feeding back the target processing result to the front end, the method further includes:

[0159] receiving task feedback information sent by the front end, wherein the task feedback information is information fed back by the front end for the target processing result.

[0160] constructing sample data based on the task feedback information.

[0161] optimizing and training the target video processing model based on the sample data.

[0162] The task feedback information is generated and displayed to the front end for the target processing result. According to the subjective evaluation and demand for the content of the target processing result, feedback information is provided for the target processing result. These feedback information is helpful to improve the detailed description of the output of the target video processing model. The task feedback information is an important part of human-computer interaction, which can reflect the real feeling and expectation of the processing result of the target video task. By collecting and utilizing these feedback information, the performance of the target video processing model can be continuously optimized to more accurately meet the actual demand and improve the quality and accuracy of the target processing result.

[0163] In practical applications, the task feedback information can be analyzed to construct sample data, and the target video processing model can be re-optimized and trained. The sample data is historical data that has obtained the target processing result corresponding to the target video task and received the task feedback information, and is used to optimize and train the target video processing model. For example, the video corresponding to the target processing result can be used as a sample video, the result label of the sample video can be generated by analyzing the task feedback information, and the target video processing model can be optimized and trained based on the sample video and the result label.

[0164] In the embodiments of the present specification, the sample data is updated through the interactive feedback mechanism, the target video processing model is further optimized and trained based on the sample data, the model performance of the target video processing model is improved, and the processing accuracy and quality of the video task are improved.

[0165] The following describes the video task processing method in further detail by taking the application of the video task processing method provided by the present specification in the video question and answer scene as an example in combination with FIG. 4. FIG. 4 shows a processing process schematic diagram of a video task processing method provided by an embodiment of the present specification, FIG. 5 shows a schematic diagram of a video task processing interface provided by an embodiment of the present specification, and FIG. 6 shows a schematic diagram of another video task processing interface provided by an embodiment of the present specification.

[0166] As shown in FIG. 5, the user can click the "upload video" control in the video task processing interface. The control can default to show upload prompt information "please upload the video that needs to be processed here". The user selects the video A that needs to be processed by the target video task, and can click the "task information" control. The control can default to show the input prompt information "please input the question that needs to be answered here". It is assumed that the user inputs "what does this video describe?".

[0167] As shown in FIG. 4, the video task processing platform can obtain the uploaded video A, determine the image information and sound information corresponding to the video A. The image information is input into the first visual feature extraction module to obtain the original visual feature 1, and the image information is sampled at at least one spatial scale to obtain the multi-scale visual feature 1; the sound information is input into the first audio feature extraction module to obtain the original audio feature 1, and the sound information is sampled at at least one spatial scale to obtain the multi-scale audio feature 1.

[0168] The multi-scale visual feature 1, the original visual feature 1, and the original audio feature 1 are input into the fusion module 1 for feature fusion to obtain the multi-scale visual spatial feature 1, and the multi-scale visual spatial feature 1 is superimposed on the original visual feature 1 to obtain the initial fusion visual feature 1. Similarly, the multi-scale audio feature 1, the original visual feature 1, and the original audio feature 1 are input into the fusion module 1 for feature fusion to obtain the multi-scale audio spatial feature 1, and the multi-scale audio spatial feature 1 is superimposed on the original audio feature 1 to obtain the initial fusion audio feature 1.

[0169] The initial fusion visual feature 1 is input into the second visual feature extraction module to obtain the original visual feature 2, and the initial fusion audio feature 1 is input into the second audio feature extraction module to obtain the original audio feature 2, and the above feature fusion process is continued until the Nth visual feature extraction module outputs the fusion visual feature; similarly, the Nth audio feature extraction module outputs the fusion audio feature.

[0170] The fusion visual feature and the fusion audio feature of the video A at the t0-t n time are input into the splicing module for splicing fusion to obtain the audio-visual fusion feature at the t0-t n time.

[0171] The fusion visual feature is input into the visual projection layer for visual projection to obtain the visual projection feature; the fusion audio feature is input into the audio projection layer for audio projection to obtain the audio projection feature; the audio-visual fusion feature is input into the time projection layer for time projection to obtain the time projection feature. The visual projection feature, the audio projection feature, the time projection feature, and the text feature of the text prompt word "what does this video describe?" are input into the video processing layer (large model) as input data for processing, and the video processing layer (large model) outputs the answer "The video guides the audience on how to make pancakes, starting with the batter in a flat pan, gradually cooking until the pancakes turn golden. The accompanying English audio commentary adds insight, highlighting the speed of the making process, and guides the audience to stack the pancakes, the combination of close-up shots and descriptive audio provides a practical and engaging tutorial that makes the preparation of pancakes look simple and enjoyable. The combination of visual steps and audio cues provides a clear, step-by-step process for making a delicious pancake."

[0172] The answer is displayed to the front end, as shown in FIG. 6.

[0173] The video task processing method provided by the embodiments of the present specification can perform fine-grained fusion of image information and sound information in a video uploaded by the front end in terms of space and time, promote the interaction between the visual mode and the auditory mode, and enable the video processing layer (large model) to more accurately understand the fine-grained characteristics of the image information and sound information of the video in terms of time and space, answer the corresponding question, and improve the accuracy and richness of the answer.

[0174] Referring to FIG. 7, FIG. 7 shows a flowchart of a training method of a video processing model according to an embodiment of the present specification, specifically including the following steps 702-706.

[0175] Step 702: Obtain training data of a training task, wherein the training data includes a sample video and a result label.

[0176] The sample data is historical data that has obtained a processing result corresponding to the training task, and is used to train an initial video model. The sample video included in the sample data is historical data that has obtained a processing result corresponding to the training task, and the result label is a processing result of the sample video on the training task, which is used to determine the prediction loss of the initial video model on the training task, so as to adjust the model parameters of the initial video model.

[0177] Step 704: Determine a sample space fusion feature and a sample time alignment feature of the sample video, wherein the sample space fusion feature is obtained based on spatial fusion of sample image information and sample sound information of the sample video, the sample space fusion feature includes a sample fusion visual feature and a sample fusion audio feature, and the sample time alignment feature is obtained based on time alignment of the sample fusion visual feature and the sample fusion audio feature.

[0178] It should be noted that step 704 is the fusion processing of the sample video in the model training stage, and its specific implementation process is similar to the fusion processing of the to-be-processed video in the model inference stage of the above-mentioned embodiments. For details, please refer to the above-mentioned embodiments.

[0179] Step 706: According to the sample space fusion feature and the sample time alignment feature, and in combination with the result label, the initial video model is trained to obtain a target video processing model after training.

[0180] In an optional implementation of the present embodiment, the training task further includes training information; according to the sample space fusion feature and the sample time alignment feature, and in combination with the result label, the initial video model is trained to obtain a target video processing model after training, including:

[0181] Project the sample space fusion features to obtain sample space projection features, and project the sample time alignment features to obtain sample time projection features based on the projection layer of the initial video model;

[0182] According to the sample space projection features, the sample time projection features, and the sample prompt words, the video processing layer of the initial video model is used to obtain a prediction processing result of the training task, wherein the sample prompt words are generated based on the training information.

[0183] According to the prediction processing result and the result label, the model parameters of the projection layer and the video processing layer of the initial video model are adjusted, and the operation step of obtaining the training data of the training task is returned until a training stop condition is reached to obtain a target video processing model.

[0184] Specifically, according to the prediction processing result and the result label, a loss value can be calculated, the loss value can be back propagated, the model parameters of the projection layer and the video processing layer of the initial video model can be adjusted, and then the step of obtaining the training data of the training task can be returned. The initial video model is continuously selected and trained until the training stop condition is reached, the training is completed, and the target video processing model is obtained. The training stop condition can be that the loss value is less than a loss value threshold, or the number of iterations reaches a number threshold, or the loss value threshold and the number threshold can be combined to determine whether the training stop condition is reached.

[0185] The loss value threshold is a critical value of the loss value. If the loss value is greater than or equal to the loss value threshold, it means that there is still a certain deviation between the prediction result of the initial video model and the true result, and the model parameters of the projection layer and the video processing layer of the initial video model still need to be adjusted. At this time, it is determined that the training stop condition is not reached. If the loss value is less than the loss value threshold, it means that the prediction result of the initial video model is close enough to the true result, and the training can be stopped. At this time, it can be determined that the training stop condition is reached. The value of the loss value threshold is selected according to the actual situation.

[0186] In addition, the number threshold is set according to the actual situation. When the number of iterations reaches the number threshold, it means that the number of training of the initial video model is sufficient, and the prediction result of the initial video model is close enough to the true result, and the training can be stopped.

[0187] In the embodiments of the present specification, the specific training situation of the initial video model can be judged according to the loss value, and the model parameters of the projection layer and the video processing layer of the initial video model can be adjusted in reverse according to the loss value in the case of unqualified training, so as to improve the processing capability of the projection layer and the video processing layer. The projection layer and the video processing layer of the initial video model can be trained jointly, the training rate is high, and the training effect is good.

[0188] The training method of the video processing model provided by the embodiments of the present specification can perform fine-grained fusion of image information and sound information of a sample video in space and time, promote the interaction between visual and auditory modalities, so that the initial video model can more accurately understand the fine-grained characteristics of image information and sound information of the sample video in time and space, obtain the prediction processing result of the training task, and combine the result label to jointly train the projection layer and the video processing layer of the initial video model, which has fast training speed and high task processing accuracy of the target video processing model after training.

[0189] Referring to FIG. 8, FIG. 8 shows a flowchart of an information processing method based on a video processing model according to an embodiment of the present specification, which is applied to a task platform and specifically includes the following steps 802-804.

[0190] Step 802: receiving a model request sent by a terminal device, wherein the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter.

[0191] It should be noted that the task platform is an online platform that provides model services, allowing developers, enterprises or individual users to submit tasks, obtain resources or achieve specific functions through API calls, web interfaces or other interaction methods. In the embodiments of the present specification, the task platform is a cloud service platform that can provide deep learning models, allowing external applications (such as e-commerce platforms, navigation applications, social applications, video production applications and online education platforms, etc.) to access and use. The task platform manages the training, updating, optimization, load balancing, scheduling and inference of the model to ensure efficient and accurate response to requests from terminal devices. The terminal device is a terminal device that requests to obtain a target video processing model.

[0192] In actual implementation, the model request is a request sent by a terminal device to a task platform to obtain or use a specific model service, which can include at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter.

[0193] The target scene is a specific environment or situation to which the target video processing model is to be applied, and it describes the purpose and expected effect of the model service. The scene identifier of the target scene is identification information for identifying the target scene, which is used to assist in quickly locating the model related to a specific scene, such as "video question answering", "video navigation", "video monitoring", "video quality evaluation", "video object positioning", "video commodity positioning and search", and the like. The scene input data of the target scene can be input data used to train the initial video model to adapt to the target scene under the target scene, such as sample videos with answers in the video question answering scene. In addition, the model specification parameter is an attribute specification parameter that defines the model architecture, performance, or configuration, such as model type, model type, model size, and the like. The model specification parameter is used to determine the model that meets the user's needs.

[0194] For example, the user selects the scene identifier of the target scene as "video question answering" on the task platform client of the terminal device, selects the required model size as 32GB, uploads the scene input data of the target scene, and the terminal device generates a model request and sends it to the task platform. The task platform receives the model request sent by the terminal device to the task platform, wherein the model request includes the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameter.

[0195] In the embodiments of the present specification, the user can independently select at least one of the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameter on the terminal device, trigger a model request, and the task platform can receive the model request sent by the terminal device, provide reference information for selecting the target video processing model based on at least one of the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameter, and the applicable scene is more extensive.

[0196] Step 804: determining a corresponding target video processing model from at least one video processing model based on the model request, wherein the at least one video processing model is trained based on the training method of the video processing model described above.

[0197] It should be noted that, based on the model request, the corresponding target video processing model is determined from at least one video processing model. One optional way is to find the corresponding target video processing model from at least one video processing model included in the model library based on the model request; another optional way is to train the target video processing model based on the model request; and still another optional way is to construct the target video processing model based on the model request, which is not limited here.

[0198] For example, at least one pre-trained video processing model can be found from the model library based on the scene identification of the target scene, and then a video processing model of a corresponding size can be selected from the at least one video processing model based on the model specification parameter, and then the video processing model of the corresponding size can be trained based on the scene input data of the target scene to obtain a target video processing model suitable for user demand.

[0199] The at least one video processing model is obtained according to a training method of a video processing model. The embodiments of the present specification and the embodiments of the description of FIG. 7 are based on the same inventive concept, and the specific manners are described above in the content of model training, which will not be described here again.

[0200] In an optional embodiment of the present embodiment, the model request includes the scene identification of the target scene.

[0201] Based on the model request, a corresponding target video processing model is determined from the at least one video processing model, including:

[0202] Based on the scene identification of the target scene, a target video processing model suitable for the target scene is found from the model library, wherein the model library stores at least one video processing model suitable for different video processing scenes.

[0203] It should be noted that the model library is a database for storing and managing various pre-trained deep learning models. The plurality of video processing models suitable for different video processing scenes cover different application scenarios and requirements. The model library allows users to select appropriate models according to their own needs, or directly use models for video task processing through API calls.

[0204] The plurality of video processing models suitable for different video processing scenes are a plurality of models stored in the model library, which are specially designed for different video processing scenes. Each model is optimized for a specific application environment. Any video processing model is obtained according to the above-mentioned video processing model training method. For example, a target video processing model suitable for the video question and answer scene can be found from the model library based on the scene identification of the target scene "video question and answer".

[0205] In the embodiments of the present specification, based on the scene demand, the target video processing model suitable for the scene is accurately found through the scene identification, so that the target processing result is more accurate and fits the scene, thereby improving the user experience and the quality of video task processing.

[0206] As an example, the task platform can provide target video processing models in various scenarios, such as a video question answering scenario, and can provide a corresponding target video processing model based on a model request sent by a terminal device. Since the target video processing model is selected from at least one video processing model trained based on the above-described video processing model training method, the target video processing model is trained in combination with the visual and auditory modalities, can accurately understand visual and audio content in fine granularity, and can answer related questions. Alternatively, the target video processing model can identify a sound source, determine the position of an object described by an audio signal in a video, and answer an object-level question. Such capabilities are important for improving perception and have important significance in the fields of navigation, monitoring, and interactive technology.

[0207] In addition, the method can also be applied to the e-commerce field. In a "video product positioning and search" scenario, based on the selected target video processing model, a video can be accurately understood or a description of the video can be obtained, and a product can be accurately searched and positioned using the sound in the video, thereby improving user experience and search efficiency.

[0208] In an optional implementation of the embodiment, the model request includes scene input data of the target scenario.

[0209] Based on the model request, a corresponding target video processing model is determined from at least one video processing model, including:

[0210] From the at least one video processing model, an initial video processing model suitable for the target scenario is determined.

[0211] Based on the scene input data of the target scenario, the initial video processing model is trained to obtain the target video processing model.

[0212] In actual implementation, the model request can include scene input data of the target scenario, and the target video processing model is a video processing model suitable for the target scenario.

[0213] By way of example, the general video processing model is a basic video processing model that is trained to be suitable for different video processing scenarios but is not optimized for any specific scenario. For example, based on scene input data of a video question answering scenario, the general video processing model is trained to obtain a target video processing model suitable for the video question answering scenario.

[0214] In the embodiments of the present specification, based on the scene requirements, the general video processing model is further trained based on the scene input data to obtain a target video processing model suitable for the scene, so that the target processing result is more accurate and more suitable for the scene, thereby improving user experience and the processing quality of the video task.

[0215] In an optional implementation of the embodiment, the model request comprises a model specification parameter;

[0216] Based on the model request, a corresponding target video processing model is determined from at least one video processing model, comprising:

[0217] Based on the model specification parameter, a corresponding target video processing model is searched from a model library, wherein the model library stores a plurality of video processing models with different model specification parameters.

[0218] The model specification parameter can be a model size, for example, based on a model size of 32 GB, a target video processing model with the corresponding size is searched from the model library.

[0219] In the embodiments of the present specification, based on the model specification requirement, the corresponding target video processing model is accurately found through the model specification parameter, ensuring efficient and stable operation of the target video processing model and improving user experience.

[0220] In an optional implementation of the embodiment, after determining the corresponding target video processing model from at least one video processing model based on the model request, the method further comprises:

[0221] The target video processing model is deployed, and a video processing interface is constructed based on the target video processing model, so that the terminal device schedules the target video processing model to execute a video task.

[0222] It should be noted that the video processing interface is an interactive programming interface for the terminal device to schedule the target video processing model, which is usually provided in the form of API. Through the video processing interface, the user can input task data of the target video task, such as a video to be processed and task information, and effectively control the output of the model, such as a description for the video.

[0223] In actual implementation, one optional way of deploying the target video processing model is to deploy the target video processing model on a distributed system of a task platform. For example, the target video processing model is deployed on the distributed system of the task platform, and a video processing interface is constructed based on the target video processing model and provided to the terminal device, so that the terminal device schedules the target video processing model to execute a target video task in a video processing scenario.

[0224] In the embodiments of the present specification, efficient calling of the terminal is achieved, video task processing is optimized, and the processing quality and response speed of the video task are improved.

[0225] The information processing method based on the video processing model provided in the embodiments of the present specification is suitable for obtaining a target video processing model according to user requirements, realizes personalized model service, provides an efficient, flexible and easy-to-use model service method for users, and improves user experience.

[0226] Corresponding to the method embodiments described above, the specification also provides task platform embodiments. FIG. 9 shows a structural schematic diagram of a task platform according to an embodiment of the specification. As shown in FIG. 9, the task platform 900 includes a request interface 902 and a response unit 904.

[0227] The request interface 902 is configured to receive a model request sent by a terminal device, where the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter.

[0228] The response unit 904 is configured to determine a corresponding target video processing model from at least one video processing model based on the model request, where the at least one video processing model is obtained based on the training method of the video processing model according to any one of claims 13-14.

[0229] Optionally, the task platform further includes a video processing interface, which is constructed based on the target video processing model.

[0230] The video processing interface is configured to allow the terminal device to schedule and execute a video task.

[0231] Optionally, the model request includes the scene identifier of the target scene.

[0232] The response unit 904 is further configured to:

[0233] Based on the scene identifier of the target scene, the target video processing model suitable for the target scene is searched from a model library, where the model library stores at least one video processing model suitable for different video processing scenes.

[0234] Optionally, the model request includes the scene input data of the target scene.

[0235] The response unit 904 is further configured to:

[0236] From the at least one video processing model, an initial video processing model suitable for the target scene is determined.

[0237] Based on the scene input data of the target scene, the initial video processing model is trained to obtain the target video processing model.

[0238] Optionally, the model request includes the model specification parameter.

[0239] The response unit 904 is further configured to:

[0240] Based on the model specification parameter, the corresponding target video processing model is searched from a model library, where the model library stores a plurality of video processing models with different model specification parameters.

[0241] Optionally, the task platform further comprises a deployment module configured to:

[0242] deploy the target video processing model, and construct a video processing interface based on the target video processing model, so that the terminal device schedules the target video processing model to execute the video task.

[0243] In the embodiments of the present specification, the task platform is adapted to obtain the target video processing model according to user demand, realizes personalized model service, provides an efficient, flexible and easy-to-use model service platform for users, and improves user experience.

[0244] The above is a schematic scheme of the task platform of the present embodiment. It should be noted that the technical scheme of the task platform belongs to the same concept as the technical scheme of the information processing method based on the video processing model described above. The details of the technical scheme of the task platform that are not described in detail can be referred to the description of the technical scheme of the information processing method based on the video processing model described above.

[0245] Corresponding to the method embodiments described above, the present specification also provides video task processing device embodiments. FIG. 10 shows a structural schematic diagram of a video task processing device according to an embodiment of the present specification. As shown in FIG. 10, the device comprises:

[0246] The first obtaining module 1002 is configured to obtain task data of a target video task, wherein the task data comprises a video to be processed;

[0247] The first determining module 1004 is configured to determine spatial fusion features and time alignment features of the video to be processed, wherein the spatial fusion features are obtained based on spatial fusion of image information and sound information of the video to be processed, the spatial fusion features comprise fusion visual features and fusion audio features, and the time alignment features are obtained based on time alignment of the fusion visual features and the fusion audio features;

[0248] The obtaining module 1006 is configured to obtain a target processing result of the target video task by using a target video processing model corresponding to the target video task according to the spatial fusion features and the time alignment features.

[0249] Optionally, the first determining module 1004 is further configured to:

[0250] determine image information and sound information corresponding to the video to be processed;

[0251] spatially fuse the image information and the sound information to obtain fusion visual features and fusion audio features, and take the fusion visual features and the fusion audio features as the spatial fusion features;

[0252] The time alignment feature is obtained by time aligning the fused visual feature and the fused audio feature.

[0253] Optionally, the first determining module 1004 is further configured to:

[0254] The multi-scale visual spatial features are obtained by respectively spatially fusing the multi-scale visual features of the image information with the image information and the sound information by using the visual model and the audio model, and fusing the multi-scale visual spatial features with the image information to obtain the fused visual feature.

[0255] The multi-scale audio spatial features are obtained by respectively spatially fusing the multi-scale audio features of the sound information with the image information and the sound information by using the visual model and the audio model, and fusing the multi-scale audio spatial features with the sound information to obtain the fused audio feature.

[0256] Optionally, the first determining module 1004 is further configured to:

[0257] The audio-visual fusion feature of the target time is obtained by fusing the target fused visual feature of the target time and the target fused audio feature of the target time, wherein the target time is any time of the to-be-processed video.

[0258] The audio-visual fusion features of each time of the to-be-processed video are taken as the time alignment feature.

[0259] Optionally, the visual model includes N visual feature extraction modules, and the audio model includes N audio feature extraction modules; the first determining module 1004 is further configured to:

[0260] The following steps are repeatedly executed until the value of i is N, and the initial value of i is 1:

[0261] The image information is input into the i-th visual feature extraction module to obtain the original visual feature extracted by the i-th visual feature extraction module, and the sound information is input into the i-th audio feature extraction module to obtain the original audio feature extracted by the i-th audio feature extraction module, wherein the initial value of i is 1.

[0262] The multi-scale visual features are obtained by downsampling the image information in at least one spatial scale.

[0263] The multi-scale visual spatial features are obtained by fusing the multi-scale visual features, the original visual feature extracted by the i-th visual feature extraction module, and the original audio feature extracted by the i-th audio feature extraction module.

[0264] fusing the multi-scale visual spatial feature and the original visual feature extracted by the i-th visual feature extraction module to obtain an initial fusion visual feature;

[0265] determining that the initial fusion visual feature is new image information, and determining that the value of i is increased by 1;

[0266] The initial fusion visual feature obtained by fusing the multi-scale visual spatial feature and the original visual feature extracted by the N-th visual feature extraction module is a fusion visual feature.

[0267] Optionally, the visual model includes N visual feature extraction modules, and the audio model includes N audio feature extraction modules; the first determination module 1004 is further configured to:

[0268] repeating the following steps until the value of i is N, and the initial value of i is 1:

[0269] inputting the image information into the i-th visual feature extraction module to obtain an original visual feature extracted by the i-th visual feature extraction module, and inputting the sound information into the i-th audio feature extraction module to obtain an original audio feature extracted by the i-th audio feature extraction module;

[0270] down-sampling the sound information at at least one spatial scale to obtain a multi-scale audio feature;

[0271] performing feature fusion on the multi-scale audio feature, the original visual feature extracted by the i-th visual feature extraction module, and the original audio feature extracted by the i-th audio feature extraction module to obtain a multi-scale audio spatial feature;

[0272] fusing the multi-scale audio spatial feature and the original audio feature extracted by the i-th audio feature extraction module to obtain an initial fusion audio feature;

[0273] determining that the initial fusion audio feature is new sound information, and determining that the value of i is increased by 1;

[0274] The initial fusion audio feature obtained by fusing the multi-scale audio spatial feature and the original audio feature extracted by the N-th audio feature extraction module is a fusion audio feature.

[0275] Optionally, the spatial scale is resolution; the first determination module 1004 is further configured to:

[0276] performing time domain transformation on the sound information to obtain a time-frequency graph of the sound information;

[0277] down-sampling the time-frequency graph at at least one resolution to obtain a multi-scale audio feature.

[0278] Optionally, the video task further comprises task information, and the target video processing model comprises a projection layer and a video processing layer; the obtaining module 1006 is further configured to:

[0279] Based on the projection layer, the spatial fusion feature is spatially projected to obtain a spatial projection feature, and the time alignment feature is temporally projected to obtain a time projection feature.

[0280] According to the spatial projection feature, the time projection feature, and the text prompt word, a target processing result of the target video task is obtained by using the video processing layer, wherein the text prompt word is generated based on the task information.

[0281] Optionally, the spatial fusion feature comprises a fusion visual feature and a fusion audio feature, and the projection layer comprises a visual projection layer and an audio projection layer; the obtaining module 1006 is further configured to:

[0282] The fusion visual feature is input into the visual projection layer for visual projection to obtain a visual projection feature;

[0283] The fusion audio feature is input into the audio projection layer for audio projection to obtain an audio projection feature;

[0284] The visual projection feature and the audio projection feature are taken as the spatial projection feature.

[0285] Optionally, the first obtaining module 1002 is further configured to:

[0286] Generate a target video task in response to the front-end inputted video to be processed and task information;

[0287] Correspondingly, the apparatus further comprises a feedback module configured to:

[0288] Feed back the target processing result to the front end.

[0289] Optionally, the apparatus further comprises an optimization module configured to:

[0290] Receive task feedback information sent by the front end, wherein the task feedback information is information fed back by the front end for the target processing result;

[0291] Based on the task feedback information, construct sample data;

[0292] Based on the sample data, the target video processing model is trained and optimized.

[0293] The video task processing apparatus provided by the embodiment of the present specification can perform fine-grained fusion of image information and sound information of a to-be-processed video in spatial and temporal dimensions, promote the interaction between visual and auditory modalities, and enable a target video processing model to more accurately understand the fine-grained characteristics of the image information and sound information of the to-be-processed video in time and space, and obtain an accurate target processing result of the target video task.

[0294] The above is a schematic scheme of the video task processing apparatus of the embodiment. It should be noted that the technical scheme of the video task processing apparatus belongs to the same concept as the technical scheme of the video task processing method described above, and the details of the technical scheme of the video task processing apparatus that are not described in detail can be referred to the description of the technical scheme of the video task processing method.

[0295] Corresponding to the method embodiment described above, the present specification also provides an embodiment of a training apparatus of a video processing model, and FIG. 11 shows a structural schematic diagram of a training apparatus of a video processing model according to an embodiment of the present specification. As shown in FIG. 11, the apparatus comprises:

[0296] The second acquisition module 1102 is configured to acquire training data of a training task, wherein the training data comprises a sample video and a result label;

[0297] The second determination module 1104 is configured to determine a sample spatial fusion feature and a sample time alignment feature of the sample video, wherein the sample spatial fusion feature is obtained by performing spatial fusion on sample image information and sample sound information of the sample video, the sample spatial fusion feature comprises a sample fusion visual feature and a sample fusion audio feature, and the sample time alignment feature is obtained by performing time alignment on the sample fusion visual feature and the sample fusion audio feature;

[0298] The training module 1106 is configured to train an initial video model according to the sample spatial fusion feature and the sample time alignment feature in combination with the result label, and obtain a target video processing model after training.

[0299] Optionally, the training task further comprises training information; and the training module 1106 is further configured to:

[0300] perform spatial projection on the sample spatial fusion feature to obtain a sample spatial projection feature, and perform time projection on the sample time alignment feature to obtain a sample time projection feature based on a projection layer of the initial video model;

[0301] According to the sample space projection feature, the sample time projection feature, and the sample prompt word, a prediction processing result of the training task is obtained by using a video processing layer of the initial video model, wherein the sample prompt word is generated based on training information;

[0302] According to the prediction processing result and the result label, the model parameters of the projection layer and the video processing layer of the initial video model are adjusted, and the operation step of obtaining the training data of the training task is returned until a training stop condition is reached, and a target video processing model trained is obtained.

[0303] The training device of the video processing model provided by the embodiments of the present specification can perform fine-grained fusion of image information and sound information of a sample video in space and time, promote the interaction between visual and auditory modalities, and enable the initial video model to more accurately understand the fine-grained features of the image information and sound information of the sample video in time and space, obtain a prediction processing result of a training task, and combine a result label to jointly train the projection layer and the video processing layer of the initial video model, thereby achieving fast training and high task processing accuracy of the target video processing model after training.

[0304] The above is a schematic scheme of the training device of the video processing model of the present embodiment. It should be noted that the technical scheme of the training device of the video processing model belongs to the same concept as the technical scheme of the training method of the video processing model described above, and the details of the technical scheme of the training device of the video processing model that are not described in detail can be referred to the description of the technical scheme of the training method of the video processing model.

[0305] FIG. 12 shows a structural block diagram of a computing device according to an embodiment of the present specification. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 through a bus 1230, and a database 1250 is used to save data.

[0306] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of networks such as the Internet. The access device 1240 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as a wired or wireless network interface, for example, an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0307] In one embodiment of the present specification, the above-mentioned components of the computing device 1200 and other components not shown in FIG. 12 can also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 12 is only for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0308] The computing device 1200 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 can also be a mobile or stationary server.

[0309] The memory 1210 is configured to store computer programs / instructions, and the processor 1220 is configured to execute the computer programs / instructions stored in the memory 1210. The computer programs / instructions, when executed by the processor, implement the steps of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model.

[0310] The above describes a schematic solution of the computing device in this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model belong to the same concept. For details of the technical solution of the computing device that are not described in detail, refer to the description of the technical solution of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model.

[0311] An embodiment of the present specification further provides a computer readable storage medium storing computer programs / instructions. The computer programs / instructions, when executed by a processor, implement the steps of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model.

[0312] The above describes a schematic solution of the computer readable storage medium in this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model belong to the same concept. For details of the technical solution of the storage medium that are not described in detail, refer to the description of the technical solution of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model.

[0313] An embodiment of the present specification further provides a computer program product including computer programs / instructions. The computer programs / instructions, when executed by a processor, implement the steps of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model.

[0314] The above describes a schematic solution of the computer program product in this embodiment. It should be noted that the technical solution of the computer program product and the technical solution of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model belong to the same concept. For details of the technical solution of the computer program product that are not described in detail, refer to the description of the technical solution of the video task processing method, the training method of the video processing model, and the information processing method based on the video processing model.

[0315] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0316] The computer program / instructions include a computer program code, which can be in a form of source code, object code, executable file, or some intermediate form etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0317] It should be noted that for the foregoing method embodiments, the acts described can be performed in a different order from that described, and that various elements can be added, deleted, modified, or rearranged without departing from the scope of the application as set forth by the claims. Further, the above-described embodiments of the application are merely possible implementations of the present application, and thus are not intended to limit the scope of the present application. It should be understood by one of ordinary skill in the art that more or less acts can be included, and further, the order of the acts can be changed without departing from the scope of the present application. Also, it should be understood that described acts can be carried out in sequence, in parallel, or in a combination of both.

[0318] In the above-described embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0319] The above-disclosed specification of the present application is only used to help explain the present application. The alternative embodiments do not describe all the details, nor limit the application to the specific embodiments described. Obviously, according to the content of the embodiments of the present application, many modifications and changes can be made. The present application selects and describes these embodiments in order to better explain the principles and practical application of the embodiments of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited by the claims and their full scope and equivalents.

Claims

1. A video task processing method, comprising: Acquire task data for the target video task, wherein the task data includes the video to be processed; The spatial fusion features and temporal alignment features of the video to be processed are determined. The spatial fusion features are obtained by spatial fusion of image information and sound information of the video to be processed. The spatial fusion features include fused visual features and fused audio features. The temporal alignment features are obtained by temporal alignment of the fused visual features and the fused audio features. Based on the spatial fusion features and the temporal alignment features, the target video processing result of the target video task is obtained using the target video processing model corresponding to the target video task.

2. The method according to claim 1, wherein determining the spatial fusion features and temporal alignment features of the video to be processed includes: Determine the image and audio information corresponding to the video to be processed; Spatial fusion is performed on the image information and the sound information to obtain fused visual features and fused audio features, and the fused visual features and the fused audio features are used as the spatial fusion features; The fused visual features and the fused audio features are time-aligned to obtain the time-aligned features.

3. The method according to claim 2, wherein spatial fusion of the image information and the sound information to obtain fused visual features and fused audio features includes: Using a visual model and an audio model, the multi-scale visual features of the image information are spatially fused with the image information and the sound information to obtain multi-scale visual spatial features, and the multi-scale visual spatial features are fused with the image information to obtain the fused visual features; Using a visual model and an audio model, the multi-scale audio features of the sound information are spatially fused with the image information and the sound information to obtain multi-scale audio spatial features. The multi-scale audio spatial features are then fused with the sound information to obtain the fused audio features.

4. The method according to claim 2, wherein performing time alignment on the fused visual features and the fused audio features to obtain the time-aligned features includes: For a target time in the video to be processed, the target fusion visual features and the target fusion audio features of the target time are determined. The target fusion visual features and the target fusion audio features are fused to obtain the audio-visual fusion features of the target time, wherein the target time is any time in the video to be processed. The audio-visual fusion features of the video to be processed at each moment are used as the time alignment features.

5. The method according to claim 3, wherein the visual model comprises N visual feature extraction modules, and the audio model comprises N audio feature extraction modules; The method involves using visual and audio models to spatially fuse the multi-scale visual features of the image information with the image information and the sound information to obtain multi-scale visual spatial features, and then fusing the multi-scale visual spatial features with the image information to obtain the fused visual features, including: Repeat the following steps until the value of i is N, where the initial value of i is 1: The image information is input into the i-th visual feature extraction module to obtain the original visual features extracted by the i-th visual feature extraction module, and the sound information is input into the i-th audio feature extraction module to obtain the original audio features extracted by the i-th audio feature extraction module, wherein the initial value of i is 1; The image information is sampled at at least one spatial scale to obtain multi-scale visual features; The multi-scale visual features, the original visual features extracted by the i-th visual feature extraction module, and the original audio features extracted by the i-th audio feature extraction module are fused to obtain multi-scale visual spatial features; The multi-scale visual spatial features are fused with the original visual features extracted by the i-th visual feature extraction module to obtain the initial fused visual features; The initial fused visual features are determined to be new image information, and the value of i is incremented by 1. The initial fused visual feature obtained by fusing the multi-scale visual spatial features with the original visual features extracted by the Nth visual feature extraction module is the fused visual feature.

6. The method according to claim 3, wherein the visual model comprises N visual feature extraction modules, and the audio model comprises N audio feature extraction modules; The process involves using visual and audio models to spatially fuse the multi-scale audio features of the sound information with the image information and the sound information, respectively, to obtain multi-scale audio spatial features. Then, the multi-scale audio spatial features are fused with the sound information to obtain the fused audio features, including: Repeat the following steps until the value of i is N, where the initial value of i is 1: The image information is input into the i-th visual feature extraction module to obtain the original visual features extracted by the i-th visual feature extraction module, and the sound information is input into the i-th audio feature extraction module to obtain the original audio features extracted by the i-th audio feature extraction module. The sound information is sampled at at least one spatial scale to obtain multi-scale audio features; The multi-scale audio features, the original visual features extracted by the i-th visual feature extraction module, and the original audio features extracted by the i-th audio feature extraction module are fused to obtain multi-scale audio spatial features; The multi-scale audio spatial features are fused with the original audio features extracted by the i-th audio feature extraction module to obtain the initial fused audio features; The initial fused audio features are determined to be new sound information, and the value of i is incremented by 1. The initial fused audio feature obtained by fusing the multi-scale audio spatial features with the original audio features extracted by the Nth audio feature extraction module is the fused audio feature.

7. The method according to claim 6, wherein the spatial scale is resolution; the step of sampling the sound information at at least one spatial scale to obtain multi-scale audio features includes: Perform a time-domain transformation on the sound information to obtain a time-frequency diagram of the sound information; The time-frequency graph is sampled at at least one resolution to obtain the multi-scale audio features.

8. The method according to any one of claims 1-7, wherein the video task further includes task information, and the target video processing model includes a projection layer and a video processing layer; the step of obtaining the target processing result of the target video task by utilizing the target video processing model corresponding to the target video task based on the spatial fusion feature and the temporal alignment feature includes: Based on the projection layer, the spatial fusion feature is spatially projected to obtain the spatial projection feature, and the temporal alignment feature is temporally projected to obtain the temporal projection feature. Based on the spatial projection features, the temporal projection features, and the text prompts, the target processing result of the target video task is obtained using the video processing layer, wherein the text prompts are generated based on the task information.

9. The method according to claim 8, wherein the spatial fusion feature includes fused visual features and fused audio features, and the projection layer includes a visual projection layer and an audio projection layer; the step of spatially projecting the spatial fusion feature to obtain a spatial projection feature includes: The fused visual features are input into the visual projection layer for visual projection to obtain visual projection features. The fused audio features are input into the audio projection layer for audio projection to obtain audio projection features; The visual projection features and the audio projection features are used as the spatial projection features.

10. The method according to any one of claims 1-7, wherein acquiring the task data of the target video task comprises: The target video task is generated in response to the video to be processed and task information input from the front end; Accordingly, after obtaining the target processing result of the target video task using the target video processing model corresponding to the target video task based on the spatial fusion feature and the temporal alignment feature, the method further includes: The target processing result is fed back to the front end.

11. A training method for a video processing model, comprising: Acquire training data for the training task, wherein the training data includes sample videos and result labels; The sample spatial fusion feature and sample temporal alignment feature of the sample video are determined. The sample spatial fusion feature is obtained by spatial fusion of sample image information and sample sound information of the sample video. The sample spatial fusion feature includes sample fusion visual features and sample fusion audio features. The sample temporal alignment feature is obtained by temporal alignment of the sample fusion visual features and sample fusion audio features. Based on the sample space fusion features and the sample time alignment features, combined with the result labels, the initial video model is trained to obtain the trained target video processing model.

12. The method according to claim 11, wherein the training task further includes training information; The step of training the initial video model based on the sample space fusion features and the sample time alignment features, combined with the result labels, to obtain the trained target video processing model includes: Based on the projection layer of the initial video model, spatial projection is performed on the sample spatial fusion features to obtain sample spatial projection features, and temporal projection is performed on the sample temporal alignment features to obtain sample temporal projection features. Based on the sample spatial projection features, the sample temporal projection features, and the sample prompt words, the prediction processing result of the training task is obtained using the video processing layer of the initial video model, wherein the sample prompt words are generated based on the training information; Based on the prediction processing results and the result labels, adjust the model parameters of the projection layer and video processing layer of the initial video model, and return to the operation steps of obtaining training data for the training task until the training stop condition is met, and obtain the target video processing model that has been trained.

13. An information processing method based on a video processing model, applied to a task platform, comprising: The device receives a model request sent by a terminal device, wherein the model request includes at least one of the following: a scene identifier of the target scene, scene input data of the target scene, and model specification parameters. Based on the model request, a corresponding target video processing model is determined from at least one video processing model, wherein the at least one video processing model is trained based on the training method of the video processing model as described in any one of claims 11-12.

14. The method according to claim 13, wherein the model request includes a scene identifier of the target scene; and determining the corresponding target video processing model from at least one video processing model based on the model request includes: Based on the scene identifier of the target scene, a target video processing model suitable for the target scene is searched from the model library, wherein the model library stores at least one video processing model suitable for different video processing scenes. The model request includes scene input data for the target scene; determining the corresponding target video processing model from at least one video processing model based on the model request includes: From at least one video processing model, determine an initial video processing model adapted to the target scene; Based on the scene input data of the target scene, the initial video processing model is trained to obtain the target video processing model; The model request includes model specification parameters; determining the corresponding target video processing model from at least one video processing model based on the model request includes: Based on the model specification parameters, the corresponding target video processing model is searched from the model library, wherein the model library stores multiple video processing models with different model specification parameters.

15. The method according to any one of claims 13-14, further comprising, after determining the corresponding target video processing model from at least one video processing model based on the model request: Deploy the target video processing model, and based on the target video processing model, construct a video processing interface so that the terminal device can schedule the target video processing model to execute video tasks.

16. A task platform, comprising a request interface and a response unit; The request interface is used to receive model requests sent by the terminal device, wherein... The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters. The response unit is configured to determine a corresponding target video processing model from at least one video processing model based on the model request, wherein the at least one video processing model is trained based on the training method of the video processing model as described in any one of claims 11-12.

17. The task platform according to claim 16, further comprising a video processing interface, wherein the video processing interface is constructed based on the target video processing model; The video processing interface is used for the terminal device to schedule and execute video tasks.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 15.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

Images