Task processing method and apparatus, device, storage medium, and program product

Abstract

Provided in the present disclosure are a task processing method and apparatus, a device, a storage medium, and a program product. The method comprises: receiving a user input of a target user, wherein the user input indicates a task processing requirement related to visual information and audio information of a target video; acquiring multi-modal information for the target video, wherein the multi-modal information comprises video frames of the target video and further comprises one of the following: audio corresponding to the target video, and text in each video frame of the target video; at least on the basis of the multi-modal information, using a machine learning model to generate from the target video a reply to the user input; and providing the reply to the target user. The present disclosure is conducive to improving the interactivity and accuracy of task processing.

Description

Method, apparatus, device, storage medium and program product for task processing TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for task processing. BACKGROUND

[0002] With the rapid development of computer technology, more and more applications, platforms or systems currently provide processing services for multimedia information, which brings great convenience to the general public. Multimedia information can include various types of content such as video, image, image set, text, audio, etc. The processing service for video can also be referred to as video processing service. The application, platform or system with video processing service can use video processing technology to process video for video editing, video understanding, video compression, etc.

[0003] SUMMARY

[0004] In a first aspect of the present disclosure, a method for task processing is provided. The method comprises: receiving a user input of a target user, the user input indicating a task processing requirement related to visual information and audio information of a target video; obtaining multi-modal information for the target video, the multi-modal information comprising video frames of the target video and further comprising one of: audio corresponding to the target video, text in each video frame of the target video; generating, by a machine learning model, a reply to the user input from the target video based at least on the multi-modal information; and providing the reply to the target user.

[0005] In a second aspect of the present disclosure, an apparatus for task processing is provided. The apparatus comprises: a slice input receiving module configured to receive a user input of a target user, the user input indicating a task processing requirement related to visual information and audio information of a target video; an information obtaining module configured to obtain multi-modal information for the target video, the multi-modal information comprising video frames of the target video and further comprising one of: audio corresponding to the target video, text in each video frame of the target video; a reply generating module configured to generate, by a machine learning model, a reply to the user input from the target video based at least on the multi-modal information; and a reply providing module configured to provide the reply to the target user.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer readable storage medium is provided. The medium has stored thereon a computer program which, when executed by a processor, implements the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided. The product includes a computer program, wherein the computer program, when executed by a processor, implements the method according to the first aspect of the present disclosure.

[0009] It should be understood that all statements herein made regarding the examples described in this section are intended to be illustrative of the key and important features of the embodiments of the present disclosure and it is not intended that the scope of the present disclosure be limited to such features. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects and advantages of embodiments of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings similar or common elements of the drawings are denoted by like reference numbers, in which:

[0011] FIG. 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG. 2 shows a schematic diagram of an example architecture for task processing according to some embodiments of the present disclosure;

[0013] FIG. 3 shows an example of performing a video processing task using a machine learning model according to some embodiments of the present disclosure;

[0014] FIG. 4 shows a flowchart of a method of task processing according to some embodiments of the present disclosure;

[0015] FIG. 5 shows an exemplary block diagram of an apparatus for task processing according to some embodiments of the present disclosure; and

[0016] FIG. 6 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather should be construed to encompass all alternatives falling within the scope of the present disclosure. It is to be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and should not be construed as limiting the scope of the present disclosure.

[0018] In the description of embodiments of the disclosure, the term "comprising" and similar terms are to be interpreted as open-ended, i.e., "including but not limited to". The term "based on" is to be interpreted as "based, at least in part, on". The term "one embodiment" or "the embodiment" is to be interpreted as "at least one embodiment". The term "some embodiments" is to be interpreted as "at least some embodiments". Other explicit and implicit definitions can also be included below.

[0019] In this document, unless explicitly stated, performing a step "in response to A" does not mean performing the step immediately after A, but can include one or more intermediate steps.

[0020] It can be understood that the data involved in the technical solutions of the present application (including but not limited to the data itself, obtaining, using, storing or deleting) should comply with the requirements of relevant laws and regulations and relevant provisions.

[0021] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type of information involved in the present disclosure, the scope of use, the use scenario, etc. should be informed to the relevant user and the authorization of the relevant user should be obtained by appropriate means according to relevant laws and regulations, wherein the relevant user can include any type of right subject, such as individual, enterprise, group.

[0022] For example, in response to receiving the active request of the user, the prompt information is sent to the relevant user to explicitly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can voluntarily choose whether to provide information to the software or hardware such as electronic device, application program, server or storage medium, etc. performing the operation of the technical solutions of the present disclosure according to the prompt information.

[0023] As an optional but non-limiting implementation manner, in response to receiving the active request of the relevant user, the prompt information is sent to the relevant user, for example, in the form of a pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide information to the electronic device.

[0024] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0025] As used herein, the term “model” can learn the relationship between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes input and provides a corresponding output by using multiple layers of processing units. Neural network model is one example of a model based on deep learning. In this document, “model” can also be referred to as “machine learning model”, “learning model”, “machine learning network” or “learning network”, which are used interchangeably herein.

[0026] “Neural network” is a machine learning network based on deep learning. Neural network is capable of processing input and providing a corresponding output, which generally includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications generally include many hidden layers, thereby increasing the depth of the network. The layers of the neural network are connected in sequence, so that the output of the previous layer is provided as the input of the next layer, where the input layer receives the input of the neural network, and the output of the output layer is the final output of the neural network. Each layer of the neural network includes one or more nodes (also known as processing nodes or neurons), each of which processes the input from the previous layer.

[0027] Generally, machine learning can include three stages, namely training stage, testing stage and application stage (also known as inference stage). In the training stage, a given model can be trained using a large amount of training data, constantly iterating and updating the parameter values until the model can obtain consistent inference from the training data that meets the expected target. Through training, the model can be considered to learn the relationship between input and output (also known as input to output mapping) from the training data. The parameter values of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, so as to determine the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values obtained by training to determine the corresponding output.

[0028] FIG. 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In the example 100, a user 140 can initiate a user request or provide a user input to a video processing system 110, which can instruct the video processing system 110 to perform a video processing task on a target video 120 to determine a processing result 150 corresponding to the target video 120. The user request or the user input can be any suitable type of request or input, such as a voice type, a text type, a gesture type, etc. For example, the video processing system 110 can provide an interactive interface for receiving the user request, which includes an input box in the interactive interface. The video processing system 110 can receive the target video 120 and the user input provided by the user 140 via the input box.

[0029] For example, the user 140 can upload the target video 120 to be processed and the user input “briefly describe the target video” to the video processing system 110. Based on the user input, the video processing system 110 can determine that the video processing task on the target video 120 is to determine description information of the target video 120. The video processing system 110 can further perform the video processing on the target video 120 to determine the description information of the target video 120.

[0030] The target video 120 can include a plurality of video clips 125 (e.g., which can include video clips 125-1, 125-2, …, 125-N, N being any suitable positive integer, for convenience of description, one or more video clips can be collectively referred to as video clips 125). The plurality of video clips 125 can each have a time length less than a predetermined time length threshold. For example, the plurality of video clips 125 can each have a time length less than 30 seconds. Alternatively or additionally, the plurality of video clips 125 can also have the same time length. For example, the plurality of video clips 125 can each be a video clip of 25 seconds. It can be understood that each video clip 125 includes at least one video frame.

[0031] In some embodiments, the video processing system 110 can perform the video processing task on each of the plurality of video clips 125 to obtain a respective processing result of the plurality of video clips 125. The video processing system 110 can further determine the processing result 150 of performing the video processing task on the target video 120 based on the respective processing results of the plurality of video clips 125.

[0032] In some embodiments, the video processing system 110 can utilize the machine learning model 130 to perform a video processing task on the target video 120 to determine the processing result 150 corresponding to the target video 120. The machine learning model 130 can be deployed locally at the video processing system 110 or at other devices / systems (e.g., remote devices). The video processing system 110 can process the target video 120 directly by utilizing the machine learning model 130 deployed locally or by invoking the machine learning model 130 deployed at other devices / systems through a communication connection between the video processing system 110 and the other devices / systems.

[0033] The machine learning model 130 can be based on any appropriate model structure, including but not limited to a Transformer model, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), etc. In some embodiments, the machine learning model 130 can be based on a visual language model (VLM). A visual language model is capable of answering questions based on learning from a large amount of images, videos, and corpus. The visual language model is capable of generating text output based on input images, videos, and text. The machine learning model 130 can also be based on other appropriate models. Although only a single machine learning model 130 is shown in the figure, there can be multiple machine learning models for the video processing system 110 to use. If the machine learning model 130 includes multiple models, the multiple models can include models configured to perform the same or similar tasks or models configured to perform different tasks.

[0034] The video processing system 110 can run on an appropriate electronic device. The electronic device herein can include any computing system with computing capability, such as various computing devices / systems, terminal devices, server devices, etc. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a palmtop computer, a portable game terminal, a VR / AR device, a Personal Communication System (PCS) device, a personal navigation device, a Personal Digital Assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0035] The server device can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network, and big data and artificial intelligence platform.

[0036] It should be understood that the structure and function of the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0037] Conventionally, a video processing system can divide a long video into multiple video clips, and perform a video understanding task on each of the multiple video clips to obtain a respective text description information of each of the multiple video clips, for briefly summarizing the content in the video clip. The video processing system further summarizes, induces, and analyzes the respective text description information of each of the multiple video clips to determine an understanding task on the entire video, to determine an induced summary of the entire video. That is, the summary of the entire video is determined based on the respective text corresponding to each of the multiple video clips. In this case, the final processing result for the entire video can only be determined based on the intermediate text description information of each video clip, without combining information of other modalities (e.g., audio of the target video, etc.), which can cause information missing problem, affecting the accuracy of processing the target video. In addition, based on the text description information of the video clip, it is often impossible to generate other task processing results for the entire video or a single video clip, resulting in task limitation. For example, some tasks may wish to extract a segment related to a certain object from a long video. However, based on only the summary description of each video clip, such a task is difficult to achieve.

[0038] In view of this, according to an embodiment of the present disclosure, an improved scheme for task processing is provided. According to the scheme of the present embodiment, a user input of a target user is received, the user input indicating a task processing requirement related to visual information and audio information of a target video. Multi-modal information for the target video is obtained, the multi-modal information including video frames of the target video and further including one of: audio corresponding to the target video, and text in each video frame of the target video. Using a machine learning model, at least based on the multi-modal information, a reply to the user input is generated from the target video. The reply is provided to the target user.

[0039] In this way, with the machine learning model, based on the information of the target video or the video segments of the target video (e.g., including at least multimodal information), various task requirements of the user on the target video or the segments therein can be supported, including not only the question and answer requirements on the visual aspects in the video, but also the questions related to the audio and / or text of the video. This helps to improve the interactivity and accuracy of task processing.

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0041] FIG. 2 shows a schematic diagram of an example architecture 200 of task processing, according to some embodiments of the present disclosure. The architecture 200 can be implemented at the video processing system 110 of FIG. 1. For ease of discussion, the architecture 200 will be described with reference to the environment 100 of FIG. 1.

[0042] The video processing system 110 can obtain a target video 120 to be processed. In some embodiments, the target video 120 can be a video provided by a user (e.g., the user 140) to the video processing system 110 in real time, or can be a video pre-stored locally at the video processing system 110 or accessible by the video processing system 110 via other means.

[0043] The target video 120 can include or can be divided into a plurality (e.g., N) of video segments 125, and the present disclosure does not limit the length of the target video and the number of the video segments. The granularity of the target video and the video segments can be set according to actual application. For example, the target video 120 can be a TV episode, and the video segments 125 can include individual episodes in the TV episode. In another example, the target video 120 can be a single continuous video (e.g., a movie or a whole video shot by a user), and the video segments 125 can be divided from the target video 120 according to predetermined time intervals or in other manners.

[0044] In some embodiments, the plurality of video segments 125 has a certain order, e.g., the plurality of video segments 125 can be sequentially ordered according to the respective start times of the plurality of video segments 125. The video processing system 110 can process the plurality of video segments 125 sequentially based on the order of the plurality of video segments, for example.

[0045] In some embodiments, the video processing system 110 can also obtain user input. The user input can be obtained synchronously with the target video 120, or separately. The video processing system 110 can obtain the user input in any suitable manner, including but not limited to obtaining user input in text form via an input box, receiving user input in voice form via a microphone, and the like. The user input can indicate a video processing task. In some embodiments, the user input can indicate a task processing requirement related to the visual information and the audio information of the target video 120, i.e., the user input can indicate a video processing task for processing the visual information and the audio information of the target video.

[0046] The video processing task can be, for example, a tracking task for at least one object in the target video 120, an identification task for the at least one object, a content extraction task for the target video 120, an editing task for the target video 120, and the like. The content extraction task for the target video 120 can indicate, for example, extracting a specified video from the target video 120, extracting a specified image from the target video 120, extracting a specified audio from the target video 120, extracting a specified text from the target video 120, and the like. The editing task for the target video 120 can include adding special effects, filters, stickers, speed, and the like to the target video.

[0047] If the video processing task is a tracking task for the at least one object, the processing result obtained by performing the video processing task on the target video 120 can include at least the coordinates of the at least one object in each video frame of the target video 120. If the video processing task includes an identification task for the at least one object, the processing result obtained by performing the video processing task on the target video 120 can include the respective identification results of the at least one object. If the objects are characters, the identification result of each object can indicate the identity, appearance, camp, and the like of the object. If the objects are items, the identification result of each object can indicate the purpose, composition, and the like of the object.

[0048] If the video processing task is a content extraction task for the target video 120, the processing result obtained by performing the video processing task on the target video 120 can include the content extracted from the target video 120, which can be determined based on the specific content of the video processing task. For example, if the video processing task indicates extracting a specific audio from the target video 120, the processing result can include the start time and the end time of the audio in the target video, and the video processing system 110 can subsequently extract the audio from the target video 120 based on the start time and the end time. Alternatively or additionally, the processing result can also directly include the audio extracted from the target video 120.

[0049] If the video processing task is an editing task on the target video 120, the processing result of performing the video processing task on the target video 120 can include an edited target video 120. For example, if the video processing task indicates adding special effects to the target video 120, the processing result is the target video after adding special effects. It can be understood that the video processing task can also be any other appropriate task, and the present disclosure does not limit the specific task and the corresponding processing result.

[0050] In some embodiments, the video processing system 110 can directly perform the video processing task indicated by the user input on the target video 120 to determine the processing result of the video processing task on the target video 120. For example, the video processing system 110 can obtain the multi-modal information of the target video 120. The multi-modal information of the target video 120 at least includes the video frames of the target video 120. The multi-modal information of the target video 120 can also include the audio corresponding to the target video 120 and / or the text in each video frame of the target video 120.

[0051] The text in each video frame can be, for example, the subtitles, comments (barrage), and text presented in the video frame of each video frame, etc. In some embodiments, the video processing system 110 can directly obtain the subtitles of the target video and determine the subtitles as the text of each video frame. In some embodiments, the video processing system 110 can also extract the text from each video frame using an optical character recognition (OCR) algorithm, etc. In some embodiments, the video processing system 110 can also extract the text from the audio of the target video 120 using an automatic speech recognition (ASR) algorithm, etc.

[0052] The video processing system 110 can determine the processing result 150 of the video processing task on the target video 120 based at least on the multi-modal information of the target video 120, for example, using the machine learning model 130. In some embodiments, due to the limitation of the model capability, to ensure the accuracy of performing the video processing on the target video 120, the video processing system 110 can sequentially process a plurality of video segments 125 of the target video 120 to determine a plurality of processing results 245 of the plurality of video segments 125 (which can include the processing result 245-1 corresponding to the video segment 125-1, the processing result 245-2 corresponding to the video segment 125-2, …, the processing result 245-N corresponding to the video segment 125-N, for convenience of description, one or more processing results corresponding to one or more video segments can be collectively referred to as the processing result 245). The video processing system 110 can further determine the processing result 150 of the video processing task on the target video 120 based on the processing result 245 of each of the plurality of video segments 125.

[0053] In some embodiments, a video segment that is currently being processed can be referred to as a target video segment. For example, if a target video 120 includes 10 video segments, and the video processing system 110 has performed a video processing task on a first video segment, and is currently performing a video processing task on a second video segment, the second video segment can be considered as the target video segment. In some embodiments, the video processing system 110 can process at least one video segment at the same time, in which case the target video segment can include the at least one video segment.

[0054] The video processing system 110 can obtain the target video segment, and determine the multi-modal information 220 corresponding to the target video segment. The multi-modal information 220 includes at least video frames 222 of the target video segment. In some embodiments, the multi-modal information 220 can further include audio 224 corresponding to the target video segment, and / or text 226 in each video frame of the target video segment. The audio 224 can be, for example, audio of the target video segment, and the text 226 can be, for example, subtitles of the target video segment. Regarding the text 226, in some embodiments, the video processing system 110 can directly obtain subtitles of the target video segment, and determine the subtitles as the text 226. In some embodiments, the video processing system 110 can further extract the text 226 from the video frames 222 and / or the audio 224 of the target video segment using an OCR algorithm, an ASR algorithm, or the like.

[0055] The video processing system 110 can determine a model input for the machine learning model 130 based at least on the multi-modal information 220, for example. For example, the video processing system 110 can determine the model input for the machine learning model 130 based on a user input and the multi-modal information 220. In some embodiments, the video processing system 110 can further obtain reference information associated with the target video segment. The reference information can include information associated with at least one video segment of the plurality of video segments that has been processed (which can also be referred to as information of the processed video segment) 212, and / or supplemental information 232 that matches the target video 120.

[0056] In some embodiments, the video processing system 110 processes the video segment using the machine learning model 130, and the information 212 associated with at least one video segment of the plurality of video segments that has been processed includes a processing result of the at least one video segment by the machine learning model 130. The video processing system 110 can store the processing result of the at least one video segment by the machine learning model 130. For example, the video processing system 110 can store the processing result of the at least one video segment by the machine learning model 130 to a processing result library 210.

[0057] In some embodiments, the information 212 associated with the at least one processed video segment of the plurality of video segments 125 can further include at least part of the multi-modal information of the at least one video segment. Illustratively, for each of the at least one processed video segment, the video processing system 110 can further store at least part of the multi-modal information of the video segment together with the processing result of the video segment. The at least part of the multi-modal information can include, for example, the audio corresponding to the video segment. The video processing system 110 can obtain the information 212 associated with the at least one processed video segment of the plurality of video segments 125 from the processing result library 210, for example, in response to a request to perform a video processing task on a target video segment.

[0058] The supplemental information 232 can be, for example, supplemental information determined from a knowledge base (e.g., the external knowledge base 230) that matches the target video 120, and the information in the knowledge base can be pre-stored in the knowledge base via any suitable manner. The supplemental information 232 can include, for example, description information of at least one object associated with the target video 120. The at least one object can be, for example, at least one character in the target video 120, and the description information of the at least one character can include information such as the character’s name, identity, appearance, affiliation, image, actor’s name, etc. The supplemental information 232 can further include meta information (also referred to as metadata) of the target video 120, which can include information such as the name of the target video, the synopsis, the creator, etc.

[0059] The video processing system 110 can determine the model input for the machine learning model 130 based at least on the multi-modal information 220 and the reference information associated with the target video segment, which can include the information 212 associated with the at least one processed video segment of the plurality of video segments 125, and / or the supplemental information 232 that matches the target video 120. The video processing system 110 can determine the model input based on, for example, the user input, the multi-modal information 220, and the reference information. The video processing system 110 can utilize the machine learning model 130 to perform the video processing task on the target video segment by providing the model input to the machine learning model 130.

[0060] Alternatively or additionally, in some embodiments, the video processing system 110 can further obtain a prompt template, and determine the prompt input for the machine learning model 130 by filling the user input, the multi-modal information 220, and the reference information into the prompt template. The prompt input is the model input for the machine learning model 130. The video processing system 110 can utilize the machine learning model 130 to perform the video processing task on the target video segment by providing the prompt input to the machine learning model 130.

[0061] With respect to the specific way that the machine learning model 130 performs the video processing task on the target video segment, reference is made to FIG. 3, which illustrates an example 300 of performing a video processing task with the machine learning model 130, according to some embodiments of the present disclosure. The machine learning model 130 can include at least an image encoder 310 and an audio encoder 320. It is noted that the machine learning model 130 can include at least one machine learning model, and the image encoder 310, the audio encoder 320 and the machine learning model for determining the processing result can belong to the same machine learning model (i.e., the image encoder 310 and the audio encoder 320 can be part of the machine learning model), or can belong to different machine learning models.

[0062] The machine learning model 130 can encode the first information 302 of image type (e.g., can include at least the video frames 222 of the target video segment) in the model input into a visual feature representation 312 that matches the input space of the machine learning model 130 with the image encoder 310. The machine learning model 130 can encode the second information 304 of audio type (e.g., can include at least the audio 224 corresponding to the target video segment) in the model input into an audio feature representation 322 that matches the input space of the machine learning model 130 with the audio encoder 320. The machine learning model 130 can also determine a text feature representation 332 corresponding to the third information 306 of text type (e.g., can include the text 226 and the reference information) in the model input. For example, the machine learning model 130 can include a text encoder, and the machine learning model 130 can determine the text feature representation 332 corresponding to the third information 306 with the text encoder. The machine learning model 130 can determine the processing result 245 for the video processing task on the target video segment based on the visual feature representation 312, the audio feature representation 322 and the text feature representation 332.

[0063] Referring back to FIG. 2, taking the target video segment as the video segment 125-2 for example, the processing result corresponding to the target video segment is the processing result 245-2. In some embodiments, the video processing system 110 can store the processing result for the target video segment for use in processing of the next video segment of the target video segment. For example, the video processing system 110 can store the processing result 245-2 to the processing result library 210 for use in processing of the next video segment of the video segment 125-2.

[0064] The video processing system 110 can perform the video processing task on the plurality of video processing segments 125 in a similar manner to obtain a plurality of processing results 245 corresponding to the plurality of video processing segments 125. The video processing system 110 can determine the processing result 150 of the video processing task for the target video 120 based on the processing results 245 of the plurality of video segments 125. The video processing system 110 may, for example, analyze the processing results 245 of the plurality of video segments 125 to determine the processing result 150 of the video processing task for the target video 120, and the disclosure does not limit the specific manner of determining the processing result 150 of the video processing task for the target video 120 based on the processing results 245 of the plurality of video segments 125.

[0065] In some embodiments, the video processing system 110 can determine the processing result 150 of the video processing task for the target video 120 as a reply to the user input. The video processing system 110 can also provide the reply to the user. For example, the video processing system 110 can provide a reply presentation interface, and can provide the reply to the user via the reply presentation interface. For another example, the interactive interface for receiving the user input can be a conversation interface between the user and the digital assistant. The video processing system 110 can provide the reply in the conversation interface. It can be understood that the video processing system 110 can also provide the reply in any appropriate manner, and the disclosure does not limit the specific manner of providing the reply.

[0066] The video processing system 110 of the disclosure can perform task processing in combination with the multi-modal information of the target video, so that even if the user's question involves requirements other than vision, such as audio-related requirements, the user's question requirements can be accurately met, and the processing result meeting the user's expectations can be generated.

[0067] In some embodiments, the video processing task for the target video 120 is also a video processing task for the plurality of video clips 125 of the target video 120. For a target video clip, if the video processing task indicates a tracking of at least one object in the target video clip, the processing result 245 includes at least coordinates of the at least one object in each video frame of the target video clip. It can be appreciated that if the at least one object includes a plurality of objects, the processing result 245 can include respective identities of the plurality of objects and respective coordinates of the plurality of objects in each video frame of the target video clip. Thus, the video processing system 110 can determine coordinates of the at least one object in each video frame of the target video 120 based on the coordinates of the at least one object in each video frame of each video clip 125. In some embodiments, the processing result 245 can further include a corresponding time of a video frame (i.e., a video frame including the at least one object) of the at least one object in the target video. For example, the video frame is a video frame corresponding to the 30th second of the target video. The video processing system 110 may, for example, determine the coordinates of the at least one object in each video frame of the target video 120 and / or the corresponding time of the video frame including the target object as a reply to the user input.

[0068] Similarly, if the video processing task indicates an acquisition of a specific audio in the target video clip, the processing result 245 indicates at least a start time and an end time of the specific audio in the target video clip. The specific audio herein can be background music, a theme song, an ending song, a speech of at least one speaker, etc. in the target video clip. The video processing system 110 can determine at least one start time and a corresponding at least one end time of the specific audio in the target video 120 based on the plurality of processing results 245 of the plurality of video clips 125.

[0069] It can be appreciated that the specific audio in the target video can be continuous or can appear intermittently. If a start time of the specific audio in a video clip A and an end time of the specific audio in a video clip B are adjacent or coincident, and the video clip A and the video clip B are two consecutive video clips, it can be determined that the specific audio is not interrupted in the two video clips, and the video processing system 110 can determine a start time of the specific audio in the two video clips as the start time of the specific audio in the video clip A and an end time of the specific audio in the two video clips as the end time of the specific audio in the video clip B. The video processing system 110 may, for example, directly determine the at least one start time and the corresponding at least one end time of the specific audio in the target video 120 as a reply to the user input. Alternatively or additionally, the video processing system 110 may, for example, extract the specific audio from an audio of the target video 120 based on the at least one start time and the corresponding at least one end time of the specific audio in the target video 120, and determine the specific audio as a reply to the user input.

[0070] If the video processing task indicates recognition of at least one object in the target video segment, the processing result 245 includes at least a respective description of the at least one object. Illustratively, for each object, if the object is a character, the respective description of the object can be a piece of text introducing the identity, appearance, camp, gender, etc. of the character. It can be appreciated that if the at least one object includes multiple objects, the processing result 245 can include respective identities and respective descriptions of the multiple objects. In some embodiments, for different video segments 125, different descriptions of the at least one object can be determined. Alternatively or additionally, in some embodiments, for the same video segment 125, respective descriptions of multiple objects can also be determined. Thus, the video processing system 110 can determine respective descriptions of the at least one object based on the multiple video segments 125. The video processing system 110 can determine the descriptions of the at least one object as the reply to the user input, for example.

[0071] In some embodiments, the video processing task can also be a task involving multiple modalities. For example, the user input indicates a task processing requirement related to visual information and audio information of the target video 120, the video processing system 110 can extract a video from the target video 120 that satisfies the requirement and determine the video as the reply to the user input (or in other words, determine the video as a reply video). Illustratively, the user input can indicate an audio requirement for the target video (or in other words, indicate to extract a video from the target video 120 that includes specified audio, the specified audio being audio that satisfies the audio requirement). Illustratively, the user input can indicate a requirement for speech of at least one speaker appearing in the target video 120. The requirement can also indicate a scenario in which the at least one speaker speaks (e.g., soliloquy, conversation, conversation in a specific emotion, etc.). For example, the user input can indicate to extract all segments in which two specified speakers have a conversation from the target video.

[0072] The multi-modality information of the target video 120 includes audio of the target video 120 (or in other words, the multi-modality information 220 of the multiple video segments includes audio 224), which can help the machine learning model 130 to better extract a video from the target video 120 that includes specified audio. The multiple processing results 245 corresponding to the multiple video segments 125 can indicate start time and end time of the audio that satisfies the audio requirement in each video segment.

[0073] The video processing system 110 can further determine, based on the plurality of processing results 245 corresponding to the plurality of video segments 125, a start time and an end time of the audio in the target video 120 that satisfies the audio requirement. The video processing system 110 can extract, based on the start time and the end time of the audio in the target video 120 that satisfies the audio requirement, a reply video from the target video, the reply video corresponding to the audio that satisfies the audio requirement, the start time and the end time of the reply video in the target video 120 being the same as the start time and the end time of the audio in the target video 120 that satisfies the audio requirement.

[0074] Exemplarily, the user input can indicate a text requirement for the target video (i.e., indicating to extract a video including a specified text from the target video 120, the specified text being a text that satisfies the text requirement). Exemplarily, the user input can indicate a requirement for a subtitle appearing in the target video 120. The multi-modal information of the target video 120 includes text of the target video 120 (i.e., the multi-modal information 220 of each of the plurality of video segments includes text 226), which can help the machine learning model 130 better extract a video including a specified text from the target video 120. The plurality of processing results 245 corresponding to the plurality of video segments 125 can indicate a time of appearance of the text that satisfies the text requirement in each of the video segments.

[0075] The video processing system 110 can further determine, based on the plurality of processing results 245 corresponding to the plurality of video segments 125, a plurality of times of appearance of the text in the target video 120 that satisfies the text requirement. The video processing system 110 can extract, based on the plurality of times of appearance of the text in the target video 120 that satisfies the text requirement, a reply video from the target video, the reply video corresponding to the text that satisfies the text requirement.

[0076] In summary, according to embodiments of the present disclosure, by using a machine learning model, based on information (e.g., at least including multi-modal information) of a target video or a video segment thereof, various task requirements of a user on the target video or a segment thereof can be supported, not only including answering questions on visual aspects in the video, but also answering questions on audio and / or text of the video. This helps to improve the interactivity and accuracy of task processing.

[0077] FIG. 4 shows a flowchart of a method 400 of task processing, according to some embodiments of the present disclosure. The method 400 can be implemented at the video processing system 110.

[0078] At block 410, the video processing system 110 receives a user input of a target user, the user input indicating a task processing requirement related to visual information and audio information of a target video.

[0079] At block 420, the video processing system 110 obtains multi-modal information for the target video, the multi-modal information including video frames of the target video and further including one of: audio corresponding to the target video, text appearing in the video frames of the target video.

[0080] At block 430, the video processing system 110 generates, using the machine learning model, a reply to the user input from the target video based at least on the multi-modal information.

[0081] At block 440, the video processing system 110 provides the reply to the target user.

[0082] In some embodiments, the user input indicates an audio requirement for the target video, the multi-modal information further includes the audio, and wherein generating the reply to the user input from the target video includes: extracting a reply video from the target video, the reply video corresponding to audio that satisfies the audio requirement.

[0083] In some embodiments, the user input indicates a requirement for speech of at least one speaker appearing in the target video.

[0084] In some embodiments, the user input indicates a text requirement for the target video, the multi-modal information further includes the text, and wherein generating the reply to the user input from the target video includes: extracting a reply video from the target video, the reply video corresponding to text that satisfies the text requirement.

[0085] In some embodiments, the target video includes a plurality of video segments, and generating the reply to the user input from the target video using the machine learning model based at least on the multi-modal information includes: obtaining reference information associated with the target video, the reference information indicating information associated with at least one video segment of the plurality of video segments that has been processed, and / or supplemental information that matches the target video; determining a model input for the machine learning model based on the multi-modal information, the user input, and the reference information; and generating the reply to the user input from the target video using the machine learning model by providing the model input to the machine learning model.

[0086] In some embodiments, the machine learning model processes each video segment of the plurality of video segments in sequence and extracts the reply to the user input from the plurality of video segments.

[0087] In some embodiments, the information associated with the at least one video segment of the plurality of video segments that has been processed includes a processing result of the at least one video segment by the machine learning model, and wherein the supplemental information includes supplemental information determined from a knowledge base that matches the target video.

[0088] Embodiments of the present disclosure also provide corresponding apparatuses for implementing the above-mentioned methods or processes. FIG. 5 shows an exemplary structural block diagram of an apparatus 500 for task processing according to some embodiments of the present disclosure. The apparatus 500 can be implemented as or included in the video processing system 110. Various modules / components in the apparatus 500 can be implemented by hardware, software, firmware, or any combination thereof.

[0089] As shown in FIG. 5, the apparatus 500 includes an input receiving module 510 configured to receive a user input of a target user, the user input indicating a task processing requirement related to visual information and audio information of a target video. The apparatus 500 further includes an information obtaining module 520 configured to obtain multi-modal information for the target video, the multi-modal information including video frames of the target video and further including one of: audio corresponding to the target video, text in the video frames of the target video. The apparatus 500 further includes a reply generating module 530 configured to generate, by using a machine learning model, a reply to the user input from the target video based at least on the multi-modal information. The apparatus 500 further includes a reply providing module 540 configured to provide the reply to the target user.

[0090] In some embodiments, the user input indicates an audio requirement for the target video, the multi-modal information further includes the audio, and the reply generating module 530 is further configured to: extract a reply video from the target video, the audio corresponding to the reply video satisfying the audio requirement.

[0091] In some embodiments, the user input indicates a requirement for speech of at least one speaker appearing in the target video.

[0092] In some embodiments, the user input indicates a text requirement for the target video, the multi-modal information further includes the text, and the reply generating module 530 is further configured to: extract a reply video from the target video, the text corresponding to the reply video satisfying the text requirement.

[0093] In some embodiments, the target video includes a plurality of video segments, and the reply generating module 530 is further configured to: obtain reference information associated with the target video, the reference information indicating information associated with at least one processed video segment of the plurality of video segments, and / or supplementary information matching the target video; determine a model input for the machine learning model based on the multi-modal information, the user input, and the reference information; and generate, by using the machine learning model, the reply to the user input from the target video by providing the model input to the machine learning model.

[0094] In some embodiments, the machine learning model processes each video segment of the plurality of video segments in sequence and extracts the reply to the user input from the plurality of video segments.

[0095] In some embodiments, the information associated with the at least one processed video segment of the plurality of video segments includes a processing result of the at least one video segment by a machine learning model, and wherein the supplemental information includes supplemental information determined from a knowledge base that matches the target video.

[0096] The units and / or modules included in the apparatus 500 can be implemented utilizing various means including software, hardware, and / or firmware. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, e.g., machine executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units and / or modules in the apparatus 500 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, specific examples of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0097] It should be understood that one or more steps in the above methods can be performed by an appropriate electronic device or combination of electronic devices. Such an electronic device or combination of electronic devices can include, for example, the video processing system 110 in FIG. 1.

[0098] FIG. 6 illustrates a block diagram of an electronic device 600 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 600 illustrated in FIG. 6 is merely exemplary and should not be construed as limiting on the functionality and scope of the embodiments described herein. The electronic device 600 illustrated in FIG. 6 can be used to implement the video processing system 110 of FIG. 1 or the apparatus 500 of FIG. 5.

[0099] As shown in FIG. 6, the electronic device 600 is in the form of a general electronic device. Components of the electronic device 600 can include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 can be a real or virtual processor and is capable of executing various processing according to programs stored in the memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0100] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any available media that is accessible by the electronic device 600 and includes both volatile and nonvolatile media, removable and non-removable media. The memory 620 can be volatile (such as register, cache, RAM), non-volatile (such as ROM, EEPROM, flash memory), or some combination of the two. The storage device 630 can be a removable or non-removable media, and can include machine-readable media, such as flash drives, magnetic disks, or any other media that can be used to store information and / or data and that can be accessed by the electronic device 600.

[0101] The electronic device 600 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 6, a disk drive for reading from or writing to a removable, non- volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk (e.g., a CD-ROM) can be provided. In such instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 620 can include a computer program product 625 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0102] The communication unit 640 enables communications with other electronic devices over a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating with one another over a communication connection. As such, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes.

[0103] The input device 650 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 660 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through the communication unit 640, as needed, one or more devices that enable a user to interact with the electronic device 600, or any devices (e.g., a network card, a modem, etc.) that enable the electronic device 600 to communicate with one or more other electronic devices. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0104] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0105] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0106] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium. The instructions stored on the computer readable storage medium can be used to program a computer, a programmable data processing apparatus, and / or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0107] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0108] The computer program product of the present disclosure can have a signal including said computer program. This signal can be electronic, electromagnetic, optical, or any other suitable type of signal. Such a signal can be provided through a communication connection, such as electrical wiring, optical fiber, wireless interface, etc. Examples of computer program products include computer program implemented on a personal computer, server, or other networked device. A non-transitory computer readable medium, such as a floppy disk, CD-ROM, DVD-ROM, Blu-ray Disc, hard disk drive, or any other suitable non-transitory computer readable medium can store the computer program product.

[0109] Having described several implementations of the present disclosure, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure. Accordingly, the foregoing description is by way of example only and is not intended to be limiting. The implementation described herein is implementations of the present disclosure. Other implementations of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure. Therefore, this disclosure is intended to cover all such modifications and variations as fall within the scope of the implementations. It is intended that the specification and depicted embodiments are to be considered exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A method for task processing, comprising: receiving a user input of a target user, the user input indicating a task processing requirement related to visual information and audio information of a target video; obtaining multi-modal information for the target video, the multi-modal information comprising video frames of the target video and further comprising one of: audio corresponding to the target video, and text in respective video frames of the target video; generating, by using a machine learning model, a reply for the user input from the target video based at least on the multi-modal information; and providing the reply to the target user. 2.The method of claim 1, wherein the user input indicates an audio requirement for the target video, the multi-modal information further comprises the audio, and wherein generating the reply for the user input from the target video comprises: extracting a reply video from the target video, the reply video corresponding to audio satisfying the audio requirement. 3.The method of claim 2, wherein the user input indicates a requirement for speech of at least one speaker appearing in the target video. 4.The method of claim 1, wherein the user input indicates a text requirement for the target video, the multi-modal information further comprises the text, and wherein generating the reply for the user input from the target video comprises: extracting a reply video from the target video, the reply video corresponding to text satisfying the text requirement. 5.The method of claim 1, wherein the target video comprises a plurality of video segments, and wherein generating the reply for the user input from the target video by using the machine learning model based at least on the multi-modal information comprises: obtaining reference information associated with the target video, the reference information indicating information associated with at least one processed video segment of the plurality of video segments, and / or supplemental information matching the target video; determining a model input for the machine learning model based on the multi-modal information, the user input, and the reference information; and generating the reply for the user input from the target video by using the machine learning model by providing the model input to the machine learning model. 6.The method of claim 5, wherein the machine learning model processes respective video segments of the plurality of video segments in sequence and extracts the reply for the user input from the plurality of video segments. 7.The method of claim 5, wherein the information associated with at least one processed video segment of the plurality of video segments comprises a processing result of the at least one video segment by the machine learning model, and wherein the supplemental information comprises supplemental information matching the target video determined from a knowledge base. 8.An apparatus for task processing, comprising: an input receiving module configured to receive a user input of a target user, the user input indicating a task processing requirement related to visual information and audio information of a target video; ​ ​ an information obtaining module configured to obtain multi-modal information for the target video, the multi-modal information including video frames of the target video and further including one of: audio corresponding to the target video, text in respective video frames of the target video; a reply generating module configured to generate, based on at least the multi-modal information, a reply to the user input from the target video using a machine learning model; and a reply providing module configured to provide the reply to the target user.

9. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-7.

11. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1-7.

Images