Video interaction method and apparatus, device and computer readable storage medium

Abstract

Embodiments of the present disclosure provide a video interaction method and apparatus, a device and a computer readable storage medium. The method comprises: receiving a user input, wherein the user input indicates a question for a target video; and on the basis of content description information of the target video and a first set of video images in the target video, determining reply content corresponding to the user input, wherein the first set of video images are obtained by respectively executing frame extraction on a plurality of video clips of the target video. Therefore, the interaction process of the video is optimized, and the quality of the video reply content is improved.

Description

Interactive methods, apparatuses, devices, and computer-readable storage media for video Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to interactive methods, apparatuses, devices, and computer-readable storage media for video. Background Technology

[0002] With the rapid development of machine learning technology, current machine learning models have relatively mature capabilities in processing text and still images. However, when faced with videos, especially long ones, the processing capabilities of machine learning models are poor, and they often cannot meet the needs of users.

[0003] Summary of the Invention

[0004] In a first aspect of this disclosure, a method for interacting with a video is provided. The method may include: receiving user input, the user input indicating a question about a target video; and determining response content corresponding to the user input based on content description information of the target video and a first set of video images in the target video, wherein the first set of video images is obtained by performing frame extraction on multiple video segments of the target video.

[0005] In a second aspect of this disclosure, an interactive device for video is provided. The device may include: a user input receiving module configured to receive user input indicating a question about a target video; and a response content determining module configured to determine response content corresponding to the user input based on content description information of the target video and a first set of video images in the target video, wherein the first set of video images is obtained by performing frame extraction on multiple video segments of the target video.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. A computer program is stored on the medium, which, when executed by a processor, implements the method of the first aspect.

[0008] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 illustrates a flowchart of video-related interactions according to some embodiments of the present disclosure;

[0012] Figure 3 illustrates a schematic diagram of a video interaction process according to some embodiments of the present disclosure;

[0013] Figure 4 shows a schematic structural block diagram of an apparatus for video interaction according to some embodiments of the present disclosure; and

[0014] Figure 5 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0017] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0018] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and authorization should be obtained from the relevant users. Among them, relevant users may include any type of rights holder, such as individuals, enterprises, and groups.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as the electronic device, application, server, or storage medium that performs the operation of the technical solution disclosed herein based on the prompt message.

[0021] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0024] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0025] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0026] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in Figure 1, environment 100 may include electronic device 110.

[0027] Electronic device 110 can acquire target video 102. Target video 102 can be provided in different formats. Furthermore, target video 102 can be a video uploaded by a user, a video stored locally on electronic device 110, or a video acquired by electronic device 110 from other electronic devices (e.g., remote devices), etc. Electronic device 110 can use target model 115 to perform processing tasks on target video 102 to determine various visual details appearing in the video. Furthermore, electronic device 110 can generate response content 112 corresponding to user input 104, combined with the visual details obtained from processing target video 102. For example, user input 104 could be "Describe the content of this video," "Help me find the female lead's outfit," or "What are they eating?" Correspondingly, electronic device 110 can obtain information such as "This video describes the process of the male and female leads from meeting to falling in love," "The female lead's outfit is a campus style," or "They are eating hot pot," etc., based on visual details. In other words, user input 104 can be any content related to target video 102. The electronic device 110 uses the target model 115 to determine the response content 112 that matches the user input based on the parsing results of the user input 104 by performing processing tasks on the target video 102 to determine various visual details appearing in the video.

[0028] Electronic device 110 may, for example, utilize a trained target model 115 to perform an object detection task. The target model 115 may be a model native to electronic device 110 or a model installed on another electronic device 110 (e.g., installed on a remote device). It should be noted that the target model 115 may be a single model or may include multiple models. As described below, in embodiments of this disclosure, the target model 115 may include at least a first model for generating video content descriptions and a second model for obtaining corresponding results based on user input, text content, and images. The multiple models may be end-to-end models. Depending on the specific scenario, the target model 115 may be a general-purpose model integrating the relevant functions of the first and second models.

[0029] Electronic device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. Terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side equipment may include computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] There is an increasing trend of using machine learning models to process text or images. However, when it comes to processing video, the limitations of machine learning models restrict their processing to a limited length (approximately a few seconds). For longer videos, machine learning models have to discard some visual information, leading to a significant decrease in processing efficiency.

[0032] This disclosure proposes a scheme for video interaction. According to the scheme, user input is received, indicating a question about a target video. Based on the content description information of the target video and a first set of video images in the target video, a response corresponding to the user input is determined. The first set of video images is obtained by extracting frames from multiple video segments of the target video. This allows for the acquisition of text-based content description information and a small number of extracted video images. Since the content description information describes the video content of the target video, and the first set of video images provides supplementary image-level information, the first set of video images is relatively concise and comprehensive, avoiding the loss of visual information due to excessive video length. Furthermore, the entire process reduces complexity and ensures the accuracy of the response during interaction.

[0033] The following description, with reference to the accompanying drawings, outlines a scheme for video processing according to embodiments of the present disclosure. Figure 2 shows a flowchart of a process 200 for video processing according to some embodiments of the present disclosure. Process 200 can be implemented at an electronic device, such as the electronic device 110 of Figure 1. For ease of discussion, process 200 will be described below with reference to Figure 1.

[0034] Referring to Figure 2, in box 201, electronic device 110 receives user input, which indicates a question about the target video. User input 104 is typically a question related to the target video, such as "Describe the content of this video," "What are they eating," etc. User input can be in text, voice, gesture, etc.

[0035] In box 202, electronic device 110 determines the response content corresponding to the user input based on the content description information of the target video and the first set of video images in the target video, wherein the first set of video images is obtained by performing frame extraction on multiple video segments of the target video respectively.

[0036] For target video 102, electronic device 110 can call the first model in target model 115 to process target video 102 to obtain content description information of target video. For example, the first model can be a proprietary model such as a video description generation model. The characteristic of this type of proprietary model is that it has relatively low requirements for computing resources and is mature enough to ensure accuracy when performing a single type of task.

[0037] The content description information of the target video 102 can indicate visual details such as people, objects, and scenes in the video. That is, the content description information of the target video 102 is a textual summary of the visual information of the target video. An example embodiment for obtaining content description information will be described below with reference to Figure 3.

[0038] Electronic device 110 obtains a first set of video images by performing frame extraction on multiple video segments of the target video. For target video 102, electronic device 110 can split it into multiple video segments. For each video segment, frame extraction can be performed separately to obtain the first set of video images.

[0039] For the target video 102, the electronic device 110 can split it into multiple video segments. For each video segment, frame extraction can be performed to obtain a first set of video images. For example, frame extraction for each video segment can be performed on a small number of frames, such as extracting one or two images for each video segment. Taking splitting the target video 102 into three video segments and extracting one image from each video segment as an example, the first set of video images will contain only three images. That is, the first set of video images can represent the image information of the target video.

[0040] The determination of content description information and the extraction of frames from the first group of video images for target video 102 can be triggered by receiving user input 104. That is, based on user input, after determining the target video corresponding to the user input, the determination of content description information and the extraction of frames from the first group of video images are performed on the target video.

[0041] Furthermore, the determination of the content description information of the target video 102 and the frame extraction of the first set of video images can also be triggered by receiving the target video 102. That is, after acquiring the target video 102, the electronic device 110 can perform the determination of the content description information of the target video and the frame extraction of the first set of video images. Therefore, upon receiving user input 104, the content description information of the target video and the first set of video images corresponding to the user input can be directly invoked to determine the response content.

[0042] For example, determining the response content can be performed using a second model. The second model can be a multimodal model. That is, the input information to the second model is multidimensional. For example, the input information to the second model includes user input 104, content description information, and a first set of video images.

[0043] Content description information and the first set of video images can serve as features of the target video 102. Content description information can convert large amounts of video data into relatively small amounts of text information. The first set of video images can serve as key illustrations of the target video 102.

[0044] Therefore, the second model can parse the user input 104 to obtain the parsing result. Based on the parsing result, the second model can determine the response content 112 corresponding to the user input 104 from the content description information and / or the first set of video images.

[0045] Through the above process, even for target videos with long processing times, the representation of the long target video can be completed based on the content description information and the first set of video images obtained through frame extraction. This method effectively reduces the resources required to perform target video processing. Furthermore, since the content description information describes the visual information of the target video in text form, the accuracy of the target video presentation can be guaranteed. Based on this, the second model, using the content description information and the first set of video images, can obtain a high-quality response corresponding to the user input.

[0046] The following describes an example embodiment for determining the content description information of a target video. In some embodiments, the electronic device 110 may divide the target video into multiple video segments. For each video segment among the multiple video segments, the electronic device 110 may determine the content description information of the video segment. Based on the content description information of the video segment, the electronic device 110 may determine the content description information of the target video.

[0047] Figure 3 shows a schematic diagram of the principle 300 used for video processing. Referring to Figure 3, the target video 102 can be divided into multiple video segments based on its duration. The duration of each video segment can be no more than a specified duration. For example, the specified duration can be 30 seconds, 1 minute, etc., and is not limited here. If the target video 102 has a duration of 10 minutes and the specified duration is 30 seconds, then 20 video segments can be obtained by splitting it.

[0048] In the example shown in Figure 3, the target video 102 is divided into three equal video segments. For each video segment, the process of determining content description information can be performed separately based on the first model 115-1. The process of determining content description information for each video segment by the first model 115-1 can be parallelized, thereby improving efficiency.

[0049] After obtaining the content description information for each video segment, the description information of each video segment can be spliced ​​together to obtain the content description information of the target video. During splicing, the context of the description information of different video segments can be considered. For example, during splicing, the last sentence of the description information of each video segment and the first sentence of the description information of the next video segment are checked to ensure logical coherence. If there are inconsistencies, corresponding adjustments or additions are performed. Furthermore, before splicing, duplicate information detection is performed on the descriptions of all video segments, marking identical or similar descriptions. If duplicates are found, the segment with more comprehensive information or clearer expression is retained, and other duplicate parts are deleted. Also, before splicing, semantic determination is performed on the description information of each video segment to identify potential errors or ambiguities. For errors or ambiguities, corrections can be made based on the context, such as adjusting for errors in the time or event sequence found in the content description information of consecutive video segments. Additionally, the sentiment and tone of the content description information of each video segment are determined. During the splicing process, if inconsistencies in sentiment or tone are found, corresponding adjustments can be made based on the context.

[0050] The following describes an example embodiment for determining the content description information of a target video. In some embodiments, the electronic device 110 obtains a second set of video images corresponding to a video segment by performing frame extraction on the video segment. The electronic device 110 can determine the content description information of the video segment based on the second set of video images corresponding to the video segment using a first model.

[0051] By performing frame extraction on video clips, the processing load on the first model 115-1 can be further reduced, freeing up resources. For example, a 1-minute video clip with a frame rate of 24 FPS has approximately 1440 images. A 1-minute video clip with a frame rate of 60 FPS has approximately 3600 images. If there are many video clips, it will also put processing pressure on the first model 115-1. Therefore, frame extraction can be performed on video clips to reduce the amount of data in the video clips. For frame extraction, the same number of frames can be extracted for each video clip, or more frames can be extracted for video clips at the beginning or end of the playback sequence, and fewer frames or no frames can be extracted for video clips in the middle of the playback sequence.

[0052] Referring again to Figure 3, for the target video 102, it can be exemplaryly divided into 3 video segments. Frame extraction is performed on each video segment to obtain a second set of video images. In the example shown in Figure 3, the second set of video images can be divided into 3 groups. Based on the first model 115-1, the process of determining content description information is performed in parallel on the 3 groups in the second set of video images, corresponding to the content description information of each video segment.

[0053] In some embodiments, the second set of video images contains more images than the first set. The purpose of the second set of video images is to extract descriptive information, therefore it needs to contain as much content as possible. Based on this, the second set of video images can contain a relatively large number of images. In contrast, the purpose of the images in the first set of video images is to serve as key illustrations of the target video; therefore, the first set of video images does not need to contain many images, but rather a few high-quality ones.

[0054] Correspondingly, for the images in the second group of video images, a high frame rate can be used, meaning an image is extracted every few frames. For the images in the first group of video images, a low frame rate can be used, meaning an image is extracted every few frames.

[0055] The following describes an example embodiment for determining the content description information of a target video. In some embodiments, the electronic device 110 may acquire associated description information, which includes the content description information of at least one adjacent video segment of the video segment. The electronic device 110 may also determine the content description information of the video segment using a first model based on the associated description information and a second set of video images corresponding to the video segment.

[0056] One efficient approach is to use the first model 115-1 to process each video segment in parallel, obtaining the content description information for each video segment. Alternatively, a highly accurate method can also be used to obtain the content description information for each video segment.

[0057] The execution process of a highly accurate method can be as follows: First, the first model 115-1 is used to process the second set of video images corresponding to the first video segment to obtain the content description information of the first video segment. For example, the first video segment can be the first video segment in the playback sequence obtained by splitting the target video.

[0058] The content description information of the first video segment is used as the associated description information. When processing the second video segment adjacent to the first video segment, the associated description information and the second set of video images corresponding to the second video segment can be used as input to the first model 115-1, thereby determining the content description information of the second video segment. The advantage of this is that the content description information can be generated by combining the correlation between the preceding and following video segments, making the content description information more coherent and accurate.

[0059] Alternatively, the first model 115-1 can be used to process the first video segment and the second video segment in parallel to obtain content description information for each segment. Then, for the first video segment, the content description information of the second video segment can be used as associated description information. The associated description information and the second set of video images corresponding to the first video segment are then used as input to the first model to update or optimize the content description information of the first video segment. Similarly, for the second video segment, after obtaining the content description information of the third video segment, the content description information of the third video segment and the content description information of the first video segment can be used simultaneously as associated description information. Therefore, based on the associated description information and the second set of video images corresponding to the second video segment, the first model 115-1 can be used to determine the content description information of the second video segment.

[0060] By combining the descriptive information of adjacent video segments with the descriptive information of adjacent video segments through the above process, the descriptive information of the determined content can be more coherent and accurate.

[0061] The following describes an example embodiment of the process for generating the second set of video images. In some embodiments, the electronic device 110 iteratively performs the following steps until a preset condition is met: an image extracted from a video clip is used as a reference image and is included as an image in the second set of video images. In the video clip, subsequent images of the reference image are sequentially compared with the reference image. If a difference exists between the first image and the reference image that satisfies a specified condition, the first image replaces the reference image. Based on the replaced reference image, the second set of video images is obtained.

[0062] For frame extraction, an efficient method is random frame extraction. For example, if you need to randomly extract 10 images from each video segment, you can quickly do so using random frame extraction. In addition, there are other frame extraction methods that can enrich the extraction results.

[0063] The execution process of a frame extraction method that enriches the frame extraction results can be as follows: Randomly extract an image from the beginning of a video segment as a reference image. Then, compare the similarity of each image after the reference image with the reference image to obtain the comparison results.

[0064] Similarity comparisons can be based on pixel difference comparisons, histogram comparisons, perceptual hashing algorithm comparisons, and so on. If the comparison result indicates that two images are similar, images after the reference image are ignored, and the comparison continues. Conversely, if the comparison result indicates that two images are dissimilar, the dissimilar images after the reference image are retained and used as new reference images. Simultaneously, the new reference image becomes one of the images in the second set of video images.

[0065] The above steps are repeated iteratively until a preset condition is met. For example, the preset condition could be that the second set of video images reaches a specified number, or that all images in the video clip have been compared, and so on. This ensures that the differences between the images in the second set of video images are sufficiently large, resulting in more comprehensive content description information.

[0066] In some embodiments, the electronic device 110 can obtain a first set of video images by performing frame extraction on a second set of video images corresponding to video segments from a plurality of video clips. For the first set of video images, frames can be directly extracted from each video segment to obtain a set of video images.

[0067] Alternatively, if the differences between the images in the second set of videos are sufficiently large, the images in the second set of videos can be considered to be representative. Therefore, extraction can also be performed from the second set of videos to obtain the first set of videos.

[0068] In some embodiments, the first set of video images includes multiple images, whereby the process by which the electronic device 110 determines the response content corresponding to the user input may include: determining the feature representation of the changing elements among the multiple images based on the feature representation of each image in the multiple images. Furthermore, the electronic device 110 determines the feature representation of the user input and the feature representation of the content description information. The electronic device 110 may also generate the response content using a second model based on at least one of the feature representation of the user input, the feature representation of each image, the feature representation of the changing elements, and the feature representation of the content description information.

[0069] If the first set of video images includes multiple images, the electronic device 110 can determine the correlation between the multiple images and extract feature representations of the correlation before generating the response content. The correlation indicates which elements have changed between the images. It is easy to understand that the feature representations of the changed elements between the multiple images are determined based on the feature representations of each image in the multiple images.

[0070] For example, a change in an element can be a change in people. This could be a change in an individual, from one person to two; a change in a group, from family members to colleagues; a change in a person's expression or actions, and so on. A change in an element can also be a change in physical objects, such as changes in everyday items, vehicles, or natural objects. A change in an element can also be a change in a scene, such as a change in an indoor scene, from an office to a stairwell; a change in an outdoor scene; a change in a scene between indoors and outdoors; a change in environmental conditions (weather, lighting, seasons, etc.), and so on. A change in an element can also be a change in time, such as changing from morning to night, or from 2021 to 2024, and so on.

[0071] Furthermore, the electronic device 110 can also determine the feature representation of the user input and the feature representation of the content description information, respectively. Finally, by using at least one of the feature representation of the user input, the feature representation of each image, the feature representation of the changing elements, and the feature representation of the content description information as input to the second model 115-2, the response content 112 generated by the second model 115-2 can be obtained.

[0072] In some embodiments, the first model and the second model are the same machine learning model. For example, the first model and the second model can be the same general-purpose model, that is, the first model and the second model can be the same target model. In this way, after training, it can be more adapted to video processing scenarios.

[0073] Figure 4 shows a schematic structural block diagram of a device 400 for device control according to some embodiments of the present disclosure. The device 400 may be implemented in or included in an electronic device 110, for example. The various modules / components in the device 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0074] As shown in the figure, the device 400 includes a user input receiving module 401, configured to receive user input indicating a question about the target video. A response content determination module 402 is configured to determine the response content corresponding to the user input based on the content description information of the target video and a first set of video images in the target video. The first set of video images is obtained by extracting frames from multiple video segments of the target video.

[0075] In some embodiments of this application, the apparatus 400 further includes a content description information determination module, which may specifically include: a video splitting submodule configured to split the target video into multiple video segments; a video segment parsing submodule configured to determine the content description information of video segments among the multiple video segments; and a content description information determination execution submodule configured to determine the content description information of the target video based on the content description information of the video segments.

[0076] In some embodiments of this application, the video segment parsing submodule may include: a frame extraction unit configured to extract frames from the video segment to obtain a second set of video images corresponding to the video segment; and a content description information determination unit configured to determine the content description information of the video segment based on the second set of video images corresponding to the video segment using a first model.

[0077] In some embodiments of this application, the number of images in the second group of video images is greater than the number of images in the first group of video images.

[0078] In some embodiments of this application, the content description information determination execution submodule may include: an associated description information acquisition unit, configured to acquire associated description information, which includes content description information of at least one adjacent video segment of the video segment. The content description information determination unit is further configured to determine the content description information of the video segment using a first model based on the associated description information and a second set of video images corresponding to the video segment.

[0079] In some embodiments of this application, the frame extraction unit is specifically configured to iteratively perform the following steps until a preset condition is met: an image extracted from a video segment is used as a reference image and as an image in the second set of video images; in the video segment, subsequent images of the reference image are compared with the reference image in sequence; if there is a difference between the first image and the reference image that satisfies a specified condition, the first image replaces the reference image; and based on the replaced reference image, the second set of video images is obtained.

[0080] In some embodiments of this application, the apparatus 400 further includes a frame extraction module. The frame extraction module is configured to obtain a first set of video images by performing frame extraction on a second set of video images corresponding to video segments from a plurality of video segments.

[0081] In some embodiments of this application, the first group of video images includes multiple images, and the response content determination module 402 includes: a feature representation determination submodule, which is configured to determine the feature representation of the changing elements among the multiple images based on the feature representation of each image in the multiple images; determine the feature representation of the user input; determine the feature representation of the content description information; and a response content determination submodule, which is configured to generate response content using a second model based on at least one of the feature representation of the user input, the feature representation of each image, the feature representation of the changing elements, and the feature representation of the content description information.

[0082] Figure 5 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 500 shown in Figure 5 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 500 shown in Figure 5 may include or be implemented as the electronic device 110 of Figure 1, or the device 400 of Figure 4.

[0083] As shown in Figure 5, the electronic device 500 is in the form of a general-purpose electronic device. Components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0084] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0085] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0086] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0087] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0088] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0089] According to an exemplary implementation of this disclosure, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative embodiments of Figures 2 and 3, which will not be described further herein.

[0090] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0091] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0092] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0094] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for interacting with video, comprising: Receive user input, which indicates a question about the target video; as well as Based on the content description information of the target video and the first set of video images in the target video, the response content corresponding to the user input is determined, wherein the first set of video images is obtained by performing frame extraction on multiple video segments of the target video respectively.

2. The method according to claim 1, wherein the content description information of the target video is determined by the following method: The target video is split into the multiple video segments; For each of the plurality of video segments, determine the content description information of that video segment; and Based on the content description information of the video segment, the content description information of the target video is determined.

3. The method according to claim 2, wherein determining the content description information of the video segment includes: By performing frame extraction on the video segment, a second set of video images corresponding to the video segment is obtained; as well as Based on the second set of video images corresponding to the video segment, the content description information of the video segment is determined using the first model.

4. The method according to claim 3, wherein the number of images in the second group of video images is greater than the number of images in the first group of video images.

5. The method according to claim 3, wherein determining the content description information of the video segment using the first model based on the second group of video images corresponding to the video segment includes: Obtain associated description information, which includes content description information of at least one adjacent video segment of the video segment; as well as Based on the associated description information and the second set of video images corresponding to the video segment. The first model is used to determine the content description information of the video segment.

6. The method according to claim 5, wherein performing frame extraction processing on the video segment to obtain the second set of video images includes: Iteratively execute the following steps until the preset condition is met: An image extracted from the video clip will be used as a reference image and will also be used as an image in the second set of video images; In the video segment, subsequent images of the reference image are compared sequentially with the reference image; If there is a difference between the first image and the reference image that meets the specified conditions, the first image replaces the reference image; as well as The second set of video images is obtained based on the replaced reference image.

7. The method of claim 3, wherein the first set of video images is determined by: The first set of video images is obtained by performing frame extraction on the second set of video images corresponding to the video segments in the plurality of video segments.

8. The method according to claim 1, wherein the first group of video images includes multiple images, and determining the response content corresponding to the user input includes: Based on the feature representation of each of the plurality of images, determine the feature representation of the changing elements among the plurality of images; Determine the feature representation of the user input; Determine the feature representation of the content description information; and The response content is generated using a second model based on at least one of the feature representations of the user input, the feature representations of each image, the feature representations of the changing elements, and the feature representations of the content description information.

9. An interactive device for video, comprising: The user input receiving module is configured to receive user input, which indicates a question about the target video; as well as The response content determination module is configured to determine the response corresponding to the user input based on the content description information of the target video and a first set of video images in the target video. The content is complex, wherein the first group of video images is obtained by extracting frames from multiple video segments of the target video.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

Images