Interaction method, device and equipment for video and computer readable storage medium

Abstract

The embodiment of the invention provides an interaction method and device for a video, equipment and a computer readable storage medium. The method comprises the following steps: receiving user input, wherein the user input indicates a question to a target video; and determining reply content corresponding to the user input based on the content description information of the target video and a first group of video images in the target video, the first group of video images being obtained by performing frame extraction on a plurality of video clips of the target video. Therefore, the interaction process of the video is optimized, and the quality of the video reply content is improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to interactive methods, devices, apparatuses, and computer-readable storage media for videos. Background Art

[0002] With the rapid development of machine learning technology, machine learning models now have relatively mature capabilities for processing text and static images. However, when it comes to videos, especially long videos, machine learning models have poor processing capabilities and usually cannot meet user needs. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for interacting with a video is provided. The method may include: receiving user input, where the user input indicates a question about a target video. Based on content description information of the target video and a first group of video images in the target video, a reply content corresponding to the user input is determined, wherein the first group of video images is obtained by respectively performing frame extraction on a plurality of video segments of the target video.

[0004] In a second aspect of the present disclosure, an interactive device for a video is provided. The device may include: a user input receiving module configured to receive user input, where the user input indicates a question about a target video. A reply content determining module configured to determine reply content corresponding to the user input based on content description information of the target video and a first group of video images in the target video, wherein the first group of video images is obtained by respectively performing frame extraction on a plurality of video segments of the target video.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A flowchart for interaction with a video according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram showing a principle example of an interactive process for a video according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic structural block diagram showing an apparatus for video interaction according to some embodiments of the present disclosure; and

[0013] Figure 5 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0016] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0017] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0018] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0019] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0021] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0023] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0024] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0025] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, environment 100 may include electronic device 110 .

[0026] The electronic device 110 can obtain a target video 102. The target video 102 can be provided in different formats. In addition, the target video 102 can be a video uploaded by a user, a video stored locally on the electronic device 110, or a video obtained by the electronic device 110 from other electronic devices (such as a remote device), etc. The electronic device 110 can use the target model 115 to perform a processing task on the target video 102 to determine various visual details appearing in the video. In addition, the electronic device 110 can generate a reply content 112 corresponding to the user input 104 based on the user input 104 and the visual details obtained by processing the target video 102. Exemplarily, the user input 104 can be "Describe the content of this video", "Help me find the outfit of this heroine", "What are they eating". Correspondingly, the electronic device 110 can obtain, based on the visual details, such as "This video describes the process from the male and female protagonists to falling in love", "The heroine's outfit is campus style", "They eat hot pot", etc. In other words, the user input 104 can be any content related to the target video 102. The electronic device 110 utilizes the target model 115 and, based on the parsing result of the user input 104 , can determine the reply content 112 matching the user input by performing a processing task on the target video 102 to determine various visual details appearing in the video.

[0027] The electronic device 110, for example, can use the trained target model 115 to perform object detection tasks. The target model 115 can be a model local to the electronic device 110, or it can be a model installed in other electronic devices 110 (for example, installed in a remote device). It should be noted that the target model 115 can be a single model or can include multiple models. As will be described below, in an embodiment of the present disclosure, the target model 115 can include at least a first model for generating a description of the video content, and a second model for obtaining corresponding results based on user input, text content, and images. Multiple models can be end-to-end models. Depending on the actual scenario, the target model 115 can be a general model that integrates the relevant functions of the first model and the second model.

[0028] The electronic device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. The terminal device may be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server-side device may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0029] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0030] Nowadays, there are more and more cases where machine learning models are used to process text or images. However, if the processing object becomes a video, due to the constraints of the model structure of the machine learning model, the machine learning model can only process a limited video length (about a few seconds). For videos with longer durations, the machine learning model has to discard part of the visual information when processing, which significantly reduces the processing effect of the machine learning model.

[0031] The embodiment of the present disclosure proposes a scheme for video interaction. According to the scheme, user input is received, and the user input indicates a question about the target video. Based on the content description information of the target video and the first group of video images in the target video, the reply content corresponding to the user input is determined, wherein the first group of video images is obtained by performing frame extraction on multiple video clips of the target video. Thus, the content description information in text form and a small number of video images after frame extraction can be obtained. Since the content description information can describe the video content of the target video, and the first group of video images can provide image-level information as a supplement. Under this premise, when performing interactive tasks for the video, the number of images in the first group of video images will not be too many, and the content description information is relatively comprehensive. This also avoids the discarding of part of the visual information due to the excessive length of the video. In addition, the whole process reduces the complexity and can ensure the accuracy of the reply content during the interaction process.

[0032] The following continues to describe a video processing solution according to an embodiment of the present disclosure with reference to the accompanying drawings. Figure 2 200 is a flowchart of a process 200 for video processing according to some embodiments of the present disclosure. The process 200 may be implemented at an electronic device, such as Figure 1 For ease of discussion, the following will refer to Figure 1 The process 200 is described below.

[0033] Combination Figure 2 As shown, in box 201, the electronic device 110 receives user input, and the user input indicates a question about the target video. The user input 104 is usually some questions related to the target video, such as "Describe the content of this video", "What do they eat", etc. The user input can be in text form, voice form, gesture form, etc.

[0034] In box 202, the electronic device 110 determines reply content corresponding to the user input based on the content description information of the target video and the first group of video images in the target video, wherein the first group of video images is obtained by performing frame extraction on multiple video segments of the target video.

[0035] For the target video 102, the electronic device 110 can call the first model in the target model 115 to process the target video 102 to obtain the content description information of the target video. Exemplarily, the first model can adopt a proprietary model such as a video description generation model. The characteristics of this type of proprietary model are that the demand for computing resources is relatively low, and it is mature enough to ensure accuracy when performing a single type of task.

[0036] The content description information of the target video 102 may indicate visual details such as characters, objects, and scenes in the video. That is, the content description information of the target video 102 is a textual retelling of the visual information of the target video. Figure 3 An example embodiment of obtaining content description information is described.

[0037] The electronic device 110 extracts frames from the target video segments to obtain a first set of video images. The target video 102 may be split to obtain multiple video segments. Frame extraction may be performed on each video segment to obtain a first set of video images.

[0038] For the target video 102, the electronic device 110 can perform splitting on it to obtain multiple video clips. For each video clip, frame extraction processing can be performed separately to obtain a first group of video images. Exemplarily, the frame extraction for each video clip can be a frame extraction with a small number of frames, such as extracting 1 image or 2 images for each video clip. Taking the target video 102 as an example of splitting it into 3 video clips and extracting 1 image from each video clip, the first group of video images only contains 3 images. In other words, the image information of the target video can be represented by the first group of video images.

[0039] The determination of content description information and the extraction of frames of the first group of video images for the target video 102 may be triggered by receiving user input 104. That is, based on the user input, after the target video corresponding to the user input is determined, the determination of content description information and the extraction of frames of the first group of video images are performed on the target video.

[0040] In addition, the determination of the content description information of the target video 102 and the extraction of the first set of video images may be triggered by the receipt of the target video 102. That is, after the electronic device 110 obtains the target video 102, it can perform the determination of the content description information of the target video and the extraction of the first set of video images. Therefore, after receiving the user input 104, the content description information of the target video and the first set of video images corresponding to the user input can be directly called to determine the reply content.

[0041] Exemplarily, determining the reply content may be performed using a second model. The second model may be a multimodal model. That is, the input information of the second model is multidimensional. For example, the input information of the second model includes user input 104, content description information, and the first set of video images.

[0042] The content description information and the first group of video images can be used as features of the target video 102. The content description information can convert video data with a large amount of data into text information with a relatively small amount of data. The first group of video images can be used as key graphics of the target video 102.

[0043] Thus, the second model can parse 104 the user input to obtain a parsing result. Thus, the second model can determine the reply content 112 corresponding to the user input 104 in the content description information and / or the first set of video images based on the parsing result.

[0044] Through the above process, even if the target video with a long processing time is processed, the expression conversion of the target video with a long processing time can be completed based on the content description information and the first set of video images obtained by frame extraction. This method effectively reduces the resources required to perform target video processing. In addition, since the content description information describes the visual information of the target video in text form, the accuracy of the target video presentation can also be guaranteed. Based on this, the second model can obtain a high-quality response corresponding to the user input based on the content description information and the first set of video images.

[0045] An example embodiment of determining the content description information of the target video is described below. In some embodiments, the electronic device 110 may split the target video into multiple video segments. For a video segment in the multiple video segments, the electronic device 110 may determine the content description information of the video segment. The electronic device 110 may determine the content description information of the target video based on the content description information of the video segment.

[0046] Figure 3 A schematic diagram of a principle 300 for video processing is shown. Figure 3 As shown, the target video 102 can be equally divided into a plurality of video segments according to the duration of the target video 102. The duration of each video segment can be no longer than a specified duration. Exemplarily, the specified duration can be 30 seconds, 1 minute, etc., which are not limited herein. If the target video 102 duration is 10 minutes and the specified duration is 30 seconds, 20 video segments can be obtained by splitting.

[0047] exist Figure 3In the example shown, the target video 102 is equally divided into three video segments, and for each video segment, the process of determining content description information can be performed based on the first model 115-1. The process of determining content description information for each video segment by the first model 115-1 can be parallel, thereby improving efficiency.

[0048] After obtaining the content description information of each video clip, the description information of each video clip can be spliced ​​to obtain the content description information of the target video. When splicing, the context of the description information of different video clips can be combined. For example, when splicing, check the last sentence of the description information of each video clip and the first sentence of the description information of the next video clip to ensure that they are logically coherent. If there is any incoherence, perform corresponding adjustments or supplements. For another example, before splicing, perform duplicate information detection on the descriptions of all video clips and mark the same or similar descriptions. When it is determined that there is a repetition, select a segment with more comprehensive information or clearer expression to retain, and delete other repeated parts. For another example, before splicing, perform semantic determination on the description information of each video clip to determine potential errors or ambiguities. For errors or ambiguities, error correction can be performed based on the context, such as finding errors in the time or event sequence in the content description information of the previous and next video clips, and performing adjustments. For another example, determine the emotion and tone of the content description information of each video clip. During the splicing process, if the emotion or tone is found to be inconsistent, the corresponding adjustment can be performed based on the context.

[0049] The following describes an example embodiment of determining the content description information of the target video. In some embodiments, the electronic device 110 extracts frames from the video segment to obtain a second set of video images corresponding to the video segment. The electronic device 110 can determine the content description information of the video segment using the first model based on the second set of video images corresponding to the video segment.

[0050] By performing frame extraction on the video clips, the processing pressure of the first model 115-1 can be further reduced and resources can be released. For example, for a video clip with a duration of 1 minute and a frame rate of 24FPS, the number of images is approximately 1440. For a video clip with a duration of 1 minute and a frame rate of 60FPS, the number of images is approximately 3600. If the number of video clips is large, it will also bring processing pressure to the first model 115-1. Therefore, it can be considered to perform frame extraction on the video clips to reduce the amount of data in the video clips. For frame extraction, the same number of frame extractions can be performed on each video clip, or more frame extractions can be performed on the video clips at the front or back of the playback sequence, and fewer or no frame extraction can be performed on the video clips in the middle of the playback sequence.

[0051] Still combined Figure 3As shown, for the target video 102, it can be divided into three video segments by way of example. Frame extraction is performed on each video segment to obtain a second set of video images. Figure 3 In the example shown, the second group of video images can be divided into 3 groups. Based on the first model 115-1, the process of determining content description information is performed in parallel on the 3 groups of the second group of video images respectively, and content description information of each video segment is obtained accordingly.

[0052] In some embodiments, the number of images in the second group of video images is greater than the number of images in the first group of video images. The purpose of the second group of video images is to extract description information, so as much content as possible is required. Based on this, the number of images in the second group of video images can be relatively large. In contrast, the purpose of the images in the first group of video images is to serve as key illustrations of the target video, so the images in the first group of video images do not need to be too many, and a few but fine images are sufficient.

[0053] Correspondingly, for the images in the second group of video images, a high frame extraction frequency can be used, that is, an image is extracted every fewer frames. For the images in the group of video images, a low frame extraction frequency can be used, that is, an image is extracted every more frames.

[0054] An example embodiment of determining the content description information of the target video is described below. In some embodiments, the electronic device 110 may obtain associated description information, where the associated description information includes content description information of at least one adjacent video segment of the video segment. The electronic device 110 may also determine the content description information of the video segment using the first model based on the associated description information and the second group of video images corresponding to the video segment.

[0055] An efficient way is to use the first model 115-1 to process each video segment in parallel to obtain the content description information of each video segment. In addition, there is also a highly accurate way to obtain the content description information of each video segment.

[0056] The execution process of the high-accuracy method may be, for example, as follows: first, the second group of video images corresponding to the first video segment is processed using the first model 115-1 to obtain content description information of the first video segment. For example, the first video segment may be the first video segment in the playback sequence obtained by splitting the target video.

[0057] The content description information of the first video clip is used as the associated description information. When processing the second video clip adjacent to the first video clip, the associated description information and the second group of video images corresponding to the second video clip can be used as the input of the first model 115-1, so that the first model 115-1 determines the content description information of the second video clip. This has the beneficial effect that the content description information can be generated by combining the correlation between the previous and next video clips, so that the content description information is more coherent and accurate.

[0058] Alternatively, the first model 115-1 can be used to process the first video clip and the second video clip in parallel to obtain content description information of the two video clips respectively. Thereafter, for the first video clip, the content description information of the second video clip can be used as the associated description information. Thus, the associated description information and the second group of video images corresponding to the first video clip are used as inputs of the first model to update or optimize the content description information of the first video clip. Similarly, for the second video clip, after obtaining the content description information of the third video clip, the content description information of the third video clip and the content description information of the first video clip can be used as the associated description information at the same time. Thus, based on the associated description information and the second group of video images corresponding to the second video clip, the content description information of the second video clip can be determined using the first model 115-1.

[0059] Through the above process, when determining the content description information of a video segment, the description information of adjacent video segments may be combined, so that the determined content description information is more coherent and accurate.

[0060] An example embodiment of the process of generating the second set of video images is described below. In some embodiments, the electronic device 110 iteratively performs the following steps until a preset condition is met: an image extracted from the video clip is used as a reference image and as an image in the second set of video images. In the video clip, subsequent images of the reference image are sequentially compared with the reference image. If there is a difference between the first image and the reference image that meets the specified condition, the first image replaces the reference image. Based on the replaced reference image, the second set of video images is obtained.

[0061] For frame extraction, an efficient method is random frame extraction. For example, if 10 images need to be randomly extracted from each video clip, they can be quickly extracted by random frame extraction. In addition, there can also be a frame extraction method that makes the frame extraction result richer.

[0062] The execution process of the frame extraction method that enriches the frame extraction result can be, for example, as follows: randomly extract an image at the front position of the video clip as a reference image. Thereafter, the images after the reference image are compared with the reference image one by one for similarity to obtain a comparison result.

[0063] The similarity comparison may be the result of pixel difference comparison, histogram comparison, perceptual hash algorithm comparison, etc. If the comparison result shows that the two images are similar, the image after the reference image is ignored and the comparison continues. On the contrary, if the comparison result shows that the two images are not similar, the image after the reference image that is not similar is retained and used as a new reference image. At the same time, the new reference image is used as one of the images in the second set of video images.

[0064] The above steps are iterated until the preset condition is met. For example, the preset condition may be that the second set of video images reaches a specified number, or that all images in the video clip are compared, etc. Thus, the difference between the images in the second set of video images is large enough, so that the content description information generated based on the difference is more comprehensive.

[0065] In some embodiments, the electronic device 110 may extract frames from a second group of video images corresponding to a video segment in the plurality of video segments to obtain a first group of video images. For the first group of video images, frames may be directly extracted from each video segment to obtain a group of video images.

[0066] Alternatively, when the differences between the images in the second group of videos are large enough, it can be considered that the images in the second group of video images are all representative images to a certain extent. Therefore, extraction can also be performed from the second group of video images to obtain the first group of video images.

[0067] In some embodiments, the first group of video images includes multiple images, and the process of the electronic device 110 determining the reply content corresponding to the user input may include: determining the feature representation of the change element between the multiple images based on the feature representation of each image in the multiple images. In addition, the electronic device 110 determines the feature representation of the user input and determines the feature representation of the content description information. The electronic device 110 can also generate the reply content using the second model based on at least one of the feature representation of the user input, the feature representation of each image, the feature representation of the change element, and the feature representation of the content description information.

[0068] If the first set of video images includes multiple images, then before generating the reply content, the electronic device 110 may also determine the correlation between the multiple images and extract the feature representation of the correlation. The correlation may indicate which elements have changed between the images. It is not difficult to understand that the feature representation of the changed elements between the multiple images is determined based on the feature representation of each of the multiple images.

[0069] Exemplarily, the changing element may be a change in a person. For example, an individual changes from one person to two people; a group changes from family members to colleagues; a change in a person's expression or action, and so on. The changing element may also be a change in a physical object, such as a change in daily objects, a change in means of transportation, a change in natural objects, and so on. The changing element may also be a change in a scene, such as a change in an indoor scene, from an office to a stairwell, a change in an outdoor scene, a change between indoor and outdoor scenes, a change in environmental conditions (weather, light, season, etc.), and so on. The changing element may also be a change in time. For example, from morning to night, from 2021 to 2024, and so on.

[0070] In addition, the electronic device 110 can also determine the feature representation of the user input and the feature representation of the content description information respectively. Finally, at least one of the feature representation of the user input, the feature representation of each image, the feature representation of the change element and the feature representation of the content description information is used as the input of the second model 115-2, and the reply content 112 generated by the second model 115-2 can be obtained.

[0071] In some embodiments, the first model and the second model are the same machine learning model. For example, the first model and the second model can be the same general model, that is, the first model and the second model can be the same target model. In this way, after training, the model can be more adaptable to the video processing scenario.

[0072] Figure 4 A schematic structural block diagram of an apparatus 400 for device control according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented in or included in the electronic device 110, for example. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0073] As shown in the figure, the apparatus 400 includes a user input receiving module 401, which is configured to receive user input, where the user input indicates a question about a target video. A reply content determining module 402, which is configured to determine reply content corresponding to the user input based on content description information of the target video and a first group of video images in the target video, wherein the first group of video images is obtained by respectively extracting frames from a plurality of video segments of the target video.

[0074] In some embodiments of the present application, the apparatus 400 further includes a content description information determination module, which may specifically include: a video segmentation submodule, which is configured to segment the target video into multiple video segments; a video segment parsing submodule, which is configured to determine content description information of the video segment for a video segment in the multiple video segments; and a content description information determination execution submodule, which is configured to determine content description information of the target video based on the content description information of the video segment.

[0075] In some embodiments of the present application, the video segment parsing submodule may include: a frame extraction unit configured to extract frames from the video segment to obtain a second set of video images corresponding to the video segment; and a content description information determination unit configured to determine content description information of the video segment using a first model based on the second set of video images corresponding to the video segment.

[0076] In some embodiments of the present application, the number of images in the second set of video images is greater than the number of images in the first set of video images.

[0077] In some embodiments of the present application, the content description information determination execution submodule may include: an associated description information acquisition unit configured to acquire associated description information, wherein the associated description information includes content description information of at least one adjacent video segment of the video segment. The content description information determination unit is further configured to determine the content description information of the video segment using the first model based on the associated description information and the second group of video images corresponding to the video segment.

[0078] In some embodiments of the present application, the frame extraction unit is specifically configured to: iteratively perform the following steps until a preset condition is met: use an image extracted from a video clip as a reference image and as an image in a second group of video images; in the video clip, compare subsequent images of the reference image with the reference image in sequence; if there is a difference between the first image and the reference image that meets a specified condition, replace the reference image with the first image; and obtain a second group of video images based on the replaced reference image.

[0079] In some embodiments of the present application, the apparatus 400 further includes a frame extraction module. The frame extraction module is configured to obtain a first group of video images by performing frame extraction on a second group of video images corresponding to a video segment in the plurality of video segments.

[0080] In some embodiments of the present application, the first group of video images includes multiple images, and the reply content determination module 402 includes: a feature representation determination submodule, which is configured to determine the feature representation of the changing elements between the multiple images based on the feature representation of each image in the multiple images; determine the feature representation of the user input; determine the feature representation of the content description information; and a reply content determination submodule, which is configured to generate reply content using the second model based on at least one of the feature representation of the user input, the feature representation of each image, the feature representation of the changing elements and the feature representation of the content description information.

[0081] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown may include or be implemented as Figure 1 electronic device 110, or Figure 4 Device 400.

[0082] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0083] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 500.

[0084] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0085] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0086] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0087] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0088] According to an exemplary implementation of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figures 2 to 3 The methods provided in the various optional methods are therefore not described in detail here.

[0089] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0090] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0091] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0092] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0093] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. An interactive method for a video, comprising: receiving a user input indicating a question about a target video; as well as Based on the content description information of the target video and a first group of video images in the target video, a reply content corresponding to the user input is determined, wherein the first group of video images is obtained by respectively performing frame extraction on a plurality of video segments of the target video.

2. The method according to claim 1, wherein the content description information of the target video is determined by: Splitting the target video into the multiple video segments; For a video segment among the multiple video segments, determining content description information of the video segment; and Based on the content description information of the video clip, content description information of the target video is determined.

3. The method according to claim 2, wherein determining the content description information of the video segment comprises: By performing frame extraction on the video segment, a second group of video images corresponding to the video segment is obtained; as well as Based on a second group of video images corresponding to the video clip, content description information of the video clip is determined using a first model. The method of claim 3 , wherein the number of images in the second set of video images is greater than the number of images in the first set of video images.

5. The method according to claim 3, wherein the determining the content description information of the video clip by using a first model based on the second group of video images corresponding to the video clip comprises: Acquire associated description information, where the associated description information includes content description information of at least one adjacent video segment of the video segment; as well as Based on the associated description information and a second group of video images corresponding to the video clip, content description information of the video clip is determined using the first model.

6. The method according to claim 5, wherein performing frame extraction processing on the video segment to obtain a second set of video images comprises: Iteratively perform the following steps until the preset conditions are met: using an image extracted from the video clip as a reference image and as an image in the second set of video images; In the video segment, sequentially comparing subsequent images of the reference image with the reference image; If there is a difference between the first image and the reference image that satisfies a specified condition, replacing the reference image with the first image; as well as Based on the replaced reference image, the second group of video images is obtained.

7. The method according to claim 3, wherein the first set of video images is determined by: The first group of video images is obtained by performing frame extraction on a second group of video images corresponding to video segments among the multiple video segments.

8. The method according to claim 1, wherein the first set of video images includes a plurality of images, and the determining of the reply content corresponding to the user input comprises: determining, based on the feature representation of each of the plurality of images, a feature representation of a variation element between the plurality of images; Determining a feature representation of the user input; Determining a characteristic representation of the content description information; and The reply content is generated using a second model based on at least one of the feature representation of the user input, the feature representation of each image, the feature representation of the changing element, and the feature representation of the content description information.

9. An interactive device for video, comprising: A user input receiving module, configured to receive user input, wherein the user input indicates a question about the target video; as well as The reply content determination module is configured to determine the reply content corresponding to the user input based on the content description information of the target video and the first group of video images in the target video, wherein the first group of video images is obtained by performing frame extraction on multiple video clips of the target video respectively.

10. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 8.

Images