Video playing interaction method and device, equipment and storage medium

Abstract

The invention relates to the technical field of intelligent interaction, in particular to a video playing interaction method and device, equipment and a storage medium, and the method comprises the steps: obtaining a screenshot of a video played in a current video playing interface in response to a real-time interaction instruction of a user; acquiring voice questioning information of a user; based on the screenshot and the voice question information, generating answer information corresponding to the voice question information; and displaying the answer information in the video playing interface. According to the method, searching is carried out through the voice question information, the video watching experience of the user is guaranteed, meanwhile, the answer information is generated based on the picture screenshot of the video and the voice question information of the user, the generated answer information is richer and more accurate through multi-mode information, and the flexibility of the interaction process is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent interaction technology, and in particular to a video playback interaction method, device, equipment and storage medium. Background Art

[0002] Video playback technology has been developing for many years, from the initial videotape and DVD playback to today's online streaming services, and has developed various major video playback platforms.

[0003] The video playback platform allows users to access and watch video content online through various devices, and integrates more interactive functions, such as commenting, liking, sharing, etc., and also includes video picture recognition function. In the prior art, users can use the picture recognition function in the video playback platform to identify elements such as characters and scenes appearing in the video.

[0004] However, since the recognition method is implemented based on pre-calibration of elements such as people and scenes, the recognition results obtained by users in the same picture are fixed and lack flexibility. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a video playback interactive method, device, equipment and storage medium to improve the flexibility of the interactive process.

[0006] In a first aspect, an embodiment of the present disclosure provides a video playback interactive method, including:

[0007] In response to a real-time interactive instruction from a user, a screenshot of the video being played in the current video playback interface is obtained;

[0008] Get the user's voice question information;

[0009] Based on the screenshot and the voice question information, generating answer information corresponding to the voice question information;

[0010] The answer information is displayed in the video playback interface.

[0011] In some embodiments, the video playback interface includes a voice recognition function entry, and the step of obtaining a screenshot of the video being played in the current video playback interface in response to a real-time interactive instruction from a user includes:

[0012] In response to a user clicking on the real-time interactive function entrance, and / or in response to a user clicking on a real-time interactive function shortcut key on an external device, displaying a real-time interactive area on the video playback interface;

[0013] Get a screenshot of the video being played in the current video playback interface;

[0014] The screen shot is displayed in the real-time interactive area.

[0015] In some embodiments, obtaining the user's voice question information includes:

[0016] Get the user's question voice;

[0017] Performing voice recognition on the question voice to obtain voice question information;

[0018] The voice question information is displayed in the real-time interactive area.

[0019] In some embodiments, generating answer information corresponding to the voice question information based on the voice question information and the screen screenshot includes:

[0020] The screen shot and the voice question information are input into a multimodal large language model to obtain answer information output by the multimodal large language model.

[0021] In some embodiments, displaying the answer information in the video playback interface includes:

[0022] When the user turns on the barrage function of the video playback interface, the answer information is displayed in the video playback interface in the form of barrage.

[0023] In some embodiments, displaying the answer information in the form of bullet screen in the video playback interface includes:

[0024] Generate an answer barrage according to the answer information;

[0025] Updating the display form of the answer barrage to distinguish the answer barrage from the regular barrage;

[0026] Display the answer barrage.

[0027] In some embodiments, displaying the answer information in the video playback interface includes:

[0028] When the user turns off the bullet screen function of the video playback interface, the answer information is displayed in the real-time interactive area of ​​the video playback interface.

[0029] In a second aspect, an embodiment of the present disclosure provides a video playback interactive device, including:

[0030] A first acquisition module, for obtaining a screenshot of the video being played in the current video playback interface in response to a real-time interactive instruction from a user;

[0031] The second acquisition module is used to obtain the user's voice question information;

[0032] A generating module, used for generating answer information corresponding to the voice question information based on the screen screenshot and the voice question information;

[0033] A display module is used to display the answer information in the video playback interface.

[0034] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0035] Memory;

[0036] Processor; and

[0037] Computer programs;

[0038] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0039] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0040] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the video playback interactive method as described above is implemented.

[0041] The video playback interactive method, device, equipment and storage medium provided by the embodiments of the present disclosure perform searches through voice question information to ensure the user's video viewing experience, and simultaneously generate answer information based on video screenshots and the user's voice question information, and utilize multimodal information to make the generated answer information richer and more accurate, thereby improving the flexibility of the interactive process. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flow chart of a video playback interactive method provided by an embodiment of the present disclosure;

[0045] Figure 2 A schematic diagram of a video playback interface provided by an embodiment of the present disclosure;

[0046] Figure 3 A flow chart of a video playback interactive method provided by another embodiment of the present disclosure;

[0047] Figure 4 A schematic diagram of a system architecture provided for another embodiment of the present disclosure;

[0048] Figure 5 A schematic diagram of the structure of a video playback interactive device provided in an embodiment of the present disclosure;

[0049] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0052] The disclosed embodiment provides a video playback interactive method, which is introduced below in conjunction with a specific embodiment.

[0053] Figure 1 This is a flow chart of the video playback interactive method provided in the embodiment of the present disclosure. The method can be applied to terminals that can access the video playback platform, such as smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, etc. It is understandable that the video playback interactive method provided in the embodiment of the present disclosure can also be applied in other scenarios.

[0054] Below Figure 1 The video playback interactive method shown in the figure is introduced, and the specific steps of the method are as follows:

[0055] S101. In response to a real-time interactive instruction from a user, a screenshot of a video being played in a current video playback interface is obtained.

[0056] During the video playback process on the video playback interface, when the user wants to search for questions related to the currently playing video, the real-time interaction command is triggered through the real-time interaction function entrance.

[0057] When the terminal receives the real-time interactive instruction from the user, it captures a screenshot of the video in the current video playback interface.

[0058] S102: Acquire user's voice question information.

[0059] The terminal supports voice collection function, which is used to obtain the user's voice question information.

[0060] Specifically, when the user triggers the real-time interactive command, he can ask the question he wants to ask; while the terminal obtains a screenshot of the video being played in the current video playback interface, it collects the user's voice through the voice collection function to obtain the user's voice question information.

[0061] Optionally, the voice capture function can be implemented based on the navigator.mediaDevices.getUserMedia API.

[0062] S103: Generate answer information corresponding to the voice question information based on the screen screenshot and the voice question information.

[0063] When a user triggers a real-time interactive command, the question he wants to ask is usually related to the currently playing video. Therefore, when generating the answer information corresponding to the voice question information, it is necessary to combine the content in the screenshot with the user's voice question information.

[0064] Optionally, the screen shot and the voice question information are input into a multimodal large language model to obtain answer information output by the multimodal large language model. The multimodal large language model can process and understand a variety of different types of data inputs, such as text, images, audio and video, to establish connections between different modalities.

[0065] In this step, prompt words are generated according to the screenshots and the voice question information, and the prompt words are aggregated so that the multimodal large language surface model can better process the screenshots and the voice question information.

[0066] Furthermore, a network retrieval-augmented generation (RAG) service is introduced into the multimodal large language model to retrieve information from external memory and use this information to enhance the generation capability of the multimodal large language model, so that the multimodal large language model can provide more accurate, richer and more relevant answer information.

[0067] S104: Display the answer information in the video playback interface.

[0068] After receiving the answer information, the answer information will be displayed in the video playback interface to provide feedback on the user's voice question information.

[0069] The disclosed embodiment obtains a screenshot of the video being played in the current video playback interface in response to a real-time interactive command from a user; obtains voice question information from the user; generates answer information corresponding to the voice question information based on the screenshot and the voice question information; displays the answer information in the video playback interface, searches through the voice question information to ensure the user's video viewing experience, and simultaneously generates answer information based on the screenshot of the video and the voice question information of the user, and utilizes multimodal information to make the generated answer information richer and more accurate, thereby improving the flexibility of the interactive process.

[0070] Based on the above embodiment, the video playback interface includes a real-time interactive function entrance, and the response to the user's real-time interactive command to obtain a screenshot of the video being played in the current video playback interface includes: responding to the user's click operation on the real-time interactive function entrance, and / or responding to the user's click operation on the real-time interactive function shortcut key on an external device, displaying a real-time interactive area on the video playback interface; obtaining a screenshot of the video being played in the current video playback interface; and displaying the screenshot in the real-time interactive area.

[0071] A real-time interactive function entrance is set in the video playback interface, and / or a real-time interactive function shortcut key is set on the external device connected to the terminal. When a user clicks on the real-time interactive function entrance in the video playback interface, or clicks on the real-time interactive function shortcut key on the external device, it is deemed that the user has initiated a real-time interactive instruction.

[0072] Correspondingly, in response to the user's real-time interaction instruction, the terminal not only obtains a screenshot of the video being played in the current video playback interface and obtains the user's voice question information, but also displays a real-time interaction area on the video playback interface.

[0073] The real-time interactive area is used for the terminal to interact with the user. For example, a screenshot of the video played in the current video playback interface is displayed in the real-time interactive area so that the user can clearly understand the screenshot obtained by the terminal.

[0074] The real-time interactive area may also display user voice question information. A possible way to obtain the voice question information is: obtain the user's question voice; perform voice recognition on the question voice to obtain voice question information; and display the voice question information in the real-time interactive area.

[0075] After the user initiates a real-time interactive command, the question voice obtained by the terminal through the voice collection function is audio.

[0076] Optionally, the question voice and the screen screenshot are directly input into the multimodal large language model to obtain the answer information output by the multimodal large language model.

[0077] Alternatively, the user's voice question is converted into voice question information in text form.

[0078] One possible implementation method is to upload the question voice to the server and convert it into text-based voice question information by the FunASR speech recognition model.

[0079] Furthermore, the voice question information and the screen screenshot in text form are directly input into the multimodal large language model to obtain the answer information output by the multimodal large language model.

[0080] Another possible implementation method is to integrate the speech recognition function into the multimodal large language model, directly input the question voice and the screenshot into the multimodal large language model, and the multimodal large language model converts the question voice into voice question information in text form, and then outputs the answer information based on the text voice question information and the screenshot.

[0081] After obtaining the voice question information in text form, the voice question information is displayed in the real-time interactive area, and the user can preview it in real time to confirm whether the voice recognition result is accurate.

[0082] Optionally, prompt information of different interaction states is also displayed in the real-time interaction area. For example, after obtaining a screenshot and a user's voice question information, in the process of generating answer information corresponding to the voice question information, a first prompt information is displayed in the real-time interaction area to prompt the user that the answer information is being generated.

[0083] The disclosed embodiment displays a real-time interactive area in the video playback interface so that the user can clearly know the current interactive status and various messages during the interactive process, thereby improving the user's experience during the interactive process.

[0084] Based on the above embodiment, the display of the answer information in the video playback interface includes: when the user turns on the barrage function of the video playback interface, the answer information is displayed in the video playback interface in the form of barrage; or, when the user turns off the barrage function of the video playback interface, the answer information is displayed in the real-time interactive area of ​​the video playback interface.

[0085] Figure 2 Schematic diagram of the video playback interface provided by the embodiment of the present disclosure. Figure 2As shown, the video playback interface includes a video playback area 21 and a real-time interaction area 22. Before the user initiates a real-time interaction command, only the video playback area 21 is displayed in the video playback interface; when the user initiates a real-time interaction command, the size of the video playback area 21 is adjusted, and the video playback area 21 and the real-time interaction area 22 are displayed in the video playback interface at the same time.

[0086] The answer information generated based on the screenshot and the voice question information needs to be displayed in the video playback interface to provide feedback to the user. Different answer information display methods are provided in the embodiments of the present disclosure.

[0087] One possible implementation is that when the user turns on the bullet screen function, the answer information is displayed in the form of bullet screen in the video playback interface, as shown in the following example: Figure 2 As shown, the answer information is displayed as one or more bullet comments in the video playback area 21.

[0088] Optionally, displaying the answer information in the form of barrage in the video playback interface includes: generating an answer barrage according to the answer information; updating the display form of the answer barrage to distinguish the answer barrage from regular barrage; and displaying the answer barrage.

[0089] When the user turns on the barrage function, the regular barrage corresponding to the video is already displayed in the video playback area. When displaying the answer information, the answer barrage corresponding to the answer information needs to be distinguished from the regular barrage. In other words, the answer barrage is equivalent to a special form of barrage.

[0090] Specifically, updating the display form of the barrage includes one or more of the following: updating the font style of the barrage, updating the font color of the barrage, updating the font size of the barrage, updating the background of the barrage, updating the border of the barrage, updating the scrolling speed of the barrage, etc.

[0091] Another possible implementation is that when the user turns off the bullet screen function of the video playback interface, the answer information is displayed in the real-time interactive area of ​​the video playback interface.

[0092] As described in the above embodiment, the real-time interaction area 22 displays various types of information in the real-time interaction process, such as screenshots obtained by the terminal, voice question information, etc. After generating answer information corresponding to the voice question information, the answer information is also displayed in the real-time interaction area 22.

[0093] Optionally, when the user turns on the barrage function of the video playback interface, the answer information is displayed in the video playback interface in the form of barrage, and the answer information is displayed in the real-time interactive area of ​​the video playback interface.

[0094] The disclosed embodiment provides different display methods for displaying answer information after generating answer information corresponding to the voice question information, flexibly adapting to the situation where the user turns on or off the barrage function, thereby further improving the flexibility of the video playback interactive method.

[0095] Figure 3 A flowchart of a video playback interactive method provided by another embodiment of the present disclosure, the method is based on Figure 4 The system architecture shown is implemented. Figure 4 The system architecture shown is Figure 3 The interactive method of video playback is introduced as shown.

[0096] like Figure 4 As shown, the user accesses the video playback website through the browser, and the browser implements the video playback function, voice recording function, video screenshot function, and information display function based on various interfaces. Among them, the video playback function allows users to watch videos; the voice recording function includes responding to the user's real-time interactive instructions and obtaining the user's question voice; the video screenshot function includes responding to the user's real-time interactive instructions and obtaining a screenshot of the video being played in the current video playback interface; the information display function includes displaying the real-time interactive area, displaying the screenshot in the real-time interactive area, displaying the voice question information in the real-time interactive area, displaying the answer information in the video playback interface in the form of bullet screen, displaying the answer information in the real-time interactive area, and so on.

[0097] In the service layer, the data processing service provides data processing support for the real-time interactive process; the speech recognition service converts the acquired user question voice into text-based voice question information; the web search RAG service retrieves information from external memory and uses this information to enhance the generation capability of the multimodal large language model, thereby providing more accurate, richer and more relevant answer information; the prompt word aggregation service generates prompt words based on screenshots and voice question information, thereby enabling the multimodal large language model to provide more accurate, richer and more relevant answer information.

[0098] The basic reservoir layer provides various basic service supports. These services are deployed through mirroring, and each service exists in the basic layer in the form of a container. Among them, the data storage service is used to store the captured screenshots; the model online service supports the call of multimodal large language models; the search engine provides external information support for the search of the network search RAG service.

[0099] Finally, the answer information generated by the multimodal large language model is displayed in the form of barrage through the video playback website, and / or displayed in the real-time interactive area.

[0100] The disclosed embodiment searches through voice question information to ensure the user's video viewing experience, and at the same time generates answer information based on video screenshots and the user's voice question information. The use of multimodal information makes the generated answer information richer and more accurate, thereby improving the flexibility of the interactive process.

[0101] Figure 5 The structure diagram of the video playback interactive device provided in the embodiment of the present disclosure is shown in FIG. The video playback interactive device may be a terminal as described in the above embodiment, or the video playback interactive device may be a component or assembly in the terminal. The video playback interactive device provided in the embodiment of the present disclosure may execute the processing flow provided in the video playback interactive method embodiment, such as Figure 5 As shown, the video playback interactive device 50 includes: a first acquisition module 51, a second acquisition module 52, a generation module 53, and a display module 54; the first acquisition module 51 is used to respond to the user's real-time interactive command and obtain a screenshot of the video played in the current video playback interface; the second acquisition module 52 is used to obtain the user's voice question information; the generation module 53 is used to generate answer information corresponding to the voice question information based on the screenshot and the voice question information; the display module 54 is used to display the answer information in the video playback interface.

[0102] Optionally, the video playback interface includes a voice recognition function entrance, and the first acquisition module 51 includes: a first display unit 511, a screenshot unit 512, and a second display unit 513; the first display unit 511 is used to respond to the user's click operation on the real-time interactive function entrance, and / or respond to the user's click operation on the real-time interactive function shortcut key on an external device, to display the real-time interactive area on the video playback interface; the screenshot unit 512 is used to obtain a screenshot of the video being played in the current video playback interface; the second display unit 513 is used to display the screenshot in the real-time interactive area.

[0103] Optionally, the second acquisition module 52 includes an acquisition unit 521, a speech recognition unit 522, and a third display unit 523; the acquisition unit 521 is used to acquire the user's question voice; the speech recognition unit 522 is used to perform speech recognition on the question voice to obtain voice question information; the third display unit 523 is used to display the voice question information in the real-time interactive area.

[0104] Optionally, the generation module 53 is used to input the screen screenshot and the voice question information into a multimodal large language model to obtain answer information output by the multimodal large language model.

[0105] Optionally, the display module 54 is used to display the answer information in the form of barrage in the video playback interface when the user turns on the barrage function of the video playback interface.

[0106] Optionally, the display module 54 includes a generation unit 541, an update unit 542; a fourth display unit 543; the generation unit 541 is used to generate an answer barrage according to the answer information; the update unit 542 is used to update the display form of the answer barrage to distinguish the answer barrage from the regular barrage; the fourth display unit 543 is used to display the answer barrage.

[0107] Optionally, the display module 54 is used to display the answer information in the real-time interactive area of ​​the video playback interface when the user turns off the barrage function of the video playback interface.

[0108] Figure 5 The video playback interactive device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar and will not be repeated here.

[0109] Figure 6 The electronic device provided in the embodiment of the present disclosure may be a terminal as described in the above embodiment. The electronic device provided in the embodiment of the present disclosure may execute the processing flow provided in the embodiment of the video playback interactive method, such as Figure 6 As shown, the electronic device 60 includes: a memory 61, a processor 62, a computer program and a communication interface 63; wherein the computer program is stored in the memory 61 and is configured so that the processor 62 executes the above-mentioned video playback interactive method.

[0110] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the video playback interactive method described in the above embodiment.

[0111] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0113] In addition, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the video playback interactive method as described above is implemented.

[0114] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0115] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0116] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0117] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0119] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video playback interactive method, characterized in that: The method comprises: In response to a real-time interactive instruction from a user, a screenshot of the video being played in the current video playback interface is obtained; Get the user's voice question information; Based on the screenshot and the voice question information, generating answer information corresponding to the voice question information; The answer information is displayed in the video playback interface.

2. The method according to claim 1, characterized in that The video playback interface includes a voice recognition function entrance, and the step of obtaining a screenshot of the video played in the current video playback interface in response to the user's real-time interactive instruction includes: In response to a user clicking on the real-time interactive function entrance, and / or in response to a user clicking on a real-time interactive function shortcut key on an external device, displaying a real-time interactive area on the video playback interface; Get a screenshot of the video being played in the current video playback interface; The screen shot is displayed in the real-time interactive area.

3. The method according to claim 2, characterized in that The obtaining of the user's voice question information includes: Get the user's question voice; Performing voice recognition on the question voice to obtain voice question information; The voice question information is displayed in the real-time interactive area.

4. The method according to claim 1, characterized in that The generating answer information corresponding to the voice question information based on the voice question information and the screen screenshot includes: The screen shot and the voice question information are input into a multimodal large language model to obtain answer information output by the multimodal large language model.

5. The method according to claim 1, characterized in that The displaying of the answer information in the video playback interface includes: When the user turns on the barrage function of the video playback interface, the answer information is displayed in the video playback interface in the form of barrage.

6. The method according to claim 5, characterized in that The step of displaying the answer information in the form of bullet screen on the video playback interface includes: Generate an answer barrage according to the answer information; Updating the display form of the answer barrage to distinguish the answer barrage from the regular barrage; Display the answer barrage.

7. The method according to claim 1, characterized in that The displaying of the answer information in the video playback interface includes: When the user turns off the bullet screen function of the video playback interface, the answer information is displayed in the real-time interactive area of ​​the video playback interface.

8. A video playback interactive device, characterized in that: include: A first acquisition module, for obtaining a screenshot of the video being played in the current video playback interface in response to a real-time interactive instruction from a user; The second acquisition module is used to obtain the user's voice question information; A generating module, used for generating answer information corresponding to the voice question information based on the screen screenshot and the voice question information; A display module is used to display the answer information in the video playback interface.

9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images