Reply information generation method and device, electronic equipment, storage medium and product

Abstract

This disclosure provides a method, apparatus, electronic device, storage medium, and product for generating response information, relating to the field of artificial intelligence technology, particularly natural language processing and large language model technology. The specific implementation scheme is as follows: acquiring a target video; analyzing the target video to determine its key information and pre-storing this key information; and during the playback of the target video, responding to a user's question and generating a response based on the pre-stored key information. Based on this scheme, the time spent on video analysis can be saved, the speed of response information generation can be improved, thereby avoiding interaction delays and ensuring a smooth interactive experience.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of natural language processing and large language model technology. Specifically, this disclosure relates to a method, apparatus, electronic device, storage medium, and product for generating response information. Background Technology

[0002] In current applications of artificial intelligence technology, users can engage in interactive question-and-answer sessions while watching videos.

[0003] In existing technologies, when a user asks a question while watching a video, it takes a long time for the system to generate a response, which can easily cause interaction delays and affect the user's interactive experience. Summary of the Invention

[0004] To address at least one of the aforementioned deficiencies, this disclosure provides a method, apparatus, electronic device, storage medium, and product for generating response information.

[0005] According to a first aspect of this disclosure, a method for generating response information is provided, the method comprising: Acquire the target video; Analyze the target video to identify key information and pre-store that information; During the playback of the target video, in response to the user's questions, a reply is generated based on pre-stored key information.

[0006] According to a second aspect of this disclosure, a response information generation apparatus is provided, the apparatus comprising: The target video acquisition module is used to acquire the target video. The target video analysis module is used to analyze the target video to determine its key information and pre-store the key information. The response information generation module is used to respond to user questions during the playback of the target video by generating response information corresponding to the questions based on pre-stored key information.

[0007] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to at least one of the aforementioned processors; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to perform the response information generation method.

[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described response information generation method.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described response information generation method.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure.

[0012] Figure 1 This is a flowchart illustrating a method for generating response information provided in an embodiment of this disclosure.

[0013] Figure 2 This is a flowchart illustrating a specific implementation of the response information generation method provided in this disclosure.

[0014] Figure 3 This is a schematic diagram of the structure of a response information generation device provided in an embodiment of this disclosure.

[0015] Figure 4 This is a block diagram of an electronic device used to implement the response information generation method provided in the embodiments of this disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] In existing technologies, when a user asks a question while watching a video, the system needs to analyze the video content in real time and then generate a response based on the analysis results. The process of analyzing the video content is time-consuming, making the generation of the response information lengthy, which can easily cause interaction delays and negatively impact the user experience.

[0018] Furthermore, user questions during video viewing may contain specific time ranges, such as "What was that just now?". Accurately perceiving the time ranges contained in user questions and precisely locating the video content within those time ranges directly impacts the understanding of the user's intent, and consequently, the quality of the generated response.

[0019] The response information generation method, apparatus, electronic device, storage medium, and product provided in this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0020] Figure 1 This is a flowchart illustrating the response information generation method provided in the embodiments of this disclosure, as shown below. Figure 1 As shown, the method may include the following steps: Step S110: Obtain the target video.

[0021] Step S120: Analyze the target video to determine the key information of the target video and pre-store the key information.

[0022] Step S130: During the playback of the target video, in response to the user's question, a reply is generated based on the pre-stored key information.

[0023] As can be seen from the above process, this disclosure pre-analyzes the target video before the user initiates a question to determine and pre-store key information. Then, upon responding to the user's question, it generates a response based on the pre-stored key information. This solution saves time spent on video analysis, increases the speed of response generation, avoids interaction delays, and ensures a smooth interactive experience.

[0024] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments.

[0025] First, the above steps S110, namely "acquiring the target video", and S120, namely "analyzing the target video to determine the key information of the target video and pre-storing the key information", will be described in detail with reference to the embodiments.

[0026] The target video can be video data in the form of a video stream or a complete video file.

[0027] For example, the target video can be an online video stream transmitted over a network, or a video stream of the real-world environment captured in real time by the image acquisition device of the user's device, or a complete video file downloaded in advance.

[0028] Key information is information obtained by analyzing the video content of the target video and used to support the generation of response content. After obtaining the key information, it can be pre-stored so that it can be directly retrieved when generating response information later.

[0029] In this solution, the step of obtaining key information from the target video is a preliminary step, moving the time-consuming video analysis process forward to before the user asks a question, and allowing it to be performed in parallel with the acquisition or playback of the target video. Compared to analyzing video content after the user asks a question, this solution saves time on video analysis.

[0030] The following describes in detail step S130, namely, "in the process of playing the target video, in response to the user's question information, generate the corresponding reply information based on the pre-stored key information".

[0031] During the playback of the target video, users can interact by asking questions.

[0032] For example, the target video can be displayed on a display page, which may include a control for submitting questions. Users can submit questions in text or voice format by triggering the control.

[0033] For example, the target video is a real-time video stream of the actual environment captured by the image acquisition device of the user's device. This video stream can be displayed through a camera interface, and the user can directly submit questions via voice dialogue.

[0034] In this embodiment of the disclosure, after receiving a user's question, the response information corresponding to the question can be generated directly based on the pre-stored key information, thereby saving the time spent on video analysis and improving the speed of response information generation.

[0035] In summary, based on steps S110 to S130, this disclosure pre-analyzes the target video before the user initiates a question to determine and pre-store key information. Then, upon responding to the user's question, it generates a response based on the pre-stored key information. This solution saves time spent on video analysis, increases the speed of response generation, avoids interaction delays, and ensures a smooth interactive experience.

[0036] In one optional approach of this disclosure, response information corresponding to the question information is generated based on pre-stored key information, including: Determine the relevance of the question information to the target video; Based on the relevance between the question information and the target video, the corresponding large language model is invoked to generate the corresponding response information based on the pre-stored key information.

[0037] Among them, the relevance of the question information to the target video can be used to assess whether the user's question information directly points to or depends on the target video that is currently playing, and needs to be understood and answered in conjunction with the visual or audio content of the target video.

[0038] The question is relevant to the target video, indicating that the user's question aims to interact with the content presented in the target video. For example, a user's question, "What was that just shown?", indicates that the user is asking a question about the content presented in the target video.

[0039] The question is unrelated to the target video, meaning the user's question is completely irrelevant to the video content. For example, the question might be a general knowledge question or casual conversation, such as "What's the weather like tomorrow?".

[0040] For example, the question information can be classified into two categories based on its intent: one related to the target video and the other unrelated to the target video. This determines the relevance between the question information and the target video.

[0041] Whether the question is relevant to the target video or not, the requirements for the large language model in generating the response differ. Therefore, the appropriate large language model can be called to generate the response based on whether the question is relevant or irrelevant, thus achieving rational resource allocation. Compared to related technologies that use a single large language model to generate the response, this solution improves the quality of the response by using a matching large language model.

[0042] In one optional approach of this disclosure, determining the relevance of the query information to the target video includes: Acquire video frame images of the target video at the time of the question, where the time of the question is the time when the user initiates the question. Based on the question information and video frame images, the correlation between the question information and the target video is determined.

[0043] The question time point refers to the moment or period during which the user raises a question. Single or multiple video frames corresponding to the question time point can be obtained from the target video that is currently playing or being processed. These frames reflect the video content of the target video at the question time point, essentially representing the video content the user visually observes when asking the question.

[0044] In this solution, by combining the question information and video frame images, the correlation between the question information and video frame images can be accurately analyzed, which helps to improve the quality of the generated response information.

[0045] For example, the question information, video frame images, and video frame images at the question time point can be input into the large language model, and the large language model can be prompted to perform a first intent classification on the question information. The result of the first intent classification is whether the question information is related to the target video or not.

[0046] In one optional approach disclosed herein, a corresponding large language model is invoked based on the relevance between the query information and the target video, including: If the question is irrelevant to the target video, the text-based large language model is invoked. In response to the relevance of the query information to the target video, a multimodal large language model is invoked.

[0047] In cases where the question is unrelated to the target video, the large language model used does not need to analyze and understand the target video when generating the response. It only needs to have text understanding and generation capabilities. Therefore, the large language model can be called to generate higher quality response information by leveraging its powerful language understanding capabilities and common sense knowledge base.

[0048] When the question information is related to the target video, the large language model called needs to have the ability to understand multimodal information such as frame images, audio and text in the target video when generating response information. Therefore, a multimodal large language model can be called to use its powerful cross-modal content understanding ability to perform cross-modal joint reasoning of multimodal content, thereby generating high-quality response information that is deeply integrated with video content.

[0049] In this solution, intent-based traffic splitting is performed based on the relevance of the question information to the target video. This allows general, simple questions (such as "How's the weather today?") to be routed to the plain text large-scale model. Since the plain text large-scale model does not require visual processing, its processing speed is far superior to that of the multimodal large-scale language model, enabling extremely fast responses to these general, simple questions and improving the smoothness of user interaction. The intelligent scheduling mechanism based on intent-based traffic splitting ensures that only questions requiring visual understanding (such as "What animal is in the picture?") are routed to the complex multimodal large-scale language model. This allows the computing power of the multimodal large-scale language model to be concentrated on processing more complex multimodal tasks, achieving rational resource allocation and improving the system's processing capacity.

[0050] In one optional approach disclosed herein, the question information is related to the target video, and based on pre-stored key information, response information corresponding to the question information is generated, including: Determine the intent behind the question; Responding to the query intent of querying content objects in the target video, the query time period is determined based on the query information, and the key information corresponding to the query time period is retrieved; The multimodal large language model is invoked to generate response information corresponding to the query information based on the key information corresponding to the query time period.

[0051] In particular, when the question information is related to the target video, it is possible to further analyze whether the specific question intent is to query the content object in the target video.

[0052] For example, the question information, video frame images, and video frame images at the time of the question can be input into a large language model, and the large language model can be prompted to perform a second intent classification on the question information. The result of the second intent classification is the specific intent category stated in the question intent. When the question intent belongs to some predefined intent categories (such as content location, information query, etc.), it can be determined that the question intent is to query the content object in the target video.

[0053] Continuing with the previous example, when inputting video frame images and video frame images at the time of the question into the large language model, the large language model can be prompted to perform first intent classification and second intent classification on the question information, so as to determine the relevance of the question to the target video and the specific intent.

[0054] If the intent of the question is determined to be to find content within a target video, the query time period can be determined. The query time period is used to precisely define the time range within the target video that is most relevant to the question information.

[0055] The query information may contain information about the specific time range that the question is targeting. For example, the query information may contain indicator pronouns such as "currently", "now", or "just now". This time range (i.e., the query period) can be obtained by analyzing the query information.

[0056] For example, if the question is "What was that just now?", the preset time period (such as 1 minute) before the question time can be used as the query period.

[0057] Once the query period is determined, the key information corresponding to the query period can be accurately extracted from the pre-stored key information. Then, the multimodal large language model can be invoked to generate the response information corresponding to the query information based on the key information corresponding to the query period.

[0058] In this solution, by pre-storing key information and performing precise retrieval based on the query time period, the scope of information search is greatly narrowed, and interference from irrelevant information is eliminated. This ensures that the information input into the multimodal large language model is highly relevant, thereby effectively improving the accuracy and reliability of the generated response information.

[0059] In one optional embodiment of this disclosure, key information is pre-stored, including: The key information is pre-stored using the video time data of the target video corresponding to the key information as an index.

[0060] By using video event data as an index to store key information, it is possible to quickly retrieve the key information corresponding to the query period when it is necessary to generate response information, thereby helping to improve the speed of generating response information.

[0061] In one alternative embodiment of this disclosure, the key information includes at least one of the following: Information about entities in the target video frame; Text within the target video frame; Semantic summarization of text within the target video frame; Audio data from the target video; Ancillary information related to the video content of the target video, obtained by searching for at least one of entity information, text, semantic summary, or audio data.

[0062] Among them, key information can be directly extracted or analyzed from the video data of the target video, such as the entities, text, semantic summaries, and audio data mentioned above.

[0063] An entity is a physical object contained within the frame of a target video. It can serve as a visual element in the video frame and carry specific semantic meaning. Entities in the frame of a target video can be realized based on technologies such as object detection and image recognition.

[0064] For example, some key categories of entity objects can be specified for storage, such as people, vehicles, fire hydrants, etc.

[0065] Text in the target video frame refers to the visually presented text content within the video frame, such as subtitles, road signs, and text on drawings. Text in the target video frame can be extracted using Optical Character Recognition (OCR) technology.

[0066] Semantic summarization of text within the target video frame is a higher-level semantic condensation and generalization based on the identified text. For example, a summary of a video clip.

[0067] Audio data from the target video can be extracted from the video's audio track and is an important supplement to visual information.

[0068] Supplementary information refers to extended information actively retrieved from external knowledge sources based on extracted entity, text, or audio information, used to enhance understanding. For example, after identifying a building entity in a video, the system can retrieve the building's construction date, historical events, etc. Another example is identifying a math problem text in a video, and then searching a question bank for detailed explanations, answers, and related knowledge points.

[0069] By introducing auxiliary information, the problems of outdated knowledge and inability to obtain real-time information in large language models can be solved. This provides external knowledge for large language models, thereby improving the accuracy, timeliness, and reliability of the generated response information.

[0070] In this solution, multi-level and multi-modal key information is extracted from the target video, which together provide a solid and efficient data foundation for the rapid and high-quality generation of subsequent response information.

[0071] In one alternative embodiment of this disclosure, the target video is a video stream of a real-time captured real-world environment.

[0072] The target video is a real-time video stream of the actual environment, displayed through a camera interface, allowing users to see the captured footage in real time. Users can ask questions about objects in the environment to obtain information, thus assisting in exploring the surroundings, enriching the interaction modes, and providing a more engaging user experience.

[0073] As an example, Figure 2 This is a flowchart illustrating a specific implementation of the response information generation method provided in this disclosure.

[0074] This includes the ability to perform video content analysis on the target video to identify key information and store it. The key information storage module can include a database to associate and store the key information with the video time information of the target video.

[0075] Users can ask questions while watching a target video. The system can respond to these questions by performing intent analysis based on the question information and video frame images at the time of the question. This analysis determines whether the question is relevant to the target video and identifies the specific intent behind the question. When the new intent of the question is to query video content within the target video, the system can analyze the query time period based on the question information. It can then retrieve corresponding key information from the key information storage module based on this query time period, allowing the large language model to generate a response based on this key information.

[0076] The steps of the response generation method provided in this example can be completed by pre-built agents. For example, a video content analysis agent can be provided to analyze the video content of the target video, extract key information, and index and store the key information. An intent distribution agent can be provided to analyze the relevance between the question and the target video, and then call different types of large language models to generate response information.

[0077] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0078] According to another embodiment, a response information generation apparatus is provided. Figure 3 A schematic diagram of the structure of the response information generation apparatus according to one embodiment is shown. Figure 3 As shown, the response information generation device 300 includes: The target video acquisition module 310 is used to acquire the target video. The target video analysis module 320 is used to analyze the target video to determine the key information of the target video and pre-store the key information; The response information generation module 330 is used to generate response information corresponding to the user's question information based on pre-stored key information during the playback of the target video.

[0079] As an optional approach, when generating response information corresponding to the question information based on pre-stored key information, the response information generation module 330 is specifically used for: Determine the relevance of the question information to the target video; Based on the relevance between the question information and the target video, the corresponding large language model is invoked to generate the corresponding response information based on the pre-stored key information.

[0080] As an optional approach, the response information generation module 330, when determining the relevance of the question information to the target video, is specifically used for: Acquire video frame images of the target video at the time of the question, where the time of the question is the time when the user initiates the question. Based on the question information and video frame images, the correlation between the question information and the target video is determined.

[0081] As an optional approach, when the response information generation module 330 calls the corresponding large language model based on the relevance between the question information and the target video, it is specifically used to: call the text large language model in response to the question information being irrelevant to the target video; In response to the relevance of the query information to the target video, a multimodal large language model is invoked.

[0082] As an optional approach, the question information is related to the target video. When the response information generation module 330 generates the response information corresponding to the question information based on pre-stored key information, it is specifically used for: Determine the intent behind the question; Responding to the query intent of querying content objects in the target video, the query time period is determined based on the query information, and the key information corresponding to the query time period is retrieved; The multimodal large language model is invoked to generate response information corresponding to the query information based on the key information corresponding to the query time period.

[0083] As an optional approach, the target video analysis module 320, when pre-storing key information, is specifically used for: The key information is pre-stored using the video time data of the target video corresponding to the key information as an index.

[0084] As an optional approach, the key information includes at least one of the following: Information about entities in the target video frame; Text within the target video frame; Semantic summarization of text within the target video frame; Audio data from the target video; Ancillary information related to the video content of the target video, obtained by searching for at least one of entity information, text, semantic summary, or audio data.

[0085] As an alternative approach, the target video is a real-time captured video stream of the actual environment.

[0086] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0087] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0088] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0089] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0090] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0091] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0092] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the response information generation method described above. For example, in some embodiments, the above response information generation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the response information generation method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the response information generation method described above by any other suitable means (e.g., by means of firmware).

[0093] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0095] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0097] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0098] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0099] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0100] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating response information, comprising: Acquire the target video; Analyze the target video to determine key information of the target video, and pre-store the key information; During the playback of the target video, in response to the user's question, a reply is generated based on the pre-stored key information.

2. The method according to claim 1, wherein, The step of generating response information corresponding to the question information based on the pre-stored key information includes: Determine the relevance between the query information and the target video; Based on the relevance between the question information and the target video, the corresponding large language model is invoked to generate the response information corresponding to the question information based on the pre-stored key information.

3. The method according to claim 2, wherein, Determining the relevance between the query information and the target video includes: Acquire video frame images of the target video at the time of the question, where the time of the question is the time when the user initiates the question. Based on the question information and the video frame image, the correlation between the question information and the target video is determined.

4. The method according to claim 3, wherein, The step of invoking the corresponding large language model based on the relevance between the question information and the target video includes: In response to the fact that the question information is irrelevant to the target video, the text large language model is invoked; In response to the fact that the question information is related to the target video, the multimodal large language model is invoked.

5. The method according to claim 4, wherein the question information is related to the target video, and the step of generating response information corresponding to the question information based on the pre-stored key information includes: Determine the intent behind the question; In response to the query intent being to query the content object in the target video, the query time period is determined based on the query information, and the key information corresponding to the query time period is retrieved; The multimodal large language model is invoked to generate response information corresponding to the question information based on the key information corresponding to the query period.

6. The method according to any one of claims 1-5, wherein, The pre-storage of the key information includes: The key information is pre-stored using the video time data of the target video corresponding to the key information as an index.

7. The method according to any one of claims 1-6, wherein, The key information includes at least one of the following: Information about entities in the target video frame; The text in the target video frame; A semantic summary of the text in the target video frame; The audio data in the target video; Auxiliary information related to the video content of the target video, wherein the auxiliary information is obtained by searching based on at least one of the entity's information, the text, the semantic summary, or the audio data.

8. The method according to any one of claims 1-7, wherein, The target video is a real-time video stream of a real-world environment.

9. A response information generation device, comprising: The target video acquisition module is used to acquire the target video. The target video analysis module is used to analyze the target video to determine key information of the target video and pre-store the key information. The response information generation module is used to generate response information corresponding to the user's question information based on the pre-stored key information during the playback of the target video.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

Images