Video question-answering method, device, electronic device and storage medium

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to a video question-and-answer method, device, electronic device, and storage medium for improving the accuracy of answers to video questions and answers. The method includes: obtaining a question input by an input object, and extracting a target entity in the question; based on the target entity, querying the scene knowledge graph corresponding to the scene to which the target video belongs, obtaining first search data, and based on the target entity, querying the time series knowledge graph corresponding to the target video, obtaining second search data, wherein the time series knowledge graph includes: the interaction relationship between each candidate entity in the target video in each time period; based on the first search data and the second search data, obtaining the answer to the question through a large language model LLM. In this way, LLM will consider scene knowledge and time series knowledge when answering video questions, thereby improving the accuracy of the answer.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video question-answering method, device, electronic device and storage medium. Background Art

[0002] Video question answering refers to generating natural language answers to given questions by understanding relevant video content. Video question answering has been widely used in many scenarios, such as multimedia information retrieval scenarios, intelligent assistants, and commercial service scenarios.

[0003] In the prior art, a contrastive language-image pre-training (CLIP) model is usually used for video question answering.

[0004] However, the CLIP model only supports questions and answers about the content of a single static screen in a video. The referenced information is too one-sided, that is, the available information is limited, resulting in poor accuracy of the answers obtained. Summary of the invention

[0005] The embodiments of the present application provide a video question-and-answer method, device, electronic device, and storage medium to improve the accuracy of answers.

[0006] The specific technical solutions provided by the embodiments of this application are as follows:

[0007] In a first aspect, a video question-answering method is provided, the method comprising:

[0008] Get the question input by the input object and extract the target entity in the question;

[0009] Based on the target entity, query the scene knowledge graph corresponding to the scene to which the target video belongs to obtain the first search data, and based on the target entity, query the time series knowledge graph corresponding to the target video to obtain the second search data, wherein the time series knowledge graph includes: the interaction relationship between each candidate entity in the target video in each time period;

[0010] Based on the first search data and the second search data, an answer to the question is obtained through a large language model LLM.

[0011] In a possible embodiment, based on the target entity, querying the scene knowledge graph corresponding to the scene to which the target video belongs to obtain the first search data, and based on the target entity, querying the time series knowledge graph corresponding to the target video to obtain the second search data, before obtaining the second search data, the method further includes:

[0012] Extracting direct relationships between first candidate entities in a target document and setting weights for the direct relationships, wherein the target document contains text related to the scene to which the target video belongs;

[0013] Extracting contextual proximity relationships between first candidate entities in the target document and setting weights for the contextual proximity relationships;

[0014] The direct relationship and context proximity relationship between the first candidate entities are merged to obtain the scene knowledge graph.

[0015] Through the above method, scenario-related documents are edited into a scenario knowledge graph, so that LLM will consider scenario knowledge when answering questions, solve the problem that LLM answers irrelevant questions, and improve the accuracy of answers.

[0016] In a possible embodiment, based on the target entity, querying the scene knowledge graph corresponding to the scene to which the target video belongs to obtain the first search data, and based on the target entity, querying the time series knowledge graph corresponding to the target video to obtain the second search data, before obtaining the second search data, the method further includes:

[0017] Editing the screen information of the target video into time text information, where the time text information includes: the time of occurrence of each event, and information of the second candidate entity corresponding to the time of occurrence of each event;

[0018] According to the occurrence time of each event, the time text information is converted into a time series graph to obtain a time series knowledge graph.

[0019] Through the above method, the picture information of the target video is stored as a temporal knowledge graph according to the event time, which can understand longer videos, obtain more information in the video, and find related video clips faster.

[0020] In a possible embodiment, editing the picture information of the target video into time text information includes:

[0021] Performing target detection on the target video to obtain attribute information of each second candidate entity;

[0022] Performing target tracking on each second candidate entity in the target video to obtain tracking information of each second candidate entity;

[0023] Based on the attribute information and tracking information, the temporal text information of the target video is generated.

[0024] In a possible embodiment, based on the first search data and the second search data, obtaining an answer to the question through a large language model LLM includes:

[0025] Through LLM, target search data that meets the relevance condition with the question is screened out from the first search data and the second search data;

[0026] The target search data is used as part of the prompt words of the LLM, and the prompt words are input into the LLM to obtain the answer to the question.

[0027] By providing target search data to LLM through the above method, the accuracy of video answers can be improved.

[0028] In a second aspect, a video question-answering device is provided, the device comprising:

[0029] The acquisition module is used to obtain the question input by the input object and extract the target entity in the question;

[0030] A query module is used to query the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain first search data, and query the time series knowledge graph corresponding to the target video based on the target entity to obtain second search data, wherein the time series knowledge graph includes: the interaction relationship between each candidate entity in the target video in each time period;

[0031] The processing module is used to obtain an answer to the question through a large language model LLM based on the first search data and the second search data.

[0032] In a possible embodiment, before querying the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data, and querying the temporal knowledge graph corresponding to the target video based on the target entity to obtain the second search data, the device further includes a first generating module, and the first generating module is used to:

[0033] Extracting direct relationships between first candidate entities in a target document and setting weights for the direct relationships, wherein the target document contains text related to the scene to which the target video belongs;

[0034] Extracting contextual proximity relationships between first candidate entities in the target document and setting weights for the contextual proximity relationships;

[0035] The direct relationship and context proximity relationship between the first candidate entities are merged to obtain the scene knowledge graph.

[0036] In a possible embodiment, before querying the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data, and querying the temporal knowledge graph corresponding to the target video based on the target entity to obtain the second search data, the device further includes a second generation module, and the second generation module is used to:

[0037] Editing the screen information of the target video into time text information, where the time text information includes: the time of occurrence of each event, and information of the second candidate entity corresponding to the time of occurrence of each event;

[0038] According to the occurrence time of each event, the time text information is converted into a time series graph to obtain a time series knowledge graph.

[0039] In a possible embodiment, when the picture information of the target video is edited into time text information, the second generating module is further used to:

[0040] Performing target detection on the target video to obtain attribute information of each second candidate entity;

[0041] Performing target tracking on each second candidate entity in the target video to obtain tracking information of each second candidate entity;

[0042] Based on the attribute information and tracking information, the temporal text information of the target video is generated.

[0043] In a possible embodiment, when obtaining an answer to a question through a large language model LLM based on the first search data and the second search data, the processing module is further configured to:

[0044] Through LLM, target search data that meets the relevance condition with the question is screened out from the first search data and the second search data;

[0045] The target search data is used as part of the prompt words of the LLM, and the prompt words are input into the LLM to obtain the answer to the question.

[0046] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods described in the first aspect above.

[0048] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect.

[0049] In the embodiment of the present application, the question input by the input object is obtained, and the target entity in the question is extracted. Then, based on the target entity, the scene knowledge graph corresponding to the scene to which the target video belongs is queried to obtain the first search data, and based on the target entity, the time series knowledge graph corresponding to the target video is queried to obtain the second search data. Finally, based on the first search data and the second search data, the answer to the question is obtained through the large language model LLM. In this way, based on LLM, combined with the scene knowledge graph and the time series knowledge graph, LLM will consider scene knowledge and time series knowledge when answering video questions, reduce the situation of hallucinations and nonsense when answering, and at the same time give LLM the ability to understand ultra-long time series videos, thereby improving the accuracy of video answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of possible application scenarios in the embodiments of the present application;

[0051] Figure 2 This is a flowchart of an implementation of a video question-and-answer method provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the construction of a scene knowledge graph in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of time text information in an embodiment of the present application;

[0054] Figure 5 is a schematic diagram of a timing diagram in an embodiment of the present application;

[0055] Figure 6 This is a schematic diagram of the structure of the video question-answering device in the embodiment of the present application;

[0056] Figure 7 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0058] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0059] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0060] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0061] The following is a brief introduction to the design concept of the embodiment of the present application:

[0062] Traditional video understanding methods (such as optical flow method and hidden Markov model based on time series, etc.) can achieve tasks such as target tracking and motion state analysis.

[0063] With the development of deep learning, most tasks now use deep learning models to understand and analyze videos, such as Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and three-dimensional convolutional neural networks (3D CNN).

[0064] Large Language Models (LLMs) have developed rapidly recently. The emergence of large language models pre-trained on large datasets has introduced a new contextual learning capability. This enables them to handle a variety of tasks with prompts without fine-tuning. The Chat Generative Pre-trained Transformer (ChatGPT) is the first breakthrough application built on this foundation, which includes the ability to generate code and call other model tools or APIs. Many studies are exploring how to use large language models like ChatGPT to call the application programming interface (API) of visual models to solve difficult problems in the field of computer vision, including Visual-ChatGPT. The emergence of instruction-tuning has further enhanced the ability of these models to effectively respond to user requests and perform specific tasks. Large language models that integrate video understanding capabilities provide more complex multimodal understanding advantages, enabling them to process and interpret complex interactions between visual and textual data. As influential as they are in the field of Natural Language Processing (NLP), these models are more general task solvers that excel at handling a wider range of tasks by leveraging their large knowledge base and contextual understanding gained from large amounts of multimodal data. This enables them to not only understand visual content, but also reason about it in a way that is closer to human understanding. Many works are also exploring the use of large language models, namely Vid-LLMs, in video understanding tasks.

[0065] Video question answering refers to generating natural language answers to given questions by understanding relevant video content. Video question answering has been widely used in many scenarios, such as multimedia information retrieval scenarios, intelligent assistants, and commercial service scenarios.

[0066] In the prior art, a contrastive language-image pre-training (CLIP) model is usually used for video question answering, or LLM is used for video question answering.

[0067] However, the CLIP model only supports questions and answers about the content of a single static screen in a video. The reference information is too one-sided, that is, the available information is limited, resulting in poor accuracy of the answers. At the same time, when LLM is used alone, due to the uniqueness of the usage scenario, LLM may not answer the question, resulting in poor accuracy of the answers.

[0068] In view of this, in an embodiment of the present application, a video question-answering method, device, electronic device and storage medium are provided to obtain the question input by the input object, and extract the target entity in the question, and then query the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data, and query the time series knowledge graph corresponding to the target video based on the target entity to obtain the second search data, wherein the time series knowledge graph includes: the interaction relationship between each candidate entity in the target video in each time period, and finally, based on the first search data and the second search data, the answer to the question is obtained through the large language model LLM. In this way, based on LLM, combined with the scene knowledge graph and the time series knowledge graph, LLM will consider the scene knowledge and time series knowledge when answering video questions, can identify scene-specific entities, reduce the situation of hallucinations and nonsense when answering, and at the same time give LLM the ability to understand ultra-long time series videos, thereby improving the accuracy of video answers.

[0069] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0070] See also Figure 1 , which is a schematic diagram of a possible application scenario in an embodiment of the present application. The application scenario diagram includes a server 110 and a terminal device 120 (including a terminal device 1201, a terminal device 1202 ... a terminal device 120n).

[0071] The server 110 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 120 and the server 110 may be directly or indirectly connected via wired or wireless communications, which is not limited in this application.

[0072] The terminal device 120 includes but is not limited to mobile phones, tablet computers, laptops, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; various software can be installed on the terminal device, such as applications and applets.

[0073] It should be noted that Figure 1The figure is only an example. In fact, the number of terminal devices 120 and servers 110 is not limited and is not specifically limited in the embodiments of the present application.

[0074] It should be noted that the video question-answering method in the embodiment of the present application can be deployed in a computing device, which can be a server or a terminal device, wherein the server can be Figure 1 The server 110 shown in FIG. 1 may also be Figure 1 Other servers other than the server 110 shown in FIG. 1 ; the terminal device may be Figure 1 The terminal device 120 shown in FIG. 1 may also be Figure 1 Other terminal devices other than the terminal device 120 shown in FIG. That is, the method can be executed by a server or a terminal device alone, or by a server and a terminal device together.

[0075] The following describes the video question-and-answer method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0076] See also Figure 2 As shown, it is an implementation flow chart of a video question-answering method provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0077] Step 20: Get the question input by the input object and extract the target entity in the question.

[0078] In an embodiment of the present application, an input interface is displayed on a terminal device. When an input object is input in the question and answer area of ​​the input interface, a video question and answer request is triggered. The server responds to the video question and answer request, obtains the question input by the input object, extracts information from the question, and obtains the target entity corresponding to the question.

[0079] For example, when the input object wants to do a video Q&A, it only needs to enter the question text in the input box of the Q&A interface of the terminal device. For example, if the input object wants to ask "Who entered the yard this morning", it can click the input box and enter: Who entered the yard this morning. Then click the confirmation button to trigger the video Q&A request, and then extract information about "Who entered the yard this morning" to obtain the target entity corresponding to the question as "person".

[0080] In addition, the input object can also click the voice recognition button to enter the question.

[0081] In this way, by presenting an input interface for the input object to input questions, it provides convenience for the input object and improves the user experience. In addition, voice input is also possible, which saves the input object's time in inputting text and improves the efficiency of intelligent question answering.

[0082] In the embodiment of the present application, a scene knowledge graph and a time series knowledge graph are constructed in advance.

[0083] Optionally, in the embodiment of the present application, a possible embodiment is provided for constructing a scene knowledge graph, and the following operations are specifically performed:

[0084] Step A1: extract the direct relationships between the first candidate entities in the target document and set weights for the direct relationships.

[0085] The target document contains text related to the scene to which the target video belongs.

[0086] For example, if the scene of the target video is a corridor, then the target document is a document related to the corridor.

[0087] In an embodiment of the present application, the target document is divided into text blocks, and the following operations are performed for each text block: first candidate entities contained in a text block and direct relationships between first candidate entities are extracted, and weights are set for the direct relationships.

[0088] Among them, the weight represents the importance of the corresponding relationship.

[0089] For example, see Figure 3 As shown, it is a schematic diagram of the construction of the scene knowledge graph in the embodiment of the present application, which splits the target document into text blocks, extracts the direct relationship between the first candidate entities contained in each text block, and sets weights for the direct relationships, wherein the direct relationship between the camera and the corridor is: the camera is deployed in the corridor, with a weight of 2, and the direct relationship between the corridor and the light is: the corridor is equipped with lights, with a weight of 1.

[0090] Step A2: extracting the contextual proximity relationship between the first candidate entities in the target document, and setting weights for the contextual proximity relationship.

[0091] In the embodiment of the present application, a contextual proximity relationship is added between each first candidate entity, and a weight of the contextual proximity relationship is set.

[0092] For example, Figure 3 As shown, the context proximity relationship between each first candidate entity is extracted, where the context proximity relationship between the camera and the corridor is: there are multiple cameras installed in the corridor, with a weight of 4, and the context proximity relationship between the corridor and the light is: there are chandeliers and spotlights installed in the corridor, with a weight of 2.

[0093] Step A3: Merge the direct relationship and context proximity relationship between the first candidate entities to obtain a scene knowledge graph.

[0094] In an embodiment of the present application, the direct relationship and the contextual proximity relationship between the first candidate entities are merged, and the weight of the direct relationship and the weight of the contextual proximity relationship are merged to obtain a scene knowledge graph.

[0095] Among them, the weight of the direct relationship and the weight of the context proximity relationship can be added to obtain a combined weight, which is not limited in the embodiments of the present application.

[0096] For example, Figure 3 As shown, after merging the direct relationship and context proximity relationship between the first candidate entities, the relationship between the camera and the corridor is: the camera is deployed in the corridor, and there are multiple cameras installed in the corridor, with a weight of 7; the relationship between the corridor and the light is: the corridor is installed with lights, and there are chandeliers and spotlights installed in the corridor, with a weight of 3.

[0097] In addition, it is worth noting that in the embodiments of the present application, the method or details of constructing the scenario knowledge graph can be adjusted according to different scenarios, and the comparison in the embodiments of the present application is not limited.

[0098] In this way, scenario-related documents are edited into a knowledge graph and put into the database. When the LLM is asked questions, the relevant scenario knowledge is matched so that the LLM will consider the scenario knowledge when answering the questions, thus solving the problem that the LLM does not answer the question and improving the accuracy of the answer.

[0099] Optionally, in the embodiment of the present application, a possible embodiment is provided for constructing a time series knowledge graph, and the following operations are specifically performed:

[0100] Step B1: Edit the picture information of the target video into time text information.

[0101] The time text information includes: the time of occurrence of each event, and the information of the second candidate entity corresponding to the time of occurrence of each event, and the information of the second candidate entity includes attribute information and tracking information.

[0102] In the embodiment of the present application, the picture information of each video frame of the target video is edited into time text information.

[0103] For example, see Figure 4As shown, it is a schematic diagram of the time text information in the embodiment of the present application, wherein the information of the second candidate entity corresponding to the event occurrence time 12:10:05 on September 1, 2026 includes: entity type: person, clothing type: short sleeve, clothing color: white, pants type: shorts, pants color: black, state: stationary, position: [(10, 15), (40, 60)], and entity id: 20. The information of the second candidate entity corresponding to the event occurrence time 12:13:25 on September 1, 2026 includes: entity type: car, car color: red, car type is type 1, state: moving, position: [(20, 14), (30, 90)], and entity id: 123.

[0104] Optionally, in the embodiment of the present application, a possible embodiment is provided for obtaining time text information, and the following operations are specifically performed:

[0105] Step B10: Perform target detection on the target video to obtain attribute information of each second candidate entity.

[0106] In the embodiment of the present application, a target detection algorithm is used to perform target detection on the target video to obtain attribute information of each second candidate entity.

[0107] Among them, the target detection algorithm can be a YOLO algorithm, and the attribute information of each second candidate entity includes but is not limited to: basic attribute information such as entity type, color, entity id, etc., which is not limited in the embodiments of the present application.

[0108] Step B11: performing target tracking on each second candidate entity in the target video to obtain tracking information of each second candidate entity.

[0109] In the embodiment of the present application, a target tracking algorithm is used to track the target video and obtain tracking information of each second candidate entity.

[0110] The tracking information of each second candidate entity includes but is not limited to: status and location, etc.

[0111] Step B12: Generate temporal text information of the target video based on the attribute information and tracking information.

[0112] Step B2: According to the occurrence time of each event, the time text information is converted into a time series graph to obtain a time series knowledge graph.

[0113] Among them, the timing diagram includes: the interaction relationship between each second candidate entity in each time period, and the interaction relationship is: stationary, far away, approaching, entering or leaving, etc., which is not limited in the embodiments of the present application.

[0114] In the embodiment of the present application, the time text information is recorded in the form of a graph according to the time when each event occurred.

[0115] For example, see Figure 5 As shown, it is a schematic diagram of the timing diagram in the embodiment of the present application, wherein, during the time period from 12:10:05 on September 1, 2026 to 12:18:08 on September 1, 2026, entity No. 20 (person) is relatively stationary with respect to entity No. 123 (car), and during the time period from 12:20:05 on September 1, 2026 to 12:21:08 on September 1, 2026, entity No. 123 (car) is away from entity No. 20 (person), and entity No. 20 (person) enters entity No. 6 (building).

[0116] In this way, the picture information of the target video is stored as a knowledge graph according to the event time, taking into account the timing information of the video. Compared with other video understanding methods, it can understand longer videos and obtain more information in the video. When performing video queries, relevant video clips can also be found faster.

[0117] Step 21: Based on the target entity, query the scene knowledge graph corresponding to the scene to which the target video belongs to obtain the first search data, and based on the target entity, query the time series knowledge graph corresponding to the target video to obtain the second search data.

[0118] Among them, the temporal knowledge graph includes: the interactive relationship between each candidate entity in the target video in each time period, and the scene knowledge graph includes: the relationship between each candidate entity related to the scene to which the target video belongs.

[0119] Step 22: Based on the first search data and the second search data, obtain the answer to the question through the large language model LLM.

[0120] In the embodiment of the present application, after obtaining the first search data and the second search data corresponding to the target entity, the answer to the question is obtained through the large language model LLM based on the first search data and the second search data.

[0121] Optionally, in the embodiment of the present application, a possible embodiment is provided for obtaining the answer to the question through LLM, and the following operations are specifically performed:

[0122] Step 220: Using LLM, select target search data from the first search data and the second search data that meet the relevance condition with the question.

[0123] In the embodiment of the present application, LLM is used to screen out data whose correlation with the question is greater than a correlation threshold from the first search data and the second search data as target search data.

[0124] Step 221: Use the target search data as part of the prompt word of the LLM, and input the prompt word into the LLM to obtain the answer to the question.

[0125] In the embodiment of the present application, the target search data is used as a part of the prompt word of the LLM, and the prompt word and the question are input into the LLM to obtain the answer to the question.

[0126] For example, suppose the question is "Who entered the yard this morning?", the prompt word and the question are input into the LLM, and the answer is "Three people entered the yard this morning, namely A, B and C".

[0127] In this way, providing LLM with useful and comprehensive target search data for answering questions can improve the accuracy of video answers.

[0128] Based on the same inventive concept, a video question-answering device is also provided in the embodiment of the present application. Figure 6 As shown, it is a schematic diagram of the structure of the video question-answering device in the embodiment of the present application, which specifically includes:

[0129] The acquisition module 601 is used to acquire the question input by the input object and extract the target entity in the question;

[0130] The query module 602 is used to query the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data, and query the time series knowledge graph corresponding to the target video based on the target entity to obtain the second search data, wherein the time series knowledge graph includes: the interaction relationship between each candidate entity in the target video in each time period;

[0131] The processing module 603 is used to obtain an answer to the question through a large language model LLM based on the first search data and the second search data.

[0132] In a possible embodiment, before querying the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data, and querying the temporal knowledge graph corresponding to the target video based on the target entity to obtain the second search data, the device further includes a first generating module 604, and the first generating module 604 is used to:

[0133] Extracting direct relationships between first candidate entities in a target document and setting weights for the direct relationships, wherein the target document contains text related to the scene to which the target video belongs;

[0134] Extracting contextual proximity relationships between first candidate entities in the target document and setting weights for the contextual proximity relationships;

[0135] The direct relationship and context proximity relationship between the first candidate entities are merged to obtain the scene knowledge graph.

[0136] In a possible embodiment, before querying the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data, and querying the temporal knowledge graph corresponding to the target video based on the target entity to obtain the second search data, the device further includes a second generation module 605, and the second generation module 605 is used to:

[0137] Editing the screen information of the target video into time text information, where the time text information includes: the time of occurrence of each event, and information of the second candidate entity corresponding to the time of occurrence of each event;

[0138] According to the occurrence time of each event, the time text information is converted into a time series graph to obtain a time series knowledge graph.

[0139] In a possible embodiment, when the picture information of the target video is edited into time text information, the second generating module 605 is further used to:

[0140] Performing target detection on the target video to obtain attribute information of each second candidate entity;

[0141] Performing target tracking on each second candidate entity in the target video to obtain tracking information of each second candidate entity;

[0142] Based on the attribute information and tracking information, the temporal text information of the target video is generated.

[0143] In a possible embodiment, when obtaining an answer to a question through a large language model LLM based on the first search data and the second search data, the processing module 603 is further configured to:

[0144] Through LLM, target search data that meets the relevance condition with the question is screened out from the first search data and the second search data;

[0145] The target search data is used as part of the prompt words of the LLM, and the prompt words are input into the LLM to obtain the answer to the question.

[0146] Based on the above embodiments, see Figure 7 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0147] An embodiment of the present application provides an electronic device, which may include a processor 710 (Center Processing Unit, CPU), a memory 720, an input device 730 and an output device 740, etc. The input device 730 may include a keyboard, a mouse, a touch screen, etc., and the output device 740 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0148] The memory 720 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 710 with program instructions and data stored in the memory 720. In the embodiment of the present application, the memory 720 may be used to store the program of any video question-answering method in the embodiment of the present application.

[0149] The processor 710 calls the program instructions stored in the memory 720, and the processor 710 is used to execute any video question-and-answer method in the embodiments of the present application according to the obtained program instructions.

[0150] Based on the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video question-and-answer method in any of the above method embodiments is implemented.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video question-answering method, characterized in that: include: Obtaining a question input by an input object, and extracting a target entity in the question; Extracting direct relationships between first candidate entities in a target document and setting weights for the direct relationships, wherein the target document contains text related to a scene to which a target video belongs; Extracting contextual proximity relationships between the first candidate entities in the target document, and setting weights for the contextual proximity relationships; Merging the direct relationship and the context proximity relationship between the first candidate entities to obtain a scene knowledge graph corresponding to the scene to which the target video belongs; Based on the target entity, the scene knowledge graph is queried to obtain first search data, and based on the target entity, the temporal knowledge graph corresponding to the target video is queried to obtain second search data, wherein the temporal knowledge graph includes: the interaction relationship between each candidate entity in the target video in each time period; Based on the first search data and the second search data, an answer to the question is obtained through a large language model LLM.

2. The method according to claim 1, characterized in that Before the method of querying the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data and querying the time series knowledge graph corresponding to the target video based on the target entity to obtain the second search data, the method further includes: Editing the screen information of the target video into time text information, wherein the time text information includes: the time of occurrence of each event, and information of the second candidate entity corresponding to the time of occurrence of each event; According to the occurrence time of each event, the time text information is converted into a time series graph to obtain the time series knowledge graph.

3. The method according to claim 2, characterized in that The step of editing the picture information of the target video into time text information includes: Performing target detection on the target video to obtain attribute information of each second candidate entity; Performing target tracking on each of the second candidate entities in the target video to obtain tracking information of each of the second candidate entities; Based on the attribute information and the tracking information, temporal text information of the target video is generated.

4. The method according to claim 1, characterized in that The obtaining the answer to the question by using a large language model (LLM) based on the first search data and the second search data includes: By using the LLM, target search data that meets a relevance condition with the question is screened out from the first search data and the second search data; The target search data is used as a part of the prompt words of the LLM, and the prompt words are input into the LLM to obtain the answer to the question.

5. A video question-answering device, characterized in that: include: An acquisition module, used to acquire a question input by an input object and extract a target entity in the question; The first generation module is used to extract the direct relationship between the first candidate entities in the target document and set the weight for the direct relationship, wherein the target document contains the relevant text of the scene to which the target video belongs; Extracting contextual proximity relationships between the first candidate entities in the target document, and setting weights for the contextual proximity relationships; Merging the direct relationship and the context proximity relationship between the first candidate entities to obtain a scene knowledge graph corresponding to the scene to which the target video belongs; A query module is used to query the scene knowledge graph based on the target entity to obtain first search data, and query the time series knowledge graph corresponding to the target video based on the target entity to obtain second search data, wherein the time series knowledge graph includes: the interaction relationship between each candidate entity in the target video in each time period; A processing module is used to obtain an answer to the question through a large language model LLM based on the first search data and the second search data.

6. The device according to claim 5, characterized in that Before the step of querying the scene knowledge graph corresponding to the scene to which the target video belongs based on the target entity to obtain the first search data, and querying the temporal knowledge graph corresponding to the target video based on the target entity to obtain the second search data, the device further includes a second generating module, wherein the second generating module is used to: Editing the screen information of the target video into time text information, wherein the time text information includes: the time of occurrence of each event, and information of the second candidate entity corresponding to the time of occurrence of each event; According to the occurrence time of each event, the time text information is converted into a time series graph to obtain the time series knowledge graph.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images