Information processing program, information processing method, and information processing device

Abstract

This information processing device acquires information relating to the structure of graph data to be retrieved, and a question sentence relating to an object included in a video. The information processing device generates a search query for the graph data on the basis of the information relating to the structure of the graph data to be retrieved. On the basis of the generated search query, the information processing device searches for the graph data in which object attribute information or inter-object interaction information is associated with the object included in the video. The information processing device outputs information relating to the object as an answer to the question sentence on the basis of the result of the search for the graph data.

Description

Information Processing Program, Information Processing Method, and Information Processing Apparatus

[0001] The present invention relates to an information processing program and the like.

[0002] In recent years, AI (Artificial Intelligence) chatbot services that answer users' questions have been increasing. For example, in the prior art, there is a dialogue system that uses a large language model such as an LLM (Large Language Models) to answer questions from users.

[0003] Here, when generating an answer to a question using a large language model, a phenomenon (hallucination) may occur where content that is different from the facts or content that is irrelevant to the context is plausibly generated.

[0004] For example, in order to suppress hallucination, a technique called RAG (Retrieval Augmented Generation) is used. RAG improves the answer accuracy by combining external information search when performing text generation by a large language model.

[0005] Japanese Patent No. 7509972

[0006] However, the above-described prior art has a problem that an accurate answer cannot be generated for a question.

[0007] In one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing apparatus that can generate an accurate answer to a question.

[0008] In the first proposal, the computer performs the following processes: The computer obtains information about the structure of the graph data to be searched and a question about the objects contained in the video. Based on the information about the structure of the graph data to be searched, the computer generates a search query for the graph data. Based on the generated search query, the computer searches for graph data in which object attribute information or interaction information between objects is associated with the objects contained in the video. Based on the search results of the retrieved graph data, the computer outputs information about the objects as an answer to the question.

[0009] It can generate accurate answers to questions.

[0010] Figure 1 is a diagram illustrating the overall processing of the information processing device according to this embodiment 1. Figure 2 is a diagram illustrating an example of the data structure of the detection pattern and matching pattern. Figure 3 is a diagram illustrating an example of a knowledge graph. Figure 4 is a diagram illustrating an example of an action scene graph. Figure 5 is a functional block diagram showing the configuration of the information processing device according to this embodiment 1. Figure 6 is a flowchart illustrating the processing procedure of the information processing device according to this embodiment 1. Figure 7 is a diagram illustrating the processing of the information processing device according to this embodiment 2. Figure 8A is a diagram (1) illustrating the processing of the first generation unit. Figure 8B is a diagram (2) illustrating the processing of the first generation unit. Figure 8C is a diagram (3) illustrating the processing of the first generation unit. Figure 9A is a diagram (1) illustrating the processing of the third generation unit. Figure 9B is a diagram (2) illustrating the processing of the third generation unit. Figure 10 is a functional block diagram showing the configuration of the information processing device according to this embodiment 2. Figure 11 is a flowchart illustrating the processing procedure of the information processing device according to this embodiment 2. Figure 12 is a diagram illustrating the processing of the information processing device according to this embodiment 3. Figure 13A is a diagram (1) illustrating the processing of the explanatory text generation unit. Figure 13B is a diagram (2) illustrating the processing of the explanatory text generation unit. Figure 14 is a diagram illustrating an example of an information acquisition query. Figure 15 is a diagram illustrating the processing of a graph information acquisition tool. Figure 16 is a functional block diagram showing the configuration of the information processing device according to this embodiment 3. Figure 17 is a flowchart showing the processing procedure of the information processing device according to this embodiment 3. Figure 18 is a diagram illustrating another example of the information processing device according to this embodiment 3. Figure 19 is a diagram illustrating an example of a computer hardware configuration that realizes similar functions to the information processing device of the embodiment.

[0011] The following describes in detail, with reference to the drawings, embodiments of the information processing program, information processing method, and information processing apparatus disclosed in this application. However, this invention is not limited to these embodiments.

[0012] Figure 1 is a diagram illustrating the overall processing of the information processing device according to this embodiment 1. For example, the information processing device 100 of this embodiment 1 is a device that outputs an answer to a question sentence 11 related to a video 10 when it receives such a question sentence 11 from a user U1. The video 10 is a time-series frame (still image).

[0013] The information processing device 100 performs KG generation processing, ASG generation processing, and graph analysis processing. For example, the KG generation processing and ASG generation processing are performed in advance. The graph analysis processing is performed to generate an answer when a question 11 is received from the user. In the following description, the KG generation processing, ASG generation processing, and graph analysis processing will be described in order.

[0014] The KG generation process performed by the information processing device 100 will now be described. The KG generation process is the process of generating a Knowledge Graph 50 that shows the conditions for detecting a certain event in the video 10. For example, the Knowledge Graph 50 is a graph corresponding to detection patterns and matching patterns.

[0015] For example, the information processing device 100 acquires text 12 related to the domain of the detected object included in the video 10. The text 12 is such as "dangerous behavior with accident risk". The information processing device 100 generates a list of detected objects from the text 12 using LLM (Large Language Models) or the like. The list of detected objects is such as "approaching a moving forklift without wearing a vest", "carrying a load for a long time", and "entering the road without checking left and right".

[0016] The information processing device 100 generates multiple candidate detection patterns and matching patterns by setting a list of detection targets as a prompt for generating detection patterns and matching patterns and inputting it into the LLM.

[0017] Figure 2 shows an example of the data structure for detection patterns and matching patterns. The example shown in Figure 2 includes detection patterns 5-1, 5-2, 5-3 and matching pattern 5-4. Detection patterns 5-1 to 5-3 each define the conditions for the object to be detected. Detection pattern 5-1 defines "Subject," "Object," and "Relationship." For example, detection pattern 5-1 shows a relationship where a person corresponding to "Subject" approaches a forklift corresponding to "Object." "Relationship" is an example of interaction information.

[0018] In detection patterns 5-2 and 5-3, "Subject" and "Attribute" are defined. For example, in detection pattern 5-2, the attribute (Attribute) that the person corresponding to "Subject" is wearing a vest is indicated. "Attribute" is an example of attribute information.

[0019] Matching pattern 5-4 further defines the conditions for the matching target for each detection target that matches the conditions of detection patterns 5-1, 5-2, and 5-3. For example, matching pattern 5-4 defines "Detection target" and "Pattern". "Pattern" defines a pattern in which a person is approaching a forklift and the forklift is moving. In such a "Pattern", whether or not the person is approaching the forklift is determined based on detection pattern 5-1. Whether or not the forklift is moving is determined based on detection pattern 5-3. In addition, as defined in detection pattern 5-2, information that the target person is wearing a vest may be further set in "Pattern".

[0020] If video 10 matches the "Pattern" of matching pattern 5-4, it is determined that the matching conditions shown in "Detection target" are met.

[0021] The information processing device 100 evaluates multiple candidate detection patterns and matching patterns, and selects the optimal detection pattern and matching pattern based on the evaluation results. The information processing device 100 generates a knowledge graph 50 based on the selected detection pattern and matching pattern.

[0022] Figure 3 shows an example of a knowledge graph. For example, the knowledge graph 50 shown in Figure 3 is generated based on detection patterns 5-1 to 5-3 and matching pattern 5-4. The knowledge graph 50 includes nodes n1-1, n1-2, n1-3, n1-4, and n1-5. Node n1-1 is the node corresponding to "Subject is wearing a vest". Node n1-2 is the node corresponding to Person. An arrow is set from node n1-1 to node n1-2, indicating that the Subject of node n1-1 is defined in node n1-2.

[0023] Nodes n1-3 correspond to the "Subject is moving" node. Node n1-2 corresponds to the forklift node. An arrow is set from node n1-3 to node n1-4, indicating that the Subject of node n1-3 is defined in node n1-4.

[0024] Nodes n1-5 are nodes corresponding to "Subject is approaching Object". An arrow is set from node n1-5 to node n1-2, indicating that the Subject of node n1-5 is defined in node n1-2. An arrow is set from node n1-5 to node n1-4, indicating that the Object of node n1-5 is defined in node n1-4. Note that the knowledge graph 50 may be generated from detection patterns only. In that case, the knowledge graph 50 may be represented using the data structures 5-1 to 5-3. Furthermore, when the knowledge graph 50 is generated from both detection patterns and matching patterns, it may be represented using the data structures 5-1 to 5-4.

[0025] The KG generation process performed by the information processing device 100 has been described above.

[0026] Returning to the explanation of Figure 1, the ASG generation process performed by the information processing device 100 will be described. The ASG generation process is a process that generates an Action Scene Graph 60 from the video 10 using the detection patterns of the knowledge graph 50. The ASG is also called a Video Scene Graph or Spatio-temporal scene graph.

[0027] For example, the information processing device 100 performs object detection using a detection pattern on time-series frames of the video 10 and tracks the detected objects. The information processing device 100 generates video clips by summarizing the detection results and tracking results for a predetermined number of frames. The information processing device 100 inputs the video clips and prompts for relationship and attribute detection generated from the detection patterns into a visual detection model such as a Vision Language Model (VLM), thereby identifying the attribute information of the detected objects contained in the video clips, interaction information between detected objects, and the time when the attribute information and interaction information occurred.

[0028] The information processing device 100 generates an action scene graph 60 based on a video clip, attribute information of the detection target identified from the video clip, interaction information between the detection targets, and time. The action scene graph 60 maintains the relationship between Subject, object, and relation, or the relationship between Subject, object, and attribute, on an event basis (attribute information, relation information).

[0029] Figure 4 shows an example of an action scene graph. As shown in Figure 4, the action scene graph 60 has time nodes n2-1, n2-2, n2-3, n2-4, n2-5, and n2-6. The action scene graph 60 has event nodes n3-1, n3-2, n3-3, n3-4, n3-5, and n3-6. The action scene graph 60 has concrete object nodes n4-1, n4-2, n4-3, n4-4, and n4-5.

[0030] Time nodes n2-1 to n2-6 are nodes that indicate time, and correspond to times T1, T2, T3, T4, T5, and T6, respectively. For example, times T1, T2, T3, T4, T5, and T6 are associated with the time (frame number) of each frame contained in the video clip.

[0031] Event nodes n3-1 to n3-6 are nodes corresponding to attribute information and interaction information. For example, event nodes n3-1 to n3-3 correspond to "wearing a vest". Event nodes n3-4 and n3-6 correspond to "moving". Event node n3-5 corresponds to "approaching".

[0032] Specific object nodes n4-1 to n4-5 are nodes corresponding to the detection target. For example, specific object nodes n4-1 to n4-4 correspond to people P1, P2, P3, and P4, respectively. Specific object node 4-5 corresponds to a forklift.

[0033] By using the action scene graph 60, it becomes possible to grasp various information about the video 10. For example, event node n3-1, which is connected to time nodes n2-1 and n2-6, is connected to concrete object node n4-2. This indicates that person P2, who is wearing a vest, is present in the video 10 during times T1 to T6.

[0034] The event node n3-2, connected to time nodes n2-1 and n2-6, is connected to the concrete object node n4-3. This indicates that person P3, wearing a vest, is present in video 10 during times T1 to T6.

[0035] The event node n3-3, connected to time nodes n2-1 and n2-6, is connected to the concrete object node n4-4. This indicates that person P4, wearing a vest, is present in video 10 during times T1 to T6.

[0036] Event nodes 3-4, connected to time nodes n2-1 and n2-3, are connected to object node n4-5. This indicates that the moving forklift is present in video 10 during times T1 to T3.

[0037] The event node n3-5, connected to time nodes n2-2 and n2-3, is connected to concrete object nodes n4-1 and n4-5. This indicates that the event of person P1 approaching a moving forklift occurred in time T2-T3 of video 10.

[0038] Event node n3-6, connected to time nodes n2-5 and n2-6, is connected to object node n4-5. This indicates that the moving forklift is present in video 10 at times T5-T6.

[0039] The above describes the ASG generation process performed by the information processing device 100.

[0040] Returning to the explanation of Figure 1, the graph analysis process performed by the information processing device 100 will be described. The graph analysis process is a process that, when a question sentence 11 related to the video 10 is received from user U1, uses LLM to analyze the behavior scene graph 60 and generates an answer.

[0041] For example, the information processing device 100 generates a search query based on the question 11 and the knowledge graph 50, and uses this search query to perform a data search on the behavior scene graph 60. The information processing device 100 then generates an answer using the results of the data search.

[0042] Next, an example of the configuration of the information processing device 100 according to this embodiment 1 will be described. Figure 5 is a functional block diagram showing the configuration of the information processing device according to this embodiment 1. As shown in Figure 5, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0043] The communication unit 110 performs data communication with the camera via the network. For example, the communication unit 110 receives video data from the camera. The video data is the video data 10 described in Figure 1.

[0044] Also, the communication unit 110 performs data communication with an external server via a network. For example, the communication unit 110 receives text data from the external server. The text data is the data of the text 12 described in FIG. 1.

[0045] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing apparatus 100. The user U1 may operate the input unit 120 to input the question sentence 11.

[0046] The display unit 130 is a display device that displays the information output from the control unit 150.

[0047] The storage unit 140 has a knowledge graph 50, an action scene graph 60, a video buffer 70, and a text table 80. The storage unit 140 is a memory or the like.

[0048] The knowledge graph 50 is graph data generated based on detection patterns and matching patterns. For example, the description of the knowledge graph 50 is the same as the description of the knowledge graph 50 in FIG. 3.

[0049] The action scene graph 60 is graph data generated based on the knowledge graph 50 and video data. The description of the action scene graph 60 is the same as the description of the action scene graph 60 in FIG. 4.

[0050] The video buffer 70 is a buffer that stores video data.

[0051] The text table 80 stores text data.

[0052] Moving on to the description of the control unit 150. The control unit 150 includes an acquisition unit 151, a KG generation unit 152, an ASG generation unit 153, and a Graph analysis unit 154. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0053] The acquisition unit 151 acquires video data captured by the camera. The acquisition unit 151 stores the video data in the video buffer 70. The acquisition unit 151 also acquires text data from an external server. The acquisition unit 151 stores the text data in the text table 80.

[0054] The KG generation unit 152 generates a knowledge graph 50 by executing the above-described KG generation process. The KG generation unit 152 stores the knowledge graph 50 in the storage unit 140.

[0055] The ASG generation unit 153 generates an action scene graph 60 by executing the above ASG generation process. The ASG generation unit 153 stores the action scene graph 60 in the storage unit 140.

[0056] When the Graph analysis unit 154 receives the input of the question text 11, it generates an answer by executing the above-described Graph analysis process. The Graph analysis unit 154 outputs the answer to the display unit 130 for display.

[0057] Next, an example of the processing procedure of the information processing device 100 according to this embodiment 1 will be described. Figure 6 is a flowchart of the processing procedure of the information processing device according to this embodiment 1. As shown in Figure 6, the acquisition unit 151 of the information processing device 100 acquires text data and stores it in the text table 80 (step S10). The KG generation unit 152 of the information processing device 100 generates a knowledge graph 50 based on the text data stored in the text table 80 (step S11). The acquisition unit 151 acquires video data and stores it in the video buffer 70 (step S12).

[0058] The ASG generation unit 153 of the information processing device 100 generates an action scene graph 60 based on the video data stored in the video buffer 70 and the knowledge graph 50 (step S13).

[0059] The graph analysis unit 154 of the information processing device 100 receives the question 11 (step S14). The graph analysis unit 154 performs graph analysis and generates an answer (step S15). The graph analysis unit 154 outputs the answer (step S16).

[0060] Next, the effects of the information processing device 100 according to this embodiment 1 will be described. The information processing device 100 generates a knowledge graph 50 through a KG generation process and generates an action scene graph 60 through an ASG generation process. When the information processing device 100 receives a question 11 from user U1, it generates an answer based on the knowledge graph 50 and the action scene graph 60. This makes it possible to generate an accurate answer to the question.

[0061] Before describing this second embodiment, we will provide some supplementary explanation regarding the challenges of conventional technologies when generating answers to questions. For example, when recognizing people or objects from video, object detection technologies such as YOLO (You Only Look Once) are used. Recognized objects are indicated by bounding boxes.

[0062] If the answer to the question is to output the recognition result of the object shown in the bounding box above, it is possible to generate an accurate answer. However, for questions that require combining multiple recognition results to generate an answer, conventional technology has not been able to generate an accurate answer.

[0063] Next, an information processing device according to this second embodiment will be described. Figure 7 is a diagram illustrating the processing of the information processing device according to this second embodiment. As shown in Figure 7, the information processing device 200 has a graph analysis unit 252. When the graph analysis unit 252 receives a question sentence 11a related to the video from user U1, it uses LLM to analyze the behavior scene graph 60 and generates an answer 11b.

[0064] For example, the graph analysis unit 252 includes a first generation unit 261, a second generation unit 262, a search unit 263, a third generation unit 264, and a response generation unit 265. Each processing unit will be described in order below.

[0065] First, the processing of the first generation unit 261 will be explained. The first generation unit 261 obtains the question sentence 11a and generates a first prompt 261a based on the question sentence 11a and the detection pattern 16. The first prompt 261a is used to cause the LLM to execute the generation of a search query for searching the behavior scene graph 60. The detection pattern 16 is a natural language sentence that represents the target of detection.

[0066] Figures 8A, 8B, and 8C are diagrams illustrating the processing of the first generation unit. As shown in Figure 8A, the first generation unit 261 generates a first prompt 261a by embedding the result of format conversion of the detection pattern 16 and the question text 11a into a pre-prepared template 17. For example, the first prompt 261a is set to information obtained from the detection pattern 16, specifically information regarding the structure of the behavior scene graph 60. An example of the detection pattern 16 is shown in Figure 8B. An example of the first prompt 261a is shown in Figure 8C.

[0067] The first generation unit 261 outputs the first prompt 261a to the second generation unit 262.

[0068] Next, the processing of the second generation unit 262 in Figure 7 will be explained. The second generation unit 262 generates a search query 262a by inputting the first prompt 261a to the LLM. For example, the search query 262a includes the node to be searched, interaction information between nodes, and attribute information of the node, with respect to the behavior scene graph 60. The second generation unit 262 outputs the search query 262a to the search unit 263.

[0069] Next, the processing of the search unit 263 in Figure 7 will be explained. Based on the search query 262a, the search unit 263 performs a search on the behavior scene graph 60 and obtains the search result 263a. The search unit 263 outputs the search result 263a to the third generation unit 264.

[0070] Here, we will use Figure 4 to supplement the processing of the search unit 263. For example, suppose the search query 262a specifies "Person" and "Forklift" as the nodes to be searched, and "Approaching" as the interaction information. In this case, the search unit 263 identifies the specific object nodes n4-1 and n4-5 and the event node n3-5 corresponding to the search query 262a, and searches for the start time and end time of the event node n3-5. In this case, the start time is T2 and the end time is T3. As a result, the search result shows that the time the person is approaching the forklift is T2 to T3.

[0071] For example, suppose the search query 262a specifies "Person" as the node to be searched and "wearing a vest" as the attribute information. In this case, the search unit 263 identifies the specific object nodes n4-1 to n4-3 and the event nodes n3-1 to n3-3 that correspond to the search query 262a, and searches for the start time to the end time of the event nodes n3-1 to n3-3, respectively.

[0072] For example, the start time of event node n3-1 is T1 and the end time is T6. The start time of event node n3-2 is T1 and the end time is T6. The start time of event node n3-3 is T1 and the end time is T6. As a result, the search result shows that each person (P2, P3, P4) was wearing the vest from T1 to T6.

[0073] In addition, a search query 262a may contain multiple search items, and the search unit 263 performs the above search for each search item and generates the information obtained from each search as search result 263a.

[0074] Next, the processing of the third generation unit 264 in Figure 7 will be explained. The third generation unit 264 generates a second prompt 264a based on the search result 263a. The second prompt 264a is used when the LLM is to generate the answer.

[0075] Figures 9A and 9B are diagrams illustrating the processing of the third generation unit. As shown in Figure 9A, the third generation unit 264 generates the second prompt 264a by embedding the question text 11a and the search results 263a into a pre-prepared template 18. Figure 9B shows an example of the second prompt 264a.

[0076] The third generation unit 264 outputs the second prompt 264a to the answer generation unit 265.

[0077] Next, the processing of the answer generation unit 265 will be explained. The answer generation unit 265 generates the answer 11b by inputting the second prompt 264a to the LLM. The answer generation unit 265 outputs the generated answer 11b.

[0078] Next, an example of the configuration of the information processing device 200 that performs the above-described process will be explained. Figure 10 is a functional block diagram showing the configuration of the information processing device according to this second embodiment. As shown in Figure 10, this information processing device 200 has a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0079] The communication unit 210 performs data communication with external devices, etc., via the network.

[0080] The input unit 220 is an input device that inputs various types of information to the control unit 250 of the information processing device 200. For example, the user operates the input unit 220 to input the question sentence 11a.

[0081] The display unit 230 is a display device that displays information output from the control unit 250. For example, the display unit 230 displays the answer 11b.

[0082] The memory unit 240 has a detection pattern 16 and an action scene graph 60. The memory unit 240 is a memory or the like.

[0083] Detection pattern 16 is a natural language expression that represents the object to be detected, and corresponds to, for example, detection pattern 16 shown in Figure 8.

[0084] The behavior scene graph 60 is graph data generated based on the detection pattern 16 and video data. The explanation of the behavior scene graph 60 is the same as the explanation of the behavior scene graph 60 in Figure 4.

[0085] We will now move on to the explanation of the control unit 250. The control unit 250 has an acquisition unit 251 and a graph analysis unit 252. The control unit 250 is a CPU, GPU, etc.

[0086] The acquisition unit 251 acquires the question text 11a, etc., from the user. The acquisition unit 251 outputs the question text 11a to the graph analysis unit 252.

[0087] When the Graph analysis unit 252 receives a question 11a related to the video, it uses LLM to analyze the behavior scene graph 60 and generates an answer 11b. For example, as explained in Figure 7, the Graph analysis unit 252 has a first generation unit 261, a second generation unit 262, a search unit 263, a third generation unit 264, and an answer generation unit 265. The Graph analysis unit 252 outputs the answer to the display unit 230.

[0088] The graph analysis unit 252 generates a search query 262a based on the question 11a and the detection pattern 16 related to the behavior scene graph 60, and searches the behavior scene graph 60 based on the search query 262a. The graph analysis unit 252 generates and outputs an answer 11b based on the search results of the behavior scene graph 60.

[0089] Next, an example of the processing procedure of the information processing device 200 according to this second embodiment will be described. Figure 11 is a flowchart of the processing procedure of the information processing device according to this second embodiment. As shown in Figure 11, the acquisition unit 251 of the information processing device 200 acquires the question text (step S100).

[0090] The Graph analysis unit 252 of the information processing device 200 generates a first prompt 261a based on the question text 11a and the detection pattern 16 (step S101). The Graph analysis unit 252 generates a search query 262a by inputting the first prompt 261a to the LLM (step S102).

[0091] The Graph analysis unit 252 performs a search on the behavior scene graph 60 based on the search query 262a (step S103). The Graph analysis unit 252 generates a second prompt 264a based on the search result 263a (step S104).

[0092] The graph analysis unit 252 generates the answer 11b by inputting the second prompt 264a to the LLM (step S105). The graph analysis unit 252 outputs the answer to the display unit 230 (step S106).

[0093] Next, the effects of the information processing device 200 according to this second embodiment will be described. The information processing device 200 generates a search query 262a based on the question text 11a and the detection pattern 16 related to the behavior scene graph 60, and searches the behavior scene graph 60 based on the search query 262a. The graph analysis unit 252 generates and outputs an answer 11b based on the search results of the behavior scene graph 60. This makes it possible to generate an accurate answer to the question text.

[0094] The information processing device 200 generates a first prompt 251a based on the question text 11a and the detection pattern 16 related to the behavior scene graph 60, and inputs it to the LLM to generate a search query 262a. This allows for the efficient generation of the search query 262a.

[0095] The information processing device 200 generates a second prompt 264a based on the search result 263a and inputs it into the LLM to generate the answer 11b. This allows for efficient generation of the answer.

[0096] The information processing device 200 identifies nodes corresponding to the search query from the behavior scene graph 60 and obtains information of nodes associated with the identified nodes as search results. For example, the information processing device 200 searches for the time at which an event related to attribute information occurred in the video, and the time at which an event related to the interaction information occurred in the video, based on the time node among the nodes corresponding to the search query. This makes it possible to find the time at which an event related to attribute information related to the search query occurred, and the time at which an event related to interaction information occurred.

[0097] Next, an information processing device according to this third embodiment will be described. Figure 12 is a diagram illustrating the processing of the information processing device according to this third embodiment. As shown in Figure 12, the information processing device 300 has an acquisition unit 351 and a graph analysis unit 352.

[0098] When the acquisition unit 351 acquires a question sentence 13a related to the video from user U1, it outputs the question sentence 13a and a prompt 352a for the agent to the graph analysis unit 352. For example, the prompt 352a is set to an instruction to generate an answer to the user's question sentence 13a using the graph information acquisition tool 363.

[0099] When the Graph analysis unit 352 receives the prompt 352a and the question 13a, it uses the LLM and the graph information acquisition tool 363 to analyze the behavior scene graph 60 and generate the answer 13b.

[0100] For example, the graph analysis unit 352 includes an explanatory text generation unit 361, an agent execution unit 362, and a graph information acquisition tool 363. Each processing unit will be described in order below.

[0101] First, the processing of the explanatory text generation unit 361 will be explained. Based on the detection pattern 16, the explanatory text generation unit 361 generates an explanatory text 361a for the graph information acquisition tool. The explanatory text 361a includes information about the structure of the behavior scene graph 60.

[0102] Figures 13A and 13B illustrate the processing of the explanatory text generation unit. As shown in Figure 13A, the explanatory text generation unit 361 generates an explanatory text 361a by embedding the result of format conversion of the detection pattern 16 into a pre-prepared template 19. Figure 13B shows an example of the explanatory text 361a.

[0103] The explanation generation unit 361 outputs the explanation 361a to the agent execution unit 362.

[0104] Next, the processing of the agent execution unit 362 in Figure 12 will be explained. The agent execution unit 362 obtains the prompt 352a, the explanatory text 361a, and the question text 13a.

[0105] The agent execution unit 362 performs actions based on the prompt 352a. For example, the agent execution unit 362 generates a prompt by setting the explanatory text 361a and the question text 13a in a pre-prepared template, and generates an information acquisition query 362a by inputting this prompt into the LLM.

[0106] Figure 14 shows an example of an information retrieval query. In the information retrieval query 362a shown in Figure 14, the event to be searched and the output format of the search results are specified. For example, the event to be searched is specified as the event when a Person approaches a Forklift that is moving. The output format is specified as outputting the sum of the durations of the events to be searched.

[0107] The agent execution unit 362 obtains graph information 363b by inputting the information acquisition query 362a to the graph information acquisition tool 363. The graph information 363b will be explained later.

[0108] The agent execution unit 362 generates a prompt with graph information 363b set in a pre-configured template, and inputs this prompt to the LLM to generate the answer 13b. The agent execution unit 362 outputs the answer 13b.

[0109] The graph information acquisition tool 363 searches the behavior scene graph 60 based on the information acquisition query 362a. Figure 15 is a diagram illustrating the processing of the graph information acquisition tool. The graph information acquisition tool 363 generates a search and statistical processing query 362b by performing a format conversion on the information acquisition query 362a. The search query 262a described in Example 2 aimed to retrieve information on the corresponding node in the behavior scene graph, but the search and statistical processing query 362b aims to further perform statistical processing based on the retrieved information. Here, statistical processing refers to the average, sum, etc. It is assumed that the search and statistical processing query 362b is written in a query language for accessing the behavior scene graph, similar to the search query 262a described in Example 2.

[0110] The graph information acquisition tool 363 performs a search and statistical processing on the behavior scene graph 60 based on the search and statistical processing query 362b, and obtains the statistical processing result 363a (corresponding to graph information 363b).

[0111] The graph information acquisition tool 363 outputs the graph information 363b to the agent execution unit 362.

[0112] Next, an example of the configuration of the information processing device 300 that performs the above-described process will be explained. Figure 16 is a functional block diagram showing the configuration of the information processing device according to this embodiment 3. As shown in Figure 16, this information processing device 300 has a communication unit 310, an input unit 320, a display unit 330, a storage unit 340, and a control unit 350.

[0113] The communication unit 310 performs data communication with external devices, etc., via the network.

[0114] The input unit 320 is an input device that inputs various types of information to the control unit 350 of the information processing device 300. For example, the user operates the input unit 320 to input the question sentence 13a.

[0115] The display unit 330 is a display device that displays information output from the control unit 350. For example, the display unit 330 displays the answer 13b.

[0116] The memory unit 340 has a detection pattern 16 and an action scene graph 60. The memory unit 340 is a memory or the like.

[0117] Detection pattern 16 is a natural language expression that represents the object to be detected, and corresponds to, for example, detection pattern 16 shown in Figure 8.

[0118] The behavior scene graph 60 is graph data generated based on the detection pattern 16 and video data. The explanation of the behavior scene graph 60 is the same as the explanation of the behavior scene graph 60 in Figure 4.

[0119] We will now move on to the explanation of the control unit 350. The control unit 350 has an acquisition unit 351 and a graph analysis unit 352. The control unit 250 is a CPU, GPU, etc.

[0120] The acquisition unit 351 acquires the question text 13a, etc., from the user. The acquisition unit 251 outputs the question text 13a to the graph analysis unit 352. The acquisition unit 251 also outputs the prompt 352a to the agent execution unit 362. The user may specify the prompt 352a by operating the input unit 320.

[0121] When the Graph analysis unit 352 receives a prompt 352a and a question 13a, it uses the LLM and the graph information acquisition tool 363 to analyze the behavior scene graph 60 and generate an answer 13b. For example, as explained in Figure 12, the Graph analysis unit 352 includes an explanation text generation unit 361, an agent execution unit 362, and a graph information acquisition tool 363.

[0122] The Graph Analysis Unit 352 uses its agent function to generate an information acquisition query 362a and inputs the information acquisition query 362a into the graph information acquisition tool 363 to search the behavior scene graph 60. Based on the search results from the graph information acquisition tool 363, the Graph Analysis Unit 352 generates and outputs the answer 11b.

[0123] Next, an example of the processing procedure of the information processing device 300 according to this third embodiment will be described. Figure 17 is a flowchart showing the processing procedure of the information processing device according to this third embodiment. As shown in Figure 17, the graph analysis unit 352 of the information processing device 200 generates an explanatory text 361a based on the detection pattern 16 (step S200).

[0124] The acquisition unit 351 of the information processing device 300 acquires the question text 13a from the user and outputs the question text 13a and prompt 352a to the graph analysis unit 352 (step S201). The graph analysis unit 352 searches the behavior scene graph 60 using the graph information acquisition tool 363 (step S202).

[0125] The graph analysis unit 352 generates an answer based on the search results (step S203). The graph analysis unit 352 outputs the answer to the display unit 330 (step S204).

[0126] Next, the effects of the information processing device 300 according to this embodiment 3 will be described. The information processing device 300 uses an agent function to generate an information acquisition query 362a and inputs the information acquisition query 362a into the graph information acquisition tool 363 to search the behavior scene graph 60. The graph analysis unit 352 generates and outputs the answer 11b based on the search results of the graph information acquisition tool 363.

[0127] Furthermore, the information processing device 300 according to this embodiment 3 may use an agent and tool framework to enable interaction with the user and utilization of graph analysis.

[0128] Figure 18 shows another example of the information processing device according to this embodiment 3. For example, the information processing device 300 has a graph analysis tool 371 and an image acquisition tool 372 in addition to the agent execution unit 362 described in Figure 12.

[0129] The graph analysis tool 371 analyzes the behavioral scene graph 60 based on the search query specified to the agent execution unit 362 and generates search results.

[0130] The video acquisition tool 372 acquires video related to the search results retrieved by the graph analysis tool 371. For example, if the search results include an event that occurred between times T1 and T2, indicating that a Person is approaching a moving Forklift, the video acquisition tool 372 acquires video from the relevant video source for times T1 and T2 and outputs it to the agent execution unit 362.

[0131] The agent execution unit 362 generates and outputs a response 13b based on the search results from the graph analysis tool 371. The agent execution unit 362 also outputs video acquired from the video acquisition tool 372 as video 13c related to the response 13b.

[0132] As explained in Figure 18, the processing by the information processing device 300 allows user U1 to refer not only to the answer 13b but also to the video 13c related to the answer 13b.

[0133] Next, an example of a computer hardware configuration that realizes the same functions as the information processing device 100 (200, 300) shown in the above embodiment will be described in order.

[0134] Figure 19 shows an example of a computer hardware configuration that realizes similar functions to the information processing device of the embodiment. As shown in Figure 19, the computer 400 has a CPU 401 that performs various calculations, an input device 402 that receives data input from the user, and a display 403. The computer 400 also has a communication device 404 that exchanges data with a camera, external devices, etc. via a wired or wireless network, and an interface device 405. The interface device 405 may have a microphone, speaker, etc. connected to it. The computer 400 also has a RAM 406 that temporarily stores various information and a hard disk drive 407. Each of the devices 401 to 407 is connected to a bus 408.

[0135] The hard disk drive 407 includes an acquisition program 407a, a KG generation program 407b, an ASG generation program 407c, and a graph analysis program 407d. The CPU 401 reads each of the programs 407a to 407d and loads them into the RAM 406.

[0136] The acquisition program 407a functions as the acquisition process 406a. The KG generation program 407b functions as the KG generation process 406b. The ASG generation program 407c functions as the ASG generation process 406c. The graph analysis program 407d functions as the graph analysis process 406d.

[0137] The processing in acquisition process 406a corresponds to the processing in acquisition units 151, 251, and 351. The processing in KG generation process 406b corresponds to the processing in KG generation unit 152. The processing in ASG generation process 406c corresponds to the processing in ASG generation unit 153. The processing in graph analysis process 406d corresponds to the processing in graph analysis units 154, 252, and 352.

[0138] Furthermore, programs 407a to 407d do not necessarily have to be stored in the hard disk drive 407 from the beginning. For example, each program can be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 400. Then, the computer 400 can read and execute each program 407a to 407d.

[0139] 50 Knowledge Graph 60 Behavioral Scene Graph 70 Video Buffer 80 Text Table 100 Information Processing Device 110 Communication Unit 120 Input Unit 130 Display Unit 140 Storage Unit 150 Control Unit 151, 251, 351 Acquisition Unit 152 KG Generation Unit 153 ASG Generation Unit 154, 252, 352 Graph Analysis Unit

Claims

1. An information processing program characterized by causing a computer to perform the following processes: obtain information about the structure of the graph data to be searched and a question about objects contained in the video; generate a search query for the graph data based on the information about the structure of the graph data to be searched; search for graph data in which attribute information of the objects or interaction information between the objects is associated with the objects contained in the video based on the generated search query; and output information about the objects as an answer to the question based on the search results of the retrieved graph data.

2. The information processing program according to claim 1, characterized in that the process for generating the search query generates the search query by inputting information about the structure of the graph data and a prompt based on the question sentence into a large-scale language model.

3. The information processing program according to claim 1, characterized in that the output process generates an answer to the question by inputting a prompt based on the search results of the search process into a large-scale language model, and outputs the answer.

4. The graph data is data with a structure in which nodes of objects, nodes of attribute information, nodes of interaction information, and nodes of time are connected to each other by related nodes; the search query includes at least one of the information that specifies the object to be searched and the interaction information, and the information that specifies the object to be searched and the attribute information; and the search process identifies a node corresponding to the search query from the graph data, and obtains information of the node associated with the identified node as a search result, characterized in that the information processing program is as described in claim 1.

5. The information processing program according to claim 4, wherein the search process further executes a process to search for the time at which an event related to the attribute information occurred in the video, and the time at which an event related to the interaction information occurred in the video, based on the time node among the nodes corresponding to the search query.

6. An information processing program characterized by obtaining information about the structure of graph data to be searched and a question about an object contained in a video; providing the question and the information about the structure of the graph data to be searched to an agent implemented with a large-scale language model, causing the agent to call a graph information acquisition tool with appropriate arguments to acquire graph information; and causing a computer to execute a process to generate an answer based on the acquired graph information.

7. The information processing program according to claim 6, characterized in that the graph information acquisition tool generates queries that perform a search on the graph data to be searched and statistical processing.

8. An information processing method characterized in that a computer performs the following processes: obtains information about the structure of the graph data to be searched and a question about objects contained in the video; generates a search query for the graph data based on the information about the structure of the graph data to be searched; searches for graph data in which attribute information of the objects or interaction information between the objects is associated with the objects contained in the video based on the generated search query; and outputs information about the objects as an answer to the question based on the search results of the retrieved graph data.

9. The information processing method according to claim 8, characterized in that the process for generating the search query generates the search query by inputting information regarding the structure of the graph data and a prompt based on the question sentence into a large-scale language model.

10. The information processing method according to claim 8, characterized in that the output process generates an answer to the question by inputting a prompt based on the search results of the search process into a large-scale language model, and outputs the answer.

11. The graph data is data with a structure in which nodes of objects, nodes of attribute information, nodes of interaction information, and nodes of time are connected to each other by related nodes; the search query includes at least one of the information that specifies the object to be searched and the interaction information, and the information that specifies the object to be searched and the attribute information; and the search process identifies a node corresponding to the search query from the graph data, and obtains information of the node associated with the identified node as a search result, characterized in that the information processing method according to 8.

12. The information processing method according to claim 11, characterized in that the search process further performs a process of searching for the time at which an event related to the attribute information occurred in the video, and the time at which an event related to the interaction information occurred in the video, based on the time node among the nodes corresponding to the search query.

13. An information processing method characterized by obtaining information about the structure of graph data to be searched and a question about an object contained in the image; providing the question and the information about the structure of the graph data to be searched to an agent implemented by a large-scale language model, causing the agent to call a graph information acquisition tool with appropriate arguments to acquire graph information; and having a computer perform a process to generate an answer based on the acquired graph information.

14. The information processing method according to 13, characterized in that the graph information acquisition tool generates a query that performs a search on the graph data to be searched and statistical processing.

15. An information processing device having a control unit that performs the following processes: obtains information about the structure of the graph data to be searched and a question about an object contained in the video; generates a search query for the graph data based on the information about the structure of the graph data to be searched; searches for graph data in which attribute information of the object or interaction information between the objects is associated with the object contained in the video based on the generated search query; and outputs information about the object as an answer to the question based on the search results of the retrieved graph data.

16. The information processing apparatus according to claim 15, characterized in that the process for generating the search query generates the search query by inputting information regarding the structure of the graph data and a prompt based on the question sentence into a large-scale language model.

17. The information processing apparatus according to claim 15, characterized in that the output process generates an answer to the question by inputting a prompt based on the search results of the search process into a large-scale language model, and outputs the answer.

18. The information processing apparatus according to 15, wherein the graph data is data with a structure in which nodes of objects, nodes of attribute information, nodes of interaction information, and nodes of time are connected to each other by related nodes, the search query includes at least one of the information that specifies the object to be searched and the interaction information, and the information that specifies the object to be searched and the attribute information, and the search process identifies a node corresponding to the search query from the graph data, and obtains information of the node associated with the identified node as a search result.

19. The information processing apparatus according to claim 18, wherein the search process further executes a process to search for the time at which an event related to the attribute information occurred in the video, and the time at which an event related to the interaction information occurred in the video, based on the time node among the nodes corresponding to the search query.

20. An information processing device having a control unit that obtains information about the structure of graph data to be searched and a question about an object contained in a video, provides the question and the information about the structure of the graph data to be searched to an agent implemented by a large-scale language model, and causes the agent to call a graph information acquisition tool with appropriate arguments to acquire graph information, and then executes a process to generate an answer based on the acquired graph information.

21. The information processing apparatus according to claim 20, characterized in that the graph information acquisition tool generates a query that performs a search on the graph data to be searched and statistical processing.

Images