Answer generation program, answer generation method, and information processing device.

Abstract

The challenge is to suppress hallucination resulting from agent-generated outcomes. [Solution] The information processing device acquires a question input to a first agent that generates information in response to input information, which concerns video footage monitoring a specific task. Based on the acquired question, the information processing device identifies a specific agent from among a plurality of second agents that can cooperate with the first agent, which has either video recognition or domain knowledge of the specific task function. The information processing device then has the first agent output the answer result based on the generated information generated by the specific agent in accordance with the instructions of the first agent as the answer to the question.

Description

[Technical Field]

[0001] The present invention relates to a response generation program, a response generation method, and an information processing device. [Background technology]

[0002] In recent years, there has been an increase in AI chatbot services where artificial intelligence (AI) answers user questions. For example, dialogue systems have been disclosed in which AI agents use large-scale language models to answer questions from users. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 7509972 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] However, with the above technology, when the agent uses a large-scale language model to answer questions about specific tasks, it may generate content that is not factual or generate output that is irrelevant to the context in a plausible way. Therefore, it is not possible to suppress hallucination of the results generated by the agent.

[0005] In one aspect, the objective is to provide a response generation program, a response generation method, and an information processing device that can suppress hallucination of results generated by an agent. [Means for solving the problem]

[0006] In the first aspect, the answer generation program causes a computer to execute a process of obtaining, for a first agent that generates information according to input information, a question input with respect to the first agent, the question being about a video for monitoring a specific operation, and identifying, based on the obtained question, a specific agent having a function of either video recognition of the specific operation or domain knowledge of the specific operation among a plurality of second agents that can cooperate with the first agent, and outputting, as an answer to the question, an answer result based on generated information generated by the specific agent according to an instruction of the first agent by the first agent.

Advantages of the Invention

[0007] According to one embodiment, it is possible to suppress hallucination of results generated by an agent.

Brief Description of the Drawings

[0008] [Figure 1] FIG. 1 is a diagram for explaining an information processing apparatus according to Example 1. [Figure 2] FIG. 2 is a functional block diagram showing the functional configuration of the information processing apparatus according to Example 1. [Figure 3] FIG. 3 is a flowchart for explaining the processing of a parent agent. [Figure 4] FIG. 4 is a diagram for explaining an example of a prompt given to a parent agent. [Figure 5] FIG. 5 is a diagram for explaining graph analysis processing. [Figure 6] FIG. 6 is a diagram showing an example of the data structure of a detection pattern and a collation pattern. [Figure 7] FIG. 7 is a diagram showing an example of a knowledge graph. [Figure 8] FIG. 8 is a diagram showing an example of an action scene graph. [Figure 9] FIG. 9 is a diagram for explaining image analysis processing. [Figure 10]Figure 10 is a diagram illustrating a specific example situation. [Figure 11] Figure 11 is a diagram illustrating an example screen for a specific example (Step 1). [Figure 12] Figure 12 is a diagram illustrating an example screen for a specific example (second step). [Figure 13] Figure 13 is a diagram illustrating an example screen for a specific example (step 3). [Figure 14] Figure 14 illustrates another example of how a child agent processes information. [Figure 15] Figure 15 illustrates an example of a hardware configuration. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, embodiments of the answer generation program, answer generation method, and information processing device disclosed in this application. However, the present invention is not limited by these embodiments. Each embodiment can be combined as appropriate within a non-consistent range. [Examples]

[0010] <Description of information processing device> Figure 1 is a diagram illustrating the information processing device 300 according to Embodiment 1. The information processing device 300 shown in Figure 1 is an example of a computer device that runs AI agents (hereinafter simply referred to as agents) and generates and outputs answers in cooperation with each agent in response to questions from the user. This embodiment will be explained as an example of considering measures to reduce accident risk in warehouse operations where forklifts and workers work together.

[0011] An agent is a software program that collects data, uses that data to perform self-determined tasks, and achieves predetermined goals. For example, an agent might proactively generate responses using a pre-trained machine learning model or a large-scale language model. The agent independently selects and executes the optimal actions necessary to achieve the goals set by the administrator or other relevant parties.

[0012] As shown in Figure 1, the information processing device 300 executes a parent agent 330, which is an example of a first agent, and multiple child agents (child agent 330X, child agent 330Y, child agent 330Z), which are examples of second agents.

[0013] The parent agent 330 requests processing from each child agent, aggregates the generated information produced by the child agents, generates a response, and outputs it to the user. For example, the parent agent 330 inputs the generated information obtained from the child agents into an LLM (Large Language Model) to generate a response and outputs it to the user.

[0014] Child agent 330X performs domain analysis on information input from parent agent 330, using its knowledge of safety and health management and user journeys to consider countermeasures. For example, child agent 330X outputs appropriate countermeasures to parent agent 330 as generated information based on business knowledge and video analysis results, based on the information input by parent agent 330.

[0015] Child agent 330Y detects events represented by abstract textual expressions from the video data based on information input from parent agent 330, and performs graph analysis to provide examples and perform statistical analysis. For example, child agent 330Y generates an Action Scene Graph (ASG) from the video data (which may be simply referred to as video below), and generates the ASG in advance. Then, child agent 330Y uses the ASG to perform graph analysis to detect information (events) input by parent agent 330, and outputs the analysis results as generated information to parent agent 330.

[0016] Child agent 330Z performs image analysis processing to appropriately recognize context (on-site characteristics) using an autoencoder that learns important information to remember and compresses visual features, based on the information input from parent agent 330. For example, child agent 330Z applies visual prompts to video data to pre-generate video with visual prompts. Then, child agent 330Z uses a Vision-Language Model (VLM) to analyze the on-site context necessary for the answer from the video data with visual prompts based on the information input from parent agent 330, and outputs the analysis results as generated information to parent agent 330.

[0017] In this system configuration, the information processing device 300 receives a question from the user to the parent agent 330, which generates information in response to the input information, and the question concerns an object in the video footage being monitored for a specific task. Based on the question, the information processing device 300 identifies a specific agent from among several child agents that can cooperate with the parent agent, which has either video recognition or domain knowledge capabilities for the specific task. Subsequently, the information processing device 300 causes the parent agent 330 to output the answer result, based on the generated information generated by the specific agent in accordance with the instructions of the parent agent 330, as the answer to the question.

[0018] For example, the information processing device 300 causes the parent agent 330 to obtain a user's question regarding the consideration of safety measures for warehouse operations, etc. (S1). Subsequently, the information processing device 300 causes the parent agent 330 to determine the processing necessary to obtain an answer to the question and requests the corresponding child agent to perform the analysis processing (S2).

[0019] Then, the information processing device 300 instructs the child agent that has been requested to perform the analysis to execute the process (S3), and outputs the processing result (generated information) to the parent agent 330 (S4). After that, the information processing device 300 instructs the parent agent 330 to aggregate the processing results of each child agent (S5), and outputs a response to the user based on the aggregated information (S6).

[0020] In this way, the information processing device 300 can perform appropriate analysis in response to the question, thereby suppressing hallucination of results generated by the agent.

[0021] <Functional Configuration> Figure 2 is a functional block diagram showing the functional configuration of the information processing device 300 according to Embodiment 1. As shown in Figure 2, the information processing device 300 has a communication unit 301, a storage unit 302, and a control unit 310.

[0022] The communication unit 301 is a processing unit that controls communication with other devices, and is implemented, for example, by a communication interface. For example, the communication unit 301 receives video from multiple cameras installed in a warehouse. The communication unit 301 also receives questions from a user terminal performing analysis and sends questions to the user terminal.

[0023] The memory unit 302 is a processing unit that stores various data and various programs executed by the control unit 310, and is implemented by, for example, memory or a hard disk. For example, this memory unit 302 stores the domain knowledge DB 303 and the video data DB 304. In addition to the above, the memory unit 302 also stores, for example, various trained machine learning models used by the control unit 310.

[0024] Domain Knowledge DB303 is a database that stores knowledge specific to a particular field. Specifically, Domain Knowledge DB303 stores information necessary for considering policies and knowledge necessary for interpreting results. In this embodiment, for example, Domain Knowledge DB303 stores knowledge related to safety and training in warehouse operations. To give one example, Domain Knowledge DB303 stores safety-related knowledge such as "In warehouses where machines and workers work in the same place, each worker should wear a vest of a conspicuous color," and training-related knowledge such as "Too many training sessions can lead to a decrease in motivation, so it is preferable to limit them to three times a year."

[0025] The video data DB304 is a database that stores the video to be analyzed. In this embodiment, as an example, the video data DB304 stores video footage captured by cameras A, B, C, and D, which are installed in different locations in the warehouse. The video data DB304 may also store the video frame by frame. Furthermore, the cameras may be installed in locations where accidents have occurred in the past or near-miss locations.

[0026] The control unit 310 is a processing unit that manages the information processing device 300 and is implemented by, for example, a processor. This control unit 310 executes the response control unit 311, the domain analysis unit 312, the graph analysis unit 313, and the image analysis unit 314. The response control unit 311, the domain analysis unit 312, the graph analysis unit 313, and the image analysis unit 314 are implemented by electronic circuits and processes executed by the processor.

[0027] (Response control unit 311: Parent agent 330) The response control unit 311 is a processing unit that executes the parent agent 330 and causes the parent agent 330 to perform various controls. Specifically, the response control unit 311 acquires a question that is input to the parent agent 330, which generates information according to the input information, and the question concerns an object in the video that monitors a specific task. Based on the question, the response control unit 311 identifies a specific agent from among several child agents that can cooperate with the parent agent 330 that has either the function of video recognition for the specific task or the function of domain knowledge for the specific task. Subsequently, the response control unit 311 causes the parent agent 330 to output the response result based on the generated information generated by the specific child agent in accordance with the instructions of the parent agent 330 as the answer to the question.

[0028] For example, the response control unit 311 instructs the parent agent 330 to perform the following processes. Figure 3 is a flowchart illustrating the processes of the parent agent 330. As shown in Figure 3, the parent agent 330 obtains a question from the user regarding warehouse operations (S101). Subsequently, the parent agent 330 determines the analysis process to generate an answer to the question according to the planning information in which instructions and aggregation conditions are predetermined (S102).

[0029] The parent agent 330 then instructs the child agent to perform the determined analysis process (S103). Here, the parent agent 330 may, if necessary, have the next child agent perform further analysis (S104), or determine whether the response results obtained from the child agent contain enough information to generate a response to the user, and if insufficient, request reprocessing (S105).

[0030] Subsequently, the parent agent 330 aggregates the response results from the child agents (S106), inputs the aggregated results into the LLM (S107), and outputs the LLM output results to the user as the answer (S108).

[0031] Here, the planning information and other details set for the parent agent 330 as described above can be set as prompts. Figure 4 illustrates an example of a prompt given to the parent agent 330. As shown in Figure 4, various pieces of information such as "behavior," "instructions," "wording," and "response format" can be set for a prompt.

[0032] "Behavior" is information that defines the behavior of the parent agent 330, and can be set to things like "knowledgeable person," "gentleman," or "expert." "Instructions" is information that defines the planning information of the parent agent 330, and sets the correspondence between the content of the question and the analysis process to be performed. For example, "Instructions" can be set to things like "If there is an image in the input of the question, perform image analysis," "If the question is about a policy, perform image analysis and then domain analysis," "If the question is about a period, perform graph analysis," "If the question is about a judgment, perform domain analysis," or "If the question is about a domain, perform image analysis." In addition, instructions can also be set to indicate the priority order, which should be prioritized when multiple conditions apply.

[0033] In this way, when a question corresponding to an instruction with a specified combination or order is input, the parent agent 330 has the child agents process the question according to the instruction and aggregates the results. On the other hand, even when a question that does not correspond to an instruction is input, the parent agent 330 interprets the information in the instruction as an example, autonomously determines the appropriate child agent, and requests processing from it.

[0034] Furthermore, "language usage" is information that defines the language used when the parent agent 330 outputs a response, and can be set to, for example, "expert". "Response format" is information that defines the format in which the parent agent 330 provides a response to the user, and can be set to, for example, "text format", "text format and image", or "audio".

[0035] Depending on the framework used by the parent agent 330, the parent agent can also determine whether the processing result of the child agent is sufficient as an answer. For example, the parent agent 330 can determine whether the information generated by the child agent is appropriate to use as an answer, and if it is determined to be inappropriate, it can request the child agent to regenerate the generated information (answer result). For example, the parent agent 330 can determine that the answer is insufficient if the processing result of the child agent does not include any of the pre-specified pieces of information, such as "who does what and how" or "whether an image corresponding to the event was detected." If the parent agent 330 determines that it is appropriate, it outputs an answer to the question.

[0036] (Domain analysis: Child agent 330X) The domain analysis unit 312 is a processing unit that executes the child agent 330X and causes the child agent 330X to perform various controls. Specifically, when the content of the question satisfies the second condition (content related to judgment) of the instruction pre-set in the parent agent 330, the domain analysis unit 312 searches for domain knowledge related to the specific task, generates the domain search results, and outputs them to the parent agent 330.

[0037] For example, the domain analysis unit 312 instructs the child agent 330X to perform the following processes. Specifically, when the child agent 330X receives a question about a policy from the parent agent 330, it refers to the domain knowledge DB 303 and answers the parent agent 330 with the most appropriate policy among several policies, or the expected results when each of the policies is implemented.

[0038] For example, child agent 330X can input domain knowledge and questions from parent agent 330 into a trained machine learning model to determine an appropriate course of action. Alternatively, child agent 330X may use digital twin or simulation techniques to identify the effects of each course of action and output this information to parent agent 330.

[0039] (Graph analysis: Child agent 330Y) The graph analysis unit 313 is a processing unit that executes the child agent 330Y and causes the child agent 330Y to perform various controls. Specifically, when the content of the question satisfies the first condition (content related to the period) of the instruction pre-set in the parent agent 330, the graph analysis unit 313 searches for graph data that represents the connections between objects in the image and generates the search results for the graph data.

[0040] For example, the graph analysis unit 313 instructs the child agent 330Y to perform the following processing. Specifically, the child agent 330Y obtains information about the structure of the graph data to be searched and a question about the objects contained in the video. Based on the information about the structure of the graph data to be searched, the child agent 330Y generates a query for searching the graph data. Based on the generated search query, the child agent 330Y searches for graph data in which object attribute information or interaction information between objects is associated with the objects contained in the video, and outputs information about the objects to the parent agent 330 by analyzing the results of the searched graph data.

[0041] Figure 5 illustrates the graph analysis process. As shown in Figure 5, when the child agent 330Y receives a question 11 related to the video 10 from user U1, it outputs an answer to the question 11. The video 10 is a time-series frame (still image).

[0042] Child agent 330Y performs KG generation, ASG generation, and graph analysis processes. For example, the KG generation and ASG generation processes are performed in advance. The graph analysis process is performed to generate an answer when the agent receives question 11 from the user. The following description will explain the KG generation process, ASG generation process, and graph analysis process in order.

[0043] The KG generation process performed by child agent 330Y is described below. The KG generation process generates a Knowledge Graph 50 that shows the conditions for detecting a certain event in the video 10. For example, the Knowledge Graph 50 is a graph corresponding to detection patterns and matching patterns.

[0044] For example, child agent 330Y obtains text 12 related to the domain of the detected object included in the video 10. The text 12 is something like "dangerous behavior with accident risk". Child agent 330Y generates a list of detected objects from the text 12 using LLM (Large Language Models) or the like. The list of detected objects is something like "approaching a moving forklift without wearing a vest", "carrying a load for a long time", "walking onto the road without checking left and right".

[0045] Child agent 330Y generates multiple candidate detection and matching patterns by setting a list of detection targets in the prompt for generating detection and matching patterns and inputting it into the LLM.

[0046] Figure 6 shows an example of the data structure for detection patterns and matching patterns. The example shown in Figure 6 includes detection patterns 5-1, 5-2, and 5-3, and matching pattern 5-4. Detection patterns 5-1 to 5-3 each define the conditions for the object to be detected. Detection pattern 5-1 defines "Subject," "Object," and "Relationship." For example, detection pattern 5-1 shows a relationship where a person corresponding to "Subject" approaches a forklift corresponding to "Object." "Relationship" is an example of interaction information.

[0047] Detection patterns 5-2 and 5-3 define "Subject" and "Attribute." For example, detection pattern 5-2 indicates that the person corresponding to "Subject" is wearing a vest. "Attribute" is an example of attribute information.

[0048] Matching pattern 5-4 further defines the matching conditions for each detection target that matches the conditions of detection patterns 5-1, 5-2, and 5-3. For example, matching pattern 5-4 defines "Detection target" and "Pattern". "Pattern" defines a pattern in which a person is approaching a forklift and the forklift is moving. In such a "Pattern", whether or not the person is approaching the forklift is determined based on detection pattern 5-1. Whether or not the forklift is moving is determined based on detection pattern 5-3. In addition, as defined in detection pattern 5-2, information that the target person is wearing a vest may be further set in "Pattern".

[0049] If video 10 matches the "Pattern" of matching pattern 5-4, it is determined that the matching conditions shown in "Detection target" are met.

[0050] Child agent 330Y evaluates multiple candidate detection and matching patterns and selects the optimal detection and matching patterns based on the evaluation results. Child agent 330Y generates a knowledge graph 50 based on the selected detection and matching patterns.

[0051] Figure 7 shows an example of a knowledge graph. For example, the knowledge graph 50 shown in Figure 7 is generated based on detection patterns 5-1 to 5-3 and matching pattern 5-4. Knowledge graph 50 includes nodes n1-1, n1-2, n1-3, n1-4, and n1-5. Node n1-1 is the node corresponding to "Subject is wearing a vest". Node n1-2 is the node corresponding to Person. An arrow is set from node n1-1 to node n1-2, indicating that the Subject of node n1-1 is defined in node n1-2.

[0052] Nodes n1-3 correspond to the "Subject is moving" node. Node n1-2 corresponds to the forklift node. An arrow is set from node n1-3 to node n1-4, indicating that the Subject of node n1-3 is defined in node n1-4.

[0053] Nodes n1-5 are nodes corresponding to "Subject is approaching Object". An arrow is set from node n1-5 to node n1-2, indicating that the Subject of node n1-5 is defined in node n1-2. An arrow is set from node n1-5 to node n1-4, indicating that the Object of node n1-5 is defined in node n1-4. Note that the knowledge graph 50 may be generated from detection patterns only. In that case, the knowledge graph 50 may be represented using the data structures 5-1 to 5-3. Furthermore, when the knowledge graph 50 is generated from both detection patterns and matching patterns, it may be represented using the data structures 5-1 to 5-4.

[0054] The above describes the KG generation process performed by child agent 330Y.

[0055] Next, we will explain the ASG generation process performed by child agent 330Y. The ASG generation process generates an Action Scene Graph (ASG) 60 from video 10 using the detection patterns of the Knowledge Graph 50. The ASG is also called a Video Scene Graph or Spatio-temporal scene graph.

[0056] For example, child agent 330Y performs object detection using a detection pattern on time-series frames of video 10 and tracks the detected objects. Child agent 330Y generates video clips by combining the detection results and tracking results into predetermined frame intervals. Child agent 330Y inputs the video clips and prompts for relationship and attribute detection generated from the detection patterns into a visual detection model such as VLM (Vision-Language Model) to identify the attribute information of the detected objects contained in the video clips, interaction information between detected objects, and the time when the attribute information and interaction information occurred.

[0057] Child agent 330Y generates an action scene graph 60 based on the video clip, the attribute information of the detected object identified from this video clip, the interaction information between the detected objects, and the time. The action scene graph 60 is generated based on events (attribute information) <attribute>Interaction information <relationship>It maintains the relationship between Subject, Object, and Relationship, or the relationship between Subject, Object, and Attribute, in units of ).

[0058] Figure 8 shows an example of an action scene graph. As shown in Figure 8, the action scene graph 60 has time nodes n2-1, n2-2, n2-3, n2-4, n2-5, n2-6. The action scene graph 60 has event nodes n3-1, n3-2, n3-3, n3-4, n3-5, n3-6. The action scene graph 60 has concrete object nodes n4-1, n4-2, n4-3, n4-4, n4-5, n4-6.

[0059] The time nodes n2-1 to n2-6 are nodes that represent time, corresponding to times T1, T2, T3, T4, T5, and T6, respectively. For example, times T1, T2, T3, T4, T5, and T6 are associated with the time (frame number) of each frame contained in a video clip.

[0060] Event nodes n3-1 to n3-6 are nodes corresponding to attribute information and interaction information. For example, event nodes n3-1 to n3-3 correspond to "wearing a vest". Event nodes n3-4 and n3-6 correspond to "moving". Event node n3-5 corresponds to "approaching".

[0061] Specific object nodes n4-1 to n4-6 are nodes that correspond to the detection target. For example, specific object nodes n4-1 to n4-4 correspond to people P1, P2, P3, and P4, respectively. Specific object node 4-5 corresponds to a forklift.

[0062] By using the action scene graph 60, it becomes possible to grasp various pieces of information related to video 10. For example, event node n3-1, which is connected to time nodes n2-1 and n2-6, is connected to concrete object node n4-2. This indicates that person P2, who is wearing a vest, is present in video 10 during times T1 to T6.

[0063] The event node n3-2, connected to time nodes n2-1 and n2-6, is connected to the concrete object node n4-3. This indicates that person P3, wearing a vest, is present in video 10 during times T1 to T6.

[0064] The event node n3-3, connected to time nodes n2-1 and n2-6, is connected to the concrete object node n4-4. This indicates that person P4, wearing a vest, is present in video 10 during times T1-T6.

[0065] Event nodes 3-4, connected to time nodes n2-1 and n2-3, are connected to object nodes n4-5. This indicates that the moving forklift was present in video 10 during times T1-T3.

[0066] The event nodes n3-5, connected to time nodes n2-2 and n2-3, are connected to concrete object nodes n4-1 and n4-5. This indicates that the event of person P1 approaching a moving forklift occurred at time T2-T3 in video 10.

[0067] Event nodes n3-6, connected to time nodes n2-5 and n2-6, are connected to object node n4-5. This indicates that the moving forklift was present in video 10 at times T5-T6.

[0068] The above describes the ASG generation process performed by child agent 330Y.

[0069] Next, the graph analysis process performed by the child agent 330Y will be described. The graph analysis process is a process that uses LLM to analyze the behavior scene graph 60 and generate an answer when a question sentence 11 related to the video 10 is received from user U1. For example, when a question sentence 11 related to the video 10 is received from user U1, the generating AI (e.g., LLM) generates an answer to the question sentence 11 based on the generated behavior scene graph 60. More specifically, when the information processing device 300 receives a question sentence from the user about a first object in the video, it identifies a result indicating interaction information associated with the first object based on the generated graph data, and the generating AI generates an answer to the question sentence based on the identified result indicating interaction information. For example, when the information processing device 300 receives a question sentence about a first object in the video, it searches the behavior scene graph 60 to identify a result indicating interaction information associated with the first object. The information processing device 300 then inputs a prompt, which consists of a question and interaction information, to the LLM to generate an answer to the question.

[0070] For example, child agent 330Y generates a search query based on question 11 and knowledge graph 50, and uses this search query to perform a data search on behavior scene graph 60. Child agent 330Y then generates an answer using the results of the data search.

[0071] (Image analysis: Child agent 330Z) The image analysis unit 314 is a processing unit that executes the child agent 330Z and causes the child agent 330Z to perform various controls. Specifically, when the content of the question satisfies the third condition (specifying a region) of the instructions pre-set for the parent agent 330, the image analysis unit 314 performs region recognition processing in the video and generates the result of the region recognition processing.

[0072] For example, the image analysis unit 314 causes the child agent 330Z to perform the following processing. Specifically, the child agent 330Z acquires video footage of the subject being monitored. The child agent 330Z identifies a first region in which a first object is located within a predetermined video frame among the multiple video frames that make up the acquired video, and a question about the first object that exists in the first region. By analyzing the acquired video footage, the child agent 330Z identifies a second object that is related to the first object that exists in the first region, among the multiple objects that each of the multiple video frames has. Based on the question about the first object, the image features of the first object and the second object, the child agent 330Z generates an answer to the question and outputs it to the parent agent 330.

[0073] Figure 9 illustrates the image analysis process. Refer to Figure 9 to get a comprehensive overview of the question-and-answer process. Figure 9 also shows the data used in each process. Each data point will be described using the name provided in Figure 9.

[0074] The video output device outputs video V. Video V contains a number of consecutive frames. The user uses a user terminal device to select a frame F from video V and set a visual prompt P for the selected frame F.

[0075] The visual encoder 101 extracts image feature quantities f from each frame of the video V. t Calculate.

[0076] The spatiotemporal feature calculation unit 102 calculates the image feature f of each frame calculated by the visual encoder 101. t From this, the spatial features f of the video V spatial and time features f temporal Calculate.

[0077] The overall projector 103 has spatial feature f spatial An embedding process is performed to match the feature quantities of the LLM decoder 110 to the spatial feature data e ν spatial generates it. Similarly, the overall projector 103 performs an embedding process to match the temporal feature amount f temporal to the feature amount space of the LLM decoder 110 for the temporal feature amount, and generates embedded data e ν temporal of the temporal feature amount.

[0078] The specified region extraction unit 104 generates a BBox 21 indicating an ROI, which is a partial region specified by the visual prompt P, based on the visual prompt P for the selected frame F.

[0079] The ROI tracker 105 performs a search for each frame of the video V using the BBox 21, and generates a BBox 22 indicating the ROI corresponding partial region of each frame.

[0080] The related region estimation unit 106 estimates the related regions in each target frame from which the ROI corresponding partial region is extracted, from the BBox 22 indicating the ROI corresponding partial region of each frame and the video V. Here, the related region estimation unit 106 estimates L related regions in descending order of the degree of relevance.

[0081] The partial region feature amount calculation unit 107 calculates the feature amount f Roi t,0 of the ROI corresponding partial region of each target frame.

[0082] Also, the partial region feature amount calculation unit 107 calculates the feature amounts f RRoi t,1 ~f RRoi t,L of the related regions of each target frame from the information indicating the related regions in each target frame. Here, since there are L related regions, the partial region feature amount calculation unit 107 calculates the feature amounts f RRoi t,1 ~f RRoi t,L for each of the L related regions.

[0083] The selection unit 108 selects the feature quantity f of the ROI-corresponding subregion of each target frame. Roi t,0 , and the feature quantities f of each related domain RRoi t,1 ~f RRoi t,L Select the features to use in your question from the options provided.

[0084] The projector 109 performs embedding processing on the feature quantities selected by the selection unit 108 to embed data e related to the ROI-corresponding subregion and related regions. RoI 0 and embedded data e RoI 1~e RoI L Generates.

[0085] The text conversion unit 111 performs text conversion processing in response to the text prompt T, according to the format of the question to the LLM decoder 110.

[0086] The embedding unit 112 performs an embedding process on the text prompt T that has undergone text conversion to embed data e t Generates.

[0087] The LLM decoder 110 uses the embedded spatial feature data e ν spatial , embedding data e for time features ν temporal , embedded data e related to ROI-corresponding sub-regions and related regions RoI 0 and embedded data e RoI 1~e RoI L , and embedded data e indicating the question t The LLM decoder 110 receives input. Based on the input data, it generates answer A to the question about the subject specified in the video and visual prompt.

[0088] <Specific example> Next, using Figures 10 to 13, we will explain an example of a specific processing flow from when a user inputs a question until they receive an answer. Figure 10 is a diagram illustrating the situation of the specific example, Figure 11 is a diagram illustrating an example screen of the specific example (Step 1), Figure 12 is a diagram illustrating an example screen of the specific example (Step 2), and Figure 13 is a diagram illustrating an example screen of the specific example (Step 3).

[0089] First, let's explain a specific example scenario. As shown in Figure 10, cameras A, B, C, and D are installed in different locations within the warehouse. Each camera captures images of workers, forklifts, and other tasks within its imaging range and outputs the images to the information processing device 300. For example, camera A captures the forklift's workspace, camera B captures the area near the entrance, camera C captures shelves where cardboard boxes are stacked, and camera D captures the workbench where workers are performing their tasks. It is assumed that ASG and KG are generated in advance.

[0090] Next, the user persona is a warehouse safety and hygiene manager. The manager is concerned about the significant risk of "workers not wearing vests coming into close proximity with forklifts" and wants to implement measures centered on education to prevent accidents.

[0091] In this situation, as shown in Figure 11, the parent agent 330 displays a screen to the user that says "Please enter a question," and receives the question from the user: "From the image data of the past three months, please display instances where a moving forklift and a worker not wearing a vest approached each other, along with the time and image."

[0092] The parent agent 330 then refers to the prompt's instructions and, in accordance with "Perform graph analysis for questions about time periods," requests child agent 330Y to perform a graph analysis. Child agent 330Y then performs the graph analysis and outputs the analysis results to the parent agent 330 as the answer result (generated information).

[0093] Subsequently, the parent agent 330 outputs to the user the results of the child agent 330Y's analysis, including images that match the user's question, information about the camera that captured the image, and the date and time of capture. For example, the parent agent 330 outputs multiple images, including one captured by camera A on "2024 / 09 / 20 13:00:05".

[0094] Next, as shown in Figure 12, the parent agent 330 displays a screen that says, "Answer provided. Do you have any further questions?" and accepts the user's screen search. For example, the user searches for images of camera A around "2024 / 09 / 20 13:00:05" obtained as an answer, and among the hit cases, their attention is drawn to the incident involving camera A. Based on this case, they consider "exploring why this incident occurred from the on-site situation (context) and receiving suggestions for improvement."

[0095] Then, the parent agent 330, after the user has selected a situation where the worker and the forklift are close together in a box (frame) on the image of camera A, receives a question from the user: "Please tell me what caused this situation to occur. Please tell me what measures can be taken to avoid it."

[0096] The parent agent 330 then refers to the prompt instructions and, in accordance with "If there is an image in the input of the question, perform image analysis" and "If the question is about a policy, perform image analysis first, then perform domain analysis," first requests image analysis from the child agent 330Z. The child agent 330Z then performs the image analysis and outputs the analysis results as the answer to the parent agent 330.

[0097] Subsequently, as shown in Figure 13, the parent agent 330 outputs the analysis results of the child agent 330Z to the child agent 330X and requests the child agent 330X to perform domain analysis. The child agent 330X then performs domain analysis based on the image analysis results and outputs the results of the consideration of each measure to the parent agent 330.

[0098] As a result of the above processing, the parent agent 330 inputs the response results from each child agent into an LLM or similar device, aggregates the results, and outputs the final response to the user. For example, the parent agent 330 outputs, as a response result, the "Cause: The worker was not wearing a vest..." for the event received in the question, and the content, advantages, and disadvantages of the measures to be taken, "Measure A" and "Measure B," received in the additional questions. For example, "Measure A" is a measure to change the color of the worker's vest, which has a small cost but a limited reduction in risk, while "Measure B" is a measure to separate the workspace of the forklift from the workspace of the worker, which has a high cost but is expected to have a significant reduction in risk.

[0099] <Effects> As described above, the information processing device 300 can perform appropriate analysis in response to the question, thereby suppressing hallucination of results generated by the agent.

[0100] Furthermore, the information processing device 300 detects objects to be detected from each frame of the video by analyzing the video using detection patterns. The information processing device 300 inputs a detection prompt generated from the detection patterns and a Visual Prompt generated from the detection results into a VLM or the like, generating results that show attribute information and interaction information of the detected objects. The information processing device 300 generates an action scene graph from the generated results. This makes it possible to generate an action scene graph that includes the Subject, Object, and Relationship that the user expects. Therefore, the information processing device 300 can generate an accurate answer to a question.

[0101] Furthermore, the information processing device 300 tracks the specified region of interest (ROI) for all frames, extracts the ROI-corresponding region, and extracts related regions that are related to and highly relevant to the ROI-corresponding region in each frame. The information processing device 300 then generates an answer using the spatial and temporal features of the entire video, the features of the ROI-corresponding region, and the features of the related regions. Therefore, the information processing device 300 can automatically take in surrounding information related to the specified target and provide it to an LMM (Large Multi-modal Model), etc. This allows the information processing device 300 to consider not only the changes in the spatial and temporal importance of the entire video and the important changes of the target of interest, but also the important changes of related objects such as people and objects that are highly relevant to the target of interest. Therefore, the information processing device 300 can improve its ability to understand images and videos. [Examples]

[0102] By the way, the process performed by the child agent described above is merely one example, and other processes can be performed. Therefore, in Example 2, as another example of a process performed by the child agent, we will explain "VQA (Visual Question Answering) that appropriately recognizes and selects context by realizing video information compression based on context as the basis for compression decisions."

[0103] Figure 14 illustrates another example of a child agent's processing. For example, the child agent inputs each frame of the video (video frame) into an encoder, extracts the visual features of each video frame, and stores each visual feature. Subsequently, the child agent inputs each visual feature into a compression mechanism, such as an autoencoder, and extracts contextual features from each visual feature.

[0104] The child agent then inputs the features of each context into the first topic extraction mechanism, which is a mechanism that predicts, extracts, and prioritizes objects and topics that may be the subject of questions, such as site characteristics and people involved, to generate topics of interest (which may be simply referred to as topics from now on) and stores them in the topic bank. Subsequently, the child agent performs sampling to extract features corresponding to the topics from the features of the context and stores the sampled features of the context in the memory bank.

[0105] In other words, during the initial video input phase, since no questions have been entered, the child agent extracts candidate topics that seem important from the video alone and generates the initial state of the topic bank using Topic extraction, which is an example of a first-level topic extraction mechanism. The child agent also stores information that is highly relevant to the features of the topic bank in the memory bank, such as when the number of frames exceeds the memory bank length.

[0106] Subsequently, when a question is input, as shown in Figure 14, the child agent inputs the question into an analysis mechanism to decompose it into morphemes. Next, the child agent inputs the obtained morphemes into a second topic extraction mechanism, which extracts the object or topic currently being questioned from the question and updates the topic bank, to extract topics. Then, the child agent inputs the extracted topics into a first transformation mechanism, which is an example of a projector that performs shape transformation into topics to be stored in the topic bank, and updates the topic bank with the shape-transformed topics.

[0107] Subsequently, the child agent performs sampling to extract features corresponding to the topics in the updated topic bank, and stores the sampled context features in the memory bank. In other words, the child agent can update (regenerate) the memory bank using the stored (stocked) image features. The child agent can also set update criteria, such as when a question on a previously unexplored topic is input. Furthermore, the child agent extracts context features that are highly relevant to the top K (where K is any number) topics in the topic bank.

[0108] The child agent then repeats the process shown in Figure 14 each time a question is input. When the question is finished, the child agent inputs the morphemes obtained from the question into a second transformation mechanism (Embedding) that converts it into an input format for the LLM, and converts them into numerical vectors. Similarly, the child agent inputs the context features stored in the memory bank into the projector, and converts (reconstructs) them into features that the LLM can understand (visual embeddings). After that, the child agent inputs the numerical vector of the question and the features (visual embeddings) into the LLM and outputs the answer obtained.

[0109] In this way, the child agent performs context-based feature compression and extracts candidate important topics from the video information. The child agent then updates the topics of interest according to the content of the questions that are input each time, and performs information compression or extracts information from stored video features based on the topics of interest, updating the memory so that it contains more information related to the topics of interest. After that, the child agent restores the compressed features in a way that the LLM can understand and inputs them into the LLM.

[0110] Therefore, the child agent can improve the accuracy of the VQA output by enabling video information storage and feature compression that can retain important information based on the video context and question content for long-duration videos. [Examples]

[0111] Now, although embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.

[0112] (Numerical values, etc.) The machine learning model, context, topic, features, video, number of child agents, instructions, prompts, etc., used in the above example are merely examples and can be changed as needed. Furthermore, the processing flow described in each flowchart can be modified as appropriate within a consistent range.

[0113] (Automatic generation of planning information) Furthermore, the parent agent 330 can also automatically generate planning information. For example, the parent agent 330 can use pre-configured instructions (example information) to generate planning information that specifies the execution order of child agents that generate information in response to questions, as well as the aggregation conditions for the generated information. For example, the parent agent 330 can also automatically generate this information using the functions of an AI agent, or it can be automatically generated using a machine learning model trained to automatically generate instructions in response to input example information and questions.

[0114] (system) Unless otherwise specified, the processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will.

[0115] Furthermore, the specific forms of distribution and integration of the components of each device are not limited to those shown in the diagram. For example, each child agent may run on a different device from the parent agent. In other words, all or part of the components may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. Moreover, each processing function of each device may be implemented, in whole or in any part, by a CPU and a program that is analyzed and executed by that CPU, or by hardware using wired logic.

[0116] Furthermore, each processing function performed by each device may be implemented, in whole or in part, by a CPU and a program executed for analysis by that CPU, or by hardware using wired logic.

[0117] (Hardware) Figure 15 illustrates an example of a hardware configuration. As shown in Figure 15, the information processing device 300 includes a communication device 300a, an HDD (Hard Disk Drive) 300b, memory 300c, and a processor 300d. Furthermore, the components shown in Figure 15 are interconnected by a bus or the like.

[0118] The communication device 300a is a network interface card or similar, and communicates with other devices. The HDD 300b stores programs and databases that operate the functions shown in Figure 2.

[0119] The processor 300d operates a process that performs the functions described in Figure 2 by reading a program that performs the same processing as each processing unit shown in Figure 2 from the HDD 300b or the like and loading it into memory 300c. For example, this process performs the same functions as each processing unit of the information processing device 300. Specifically, the processor 300d reads a program that has the same functions as the answer control unit 311, domain analysis unit 312, graph analysis unit 313, image analysis unit 314, etc. from the HDD 300b or the like. Then, the processor 300d executes a process that performs the same processing as the answer control unit 311, domain analysis unit 312, graph analysis unit 313, image analysis unit 314, etc.

[0120] Thus, the information processing device 300 operates as an information processing device that executes an information processing method by reading and executing a program. Furthermore, the information processing device 300 can also achieve the same functionality as the embodiment described above by reading the program from a recording medium using a media reader and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the information processing device 300. For example, the above embodiment may also be applied to cases where another computer or server executes the program, or where these computers or servers collaborate to execute the program.

[0121] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and executed by being read from the recording medium by a computer. [Explanation of Symbols]

[0122] 300 Information Processing Devices 301 Communications Department 302 Storage section 303 Domain Knowledge Database 304 Video Data Database 310 Control Unit 311 Response Control Unit 312 Domain Analysis Department 313 Graph Analysis Department 314 Image Analysis Department< / relationship> < / attribute>

Claims

1. On the computer, A question is entered into a first agent that generates information in response to the input information, and the agent obtains the aforementioned question regarding video footage monitoring a specific task. Based on the questions obtained, a specific agent is identified from among a plurality of second agents capable of cooperating with the first agent, which has either the function of image recognition for the specific task or domain knowledge for the specific task. The first agent outputs the response result based on the generated information generated by the specific agent in accordance with the instructions of the first agent as the answer to the question. A response generation program characterized by executing a process.

2. The output process described above is: Based on planning information that defines the aggregation conditions for the generation information of the specified multiple agents, the generation information generated by the specified multiple agents is aggregated. The first agent outputs the response result based on the aggregated generated information as the answer to the question. The answer generation program according to feature 1.

3. A video is acquired that includes an object and a person performing a specific task using the object. The aforementioned questions regarding measures to be taken in response to an incident that occurred while the person present in the aforementioned video was working were obtained. By analyzing the aforementioned video, the type of action taken by the person toward the object that caused the aforementioned event can be identified. Based on the domain knowledge of the aforementioned specific business, business knowledge information is identified, By inputting a prompt containing the identified type of action and the business knowledge information into a large-scale language model, an answer to the question is generated. The answer generation program according to claim 1 or 2, characterized in that it causes the computer to perform the processing.

4. The first agent is instructed to determine whether the information generated by the specific agent is appropriate to be used in the response result. If it is determined that it is inappropriate, the computer will perform a process that causes the first agent to request the specific agent to regenerate the generated information. The output process described above is: If deemed appropriate, the first agent outputs the response result based on the information generated by the specific agent as the answer to the question. The answer generation program according to feature 1 or 2.

5. The output process described above is: Using instructions pre-configured for the first agent, the first agent is instructed to generate planning information that specifies the execution order of the second agent, which generates generated information in response to the question, and the aggregation conditions for the generated information. The response results are generated according to the planning information. The answer generation program according to feature 1 or 2.

6. The aforementioned multiple second agents, This includes an agent responsible for searching domain knowledge related to the aforementioned specific task, an agent responsible for searching graph data representing the connections between objects in the video, and an agent responsible for recognizing regions within the video. The process to be identified is, Based on the content of the above questions, the order in which each agent will execute is determined. The answer generation program according to feature 1 or 2.

7. We obtain questions about the events of objects present in the monitored video. When the content of the question concerning the object satisfies the first condition of the instruction pre-set for the first agent, the second agent to which the question was input searches for graph data representing the connections between objects in the image, and generates the search results for the graph data. When the content of the aforementioned question satisfies the second condition of the instructions pre-configured for the first agent, the second agent to which the question was entered searches for domain knowledge relating to the specific task and generates domain search results. The answer to the question is generated by inputting a prompt containing the search results of the graph data and the search results of the domain knowledge into a large-scale language model. The answer generation program according to claim 1, characterized in that it causes the computer to perform the processing.

8. We obtain questions about the events of objects present in the monitored video. When the content of the question concerning the object satisfies the first condition of the instruction pre-set for the first agent, the second agent to which the question was input searches for graph data representing the connections between objects in the image, and generates the search results for the graph data. When the content of the aforementioned question satisfies the second condition of the instructions pre-configured for the first agent, the second agent to which the question was entered searches for domain knowledge relating to the specific task and generates domain search results. When the content of the question satisfies the third condition of the instructions pre-set for the first agent, the second agent that received the question performs region recognition processing within the video and generates the result of the region recognition processing. The answer to the question is generated by inputting a prompt to a large-scale language model that includes the search results of the graph data, the search results of the domain knowledge, and the execution result of the domain recognition process. The answer generation program according to claim 1, characterized in that it causes the computer to perform the processing.

9. We obtain information about the structure of the graph data to be searched and a question about the objects contained in the video. The process of the aforementioned specific agent is executed, and based on information about the structure of the graph data to be searched, a query for searching the graph data is generated. The process of the specified agent is executed, and based on the generated search query, graph data is searched in which object attribute information or interaction information between objects is associated with objects contained in the video. The first agent performs the processing described above and analyzes the results of the retrieved graph data to output information about the object. The answer generation program according to claim 1, characterized in that it causes the computer to perform the processing.

10. Acquire video footage of the monitored target, Identify the first region in which a first object is located within a predetermined video frame among the multiple video frames that make up the acquired video, and identify the question concerning the first object that exists in the first region. By executing the processing of the aforementioned specific agent and analyzing the acquired video, a second object is identified that is related to the first object located in the first region, among the multiple objects present in each of the multiple video frames. The first agent generates an answer to a question based on a question about a first object and image features of the first and second objects. The answer generation program according to claim 1, characterized in that it causes the computer to perform the processing.

11. Computers A question is entered into a first agent that generates information in response to the input information, and the agent obtains the aforementioned question regarding video footage monitoring a specific task. Based on the questions obtained, a specific agent is identified from among a plurality of second agents capable of cooperating with the first agent, which has either the function of image recognition for the specific task or domain knowledge for the specific task. The first agent outputs the response result based on the generated information generated by the specific agent in accordance with the instructions of the first agent as the answer to the question. A method for generating an answer, characterized by performing a process.

12. A question is entered into a first agent that generates information in response to the input information, and the agent obtains the aforementioned question regarding video footage monitoring a specific task. Based on the questions obtained, a specific agent is identified from among a plurality of second agents capable of cooperating with the first agent, which has either the function of image recognition for the specific task or domain knowledge for the specific task. The first agent outputs the response result based on the generated information generated by the specific agent in accordance with the instructions of the first agent as the answer to the question. An information processing device characterized by having a control unit.

Images