Generation program, generation method, and information processing device

Abstract

An information processing device according to the present invention performs the following: acquiring input data in which objection attribute information of an object or interaction information between objects is associated with a to-be-detected object; detecting a first object, which represents the to-be-detected object, from video frames constituting a video by analyzing a to-be-analyzed video by using the input data; generating a result indicating the attribute information of the first object or the interaction information with the first object by inputting a prompt, which includes the input data, and the detected first object into a visual language model; and generating graph data in which the generated result and the detected first object have been associated.

Description

Generation Program, Generation Method, and Information Processing Apparatus

[0001] The present invention relates to a generation program and the like.

[0002] In recent years, AI (Artificial Intelligence) chatbot services that answer users' questions have been increasing. For example, in the prior art, there is a dialogue system that uses a large language model such as an LLM (Large Language Models) to answer questions from users.

[0003] Here, when generating an answer to a question using a large language model, a phenomenon (hallucination) may occur where content different from the facts or content unrelated to the context is plausibly generated.

[0004] For example, in order to suppress hallucination, a technique called RAG (Retrieval Augmented Generation) is used. RAG improves the answer accuracy by combining external information retrieval when performing text generation by a large language model.

[0005] Patent No. 7509972

[0006] However, the above prior art has a problem that an accurate answer cannot be generated for a question.

[0007] For example, even when using RAG in the prior art, external information that can improve the answer accuracy cannot be set. Also, although it is conceivable to set graph data related to video as external information, it is difficult to generate graph data related to the detected object in the video.

[0008] In one aspect, an object of the present invention is to provide a generation program, a generation method, and an information processing apparatus capable of generating graph data related to an object to be detected.

[0009] In the first proposal, the computer is instructed to perform the following processes: The computer obtains input data to which object attribute information or interaction information between objects is associated with the object to be detected, analyzes the video to be analyzed using the input data to detect a first object representing the object to be detected from among the video frames that make up the video, inputs a prompt containing the input data and the detected first object into a visual language model to generate a result showing the attribute information or interaction information related to the first object, and generates graph data to which the generated result is associated with the detected first object.

[0010] It can generate graph data related to the detected object.

[0011] Figure 1 is a diagram illustrating the overall processing of the information processing device according to this embodiment 1. Figure 2 is a diagram illustrating an example of the data structure of the detection pattern and the matching pattern. Figure 3 is a diagram illustrating an example of a knowledge graph. Figure 4 is a diagram illustrating an example of an action scene graph. Figure 5 is a functional block diagram showing the configuration of the information processing device according to this embodiment 1. Figure 6 is a flowchart illustrating the processing procedure of the information processing device according to this embodiment 1. Figure 7 is a diagram illustrating the processing of the information processing device according to this embodiment 2. Figure 8 is a diagram illustrating the processing of the frame division unit. Figure 9 is a diagram illustrating the processing of the object detection unit. Figure 10 is a diagram illustrating the processing of the tracking unit. Figure 11 is a diagram illustrating the processing of the clipping unit. Figure 12 is a diagram illustrating the Visual Prompt creation process executed by the determination unit. Figure 13 is a diagram illustrating the format conversion process executed by the determination unit. Figure 14 is a diagram illustrating the determination process executed by the determination unit. Figure 15 is a diagram illustrating other processes of the determination unit. Figure 16 is a diagram illustrating another example of an action scene graph (1). Figure 17 is a diagram illustrating another example of an action scene graph (2). Figure 18 is a functional block diagram showing the configuration of the information processing device according to this embodiment 2. Figure 19 is a flowchart showing the processing procedure of the information processing device according to this embodiment 2. Figure 20 is a diagram showing an example of a computer hardware configuration that realizes the same functions as the information processing device in the embodiment.

[0012] The following describes in detail, with reference to the drawings, embodiments of the generation program, generation method, and information processing device disclosed in this application. However, this invention is not limited by these embodiments.

[0013] Figure 1 is a diagram illustrating the overall processing of the information processing device according to this embodiment 1. For example, the information processing device 100 of this embodiment 1 is a device that outputs an answer to a question sentence 11 related to a video 10 when it receives such a question sentence 11 from a user U1. The video 10 is a time-series frame (still image).

[0014] The information processing device 100 performs KG generation processing, ASG generation processing, and graph analysis processing. For example, the KG generation processing and ASG generation processing are performed in advance. The graph analysis processing is performed to generate an answer when a question 11 is received from the user. In the following description, the KG generation processing, ASG generation processing, and graph analysis processing will be described in order.

[0015] The KG generation process performed by the information processing device 100 will now be described. The KG generation process is the process of generating a Knowledge Graph 50 that shows the conditions for detecting a certain event in the video 10. For example, the Knowledge Graph 50 is a graph corresponding to detection patterns and matching patterns.

[0016] For example, the information processing device 100 acquires text 12 related to the domain of the detected object included in the video 10. The text 12 is such as "dangerous behavior with accident risk". The information processing device 100 generates a list of detected objects from the text 12 using LLM (Large Language Models) or the like. The list of detected objects is such as "approaching a moving forklift without wearing a vest", "carrying a load for a long time", and "entering the road without checking left and right".

[0017] The information processing device 100 generates multiple candidate detection patterns and matching patterns by setting a list of detection targets as a prompt for generating detection patterns and matching patterns and inputting it into the LLM.

[0018] Figure 2 shows an example of the data structure for detection patterns and matching patterns. The example shown in Figure 2 includes detection patterns 5-1, 5-2, 5-3 and matching pattern 5-4. Detection patterns 5-1 to 5-3 each define the conditions for the object to be detected. Detection pattern 5-1 defines "Subject," "Object," and "Relationship." For example, detection pattern 5-1 shows a relationship where a person corresponding to "Subject" approaches a forklift corresponding to "Object." "Relationship" is an example of interaction information.

[0019] In detection patterns 5-2 and 5-3, "Subject" and "Attribute" are defined. For example, in detection pattern 5-2, the attribute (Attribute) that the person corresponding to "Subject" is wearing a vest is indicated. "Attribute" is an example of attribute information.

[0020] Matching pattern 5-4 further defines the conditions for the matching target for each detection target that matches the conditions of detection patterns 5-1, 5-2, and 5-3. For example, matching pattern 5-4 defines "Detection target" and "Pattern". "Pattern" defines a pattern in which a person is approaching a forklift and the forklift is moving. In such a "Pattern", whether or not the person is approaching the forklift is determined based on detection pattern 5-1. Whether or not the forklift is moving is determined based on detection pattern 5-3. In addition, as defined in detection pattern 5-2, information that the target person is wearing a vest may be further set in "Pattern".

[0021] If video 10 matches the "Pattern" of matching pattern 5-4, it is determined that the matching conditions shown in "Detection target" are met.

[0022] The information processing device 100 evaluates multiple candidate detection patterns and matching patterns, and selects the optimal detection pattern and matching pattern based on the evaluation results. The information processing device 100 generates a knowledge graph 50 based on the selected detection pattern and matching pattern.

[0023] Figure 3 shows an example of a knowledge graph. For example, the knowledge graph 50 shown in Figure 3 is generated based on detection patterns 5-1 to 5-3 and matching pattern 5-4. The knowledge graph 50 includes nodes n1-1, n1-2, n1-3, n1-4, and n1-5. Node n1-1 is the node corresponding to "Subject is wearing a vest". Node n1-2 is the node corresponding to Person. An arrow is set from node n1-1 to node n1-2, indicating that the Subject of node n1-1 is defined in node n1-2.

[0024] Nodes n1-3 correspond to the "Subject is moving" node. Node n1-2 corresponds to the forklift node. An arrow is set from node n1-3 to node n1-4, indicating that the Subject of node n1-3 is defined in node n1-4.

[0025] Nodes n1-5 are nodes corresponding to "Subject is approaching Object". An arrow is set from node n1-5 to node n1-2, indicating that the Subject of node n1-5 is defined in node n1-2. An arrow is set from node n1-5 to node n1-4, indicating that the Object of node n1-5 is defined in node n1-4. Note that the knowledge graph 50 may be generated from detection patterns only. In that case, the knowledge graph 50 may be represented using the data structures 5-1 to 5-3. Furthermore, when the knowledge graph 50 is generated from both detection patterns and matching patterns, it may be represented using the data structures 5-1 to 5-4.

[0026] The KG generation process performed by the information processing device 100 has been described above.

[0027] Returning to the explanation of Figure 1, the ASG generation process performed by the information processing device 100 will be described. The ASG generation process is a process that generates an Action Scene Graph 60 from the video 10 using the detection patterns of the knowledge graph 50. The ASG is also called a Video Scene Graph or Spatio-temporal scene graph.

[0028] For example, the information processing device 100 performs object detection using a detection pattern on time-series frames of the video 10 and tracks the detected objects. The information processing device 100 generates video clips by summarizing the detection results and tracking results for a predetermined number of frames. The information processing device 100 inputs the video clips and prompts for relationship and attribute detection generated from the detection patterns into a visual detection model such as a Vision Language Model (VLM), thereby identifying the attribute information of the detected objects contained in the video clips, interaction information between detected objects, and the time when the attribute information and interaction information occurred.

[0029] The information processing device 100 generates an action scene graph 60 based on a video clip, attribute information of the detection target identified from the video clip, interaction information between the detection targets, and time. The action scene graph 60 maintains the relationship between Subject, object, and relation, or the relationship between Subject, object, and attribute, on an event basis (attribute information, relation information).

[0030] Figure 4 shows an example of an action scene graph. As shown in Figure 4, the action scene graph 60 has time nodes n2-1, n2-2, n2-3, n2-4, n2-5, and n2-6. The action scene graph 60 has event nodes n3-1, n3-2, n3-3, n3-4, n3-5, and n3-6. The action scene graph 60 has concrete object nodes n4-1, n4-2, n4-3, n4-4, and n4-5.

[0031] Time nodes n2-1 to n2-6 are nodes that indicate time, and correspond to times T1, T2, T3, T4, T5, and T6, respectively. For example, times T1, T2, T3, T4, T5, and T6 are associated with the time (frame number) of each frame contained in the video clip.

[0032] Event nodes n3-1 to n3-6 are nodes corresponding to attribute information and interaction information. For example, event nodes n3-1 to n3-3 correspond to "wearing a vest". Event nodes n3-4 and n3-6 correspond to "moving". Event node n3-5 corresponds to "approaching".

[0033] Specific object nodes n4-1 to n4-5 are nodes corresponding to the detection target. For example, specific object nodes n4-1 to n4-4 correspond to people P1, P2, P3, and P4, respectively. Specific object node 4-5 corresponds to a forklift.

[0034] By using the action scene graph 60, it becomes possible to grasp various information about the video 10. For example, event node n3-1, which is connected to time nodes n2-1 and n2-6, is connected to concrete object node n4-2. This indicates that person P2, who is wearing a vest, is present in the video 10 during times T1 to T6.

[0035] The event node n3-2, connected to time nodes n2-1 and n2-6, is connected to the concrete object node n4-3. This indicates that person P3, wearing a vest, is present in video 10 during times T1 to T6.

[0036] The event node n3-3, connected to time nodes n2-1 and n2-6, is connected to the concrete object node n4-4. This indicates that person P4, wearing a vest, is present in video 10 during times T1 to T6.

[0037] Event nodes 3-4, connected to time nodes n2-1 and n2-3, are connected to object node n4-5. This indicates that the moving forklift is present in video 10 during times T1 to T3.

[0038] The event node n3-5, connected to time nodes n2-2 and n2-3, is connected to concrete object nodes n4-1 and n4-5. This indicates that the event of person P1 approaching a moving forklift occurred in time T2-T3 of video 10.

[0039] Event node n3-6, connected to time nodes n2-5 and n2-6, is connected to object node n4-5. This indicates that the moving forklift is present in video 10 at times T5-T6.

[0040] The above describes the ASG generation process performed by the information processing device 100.

[0041] Returning to the explanation of Figure 1, the graph analysis process performed by the information processing device 100 will be described. The graph analysis process is a process that uses LLM to analyze the behavior scene graph 60 and generate an answer when a question sentence 11 related to the video 10 is received from user U1. For example, when a question sentence 11 related to the video 10 is received from user U1, the generating AI (e.g., LLM) generates an answer to the question sentence 11 based on the generated behavior scene graph 60. More specifically, when the information processing device 100 receives a question sentence from the user regarding a first object in the video, it identifies a result indicating interaction information associated with the first object based on the generated graph data, and the generating AI generates an answer to the question sentence based on the identified result indicating interaction information. For example, when the information processing device 100 receives a question sentence regarding a first object in the video, it searches the behavior scene graph 60 to identify a result indicating interaction information associated with the first object. The information processing device 100 then inputs a prompt, which consists of a question and interaction information, to the LLM to generate an answer to the question.

[0042] Furthermore, for example, the information processing device 100 generates a search query based on the question 11 and the knowledge graph 50, and uses this search query to perform a data search on the behavior scene graph 60. The information processing device 100 then generates an answer using the results of the data search.

[0043] Next, a configuration example of the information processing apparatus 100 according to the first embodiment will be described. FIG. 5 is a functional block diagram showing the configuration of the information processing apparatus according to the first embodiment. As shown in FIG. 5, the information processing apparatus 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0044] The communication unit 110 performs data communication with a camera via a network. For example, the communication unit 110 receives video data from the camera. The video data is the data of the video 10 described in FIG. 1.

[0045] Also, the communication unit 110 performs data communication with an external server via a network. For example, the communication unit 110 receives text data from the external server. The text data is the data of the text 12 described in FIG. 1.

[0046] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing apparatus 100. The user U1 may operate the input unit 120 to input the question sentence 11.

[0047] The display unit 130 is a display device that displays information output from the control unit 150.

[0048] The storage unit 140 includes a knowledge graph 50, an action scene graph 60, a video buffer 70, and a text table 80. The storage unit 140 is a memory or the like.

[0049] The knowledge graph 50 is graph data generated based on detection patterns and matching patterns. For example, the description of the knowledge graph 50 is the same as the description of the knowledge graph 50 in FIG. 3.

[0050] The action scene graph 60 is graph data generated based on the knowledge graph 50 and video data. The description of the action scene graph 60 is the same as the description of the action scene graph 60 in FIG. 4.

[0051] The video buffer 70 is a buffer that stores video data.

[0052] Text table 80 stores text data.

[0053] We will now move on to the explanation of the control unit 150. The control unit 150 includes an acquisition unit 151, a KG generation unit 152, an ASG generation unit 153, and a graph analysis unit 154. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0054] The acquisition unit 151 acquires video data captured by the camera. The acquisition unit 151 stores the video data in the video buffer 70. The acquisition unit 151 also acquires text data from an external server. The acquisition unit 151 stores the text data in the text table 80.

[0055] The KG generation unit 152 generates a knowledge graph 50 by executing the above-described KG generation process. The KG generation unit 152 stores the knowledge graph 50 in the storage unit 140.

[0056] The ASG generation unit 153 generates an action scene graph 60 by executing the above ASG generation process. The ASG generation unit 153 stores the action scene graph 60 in the storage unit 140.

[0057] When the Graph analysis unit 154 receives the input of the question text 11, it generates an answer by executing the above-described Graph analysis process. The Graph analysis unit 154 outputs the answer to the display unit 130 for display.

[0058] Next, an example of the processing procedure of the information processing device 100 according to this embodiment 1 will be described. Figure 6 is a flowchart of the processing procedure of the information processing device according to this embodiment 1. As shown in Figure 6, the acquisition unit 151 of the information processing device 100 acquires text data and stores it in the text table 80 (step S10). The KG generation unit 152 of the information processing device 100 generates a knowledge graph 50 based on the text data stored in the text table 80 (step S11). The acquisition unit 151 acquires video data and stores it in the video buffer 70 (step S12).

[0059] The ASG generation unit 153 of the information processing device 100 generates an action scene graph 60 based on the video data stored in the video buffer 70 and the knowledge graph 50 (step S13).

[0060] The graph analysis unit 154 of the information processing device 100 receives the question 11 (step S14). The graph analysis unit 154 performs graph analysis and generates an answer (step S15). The graph analysis unit 154 outputs the answer (step S16).

[0061] Next, the effects of the information processing device 100 according to this embodiment 1 will be described. The information processing device 100 generates a knowledge graph 50 through a KG generation process and generates an action scene graph 60 through an ASG generation process. When the information processing device 100 receives a question 11 from user U1, it generates an answer based on the knowledge graph 50 and the action scene graph 60. This makes it possible to generate an accurate answer to the question.

[0062] Before describing this second embodiment, we will explain the challenges of the conventional technology for generating behavioral scene graphs. In the conventional technology, it is common to detect predetermined subjects, objects, and relationships in the input video and generate a behavioral scene graph.

[0063] Furthermore, a technique has been proposed in which captions are generated for images using VLM, and a Scene Graph (SG) is generated based on the generated captions. This conventional technique can generate an SG for any subject, object, and relationship.

[0064] However, the conventional technology described above cannot generate an action scene graph that includes the subject, object, and relationship that the user expects. If such an action scene graph is used as is, as the action scene graph 60 described in Example 1, it is not possible to generate an answer that the user desires in response to a question.

[0065] Next, an information processing device according to this second embodiment will be described. Figure 7 is a diagram illustrating the processing of the information processing device according to this second embodiment. As shown in Figure 7, the information processing device 200 has an ASG generation unit 252. The ASG generation unit 252 generates an action scene graph 60 based on the video 15 and the detection pattern 16.

[0066] For example, the ASG generation unit 252 includes a frame division unit 261, an object detection unit 262, a tracking unit 263, a clipping unit 264, a determination unit 265, and an ASG construction unit 266. Each processing unit will be described in order below.

[0067] The processing of the frame division unit 261 will now be explained. The frame division unit 261 acquires the video 15 and divides the video into frames. The video 15 is the "video to be analyzed".

[0068] Figure 8 is a diagram illustrating the processing of the frame division unit. The video 15 is data having time-series frames (still images). As shown in Figure 8, the frame division unit 261 divides the video 15 into a plurality of frames 15a. The frame division unit 261 outputs the plurality of frames 15a to the object detection unit 262. For example, each frame is assigned a frame number.

[0069] Next, the processing of the object detection unit 262 in Figure 7 will be explained. The object detection unit 262 analyzes multiple frames 15a based on the detection pattern 16 and detects the object to be detected from each frame. The detection pattern 16 is natural language that represents the object to be detected. The detection pattern 16 corresponds to the "input data".

[0070] Figure 9 is a diagram illustrating the processing of the object detection unit. For example, detection pattern 16 includes detection patterns 16-1, 16-2, and 16-3. Detection patterns 16-1 to 16-3 each define the conditions for the object to be detected. In detection pattern 16-1, "Subject," "Object," and "Relationship" are defined. For example, in detection pattern 16-1, the relationship (Relationship) is that a person (Person) corresponding to "Subject" approaches a forklift corresponding to "Object." "Relationship" is an example of interaction information.

[0071] Detection patterns 16-2 and 16-3 define "Subject" and "Attribute". For example, detection pattern 16-2 indicates that the person corresponding to "Subject" is wearing a vest. "Attribute" is an example of attribute information.

[0072] In detection patterns 16-1 to 16-3, a more detailed definition can be provided by using "Expression". For example, in detection pattern 16-1, the "Expression" states that when a person (Person) corresponding to "Subject" approaches a forklift (Object), it means that the distance between the person (Person) and the forklift is less than 1 meter.

[0073] Furthermore, in detection patterns 16-1 to 16-3, the "label" of the object to be detected is also defined.

[0074] The object detection unit 262 outputs the coordinates of the bounding box for each label using Open-Vocabulary object detection technology. The object detection unit 262 may use existing technologies such as "Cheng, Tianheng, et al. "Yolo-world: Real-time open-vocabulary object detection." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024." as the Open-Vocabulary object detection technology.

[0075] The object detection unit 262 uses Open-Vocabulary object detection technology to generate a detection result 18-1 from frame 15a-1 based on the detection pattern 16. The detection result 18-1 shown in Figure 9 includes the coordinates of bounding boxes b1-1, b1-2, b1-3, and b1-4 corresponding to the label "Person," and the coordinates of the bounding box b2-1 corresponding to the label "Forklift."

[0076] The object detection unit 262 generates a detection result 18 for each frame by repeatedly executing the above process for multiple frames 15a. The object detection unit 262 outputs the detection result 18 to the tracking unit 263. For example, the detection result 18 contains the frame number and the coordinates of the bounding box for each label. The object detection unit 262 also outputs multiple frames 15a to the tracking unit 263.

[0077] Next, the processing of the tracking unit 263 in Figure 7 will be explained. Based on the detection result 18, the tracking unit 263 tracks the coordinates of the bounding box of each label and generates tracking information for each label. Each piece of tracking information is assigned a tracking ID. For example, when performing tracking, the tracking unit 263 may use existing technologies such as "Bewley, Alex, et al. "Simple online and realtime tracking." 2016 IEEE international conference on image processing (ICIP). IEEE, 2016."

[0078] Figure 10 is a diagram illustrating the processing of the tracking unit. For example, the tracking unit 263 acquires the detection result 18, tracks the coordinates of the bounding box b1-1 of the label "Person" included in the detection result 18, and generates tracking information. This tracking information is assigned the tracking ID "P1", and the tracking result of the coordinates of the bounding box b1-1 is set. In the tracking information, the tracking unit 263 may associate the frame number of each frame with the coordinates of the bounding box being tracked.

[0079] Similarly, the tracking unit 263 tracks the coordinates of the bounding box b1-2 of the label "Person" and generates tracking information. This tracking information is assigned the tracking ID "P2" and contains the tracking result for the coordinates of the bounding box b1-2. The tracking unit 263 tracks the coordinates of the bounding box b1-3 of the label "Person" and generates tracking information. This tracking information is assigned the tracking ID "P3" and contains the tracking result for the coordinates of the bounding box b1-3. The tracking unit 263 tracks the coordinates of the bounding box b1-4 of the label "Person" and generates tracking information. This tracking information is assigned the tracking ID "P4" and contains the tracking result for the coordinates of the bounding box b1-4.

[0080] The tracking unit 263 tracks the coordinates of the bounding box b2-1 of the label "Forklift" and generates tracking information. This tracking information is assigned the tracking ID "F1" and the tracking result of the bounding box b2-1 coordinates is set. The tracking unit 263 outputs the tracking information with tracking IDs "P1, P2, P3, P4, F1" as tracking result information 19 to the clipping unit 264. The tracking unit 263 also outputs multiple frames 15a to the clipping unit 264.

[0081] Next, the clipping unit 264 in Figure 7 will be described. The clipping unit 264 divides the multiple frames 15a into clips of a predetermined number of frames each. For example, the number of frames included in one clip may be 10.

[0082] Figure 11 is a diagram illustrating the processing of the clipping unit. For example, the clipping unit 264 generates a clip C1 that combines frame numbers fr1 to fr10 of multiple frames 15a. The clipping unit 264 generates a clip C2 that combines frame numbers fr11 to fr20 of multiple frames 15a. The clipping unit 264 repeatedly performs the above processing for frames from frame number fr21 onwards to generate clip information 20 containing multiple clips.

[0083] The clipping unit 264 outputs clip information 20 and tracking result information 19 to the determination unit 265.

[0084] Next, the determination unit 265 in Figure 7 will be explained. Based on the detection pattern 16, tracking result information 19, and clip information 20, the determination unit 265 determines whether each clip satisfies the relationship and attributes specified in the detection pattern 16.

[0085] For example, the determination unit 265 executes a Visual Prompt creation process, a format conversion process, and a determination process for all Subjects and Objects shown in the detection pattern 16.

[0086] First, let's explain the Visual Prompt creation process executed by the determination unit 265. Figure 12 is a diagram illustrating the Visual Prompt creation process executed by the determination unit. In Figure 12, as an example, the Subject is set to a person with tracking ID "P2" and the Object to a forklift with tracking ID "F1". The determination unit 265 also uses the tracking information of the person with tracking ID "P2" and the tracking information of the forklift with tracking ID "F1" included in the tracking result information 19.

[0087] The determination unit 265 draws a bounding box in red for the person with tracking ID "P2" in each frame of clip C1 included in clip information 20, based on the tracking information of the person with tracking ID "P2". The determination unit 265 also draws a bounding box in green for the forklift with tracking ID "F1" in each frame of clip C1 included in clip information 20, based on the tracking information of the forklift with tracking ID "F1". For example, the result of drawing the bounding box on clip C1 is denoted as prompt V1.

[0088] The determination unit 265 generates a Visual Prompt 21 by performing the same processing on clip C2 and other clips included in the clip information 20 as it did on clip C1. For example, the result of drawing a bounding box on clip C2 is denoted as prompt V2.

[0089] Next, the conversion process performed by the determination unit 265 will be described. Figure 13 is a diagram illustrating the format conversion process performed by the determination unit. As shown in Figure 13, the determination unit 265 generates a detection prompt 17 by performing a format conversion process on the detection pattern 16. The detection prompt 17 is a prompt to cause the VLM to respond whether the Subject and Object of the detection pattern 16 satisfy the relationships and attributes defined in the detection pattern 16. For example, the detection prompt 17 is a prompt to respond whether the distance between the person surrounded by the red bounding box and the forklift surrounded by the green bounding box is within 1 meter.

[0090] Next, the determination process executed by the determination unit 265 will be described. Figure 14 is a diagram illustrating the determination process executed by the determination unit. For example, the determination unit 265 obtains the determination result 22 by inputting the Visual Prompt 21 and the detection prompt 17 to the VLM 30.

[0091] For example, the determination unit 265 inputs the prompt V1 of the Visual Prompt 21 and the detection prompt 17 to the VLM 30, and based on the determination result 22, it can determine that in clip C1 (frame numbers fr1 to fr10), an event has occurred in which the distance between the person with tracking ID "P2" and the forklift with tracking ID "F1" has become shorter. For example, the determination unit 265 can determine that in clip C1 (frame numbers fr1 to fr10), the relationship between the person with tracking ID "P2" and the forklift with tracking ID "F1" matches the conditions of the interaction information.

[0092] Furthermore, the determination unit 265 inputs the prompt from Visual Prompt 21 and the detection prompt 17 to VLM 30, and based on the determination result 22, it can determine that in clip C1 (frame numbers fr1 to fr10), the event that tracking ID "P2" is wearing a vest has occurred. For example, the determination unit 265 can determine that the attributes of the person with tracking ID "P2" match the conditions of the attribute information.

[0093] The determination unit 265 performs the Visual Prompt creation process, format conversion process, and determination process described above for all Subjects and Objects shown in the detection pattern 16. This allows the determination unit 265 to identify the time period in the video 15 where an event satisfying the "Relationship (interaction information)" defined in the detection pattern 16 occurs for a pair of Subject (e.g., a person) and Object (e.g., a forklift). Furthermore, the determination unit 265 can identify the time period in the video 15 where an event satisfying the "Attribute (attribute information)" defined in the detection pattern 16 occurs for a Subject (e.g., a person).

[0094] In the processing of the determination unit 265 described above, the number of queries to the VLM 30 is "number of clips" × "number of detection patterns" × "number of Subject and Object pairs". For this reason, the following processing may be performed to reduce the computational load.

[0095] Figure 15 is a diagram illustrating other processes of the determination unit. As shown in Figure 15, the determination unit 265 draws bounding boxes in different colors for all Subjects and Objects included in the detection pattern 16 and generates a Visual Prompt 21-1. For example, it draws bounding box b1-1 in "orange", bounding box b1-2 in "red", bounding box b1-3 in "purple", and bounding box b1-4 in "yellow".

[0096] The determination unit 265 creates prompt 17-1 to respond once regarding the presence or absence of all Relationships and Attributes in the detection pattern 16. By having the determination unit 265 perform this process, the number of queries to the VLM 30 can be reduced, thereby reducing the computational load.

[0097] Next, the ASG construction unit 266 in Figure 7 will be described. The ASG construction unit 266 generates an action scene graph 60 based on the judgment result of the judgment unit 265. The action scene graph 60 corresponds to the action scene graph 60 shown in Figure 4.

[0098] An example of the processing of the ASG construction unit 266 will be explained using Figure 4. For example, if the determination result 22 includes the event that the distance between the person with tracking ID "P2" and the forklift with tracking ID "F1" becomes close in clip C1 (frame numbers fr1 to fr10), the ASG construction unit 266 performs the following processing. For convenience, the time corresponding to frame number fr1 is denoted as time T2, and the time corresponding to frame number fr10 is denoted as time T3.

[0099] In this case, the ASG construction unit 266 generates a concrete object node n4-1 corresponding to tracking ID "P2", a concrete object node n4-5 corresponding to tracking ID "F1", and an event node n3-5 corresponding to the interaction information "approaching". The ASG construction unit 266 connects concrete object node n4-1 to event node n3-5, and concrete object node n4-5 to event node n3-5. The ASG construction unit 266 also connects event node n3-5 to time nodes n2-2 and n2-3.

[0100] This allows the ASG construction unit 266 to set in the action scene graph 60 that, during time T2 to T3, an event occurred in which the distance between the person with tracking ID "P2" and the forklift with tracking ID "F1" became shorter.

[0101] Furthermore, if the judgment result 22 includes the fact that in a certain clip, an event has occurred that satisfies the "Attribute (wearing a vest)" for the person with tracking ID "P3," the ASG construction unit 266 performs the following processing. For convenience, the time corresponding to the starting frame number of a certain clip is defined as time T1, and the time corresponding to the ending frame number is defined as time T6.

[0102] In this case, the ASG construction unit 266 generates a concrete object node n4-3 corresponding to the tracking ID "P3" and an event node n3-2 corresponding to the attribute information "wearing a vest". The ASG construction unit 266 connects the concrete object node n4-3 and the event node n3-2. The ASG construction unit 266 also connects the event node n3-2 to the time nodes n2-1 and n2-6.

[0103] This allows the ASG construction unit 266 to set in the behavior scene graph 60 that, during times T1 to T6, an event satisfying the "Attribute (wearing a vest)" has occurred for the person with tracking ID "P3".

[0104] The above describes an example of the process by which the ASG generation unit 252 generates an action scene graph 60 based on the video 15 and the detection pattern 16.

[0105] The ASG generation unit 252 may also acquire video analysis results from other modules or external data and add various types of information to the action scene graph 60.

[0106] For example, the video analysis results are the video analysis results of a person included in video 15, such as the person having poor posture, limping, or making a surprised expression. When the ASG generation unit 252 receives information such as "poor posture" as a video analysis result for the person corresponding to the tracking ID "P2" included in video 15, it associates and records the information "poor posture" with the concrete object node n4-1 associated with the tracking ID "P2" shown in Figure 4.

[0107] Furthermore, the external data is information that associates a person's facial image with the person's employee ID. The ASG generation unit 252 matches the person's facial image with each frame of the video 15 and associates the detected object in each frame with the employee ID. Based on the results of the association between the detected object in each frame and the employee ID, the ASG generation unit 252 associates and records the employee ID with the specific object nodes n4-1 to n4-4 shown in Figure 4.

[0108] By the way, the format of the behavior scene graph 60 generated by the ASG generation unit 252 is not limited to that described in Figure 4. For example, the ASG generation unit 252 may generate a behavior scene graph in the format shown in Figures 16 and 17.

[0109] Figure 16 shows another example (1) of an action scene graph. The action scene graph 60a shown in Figure 16 maintains the relationships between subject, object, relation, and attribute on a frame-by-frame basis. In the action scene graph 60a-1 for a certain frame (frame number fr1), event nodes n3-1, n3-2 and concrete object nodes n4-1 to n4-5 are included. Event nodes n3-1, n3-2 and concrete object nodes n4-1 to n4-5 correspond to event nodes n3-1, n3-2 and concrete object nodes n4-1 to n4-5 in Figure 4.

[0110] Figure 17 shows another example (2) of the action scene graph. The action scene graph 60b shown in Figure 17 has time nodes n2-1, n2-2, and n2-3. The action scene graph 60b has event nodes n3-4 and n3-5. The action scene graph 60b has concrete object nodes n4-1, n4-2, and n4-5. The time nodes n2-1 to n2-3, event nodes n3-4 and n3-5, and concrete object nodes n4-1, n4-2, and n4-5 correspond to the time nodes n2-1 to n2-3, event nodes n3-4 and n3-5, and concrete object nodes n4-1, n4-2, and n4-5 in Figure 4. The connection relationships and meanings of the time nodes, event nodes, and concrete object nodes are the same as those explained in Figure 4.

[0111] The action scene graph 60b further includes a camera node n5 and frame nodes n6-1, n6-2, n6-3, n6-4, n6-5, and n6-6. Camera node n5 is the node corresponding to the camera that captured the video. Frame nodes n6-1 to n6-6 are nodes corresponding to each frame.

[0112] Frame nodes n6-1 to n6-6 are connected to camera node n5, and the frames corresponding to frame nodes n6-1 to n6-6 indicate that they were captured by the camera on camera node n5.

[0113] Frame nodes n6-1 to n6-3 are connected to time node n2-1, and it is shown that the time at which each frame corresponding to frame nodes n6-1 to n6-3 was captured is included in the time "Time-0" of time node n2-1. Frame nodes n6-4 to n6-6 are connected to time node n2-2, and it is shown that the time at which the frames corresponding to frame nodes n6-4 to n6-6 were captured is included in the time "Time-1" of time node n2-2.

[0114] The concrete object node n4-1 is connected to frame nodes n6-1 and n6-4, indicating that the person corresponding to concrete object node n4-1 is included in frame nodes n6-1 to n6-4. The concrete object node n4-2 is connected to frame nodes n6-2 and n6-5, indicating that the person corresponding to concrete object node n4-2 is included in frame nodes n6-2 to n6-5. The concrete object node n4-5 is connected to frame nodes n6-3 and n6-6, indicating that the forklift corresponding to concrete object node n4-1 is included in frame nodes n6-3 to n6-6.

[0115] Although not shown in the diagram, frame nodes n6-1 to n6-6 record information about the coordinates of the bounding boxes contained in the corresponding frames.

[0116] Next, an example of the configuration of the information processing device 200 that performs the above-described process will be explained. Figure 18 is a functional block diagram showing the configuration of the information processing device according to this embodiment 2. As shown in Figure 18, this information processing device 200 has a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0117] The communication unit 210 performs data communication with the camera via the network. For example, the communication unit 210 receives video data from the camera. The video data is the video data 15 described in Figure 8.

[0118] The input unit 220 is an input device that inputs various types of information to the control unit 250 of the information processing device 200.

[0119] The display unit 230 is a display device that displays information output from the control unit 250.

[0120] The memory unit 240 includes a detection pattern 16, an action scene graph 60, and a video buffer 70. The memory unit 240 is a memory or the like.

[0121] Detection pattern 16 is a natural language expression representing the object to be detected, and corresponds to, for example, detection pattern 16 shown in Figure 9.

[0122] The behavior scene graph 60 is graph data generated based on the detection pattern 16 and video data. The explanation of the behavior scene graph 60 is the same as the explanation of the behavior scene graph 60 in Figure 4.

[0123] The video buffer 70 is a buffer for storing video data.

[0124] We will now move on to the explanation of the control unit 250. The control unit 250 has an ASG generation unit 252. The control unit 250 is a CPU, GPU, etc.

[0125] The acquisition unit 251 acquires video data captured by the camera. The acquisition unit 251 stores the video data in the video buffer 70.

[0126] The ASG generation unit 252 generates an action scene graph 60 based on the video data and detection pattern 16 in the video buffer 70. For example, as explained in Figure 7, the ASG generation unit 252 includes a frame division unit 261, an object detection unit 262, a tracking unit 263, a clipping unit 264, a determination unit 265, and an ASG construction unit 266.

[0127] For example, the ASG generation unit 252 detects objects to be detected from each frame of the video by analyzing the video to be analyzed using the detection pattern 16. The ASG generation unit 252 inputs the detection prompt 17 generated from the detection pattern 16 and the Visual Prompt generated from the detection results to the VLM 30 to generate results showing attribute information and interaction information of the detected objects. The ASG generation unit 252 generates an action scene graph 60 from the generated results. Further explanation of the ASG generation unit 252 is the same as the explanation in Figure 7, etc.

[0128] Next, an example of the processing procedure of the information processing device 200 according to this second embodiment will be described. Figure 19 is a flowchart showing the processing procedure of the information processing device according to this second embodiment. As shown in Figure 19, the acquisition unit 251 of the information processing device 200 acquires video data and stores it in the video buffer 70 (step S100).

[0129] The ASG generation unit 252 of the information processing device 200 divides the video data into multiple frames (step S101). The ASG generation unit 252 performs Open-Vocabulary object detection (step S102).

[0130] The ASG generation unit 252 performs tracking (step S103). The ASG generation unit 252 performs clipping (step S104). The ASG generation unit 252 creates a Visual Prompt (step S105).

[0131] The ASG generation unit 252 converts the detection pattern 16 into a detection prompt 17 (step S106). The ASG generation unit 252 inputs the Visual Prompt and the detection prompt 17 to the VLM 30 and determines the attribute information and interaction information of the detected object (step S107).

[0132] The ASG generation unit 252 generates an action scene graph 60 based on the judgment result (step S108).

[0133] Next, the effects of the information processing device 200 according to this second embodiment will be described. The information processing device 200 detects objects to be detected from each frame of the video by analyzing the video to be analyzed using the detection pattern 16. The ASG generation unit 252 inputs the detection prompt 17 generated from the detection pattern 16 and the Visual Prompt generated from the detection results to the VLM 30, thereby generating results that show attribute information and interaction information of the detected objects. The ASG generation unit 252 generates an action scene graph 60 from the generated results. This makes it possible to generate an action scene graph that includes the subject, object, and relationship that the user expects.

[0134] By applying the behavioral scene graph 60 generated by the information processing device 200 to the information processing device 100 of Example 1, it is possible to generate answers that the user desires in response to a question.

[0135] The information processing device 200 can efficiently generate results showing attribute information and interaction information of the object to be detected by inputting the Visual Prompt and the detection prompt 17 to the VLM 30.

[0136] The information processing device 200 acquires external data that associates a person's facial image with the person's employee ID, matches the person's facial image with each frame of the video, and associates the detected object in each frame with the employee ID. This allows useful information to be added to the behavior scene graph 60.

[0137] Next, an example of a computer hardware configuration that realizes the same functions as the information processing device 100 (200) shown in the above embodiment will be described in order.

[0138] Figure 20 shows an example of a computer hardware configuration that realizes similar functions to the information processing apparatus of the embodiment. As shown in Figure 20, the computer 300 has a CPU 301 that performs various calculations, an input device 302 that receives data input from the user, and a display 303. The computer 300 also has a communication device 304 that exchanges data with a camera, external devices, etc. via a wired or wireless network, and an interface device 305. The interface device 305 may have a microphone, speaker, etc. connected to it. The computer 300 also has a RAM 306 that temporarily stores various information and a hard disk drive 307. Each of the devices 301 to 307 is connected to a bus 308.

[0139] The hard disk drive 307 includes an acquisition program 307a, a KG generation program 307b, an ASG generation program 307c, and a graph analysis program 307d. The CPU 301 reads each of the programs 307a to 307d and loads them into the RAM 306.

[0140] The acquisition program 307a functions as the acquisition process 306a. The KG generation program 307b functions as the KG generation process 306b. The ASG generation program 307c functions as the ASG generation process 306c. The graph analysis program 307d functions as the graph analysis process 306d.

[0141] The processing in acquisition process 306a corresponds to the processing in acquisition units 151 and 251. The processing in KG generation process 306b corresponds to the processing in KG generation unit 152. The processing in ASG generation process 306c corresponds to the processing in ASG generation units 153 and 252. The processing in graph analysis process 306d corresponds to the processing in graph analysis unit 154.

[0142] Furthermore, it is not necessary to store each program 307a to 307d in the hard disk drive 307 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 300. Then, the computer 300 may read and execute each program 307a to 307d.

[0143] 50 Knowledge Graph 60 Behavioral Scene Graph 70 Video Buffer 80 Text Table 100 Information Processing Unit 110 Communication Unit 120 Input Unit 130 Display Unit 140 Storage Unit 150 Control Unit 151, 251 Acquisition Unit 152 KG Generation Unit 153, 252 ASG Generation Unit 154 Graph Analysis Unit

Claims

The system acquires input data to which the object to be detected is associated with attribute information of the object or interaction information between the objects. By analyzing the video to be analyzed using the input data, a first object representing the object to be detected is detected from among the video frames that make up the video. By inputting the prompt containing the aforementioned input data and the detected first object into the visual language model, a result is generated that shows attribute information of the first object or interaction information related to the first object. Generate graph data that associates the generated results with the detected first object. A generation program characterized by having a computer perform the processing. 　 The detection process involves detecting the region information of the first object, The generation program according to claim 1 is characterized in that the process for generating the result involves inputting the information obtained by drawing the region information on the video frame and the prompt to the visual language model to generate the result. 　 The generation program according to claim 1, characterized in that the process for generating the graph data generates an ASG (Action Scene Graph) as the graph data. 　 The generation program according to claim 1, characterized in that it obtains external data that associates an image of a person with identification information of the person, identifies a detection target corresponding to the image of the person based on the object to be detected and the image of the person, and further causes the computer to perform a process that associates the detection target with the identification information of the person. 　 The video to be analyzed is analyzed using input data to which the object to be detected is associated with attribute information of the object or interaction information between the objects. Based on the video frames constituting the video, a first object representing the detected object is identified, and graph data is identified which is generated by the visual language model based on the input data, and which associates the attribute information of the first object or interaction information related to the first object with the detected first object. When a user asks a question regarding the first object in the video, the system searches the identified graph data to identify a result that shows the interaction information associated with the first object. Based on the results showing the identified interaction information, the generative AI model generates an answer to the question. A generation program characterized by having a computer perform the processing. 　 The system acquires input data to which the object to be detected is associated with attribute information of the object or interaction information between the objects. By analyzing the video to be analyzed using the input data, a first object representing the object to be detected is detected from among the video frames that make up the video. By inputting the prompt containing the aforementioned input data and the detected first object into the visual language model, a result is generated that shows attribute information of the first object or interaction information related to the first object. Generate graph data that associates the generated results with the detected first object. A generation method characterized by the processing being performed by a computer. 　 The detection process involves detecting the region information of the first object, The generation method according to claim 6, characterized in that the process for generating the result generates the result by inputting the information obtained by drawing the region information on the video frame and the prompt to the visual language model. 　 The generation method according to claim 6, characterized in that the process for generating the graph data generates an ASG (Action Scene Graph) as the graph data. 　 The generation method according to claim 6, characterized in that an external data associating an image of a person with identification information of the person is acquired, a computer is further instructed to identify a detection target corresponding to the image of the person based on the object to be detected and the image of the person, and to associate the detection target with the identification information of the person. 　 The system acquires input data to which the object to be detected is associated with attribute information of the object or interaction information between the objects. By analyzing the video to be analyzed using the input data, a first object representing the object to be detected is detected from among the video frames that make up the video. By inputting the prompt containing the aforementioned input data and the detected first object into the visual language model, a result is generated that shows attribute information of the first object or interaction information related to the first object. Generate graph data that associates the generated results with the detected first object. An information processing device having a control unit that performs processing. 　 The detection process involves detecting the region information of the first object, The information processing apparatus according to claim 10, characterized in that the process for generating the result generates the result by inputting the information obtained by drawing the region information on the video frame, the prompt, and the visual language model. 　 The information processing apparatus according to claim 10, characterized in that the process for generating the graph data generates an ASG (Action Scene Graph) as the graph data. 　 The information processing apparatus according to claim 10, characterized in that it acquires external data that associates an image of a person with identification information of the person, identifies a detection target corresponding to the image of the person based on the object to be detected and the image of the person, and further causes a computer to perform a process that associates the detection target with the identification information of the person.

Images