Information notification program, information notification method, and information processing device.

Abstract

The objective is to generate information regarding appropriate work procedures. [Solution] The information processing device acquires domain knowledge information, including the work procedures of the monitored task. By analyzing the monitored video, the information processing device identifies the image regions of people and the image regions of objects that the people are working on. The information processing device inputs a prompt, composed of the feature quantities of the identified person's image region and the object's image region, along with the domain knowledge information, into a large-scale multimodal model. When the person's actions in the video are not based on the work procedures, the large-scale multimodal model generates information related to the work procedures. The information processing device then notifies the user of the generated information related to the work procedures.

Claims

1. On the computer, We obtain domain knowledge information, including the work procedures of the monitored tasks. By analyzing the monitored video footage, the image regions of the person and the image regions of the object the person is working on are identified within the video footage. By inputting a prompt consisting of the feature quantities of the image region of the identified person and the image region of the object, and the domain knowledge information, into a large-scale multimodal model, when the person's actions in the video are not actions based on the work procedure, the large-scale multimodal model generates information related to the work procedure. Information related to the generated work procedure will be notified. An information notification program characterized by causing a process to be executed.

2. The video of the subject being monitored is acquired, Identify a first region in which a first object is located within a predetermined video frame among a plurality of video frames included in the acquired video, and identify a request relating to the first object that exists in the first region. By analyzing the acquired video, a second object related to the first object located in the first region is identified from among the multiple objects present in each of the multiple video frames. Based on the requirements relating to the first object, and the image features of the first and second objects, a response to the requirements is generated. The information notification program according to claim 1, characterized in that it causes the computer to perform the processing.

3. The process for identifying the first region and the request includes an operation to specify the first region in which the first object is located within the predetermined video frame displayed on the display screen, and a process for receiving a request document from the user relating to the first object in the first region, and identifying the request based on the request document. The process for generating the aforementioned response includes a process for generating a response to the request by inputting a prompt containing the request, the image features of the first object and the second object into a large-scale multimodal model, The response to the aforementioned request is displayed on the screen. The information notification program according to feature 2.

4. When an operation is performed on the agent, the person who performed the operation is identified from the video footage. The computer is instructed to perform a process that targets the identified person and sets up the first area. The agent provides coaching to the person who performed the operation, based on the information related to the work procedure generated by the large-scale multimodal model. The information notification program according to feature 2.

5. The aforementioned acquisition process is, The domain knowledge information, including the work procedures for specific tasks performed in the retail industry, is acquired. The process to be identified is, By analyzing video footage of the work area in the aforementioned retail industry, the image region of the person and the image region of the object being worked on by the person are identified. The aforementioned generation process is, By inputting the prompt, which is composed of the feature quantities of the image region of the person and the feature quantities of the image region of the object, and the domain knowledge information, into the large-scale multimodal model, the large-scale multimodal model generates information related to the work procedure when the person's actions are not based on the work procedure of the specific task. The information notification program according to feature 1.

6. The process to be identified is, By analyzing the video of the work area, the first image region is identified for each task included in the work procedure of the specific task, the second image region is identified for each object that is the work target when each task is performed, and the third image region is identified for each task not included in the specific task. The aforementioned generation process is, By inputting the prompt, which is composed of the feature quantities of each of the first, second, and third image regions, and the domain knowledge information, into the large-scale multimodal model, information is generated regarding the impact that tasks not included in the specific task have on the specific task. The process to be notified is, The large-scale multimodal model outputs information regarding the impact on the specific business operation. The information notification program according to feature 5.

7. The aforementioned generation process is, The technical information is obtained from a storage DB in which technical information, which is specialized knowledge and skills related to the aforementioned specific business, is stored. By generating the prompt using the aforementioned technical information and inputting it into the large-scale multimodal model, coaching information for the person who performed a task not included in the specific task is generated to help them perform the specific task. The process to be notified is, Outputting the aforementioned coaching information, The information notification program according to feature 6.

8. The aforementioned workplace is a kitchen where prepared foods are prepared in a store of the retail industry. The aforementioned specific tasks are cooking tasks performed at the aforementioned cooking location. The aforementioned work procedure is the cooking process until the prepared food is completed. Each of the first image regions is an image region of each operation included in the cooking process, Each of the second image regions is an image region of the prepared food at the location where each of the operations included in the cooking process was performed. The third image region is the image region of the location where the person performed an action not included in the cooking process. The information notification program according to feature 6.

9. The information notification program according to claim 1, wherein the large-scale multimodal model is a neural network trained using a token set obtained by masking some tokens from a plurality of tokens, and the large-scale multimodal model is trained to generate a response to a request when a prompt including image features of a request, a person, and an object is input to the large-scale multimodal model.

10. When a target is given to the AI ​​agent, the AI ​​agent generates tasks to achieve the target, collects information from the storage unit to cause the large-scale multimodal model to execute the generated tasks, and inputs the information collected from the storage unit into the large-scale multimodal model to perform a process that generates information related to the work procedure. The information notification program according to claim 1, characterized in that the information collected from the memory unit is domain knowledge of the work required in the area of ​​the monitored video, and an image in which the image area of ​​a person and the image area of ​​the object that the person is working on are identified.

11. Computers We obtain domain knowledge information, including the work procedures of the monitored tasks. By analyzing the monitored video footage, the image regions of the person and the image regions of the object the person is working on are identified within the video footage. By inputting a prompt consisting of the feature quantities of the image region of the identified person and the image region of the object, and the domain knowledge information, into a large-scale multimodal model, when the person's actions in the video are not actions based on the work procedure, the large-scale multimodal model generates information related to the work procedure. Information related to the generated work procedure will be notified. A method for notifying information, characterized by performing a process.

12. We obtain domain knowledge information, including the work procedures of the monitored tasks. By analyzing the monitored video footage, the image regions of the person and the image regions of the object the person is working on are identified within the video footage. By inputting a prompt consisting of the feature quantities of the image region of the identified person and the image region of the object, and the domain knowledge information, into a large-scale multimodal model, when the person's actions in the video are not actions based on the work procedure, the large-scale multimodal model generates information related to the work procedure. Information related to the generated work procedure will be notified. An information processing device characterized by having a control unit.

Images