Information processing system, information processing method, and information processing program

Abstract

An information processing system (10) according to an embodiment comprises at least one processor. The at least one processor: inputs an image to a detector (25A) that detects an object related to request information; inputs a selection image, which is an image from which the object was detected by the detector (25A), to a language model (LLM (25B)); and outputs generation information relating to the request information and generated by the language model to an output device (17).

Description

Information processing system, information processing method, and information processing program

[0001] An embodiment of the present invention relates to an information processing system, an information processing method, and an information processing program.

[0002] Image recognition tasks using AI (Artificial Intelligence) are currently being performed. However, image recognition tasks using conventional AI can require a huge amount of work to build a judgment algorithm and to annotate and train the network, which has been a barrier to application. While attempts have been made to reduce these work hours by using generative AI, the increased computation time and amount of calculations can be problematic.

[0003] Japanese Patent Application Laid-Open No. 2022-91787

[0004] The present invention has been made in view of the above, and has an object to provide an information processing system, an information processing method, and an information processing program that can improve the efficiency of computational resources.

[0005] The information processing system of the embodiment includes at least one processor, which inputs an image to a detector that detects an object related to requested information, inputs a selected image, which is the image of the object detected by the detector, to a language model, and outputs generated information related to the requested information generated by the language model to an output device.

[0006] FIG. 1 is an explanatory diagram of an example of a system according to an embodiment. FIG. 2 is a sequence diagram showing an example of the flow of setting processing executed by a setting processing unit. FIG. 3 is a sequence diagram showing an example of the flow of generation processing executed by a generation processing unit. FIG. 4 is a flowchart showing an example of the flow of setting processing executed by a setting processing unit of an information processing system. FIG. 5 is a flowchart showing an example of the flow of generation processing executed by an information processing system. FIG. 6A is an explanatory diagram of an example of processing by a generation processing unit. FIG. 6B is an explanatory diagram of an example of processing by a generation processing unit. FIG. 7 is a hardware configuration diagram.

[0007] Hereinafter, embodiments of an information processing system, an information processing method, and an information processing program will be described in detail with reference to the accompanying drawings.

[0008] FIG. 1 is an explanatory diagram of an example of a system 1 according to the present embodiment.

[0009] The system 1 is a system for obtaining predetermined generated information from an image. The system 1 includes an information processing system 10 and an external system 20. The information processing system 10 and the external system 20 are connected to each other so as to be able to communicate with each other via a network NW or the like.

[0010] The external system 20 is one or more information processing devices provided outside the information processing system 10. The external system 20 is configured by one or more dedicated or general-purpose computers.

[0011] The external system 20 includes a communication unit 21, an input unit 22, an output unit 23, a storage unit 24, and a control unit 25. The communication unit 21, the input unit 22, the output unit 23, the storage unit 24, and the control unit 25 are communicatively connected via a bus or the like.

[0012] The communication unit 11 communicates with the information processing system 10 via a network NW or the like. The input unit 12 accepts various operations by a user. The output unit 23 outputs various information. The output unit 23 may be a display that displays various information, a speaker that outputs various sounds, or the like.

[0013] The storage unit 24 stores various types of data. The storage unit 24 is, for example, a random access memory (RAM), a semiconductor memory element such as a flash memory, a hard disk, an optical disk, etc. The storage unit 27 may be a storage device provided outside the external system 20. The storage unit 27 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a local area network (LAN), the internet, etc.

[0014] The control unit 25 executes information processing in the external system 20. The control unit 25 is realized by one or more processors. For example, the control unit 25 may be realized by causing a processor such as a central processing unit (CPU) to execute a program, i.e., by software.

[0015] The control unit 25 includes a detector 25A and a language model. The detector 25A and the language model may be realized by a processor such as a dedicated integrated circuit (IC), i.e., hardware. The above-mentioned units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units. Furthermore, at least one of the detector 25A and the language model may be provided in the information processing system 10 described below.

[0016] The detector 25A is a model that outputs a detection result for input information. In this embodiment, the detector 25A is a model that outputs a detection result for an input image. The detector 25A uses image recognition AI (Artificial Intelligence) that distinguishes and identifies information contained in an image. In this embodiment, the detector 25A detects an object contained in an image. The object is an object to be detected by the detector 25A. The object is also an object related to generated information generated by the information processing system 10, which will be described later. The generated information and the object will be described in detail later.

[0017] The external system 20 includes detectors 25A1 to 25An as the detector 25A, where n is an integer equal to or greater than 2. These multiple detectors 25A1 to 25An differ in at least one of the detection target and the detection method for the detection target.

[0018] For example, a detector 25A uses skeletal information of a person included in an input image to detect whether the image contains a body part of the person to be detected, such as, but not limited to, a wrist, ankle, palm, leg, head, neck, etc.

[0019] Furthermore, for example, a certain detector 25A detects whether or not an object to be detected (for example, a person) is included in the image by detecting an object included in the input image.

[0020] In this way, the external system 20 includes a plurality of types of detectors 25A that differ in at least one of the detection target and the detection method.

[0021] A language model is a generative AI used in the field of natural language processing (NLP) for question answering, sentence generation, etc., by computer programs and systems. Language models can include large language models (LLMs), generative language models (e.g., Generative Pretrained Transformers (GPTs)), representational language models (e.g., Bidirectional Encoder Representations from Transformers (BERTs)), and / or any other type of language model.

[0022] In this embodiment, a form in which the external system 20 includes the LLM 25B as a language model will be described as an example.

[0023] Next, the information processing system 10 will be described.

[0024] The information processing system 10 is one or more information processing devices. The information processing system 10 is configured by one or more dedicated or general-purpose computers.

[0025] The information processing system 10 includes a communication unit 11, an input unit 12, an output unit 13, an image capturing unit 14, a storage unit 15, and a control unit 16. The communication unit 11, the input unit 12, the output unit 13, the image capturing unit 14, the storage unit 15, and the control unit 16 are communicatively connected via a bus or the like. At least one of the external system 20, the output unit 13, and the storage unit 15 functions as an output device 17 to which the control unit 16 outputs various information.

[0026] The communication unit 11 communicates with an external system 20 or the like via a network NW or the like. The input unit 12 accepts various operations by a user. The output unit 13 outputs various information. The output unit 13 is a display that displays various information, a speaker that outputs various sounds, or the like. At least one of the input unit 12 and the output unit 13 may be configured to be provided outside the information processing system 10 and connected to the control unit 16 so as to be able to communicate with it.

[0027] The photographing unit 14 obtains photographed image data by photographing. In the following description, the photographed image data will be simply referred to as an image. The photographing unit 14 sequentially outputs the images photographed in chronological order to the control unit 16. Note that the photographing unit 14 may be provided outside the information processing system 10. In this case, the photographing unit 14 and the control unit 16 may be connected to each other wirelessly or via a wire so as to be able to communicate with each other.

[0028] The storage unit 15 stores various types of data. The storage unit 15 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 15 may be a storage device provided outside the information processing system 10. Also, at least a portion of the information stored in the storage unit 15 may be stored in a storage device provided outside the information processing system 10. Also, the storage unit 15 may be a storage medium that stores or temporarily stores programs and various types of information after downloading them via a LAN (Local Area Network), the Internet, or the like.

[0029] The control unit 16 executes information processing in the information processing system 10 .

[0030] The control unit 16 includes a setting processing unit 16A, a generation processing unit 16B, and an output control unit 16C. The setting processing unit 16A, the generation processing unit 16B, and the output control unit 16C are realized by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0031] Furthermore, at least one of the above-mentioned components included in the control unit 16 may be mounted on an external information processing device such as an external system 20 communicatively connected to the information processing system 10 via a network NW or the like.

[0032] The setting processing unit 16A sets the processing to be executed by the generation processing unit 16B (described later) and executes various setting processes for the processing.

[0033] Although details will be described later, the generation processing unit 16B selects images that satisfy predetermined conditions from among the images sequentially captured by the image capturing unit 14, and executes a process of generating generation information related to the requested information by inputting the selected images to the LLM 25B. The setting processing unit 16A performs various setting processes for the processes executed by this generation processing unit 16B.

[0034] The setting processing unit 16A uses the LLM 25B to perform setting processing such as setting the detector 25A used in the generation processing unit 16B and additional selection processing to be executed by the generation processing unit 16B.

[0035] FIG. 2 is a sequence diagram showing an example of the flow of the setting process executed by the setting processing unit 16A.

[0036] The setting processing unit 16A of the information processing system 10 receives the request information (step S100).

[0037] The request information is information that represents a generation rule for generation information related to an object using an image that the user desires the information processing system 10 to generate. In detail, the request information includes information that represents the object, the content of the generation information, etc.

[0038] The generated information is information relating to the object. In this embodiment, the generated information is generated by processing of the generation processing unit 16B, which will be described later.

[0039] Specifically, for example, the request information is "monitor whether gloves are worn" or "determine that a person is working if they are heading towards the workbench," etc. When the request information is "monitor whether gloves are worn," "gloves" corresponds to the object, and "monitor whether gloves are worn" corresponds to information about the content of the generated information. When the request information is "determine that a person is working if they are heading towards the workbench," "person" corresponds to the object, and "determine that a person is working if they are heading towards the workbench" corresponds to information about the content of the generated information.

[0040] The setting processing unit 16A acquires request information input by a user through an operation instruction on the input unit 12, thereby accepting the request information.

[0041] Based on the request information received in step S100, the setting processing unit 16A generates a second prompt representing an instruction to specify the detector 25A to be used for selecting the selected images that satisfy the predetermined conditions (step S102).

[0042] That is, the setting processing unit 16A generates, as the second prompt, a prompt representing a designation instruction for the detector 25A used to select a selected image from the images in order to obtain generation information related to the object represented by the request information. The setting processing unit 16A may generate the second prompt representing a designation instruction for the detector 25A and the object to be detected by the detector 25A. The prompt is an instruction or input sentence for the AI ​​model. The setting processing unit 16A may generate the second prompt using a known method.

[0043] For example, assume that the request information is "monitoring whether gloves are worn." In this case, the setting processing unit 16A generates a prompt indicating, "To monitor whether gloves are worn, narrow down the image frames. Output an appropriate detector and detection class to find frames in which gloves are worn," as a second prompt indicating an instruction to the detector 25A used to obtain generation information according to the request information.

[0044] The setting processing unit 16A inputs the generated second prompt to the LLM 25B (step S104). The LLM 25B processes the input second prompt to determine the detector 25A to be used for selecting selected images that satisfy predetermined conditions based on the request information and the object to be detected by the detector 25A (step S106).

[0045] That is, by processing the second prompt, the setting processing unit 16A determines an appropriate detector 25A to use for selecting the selected image based on the request information from among the multiple detectors 25A1 to 25An. Furthermore, by processing the second prompt, the setting processing unit 16A determines the object to be detected by the determined detector 25A. The detection object may be referred to as a detection class. Note that the determined object to be detected by the detector 25A may match an object such as a "glove" directly represented by the request information, but it may also be a related object such as a "wrist" indirectly represented by the request information. For example, the determined object to be detected by the detector 25A may be a "wrist" on which the "glove" is worn, directly represented by the request information "monitoring glove wear." That is, by processing the input second prompt, the LLM 25B determines the detector 25A that detects the object related to the request information and the object to be detected by the detector 25A.

[0046] The setting processor 16A of the information processing system 10 receives the determination result, which is the processing result of the second prompt, from the LLM 25B (step S108). The determination result includes information indicating the determined detector 25A and the object to be detected by the detector 25A.

[0047] For example, assume that the request information is "monitoring the wearing of gloves." In this case, for example, the LLM 25B determines that the object to be detected is a wrist, and that the detector 25A that detects whether the wrist, which is the object to be detected, is included in the image using skeletal information of a person included in the image is the detector 25A to be used by the generation processing unit 16B. Also assume that the request information is "determine that a person is working if they are facing a workbench." In this case, for example, the LLM 25B determines that the object to be detected is a person, and that the detector 25A that detects a person included in the image is the detector 25A to be used by the generation processing unit 16B.

[0048] The setting processing unit 16A of the information processing system 10 sets the detector 25A and the object to be detected by the detector 25A, which are represented by the determination result received from the LLM 25B, as the detector 25A and the object of the detector 25A to be used in the generation processing unit 16B (step S110). That is, the setting processing unit 16A sets the detector 25A and the object determined by the LLM 25B based on the second prompt as the detector 25A and the object to be used in the generation processing unit 16B to acquire a selected image.

[0049] If the LLM 25B cannot determine an appropriate detector 25A according to the request information, the setting processing unit 16A may set the detector 25A and the object specified by the user operating the input unit 12 as the detector 25A and the object to be used by the generation processing unit 16B. In this case, the setting processing unit 16A may generate interactive second prompts based on the request information and input them sequentially to the LLM 25B, causing the LLM 25B to interactively re-determine an appropriate detector 25A according to the request information.

[0050] Furthermore, the setting processing unit 16A of the information processing system 10 may accept additional condition information (step S112). The additional condition information is information that indicates additional conditions that are set when an image that satisfies additional conditions is selected as a selected image from among images of the object detected by the detector 25A. For example, the setting processing unit 16A accepts the additional condition information by acquiring the additional condition information input by the user through an operation instruction of the input unit 12.

[0051] The additional condition information may be information that indicates the conditions for acquiring, as selected images, images that satisfy additional conditions from among the images of the object detected by the detector 25A.

[0052] For example, assume that the detector 25A determined in step S106 is a detector 25A that detects whether the wrist, which is the detection target, is included in the image using skeletal information of a person included in the image. In this case, the additional condition information is, for example, information indicating that "the y coordinate of the right wrist included in the image is greater than the right shoulder." Furthermore, the additional condition information is, for example, information indicating that "the position of the wrist included in the image is located in the upper half of the image." Furthermore, assume that the detector 25A determined in step S106 is a detector 25A that detects a person, which is the detection target, in the image. In this case, the additional condition information is, for example, information indicating that "the size of the person included in the image is 10 pixels or more."

[0053] The setting processing unit 16A generates a third prompt representing a setting instruction for additional screening processing for the image in accordance with the additional condition information received in step S112 (step S114), and inputs the third prompt to the LLM 25B (step S116). The setting processing unit 16A may generate the third prompt using a known method.

[0054] The LLM 25B processes the input third prompt to generate an additional selection process, which is a judgment program for selecting images that satisfy the additional conditions represented by the additional condition information (step S118).

[0055] The setting processor 16A of the information processing system 10 receives the additional selection process, which is the processing result of the third prompt, from the LLM 25B (step S120). The setting processor 16A of the information processing system 10 sets the additional selection process received from the LLM 25B in the processing program executed by the generation processor 16B (step S122).

[0056] If the LLM 25B is unable to generate an additional screening process by processing the third prompt, the setting processing unit 16A may set the additional screening process set by the user's operation of the input unit 12 in the processing program executed by the generation processing unit 16B. In this case, the setting processing unit 16A may generate an interactive third prompt and sequentially input the third prompt to the LLM 25B, thereby causing the LLM 25B to generate the additional screening process again in an interactive manner.

[0057] Then, this sequence ends.

[0058] By performing these processes, the setting processing unit 16A sets the detector 25A and object (detection target) used by the generation processing unit 16B to select images, as well as the additional selection process to be performed by the generation processing unit 16B, in the processing program of the generation processing unit 16B using LLM25B.

[0059] Returning to FIG. 1, the explanation will be continued.

[0060] The generation processing unit 16B acquires, as selected images, images that satisfy predetermined conditions from the images sequentially captured by the imaging unit 14, and executes processing to generate generated information regarding the object included in the selected images. In detail, the generation processing unit 16B regards, as selected images, images of the object detected by the detector 25A set by the setting processing unit 16A, and obtains generated information by inputting the selected images to the LLM 25B.

[0061] In addition, the generation processing unit 16B may obtain generation information by selecting an image that is an image of an object detected by a detector 25A set by the setting processing unit 16A and that satisfies the additional conditions represented by the additional selection processing set by the setting processing unit 16A as a selected image, and inputting the selected image into the LLM 25B.

[0062] In addition, the generation processing unit 16B may obtain generation information by selecting an image that is an image of an object detected by a detector 25A set by the setting processing unit 16A, and that has been further processed to satisfy the additional conditions represented by the additional selection processing set by the setting processing unit 16A, and inputting the selected image into the LLM 25B.

[0063] The output control unit 16C outputs the generated information to the output device 17. The output device 17 is at least one of the output unit 13, the storage unit 15, and the external system 20. For example, the output control unit 16C outputs the generated information to the output unit 13, such as a display, by outputting the generated information to the output unit 13. The output control unit 16C also outputs the generated information by storing the generated information in the storage unit 15. The output control unit 16C may also output the generated information by transmitting the generated information to an external information processing system, such as the external system 20.

[0064] The processing by the generation processing unit 16B and the output control unit 16C will be described in detail.

[0065] 3 is a sequence diagram showing an example of the flow of the generation process executed by the generation processing unit 16 B. The control unit 16 executes the processes of steps S200 to S216 each time one image (one frame) is acquired from the imaging unit 14.

[0066] The generation processing unit 16B acquires an image from the photographing unit 14 (step 200). The generation processing unit 16B may acquire the image by reading an image photographed by the photographing unit 14 and stored in the storage unit 15. Alternatively, the generation processing unit 16B may acquire the image by reading an image photographed by the photographing unit 14 and stored in an external information processing device such as the external system 20.

[0067] The generation processing unit 16B inputs the image acquired in step S200 to the detector 25A determined by the LLM 25B by processing the second prompt (step S202).

[0068] The detector 25A outputs a detection result indicating the presence or absence of an object in the input image to the information processing system 10 (step S204). For example, assume that the detector 25A configured by the setting processing unit 16A is a detector 25A that uses skeletal information of a person included in the image to detect whether the image contains a wrist, which is the detection target. In this case, the detector 25A detects the wrist using the skeletal information of the person included in the image input in step S202, and outputs a detection result indicating whether the wrist is included in the image. Also, assume that the detector 25A configured by the setting processing unit 16A is a detector 25A that detects a person as the detection target object and detects the person included in the image. In this case, the detector 25A detects the person included in the image input in step S202, and outputs a detection result indicating whether the person is included in the image.

[0069] Next, the generation processing unit 16B of the information processing system 10 acquires, as a selected image, an image acquired in step S200 in which the detection result by the detector 25A indicates the detection of an object. For example, the generation processing unit 16B acquires, as the selected image, an image in which a wrist or a person is detected.

[0070] In addition, the generation processing unit 16B of the information processing system 10 may further perform additional selection processing set by the setting processing unit 16A on the image acquired in step S200, where the detection result by the detector 25A represents the detection of an object (step S206).

[0071] Assume that the additional selection process set by the setting processing unit 16A indicates that an image that satisfies a predetermined condition is to be acquired as a selected image. In this case, if the generation processing unit 16B determines that the image acquired in step S200, which is an image in which the detection result by the detector 25A indicates the detection of an object, satisfies the predetermined condition, the generation processing unit 16B acquires the image as a selected image.

[0072] Also, assume that the additional screening process set by the setting processing unit 16A is the additional screening process set by the LLM 25B. In this case, the generation processing unit 16B acquires, as a selected image, an image selected by the additional screening process for the image acquired in step S200, in which the detection result by the detector 25A indicates the detection of an object.

[0073] For example, assume that the additional selection process set by the LLM 25B is a process of determining whether "the y coordinate of the right wrist included in the image is greater than the right shoulder." In this case, the generation processing unit 16B acquires, as a selected image, an image acquired in step S200 in which the detection result by the detector 25A indicates the detection of the wrist as the target object, and which has been determined to have "the y coordinate of the right wrist included in the image greater than the right shoulder."

[0074] Also, for example, assume that the additional selection process set by the LLM 25B is a process of determining whether or not the position of a wrist included in an image is located in the upper half of the image. In this case, the generation processing unit 16B acquires, as selected images, images acquired in step S200 in which the detection result by the detector 25A indicates the detection of a wrist as the target object, and which have been determined to be located in the upper half of the image.

[0075] Also, for example, assume that the additional selection process set by the LLM 25B is a process of determining whether or not the size of a person included in an image is 10 pixels or more. In this case, the generation processing unit 16B acquires, as selected images, images acquired in step S200 in which the detection result by the detector 25A indicates the detection of a person as a target, and which have been determined to have the size of a person included in the image as 10 pixels or more.

[0076] Furthermore, the generation processing unit 16B of the information processing system 10 may further perform a predetermined processing operation on the image acquired as the selected image by the additional selection processing in step S206 (step S208). The processing operation may be, for example, a process of trimming an area including an object included in the image. The generation processing unit 16B then acquires the image after the processing operation, for example, the trimmed area included in the image, as the selected image. This processing operation may also be set in the LLM 25B, as in the additional selection processing.

[0077] Then, the generation processing unit 16B inputs the selected image and the first prompt to the LLM 25B (step 210).

[0078] The first prompt is a prompt that represents an instruction to generate generation information related to the requested information. That is, the first prompt is a prompt that represents an instruction to generate generation information related to an object included in the selected image. The generation processing unit 16B generates the first prompt, for example, by a known method based on the requested information.

[0079] For example, assume that the request information is "monitoring glove wear." Also assume that the detector 25A determined by the LLM 25B is a detector 25A that detects a human body part, such as a wrist, as a detection target (object). Also assume that the object represented by the request information is an object worn on the wrist, for example, a glove. In this case, the generation processing unit 16B generates, as the first prompt, a prompt representing an instruction to generate generation information regarding whether or not the target glove included in the selected image is worn on the target wrist. In other words, the generation processing unit 16B generates, as the first prompt, a prompt representing an instruction to generate generation information representing a determination result of whether or not a glove is worn on the wrist of a person included in the selected image.

[0080] Also, for example, assume that the request information is "determine that a person is working if they are facing the workbench." Also, assume that the detector 25A determined by the LLM 25B is a detector 25A that detects a person as a detection target (object). Also, assume that the object represented by the request information is a person. In this case, the generation processing unit 16B generates, as the first prompt, a prompt representing an instruction to generate generated information related to the operation of the person, which is the object, included in the selected image. Specifically, the generation processing unit 16B generates, as the first prompt, a prompt representing an instruction to generate generated information related to the operation of the person, which is the object. Even more specifically, the generation processing unit 16B generates, as the first prompt, a prompt representing an instruction to generate generated information representing the determination result of whether the person, which is included in the selected image, is facing the workbench.

[0081] The generation processing unit 16B then inputs the first prompt and the selected image to the LLM 25B (step 210). The LLM 25B processes the first prompt to generate generated information from the selected image.

[0082] For example, in response to the first prompt, LLM25B generates information such as information representing the determination result of whether gloves are worn on the wrist of a person included in the selected image, or information representing the determination result of whether a person included in the selected image is facing a workbench (step S212).

[0083] The generation processing unit 16B of the information processing system 10 receives the generated information, which is the processing result of the first prompt, from the LLM 25B (step S214). The output control unit 16C of the information processing system 10 outputs the generated information received in step S214 to the output device 17 (step S216). Then, this sequence ends.

[0084] Next, an example of the flow of the setting process executed by the information processing system 10 of this embodiment will be described.

[0085] FIG. 4 is a flowchart showing an example of the flow of the setting process executed by the setting processing unit 16A of the information processing system 10.

[0086] The setting processing unit 16A receives request information from the input unit 12 or the like (step S300). Based on the request information received in step S300, the setting processing unit 16A generates a second prompt representing an instruction to specify the detector 25A to be used to select selected images that satisfy predetermined conditions (step S302). The setting processing unit 16A inputs the second prompt generated in step S302 to the LLM 25B (step S304). The setting processing unit 16A then obtains from the LLM 25B a determination result determined by the LLM 25B, including information representing the detector 25A and the object to be detected by the detector 25A (step S306).

[0087] The setting processing unit 16A sets the detector 25A and the object to be detected by the detector 25A represented by the judgment result received from the LLM 25B as the detector 25A and the object to be detected by the detector 25A to be used in the generation processing unit 16B (step S308).

[0088] The setting processing unit 16A receives additional condition information from the input unit 12 or the like (step S310). The setting processing unit 16A creates a third prompt representing a setting instruction for additional screening processing for images according to the additional condition information received in step S310 (step S312), and inputs the third prompt to the LLM 25B (step S314).

[0089] The setting processing unit 16A then obtains the additional selection process, which is the processing result of the third prompt, from the LLM 25B (step S316), and sets it in the processing program to be executed by the generation processing unit 16B (step S318).Then, this routine ends.

[0090] Next, an example of the flow of the generation information generation process executed by the information processing system 10 will be described.

[0091] FIG. 5 is a flowchart showing an example of the flow of the generation process executed by the information processing system 10.

[0092] The generation processing unit 16B acquires an image from the photographing unit 14 (step 400).

[0093] The generation processing unit 16B inputs the acquired image to the detector 25A determined by the LLM 25B by processing the second prompt (step S402). The detector 25A outputs the detection result of the presence or absence of an object in the input image to the information processing system 10. The generation processing unit 16B acquires the detection result from the detector 25A (step S404).

[0094] The setting processing unit 16A determines whether the detection result acquired in step S404 indicates the detection of an object that is the detection target of the detector 25A (step S406). If the setting processing unit 16A determines that the detection result indicates the detection of an object (step S406: Yes), the process proceeds to step S408.

[0095] In step S408, the generation processing unit 16B executes the additional selection process set by the LLM 25B through the processing of the setting processing unit 16A on the image for which a positive determination was made in step S406 (step S408).

[0096] Then, the generation processing unit 16B determines whether the determination result of the additional selection process in step S408 satisfies the determination condition for the additional selection process (step S410). If the determination in step S410 is affirmative (step S410: Yes), the process proceeds to step S412.

[0097] Next, the generation processing unit 16B further performs a predetermined processing on the image for which a positive determination was made in step S410 (step S412). The processing is, for example, processing such as trimming an area including an object included in the image. The generation processing unit 16B then acquires the image after the processing, for example, the trimmed area included in the image, as a selected image.

[0098] Then, the generation processing unit 16B acquires the image after the processing processing in step S412 as a selected image (step S414).

[0099] Next, the generation processing unit 16B inputs the selected image and the first prompt acquired in step S414 to the LLM 25B (step S416).

[0100] The generation processing unit 16B then acquires the generated information generated from the selected image by processing the first prompt from the LLM 25B (step S418). The output control unit 16C then outputs the generated information acquired in step S418 to the output device 17 (step S420).

[0101] The control unit 16 determines whether to end the process (step S422). For example, the control unit 16 executes the determination of step S422 by determining whether a signal indicating the end of the process has been received, such as by a user operating the input unit 12. If the determination in step S422 is affirmative (step S422: Yes), the control unit 16 ends this routine. If the determination in step S422 is negative (step S422: No), the control unit 16 proceeds to step S426.

[0102] On the other hand, if a negative determination is made in step S406 (step S406: No), the process proceeds to step S424. Also, if a negative determination is made in step S410 (step S410: No), the process proceeds to step S424. In step S424, the generation processing unit 16B excludes the acquired image from the generation information (step S424), and the process proceeds to step S426. In step S426, the generation processing unit 16B acquires the next unprocessed image from the imaging unit 14 (step S426), and the process returns to step S402.

[0103] A specific example will be described in which the setting processing unit 16A and the generation processing unit 16B execute the processes of steps S300 to S318 and steps S400 to S426, respectively.

[0104] For example, it is assumed that the request information is "monitoring the wearing of gloves."

[0105] In this case, for example, based on the second prompt, the LLM 25B determines that the object to be detected is a wrist, and determines, using skeletal information of a person included in the image, the detector 25A that detects whether the image contains the wrist as the detection object, as the detector 25A to be used by the generation processing unit 16B. Then, the setting processing unit 16A sets the detector 25A and the object "wrist" determined by the LLM 25B based on the second prompt as the detector 25A and the object to be used in the generation processing unit 16B to acquire a selected image.

[0106] In addition, it is assumed that the additional selection process generated by the LLM 25B based on the third prompt corresponding to the additional condition information is a process for determining whether or not the position of the wrist included in the image is in the lower half of the image. In this case, the setting processing unit 16A sets the additional selection process in the generation processing unit 16B.

[0107] Then, each time a new image is acquired, the generation processing unit 16B inputs the acquired image into the detector 25A, and obtains from the detector 25A a detection result as to whether or not the image contains a wrist, which is detected using skeletal information of the person contained in the input image.

[0108] For images determined to include a wrist, the generation processing unit 16B further executes a preset additional selection process, determining whether the wrist in the image is located in the lower half of the image. Then, for images determined to include a wrist in the upper half of the image, the generation processing unit 16B further executes a predetermined processing process. Assume that the processing process is a process of trimming an area including an object in the image. In this case, the generation processing unit 16B acquires the image after the processing process, for example, the trimmed area in the image, as a selected image.

[0109] The generation processing unit 16B then inputs the selected image and the first prompt to the LLM 25B, thereby obtaining, as an output from the LLM 25B, generation information representing the determination result as to whether or not a glove is worn on the wrist of a person included in the selected image. The output control unit 16C outputs this generation information to the output device 17.

[0110] 6A and 6B are explanatory diagrams of an example of processing by the generation processing unit 16B.

[0111] For example, as shown in Fig. 6A, the generation processing unit 16B inputs an image 30 to the detector 25A, and determines whether the image 30 includes a wrist, which is the object F relative to the glove T. The generation processing unit 16B then performs the additional selection process on the image 30 to determine whether the wrist is located in the lower half of the image. Then, as shown in Fig. 6B, the generation processing unit 16B acquires, as a selected image 32, an area E obtained by trimming an area including the wrist, which is the object F, from the image 30 that includes the wrist and is located in the lower half of the image 30.

[0112] The generation processing unit 16B may acquire at least the image 30 that is determined to include the object F by inputting the image 30 to the detector 25A as the selected image 32. Furthermore, the generation processing unit 16B may acquire the image 30 that satisfies the conditions for additional selection through additional selection processing as the selected image 32. Furthermore, the generation processing unit 16B may acquire the image 30 that has been further processed as the selected image 32.

[0113] Also, for example, it is assumed that the request information is "determine that work is in progress if a person is facing the workbench."

[0114] In this case, for example, the LLM 25B determines, based on the second prompt, that the object to be detected is a person, and determines the detector 25A that detects the person included in the image as the detector 25A to be used by the generation processing unit 16B. Then, the setting processing unit 16A sets the detector 25A and the object "person" determined by the LLM 25B based on the second prompt as the detector 25A and the object to be used in the generation processing unit 16B to acquire the selected image.

[0115] Also, assume that the additional selection process generated by the LLM 25B based on the third prompt corresponding to the additional condition information is a process for determining whether the size of a person included in an image is 10 pixels or more. In this case, the setting processing unit 16A sets the additional selection process in the generation processing unit 16B.

[0116] Then, each time a new image is acquired, the generation processing unit 16B inputs the acquired image into the detector 25A, and obtains a detection result from the detector 25A as to whether or not a person is included in the input image.

[0117] For images determined to contain a person, the generation processing unit 16B further executes a preset additional selection process, determining whether the size of the person contained in the image is 10 pixels or more. Then, for images determined to contain a person whose size is 10 pixels or more, the generation processing unit 16B further executes a predetermined processing process. Assume that the processing process is a process of trimming an area containing an object contained in the image. In this case, the generation processing unit 16B acquires the image after the processing process, for example, the trimmed area contained in the image, as a selected image.

[0118] The generation processing unit 16B then inputs the selected image and the first prompt to the LLM 25B, and obtains, as an output from the LLM 25B, generation information representing a determination result that a person included in the selected image is working if the person is facing a workbench. The output control unit 16C outputs this generation information to the output device 17.

[0119] As described above, the information processing system 10 of this embodiment includes at least one processor. The at least one processor inputs an image to the detector 25A that detects an object related to the requested information, inputs a selected image, which is an image of the object detected by the detector 25A, to the language model (LLM 25B), and outputs generated information related to the requested information generated by the language model to the output device 17.

[0120] In this way, the processor of the information processing system 10 of this embodiment does not directly input the acquired image to the language model, but inputs a selected image, which is an image of an object detected by the detector 25A, from one or more images to the language model. Then, the processor outputs generated information related to the requested information, which is generated by the language model, to the output device 17.

[0121] That is, the information processing system 10 of this embodiment uses a language model such as LLM25B in combination with a detector 25A, and generates generated information by selectively inputting only limited images in which the object is detected by the detector 25A into the language model, rather than inputting all images into the language model.

[0122] Therefore, the information processing system 10 of this embodiment can suppress an increase in the calculation time and amount due to the construction of a determination algorithm and the annotation and learning of a network, thereby making efficient use of calculation resources.

[0123] Therefore, the information processing system 10 of this embodiment can improve the efficiency of computational resources.

[0124] Furthermore, the information processing system 10 of this embodiment can reduce the total calculation time and amount of calculations for a series of processes for outputting generated information from an image, as well as shorten the calculation time, by improving the efficiency of calculation resources.

[0125] Furthermore, the information processing system 10 of this embodiment can provide generated information generated from an image without requiring the user to perform complicated setting processes, etc. Therefore, by using the generated information, the user can easily and quickly check the shooting environment, such as the site where the image was taken, and can also take improvement activities starting from the shooting environment.

[0126] At least one processor of the information processing system 10 inputs the selected image and a first prompt representing an instruction to generate generation information related to the requested information to the language model, and outputs the generation information generated by the language model based on the selected image and the first prompt to the output device 17.

[0127] Therefore, the information processing system 10 of this embodiment can specify an instruction to generate desired generation information represented by the first prompt to the language model.

[0128] The language model is a large-scale language model (LLM25B). By using the LLM25B as the language model, it is possible to provide generated information with higher accuracy.

[0129] In addition, at least one processor of the information processing system 10 of this embodiment inputs a second prompt representing a judgment instruction for the detector 25A to the language model based on the request information, and sets the detector 25A judged by the language model based on the second prompt as the detector 25A to be used for acquiring the selected image.

[0130] Therefore, the information processing system 10 of this embodiment can determine the optimal detector 25A using a language model without requiring detailed setting operations by the user, and can use this as the detector 25A to select the selected images.

[0131] Furthermore, at least one processor of the information processing system 10 of this embodiment inputs a second prompt, which represents an instruction to determine the detector 25A and the object to be detected by the detector 25A, to the language model based on the request information. Then, the processor sets the detector 25A and the object determined by the language model based on the second prompt as the detector 25A and the object to be used in acquiring the selected image.

[0132] Therefore, the information processing system 10 of this embodiment allows the language model to determine the optimal detector 25A and object (detection target, detection class) without requiring detailed setting operations by the user, and can use them as the detector 25A and object to be used for selecting the selected image.

[0133] Furthermore, at least one processor of the information processing system 10 of this embodiment acquires, as selected images, images that satisfy predetermined conditions among images of the object detected by the detector 25A.

[0134] Therefore, the information processing system 10 of this embodiment can generate generated information by selectively inputting only a further limited number of images into the language model, thereby further improving the efficiency of computational resources.

[0135] Furthermore, at least one processor of the information processing system 10 of this embodiment inputs a third prompt representing an instruction to set additional screening processing for the image to the language model, and executes the additional screening processing set by the language model.

[0136] Therefore, the information processing system 10 of this embodiment can generate generated information by selectively inputting only images further limited by the additional selection process set by the language model into the language model, thereby making it possible to further improve the efficiency of computational resources.

[0137] Furthermore, at least one processor of the information processing system 10 of this embodiment acquires, as a selected image, a processed image obtained by processing the image of the object detected by the detector 25A.

[0138] Therefore, the information processing system 10 of this embodiment can generate generated information by selectively inputting only a further limited number of images into the language model, thereby further improving the efficiency of computational resources.

[0139] In the present embodiment, the generation processing unit 16B processes each image (frame) captured by the image capturing unit 14. However, the generation processing unit 16B may acquire a plurality of images, and among the acquired images, identify an image in which an object is detected by the detector 25A and which satisfies the discrimination conditions of the additional selection process as a selection target, and input the image to the LLM 25B together with the first prompt.

[0140] The application of the information processing system 10 of this embodiment is not limited. For example, the information processing system 10 of this embodiment is applied to a system that monitors the status of people using captured images of people inside a building such as a factory or a house. Note that the information processing system 10 can be applied to any system that handles images, and its application is not limited to a system that monitors the status of people.

[0141] Next, an example of the hardware configuration of the information processing system 10 and the external system 20 of the above embodiment will be described.

[0142] FIG. 7 is a diagram illustrating an example of a hardware configuration of the information processing system 10 and the external system 20 according to the above embodiment.

[0143] The information processing system 10 and the external system 20 of the above embodiment have a hardware configuration that utilizes a conventional computer, in which a CPU (Central Processing Unit) 80, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 84, an I / F 86, etc. are interconnected by a bus 88.

[0144] The CPU 80 is a computing device that controls the information processing system 10 and the external system 20 of the above embodiment. The ROM 82 stores programs and the like that realize information processing by the CPU 80. The RAM 84 stores data necessary for various processes by the CPU 80. The I / F 86 is an interface that is connected to a storage unit, an input unit, an output unit, a sensor, a communication unit, and the like, and is used to send and receive data.

[0145] In the information processing system 10 and the external system 20 of the above embodiment, the CPU 80 reads a program from the ROM 82 onto the RAM 84 and executes it, thereby realizing each of the above functional units on the computer.

[0146] The programs for executing the above processes executed in the information processing system 10 and the external system 20 of the above embodiment may be stored in a hard disk drive (HDD). Also, the programs for executing the above processes executed in the information processing system 10 and the external system 20 of the above embodiment may be provided by being pre-installed in the ROM 82.

[0147] Furthermore, the programs for executing the above processes executed by the information processing system 10 and the external system 20 of the above embodiments may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) and provided as a computer program product. Furthermore, the programs for executing the above information processes executed by the information processing system 10 and the external system 20 of the above embodiments may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. Furthermore, the programs for executing the above information processes executed by the information processing system 10 and the external system 20 of the above embodiments may be provided or distributed via a network such as the Internet.

[0148] Although the above describes an embodiment, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0149] Note that the present technology can also be configured as follows. (1) An information processing system including at least one processor, wherein the at least one processor inputs an image to a detector that detects an object related to request information, inputs a selected image, which is the image of the object detected by the detector, to a language model, and outputs generated information related to the request information generated by the language model to an output device. (2) The information processing system described in (1), wherein the at least one processor inputs the selected image and a first prompt representing an instruction to generate the generated information related to the request information to the language model, and outputs the generated information generated by the language model based on the selected image and the first prompt to the output device. (3) The information processing system described in (1) or (2), wherein the language model is a large-scale language model. (4) The information processing system according to any one of (1) to (3), wherein the at least one processor inputs a second prompt representing an instruction for determination of the detector to the language model based on the request information, and sets the detector determined by the language model based on the second prompt as the detector to be used for acquiring the selected image. (5) The information processing system according to any one of (1) to (4), wherein the at least one processor inputs a second prompt representing an instruction for determination of the detector and the object to be detected by the detector to the language model based on the request information, and sets the detector and the object determined by the language model based on the second prompt as the detector and the object to be used for acquiring the selected image. (6) The information processing system according to any one of (1) to (5), wherein the at least one processor acquires, from among the images in which the object is detected by the detector, an image that satisfies a predetermined condition as the selected image. (7) The information processing system according to any one of (1) to (6), wherein the at least one processor inputs a third prompt representing an instruction to set an additional selection process for the image to the language model, and executes the additional selection process set by the language model.(8) The information processing system according to any one of (1) to (7), wherein the at least one processor acquires, as the selected image, a processed image obtained by processing the image in which the object is detected by the detector. (9) The information processing system according to any one of (1) to (8), wherein the detector detects a body part of the person that is the accessory using skeletal information of the person included in the image. (10) The information processing system according to any one of (1) to (9), wherein the detector detects an object that is the accessory included in the image. (11) The information processing system according to any one of (2) to (10), wherein the object is a body part, and the first prompt represents an instruction to generate the generated information regarding the presence or absence of an article of wear for the body part included in the selected image. (12) The information processing system according to any one of (2) to (11), wherein the object is a person, and the first prompt represents an instruction to generate the generated information related to an operation of the person. (13) An information processing method executed by an information processing system, comprising: inputting an image to a detector that detects an object related to request information, inputting a selected image, which is the image of the object detected by the detector, to a language model, and outputting the generated information related to the request information generated by the language model to an output device. (14) An information processing program for causing a computer to input an image to a detector that detects an object related to request information, inputting the selected image, which is the image of the object detected by the detector, to a language model, and outputting the generated information related to the request information generated by the language model to an output device.

[0150] 10 Information processing system 16 Control unit 17 Output device 25A Detector 25B LLM

Claims

1. An information processing system comprising at least one processor, wherein the at least one processor inputs an image to a detector that detects an object related to requested information, inputs a selected image, which is the image of the object detected by the detector, to a language model, and outputs generated information related to the requested information generated by the language model to an output device.

2. The information processing system of claim 1, wherein the at least one processor inputs the selected image and a first prompt representing an instruction to generate the generated information related to the requested information into the language model, and outputs the generated information generated by the language model based on the selected image and the first prompt to the output device.

3. The information processing system according to claim 1, wherein the language model is a large-scale language model.

4. The information processing system of claim 1, wherein the at least one processor inputs a second prompt representing a determination instruction for the detector to the language model based on the request information, and sets the detector determined by the language model based on the second prompt as the detector to be used for acquiring the selected image.

5. The information processing system of claim 1, wherein the at least one processor inputs a second prompt representing an instruction to determine the detector and the object detected by the detector to the language model based on the request information, and sets the detector and the object determined by the language model based on the second prompt as the detector and the object to be used in acquiring the selected image.

6. The information processing system according to claim 1, wherein the at least one processor acquires, as the selected image, an image that satisfies a predetermined condition from among the images in which the object is detected by the detector.

7. The information processing system of claim 1, wherein the at least one processor inputs a third prompt representing an instruction to set an additional selection process for the image to the language model, and executes the additional selection process set by the language model.

8. The information processing system according to claim 1, wherein the at least one processor acquires, as the selected image, a processed image obtained by processing the image in which the object is detected by the detector.

9. The information processing system according to claim 1, wherein the detector detects body parts of the person, which is the target object, using skeletal information of the person included in the image.

10. The information processing system according to claim 1, wherein the detector detects an object that is the target object and is included in the image.

11. The information processing system of claim 2, wherein the object is a body part, and the first prompt represents an instruction to generate the generated information regarding the presence or absence of clothing for the body part included in the selected image.

12. The information processing system according to claim 2, wherein the object is a person, and the first prompt represents an instruction to generate the information regarding the person's activities.

13. An information processing method executed by an information processing system, comprising: inputting an image into a detector that detects an object related to requested information; inputting a selected image, which is the image of the object detected by the detector, into a language model; and outputting generated information related to the requested information generated by the language model to an output device.

14. An information processing program for causing a computer to input an image into a detector that detects an object related to requested information, input a selected image, which is the image of the object detected by the detector, into a language model, and output generated information related to the requested information generated by the language model to an output device.

Images