Information processing method, information processing device, and information processing program

Abstract

This information processing method includes: acquiring a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies an object included in image data in accordance with a text prompt; modifying the first prompt into a second prompt by adding a second identification class, which is different from a first identification class defined in the first prompt, to the first identification class; acquiring an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and outputting the identification result.

Description

Information processing method, information processing device, and information processing program

[0001] The present disclosure relates to a technique for recognizing an object contained in image data.

[0002] Vision and Language Models (VLMs) are known that receive image data and text prompts as input and perform captioning or the like on the image data. For example, Patent Literature 1 discloses a technique for setting a bounding box on an image using a VLM. When a user specifies a class of object to be identified in a text prompt, for example, the VLM identifies objects that correspond to the specified class from the input image data and outputs the identification result.

[0003] However, when the number of classes specified in the text prompt is small, the VLM is more likely to generate false positives, in which an object belonging to a class different from the specified class is identified as belonging to the specified class.

[0004] US Patent Application Publication No. 2023 / 0154213

[0005] The present disclosure aims to provide a technology that can prevent an AI model from outputting false positive identification results.

[0006] An information processing method in one aspect of the present disclosure is an information processing method executed by a computer, and includes obtaining a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies an object included in image data according to the text prompt; modifying the first prompt to a second prompt by adding a second identification class different from the first identification class, which is the identification class defined in the first prompt; obtaining an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and outputting the identification result.

[0007] According to the present disclosure, it is possible to prevent the AI ​​model from outputting false positive identification results.

[0008] 7 is a block diagram showing an example of the configuration of an information processing system according to embodiment 1 of the present disclosure. FIG. 8 is a flowchart showing an example of processing of an information processing device according to embodiment 1 of the present disclosure. FIG. 9 is a diagram showing an input screen of a first example according to embodiment 1 of the present disclosure. FIG. 10 is a diagram showing an input screen of a second example according to embodiment 1 of the present disclosure. FIG. 11 is a block diagram showing an example of the configuration of an information processing system according to embodiment 2 of the present disclosure. FIG. 12 is a flowchart showing an example of processing of an information processing device according to embodiment 2 of the present disclosure. FIG. 13 is a flowchart showing details of the correction processing shown in FIG. 7. FIG. 14 is a block diagram showing an example of the configuration of an information processing system according to embodiment 3 of the present disclosure. FIG. 15 is a diagram showing an input screen of a first example and an input screen of a second example according to embodiment 3 of the present disclosure. FIG. 16 is a flowchart showing a first example of processing of an information processing device according to embodiment 3 of the present disclosure. FIG. 17 is a flowchart showing details of a first prompt acquisition process. FIG. 18 is a diagram showing an example of processing of an information processing device according to embodiment 3 of the present disclosure. FIG. 19 is a diagram showing an example of processing of a first prompt acquisition process. FIG. 10 is a diagram showing an input screen according to a fourth embodiment of the present disclosure. FIG. 11 is a block diagram showing an example of the configuration of an information processing system according to a fifth embodiment of the present disclosure. FIG. 12 is a flowchart showing an example of processing by the information processing system according to the fifth embodiment of the present disclosure. FIG. 13 is a block diagram showing an example of the configuration of an information processing system according to a sixth embodiment of the present disclosure. FIG. 14 is a flowchart showing an example of processing by an information processing device according to the sixth embodiment of the present disclosure. FIG. 15 is a block diagram showing an example of the configuration of an information processing system according to a seventh embodiment of the present disclosure. FIG. 16 is a flowchart showing an example of processing by an information processing device according to the seventh embodiment of the present disclosure.

[0009] (Findings underlying the present disclosure) Because VLMs are trained using a large amount of training data, they can respond to new classification classes specified in text prompts without retraining for the new classification classes. Furthermore, VLMs can specify various tasks in text prompts, eliminating the need to switch models for each task. For example, if segmentation is specified as the task in the text prompt, the VLM outputs classification results that identify objects at the contour level. Furthermore, if object detection is specified as the task in the text prompt, the VLM outputs classification results that enclose objects in bounding boxes. In this way, VLMs can respond very flexibly to user needs.

[0010] However, if the number of classification classes specified in the text prompt is small, the likelihood of outputting a false positive classification result increases. This is because, when the number of classification classes specified is small, the VLM attempts to classify various objects into the specified classification classes as much as possible. Furthermore, the technology of Patent Document 1 merely discloses setting a pseudo-bounding box for image data using a VLM, and is unable to suppress the false positives.

[0011] Therefore, the inventor discovered that if an identification class different from the identification class specified in the text prompt is added to increase the number of identification classes, the possibility of the VLM outputting a false positive identification result can be reduced, leading to the present disclosure.

[0012] (1) An information processing method in one aspect of the present disclosure is an information processing method executed by a computer, and includes: obtaining a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies an object included in image data according to the text prompt; modifying the first prompt to a second prompt by adding a second identification class different from the first identification class, which is the identification class defined in the first prompt; obtaining an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and outputting the identification result.

[0013] This configuration generates a second prompt by adding a second identification class different from the first identification class to the first identification class defined in the first prompt, and inputs the generated second prompt to the first AI model. This allows the first AI model to identify an object that should originally be identified as the second identification class as the second identification class without forcing it to be identified as the first identification class. As a result, this configuration can prevent false positive identification results from being obtained for the first identification class.

[0014] (2) In the information processing method described in (1) above, the second identification class may be a class defined by a user.

[0015] This arrangement allows the second identification class to be defined by the user.

[0016] (3) In the information processing method described in (1) or (2) above, the first identification class may be a class defined by a user.

[0017] This arrangement allows the first identification class to be defined by the user.

[0018] (4) In the information processing method described in any of (1) to (3) above, the modifying may include presenting a plurality of candidate identification classes and determining a candidate identification class selected by a user from the plurality of candidate identification classes as the second identification class.

[0019] According to this configuration, the user can select the second identification class from among a plurality of candidate identification classes, and therefore the user can easily select the second identification class.

[0020] (5) In the information processing method described in any one of (1) to (4) above, the second identification class may be a predefined class that is defined in advance.

[0021] This configuration can automatically determine the second identification class without requiring the user to input the second identification class, thereby eliminating the need to accept a second prompt from the user and reducing the number of processing steps in the information processing system 1A.

[0022] (6) In the information processing method described in any one of (1) to (5) above, the second identification class may include a plurality of predefined classes that have been defined in advance, and the modifying may include calculating the similarity between the first identification class and each of the plurality of predefined classes, identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity, and if the similar predefined class is identified, replacing the similar predefined class with the first identification class, and generating the text prompt including the replaced first identification class and the remaining predefined classes as the second prompt.

[0023] According to this configuration, if there is a predefined class similar to the first identification class, the predefined class is replaced with the first identification class, thereby preventing an object that should be identified as the first identification class from being identified as the predefined class, thereby reducing false positives.

[0024] (7) In the information processing method described in any one of (1) to (6) above, the second identification class may include a plurality of predefined classes that have been predefined, and the modifying may include calculating the similarity between the first identification class and each of the plurality of predefined classes that have been predefined, identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity, and if the similar predefined class cannot be identified, generating the text prompt in which the plurality of predefined classes have been added to the first identification class as the second prompt.

[0025] According to this configuration, if there is no predefined class similar to the first identification class, the predefined class is added to the first identification class as is to generate the second prompt, thereby expanding the identification classes included in the second prompt and reducing false positives.

[0026] (8) In the information processing method described in any of (1) to (7) above, obtaining the first prompt may include obtaining the first prompt by inputting the target image data into a second AI model that explains the content of the image data in natural language.

[0027] When it is difficult for a user to name an object contained in the target image data and to explain the situation of the target image data, the user may be unable to come up with an appropriate prompt. According to this configuration, the first prompt is automatically generated by the second AI model. Therefore, even if the user is unable to come up with an appropriate first prompt, an appropriate first prompt can be generated.

[0028] (9) In the information processing method described in (8) above, obtaining the first prompt may include extracting a plurality of areas from the target image data and inputting the extracted plurality of areas into the second AI model to obtain the first prompt.

[0029] When a single target image data piece contains multiple objects, the second AI model may not be able to generate an appropriate description that identifies each of the multiple objects. According to this configuration, the target image data piece is divided into multiple regions, which increases the likelihood that each region contains one object, allowing the second AI model to generate an appropriate first prompt.

[0030] (10) In the information processing method described in (8) above, cutting out multiple areas from the target image data may include detecting multiple objects from the target image data and setting an area for each of the detected objects.

[0031] With this configuration, one area is set for each of multiple objects, allowing the second AI model to generate an appropriate first prompt.

[0032] (11) In the information processing method described in any one of (8) to (10) above, the second AI model is a model that outputs an explanation of the image data in accordance with a third prompt, and the third prompt may include a question in natural language indicating at least one of the size of the object and the display state of the object.

[0033] When the target image data includes a small object or when an occlusion occurs in which multiple objects overlap, the second AI model may not be able to generate an appropriate first prompt. According to this configuration, the third prompt indicating at least one of the size and display mode of the object is input to the second learning model, so that the second AI model can generate an appropriate first prompt even when the target image data includes a small object or an occlusion occurs.

[0034] (12) In the information processing method described in any of (1) to (11) above, the correction may further include presenting to the user the recognition result when the second prompt and sample image data are input into the first AI model, obtaining a selection instruction from the user as to whether to return the second prompt to the first prompt, and switching the text prompt input into the first AI model to the first prompt or the second prompt in accordance with the selection instruction.

[0035] According to this configuration, the user can select whether to use the first prompt or the second prompt based on the classification result of the sample image data when the second prompt is used. Therefore, if the classification result of the second prompt does not meet the user's intention, the user can perform classification processing on the target image data using the first prompt. Therefore, this configuration can realize classification processing that meets the user's intention.

[0036] (13) In the information processing method described in any of (1) to (12) above, obtaining the identification result may include the computer transmitting the second prompt and the target image data to an external computer equipped with the first AI model, and the external computer inputting the second prompt and the target image data into the first AI model to obtain the identification result, and transmitting the obtained identification result to the computer.

[0037] The classification process using the first AI model requires a higher processing load than the process of correcting the first prompt. With this configuration, the AI ​​process requiring a higher processing load is executed by an external computer, thereby reducing the processing load on the computer.

[0038] (14) In the information processing method described in any of (1) to (13) above, the second identification class may include a plurality of predefined classes that have been defined in advance, and the modifying may include calculating the similarity between the first identification class and each of the plurality of predefined classes and identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity, and the identifying may include, if the similar predefined class can be identified, inputting the first prompt into the first AI model instead of the second prompt.

[0039] According to this configuration, if there is a predefined class similar to the first identification class, the first prompt is input into the first AI model instead of the second prompt, thereby preventing the addition of the second identification class from actually decreasing the identification accuracy of the first identification class.

[0040] (15) In the information processing method described in any one of (1) to (14) above, the second identification class may include a plurality of predefined classes that have been defined in advance, and the modifying may include calculating the similarity between the first identification class and each of the plurality of predefined classes, identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity, and if the similar predefined class is identified, deleting the similar predefined class from the second identification class, and generating the text prompt as the second prompt, the text prompt including the first identification class and the second identification class from which the similar predefined class has been deleted.

[0041] According to this configuration, if there is a predefined class similar to the first identification class, the predefined class is omitted from the second prompt, so that an object that should be identified as the first identification class is prevented from being identified as the predefined class, thereby reducing false positives.

[0042] (16) In the information processing method described in any of (1) to (15) above, the correcting may include identifying a first word corresponding to the first discrimination class from a database that stores a correspondence between a first word and a second word that co-occurs with the first word, and identifying the second discrimination class based on the second word that corresponds to the identified first word.

[0043] According to this configuration, an identification class that co-occurs with the first identification class is identified as the second identification class, thereby reducing the possibility that an object that should be identified as the first identification class will be identified as the second identification class, thereby reducing false positives.

[0044] The present disclosure can also be realized as an information processing program that causes a computer to execute each characteristic configuration included in such an information processing method, or as an information processing system that operates by this information processing program. Needless to say, such a computer program can be distributed on a computer-readable non-transitory recording medium such as a CD-ROM or via a communication network such as the Internet.

[0045] (17) An information processing device in another aspect of the present disclosure is an information processing device including a processor, wherein the processor includes: obtaining a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies an object included in image data according to a text prompt; modifying the first prompt to a second prompt by adding a second identification class different from the first identification class, which is the identification class defined in the first prompt; obtaining an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and outputting the identification result.

[0046] (18) In another aspect of the present disclosure, an information processing program causes a computer to perform the following: obtain a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies an object included in image data according to the text prompt; modify the first prompt into a second prompt by adding a second identification class different from the first identification class, which is the identification class defined in the first prompt; obtain an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and output the identification result.

[0047] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined.

[0048] 1 is a block diagram showing an example of a configuration of an information processing system 1 according to a first embodiment of the present disclosure. The information processing system 1 is a system that modifies a first prompt input by a user into a second prompt and causes a first AI (artificial intelligence) model 122 to execute a classification process for target image data using the second prompt.

[0049] The information processing system 1 includes an information processing device 10, an image input device 20, an input device 30, and a display device 40. The information processing device 10 is a computer such as a personal computer, a portable computer, a cloud server, or an edge server. When the information processing device 10 is configured as a server such as a cloud server or an edge server, the information processing device 10 is connected to the image input device 20, the input device 30, and the display device 40 so as to be able to communicate with each other via a communication network. The communication network is, for example, a wide area communication network including the Internet and a mobile phone communication network, or a local area network.

[0050] The image input device 20 is an information source of target image data, which is image data that is the subject of classification processing. The image input device 20 may be configured, for example, as a computer such as a web server or a personal computer, or as an imaging device such as a camera. The imaging device may be, for example, a portable computer such as a smartphone or a tablet computer, or a digital camera. The imaging device may also be a video camera that performs fixed-point observation. The target image data may be stored in the memory 12 of the information processing device 10. Classification processing refers to processing that identifies a classification class to which an object included in the target image data belongs. A classification class refers to a label or category in a classification problem. Examples of classification classes are "dog," "cat," "bird," etc.

[0051] The input device 30 is a computer equipped with input devices such as a keyboard, a mouse, and a touch panel. The input device 30 may be, for example, a computer such as a portable computer or a desktop computer. When the information processing device 10 is a personal computer, the input device 30 may be composed of input devices such as a keyboard and a mouse. The input device 30 accepts various operations from a user.

[0052] The display device 40 is a display device including an organic EL panel or the like. The display device 40 may be configured as a display device provided in a portable computer or the like, or may be configured as a display device in a desktop computer. The display device 40 displays the identification result of the identification process performed by the information processing device 10.

[0053] The information processing device 10 includes a processor 11 and a memory 12. The processor 11 is configured with a processor such as a central processing unit (CPU). The processor 11 includes an image acquisition unit 111, a first prompt acquisition unit 112, a correction unit 113, an identification unit 114, and an output unit 115. The image acquisition unit 111 to the output unit 115 are realized by the processor 11 executing an information processing program stored in the memory 12. Alternatively, the image acquisition unit 111 to the output unit 115 may be configured with dedicated hardware circuits.

[0054] The image acquisition unit 111 acquires target image data from the image input device 20. For example, the image acquisition unit 111 may acquire the target image data from the image input device 20 using a communication interface (not shown). The target image data may be still image data or image data of each frame of moving image data.

[0055] The first prompt acquisition unit 112 acquires a first prompt, which is a text prompt that defines an identification class (hereinafter referred to as a first identification class) for the first AI model 122. The first AI model 122 is configured, for example, by a generative AI model such as a trained vision and language model that identifies an object included in image data according to the text prompt.

[0056] In the first embodiment, the first identification class is defined by the user, and the first prompt acquisition unit 112 acquires the first prompt from the input device 30 using, for example, a communication interface (not shown).

[0057] The first prompt defines the first discrimination class using natural language input by the user using the input device 30. An example of the description format of the first prompt is a dictionary format such as {discrimination class: text}. The first prompt can also define a task using natural language. Examples of tasks include segmentation and object detection. An example of the description format of the first prompt including the discrimination class and task is {discrimination class: dog, discrimination class: cat, task: object detection}. Note that the text specified as the discrimination class and task may be a sentence or a phrase. An example of text specifying the discrimination class is "the man in white shirt."

[0058] The modifying unit 113 modifies the first prompt to a second prompt by adding a second identification class that is different from the first identification class, which is the identification class defined in the first prompt.

[0059] The second identification class is a class defined by the user. For example, the user may define the identification class of an object expected to be included in a dataset of multiple target image data as the second identification class. In this case, the correction unit 113 may acquire the second identification class input by the user using the input device 30. Then, the correction unit 113 may generate the second prompt by adding the second identification class to the first identification class.

[0060] Alternatively, the correction unit 113 may present a plurality of candidate identification classes and determine a candidate identification class selected by the user from the plurality of candidate identification classes as the second identification class. The candidate identification class may be an identification class defined in advance by the user. For example, the user may visually confirm the identification classes contained in all target image data or one or more selected target image data, list the confirmed identification classes, and delete duplicate identification classes from the listed identification classes to determine the candidate identification class.

[0061] Alternatively, the candidate discrimination classes may be discrimination classes determined based on the clustering results for the target image data. For example, all or part of the target image data is input into a classifier to determine which cluster each piece of target image data falls into. The user then determines representative image data for each cluster from among the classified image data. The user visually checks the discrimination classes contained in the representative image data for each cluster, lists the identified discrimination classes, and deletes any overlapping discrimination classes from the list to determine the candidate discrimination classes.

[0062] The classification unit 114 obtains a classification result for the first classification class by inputting the second prompt and the target image data into the first AI model 122. An example of the classification result when the first classification class is dog may be target image data in which the dog area is surrounded by a bounding box or target image data in which the dog area is filled in along the outline. The classification result may include a degree of confidence in the classification result.

[0063] The output unit 115 outputs the identification result. For example, the output unit 115 may output the identification result to the display device 40 via a communication interface (not shown).

[0064] The memory 12 is configured as a non-volatile rewritable storage device such as a solid state drive (SSD), etc. The memory 12 includes a candidate discrimination class storage unit 121. The memory 12 stores a first AI model 122.

[0065] The candidate identification class storage unit 121 stores a plurality of candidate identification classes that the modifying unit 113 presents to the user.

[0066] The information processing device 10 may be configured as an edge computer integrated with the input device 30 and the display device 40. Also, some functions from the image acquisition unit 111 to the output unit 115 may be distributed to at least one of the input device 30 and the display device 40. The first AI model 122 may be stored in an external server connected to the information processing device 10 via a communication interface (not shown). In this case, the identification unit 114 may transmit the second prompt to the external server via the communication interface (not shown) and obtain the identification result from the external server.

[0067] FIG. 2 is a flowchart showing an example of processing by the information processing device 10 according to the first embodiment of the present disclosure.

[0068] (Step S1) The first prompt acquisition unit 112 acquires a first prompt input by the user from the input device 30. The first prompt defines a first identification class that the user wishes to identify. The number of first identification classes may be one or more.

[0069] (Step S2) Correction unit 113 corrects the first prompt to a second prompt. Fig. 3 is a diagram showing a first example of input screen 200 according to the first embodiment of the present disclosure. Input screen 200 is displayed on display device 40. In input screen 200 of the first example, the user can freely input the second identification class.

[0070] The input screen 200 includes an input field 201. The input field 201 accepts input of a second identification class by the user. For example, the user inputs an object that is expected to be included in the dataset of the target image data as the second identification class. In this example, "car," "road," and the like are input in the input field 201 as the second identification class.

[0071] On the other hand, in this example, the first prompt 210 includes two discrimination classes, "person" and "sky." Therefore, the corrector 113 generates a text prompt in which discrimination classes such as "car" and "road" are added to the first prompt 210 as the second prompt 220. The second prompt 220 includes a discrimination class in which the second discrimination class is added to the first discrimination class, and the task defined in the first prompt 210.

[0072] 4 is a diagram showing a second example input screen 300 according to the first embodiment of the present disclosure. The input screen 300 is displayed on the display device 40. On the second example input screen 300, the user can freely select a second identification class from among a plurality of candidate identification classes. For example, the user selects, from among the plurality of candidate identification classes, an object that is expected to be included in the dataset of the target image data as the second identification class.

[0073] The input screen 300 includes a plurality of selection buttons 301. The plurality of selection buttons 301 correspond to a plurality of candidate identification classes, respectively. In this example, the input screen 300 displays selection buttons 301 corresponding to "car," "sea," "road," and "house." Note that the input screen 300 displays four selection buttons 301, but may display five or more selection buttons 301, or may display three or fewer selection buttons 301. If the input screen 300 cannot display all candidate identification classes at once, it may display a switching button (not shown). Each time the switching button is pressed, the input screen 300 displays a selection button 301 corresponding to a candidate identification class that was previously hidden. This allows the user to check all candidate identification classes.

[0074] In this example, the candidate identification classes "car" and "road" are selected from among the candidate identification classes "car," "sea," etc. Therefore, in the second prompt 220, identification classes including "car" and "road" are added in addition to the "person" and "sky" defined in the first prompt 210.

[0075] (Step S3) The identification unit 114 inputs the second prompt 220 and the target image data acquired by the image acquisition unit 111 into the first AI model 122 to obtain an identification result for the first identification class. The first AI model 122 identifies the first and second identification classes included in the second prompt 220 from the target image data and outputs an identification result for each identification class. Of these identification results, the identification unit 114 obtains the identification results for the first identification class, "person" and "sky," as the final identification result for the target image data. Specifically, the identification unit 114 obtains the target image data in which the areas of "person" and "sky" are surrounded by bounding boxes or the like as the identification result. The identification unit 114 inputs all target image data to the first AI model 122 to obtain an identification result for each of all target image data.

[0076] (Step S4) The output unit 115 outputs the classification result to the display device 40. As a result, the display device 40 displays the classification result. The display device 40 may sequentially display the classification results of all the target image data in response to a user instruction.

[0077] In this way, information processing device 10 generates second prompt 220 by adding a second identification class different from the first identification class to the first identification class defined in the first prompt, and inputs the generated second prompt 220 to first AI model 122. This allows first AI model 122 to identify an object that should originally be identified as the second identification class as the second identification class without forcing it to be identified as the first identification class. As a result, this configuration can prevent false positive identification results from being obtained for the first identification class.

[0078] Second Embodiment An information processing system 1A according to a second embodiment employs a predefined class defined in advance as the second defined class. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0079] 5 is a block diagram showing an example of the configuration of an information processing system 1A according to a second embodiment of the present disclosure. The memory 12A of the information processing device 10A further includes a predefined class storage unit 123. The predefined class storage unit 123 stores a group of predefined classes. The group of predefined classes includes a plurality of predefined classes defined in advance on the information processing device 10A side. The predefined classes are identification classes used in general object identification processing. An example of an identification class used in general object identification processing is an identification class used in AI competitions that perform object identification. The candidate identification class storage unit 121 is omitted from the memory 12A.

[0080] 6 is an explanatory diagram of processing in the second embodiment of the present disclosure. In this example, first prompt 210 has two classes, "person" and "sky," defined as first identification classes. Predefined class group 230 includes predefined classes such as "person," "car," and "road." Correction unit 113A generates second prompt 220 by adding predefined classes defined in predefined class group 230 to the first identification classes. In this example, "person" is included in both the first identification class and predefined class group 230, so correction unit 113A deletes "person" from predefined class group 230 and then generates second prompt 220.

[0081] 7 is a flowchart showing an example of the process of information processing device 10A according to Embodiment 2 of the present disclosure. Steps S11, S13, and S14 are the same as steps S1, S3, and S4, and therefore description thereof will be omitted.

[0082] (Step S12) The correction unit 113A executes a correction process, the details of which will be described later with reference to FIG.

[0083] FIG. 8 is a flowchart showing the details of the correction process shown in FIG.

[0084] (Step S101) The correction unit 113A searches for a subject in text defining a first identification class (hereinafter referred to as target text). For example, if the target text is "the man wears a white shirt," the correction unit 113A identifies the first noun "man" as the subject. The correction unit 113A uses morphological analysis or the like to classify the text defining the first identification class into words and parts of speech, determines whether the target text is a sentence, and if it is a sentence, identifies the first noun as the subject. The correction unit 113A determines that the target text is a sentence if the text contains a subject and a verb, and determines that it is not a sentence if the text does not contain a verb.

[0085] (Step S102) If the target text has a subject (YES in step S102), the correction unit 113A proceeds to step S104. On the other hand, if the target text does not have a subject (NO in step S102), the correction unit 113A proceeds to step S103.

[0086] (Step S103) The correction unit 113A detects words that can be the subject from the target text. For example, the correction unit 113A may detect the first noun that appears in the target text and proper nouns, pronouns, personal pronouns, etc. included in the target text as the subjects. Note that if the target text includes a compound noun, the correction unit 113A may delete the compound noun from the subject candidates. For example, if the target text is "the man in the white shirt," the first noun "man" is identified as the subject word.

[0087] (Step S104) The correction unit 113A calculates the similarity between words indicating the subject or entity (hereinafter referred to as main words) included in the target text and each predefined class. For example, if the main word is "man" and multiple predefined classes are [person, car, dog, sky], the correction unit 113A calculates the similarity between the main word "man" and each of the predefined classes "person," "car," "dog," and "sky." The correction unit 113A calculates word vectors between the main word and each predefined class. For example, word2vec can be used as the word vector. For example, cosine similarity or L2 distance can be used as the similarity. For example, vectors including multiple feature amounts output from multiple nodes constituting an intermediate layer of a recognition model used in region segmentation, object detection, etc. can be used as the word vector.

[0088] (Step S105) Correction unit 113A determines whether there is a predefined class whose similarity is equal to or greater than a threshold. The threshold may be, for example, 0.7, 0.8, 0.9, or the like. If there is a predefined class whose similarity is equal to or greater than the threshold (YES in step S105), the process proceeds to step S106. If there is no predefined class whose similarity is equal to or greater than the threshold (NO in step S105), the process proceeds to step S107.

[0089] (Step S106) Correction unit 113A replaces the predefined class whose similarity is equal to or greater than a threshold with the target text. For example, if "person" is the predefined class whose similarity to "man" is equal to or greater than a threshold among [person, car, dog, sky], correction unit 113A replaces the predefined class "person" with the target text "the man in the white shirt." Therefore, correction unit 113A generates a second prompt 220 that defines "class: the man in the white shirt, class: car, class: dog, class: sky" as first and second identification classes. Note that if there are multiple predefined classes whose similarity is equal to or greater than a threshold, correction unit 113A may replace the predefined class with the greatest similarity with the target text.

[0090] (Step S107) Correction unit 113A generates second prompt 220 by adding predefined class group 230 to the target text. For example, correction unit 113A generates second prompt 220 that defines "class: the man in the white shirt, class: person, class: car, class: dog, class: sky" as the first and second discrimination classes. When the processing of step S106 or step S107 ends, the processing returns to FIG. 7.

[0091] In this way, according to the information processing system 1A of the second embodiment, the second prompt 220 can be generated automatically without requiring the user to input the second identification class. This eliminates the need for the user to input the second prompt 220, thereby reducing the number of processing steps in the information processing system 1A. If there is a predefined class similar to the first identification class, the predefined class is replaced with the first identification class, thereby preventing an object that should be identified as the first identification class from being identified as the predefined class. As a result, false positives can be reduced.

[0092] 7, the correction process (step S12) is performed according to the flowchart shown in FIG. 8. However, the present disclosure is not limited to this. In a modification of the second embodiment, correction unit 113A may generate second prompt 220 by simply adding a predefined class to the target text.

[0093] (Embodiment 3) In embodiment 3, the first prompt 210 is generated by the second AI model 124. In embodiment 3, the same components as those in embodiment 1 are given the same reference numerals and will not be described again.

[0094] 9 is a block diagram showing an example of the configuration of an information processing system 1B according to the third embodiment of the present disclosure. The memory 12B of the information processing device 10B stores a second AI model 124 in addition to a candidate identification class storage unit 121, a predefined class storage unit 123, and a first AI model 122. The second AI model 124 is an AI that explains the contents of image data in natural language. An example of the second AI model 124 is a generative AI model such as a vision and language model. Note that the second AI model 124 may also serve as the first AI model 122.

[0095] The first prompt acquisition unit 112B acquires the first prompt 210 by inputting the target image data acquired by the image acquisition unit 111 into the second AI model 124.

[0096] 10 is a diagram illustrating a first example of an input screen 810 and a second example of an input screen 820 according to the third embodiment of the present disclosure. The first prompt acquisition unit 112B causes the display device 40 to display the input screen 810 or the input screen 820.

[0097] The input screen 810 includes an image display field 811, a prompt selection button 812, and an image analysis selection button 813. The image display field 811 displays target image data. The prompt selection button 812 is selected by the user when the user wishes to input the first prompt 210 themselves. Hereinafter, a prompt input by the user is referred to as a user prompt. The image analysis selection button 813 is selected by the user when the user wishes to have the second AI model 124 generate the first prompt 210.

[0098] The input screen 820 has the same configuration as the input screen 810, except that the prompt selection button 812 and the image analysis selection button 813 are replaced with square buttons instead of radio buttons, namely a prompt selection button 822 and an image analysis selection button 823.

[0099] 11 is a flowchart showing a first example of processing by the information processing device 10B according to the third embodiment of the present disclosure. The first example of the third embodiment causes the second AI model 124 to generate the first prompt 210 without requiring the user to select whether or not to use a user prompt. Therefore, when the first example of the third embodiment is adopted, the first prompt acquisition unit 112B does not cause the input screens 810 and 820 to be displayed on the display device 40. Steps S32, S33, and S34 are the same as steps S12, S13, and S14 in FIG. 7 , and therefore will not be described here.

[0100] (Step S31) First prompt acquisition unit 112B executes a process for acquiring first prompt 210. Details of the process for acquiring first prompt 210 will be described later with reference to FIG.

[0101] 12 is a flowchart showing a second example of processing by information processing device 10B according to embodiment 3 of the present disclosure. The second example of embodiment 3 allows the user to select whether or not to use a user prompt. Therefore, when the second example of embodiment 3 is employed, first prompt acquisition unit 112B displays input screen 810 or input screen 820 on display device 40. Steps S43, S44, S45, and S46 are the same as steps S11, S12, S13, and S14 of FIG. 7 , and therefore will not be described here.

[0102] (Step S41) The first prompt acquisition unit 112B acquires a selection instruction received from the user by the input device 30 and determines, based on the acquired selection instruction, whether to use a user prompt as the first prompt 210. For example, if the prompt selection button 812 is selected on the input screen 810, or if the prompt selection button 822 is selected on the input screen 820, the first prompt acquisition unit 112B determines that a user prompt will be used. On the other hand, if the image analysis selection button 813 is selected on the input screen 810, or if the image analysis selection button 823 is selected on the input screen 820, the first prompt acquisition unit 112B determines that a user prompt will not be used. If a user prompt will be used (YES in step S41), the process proceeds to step S43. If a user prompt will not be used (NO in step S41), the process proceeds to step S42.

[0103] (Step S42) First prompt acquisition unit 112B performs processing to acquire first prompt 210. Details of the processing to acquire first prompt 210 will be described later with reference to FIG.

[0104] FIG. 13 is a flowchart showing the details of the process for obtaining the first prompt 210.

[0105] (Step S201) The image acquisition unit 111 acquires target image data from the image input device 20.

[0106] (Step S202) The first prompt acquisition unit 112B acquires an explanatory text for the target image data by inputting the target image data to the second AI model 124. At this time, the first prompt acquisition unit 112B may input a third prompt to the second AI model 124 along with the target image data. The third prompt is a prompt that provides an instruction to the second AI model 124. The third prompt includes, for example, an instruction to output an explanatory text including a class of an object included in the target image data. The explanatory text output from the second AI model 124 includes, for example, a class indicating an object included in the target image data.

[0107] (Step S203) The first prompt acquisition unit 112B generates a first prompt 210 from the explanatory text. For example, the first prompt acquisition unit 112B may generate the first prompt 210 by representing the classes (first identification classes) included in the explanatory text in a dictionary format. The first prompt acquisition unit 112B may also include a task in the first prompt 210. In this case, the first prompt 210 includes, for example, a prompt that defines a predetermined task. For example, the predetermined task is the above-mentioned segmentation or object detection.

[0108] When step S203 is completed, the process returns to FIG. 11 or FIG.

[0109] FIG. 14 is a diagram illustrating an example of processing in the information processing device 10B according to the third embodiment of the present disclosure. The first prompt acquisition unit 112B inputs target image data 400 to the second AI model 124. The second AI model 124 outputs a description 240 of the target image data 400. In this example, "the man in shirt" is output as the description 240. The first prompt acquisition unit 112B converts the description 240 into a dictionary format such as {identification class: the man in shirt} to generate the first prompt 210. The correction unit 113B corrects the first prompt 210 using the same method as in the second embodiment to generate the second prompt 220. That is, the correction unit 113B generates the second prompt 220 by adding a predefined class group 230 to the first identification class or by replacing one predefined class in the predefined class group 230 with the first identification class. In this example, the similarity between {the man in shirt} and the predefined class {person} is greater than or equal to a threshold, so a second prompt 220 is generated in which {person} is replaced with {the man in shirt}.

[0110] Note that first prompt acquisition unit 112B may generate second prompt 220 using the technique of the first embodiment.

[0111] When it is difficult to explain the class included in the target image data 400 and the situation for the target image data 400, the user may not be able to come up with an appropriate prompt. According to the third embodiment, the first prompt 210 is automatically generated by the second AI model 124. Therefore, even if the user is unable to come up with an appropriate first prompt 210, an appropriate first prompt 210 can be generated.

[0112] 15 is an explanatory diagram of processing in Modification 1 of Embodiment 3 of the present disclosure. In Modification 1, first prompt acquirer 112B randomly cuts out multiple regions 410 from target image data 400. For example, first prompt acquirer 112B may randomly determine the number of divisions and the ratio of the vertical to horizontal lengths (aspect ratio) of each divided region 410.

[0113] 15 , the target image data 400 includes a person 401 and a car 402. The first prompt acquisition unit 112B randomly determines the number of divisions of the target image data 400 to be 2. The first prompt acquisition unit 112B also randomly determines the aspect ratios of the two regions 410. As a result, the region 410 including the person 401 and the region 410 including the car 402 are cut out from the target image data 400.

[0114] 15, of the two cut-out regions 410, one region 410 includes the entire area of ​​the person 401, and the other region 410 includes the entire area of ​​the car 402. However, this is just one example, and depending on the aspect ratio determined randomly, there may be cases where one region 410 includes part of the person 401 and the other region 410 includes part of the car 402.

[0115] The two regions 410 are input to the second AI model 124. The second AI model 124 outputs an explanation 240 for each of the two regions 410. In this example, two explanations 240 corresponding to the two regions 410 are output. That is, {The man in shirt} is output as the explanation 240 corresponding to one region 410, and {Car is running} is output as the explanation for the other region 410.

[0116] As in the second embodiment, the correction unit 113B generates the second prompt 220 by adding these two explanations 240 to a pre-identified class or by replacing a pre-identified class with these two explanations 240. For example, the correction unit 113B identifies the main word {man} of {The man in shirt} and the main word {car} of {Car is running}. If there is a pre-defined class whose similarity to at least one of these two main words is equal to or greater than a threshold, the correction unit 113B replaces the pre-defined class with the highest similarity, which is a pre-defined class whose similarity is equal to or greater than a threshold, with the explanation 240 including at least one of the main words. On the other hand, if there is no pre-defined class whose similarity to both of these two main words is equal to or greater than the threshold, the correction unit 113B generates the second prompt 220 by adding these two explanations 240 to a pre-defined class.

[0117] As described above, according to the first modification of the third embodiment, when the target image data 400 includes multiple objects, multiple regions 410 are extracted from the target image data 400, and the extracted multiple regions 410 are input to the second AI model 124. This allows the first prompt acquisition unit 112B to narrow down the objects included in the target image data 400 and input them to the second AI model 124. Therefore, even when the target image data 400 includes multiple objects, a description 240 that describes all of the objects included in the target image data 400 can be obtained in a simple manner, and an appropriate first prompt 210 can be generated.

[0118] If the location of an important area in the target image data 400 is known in advance depending on the domain or task, the first prompt acquisition unit 112B can simply extract the important area 410 from the target image data 400 according to that location. For example, if the target image data 400 is in-vehicle image data, the sky area is not important, so the lower half area is important. In this case, the first prompt acquisition unit 112B can simply extract the lower half area 410 from the target image data 400.

[0119] (Variation 2 of Embodiment 3) In this variation 2, the first prompt acquisition unit 112B applies unsupervised clustering to the target image data 400 to extract multiple regions 410 from the target image data 400. An example of unsupervised clustering is the k-means method.

[0120] FIG. 16 is an explanatory diagram of processing in Variation 2 of Embodiment 3 of the present disclosure. The first prompt acquisition unit 112B applies clustering to the target image data 400. As a result, a cluster consisting of pixels representing a person 401 and a cluster consisting of pixels representing a car 402 are extracted from the target image data 400. The first prompt acquisition unit 112B cuts out a rectangular region surrounding the entire cluster including the person 401 as region 410 from the target image data 400. The first prompt acquisition unit 112B also cuts out a rectangular region surrounding the entire cluster including the car 402 as region 410. The first prompt acquisition unit 112B inputs the two cut-out regions 410 into the second AI model 124 and acquires the explanatory text 240. In this case, the explanatory text 240 identical to that of FIG. 15 is acquired.

[0121] As described above, according to the second modification of the third embodiment, the same effects as those of the first modification can be obtained.

[0122] (Variation 3 of Embodiment 3) In this variation 3, in step S202 of Fig. 13, the first prompt acquisition unit 112B inputs a third prompt to the second AI model 124 along with the target image data 400. The third prompt includes a question in natural language indicating at least one of the size of an object included in the target image data 400 and the display state of the object.

[0123] FIG. 17 is an explanatory diagram of processing in Variation 3 of Embodiment 3 of the present disclosure. In the example of FIG. 17 , target image data 400 includes a person 401, a car 402, and an apple 403. The car 402 is located behind the person 401, resulting in occlusion. The apple 403 is significantly smaller than the person 401 and the car 402. If the target image data 400 includes occlusions or relatively small objects, the second AI model 124 may not be able to output appropriate descriptions for all of the objects. Therefore, in Variation 3, the third prompt 250 includes a question indicating at least one of the size and display mode of the objects included in the target image data 400. The third prompt 250 is input by the user via the image input device 20.

[0124] In the example on the left side of FIG. 17 , the third prompt 250 includes a question requesting an explanation about a small object contained in the image. This question is an example of a question about the size of an object. Therefore, the second AI model 124 outputs an explanatory sentence 240 indicating that a small apple is present in the upper right corner of the image. The corrector 113B uses this explanatory sentence 240 to generate the first prompt 210. For example, the corrector 113B generates the first prompt 210 as {Class: Small Apple}.

[0125] In the example on the right side of FIG. 17 , the third prompt 250 includes a question about an object being hidden within another object. This question is an example of a question about the display mode of an object. Therefore, the second AI model 124 outputs an explanatory sentence 240 about a car being hidden by a person. The correcting unit 113B uses this explanatory sentence 240 to generate the first prompt 210. For example, the correcting unit 113B generates the first prompt 210 as {Class: Car hidden by a person}.

[0126] (Fourth Embodiment) In a fourth embodiment, when the user determines that the second prompt 220 is inappropriate, the second prompt 220 is switched to the first prompt 210. In the fourth embodiment, the same components as those in the first to third embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

[0127] FIG. 18 is a block diagram showing an example of the configuration of an information processing system 1C according to the fourth embodiment of the present disclosure.

[0128] The correction unit 113C of the information processing device 10C presents to the user the recognition result when the second prompt 220 and sample image data are input into the first AI model 122. The correction unit 113C acquires a selection instruction from the user as to whether to switch the second prompt 220 to the first prompt 210. The correction unit 113C switches the text prompt to be input into the first AI model 122 to the first prompt 210 or the second prompt 220 in accordance with the selection instruction. The sample image data is, for example, image data selected from the target image data 400. The selection instruction is input by the user using the input device 30. The correction unit 113C acquires the selection instruction from the input device 30. The memory 12C is the same as the memory 12B.

[0129] 19 is a flowchart illustrating an example of processing by the information processing device 10C according to the fourth embodiment of the present disclosure. The processing in steps S51 and S52 is the same as steps S1 and S2 in FIG.

[0130] (Step S53) The correction unit 113C obtains a classification result for the first classification class by inputting the second prompt 220 and the sample image data into the first AI model 122. Details of this process are the same as the process of step S3.

[0131] (Step S54) The correction unit 113C outputs the classification result to the display device 40. In this way, the classification result is presented to the user.

[0132] (Step S55) The correction unit 113C receives, from the input device 30, a selection instruction for the identification result received by the input device 30 from the user. Fig. 20 is a diagram showing an input screen 900 according to the fourth embodiment of the present disclosure. The input screen 900 includes an identification result display field 910, a YES button 901, and a NO button 902.

[0133] The classification result display field 910 displays the classification result for the sample image data. For example, the classification result display field 910 displays a bounding box surrounding an object of a first classification class included in the sample image data. The YES button 901 is selected by a user who wishes to switch from the second prompt 220 to the first prompt 210. This allows a user who is not satisfied with the classification result displayed in the classification result display field 910 to select classification processing using the first prompt 210 instead of the second prompt 220.

[0134] The NO button 902 is selected by a user who wishes to use the second prompt 220. This allows a user who is satisfied with the identification result displayed in the identification result display field 910 to select the identification process using the second prompt 220.

[0135] A user who wishes to reserve the classification result displayed in the classification result display field 910 can select a next button (not shown). When the next button is selected, the correction unit 113C inputs the next sample image data and the second prompt 220 into the first AI model 122 and obtains the classification result of the first classification class. The correction unit 113C outputs the classification result to the display device 40. As a result, the classification result for the next sample image is displayed in the classification result display field 910. In this case, the correction unit 113C repeats the processes of steps S53 to S55 until the user selects the YES button 901 or the NO button 902.

[0136] (Step S56) The correction unit 113C determines whether or not a selection instruction to switch the second prompt 220 to the first prompt 210 has been input by the user. If a selection instruction to switch has been input (YES in step S56), the process proceeds to step S57. If a selection instruction to switch has not been input (NO in step S56), the process proceeds to step S58. If the YES button 901 has been selected, the correction unit 113C may determine that a selection instruction to switch has been input. If the NO button 902 has been selected, the correction unit 113C may determine that a selection instruction to switch has not been input.

[0137] (Step S57 ) The modifying unit 113C determines to adopt the first prompt 210 instead of the second prompt 220 .

[0138] (Step S58) The modifying unit 113C determines to adopt the second prompt 220.

[0139] (Step S59) If the first prompt 210 is adopted, the classification unit 114 inputs the first prompt 210 and the target image data 400 into the first AI model 122 to obtain a classification result for the first classification class. On the other hand, if the second prompt 220 is adopted, the classification unit 114 inputs the second prompt 220 and the target image data 400 into the first AI model 122 to obtain a classification result for the first classification class.

[0140] (Step S60) The output unit 115 outputs the classification result to the display device 40. As a result, the display device 40 displays the classification result.

[0141] According to the information processing system 1C of the fourth embodiment, the user can select whether to use the first prompt 210 or the second prompt 220 based on the classification result of the sample image data when the second prompt 220 is used. Therefore, if the classification result of the second prompt 220 does not meet the user's intention, the user can perform classification processing on the target image data using the first prompt 210. Therefore, this configuration can realize classification processing that meets the user's intention.

[0142] (Variation of Embodiment 4) The correction unit 113C may display on the input screen 900 the classification result when the first prompt 210 and sample image data are input to the first AI model 122, in addition to the classification result when the second prompt 220 and sample image data are input to the first AI model 122. The user may determine which prompt to adopt after checking the classification results between the first prompt 210 and the second prompt 220. In this case, the user can more directly check which of the first prompt 210 and the second prompt 220 to adopt.

[0143] (Fifth Embodiment) In a fifth embodiment, a second information processing device 60 is caused to execute AI processing with a high processing load. In the fifth embodiment, the same components as those in the first to fourth embodiments are denoted by the same reference numerals, and description thereof will be omitted. Fig. 21 is a block diagram showing an example of the configuration of an information processing system 1D according to the fifth embodiment of the present disclosure.

[0144] The information processing system 1D includes a first information processing device 10D and a second information processing device 60.

[0145] The first information processing device 10D includes a processor 11D, a memory 12D, and a communication device 13.

[0146] First prompt acquisition unit 112D is the same as first prompt acquisition unit 112B described in embodiment 3. However, this is just an example, and first prompt acquisition unit 112D may be first prompt acquisition unit 112 or 112A described in embodiments 1 and 2.

[0147] The classification unit 114D outputs the second prompt 220 and the target image data 400 to the second information processing device 60 using the communication device 13. The classification unit 114D obtains the classification result for the first classification class from the second information processing device 60 using the communication device 13.

[0148] The communication device 13 is a communication interface that connects the first information processing device 10D to the second information processing device 60 via a network. The network is, for example, a wide area communication network including the Internet and a mobile phone communication network. The network may also be a local area network.

[0149] The second information processing device 60 is configured with a computer having higher processing performance and suitable for AI processing than the first information processing device 10D. The second information processing device 60 is, for example, a server such as a cloud server or an edge server. The second information processing device 60 includes a communication device 610, a processing unit 620, and a memory 630. The communication device 610 is a communication interface that connects the second information processing device 60 to the first information processing device 10D via a network.

[0150] The processing unit 620 is a processor such as a CPU.

[0151] The memory 630 stores the first AI model 122 and the second AI model 124. Therefore, the first AI model 122 and the second AI model 124 are omitted from the memory 12D.

[0152] FIG. 22 is a flowchart illustrating an example of processing of the information processing system 1D according to the fifth embodiment of the present disclosure.

[0153] The first prompt acquisition unit 112D outputs the target image data 400 to the second information processing device 60 using the communication device 610 (step S71).

[0154] The processing unit 620 acquires target image data using the communication device 610 (step S81).

[0155] The processing unit 620 acquires the description of the target image data by inputting the target image data to the second AI model 124 (step S82). Details of this process are the same as step S202 in FIG.

[0156] The processing unit 620 outputs the explanatory text to the first information processing device 10D using the communication device 610 (step S83).

[0157] The first prompt acquisition unit 112D acquires the explanatory text using the communication device 13 (Step S72).

[0158] The first prompt acquiring unit 112D generates the first prompt 210 from the explanatory text (step S73). Details of this process are the same as those of step S203 in FIG.

[0159] The correction unit 113 executes the correction process (step S74). The details of this process are the same as those of step S12 in Fig. 7. As a result, the second prompt 220 is obtained.

[0160] The identifying unit 114D outputs the second prompt 220 to the second information processing device 60 using the communication device 13 (step S75).

[0161] The processing unit 620 obtains the second prompt 220 using the communication device 610 (step S84).

[0162] The processing unit 620 inputs the second prompt 220 and the target image data 400 into the first AI model 122 and obtains a classification result for the first classification class (step S85).

[0163] The processing unit 620 outputs the identification result using the communication device 610 (step S86).

[0164] The identification unit 114D acquires the identification result using the communication device 13 (step S76).

[0165] The output unit 115 outputs the identification result to the display device 40 (step S77).

[0166] Thus, according to the information processing system 1D of embodiment 4, AI processing with a high processing load is executed by the second information processing device 60 (an example of an external computer), thereby reducing the processing load of the first information processing device 10D (an example of a computer).

[0167] (Variation of Embodiment 5) When first prompt acquisition unit 112 or first prompt acquisition unit 112A is used as first prompt acquisition unit 112D, second AI model 124 is not required. In this case, memory 630 does not need to store second AI model 124.

[0168] Sixth Embodiment In a sixth embodiment, a first prompt 210 and a second prompt 220 are switched depending on the similarity between a first identification class and a predefined class. In the sixth embodiment, the same components as those in the first to fifth embodiments are denoted by the same reference numerals, and a description thereof will be omitted.

[0169] FIG. 23 is a block diagram illustrating an example of a configuration of an information processing system 1E according to the sixth embodiment of the present disclosure.

[0170] The correction unit 113E calculates the similarity between the first identification class and each of the plurality of predefined classes, and identifies, from among the plurality of predefined classes, a predefined class that is similar to the first identification class, based on the similarity.

[0171] If the identification unit 114E can identify a predefined class similar to the first identification class, it inputs the first prompt 210 to the first AI model 122 instead of the second prompt 220. As a result, the identification unit 114E obtains an identification result for the first identification class.

[0172] The memory 12E is the same as the memory 12A shown in FIG.

[0173] 24 is a flowchart showing an example of processing by the information processing device 10E according to the sixth embodiment of the present disclosure. Steps S91 and S92 are the same as steps S11 and S12 in FIG.

[0174] (Step S93) The correction unit 113E calculates the similarity between the first identification class included in the first prompt 210 and each of the multiple predefined classes. The correction unit 113E may calculate the similarity between the word vector of the first identification class and each of the multiple word vectors of the multiple predefined classes. For example, word2vec may be used as the word vector. For the similarity, for example, cosine similarity and L2 distance may be used. When the first identification class consists of multiple words, the correction unit 113E may calculate the word vector of the first class by obtaining a word vector for each word and combining the word vectors. The similarity may be calculated using the main words of the first identification class and the predefined identification class described in embodiment 2.

[0175] For example, assume that the first identification class is {the person in shirt} and the predefined class group 230 is {car, bike, shirt}. The correction unit 113E calculates a word vector (first word vector) of the first identification class {the person in shirt} and word vectors (second word vectors) of the predefined classes {car}, {bike}, and {shirt}. The correction unit 113F calculates the similarity between the first word vector and each of the multiple second word vectors. The similarity may be, for example, an inter-vector distance such as cosine similarity or L2 distance. Note that the word vector may be a vector including multiple feature amounts output from multiple nodes constituting an intermediate layer in a recognition model used in region segmentation, object detection, or the like.

[0176] (Step S94) Correction unit 113E determines whether there is a predefined class whose similarity is equal to or greater than the threshold. If there is a predefined class whose similarity is equal to or greater than the threshold (YES in step S94), the process proceeds to step S95. If there is no predefined class whose similarity is equal to or greater than the threshold (NO in step S94), the process proceeds to step S96.

[0177] (Step S95) The correction unit 113E decides to adopt the first prompt 210 as the prompt to be input to the first AI model 122. For example, the correction unit 113E adopts the first prompt 210 that defines the first identification class {the person in shirt} as the prompt to be input to the first AI model 122.

[0178] (Step S96) The correction unit 113E decides to adopt the second prompt 220 as the prompt to be input to the first AI model 122. For example, the correction unit 113E adopts the second prompt 220, which adds the predefined class group {car, bike, shirt} to the first identification class {the person in shirt}, as the prompt to be input to the first AI model 122.

[0179] (Step S97) If the first prompt 210 is adopted, the classification unit 114E inputs the first prompt 210 and the target image data 400 into the first AI model 122 to obtain a classification result for the first classification class. If the second prompt 220 is adopted, the classification unit 114E inputs the second prompt 220 and the target image data 400 into the first AI model 122 to obtain a classification result for the first classification class.

[0180] (Step S98) The output unit 115 outputs the identification result to the display device 40.

[0181] Thus, according to the information processing system 1E of embodiment 6, if there is a predefined class similar to the first identification class, the first prompt 210 is input to the first AI model 122 instead of the second prompt 220, so that adding the second identification class can actually prevent the identification accuracy of the first identification class from decreasing.

[0182] (Variation of Embodiment 6) In Embodiment 6, the first prompt 210 is generated based on a user input, but the first prompt 210 may be generated using the second AI model 124, as in Embodiment 3. In this case, the memory 12B described in Embodiment 3 may be used as the memory 12E.

[0183] (Seventh Embodiment) In the seventh embodiment, a predefined class similar to the first identification class is deleted from among a plurality of predefined classes to generate a second prompt 220. In the seventh embodiment, the same components as those in the first to sixth embodiments are denoted by the same reference numerals, and a description thereof will be omitted.

[0184] FIG. 25 is a block diagram illustrating an example of a configuration of an information processing system 1F according to the seventh embodiment of the present disclosure.

[0185] The correction unit 113F calculates the similarity between the first identification class and each of the plurality of predefined classes. The correction unit 113F identifies a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity. If the correction unit 113F identifies a similar predefined class, the correction unit 113F deletes the similar predefined class from the second identification class. The correction unit 113F generates, as the second prompt 220, a text prompt that includes the first identification class and the second identification class from which the similar predefined class has been deleted.

[0186] The memory 12F is the same as the memory 12E.

[0187] 26 is a flowchart showing an example of processing by the information processing device 10F according to the seventh embodiment of the present disclosure. Steps S401 and S402 are the same as steps S11 and S12 in FIG.

[0188] (Step S403) Correction unit 113F calculates the similarity between the first identification class included in first prompt 210 and each of the multiple predefined classes. The details of this process are the same as the process by correction unit 113E described in the sixth embodiment.

[0189] (Step S404) Correction unit 113F determines whether there is a predefined class whose similarity is equal to or greater than the threshold. If there is a predefined class whose similarity is equal to or greater than the threshold (YES in step S404), the process proceeds to step S405. If there is no predefined class whose similarity is equal to or greater than the threshold (NO in step S404), the process proceeds to step S406.

[0190] (Step S405) The correction unit 113F deletes predefined classes from among the multiple predefined classes whose similarity is equal to or greater than a threshold. For example, assume that the similarity of {shirt} from the predefined class group {car, bike, shirt} to the first identification class {the person in shirt} is equal to or greater than the threshold. In this case, the correction unit 113F deletes {shirt} from the predefined class group. The correction unit 113F then generates a second prompt 220 using the remaining predefined classes {car, bike} and the first identification class {the person in shirt}. Hereinafter, this second prompt 220 will be referred to as a corrected second prompt.

[0191] (Step S406) The correction unit 113F adopts the second prompt 220 generated in step S402 as a prompt to be input to the first AI model 122. Hereinafter, this second prompt 220 will be referred to as an initial second prompt.

[0192] (Step S407) If the modified second prompt is adopted, the classification unit 114F inputs the modified second prompt and the target image data 400 into the first AI model 122 to obtain a classification result of the first classification class. If the initial second prompt is adopted, the classification unit 114F inputs the initial second prompt and the target image data 400 into the first AI model 122 to obtain a classification result of the first classification class.

[0193] (Step S408) The output unit 115 outputs the identification result to the display device 40.

[0194] In this way, according to the information processing system 1F of the seventh embodiment, if there is a predefined class similar to the first identification class, that predefined class is omitted from the second prompt, thereby preventing an object that should be identified as the first identification class from being identified as the predefined class, thereby reducing false positives.

[0195] (Variation of Embodiment 7) In Embodiment 7, the first prompt 210 is generated based on a user input, but the first prompt 210 may be generated using the second AI model 124, as in Embodiment 3. In this case, the memory 12B described in Embodiment 3 may be used as the memory 12E.

[0196] (Embodiment 8) In embodiment 8, predefined classes are defined taking into consideration the relationships between co-occurring words. Embodiment 8 can be applied to any of embodiments 2 to 7. The configuration of embodiment 8 will be described below with reference to FIG. 5, which was described in embodiment 2.

[0197] The predefined class storage unit 123 stores a co-occurrence relation database that stores a first word and a second word that co-occurs with the first word in association with each other.

[0198] The co-occurrence relation database is generated in advance as follows. First, an AI model configured with a vision and language model, a large number of prompt candidates, and a dataset of evaluation image data are prepared. The large number of prompt candidates include prompts containing words used in Wikipedia and the like, as well as prompts previously input to the first AI model 122.

[0199] Next, the AI ​​model receives the evaluation image data and prompt candidates and outputs a classification result. The evaluation image data includes object P. The prompt candidates include words representing object P. Assume that this classification result includes objects P1, P2, ..., Pn other than object P. In this case, it is estimated that word W representing object P and words W1, W2, ..., Wn representing objects P1, P2, ..., Pn have a co-occurrence relationship. Then, co-occurrence count values ​​K[P, P1], K[P, P2], ..., K[P, Pn] indicating the co-occurrence relationship between word W and each of words W1, W2, ..., Wn are incremented by one.

[0200] This process is performed on all evaluation image data, and each of the finally obtained co-occurrence count values ​​K[P, Pi] (i = 1, 2, ..., n) is divided by ΣP to normalize the co-occurrence count values ​​K[P, Pi]. Words Wi whose normalized co-occurrence count values ​​K[P, Pi] are equal to or greater than a threshold value are determined to have a co-occurrence relationship with word W. Word W is registered as the first word and word Wi as the second word in the co-occurrence relationship database. As a result, various first words and second words that have a co-occurrence relationship with the first word are associated and registered in the co-occurrence relationship database.

[0201] The correction unit 113A identifies a first word corresponding to the first identification class included in the first prompt 210 from the co-occurrence relation database. For example, assume that the first identification class includes {man} and {man} is registered as the first word in the co-occurrence relation database. In this case, the correction unit 113A identifies {man} from the co-occurrence relation database. Assume that {woman} is registered as the second word corresponding to {man} in the co-occurrence relation database. In this case, the correction unit 113A identifies {women} as a predefined class. The correction unit 113A adds the predefined class {woman} to the first identification class {man} to generate the second prompt 220. The generated second prompt 220 is input to the first AI model 122, and an identification result of the target image data 400 for the first identification class {man} is obtained.

[0202] When a plurality of words are registered in the co-occurrence relation database as second words corresponding to the first word {man}, correction unit 113A may identify the top L words with the co-occurrence count value K[man, Pi] as predefined classes. Then, correction unit 113A may generate second prompt 220 by adding the top L predefined classes to word {man}.

[0203] For example, if the first identification class is "license plate," the first AI model 122 may identify the entire vehicle including the license plate as the license plate, rather than the license plate itself. In this case, the second prompt 220 includes "license plate" and "vehicle." This allows the first AI model 122 to accurately distinguish and identify the license plate from the vehicle.

[0204] According to the present disclosure, false positives in AI models are suppressed, which is useful in performing processes for classifying objects using AI models.

Claims

1. An information processing method executed by a computer, comprising: obtaining a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies an object included in image data according to the text prompt; modifying the first prompt to a second prompt by adding a second identification class different from the first identification class, which is the identification class defined in the first prompt; obtaining an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and outputting the identification result.

2. The information processing method according to claim 1, wherein the second identification class is a class defined by a user.

3. The information processing method according to claim 1 or 2, wherein the first identification class is a class defined by a user.

4. An information processing method according to claim 1 or 2, wherein the modifying step includes: presenting a plurality of candidate identification classes; and determining a candidate identification class selected by a user from the plurality of candidate identification classes as the second identification class.

5. The information processing method according to claim 1 or 2, wherein the second identification class is a predefined class.

6. An information processing method according to claim 1 or 2, wherein the second identification class includes a plurality of predefined classes that have been defined in advance, and the modifying step includes: calculating a similarity between the first identification class and each of the plurality of predefined classes; identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity; if the similar predefined class is identified, replacing the similar predefined class with the first identification class; and generating the text prompt including the replaced first identification class and the remaining predefined classes as the second prompt.

7. An information processing method according to claim 1 or 2, wherein the second identification class includes a plurality of predefined classes, and the modifying step includes: calculating a similarity between the first identification class and each of the plurality of predefined classes; identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity; and, if the similar predefined class cannot be identified, generating, as the second prompt, the text prompt in which the plurality of predefined classes are added to the first identification class.

8. An information processing method as described in claim 1 or 2, wherein obtaining the first prompt includes obtaining the first prompt by inputting the target image data into a second AI model that explains the content of the image data in natural language.

9. An information processing method as described in claim 8, wherein obtaining the first prompt includes: extracting a plurality of regions from the target image data; and inputting the extracted regions into the second AI model to obtain the first prompt.

10. An information processing method according to claim 8, wherein cutting out a plurality of regions from the target image data includes: detecting a plurality of objects from the target image data; and setting a region for each of the plurality of detected objects.

11. The information processing method of claim 8, wherein the second AI model is a model that outputs an explanatory text for the image data in accordance with a third prompt, and the third prompt includes a question in natural language indicating at least one of the size of the object and the display state of the object.

12. An information processing method as described in claim 1 or 2, wherein the modifying step further includes: presenting to the user the recognition result when the second prompt and sample image data are input into the first AI model; obtaining a selection instruction from the user as to whether or not to return the second prompt to the first prompt; and switching the text prompt input into the first AI model to the first prompt or the second prompt in accordance with the selection instruction.

13. An information processing method as described in claim 1 or 2, wherein obtaining the identification result includes: the computer transmitting the second prompt and the target image data to an external computer equipped with the first AI model; and the external computer inputting the second prompt and the target image data into the first AI model to obtain the identification result, and transmitting the obtained identification result to the computer.

14. An information processing method as described in claim 1 or 2, wherein the second identification class includes a plurality of predefined classes that have been defined in advance, the modifying includes calculating the similarity between the first identification class and each of the plurality of predefined classes, and identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity, and the identifying includes, when the similar predefined class can be identified, inputting the first prompt into the first AI model instead of the second prompt.

15. An information processing method according to claim 1 or 2, wherein the second identification class includes a plurality of predefined classes that have been defined in advance, and the modifying step includes: calculating a similarity between the first identification class and each of the plurality of predefined classes; identifying a predefined class from the plurality of predefined classes that is similar to the first identification class based on the similarity; if the similar predefined class is identified, deleting the similar predefined class from the second identification class; and generating, as the second prompt, the text prompt that includes the first identification class and the second identification class from which the similar predefined class has been deleted.

16. An information processing method as described in claim 1 or 2, wherein the modifying step includes: identifying a first word corresponding to the first discrimination class from a database that stores a correspondence between a first word and a second word that co-occurs with the first word; and identifying the second discrimination class based on the second word that corresponds to the identified first word.

17. An information processing device including a processor, wherein the processor performs the following steps: obtain a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies an object included in image data according to the text prompt; modify the first prompt into a second prompt by adding a second identification class different from the first identification class, which is the identification class defined in the first prompt; obtain an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and output the identification result.

18. An information processing program that causes a computer to perform the following steps: obtain a first prompt, which is a text prompt that defines an identification class for a first AI model that identifies objects included in image data according to the text prompt; modify the first prompt into a second prompt by adding a second identification class different from the first identification class, which is the identification class defined in the first prompt; obtain an identification result for the first identification class by inputting the second prompt and target image data into the first AI model; and output the identification result.

Images