Information processing methods, information processing devices, and information processing programs
By adding recognition categories to the visual and language models to correct the prompt words, the false positive recognition problem of VLM when there are few categories is solved, and the recognition accuracy is improved.
Patent Information
- Application Number
- CN202580024429.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-12-19
- Filing Date
- 2025-03-21
- Publication Date
- 2026-11-10
AI Technical Summary
Visual and Language Models (VLMs) are prone to producing false positives when the text prompts specify few categories, and existing technologies struggle to effectively suppress this.
The prompts are corrected by appending a category different from the original recognition category to the text prompts, and the corrected prompts are then input into the VLM to obtain the recognition results, thus suppressing false positive recognition.
It effectively suppressed false positive recognition results of VLM when the text prompt words specified few categories, and improved the recognition accuracy.
Smart Images

Figure CN123076872A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to techniques for recognizing objects contained in image data. Background Technology
[0002] Visual and Language Models (VLMs) are known to accept image data and text prompts as input, and to annotate the image data. For example, Patent Document 1 discloses a technique for setting bounding boxes in an image using a VLM. In a VLM, if the user specifies, for example, the category of an object to be recognized in a text prompt, the VLM identifies objects that match the specified category based on the input image data and outputs the recognition result.
[0003] However, when VLM has fewer categories specified by text prompts, it is more likely to produce false positives, identifying objects belonging to a different category as belonging to the specified category.
[0004] Prior art literature
[0005] Patent documents
[0006] Patent Document 1: U.S. Patent Application Publication No. 2023 / 0154213 Summary of the Invention
[0007] The purpose of this disclosure is to provide a technique that can suppress false positive identification results output by AI models.
[0008] The information processing method in one aspect of this disclosure is a computer-executed information processing method, comprising: obtaining a first prompt word, wherein the first prompt word is the text prompt word that defines a recognition category for a first AI model that identifies objects contained in image data according to the text prompt word; modifying the first prompt word into a second prompt word by adding a second recognition category different from the first recognition category to the recognition category defined in the first prompt word; obtaining a recognition result for the first recognition category by inputting the second prompt word and object image data into the first AI model; and outputting the recognition result.
[0009] According to this disclosure, it is possible to suppress the output of false positive identification results by AI models. Attached Figure Description
[0010] Figure 1 This is a block diagram illustrating an example of the structure of the information processing system in Embodiment 1 of this disclosure.
[0011] Figure 2 This is a flowchart illustrating an example of the processing of the information processing apparatus in Embodiment 1 of this disclosure.
[0012] Figure 3 This is a diagram showing the input screen of the first example in Embodiment 1 of this disclosure.
[0013] Figure 4 This is a diagram showing the input screen of the second example in Embodiment 1 of this disclosure.
[0014] Figure 5 This is a block diagram illustrating an example of the structure of the information processing system in Embodiment 2 of this disclosure.
[0015] Figure 6 This is an explanatory diagram of the process in Embodiment 2 of this disclosure.
[0016] Figure 7 This is a flowchart illustrating an example of the processing of the information processing apparatus in Embodiment 2 of this disclosure.
[0017] Figure 8 It means Figure 7 The detailed flowchart of the correction process is shown.
[0018] Figure 9 This is a block diagram illustrating an example of the structure of the information processing system in Embodiment 3 of this disclosure.
[0019] Figure 10 This diagram shows the input screen of the first example and the input screen of the second example in Embodiment 3 of this disclosure.
[0020] Figure 11 This is a flowchart illustrating a first example of the processing of the information processing apparatus in Embodiment 3 of this disclosure.
[0021] Figure 12 This is a flowchart illustrating a second example of the processing of the information processing apparatus in Embodiment 3 of this disclosure.
[0022] Figure 13 This is a detailed flowchart showing the acquisition and processing of the first prompt word.
[0023] Figure 14 This is a diagram illustrating an example of processing in the information processing apparatus of Embodiment 3 of this disclosure.
[0024] Figure 15 This is an explanatory diagram of the process in a variation of Embodiment 3 of this disclosure.
[0025] Figure 16 This is an explanatory diagram of the processing in a variation 2 of Embodiment 3 of this disclosure.
[0026] Figure 17 This is an explanatory diagram of the processing in a variation of Embodiment 3 of this disclosure.
[0027] Figure 18 This is a block diagram illustrating an example of the structure of the information processing system in Embodiment 4 of this disclosure.
[0028] Figure 19 This is a flowchart illustrating an example of the processing of the information processing apparatus in Embodiment 4 of this disclosure.
[0029] Figure 20 This is a diagram showing the input screen in Embodiment 4 of this disclosure.
[0030] Figure 21 This is a block diagram illustrating an example of the structure of the information processing system in Embodiment 5 of this disclosure.
[0031] Figure 22 This is a flowchart illustrating an example of the processing of the information processing system according to Embodiment 5 of this disclosure.
[0032] Figure 23 This is a block diagram illustrating an example of the structure of the information processing system according to Embodiment 6 of this disclosure.
[0033] Figure 24 This is a flowchart illustrating an example of the processing of the information processing apparatus according to Embodiment 6 of this disclosure.
[0034] Figure 25 This is a block diagram illustrating an example of the structure of the information processing system according to Embodiment 7 of this disclosure.
[0035] Figure 26 This is a flowchart illustrating an example of the processing of the information processing apparatus according to Embodiment 7 of this disclosure. Detailed Implementation
[0036] (The insights that form the basis of this disclosure)
[0037] Because VLM learns from a large amount of training data, it can adapt to new recognition categories even when specified in text prompts without relearning. Furthermore, since VLM can specify various tasks in text prompts, it avoids switching models for each task. For example, if segmentation is specified as the task in the text prompt, VLM outputs object recognition results at the contour level. Similarly, if object detection is specified as the task, VLM outputs recognition results of objects enclosed by bounding boxes. Thus, VLM can flexibly respond to user needs.
[0038] However, when the number of recognition categories specified in the text prompt is small, the possibility of outputting false positive results increases. This is because, with few specified recognition categories, the VLM needs to classify as many objects as possible into the specified categories. Furthermore, the technology in Patent Document 1 only discloses using a VLM to set virtual bounding boxes in image data, and cannot suppress the aforementioned false positives.
[0039] Therefore, the inventors came to the insight that if a different recognition category is added to the recognition category specified in the text prompt to supplement the number of recognition categories, the possibility of VLM outputting false positive recognition results can be suppressed, and thus this disclosure was conceived.
[0040] (1) The information processing method in one aspect of this disclosure is an information processing method executed by a computer, comprising: obtaining a first prompt word, wherein the first prompt word is the text prompt word for defining a recognition category for an object contained in image data according to the text prompt word recognition; modifying the first prompt word into a second prompt word by adding a second recognition category different from the first recognition category to the recognition category defined in the first prompt word, i.e., the first recognition category; obtaining a recognition result for the first recognition category by inputting the second prompt word and object image data into the first AI model; and outputting the recognition result.
[0041] This structure adds a second, different category to the first recognition category defined by the first cue word to generate a second cue word, and then inputs the generated second cue word into the first AI model. As a result, the first AI model is able to identify objects that should be identified as the second category as the first category instead of the first. Consequently, this structure can suppress false positives related to the first recognition category.
[0042] (2) In the information processing method described in (1) above, the second identification category may also be a user-defined category.
[0043] This structure allows users to define a second identification category.
[0044] (3) In the information processing method described in (1) or (2) above, the first identification category may also be a user-defined category.
[0045] This structure allows users to define the first identification category.
[0046] (4) In any of the information processing methods described in (1) to (3) above, the correction may include: prompting multiple candidate identification categories; and determining the candidate identification category selected by the user from the multiple candidate identification categories as the second identification category.
[0047] According to this structure, users can easily select the second identification category because they can choose from multiple candidate identification categories.
[0048] (5) In any of the information processing methods described in (1) to (4) above, the second identification category may be a predefined category.
[0049] This structure can automatically determine the second recognition category without requiring the user to input it. Therefore, it eliminates the need for the user to receive the second prompt, thus reducing the number of processing steps in the information processing system 1A.
[0050] (6) In any of the information processing methods described in (1) to (5) above, the second identification category may include a plurality of predefined categories, and the correction includes: calculating the similarity between the first identification category and each of the plurality of predefined categories; determining, based on the similarity, a predefined category among the plurality of predefined categories that is similar to the first identification category; if the similar predefined category can be determined, replacing the similar predefined category with the first identification category; and generating the text prompt word including the replaced first identification category and the remaining predefined categories as the second prompt word.
[0051] According to this structure, when there is a predefined category similar to the first identification category, the predefined category is replaced with the first identification category. Therefore, the situation where an object that should be identified as the first identification category is identified as a predefined category is suppressed. As a result, false positives can be suppressed.
[0052] (7) In any of the information processing methods described in (1) to (6) above, the second identification category may include a plurality of predefined categories, and the correction includes: calculating the similarity between the first identification category and each of the plurality of predefined categories; determining the predefined category among the plurality of predefined categories that is similar to the first identification category based on the similarity; and generating the text prompt word obtained by adding the plurality of predefined categories to the first identification category when the similar predefined category cannot be determined, as the second prompt word.
[0053] According to this structure, in the absence of a predefined category similar to the first identification category, the predefined category is added unchanged to the first identification category to generate the second prompt word. Therefore, the identification categories contained in the second prompt word can be expanded. As a result, false positives can be suppressed.
[0054] (8) In any of the information processing methods described in (1) to (7) above, obtaining the first prompt word may also include: obtaining the first prompt word by inputting the object image data into a second AI model that describes the content of the image data in natural language.
[0055] When users struggle to identify object names or describe the state of an object in an image, they may be unable to come up with suitable prompts. Based on this structure, a second AI model automatically generates a first prompt. Therefore, even when the user cannot think of a suitable first prompt, it is possible to generate one.
[0056] (9) In the information processing method described in (8) above, obtaining the first prompt word may also include: cropping out multiple regions based on the object image data; and obtaining the first prompt word by inputting the cropped multiple regions into the second AI model.
[0057] When a single object image contains multiple objects, the second AI model may fail to generate appropriate descriptive sentences that identify each object separately. According to this structure, by segmenting the object image data into multiple regions, the probability of each region containing an object is increased, enabling the second AI model to generate appropriate first cue words.
[0058] (10) In the information processing method described in (8) above, cropping multiple regions based on the object image data may also include: detecting multiple objects based on the object image data; and setting a region for each of the detected multiple objects.
[0059] Based on this structure, since a region is set for each of the multiple objects, the second AI model can generate a suitable first prompt word.
[0060] (11) In any of the information processing methods described in (8) to (10) above, the second AI model may also be a model that outputs a descriptive sentence of the image data according to a third prompt word, the third prompt word being a natural language-based question indicating at least one of the size of the object and the display state of the object.
[0061] In cases where the object image data contains small objects, or in cases of occlusion where multiple objects overlap, the second AI model may fail to generate a suitable first cue word. According to this structure, since a third cue word representing at least one of the object's size and display method is input into the second learning model, the second AI model can generate a suitable first cue word even in cases where the object image data contains small objects or occlusion occurs.
[0062] (12) In any of the information processing methods described in (1) to (11) above, the modification may further include: prompting the user to input the second prompt word and the sampled image data into the first AI model; obtaining a selection instruction from the user on whether to return the second prompt word to the first prompt word; and switching the text prompt word input into the first AI model to the first prompt word or the second prompt word in accordance with the selection instruction.
[0063] According to this structure, the user can choose between using the first and second prompt words based on the recognition results of the sampled image data when the second prompt word is used. Therefore, if the recognition result of the second prompt word does not match the user's intention, the user can use the first prompt word to perform recognition processing on the object image data. Thus, this structure enables recognition processing that aligns with the user's intention.
[0064] (13) In any of the information processing methods described in (1) to (12) above, obtaining the recognition result may also include: the computer sending the second prompt word and the object image data to an external computer equipped with the first AI model; and the external computer obtaining the recognition result by inputting the second prompt word and the object image data into the first AI model, and sending the obtained recognition result to the computer.
[0065] The recognition process using the first AI model has a higher processing burden compared to the process of correcting the first prompt word. According to this structure, since the processing-intensive AI processing is performed by an external computer, the processing burden on the computer can be reduced.
[0066] (14) In any of the information processing methods described in (1) to (13) above, the second identification category may include a plurality of predefined categories, and the correction includes: calculating the similarity between the first identification category and each of the plurality of predefined categories; determining, based on the similarity, a predefined category among the plurality of predefined categories that is similar to the first identification category, and the identification includes: when the similar predefined category can be determined, replacing the second prompt word with the first prompt word and inputting it into the first AI model.
[0067] According to this structure, when there is a predefined category similar to the first recognition category, the first prompt word is input into the first AI model instead of the second prompt word. Therefore, it is possible to suppress the decrease in recognition accuracy of the first recognition category due to the addition of the second recognition category.
[0068] (15) In any of the information processing methods described in (1) to (14) above, the second identification category may include a plurality of predefined categories, and the correction includes: calculating the similarity between the first identification category and each of the plurality of predefined categories; determining, based on the similarity, a predefined category among the plurality of predefined categories that is similar to the first identification category; deleting the similar predefined category from the second identification category if the similar predefined category can be determined; and generating the text prompt word that includes the first identification category and the second identification category from which the similar predefined category has been deleted, as the second prompt word.
[0069] According to this structure, when there is a predefined category similar to the first identification category, the predefined category is omitted from the second prompt, thus suppressing the situation where an object that should be identified as the first identification category is identified as a predefined category. As a result, false positives can be suppressed.
[0070] (16) In any of the information processing methods described in (1) to (15) above, the correction may also include: establishing a database that stores a first word and a second word that co-occurs with the first word, determining the first word corresponding to the first identification category; and determining the second identification category based on the second word corresponding to the determined first word.
[0071] According to this structure, since the identification category that has co-occurrence with the first identification category is determined as the second identification category, the possibility of identifying an object that should be identified as the first identification category as the second identification category can be suppressed, and false positives can be suppressed.
[0072] This disclosure also enables the realization of information processing programs that enable computers to execute the characteristic structures contained in such information processing methods, or information processing systems that operate through such information processing programs. Furthermore, it is self-evident that such computer programs can be distributed via computer-readable, non-transitory recording media such as CD-ROMs or communication networks such as the Internet.
[0073] (17) In another aspect of this disclosure, the information processing apparatus is an information processing apparatus including a processor, the processor performing the following processing: obtaining a first prompt word, the first prompt word being the text prompt word for defining a recognition category for an object contained in image data according to the text prompt word; modifying the first prompt word to a second prompt word by adding a second recognition category different from the first recognition category to the recognition category defined in the first prompt word, i.e., the first recognition category; obtaining a recognition result for the first recognition category by inputting the second prompt word and object image data into the first AI model; and outputting the recognition result.
[0074] (18) In another embodiment of this disclosure, the information processing procedure causes a computer to perform: obtaining a first prompt word, the first prompt word being the text prompt word for a first AI model that defines a recognition category for an object contained in image data according to the text prompt word; modifying the first prompt word to a second prompt word by adding a second recognition category different from the first recognition category to the recognition category defined in the first prompt word, i.e., the first recognition category; obtaining a recognition result for the first recognition category by inputting the second prompt word and the object image data into the first AI model; and outputting the recognition result.
[0075] Furthermore, the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit the scope of this disclosure. Additionally, any constituent element in the following embodiments that is not described in the independent claim representing the highest-level concept is described as an arbitrary constituent element. Furthermore, the various contents can be combined in all embodiments.
[0076] (Implementation Method 1)
[0077] Figure 1 This is a block diagram illustrating an example of the structure of the information processing system 1 in Embodiment 1 of this disclosure. The information processing system 1 is a system that corrects a first prompt word input by the user to a second prompt word, and uses the second prompt word to cause the first AI (artificial intelligence) model 122 to perform object image data recognition processing.
[0078] Information processing system 1 includes an information processing device 10, an image input device 20, an input device 30, and a display device 40. The information processing device 10 is a computer such as a personal computer, a portable computer, a cloud server, or an edge server. When the information processing device 10 is composed of a server such as a cloud server or an edge server, the information processing device 10, the image input device 20, the input device 30, and the display device 40 can be interconnected via a communication network. The communication network may be, for example, a wide area network (WAN) including the Internet and mobile phone communication networks, or a local area network (LAN).
[0079] Image input device 20 is the information source of image data, i.e., object image data, which becomes the object of recognition processing. Image input device 20 can be, for example, a computer such as a web server or personal computer, or a camera or other imaging device. Imaging devices can be, for example, portable computers such as smartphones and tablets, or digital cameras. Imaging devices can also be dynamic image cameras for fixed-point observation. Object image data can be stored in the memory 12 of information processing device 10. Recognition processing refers to the process of identifying the recognition category to which the object contained in the object image data belongs. Recognition category refers to a label or category in a classification problem. Recognition categories can be, for example, "dog," "cat," or "bird."
[0080] Input device 30 is a computer equipped with input devices such as a keyboard, mouse, and touch panel. Input device 30 can be, for example, a portable computer or a desktop computer. When information processing device 10 is a personal computer, input device 30 can be composed of input devices such as a keyboard and mouse. Input device 30 receives various operations from the user.
[0081] Display device 40 is a display device that includes an organic EL panel or the like. Display device 40 can be a display device found in portable computers or desktop computers. Display device 40 displays the recognition result of the recognition processing by information processing device 10.
[0082] The information processing device 10 includes a processor 11 and a memory 12. The processor 11 is composed of a processor such as a central processing unit (CPU). The processor 11 includes an image acquisition unit 111, a first prompt word acquisition unit 112, a correction unit 113, a recognition unit 114, and an output unit 115. The image acquisition unit 111 to the output unit 115 are implemented by the processor 11 executing an information processing program stored in the memory 12. Alternatively, the image acquisition unit 111 to the output unit 115 may also be composed of dedicated hardware circuitry.
[0083] The image acquisition unit 111 acquires object image data from the image input device 20. For example, the image acquisition unit 111 can acquire object image data from the image input device 20 using a communication interface (not shown). The object image data can be still image data or image data of each frame of moving image data.
[0084] The first prompt word acquisition unit 112 acquires the text prompt word, i.e., the first prompt word, which defines the recognition category (hereinafter referred to as the first recognition category) for the first AI model 122. The first AI model 122 is composed, for example, a visual and language model that has been learned to recognize objects contained in image data according to the text prompt word, etc.
[0085] In Implementation 1, the first identification category is defined by the user. Therefore, the first prompt word acquisition unit 112 acquires the first prompt word from the input device 30, for example, using a communication interface (not shown).
[0086] The first prompt word defines the first recognition category using natural language input by the user through input device 30. An example of the first prompt word's format is a dictionary form such as {Recognition Category: Text}. The first prompt word can also define a task using natural language. Examples of tasks include segmentation and object detection. An example of the format of the first prompt word containing the recognition category and task is {Recognition Category: Dog, Recognition Category: Cat, Task: Object Detection}, etc. Furthermore, the text specified as the recognition category and task can be a sentence or a phrase. An example of text specifying the recognition category is "the man in white shirt".
[0087] The correction unit 113 corrects the first prompt word to the second prompt word by adding a second recognition category, which is different from the first recognition category, to the recognition category defined in the first prompt word.
[0088] The second recognition category is a user-defined category. For example, the user can define the recognition category of an object in a dataset containing multiple object image data as the second recognition category. In this case, the correction unit 113 can obtain the second recognition category input by the user using the input device 30. Furthermore, the correction unit 113 can generate a second prompt word by adding the second recognition category to the first recognition category.
[0089] Alternatively, the correction unit 113 may suggest multiple candidate recognition categories, and the candidate recognition category selected by the user from these multiple categories will be determined as the second recognition category. The candidate recognition category can be a user-defined recognition category. For example, the user can visually confirm the recognition categories contained in all object image data or one or more captured object image data, list the confirmed recognition categories, and delete duplicate recognition categories from the list, thereby determining the candidate recognition category.
[0090] Alternatively, candidate recognition categories can be determined based on clustering results of the object image data. For example, by inputting all or a portion of the object image data into a classifier, each object image data is classified into a specific cluster. The user then selects representative image data from each cluster from the classified image data. The user visually confirms the recognition categories contained in the representative image data of each cluster, lists the confirmed recognition categories, and removes duplicate recognition categories from the list, thereby determining the candidate recognition categories.
[0091] The recognition unit 114 obtains a recognition result for the first recognition category by inputting the second prompt word and object image data into the first AI model 122. An example of a recognition result for the first recognition category being a dog can be object image data with the area of the dog enclosed by a bounding box, or object image data with the area of the dog highlighted along its contour. The recognition result may include a rating of the reliability of the recognition result.
[0092] The output unit 115 outputs the recognition result. For example, the output unit 115 can output the recognition result to the display device 40 via a communication interface (not shown).
[0093] The memory 12 is composed of a non-volatile, rewritable storage device such as a solid-state drive (SSD). The memory 12 includes a candidate identification category storage unit 121. The memory 12 stores a first AI model 122.
[0094] The candidate identification category storage unit 121 stores the correction unit 113 to prompt the user with multiple candidate identification categories.
[0095] Alternatively, the information processing device 10 can be configured as an edge computer integrated with the input device 30 and the display device 40. Furthermore, some functions of the image acquisition unit 111 to the output unit 115 can be distributed across at least one of the input device 30 and the display device 40. The first AI model 122 can be stored on an external server connected to the information processing device 10 via a communication interface (not shown). In this case, the recognition unit 114 sends the second prompt word to the external server via the communication interface (not shown), and obtains the recognition result from the external server.
[0096] Figure 2 This is a flowchart illustrating an example of the processing of the information processing apparatus 10 in Embodiment 1 of this disclosure.
[0097] (Step S1)
[0098] The first prompt word acquisition unit 112 acquires the first prompt word input by the user from the input device 30. The first prompt word defines a first recognition category that the user wishes to identify. The number of first recognition categories can be one or more.
[0099] (Step S2)
[0100] The correction unit 113 corrects the first prompt word to the second prompt word. Figure 3 This is a diagram showing an input screen 200 in the first embodiment 1 of this disclosure. The input screen 200 is displayed on the display device 40. In the first example of the input screen 200, the user can freely input a second identification category.
[0101] Input screen 200 includes input field 201. Input field 201 accepts the user's input of a second recognition category. For example, the user may input an object contained in a dataset of object image data as the second recognition category. In this example, the user may input "car" or "road" as the second recognition category in input field 201.
[0102] On the other hand, in this example, the first prompt word 210 includes two recognition categories: "person" and "sky". Therefore, the correction unit 113 generates a text prompt word 220 by adding recognition categories such as "car" and "road" to the first prompt word 210. The second prompt word 220 includes the recognition category obtained by adding a second recognition category to the first recognition category, and the task defined by the first prompt word 210.
[0103] Figure 4 This is a diagram illustrating an input screen 300 in a second example of Embodiment 1 of this disclosure. The input screen 300 is displayed on the display device 40. In the second example of the input screen 300, the user can freely select a second recognition category from multiple candidate recognition categories. For example, the user selects an object from a dataset of object image data as the second recognition category.
[0104] Input screen 300 includes multiple selection buttons 301. Each selection button 301 corresponds to a different candidate recognition category. In this example, input screen 300 displays selection buttons 301 corresponding to "car," "sea," "road," and "house," respectively. Input screen 300 can display four selection buttons 301, but it can also display five or more, or three or fewer. If all candidate recognition categories cannot be displayed at once, input screen 300 can also display a toggle button (figure omitted). Whenever a toggle button is pressed, input screen 300 displays the hidden selection button 301 corresponding to a candidate recognition category. This allows the user to confirm all candidate recognition categories.
[0105] In this example, the candidate recognition categories "car" and "road" are selected from candidate recognition categories such as "car" and "sea". To this end, in the second prompt word 220, in addition to "person" and "sky" defined by the first prompt word 210, the recognition categories containing "car" and "road" are added.
[0106] (Step S3)
[0107] The recognition unit 114 inputs the second prompt word 220 and the object image data acquired by the image acquisition unit 111 into the first AI model 122 to obtain a recognition result for the first recognition category. The first AI model 122 recognizes the first and second recognition categories contained in the second prompt word 220 based on the object image data and outputs the recognition result for each recognition category. The recognition unit 114 acquires the recognition results for "person" and "sky" as the first recognition category from these recognition results, and uses them as the final recognition result for the object image data. Specifically, it acquires the object image data of the regions surrounding "person" and "sky" with bounding boxes, etc., and uses them as recognition results. The recognition unit 114 inputs all object image data of the object into the first AI model 122 and obtains the recognition results for each of the object image data.
[0108] (Step S4)
[0109] The output unit 115 outputs the recognition result to the display device 40. The display device 40 then displays the recognition result. The display device 40 can sequentially display the recognition results of all object image data according to the user's instructions.
[0110] In this way, the information processing device 10 adds a second identification category, different from the first identification category, to the first identification category defined by the first prompt word to generate a second prompt word 220, and inputs the generated second prompt word 220 into the first AI model 122. As a result, the first AI model 122 can identify objects that should be identified as the second identification category as the second identification category instead of forcibly identifying them as the first identification category. Consequently, this structure can suppress false positive identification results obtained for the first identification category.
[0111] (Implementation Method 2)
[0112] The information processing system 1A in Embodiment 2 uses a predefined, pre-defined category as the second defined category. Furthermore, in Embodiment 2, the same reference numerals are used for the same constituent elements as in Embodiment 1, and descriptions are omitted.
[0113] Figure 5 This is a block diagram illustrating an example of the structure of the information processing system 1A in Embodiment 2 of this disclosure. The memory 12A of the information processing apparatus 10A further includes a predefined category storage unit 123. The predefined category storage unit 123 stores a predefined category group. The predefined category group includes a plurality of predefined categories predefined on the information processing apparatus 10A side. The predefined categories employ recognition categories used in general object recognition processing. An example of recognition categories used in general object recognition processing is the recognition category used in an AI competition for object recognition. The candidate recognition category storage unit 121 is omitted from the memory 12A.
[0114] Figure 6 This is an explanatory diagram of the process in Embodiment 2 of this disclosure. In this example, the first prompt word 210 defines "person" and "sky" as the first identification categories. The predefined category group 230 includes predefined categories such as "person," "car," and "road." The correction unit 113A generates the second prompt word 220 by adding the predefined categories specified by the predefined category group 230 to the first identification category. In this example, since "person" is included in both the first identification category and the predefined category group 230, the correction unit 113A generates the second prompt word 220 after deleting "person" from the predefined category group 230.
[0115] Figure 7 This is a flowchart illustrating an example of the processing of the information processing apparatus 10A in Embodiment 2 of this disclosure. Steps S11, S13, and S14 are the same as steps S1, S3, and S4, and therefore are omitted from the description.
[0116] (Step S12)
[0117] The correction unit 113A performs the correction process. Details of the correction process are as follows: Figure 8 This will be discussed later.
[0118] Figure 8 It means Figure 7 The detailed flowchart of the correction process is shown.
[0119] (Step S101)
[0120] The correction unit 113A retrieves the subject of the text that defines the first identification category (hereinafter referred to as the object text). For example, if the object text is "the man wears a white shirt," the correction unit 113A identifies the first noun, "man," as the subject. The correction unit 113A uses morpheme analysis to distinguish the text defining the first identification category into words and parts of speech, and determines whether the object text is a sentence. If it is a sentence, the first noun is identified as the subject. If the text contains both a subject and a verb, the correction unit 113A identifies the object text as a sentence; if it does not contain a verb, the correction unit 113A identifies it as not a sentence.
[0121] (Step S102)
[0122] If the object text has a subject (step S102 "Yes"), the correction unit 113A proceeds to step S104. On the other hand, if the object text does not have a subject (step S102 "No"), the correction unit 113A proceeds to step S103.
[0123] (Step S103)
[0124] The correction unit 113A detects words that will become the subject based on the object text. For example, in the object text, the correction unit 113A can detect the first noun that appears and proper nouns, pronouns, or personal pronouns contained in the object text as the subject. In addition, if the object text contains compound nouns, the correction unit 113A can also remove the compound nouns from the candidates for subject. For example, if the object text is "the man in the white shirt," the first noun "man" that appears is identified as the word that will become the subject.
[0125] (Step S104)
[0126] The correction unit 113A calculates the similarity between the words representing the subject or main body contained in the object text (hereinafter referred to as the main words) and each predefined category. For example, if the main word is "man" and the multiple predefined categories are [person, car, dog, sky], the correction unit 113A calculates the similarity between the main word "man" and each of the predefined categories "person", "car", "dog", and "sky". The correction unit 113A calculates the word vectors for the main words and each predefined category. The word vectors can be, for example, word2vec. The similarity can be, for example, cosine similarity or L2 distance. The word vectors can be vectors containing multiple features output from multiple nodes constituting the intermediate layer by a cognitive model used in region segmentation or object detection, etc.
[0127] (Step S105)
[0128] The correction unit 113A determines whether there is a predefined category with a similarity value of 0.7, 0.8, or 0.9. If there is a predefined category with a similarity value of 0.7 or 0.8 (step S105 "Yes"), the process proceeds to step S106; if there is no predefined category with a similarity value of 0.7 or 0.9 (step S105 "No"), the process proceeds to step S107.
[0129] (Step S106)
[0130] The correction unit 113A replaces predefined categories with similarity thresholds of 1 or higher with object text. For example, if the predefined category "person" has a similarity threshold of 1 or higher with "man" in [person, car, dog, sky], the correction unit 113A replaces the predefined category "person" with the object text "the man wearing that white shirt". Therefore, the correction unit 113A generates a second prompt word 220 that defines "category: the man wearing that white shirt, category: car, category: dog, category: sky" as the first and second identification categories. Furthermore, if there are multiple predefined categories with similarity thresholds of 1 or higher, the correction unit 113A simply replaces the predefined category with the highest similarity with the object text.
[0131] (Step S107)
[0132] The correction unit 113A generates a second prompt word 220 by appending a predefined category group 230 to the object text. For example, the correction unit 113A generates a second prompt word 220 that defines "Category: The man wearing that white shirt, Category: Person, Category: Car, Category: Dog, Category: Sky" as the first and second identification categories. If the processing of step S106 or step S107 ends, the processing returns to... Figure 7 .
[0133] Thus, according to the information processing system 1A of embodiment 2, the second prompt word 220 can be automatically generated without requiring the user to input the second identification category. Therefore, by eliminating the need for the user to receive the second prompt word 220, the number of processing steps in the information processing system 1A can be reduced. In the case of a predefined category similar to the first identification category, the predefined category is replaced with the first identification category, thereby suppressing the identification of objects that should be identified as the first identification category as predefined categories. As a result, false positives can be suppressed.
[0134] (A variation of Implementation Method 2)
[0135] exist Figure 7 In this process, as a correction process (step S12), the following is adopted: Figure 8 The flowchart shown is for illustrative purposes only, but this disclosure is not limited thereto. In a variation of Embodiment 2, the correction unit 113A can generate the second prompt word 220 simply by adding a predefined category to the object text.
[0136] (Implementation Method 3)
[0137] Implementation 3 enables the second AI model 124 to generate the first prompt word 210. Furthermore, in Implementation 3, the same reference numerals are used for the same constituent elements as in Implementation 1, and descriptions are omitted.
[0138] Figure 9 This is a block diagram illustrating an example of the structure of the information processing system 1B in Embodiment 3 of this disclosure. In addition to the candidate identification category storage unit 121, the predefined category storage unit 123, and the first AI model 122, the memory 12B of the information processing device 10B also stores a second AI model 124. The second AI model 124 is an AI that describes the content of image data using natural language. An example of the second AI model 124 is a generated AI model such as a visual and language model. Furthermore, the second AI model 124 can also be used as the first AI model 122.
[0139] The first prompt word acquisition unit 112B acquires the first prompt word 210 by inputting the object image data acquired by the image acquisition unit 111 into the second AI model 124.
[0140] Figure 10 This diagram shows the input screen 810 in the first example and the input screen 820 in the second example of Embodiment 3 of this disclosure. The first prompt word acquisition unit 112B causes the display device 40 to display the input screen 810 or the input screen 820.
[0141] The input screen 810 includes an image display bar 811, a prompt selection button 812, and an image parsing selection button 813. The image display bar 811 displays object image data. When the user wishes to input the first prompt 210, the prompt selection button 812 is selected by the user. Hereinafter, the prompt input by the user will be referred to as the user prompt. When the user wishes for the second AI model 124 to generate the first prompt 210, the image parsing selection button 813 is selected by the user.
[0142] The input screen 820 has the same structure as the input screen 810, except that it uses quadrilateral buttons to replace radio buttons, namely, prompt word selection button 822 and image analysis selection button 823, which replace radio buttons with prompt word selection button 812 and image analysis selection button 813.
[0143] Figure 11 This is a flowchart illustrating a first example of the processing of the information processing apparatus 10B in Embodiment 3 of this disclosure. In the first example of Embodiment 3, the user is not allowed to select whether or not a user prompt word is used, and the second AI model 124 generates the first prompt word 210. Therefore, in the case of the first example of Embodiment 3, the first prompt word acquisition unit 112B does not cause the display device 40 to display the input screens 810 and 820. Steps S32, S33, and S34 and Figure 7 Steps S12, S13, and S14 are the same, so the explanation is omitted.
[0144] (Step S31)
[0145] The first prompt word acquisition unit 112B performs the acquisition process for the first prompt word 210. Details of the acquisition process for the first prompt word 210 are as follows: Figure 13 This will be discussed later.
[0146] Figure 12 This is a flowchart illustrating a second example of the processing of the information processing apparatus 10B in Embodiment 3 of this disclosure. In the second example of Embodiment 3, the user selects whether or not a user prompt word is used. Therefore, in the case of the second example of Embodiment 3, the first prompt word acquisition unit 112B displays the input screen 810 or the input screen 820 on the display device 40. Steps S43, S44, S45, S46 and Figure 7 Steps S11, S12, S13, and S14 are the same, so the explanation is omitted.
[0147] (Step S41)
[0148] The first prompt word acquisition unit 112B acquires the selection instruction received by the input device 30 from the user, and determines whether to use the user prompt word as the first prompt word 210 based on the acquired selection instruction. For example, if the prompt word selection button 812 is selected on the input screen 810, or if the prompt word selection button 822 is selected on the input screen 820, the first prompt word acquisition unit 112B determines that the user prompt word is being used. On the other hand, if the image analysis selection button 813 is selected on the input screen 810, or if the image analysis selection button 823 is selected on the input screen 820, the first prompt word acquisition unit 112B determines that the user prompt word is not being used. If the user prompt word is used (step S41 "Yes"), the process proceeds to step S43; if the user prompt word is not used (step S41 "No"), the process proceeds to step S42.
[0149] (Step S42)
[0150] The first prompt word acquisition unit 112B performs the acquisition process for the first prompt word 210. Details of the acquisition process for the first prompt word 210 are as follows: Figure 13 This will be discussed later.
[0151] Figure 13 This is a detailed flowchart showing the acquisition and processing of the first prompt word 210.
[0152] (Step S201)
[0153] The image acquisition unit 111 acquires object image data from the image input device 20.
[0154] (Step S202)
[0155] The first prompt word acquisition unit 112B acquires a descriptive sentence for the object image data by inputting the object image data into the second AI model 124. At this time, the first prompt word acquisition unit 112B can input a third prompt word along with the object image data into the second AI model 124. The third prompt word is a prompt word that gives instructions to the second AI model 124. For example, the third prompt word may contain an instruction to output a descriptive sentence indicating the category of the object contained in the object image data. The descriptive sentence output from the second AI model 124 may, for example, contain an indication of the category of the object contained in the object image data.
[0156] (Step S203)
[0157] The first prompt word acquisition unit 112B generates the first prompt word 210 based on the descriptive sentence. For example, the first prompt word acquisition unit 112B can generate the first prompt word 210 in dictionary form, representing the category (first identification category) contained in the descriptive sentence. Furthermore, the first prompt word acquisition unit 112B can also make the first prompt word 210 include a task. In this case, the first prompt word 210 may include, for example, a prompt word defining a predetermined task. For example, the predetermined task is the aforementioned segmentation or object detection, etc.
[0158] If step S203 ends, the process returns to... Figure 11 or Figure 12 .
[0159] Figure 14 This diagram illustrates an example of processing in the information processing apparatus 10B according to Embodiment 3 of this disclosure. The first prompt word acquisition unit 112B inputs object image data 400 to the second AI model 124. The second AI model 124 outputs a descriptive sentence 240 for the object image data 400. Here, the descriptive sentence 240 is "the man in shirt". The first prompt word acquisition unit 112B transforms the descriptive sentence 240 into dictionary form, such as {identification category: the man in shirt}, to generate the first prompt word 210. The correction unit 113B corrects the first prompt word 210 using the same method as in Embodiment 2, generating the second prompt word 220. That is, the correction unit 113B adds a predefined category group 230 to the first identification category, or replaces one predefined category in the predefined category group 230 with the first identification category, thereby generating the second prompt word 220. In this example, since the similarity between {the man in the shirt} and the predefined category {person} is above the threshold, the second prompt word 220 is generated by replacing {person} with {the man in the shirt}.
[0160] In addition, the first prompt word acquisition unit 112B can use the method of embodiment 1 to generate the second prompt word 220.
[0161] In cases where it is difficult to describe the categories contained in the object image data 400 and the condition of the object image data 400, the user may not be able to think of a suitable prompt word. According to embodiment 3, the first prompt word 210 is automatically created by the second AI model 124. Therefore, even if the user cannot think of a suitable first prompt word 210, a suitable first prompt word 210 can be generated.
[0162] (Modification 1 of Implementation Method 3)
[0163] Figure 15This is an explanatory diagram of the processing in a variation of Embodiment 3 of this disclosure. In this variation, the first prompt word acquisition unit 112B randomly crops multiple regions 410 from the object image data 400. For example, the first prompt word acquisition unit 112B randomly determines the number of segments, and randomly determines the aspect ratio (vertical-to-horizontal ratio) of the length of each segmented region 410.
[0164] exist Figure 15 In the example, the object image data 400 includes a person 401 and a car 402. The first prompt word acquisition unit 112B randomly determines the number of segments of the object image data 400 to be 2. The first prompt word acquisition unit 112B also randomly determines the aspect ratio of each of the two regions 410. Thus, the region 410 containing the person 401 and the region 410 containing the car 402 are cropped from the object image data 400.
[0165] In addition, Figure 15 In the example, one of the two cut-out areas 410 contains the entire area of person 401, and the other area 410 contains the entire area of car 402. However, this is just one example; depending on the randomly determined aspect ratio, it is also possible for one area 410 to contain a portion of person 401 and the other area 410 to contain a portion of car 402.
[0166] Two regions 410 are input into the second AI model 124. The second AI model 124 outputs a description sentence 240 for each of the two regions 410. In this example, two description sentences 240 corresponding to the two regions 410 are output. That is, as the description sentence 240 corresponding to one region 410, {The man wearing a shirt} is output, and as the description sentence for the other region 410, {The car is running} is output.
[0167] Similar to Embodiment 2, the correction unit 113B generates the second prompt word 220 by adding the two explanatory sentences 240 to the pre-identified categories or by replacing the pre-identified categories with the two explanatory sentences 240. For example, the correction unit 113B determines the main word {man} for {the man wearing a shirt} and the main word {car} for {the car is driving}. If there is a pre-defined category with a similarity threshold or higher for the main word of at least one of these two main words, the correction unit 113B replaces the pre-defined category with the highest similarity threshold with the explanatory sentence 240 containing the main word of at least one of them. On the other hand, if there is no pre-defined category with a similarity threshold or higher for either of these two main words, the correction unit 113B generates the second prompt word 220 by adding the two explanatory sentences 240 to the pre-defined categories.
[0168] Thus, according to Variation 1 of Embodiment 3, when the object image data 400 contains multiple objects, multiple regions 410 are cropped from the object image data 400, and the cropped regions 410 are input into the second AI model 124. Therefore, the first prompt word acquisition unit 112B can input the objects contained in the object image data 400 into the second AI model 124 in a narrowed manner. For this reason, when the object image data 400 contains multiple objects, a descriptive sentence 240 that represents all the objects contained in the object image data 400 without omission can be obtained in a simple way, and a suitable first prompt word 210 can be generated.
[0169] Furthermore, if the location of an important region in the object image data 400 is known in advance according to a domain or task, the first prompt word acquisition unit 112B can crop out the important region 410 from the object image data 400 according to that location. For example, if the object image data 400 is vehicle image data, since the sky area is not important, the lower half of the area becomes important.
[0170] In this case, the first prompt word acquisition unit 112B can crop out the lower half region 410 from the object image data 400.
[0171] (Modification 2 of Implementation Method 3)
[0172] In this variation 2, the first prompt word acquisition unit 112B extracts multiple regions 410 from the object image data 400 by applying unsupervised clustering to the object image data 400. An example of unsupervised clustering is the k-means method.
[0173] Figure 16 This is an explanatory diagram of the processing in a variation 2 of Embodiment 3 of this disclosure. The first prompt word acquisition unit 112B applies clustering to the object image data 400. Thus, clusters consisting of pixels representing people 401 and pixels representing cars 402 are extracted from the object image data 400. The first prompt word acquisition unit 112B crops out a quadrilateral region 410 from the object image data 400, encompassing the entire area of the cluster containing people 401. Furthermore, the first prompt word acquisition unit 112B also crops out a quadrilateral region 410, encompassing the entire area of the cluster containing cars 402. The first prompt word acquisition unit 112B inputs the two cropped regions 410 into the second AI model 124 to acquire the description sentence 240. In this case, the acquisition and... Figure 15 The same explanatory sentence 240.
[0174] Thus, according to variation 2 of embodiment 3, the same effect as in variation 1 can be obtained.
[0175] (Modification 3 of Implementation Method 3)
[0176] In this variation 3, the first prompt word acquisition unit 112B is in Figure 13 In step S202, the third prompt word and the object image data 400 are input into the second AI model 124. The third prompt word is a natural language-based question that represents at least one of the size of the object contained in the object image data 400 and the display state of the object.
[0177] Figure 17 This is an explanatory diagram of the process in a variation of Embodiment 3 of this disclosure. Figure 17 In the example, the object image data 400 includes a person 401, a car 402, and an apple 403. The car 402 is behind the person 401, causing occlusion. The apple 403 is significantly smaller than both the person 401 and the car 402. Thus, when there is occlusion or relatively small objects in the object image data 400, the second AI model 124 may not be able to output appropriate descriptions for all objects individually. Therefore, in Variation 3, the third prompt 250 includes a question indicating at least one of the size and display method of the objects contained in the object image data 400. Furthermore, the third prompt 250 is input by the user in the image input device 20.
[0178] exist Figure 17 In the example on the left, the third prompt 250 contains a question requesting an explanation related to a small object contained in the image. This question is an example of a question related to the size of the object. Therefore, the second AI model 124 outputs a descriptive sentence 240 indicating that a small apple exists in the upper right corner of the image. The correction unit 113B uses this descriptive sentence 240 to generate the first prompt 210. For example, the correction unit 113B generates a first prompt 210 such as {Category: Small Apple}.
[0179] exist Figure 17 In the example on the right, the third cue word 250 contains a question about the meaning of hiding another target within another target. This question is an example of a question related to the way an object is displayed. For this purpose, the second AI model 124 outputs a descriptive sentence 240 that means the car is hidden within the person. The correction unit 113B uses the descriptive sentence 240 to generate the first cue word 210. For example, the correction unit 113B generates a first cue word 210 such as {Category: Car hidden within a person}.
[0180] (Implementation Method 4)
[0181] In Embodiment 4, if the user determines that the second prompt word 220 is inappropriate, the second prompt word 220 is switched to the first prompt word 210. In Embodiment 4, the same reference numerals are used for the same constituent elements as in Embodiments 1 to 3, and descriptions are omitted.
[0182] Figure 18 This is a block diagram illustrating an example of the structure of the information processing system 1C in Embodiment 4 of this disclosure.
[0183] The correction unit 113C of the information processing device 10C prompts the user with the recognition result when the second prompt word 220 and the sampled image data are input into the first AI model 122. The correction unit 113C obtains a selection instruction from the user regarding whether to switch the second prompt word 220 to the first prompt word 210. The correction unit 113C switches the text prompt word input into the first AI model 122 to either the first prompt word 210 or the second prompt word 220 according to the selection instruction. The sampled image data is, for example, image data selected from the object image data 400. The selection instruction is input by the user using the input device 30. The correction unit 113C obtains the selection instruction from the input device 30. The memory 12C is the same as the memory 12B.
[0184] Figure 19 This is a flowchart illustrating an example of the processing of the information processing apparatus 10C in Embodiment 4 of this disclosure. The processing of steps S51 and S52 is... Figure 2 Steps S1 and S2 are the same.
[0185] (Step S53)
[0186] The correction unit 113C obtains the recognition result for the first recognition category by inputting the second prompt word 220 and the sampled image data into the first AI model 122. The details of this process are the same as those in step S3.
[0187] (Step S54)
[0188] The correction unit 113C outputs the recognition result to the display device 40. This displays the recognition result to the user.
[0189] (Step S55)
[0190] The correction unit 113C receives the selection instructions for the recognition results from the user received by the input device 30. Figure 20 This is a diagram showing the input screen 900 in Embodiment 4 of this disclosure. The input screen 900 includes a recognition result display bar 910, a "Yes" button 901, and a "No" button 902.
[0191] The recognition result display bar 910 displays the recognition results for the sampled image data. For example, the recognition result display bar 910 displays the bounding box surrounding the object of the first recognition category contained in the sampled image data. The "Yes" button 901 is selected by the user who wants to switch the second prompt 220 to the first prompt 210. Thus, the user who is not satisfied with the recognition results displayed in the recognition result display bar 910 can choose to use the recognition process using the first prompt 210 instead of the second prompt 220.
[0192] The "No" button 902 is selected by the user who wishes to use the second prompt word 220. Thus, the user who is satisfied with the recognition results displayed in the recognition result display bar 910 can choose to use the recognition process with the second prompt word 220.
[0193] Additionally, users who have reservations about the recognition results displayed in the recognition result display bar 910 can select the "Next" button (figure omitted). If the "Next" button is selected, the correction unit 113C inputs the next sampled image data and the second prompt word 220 into the first AI model 122 to obtain the recognition result for the first recognition category. The correction unit 113C outputs the recognition result to the display device 40. Thus, the recognition result for the next sampled image is displayed in the recognition result display bar 910. In this case, the correction unit 113C repeats the processing steps S53 to S55 until the user selects the "Yes" button 901 or the user selects the "No" button 902.
[0194] (Step S56)
[0195] The correction unit 113C determines whether the user has input a selection instruction to switch the second prompt word 220 to the first prompt word 210. If the selection instruction to switch has been input (step S56 "Yes"), the process proceeds to step S57. If the selection instruction to switch has not been input (step S56 "No"), the process proceeds to step S58. If the "Yes" button 901 is selected, the correction unit 113C determines that a selection instruction to switch has been input. If the "No" button 902 is selected, the correction unit 113C determines that no selection instruction to switch has been input.
[0196] (Step S57)
[0197] The revision section 113C replaces the second prompt word 220 and decides to adopt the first prompt word 210.
[0198] (Step S58)
[0199] The revision department 113C decided to adopt the second prompt word 220.
[0200] (Step S59)
[0201] When the first prompt word 210 is used, the recognition unit 114 inputs the first prompt word 210 and the object image data 400 into the first AI model 122 and obtains the recognition result for the first recognition category. On the other hand, when the second prompt word 220 is used, the recognition unit 114 inputs the second prompt word 220 and the object image data 400 into the first AI model 122 and obtains the recognition result for the first recognition category.
[0202] (Step S60)
[0203] The output unit 115 outputs the recognition result to the display device 40. The display device 40 then displays the recognition result.
[0204] According to the information processing system 1C of Embodiment 4, the user can select whether to use the first prompt word 210 or the second prompt word 220 based on the recognition result of the sampled image data when the second prompt word 220 is used. Therefore, if the recognition result of the second prompt word 220 does not match the user's intention, the user can use the first prompt word 210 to perform recognition processing on the target image data. Thus, this structure can achieve recognition processing that matches the user's intention.
[0205] (A variation of implementation method 4)
[0206] The correction unit 113C can display, on the input screen 900, not only the recognition result when the second prompt word 220 and the sampled image data are input into the first AI model 122, but also the recognition result when the first prompt word 210 and the sampled image data are input into the first AI model 122. The user can determine which prompt word to use based on the recognition results of the first prompt word 210 and the second prompt word 220. In this case, the user can more directly determine whether the first prompt word 210 or the second prompt word 220 should be used.
[0207] (Implementation Method 5)
[0208] In Embodiment 5, the second information processing device 60 performs AI processing, which has a high processing load. Furthermore, in Embodiment 5, the same reference numerals are used for the same components as in Embodiments 1-4, and descriptions are omitted. Figure 21 This is a block diagram illustrating an example of the structure of the information processing system 1D in Embodiment 5 of this disclosure.
[0209] The information processing system 1D includes a first information processing device 10D and a second information processing device 60.
[0210] The first information processing device 10D includes a processor 11D, a memory 12D, and a communication device 13.
[0211] The first prompt word acquisition unit 112D is the same as the first prompt word acquisition unit 112B described in Embodiment 3. However, this is an example, and the first prompt word acquisition unit 112D may also be the first prompt word acquisition unit 112, 112A described in Embodiments 1 and 2.
[0212] The recognition unit 114D uses the communication device 13 to output the second prompt word 220 and the object image data 400 to the second information processing device 60. The recognition unit 114D uses the communication device 13 to obtain the recognition result for the first recognition category from the second information processing device 60.
[0213] The communication device 13 is a communication interface that connects the first information processing device 10D to the second information processing device 60 via a network. The network may be, for example, a wide area network including the Internet and mobile phone communication networks. Alternatively, the network may be a local area network (LAN).
[0214] The second information processing device 60 is composed of a computer with higher processing performance than the first information processing device 10D and suitable for AI processing. The second information processing device 60 is, for example, a cloud server or an edge server. The second information processing device 60 includes a communication device 610, a processing unit 620, and a memory 630. The communication device 610 is a communication interface that connects the second information processing device 60 to the first information processing device 10D via a network.
[0215] The processing unit 620 is a processor such as a CPU.
[0216] The memory 630 stores the first AI model 122 and the second AI model 124. For this purpose, the first AI model 122 and the second AI model 124 are omitted from the memory 12D.
[0217] Figure 22 This is a flowchart illustrating an example of the processing of the information processing system 1D according to Embodiment 5 of this disclosure.
[0218] The first prompt word acquisition unit 112D uses the communication device 610 to output the object image data 400 to the second information processing device 60 (step S71).
[0219] The processing unit 620 uses the communication device 610 to acquire object image data (step S81).
[0220] The processing unit 620 obtains a description of the object image data by inputting the object image data into the second AI model 124 (step S82). The details of this processing are as follows... Figure 13 The steps are the same as in step S202.
[0221] The processing unit 620 uses the communication device 610 to output the explanatory sentence to the first information processing device 10D (step S83).
[0222] The first prompt word acquisition unit 112D acquires the description sentence using the communication device 13 (step S72).
[0223] The first prompt word acquisition unit 112D generates the first prompt word 210 based on the description sentence (step S73). The details of this process are as follows... Figure 13 The steps are the same as in step S203.
[0224] The correction unit 113 performs a correction process (step S74). Details of this process are as follows... Figure 7 The same steps are used for step S12. Thus, the second prompt word 220 is obtained.
[0225] The identification unit 114D uses the communication device 13 to output the second prompt word 220 to the second information processing device 60 (step S75).
[0226] The processing unit 620 uses the communication device 610 to obtain the second prompt word 220 (step S84).
[0227] The processing unit 620 inputs the second prompt word 220 and the object image data 400 into the first AI model 122 to obtain the recognition result for the first recognition category (step S85).
[0228] The processing unit 620 outputs the identification result using the communication device 610 (step S86).
[0229] The identification unit 114D uses the communication device 13 to obtain the identification result (step S76).
[0230] The output unit 115 outputs the recognition result to the display device 40 (step S77).
[0231] Thus, according to the information processing system 1D of embodiment 4, since the AI processing with high processing burden is performed by the second information processing device 60 (an example of an external computer), the processing burden of the first information processing device 10D (an example of a computer) can be reduced.
[0232] (A variation of implementation method 5)
[0233] When the first prompt word acquisition unit 112 or the first prompt word acquisition unit 112A is used as the first prompt word acquisition unit 112D, the second AI model 124 is not required. In this case, the memory 630 may also not store the second AI model 124.
[0234] (Implementation Method 6)
[0235] In Embodiment 6, the first prompt word 210 and the second prompt word 220 are switched according to the similarity between the first identification category and the predefined category. In Embodiment 6, the same reference numerals are used for the same constituent elements as in Embodiments 1 to 5, and descriptions are omitted.
[0236] Figure 23 This is a block diagram illustrating an example of the structure of the information processing system 1E according to Embodiment 6 of this disclosure.
[0237] The correction unit 113E calculates the similarity between the first identification category and each of the multiple predefined categories. Based on the similarity, the correction unit 113E determines the predefined category among the multiple predefined categories that is similar to the first identification category.
[0238] When the recognition unit 114E is able to determine a predefined category similar to the first recognition category, it replaces the second prompt word 220 with the first prompt word 210 and inputs it into the first AI model 122. As a result, the recognition unit 114E obtains the recognition result of the first recognition category.
[0239] Memory 12E and Figure 5 The memory shown is the same as 12A.
[0240] Figure 24 This is a flowchart illustrating an example of the processing of the information processing apparatus 10E according to Embodiment 6 of this disclosure. Steps S91, S92 and... Figure 7 Steps S11 and S12 are the same.
[0241] (Step S93)
[0242] The correction unit 113E calculates the similarity between the first identification category contained in the first prompt word 210 and each of the multiple predefined categories. The correction unit 113E calculates the similarity between the word vector of the first identification category and the word vectors of the multiple predefined categories. For example, word2vec can be used for the word vectors. For example, cosine similarity and L2 distance can be used for the similarity. When the first identification category consists of multiple words, the correction unit 113E calculates the word vector for each word and calculates the word vector of the first category by combining the word vectors. The similarity can be calculated using the main words of the first identification category and the predefined categories described in Embodiment 2.
[0243] For example, consider a case where the first recognition category is {the person in shirt}, and the predefined category group 230 is {car, bicycle, shirt}. The correction unit 113E calculates the word vector (first word vector) for the first recognition category {the person in shirt} and the word vectors (second word vectors) for each of the predefined categories {car}, {bicycle}, and {shirt}. The correction unit 113F calculates the similarity between the first word vector and each of the multiple second word vectors. The similarity can be, for example, cosine similarity or L2 distance, etc., between vectors. Furthermore, the word vectors can be vectors containing multiple features output from multiple nodes constituting the intermediate layer using a recognition model used in region segmentation or object detection.
[0244] (Step S94)
[0245] The correction unit 113E determines whether there is a predefined category with a similarity of more than a threshold. If there is a predefined category with a similarity of more than a threshold (step S94 "Yes"), the process proceeds to step S95; if there is no predefined category with a similarity of more than a threshold (step S94 "No"), the process proceeds to step S96.
[0246] (Step S95)
[0247] The correction unit 113E uses the first prompt word 210 as the prompt word input to the first AI model 122. For example, the correction unit 113E uses the first prompt word 210 that defines the first recognition category {the person wearing a shirt} as the prompt word input to the first AI model 122.
[0248] (Step S96)
[0249] The correction unit 113E decides to use the second prompt word 220 as the prompt word input to the first AI model 122. For example, the correction unit 113E uses the second prompt word 220, which is obtained by adding a predefined category group {car, bicycle, shirt} to the first recognition category {the person wearing a shirt}, as the prompt word input to the first AI model 122.
[0250] (Step S97)
[0251] When the recognition unit 114E uses the first prompt word 210, it inputs the first prompt word 210 and the object image data 400 into the first AI model 122 to obtain a recognition result for the first recognition category. When the recognition unit 114E uses the second prompt word 220, it inputs the second prompt word 220 and the object image data 400 into the first AI model 122 to obtain a recognition result for the first recognition category.
[0252] (Step S98)
[0253] The output unit 115 outputs the recognition result to the display device 40.
[0254] Thus, according to the information processing system 1E of embodiment 6, when there is a predefined category similar to the first recognition category, the first prompt word 210 is input into the first AI model 122 instead of the second prompt word 220. Therefore, it is possible to suppress the situation where the recognition accuracy of the first recognition category decreases due to the addition of the second recognition category.
[0255] (A variation of Implementation Method 6)
[0256] In embodiment 6, the first prompt word 210 is generated based on user input, but it can also be generated using the second AI model 124, similar to embodiment 3. In this case, the memory 12B described in embodiment 3 can be used as the memory 12E.
[0257] (Implementation Method 7)
[0258] In Embodiment 7, a second prompt word 220 is generated by deleting predefined categories that are similar to the first identification category from among a plurality of predefined categories. In Embodiment 7, the same reference numerals are used to label the same constituent elements as in Embodiments 1 to 6, and descriptions are omitted.
[0259] Figure 25 This is a block diagram illustrating an example of the structure of the information processing system 1F according to Embodiment 7 of this disclosure.
[0260] The correction unit 113F calculates the similarity between the first identification category and each of the multiple predefined categories. Based on the similarity, the correction unit 113F determines the predefined category among the multiple predefined categories that is similar to the first identification category. If a similar predefined category can be determined, the correction unit 113F removes the similar predefined category from the second identification category. The correction unit 113F generates a text prompt word that includes the first identification category and the second identification category from which the similar predefined category has been removed, as the second prompt word 220.
[0261] Memory 12F is the same as memory 12E.
[0262] Figure 26 This is a flowchart illustrating an example of the processing of the information processing apparatus 10F according to Embodiment 7 of this disclosure. Steps S401, S402 and... Figure 7 Steps S11 and S12 are the same.
[0263] (Step S403)
[0264] The correction unit 113F calculates the similarity between the first identification category contained in the first prompt word 210 and each of the multiple predefined categories. The details of this process are the same as those of the correction unit 113E described in Embodiment 6.
[0265] (Step S404)
[0266] The correction unit 113F determines whether there is a predefined category with a similarity of more than a threshold. If there is a predefined category with a similarity of more than a threshold (step S404 "Yes"), the process proceeds to step S405; if there is no predefined category with a similarity of more than a threshold (step S404 "No"), the process proceeds to step S406.
[0267] (Step S405)
[0268] The correction unit 113F deletes predefined categories whose similarity to a threshold is greater than or equal to a given threshold. For example, suppose the similarity of the category {shirt} in the predefined category group {car, bicycle, shirt} to the first identification category {the person wearing a shirt} is greater than or equal to a threshold. In this case, the correction unit 113F deletes {shirt} from the predefined category group. Then, the correction unit 113F uses the remaining predefined categories {car, bicycle} and the first identification category {the person wearing a shirt} to generate a second prompt word 220. Hereinafter, this second prompt word 220 will be referred to as the corrected second prompt word.
[0269] (Step S406)
[0270] The correction unit 113F uses the second prompt word 220 generated in step S402 as the prompt word input to the first AI model 122. Hereinafter, this second prompt word 220 will be referred to as the initial second prompt word.
[0271] (Step S407)
[0272] When using the corrected second prompt word, the recognition unit 114F inputs the corrected second prompt word and the object image data 400 into the first AI model 122 to obtain the recognition result for the first recognition category. When using the initial second prompt word, the recognition unit 114F inputs the initial second prompt word and the object image data 400 into the first AI model 122 to obtain the recognition result for the first recognition category.
[0273] (Step S408)
[0274] The output unit 115 outputs the recognition result to the display device 40.
[0275] Thus, according to the information processing system 1F of embodiment 7, when there is a predefined category similar to the first identification category, the predefined category is omitted from the second prompt word, thereby suppressing the situation where an object that should be identified as the first identification category is identified as a predefined category. As a result, false positives can be suppressed.
[0276] (A variation of Implementation Method 7)
[0277] In embodiment 7, the first prompt word 210 is generated based on user input, but it can also be generated using the second AI model 124, similar to embodiment 3. In this case, the memory 12B described in embodiment 3 can be used as the memory 12E.
[0278] (Implementation Method 8)
[0279] In Implementation 8, a predefined category is defined by considering the relational relationship of words with co-occurrence. Implementation 8 can be applied to any of Implementations 2 to 7. Hereinafter, the method described in Implementation 2 will be used. Figure 5 The structure of implementation method 8 will be explained below.
[0280] The category storage unit 123 is predefined to store a co-occurrence relationship database that establishes a corresponding storage for the first word and the second word that co-occurs with the first word.
[0281] The co-occurrence relation database is generated in advance as follows. First, an AI model consisting of a visual and a language model, a large number of cue word candidates, and a dataset of evaluation image data are prepared. The large number of cue word candidates includes cue words used in Wikipedia, etc., as well as cue words previously input to AI model 122.
[0282] Next, the AI model is input with evaluation image data and candidate prompts, and outputs a recognition result. The evaluation image data contains object P. The candidate prompts contain words representing object P. In this recognition result, objects P1, P2, ..., Pn other than object P are included. In this case, it is presumed that the word W representing object P and the words W1, W2, ..., Pn representing objects P1, P2, ..., Pn have a co-occurrence relationship. Furthermore, the co-occurrence counts K[P, P1], K[P, P2], ..., K[P, Pn] representing the co-occurrence relationship between word W and each of the words W1, W2, ..., Wn are incremented by 1.
[0283] This processing is performed on all evaluation image data, and the resulting co-occurrence counts K[P, Pi] (i=1, 2, ..., n) are divided by ΣP to normalize the co-occurrence counts K[P, Pi]. Words Wi and W are determined to have a co-occurrence relationship if their normalized co-occurrence counts K[P, Pi] are above a certain threshold. In the co-occurrence relationship database, word W is registered as the first word, and word Wi is registered as the second word. Thus, all first words and second words that have a co-occurrence relationship with the first word are correspondingly registered in the co-occurrence relationship database.
[0284] The correction unit 113A determines the first word corresponding to the first recognition category contained in the first prompt word 210 based on the co-occurrence relation database. For example, assuming that {man} is included in the first recognition category, {man} is registered as the first word in the co-occurrence relation database. In this case, the correction unit 113A determines {man} from the co-occurrence relation database. In the co-occurrence relation database, {woman} is registered as the second word corresponding to {man}. In this case, the correction unit 113A determines {woman} as a predefined category. The correction unit 113A adds the predefined category {woman} to the first recognition category {man} to generate the second prompt word 220. The generated second prompt word 220 is input into the first AI model 122 to obtain the recognition result of the object image data 400 for the first recognition category {man}.
[0285] When multiple words are registered in the co-occurrence relation database as the second word corresponding to the first word {man}, the correction unit 113A can determine the L words above the co-occurrence count value K[man, Pi] as predefined categories. Then, the correction unit 113A generates the second prompt word 220 by adding the L predefined categories above the word {man}.
[0286] For example, when the first recognition category is "license plate," the first AI model 122 sometimes identifies the entire area of a car containing the license plate as a license plate, rather than the license plate itself. In this case, the second prompt 220 is made to include both "license plate" and "car." Thus, the first AI model 122 is able to correctly distinguish between the license plate and the car.
[0287] Industrial availability
[0288] According to this disclosure, since false positives of the AI model are suppressed, it is useful in performing the process of classifying objects using the AI model.
Claims
1. An information processing method, which is an information processing method executed by a computer, comprising: Obtain the first prompt word, wherein the first prompt word is the text prompt word for which the first AI model defines the recognition category of the object contained in the image data according to the text prompt word recognition; The first prompt word is corrected to a second prompt word by adding a second identification category, which is different from the first identification category, to the identification category defined in the first prompt word. The recognition result for the first recognition category is obtained by inputting the second prompt word and the object image data into the first AI model; and Output the recognition result.
2. The information processing method according to claim 1, wherein, The second identification category is a user-defined category.
3. The information processing method according to claim 1 or 2, wherein, The first identification category is a user-defined category.
4. The information processing method according to claim 1 or 2, wherein, The correction includes: Prompts for multiple candidate recognition categories; and The candidate recognition category selected by the user from the plurality of candidate recognition categories will be determined as the second recognition category.
5. The information processing method according to claim 1 or 2, wherein, The second identification category is a predefined, predefined category.
6. The information processing method according to claim 1 or 2, wherein, The second identification category includes multiple predefined categories. The correction includes: Calculate the similarity between the first identification category and each of the multiple predefined categories; Based on the similarity, determine the predefined category among the plurality of predefined categories that is similar to the first identification category; If the similar predefined categories can be determined, the similar predefined categories are replaced with the first identification category; and The text prompt word, which includes the replaced first identification category and the remaining predefined categories, is generated as the second prompt word.
7. The information processing method according to claim 1 or 2, wherein, The second identification category includes multiple predefined categories. The correction includes: Calculate the similarity between the first identification category and each of the predefined categories; Based on the similarity, determine the predefined category among the plurality of predefined categories that is similar to the first identification category; and If the predefined similar categories cannot be determined, a text prompt word is generated by adding the multiple predefined categories to the first identification category, and this text prompt word is used as the second prompt word.
8. The information processing method according to claim 1 or 2, wherein, Obtaining the first prompt word includes: The first prompt word is obtained by inputting the object image data into a second AI model that describes the content of the image data in natural language.
9. The information processing method according to claim 8, wherein, Obtaining the first prompt word includes: Multiple regions are cropped based on the object image data; and The first prompt word is obtained by inputting the cut-out regions into the second AI model.
10. The information processing method according to claim 8, wherein, Multiple regions are cropped from the object image data, including: Detect multiple objects based on the object image data; and According to each of the multiple objects detected, a set area is defined.
11. The information processing method according to claim 8, wherein, The second AI model is a model that outputs a descriptive sentence for the image data according to the third prompt word. The third prompt word contains a natural language-based question indicating at least one of the object's size and the object's display status.
12. The information processing method according to claim 1 or 2, wherein, The amendment also includes: The recognition result when the user is prompted to input the second prompt word and the sampled image data into the first AI model; Obtain the user's selection instruction on whether to revert the second prompt to the first prompt; and The text prompt word input into the first AI model is switched to either the first prompt word or the second prompt word according to the selection instruction.
13. The information processing method according to claim 1 or 2, wherein, Obtaining the recognition result includes: The computer sends the second prompt word and the object image data to an external computer equipped with the first AI model; and The external computer obtains the recognition result by inputting the second prompt word and the object image data into the first AI model, and then sends the obtained recognition result to the computer.
14. The information processing method according to claim 1 or 2, wherein, The second identification category includes multiple predefined categories. The correction includes: Calculate the similarity between the first identification category and each of the plurality of predefined categories; and Based on the similarity, determine the predefined category among the plurality of predefined categories that is similar to the first identification category. The identification includes: when the predefined similar categories can be determined, inputting the first prompt word into the first AI model instead of the second prompt word.
15. The information processing method according to claim 1 or 2, wherein, The second identification category includes multiple predefined categories. The correction includes: Calculate the similarity between the first identification category and each of the multiple predefined categories; Based on the similarity, determine the predefined category among the plurality of predefined categories that is similar to the first identification category; If the similar predefined categories can be identified, the similar predefined categories are removed from the second identification category; and Generate a text prompt word that includes the first identification category and a second identification category that removes the similar predefined categories, and use this as the second prompt word.
16. The information processing method according to claim 1 or 2, wherein, The correction includes: A database is established to store the first word and the second word that co-occurs with the first word, and the first word is determined to be the first word corresponding to the first identification category; and The second identification category is determined based on the second word corresponding to the determined first word.
17. An information processing apparatus comprising a processor, The processor performs the following processes: Obtain the first prompt word, wherein the first prompt word is the text prompt word for which the first AI model defines the recognition category of the object contained in the image data according to the text prompt word recognition; The first prompt word is corrected to a second prompt word by adding a second identification category, which is different from the first identification category, to the identification category defined in the first prompt word. The recognition result for the first recognition category is obtained by inputting the second prompt word and the object image data into the first AI model; and Output the recognition result.
18. An information processing program that causes a computer to execute: Obtain the first prompt word, wherein the first prompt word is the text prompt word that defines the recognition category for the first AI model that identifies objects contained in image data according to text prompt words; The first prompt word is corrected to a second prompt word by adding a second identification category, which is different from the first identification category, to the identification category defined in the first prompt word. The recognition result for the first recognition category is obtained by inputting the second prompt word and the object image data into the first AI model; and Output the recognition result.
Citation Information
Patent Citations
Systems and methods for open vocabulary object detection
US20230154213A1
