Information processing apparatus and information processing method

Abstract

To provide an information processing apparatus and an information processing method for enabling accurate estimation.SOLUTION: In an information processing system in which a terminal device and an information processing apparatus are communicably connected with each other wirelessly or by wire via a predetermined communication network, the information processing apparatus comprises an acquisition unit and an estimation unit. The acquisition unit acquires a plurality of questions inquiring the occurrence of a predetermined situation. The estimation unit estimates the occurrence of the predetermined situation according to results of answers to the respective plurality of questions acquired by the acquisition unit.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Conventionally, there are known techniques for estimating a specific state based on predetermined information. For example, there is a known technique that uses an AI technology called VQA (Visual Question Answering) to estimate equipment states that violate safety manuals or dangerous states based on captured images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-071675 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional techniques, the accuracy of the answer can vary depending on the question (question sentence), so there is room for further improvement in order to perform more accurate estimation.

[0005] The present application has been made in consideration of the above, and has an object to provide an information processing device and an information processing method that enable highly accurate estimation. [Means for solving the problem]

[0006] The information processing device of the present application is characterized by having an acquisition unit that acquires a plurality of questions inquiring about the occurrence of a specified situation, and an estimation unit that estimates the occurrence of the specified situation based on the answer results to each of the plurality of questions acquired by the acquisition unit. Effect of the Invention

[0007] According to one aspect of the embodiment, an effect is achieved in that highly accurate estimation can be made possible. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of information processing according to the embodiment. [Diagram 3] FIG. 3 is a diagram (1) showing an example of a plurality of questions according to the embodiment. [Figure 4] FIG. 4 is a diagram (2) showing an example of a plurality of questions according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a question information storage unit according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a correct image data storage unit according to the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of information processing according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a detailed description will be given of an information processing device and an information processing method according to the present application (hereinafter, referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to the embodiment. In addition, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions are omitted.

[0010] Conventionally, a technique for estimating a specific state based on predetermined information is known. For example, a technique for estimating an equipment state that violates a safety manual or a dangerous state based on a captured image by using an AI technique called VQA is known. However, in the conventional technique, the accuracy of the answer may change depending on the question input to the VQA, so there is room for further improvement to perform highly accurate estimation using the VQA. The present application has been made in view of the above, and aims to provide an information processing device and an information processing method that enable highly accurate estimation. The present application is suitable for a system that detects the occurrence of an incident or the like in a facility such as a factory. Note that the following embodiment may not be limited to processing using VQA.

[0011] Moreover, VQA is a model that generates an answer to a question based on an image when an image and a question are input. For example, when an image including a person wearing a helmet and the question "Are you wearing a helmet?" are input, VQA outputs "YES" based on the image. However, VQA may output different answers to questions with similar content. For example, even questions with similar content such as "Are you wearing a helmet?", "Is the photographer wearing a helmet?", and "Are you wearing a helmet on your head?" may change YES / NO depending on the question for an image including a person wearing a helmet. In the following embodiment, multiple questions are prepared for a predetermined event (predetermined situation), and the occurrence of the predetermined situation is estimated by a majority vote of the VQA answers to each of the multiple questions. Since the answer output by VQA may vary depending on the selection of the subject and object and the way of expression, in order to accurately detect incidents while absorbing such variations, multiple sentences with various words and different ways of expression are used instead of a single question sentence representing the incident to be detected, thereby enabling accurate incident detection.

[0012] (Embodiment) [Configuration of information processing system] An information processing system 1 shown in Fig. 1 will be described. As shown in Fig. 1, the information processing system 1 includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other in a wired or wireless manner. Fig. 1 is a diagram showing an example of the configuration of the information processing system 1 according to an embodiment.

[0013] The terminal device 10 is an information processing device used by a user who wishes to check the occurrence of a predetermined situation (or the state of a predetermined target). The user may be, for example, a manager who manages whether workers (e.g., the user's employees) working at a predetermined site are in a dangerous state (for example, whether the equipment is in a state that violates a safety manual) (the user is not particularly limited to such an example). The terminal device 10 may be any device that can realize the processing in the embodiment. The terminal device 10 may be, for example, a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA, or other device. FIG. 2 shows a case where the terminal device 10 is a smartphone.

[0014] The terminal device 10 is, for example, a smart device such as a smartphone or smart glasses, and is a mobile terminal device that can communicate with any server device via a wireless communication network such as 4G to 5G (Generation) or LTE (Long Term Evolution). The terminal device 10 has a screen such as a liquid crystal display with a touch panel function, and may accept various operations on display data such as content, such as tapping, sliding, and scrolling, performed by a user's finger or a stylus. In FIG. 2, the terminal device 10 is used by a user U1.

[0015] The information processing device 100 is an information processing device intended to enable highly accurate estimation, and may be any device capable of implementing the processes in the embodiment. For example, the information processing device 100 may be an information processing device intended to enable highly accurate estimation using VQA in order to meet the desire of a user to confirm the occurrence of a predetermined situation. The information processing device 100 is realized, for example, by a server device or a cloud system that manages a predetermined site (for example, manages the safety of the site). For example, the information processing device 100 is realized by a server device or a cloud system such as a security system. For example, the information processing device 100 acquires a plurality of questions inquiring about the occurrence of a predetermined situation, and estimates the occurrence of the predetermined situation according to the answer results to each of the plurality of questions. This enables, for example, a highly accurate estimation regarding the occurrence of the predetermined situation.

[0016] Although FIG. 1 illustrates a case in which the terminal device 10 and the information processing device 100 are separate devices, the terminal device 10 and the information processing device 100 may be integrated.

[0017] [An example of information processing] Fig. 2 is a diagram showing an example of information processing of the information processing system 1 according to the embodiment. Figs. 3 and 4 are diagrams showing an example of a plurality of questions according to the embodiment. In the following embodiment, the plurality of questions shown in Figs. 3 and 4 are input to the VQA for each question. Fig. 2 will be described below with reference to Fig. 3 as appropriate.

[0018] The user U1 selects an image (which may be a video) to be used for checking the occurrence of a predetermined situation via the terminal device 10 (step S11). Hereinafter, the image selected for checking the occurrence of a predetermined situation is referred to as a "target image" as appropriate. The information processing device 100 acquires the target image (step S101). The information processing device 100 may acquire the target image in any way. For example, the information processing device 100 may acquire the target image from the terminal device 10, or may acquire the target image by acquiring selection information selected by the user U1 (which may be selection information of the target image or selection information of a video including the target image) from the terminal device 10. For example, when the terminal device 10 has an imaging device such as a camera, the information processing device 100 may acquire the target image from the terminal device 10, or may acquire the target image from an external information processing device having an imaging device. In the latter case, the information processing device 100 may acquire the target image transmitted from an external information processing device based on the selection information by acquiring the selection information of the user U1, or may acquire the target image transmitted by the external information processing device acquiring the selection information of the user U1. Furthermore, in the case where the information processing device 100 has an imaging device, the information processing device 100 may acquire a target image captured by the information processing device 100 by acquiring selection information of the user U1.

[0019] Here, a predetermined situation according to the embodiment will be described. The predetermined situation according to the embodiment may be any predetermined situation, and may be, for example, an accident or incident such as a person falling or an object falling. Furthermore, the predetermined situation according to the embodiment may be related to the state of a predetermined object (for example, a worker). For example, it may be related to the state of equipment, such as not wearing a helmet or not wearing goggles, or may be related to a dangerous state.

[0020] The information processing device 100 acquires a plurality of questions inquiring about the occurrence of a predetermined situation with respect to the target image (step S102). At this time, the information processing device 100 may acquire the plurality of questions in any manner. For example, when a plurality of questions are predefined in association with a predetermined situation, the information processing device 100 may acquire the plurality of questions predefined in association with the predetermined situation. For example, the information processing device 100 may acquire the plurality of questions predefined in association with a predetermined situation selected by the user U1. In FIG. 3, five questions ("Is that person wearing a hard hat?", "Does that individual have a safety helmet on?", "Are they sporting a helmet?", "Is he / she wearing a hard hat?", "Does that person have a safety helmet on their head?") are listed in order to identify a person not wearing a helmet, and these five questions correspond to the plurality of questions predefined in association with the predetermined situation. In addition, in FIG. 4, five questions ("Is that person wearing safety goggles?", "Does that individual have laboratory goggles on?", "Are they sporting welding goggles?", "Is he / she wearing laser safety goggles?", "Does that person have industrial safety glasses on?") are listed to identify a person not wearing goggles, and these five questions correspond to a plurality of questions that are predetermined in association with a predetermined situation. In addition, for example, when the user U1 directly sets (inputs) a plurality of questions (such as the questions shown in FIG. 3 and FIG. 4) (including a case where the questions are set by checking corresponding checkboxes from among a plurality of questions prepared in advance), the information processing device 100 may acquire the plurality of questions set by the user U1.Furthermore, the information processing device 100 may acquire a plurality of questions for identifying a person not wearing a helmet and a plurality of questions for identifying a person not wearing goggles for one target image. For example, the information processing device 100 may acquire both the questions shown in FIG. 3 and the questions shown in FIG. 4 in order to identify a person not wearing a helmet and a person not wearing goggles (which may be different from a person not wearing a helmet).

[0021] The information processing device 100 may use a model in acquiring a plurality of questions. The information processing device 100 may have a model for acquiring a plurality of questions. The model is, for example, a sentence generation model that is trained on data published on the Internet. For example, the model is a GPT (Generative Pre-trained Transformer) model that generates answers to questions. Such a model generates answers to questions from data published on the Internet. Since new data is published on the Internet every day, the answers provided by the model are updated every day. In other words, even if the same question is asked, different data is referenced by the model when generating the answer depending on the time of the question, and therefore different answers are generated.

[0022] The information processing device 100 may acquire a plurality of questions generated by a sentence generation model such as a GPT model. For example, the information processing device 100 may acquire a plurality of questions generated by inputting a prompt including an instruction sentence instructing to generate a question (but a question with a different expression) similar to one or a plurality of predetermined questions (the predetermined question may be, for example, a question previously determined in association with a predetermined situation, a question directly set by the user U1, or a question determined according to the selection of the predetermined situation by the user U1) into the sentence generation model. In FIG. 3, for example, if "Is that person wearing a hard hat?" is a predetermined question (there is no question similar to this question), the information processing device 100 may acquire questions "Does that individual have a safety helmet on?", "Are they sporting a helmet?", "Is he / she wearing a hard hat?", and "Does that person have a safety helmet on their head?" generated from this question. In addition, if the specified question is "Does that individual have a safety helmet on?" instead of "Is that person wearing a hard hat?" (and there is no other question with a similar content), the information processing device 100 may acquire questions generated from this question, such as "Is that person wearing a hard hat?", "Are they sporting a helmet?", "Is he / she wearing a hard hat?", and "Does that person have a safety helmet on their head?"

[0023] 4, for example, if "Does that person have industrial safety glasses on?" is a predetermined question (there is no question similar to this question), the information processing device 100 may acquire questions "Is that person wearing safety goggles?", "Does that individual have laboratory goggles on?", "Are they sporting welding goggles?", and "Is he / she wearing laser safety goggles?" generated from this question. Note that, if "Is that person wearing safety goggles?" is a predetermined question instead of "Does that person have industrial safety glasses on?" (and there is no question similar to this question), the information processing device 100 may acquire questions "Does that person have industrial safety glasses on?", "Does that individual have laboratory goggles on?", "Are they sporting welding goggles?", and "Is he / she wearing laser safety goggles?" generated from this question.

[0024] In this case, the prompt may be, for example, a prompt including an instruction sentence that instructs that the number of questions among the multiple questions to be generated be equal to or less than a predetermined threshold. For example, the prompt may be a prompt including an instruction sentence that instructs that the number of questions be 5 to 10 on average. Thereby, for example, since a large number of questions requires time for analysis, efficient sentence generation can be achieved by controlling the number of questions to a certain extent. In addition, the prompt may be, for example, a prompt including an instruction sentence that instructs that each of the multiple questions to be generated includes a predetermined word or expression. For example, the prompt may be a prompt including an instruction sentence that instructs that a word or expression (for example, "helmet" or "fallen down") that is essential for confirming the occurrence of a predetermined situation is included.

[0025] The prompt may also be, for example, a prompt including an instruction sentence that instructs the user to obtain an expected answer result as an answer to each of the multiple questions generated. For example, the prompt may be a prompt including an instruction sentence that instructs the user to obtain an answer result by selecting between two options such as "YES" or "NO". The prompt may also be, for example, a prompt including an instruction sentence that instructs the user to construct a sentence using words or expressions that do not specify attributes of people or things (including the number of people or things) (for example, "PERSON" that does not specify attributes, rather than "HE" or "SHE"), for any of the multiple questions generated.

[0026] That is, the information processing device 100 may acquire the multiple questions generated by, for example, inputting a prompt including an instruction sentence to instruct the number of questions in the multiple questions to be generated to be equal to or less than a predetermined threshold (for example, a predetermined threshold linked to a predetermined situation, etc.) into the sentence generation model. Also, the information processing device 100 may acquire the multiple questions generated by, for example, inputting a prompt including an instruction sentence to instruct the multiple questions to be generated to include predetermined words or expressions (for example, predetermined words or expressions linked to a predetermined situation, etc.) into the sentence generation model. Also, the information processing device 100 may acquire the multiple questions generated by, for example, inputting a prompt including an instruction sentence to instruct the multiple questions to be generated to be expected answer results (for example, predetermined answer results linked to a predetermined situation, etc.) as answer results for each of the multiple questions to be generated into the sentence generation model. In addition, the information processing device 100 may obtain multiple questions generated by inputting a prompt including an instruction sentence that instructs the sentence to be constructed using words or expressions that do not specify attributes for any of the multiple questions to be generated (e.g., words or expressions that are predetermined and linked to a specified situation) into the sentence generation model.

[0027] The information processing device 100 generates an answer result for each of the multiple questions by inputting the multiple questions and the target image thus acquired to the VQA for each question (step S103). In other words, the information processing device 100 generates an answer result for each of the multiple questions by inputting a combination of each of the multiple questions acquired and the target image to the VQA for each combination. Hereinafter, for convenience of explanation, an example will be described in which the answer result output by the VQA is output as a binary option of "YES" or "NO", but this is not particularly limited. For example, an answer including a specific word or expression such as "This is a helmet" or "There are gloves here" may be output by the VQA. Also, for example, instead of a binary option of "YES" or "NO", a three-option option such as "YES", "NO" or "PROBABLY" may be output.

[0028] The information processing device 100 estimates the occurrence of a predetermined situation based on the answer results to each of the multiple questions (step S104). Specifically, the information processing device 100 determines whether or not the majority vote of the answers to each of the multiple questions satisfies a predetermined condition. At this time, the majority vote may be performed by grouping answers with similar content among the answers to each of the multiple questions and counting the number of answers. For example, when the result is output as four options such as "YES", "NO", "PROBABLY", or "NOT YES", the affirmative answers of "YES" and "PROBABLY" may be grouped, and the negative answers of "NO" and "NOT YES" may be grouped, and a majority vote may be performed among the groups.

[0029] Using the example of FIG. 3, for example, assume that an image including a wearer wearing a helmet (assumed to be image G1) is used as a target image, and the image G1 and a question "Is that person wearing a hard hat?" are input to the VQA, generating an answer "YES" as an answer result by the VQA. Similarly, assume that the information processing device 100 inputs the image G1 and the question "Does that individual have a safety helmet on?" to the VQA, generating an answer "YES" as an answer result by the VQA. Similarly, assume that the information processing device 100 inputs the image G1 and the question "Are they sporting a helmet?" to the VQA, generating an answer "YES" as an answer result by the VQA. Similarly, assume that the information processing device 100 inputs the image G1 and the question "Is he / she wearing a hard hat?" to the VQA, generating an answer "YES" as an answer result by the VQA. Similarly, it is assumed that the information processing device 100 generates an answer of "YES" as an answer result by VQA by inputting the image G1 and the question "Does that person have a safety helmet on their head?" to the VQA. In this case, since all the answers to the five questions are "YES", the information processing device 100 estimates that the predetermined target shown in the image G1 is wearing a helmet. Note that even if not all the answers to the five questions are "YES", but rather three questions are "YES" and two questions are "NO", since "YES" prevails by majority vote, the information processing device 100 estimates that the predetermined target shown in the image G1 is wearing a helmet. On the other hand, if the answers to the three questions are "NO" and the answers to the two questions are "YES", since "NO" prevails by majority vote, the information processing device 100 estimates that the predetermined target shown in the image G1 is not wearing a helmet.

[0030] The information processing device 100 judges whether the majority vote of the answers to each question of the multiple questions satisfies a predetermined condition, and when it is judged that the predetermined condition is satisfied, it estimates the occurrence of a predetermined situation. For example, when 80% or more of the answers are answers that support a predetermined result in the majority vote (for example, when there are positive answers and negative answers, and 80% or more of the answers are answers that support either one of them, etc.), the information processing device 100 judges that the predetermined condition is satisfied, and estimates the occurrence of a predetermined situation based on the answers of 80% or more. On the other hand, when 80% or more of the answers that support the predetermined result are not obtained in the majority vote (for example, when there are positive answers and negative answers, and 70% or less than 80% of the answers are answers that support either one of them, etc.), the information processing device 100 judges that the predetermined condition is not satisfied.

[0031] When the information processing device 100 determines that the predetermined condition is not satisfied, it modifies at least one of the multiple questions that were the subject of the majority vote, re-inputs the modified question into the VQA, and performs the determination process again by majority vote, thereby estimating the occurrence of the predetermined situation. That is, the information processing device 100 estimates the occurrence of the predetermined situation when the majority vote of the answers to each question of the multiple questions including the modified question satisfies the predetermined condition (for example, when there are positive and negative answers and 80% or more of the answers support either one of them). Note that the information processing device 100 repeats the process of modifying the questions and performing the determination by majority vote in this manner until a determination result that satisfies the predetermined condition is obtained.

[0032] Then, when the information processing device 100 determines that the majority vote of the answers to each question of the multiple questions satisfies a predetermined condition, it transmits information on the estimation result of the occurrence of the predetermined situation to the terminal device 10 (step S105). Note that the information on the estimation result of the occurrence of the predetermined situation is not limited to the estimation result itself, and may include, for example, a result of accuracy based on the majority vote (majority vote of the answer results) during the determination process. For example, the information processing device 100 may transmit to the terminal device 10, not only information indicating that the predetermined target is estimated to be wearing a helmet, but also information indicating that the predetermined target is estimated to be wearing a helmet and that the accuracy is about 70%, information indicating the content of the question used for the estimation, and the like. Also, for example, when the accuracy is about 60%, the information processing device 100 may transmit to the terminal device 10 information indicating that the accuracy is low and that it is better to change or add a question and perform analysis again. Also, for example, when the accuracy is 50% (the same number or points by majority vote) and there is a tie for first place, the information processing device 100 may transmit information indicating that either possibility is possible to the terminal device 10. When the terminal device 10 receives information regarding the estimated result of the occurrence of the predetermined situation from the information processing device 100, it displays the information regarding the estimated result of the occurrence of the predetermined situation. For example, the terminal device 10 displays information indicating that the predetermined target is estimated to be wearing a helmet (for example, a sentence indicating that "worker A is estimated to be wearing a helmet"). Also, for example, the terminal device 10 displays information indicating that the predetermined target is estimated to be wearing a helmet and information indicating that the accuracy is about 70% (for example, a sentence indicating that "worker A is estimated to be wearing a helmet, but the accuracy is about 70%). Also, for example, when the accuracy is 50% and there is a tie for first place, the terminal device 10 displays information indicating that either possibility is possible (for example, a sentence indicating that "worker A is estimated to be either wearing a helmet or not wearing a helmet").

[0033] (Information processing variations: question weighting) In the above embodiment, the information processing device 100 determines whether a predetermined condition is satisfied based on a majority vote of the answers to each question of the multiple questions. The information processing device 100 may calculate a score of the answer to each question by weighting each question of the multiple questions, and determine whether a predetermined condition is satisfied based on the calculated score result for each question of the multiple questions. That is, the information processing device 100 may perform a majority vote using a score considering the weighting for each question of the multiple questions. Using the example of FIG. 3, for example, suppose that the image G1 and the question "Is that person wearing a hard hat?" are input to the VQA, and an answer "YES" is generated as the answer result by the VQA. In the case where the question is weighted by "1.2", the information processing device 100 calculates a score of "+1.2" by considering the answer "YES" as "+1". This score corresponds to the score of the answer to the question "Is that person wearing a hard hat?". Similarly, for example, suppose that an image G1 and a question "Are they sporting a helmet?" are input to the VQA, and an answer "YES" is generated as the answer result by the VQA. If a weight of "0.8" is assigned to this question, the information processing device 100 calculates a score of "+0.8" by adding the answer "YES" to "+1". This score corresponds to the score of the answer to the question "Are they sporting a helmet?". Then, when taking a majority vote, the information processing device 100 obtains a score result by adding (or subtracting) all the scores calculated in this way. For example, in the case of the above two questions, the information processing device 100 obtains a score result of "+2.0" by adding the score of "+1.2" of the answer to the question "Is that person wearing a hard hat?" and the score of "+0.8" of the answer to the question "Are they sporting a helmet?".If the answers to each of the multiple questions include a "NO" answer, the information processing device 100 will set the "NO" answer to "-1", calculate the score of the answer to that question, and reflect this in the score result (in this case, subtraction rather than addition). Based on the score result calculated in this way, the information processing device 100 may determine that a predetermined condition is met if the score is equal to or greater than a predetermined threshold. In the above case, if the score result is a positive score, the answer "YES" will prevail, and if the score result is a negative score, the answer "NO" will prevail.

[0034] When each question is evaluated based on the answer result to each question of a plurality of questions, the information processing device 100 may reflect the evaluation in the weighting of each question. Using the example of FIG. 3, for example, by inputting the image G1 and the question "Is that person wearing a hard hat?" to the VQA, an answer "YES" is generated as the answer result by the VQA, and when the question is weighted by "1.2", the information processing device 100 calculates a score of "+1.2" by setting the answer "YES" as "+1". However, when the answer "YES" is evaluated to be wrong (i.e., an incorrect answer), the weighting may be changed to reflect the evaluation. For example, the information processing device 100 may change the weighting of the question "Is that person wearing a hard hat?" from "1.2" to "1.0". On the other hand, when the answer "YES" is evaluated to be not wrong (i.e., a correct answer), the information processing device 100 may change the weighting to reflect the evaluation. For example, the information processing device 100 may change the weighting for the question "Is that person wearing a hard hat?" from "1.2" to "1.5." The evaluation may be, for example, an evaluation of the estimation result by the user U1, or an evaluation using an image (correct answer image data) prepared in advance as correct answer data. In the former case, for example, when a predetermined target is estimated to be not wearing a helmet even though the target is wearing a helmet, if the user U1 evaluates that the estimation result is incorrect, the information processing device 100 changes the weighting for each question of the plurality of questions that are the subject of the majority vote based on the evaluation result of the user U1 who evaluated that the estimation result is incorrect. In the latter case, for example, the information processing device 100 changes the weighting for each question of the plurality of questions that are the subject of the majority vote based on the answer result using the correct answer image data as the target image. For example, the information processing device 100 performs evaluation based on the answer result using the correct answer image data, and determines whether the answer is correct or incorrect, thereby changing the weighting for each question of the plurality of questions that are the subject of the majority vote.This allows, for example, automatic evaluation of each question, thus allowing automatic weighting changes, automatic question changes, etc.

[0035] [Configuration of terminal device] Next, the configuration of the terminal device 10 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the terminal device 10 according to the embodiment. As shown in Fig. 5, the terminal device 10 has a communication unit 11, an input unit 12, an output unit 13, and a control unit 14.

[0036] (Communications Department 11) The communication unit 11 is realized by, for example, a network interface card (NIC) etc. The communication unit 11 is connected to a predetermined network N by wire or wirelessly, and transmits and receives information to and from the information processing device 100 etc. via the predetermined network N.

[0037] (Input section 12) The input unit 12 accepts various operations from a user. In Fig. 2, the input unit 12 accepts various operations from a user U1. For example, the input unit 12 may accept various operations from a user via a display surface using a touch panel function. The input unit 12 may also accept various operations from buttons provided on the terminal device 10, or a keyboard or mouse connected to the terminal device 10.

[0038] (Output section 13) The output unit 13 is a display screen of a tablet terminal or the like realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. The output unit 13 displays, for example, information on the result of estimation of the occurrence of a predetermined situation received from the information processing device 100. For example, the output unit 13 displays information indicating whether or not it is estimated that the predetermined situation has occurred. In addition, for example, when it is estimated that the predetermined situation has occurred (for example, when displaying information indicating that it is estimated that the predetermined situation has occurred), the output unit 13 displays information indicating the degree of certainty (the degree of certainty based on the majority vote) that the predetermined situation is estimated to have occurred. For example, when 80% of the votes are cast for either one of the two options, "YES" or "NO," and the answer prevails, the output unit 13 displays information indicating that the degree of certainty is about 80%. In addition, for example, when the degree of certainty is low (when the degree of certainty of the answer that prevails in the majority vote is low), the output unit 13 displays information indicating that it is better to perform analysis again by changing or adding questions. Also, for example, when the probability is 50% (a tie or a tie by majority vote) and there is a tie for first place, the output unit 13 displays information indicating that either possibility is possible.

[0039] (Control unit 14) The control unit 14 is, for example, a controller, and is realized by a central processing unit (CPU) or a micro processing unit (MPU) executing various programs stored in a storage device inside the terminal device 10 using a random access memory (RAM) as a working area. For example, the various programs include application programs installed in the terminal device 10. For example, the various programs include application programs that display information related to an estimated result of the occurrence of a predetermined situation received from the information processing device 100. The control unit 14 is also realized by an integrated circuit, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0040] As shown in FIG. 5, the control unit 14 has a receiving unit 141 and a transmitting unit 142, and realizes or executes the information processing operations described below.

[0041] (Receiving unit 141) The receiving unit 141 receives, for example, information on the result of estimation of the occurrence of a predetermined situation. For example, the receiving unit 141 receives information indicating whether or not it is estimated that the predetermined situation has occurred. In addition, for example, when it is estimated that the predetermined situation has occurred (for example, when information indicating that the predetermined situation is estimated to have occurred is received, etc.), the receiving unit 141 receives information indicating the degree of certainty (the degree of certainty based on the majority vote) that the predetermined situation is estimated to have occurred. For example, when either one of the two options of "YES" or "NO" is voted for by 80% of the votes and becomes dominant in the majority vote, the receiving unit 141 receives information indicating that the degree of certainty is about 80%. In addition, for example, when the degree of certainty is low (when the degree of certainty of the answer that has become dominant in the majority vote is low), the receiving unit 141 receives information indicating that it is better to analyze again by changing or adding a question. In addition, for example, when the degree of certainty is 50% (the number of votes is the same or the same score in the majority vote) and the two answers are tied for first place, the receiving unit 141 receives information indicating that either of the two is possible.

[0042] (Transmitter 142) The transmitting unit 142 transmits, for example, selection information selected to confirm the occurrence of a predetermined situation. For example, the transmitting unit 142 transmits selection information of a target image. The transmitting unit 142 also transmits evaluation information on the information on the estimated result of the occurrence of a predetermined situation received by the receiving unit 141. For example, the transmitting unit 142 transmits evaluation information made by a user on the information on the estimated result of the occurrence of a predetermined situation.

[0043] [Configuration of information processing device] Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 6, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0044] (Communication unit 110) The communication unit 110 is realized by, for example, a NIC etc. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the terminal device 10 etc. via the network N.

[0045] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6, the storage unit 120 has a question information storage unit 121 and a correct answer image data storage unit 122.

[0046] The question information storage unit 121 stores information about a plurality of questions that are predetermined in association with a predetermined situation. The question information storage unit 121 may store information about a plurality of questions generated by a sentence generation model such as a GPT model. For example, the question information storage unit 121 may store information about weighting for each of the plurality of questions. For example, when the weighting for each of the plurality of questions is changed, the question information storage unit 121 may update the information to the weighting after the change (update the information stored in the question information storage unit 121). Here, FIG. 7 shows an example of the question information storage unit 121 according to the embodiment. As shown in FIG. 7, the question information storage unit 121 has items such as "Predetermined situation ID", "Question", "Weighting", and "Evaluation".

[0047] The "predetermined situation ID" indicates identification information for identifying the predetermined situation. The "question" indicates the content of each question of the multiple questions. The "weighting" indicates the weighting for each question of the multiple questions. For example, information indicating the transition of the weighting may be stored. The "evaluation" indicates an evaluation based on the answer result for each question of the multiple questions (evaluation for each question of the multiple questions). In the example shown in FIG. 7, conceptual information such as "evaluation #1" and "evaluation #2" is stored in the "evaluation", but in reality, information such as evaluations such as "good" / "bad", "90 points" / "80 points" / "70 points" / "60 points", "A rank" / "B rank" / "C rank" is stored. For example, information indicating the history of the evaluation may be stored. Then, the weighting for each question of the multiple questions may be determined based on this evaluation.

[0048] The correct image data storage unit 122 stores information about correct image data for evaluating a plurality of questions. For example, the correct image data storage unit 122 stores image data in which the occurrence of a predetermined situation is known (i.e., stores image data indicating the occurrence of a known predetermined situation). Here, FIG. 8 shows an example of the correct image data storage unit 122 according to the embodiment. As shown in FIG. 8, the correct image data storage unit 122 has items such as "predetermined situation ID" and "correct image data".

[0049] The "predetermined situation ID" indicates identification information for identifying a predetermined situation. The "correct image data" indicates correct image data for evaluating a plurality of questions. In the example shown in FIG. 8, conceptual information such as "correct image data #1" and "correct image data #2" is stored in the "correct image data", but in reality, information such as image data is stored. In addition, for example, the URL where the image data is located, or a file path name indicating the storage location, may be stored.

[0050] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU, an MPU, or the like, executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also realized, for example, by an integrated circuit such as an ASIC or an FPGA.

[0051] 6, the control unit 130 has an acquisition unit 131, a generation unit 132, a determination unit 133, an estimation unit 134, and a provision unit 135, and realizes or executes the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 6, and may be other configurations as long as they perform the information processing described below.

[0052] (Acquisition part 131) The acquiring unit 131 acquires various types of information from the storage unit 120. In addition, the acquiring unit 131 stores the acquired various types of information in the storage unit 120.

[0053] The acquisition unit 131 acquires various pieces of information from an external information processing device. The acquisition unit 131 acquires various pieces of information from other information processing devices such as the terminal device 10.

[0054] The acquisition unit 131 acquires, for example, a target image. For example, the user selects an image (which may be a video) that is to be checked for occurrence of a predetermined situation, and the acquisition unit 131 acquires the target image based on the user's selection.

[0055] The acquiring unit 131 acquires, for example, a plurality of questions inquiring about the occurrence of a predetermined situation (i.e., acquires a plurality of questions inquiring about the occurrence of a predetermined situation regarding the target image). For example, the acquiring unit 131 acquires a plurality of questions that are predetermined in association with a predetermined situation. Also, for example, when a user directly sets a plurality of questions, the acquiring unit 131 acquires a plurality of questions set by the user.

[0056] The acquisition unit 131 acquires, for example, a model for acquiring a plurality of questions. For example, the acquisition unit 131 acquires a sentence generation model (for example, a GPT model, etc.) that has been trained using data published on the Internet, etc. Then, the acquisition unit 131 acquires, for example, a plurality of questions generated by such a sentence generation model. For example, the acquisition unit 131 acquires a plurality of questions generated by inputting a predetermined prompt (for example, a prompt including an instruction sentence that instructs to generate a question with the same content as the predetermined question, etc.) to the sentence generation model.

[0057] (Generation unit 132) The generation unit 132 generates an answer result for each of the multiple questions by, for example, inputting the multiple questions and the target image acquired by the acquisition unit 131 to the VQA for each question (by inputting a combination of each question of the multiple questions and the target image for each combination to the VQA). For example, when generating an answer result with two choices such as "YES" or "NO" (for example, when inputting a question generated using a prompt instructing to answer with two choices such as "YES" or "NO" to the VQA), the generation unit 132 generates an answer result of either "YES" or "NO".

[0058] (Judgment unit 133) The determination unit 133 determines whether or not the majority vote of the VQA answers to each question of the multiple questions satisfies a predetermined condition based on the answer result generated by the generation unit 132, for example. If the majority vote satisfies the predetermined condition, for example, the determination unit 133 determines that the occurrence of a predetermined situation (for example, a state of a predetermined target) is estimated based on the answer result generated by the generation unit 132. On the other hand, if the majority vote does not satisfy the predetermined condition, for example, the determination unit 133 changes at least one of the multiple questions that are the subject of the majority vote, re-inputs the changed question to the VQA, and performs a determination process again by majority vote. Note that at this time, the process of changing at least one of the multiple questions that are the subject of the majority vote may be performed by the estimation unit 134 described later. In this way, the determination unit 133 repeats the process of changing the question and making a determination by majority vote until a determination result that satisfies the predetermined condition is obtained, for example. In addition, the determination unit 133 performs a determination process, for example, taking into consideration weighting for each question of the multiple questions. For example, the determination unit 133 determines whether or not a score based on the answer result and the weighting satisfies a predetermined condition, and if the score satisfies the predetermined condition, determines to infer the occurrence of a predetermined situation based on the answer result.

[0059] (Estimation part 134) The estimation unit 134 estimates the occurrence of a predetermined situation, for example, according to an answer result to each of the multiple questions acquired by the acquisition unit 131. For example, the estimation unit 134 estimates the occurrence of a predetermined situation according to an answer result to each question generated by inputting each question and a target image for each of the multiple questions to a model (e.g., VQA) that generates an answer to a question based on an image when an image and a question are input.

[0060] The estimation unit 134 estimates the occurrence of the predetermined situation, for example, when the determination unit 133 determines that the majority vote satisfies a predetermined condition. Furthermore, when the determination unit 133 determines that the majority vote does not satisfy the predetermined condition, the estimation unit 134 determines whether or not the majority vote of the answers to each of the multiple questions including the changed question changed by the determination unit 133 satisfies the predetermined condition, and then estimates the occurrence of the predetermined situation when the determination unit 133 determines that the majority vote satisfies the predetermined condition.

[0061] The estimation unit 134 estimates the occurrence of a specified situation when the determination unit 133 determines that the score calculated based on the answer result to each question of the multiple questions and the weighting of each question of the multiple questions satisfies a specified condition, for example, taking into account the weighting of each question of the multiple questions.

[0062] (Provider 135) The providing unit 135 provides, for example, information on the occurrence of a predetermined situation estimated by the estimation unit 134. That is, the providing unit 135 provides, for example, information on the estimation result of the occurrence of a predetermined situation. In addition, the providing unit 135 provides, for example, information indicating the accuracy of the occurrence of a predetermined situation. For example, the providing unit 135 provides information indicating the accuracy determined based on the answer results (majority vote of answer results) to each question of a plurality of questions. For example, the providing unit 135 provides information indicating that the accuracy is about 80% when either one (for example, one of two options of "YES" or "NO") receives 80% of the votes and becomes dominant in the majority vote. In addition, for example, the providing unit 135 provides information indicating that it is better to analyze again by changing or adding a question when the accuracy is low (when the accuracy of the answer that became dominant in the majority vote is low). In addition, for example, the providing unit 135 provides information indicating that either one is possible when the accuracy is 50% (the number of votes is the same or the score is the same in the majority vote) and the first place is tied.

[0063] [Information processing flow] Next, the procedure of information processing by the information processing system 1 according to the embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the procedure of information processing by the information processing system 1 according to the embodiment.

[0064] As shown in FIG. 9, the information processing device 100 acquires a target image (step S201).

[0065] The information processing device 100 acquires a plurality of questions inquiring about the occurrence of a predetermined situation regarding the target image (step S202).

[0066] The information processing device 100 generates an answer result for each question by inputting a plurality of questions and a target image to the VQA for each question (step S203).

[0067] The information processing device 100 determines whether the answer to each question satisfies a predetermined condition (step S204).

[0068] When it is determined that the answer results to each question satisfy a predetermined condition (step S204; YES), the information processing device 100 estimates the occurrence of a predetermined situation (step S205).

[0069] On the other hand, when the information processing device 100 determines that the answer results to each question do not satisfy the predetermined condition (step S204; NO), it changes at least one of the multiple questions (step S206). In this case, the information processing device 100 returns to step S203 and inputs the multiple questions including the changed question and the target image to the VQA again for each question to generate an answer result to each question.

[0070] [Other system configuration examples] The above-described configuration of the information processing system 1 is merely an example, and the information processing system 1 may have any device configuration as long as the desired processing is possible. For example, the information processing system 1 may not have the terminal device 10. Furthermore, the various processes performed by the information processing device 100 may be executed by any device included in the information processing system 1, not limited to the information processing device 100. For example, at least a part of the various processes performed by the information processing device 100 (e.g., the generation unit 132, the determination unit 133, etc.) may be executed by the terminal device 10. In this way, the various processes performed by the information processing device 100 may be distributed and processed by multiple devices included in the information processing system 1.

[0071] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as well as the scope and gist of the invention. In addition, these embodiments and their modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0072] 1. Information Processing Systems 10 Terminal Equipment 11 Communications Department 12 Input section 13 Output section 14 Control section 100 Information processing device 110 Communications Department 120 Storage section 121 Question information storage unit 122 Correct image data storage unit 130 Control section 131 Acquisition Department 132 Generation part 133 Judgment section 134 Estimation Department 135 Provision Department 141 Receiving unit 142 Transmitter N Network

Claims

1. an acquisition unit that acquires a plurality of questions inquiring about the occurrence of a predetermined situation; an estimation unit that estimates the occurrence of the predetermined situation according to answers to each of the plurality of questions acquired by the acquisition unit; 13. An information processing device comprising:

2. The estimation unit is A model that generates an answer to a question based on an image when an image and a question are input, and the occurrence of the predetermined situation is predicted according to the answer result to each question generated by inputting each question and a target image for each of the plurality of questions.

2. The information processing apparatus according to claim 1,

3. The estimation unit is Based on a majority vote of answers to each question of the plurality of questions, if the majority vote satisfies a predetermined condition, the occurrence of the predetermined situation is predicted.

2. The information processing apparatus according to claim 1,

4. The estimation unit is If a majority vote of answers to each of the plurality of questions does not satisfy a predetermined condition, at least one of the plurality of questions is changed, and if a majority vote of answers to each of the plurality of questions including the changed question satisfies the predetermined condition, the occurrence of the predetermined situation is predicted.

2. The information processing apparatus according to claim 1,

5. The acquisition unit is Acquire the plurality of questions that are predetermined in association with the predetermined situation.

2. The information processing apparatus according to claim 1,

6. The acquisition unit is Obtaining the plurality of questions generated by inputting prompts to the GPT model that instruct the model to generate questions similar to the given question.

2. The information processing apparatus according to claim 1,

7. The acquisition unit is Acquiring the plurality of questions generated based on the prompt instructing the number of questions in the plurality of questions to be equal to or less than a predetermined threshold.

7. The information processing apparatus according to claim 6,

8. The acquisition unit is Obtaining the plurality of questions generated based on the prompt, the prompt instructing that any of the plurality of questions generated include a predetermined word or expression.

7. The information processing apparatus according to claim 6,

9. The acquisition unit is Acquire the plurality of questions generated based on the prompt that instructs the user to obtain an expected answer result as an answer to each question of the plurality of questions generated.

7. The information processing apparatus according to claim 6,

10. The estimation unit is Predicting the occurrence of the predetermined situation based on a score of an answer to each question, the score being calculated based on an answer result to each question of the plurality of questions and a weighting of each question.

2. The information processing apparatus according to claim 1,

11. The estimation unit is Predicting the occurrence of the predetermined situation based on the score of the answer to each of the plurality of questions, the score being calculated based on the weighting of each of the plurality of questions, the score being determined based on an evaluation of each of the plurality of questions based on an answer result to each of the plurality of questions.

11. The information processing apparatus according to claim 10,

12. The estimation unit is The occurrence of the predetermined situation is estimated based on the score of the answer to each of the questions, which is determined based on an evaluation of each of the questions based on an answer result to each of the plurality of questions using an image prepared in advance as correct answer data as a target image, and is calculated based on the weighting of each of the questions.

11. The information processing apparatus according to claim 10,

13. 1. A computer-implemented information processing method, comprising: acquiring a plurality of questions inquiring about the occurrence of a predetermined situation; an estimation step of estimating the occurrence of the predetermined situation according to answers to each of the plurality of questions acquired in the acquisition step; 13. An information processing method comprising:

Images