Information processing program, information processing method, and information processing device

Abstract

To provide an information processing program that can reduce the time and effort required for a user to reach desired information.SOLUTION: An information processing program according to one embodiment causes a computer to execute an information processing method including the steps of: acquiring a first question according to an input query; estimating an answer to the first question for each of multiple contents to generate a first answer estimation result including the answer to the first question for each of the multiple contents; and classifying the multiple contents into multiple first groups by using the first answer estimation result to generate the first search result indicating the content belonging to the first group for each of the multiple first groups.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] There is a need to search for scenes that meet or do not meet specific criteria, such as "wearing a helmet," in on-site videos taken with cameras installed at manufacturing sites, maintenance inspection sites, etc. In addition to scene searches for on-site videos, there is also a need to search for scenes that meet or do not meet specific criteria from videos on video sharing sites.

[0003] There are services that search for images using keywords as queries, and services that search for images similar to the input image when an image is input. The image search technology used in these services can be applied to searching for scenes within videos by applying it to video frame images.

[0004] However, conventional techniques such as keyword-based image search and similar image search cannot search for images or scenes that do not meet specific conditions. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7014232 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to provide an information processing program, an information processing method, and an information processing device that can reduce the effort required for a user to reach desired information such as a desired image or scene. [Means for solving the problem]

[0007] An information processing program according to one embodiment causes a computer to execute an information processing method including: acquiring a first question corresponding to an input query; estimating an answer to the first question for each of a plurality of pieces of content to generate a first answer estimation result including the answer to the first question for each of the plurality of pieces of content; and using the first answer estimation result to classify the plurality of pieces of content into a plurality of first groups to generate a first search result indicating the content belonging to each of the plurality of first groups. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an information processing system according to a first embodiment. [Figure 2] FIG. 3 is a diagram showing information stored in a question storage unit according to the first embodiment. [Figure 3] FIG. 4 is a diagram showing information stored in an estimation result storage unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing information stored in an estimation result storage unit according to the first embodiment. [Figure 5] FIG. 4 is a diagram for explaining processing in a grouping unit according to the first embodiment. [Figure 6] FIG. 4 is a diagram for explaining processing in a grouping unit according to the first embodiment. [Figure 7] FIG. 3 is a diagram showing a search screen according to the first embodiment. [Figure 8] FIG. 3 is a diagram showing a search screen according to the first embodiment. [Figure 9] FIG. 3 is a diagram showing a search screen according to the first embodiment. [Figure 10] FIG. 3 is a diagram showing a search screen according to the first embodiment. [Figure 11] FIG. 2 is a block diagram showing the hardware configuration of the server shown in FIG. 1. [Figure 12A] 3 is a flowchart showing an image search method according to the first embodiment. [Figure 12B]3 is a flowchart showing an image search method according to the first embodiment. [Figure 13] FIG. 12C is a diagram for explaining the image search method shown in FIGS. 12A and 12B. [Figure 14] FIG. 10 is a diagram showing a search screen generated by a display control unit according to the second embodiment. [Figure 15] FIG. 10 is a diagram showing an answer estimation result according to the second embodiment. [Figure 16] FIG. 10 is a diagram showing information stored in an estimation result storage unit according to the second embodiment. [Figure 17] FIG. 10 is a diagram showing a search screen according to the second embodiment. [Figure 18] 10 is a flowchart showing a video adding method according to the second embodiment. [Figure 19A] 10 is a flowchart showing a moving image scene search method according to a second embodiment. [Figure 19B] 10 is a flowchart showing a moving image scene search method according to a second embodiment. [Figure 20] FIG. 10 is a block diagram showing an information processing system according to a third embodiment. [Figure 21] FIG. 11 is a diagram showing information stored in a question storage unit according to the third embodiment. [Figure 22] FIG. 11 is a diagram showing information stored in a question storage unit according to the third embodiment. [Figure 23] FIG. 11 is a diagram showing a search screen according to the third embodiment. [Figure 24] 10 is a flowchart showing a method for presenting narrowed-down candidates according to the third embodiment. [Figure 25] FIG. 1 is a diagram illustrating an application of information processing technology according to an embodiment. [Figure 26] FIG. 1 is a diagram illustrating an application of information processing technology according to an embodiment. [Figure 27] FIG. 13 is a diagram showing a screen according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment will be described with reference to the drawings. In the embodiment described below, images are used as content to be searched. The content to be searched may be other data such as text or audio.

[0010] [First embodiment] Fig. 1 schematically shows an information processing system 100 according to the first embodiment. As shown in Fig. 1, the information processing system 100 includes a client 110 and a server 120 as an information processing device according to the first embodiment. The client 110 and the server 120 are connected to a communication network that may include the Internet, and communicate with each other via the communication network.

[0011] The server 120 implements an image search function that performs an image search in response to a query input, and provides an image search service to the client 110. In one example, the image search function is implemented in the server 120 as a Web application, and the client 110 accesses the image search service using a Web browser. When the client 110 accesses the image search service, a search screen including a text input form for inputting a query in text format is displayed on the Web browser. When the user inputs a query in the text input form, the server 120 receives the query input by the user from the client 110 (hereinafter also referred to as an input query), and performs an image search using the received input query to obtain search results. The server 120 adds the search results to the search screen to present the search results to the user through the client 110.

[0012] The image search function is implemented in the server 120 in the example shown in Fig. 1, but may also be implemented in a local computer terminal such as the client 110 shown in Fig. 1. When the image search function is implemented in a local computer terminal, the image search function may be realized as an application executed on the local computer terminal.

[0013] The client 110 is a computer terminal such as a personal computer (PC) or a smartphone. The client 110 includes an input unit 111 and an output unit 112. The input unit 111 includes an input device that allows a user to input information. Examples of the input device include a keyboard, a mouse, a touch panel, and a microphone. The input unit 111 is used to input a query. The output unit 112 includes an output device that outputs information to the user. Examples of the output device include a display device and a speaker. In the first embodiment, the output unit 112 includes a display device, and the display device is used to display a search screen generated by the server 120.

[0014] The server 120 includes a search target storage unit 121, a query input unit 122, a question acquisition unit 123, a question storage unit 124, an answer estimation unit 125, a model storage unit 126, an estimation result storage unit 127, a grouping unit 128, and a display control unit 129. Note that the server 120 is not limited to being implemented as a single device (e.g., a computer), and may be configured by multiple devices. The search target storage unit 121, the question storage unit 124, the model storage unit 126, and the estimation result storage unit 127 may be provided in an external device such as a cloud storage.

[0015] The search target storage unit 121 stores a plurality of images (specifically, still images) to be searched in association with an image ID. The image ID is identification information for identifying an image. For example, if the image ID is 1 to N, I N I images are stored in the search target storage unit 121.

[0016] The query input unit 122 is a user interface for inputting a query. When the query is in text format, the query input unit 122 may be the text input form described above. The text input as the query may be either a keyword or a sentence. The sentence may be any of an assertion, an interrogative sentence, and a negative sentence.

[0017] The question acquiring unit 123 acquires a question corresponding to a query input by a user. Specifically, the question acquiring unit 123 converts the input query into a question in a predetermined format. The predetermined format refers to a format that can be handled by the answer estimation unit 125 at a subsequent stage. In the first embodiment, the format that can be handled by the answer estimation unit 125 is text, and the question acquiring unit 123 converts the input query into a question sentence (a question in text format).

[0018] Whether an input query is a keyword or a sentence can be determined by performing morphological analysis on the input query and checking the distribution of parts of speech. Whether a sentence is an assertion, question, or negation can be determined by using parts of speech and rules related to them, such as whether or not there is an interrogative word or a negation expression. Other determination methods, such as using a model trained by machine learning, may also be used.

[0019] If the input query is a question, the question acquiring unit 123 uses the input query as it is as a question.

[0020] If the input query is an assertive sentence or a negative sentence, the question acquisition unit 123 converts the input query into an interrogative sentence and uses it as a question. A method for converting an assertive sentence or a negative sentence into an interrogative sentence may be to add an interrogative final particle "ka" to the end of the sentence. For example, by adding an interrogative final particle "ka" to the assertive sentence "Are you wearing a helmet?", the interrogative sentence "Are you wearing a helmet?" is obtained.

[0021] Another method for converting an assertive or negative sentence into a question is to perform morphological analysis on the assertive or negative sentence, converting parts of a given speech into interrogative words, and adding an interrogative particle at the end of the sentence. For example, if the adjective "yellow" in the assertive sentence "He is wearing a yellow helmet" is converted into an interrogative word and then the interrogative particle "ka" is added to the end of the sentence, the question "What kind of helmet are you wearing?" is obtained. Furthermore, if the noun "helmet" in this assertive sentence is converted into an interrogative word and then the interrogative particle "ka" is added to the end of the sentence, the question "What yellow thing are you wearing?" is obtained.

[0022] In the example above, the adjective "yellow" is converted to the interrogative "what kind," but the conversion destination can be changed depending on the content of the adjective. For example, by using ontology, it can be determined that the adjective "yellow" describes a color. If a rule is established that adjectives that describe color, such as "yellow," are converted to the interrogative "what color," the question "What color helmet are they wearing?" can be obtained. Similarly, if a rule is established that converts "number + counter particle" (such as 3) to "what + counter particle," the question "How many people are working" can be obtained from the sentence "Two people are working."

[0023] When the input query is a keyword, the question acquisition unit 123 identifies the class of the keyword using, for example, an ontology, and creates a question from the keyword using a template corresponding to the identified class. In this case, a template is prepared for each class. For example, suppose that tracing the ontology for the keyword "helmet" yields the class "safety equipment." In response to this, a rule is prepared that when the keyword class is "safety equipment," the template "Are you wearing (input)?" is used. As a result, the question "Are you wearing a helmet?" is obtained from the keyword "helmet." The template may be generated manually, or may be automatically generated based on keywords extracted from a corpus of questions.

[0024] As described above, there are multiple ways to obtain a question, and therefore, multiple questions may be obtained by converting an input query.

[0025] In the first embodiment, text is used as the query, but the query is not limited to text and may be an image or audio. When audio is input as the query, the question acquisition unit 123 may convert the audio into text using speech recognition and generate a question from the text. When an image is input as the query, the question acquisition unit 123 may generate a caption from the image using image captioning technology and generate a question from the caption.

[0026] Furthermore, the format that can be handled by the answer estimation unit 125 is not limited to text, but may also be audio, image, numerical value (e.g., vector), etc. When text is input as a query and the format that can be handled by the answer estimation unit 125 is audio, the question acquisition unit 123 may convert the input text into audio using text-to-speech (TTS). When audio or an image is input as a query and the format that can be handled by the answer estimation unit 125 is audio or image, the question acquisition unit 123 may use the input audio or image as the question as is. When text, audio, or an image is input as a query and the format that can be handled by the answer estimation unit 125 is vector, the question acquisition unit 123 may convert the input text, audio, or image into a vector by encoding or embedding the input text, audio, or image.

[0027] The question acquisition unit 123 sends the question to the answer estimation unit 125 and stores it in the question storage unit 124. The question storage unit 124 stores a question list in which multiple questions can be registered. For example, as shown in FIG. 2, the question storage unit 124 stores the question acquired by the question acquisition unit 123 in association with a question ID. The question ID is identification information for identifying the question.

[0028] When the question acquisition unit 123 acquires a question, it determines whether a question identical to the acquired question exists in the question storage unit 124. If a question identical to the acquired question does not exist in the question storage unit 124, the question acquisition unit 123 adds the acquired question to a question list. Specifically, the question acquisition unit 123 assigns a question ID to the acquired question and stores the acquired question in the question storage unit 124 in association with the assigned question ID. If a question identical to the acquired question exists in the question storage unit 124, the question acquisition unit 123 omits adding the acquired question to the question list. The identity of the question sentences may be determined, for example, by calculating the literal similarity between the question sentences. For example, if the calculated similarity exceeds a predetermined threshold, the question acquisition unit 123 determines that the question sentences are identical to each other, and if the calculated similarity is equal to or less than the predetermined threshold, the question acquisition unit 123 determines that the question sentences are not identical to each other. Alternatively, the identity of the question sentences may be determined by encoding or embedding the question sentences to obtain vectors and calculating the similarity between the vectors. In addition, the identity of the questions may be determined by performing morphological analysis on the questions and calculating the similarity between words obtained from one question and words obtained from the other question using word2vec or the like.

[0029] When determining the identity of a question sentence based on the similarity between vectors, the result (vector) obtained by encoding or embedding the question sentence may be stored in the question storage unit 124 instead of the question sentence itself. This eliminates the need to perform encoding or embedding every time the identity with the question sentence obtained by the question acquisition unit 123 is determined.

[0030] The question storage unit 124 may store not only questions resulting from a query input by a single user, but also questions resulting from queries input by other users. For example, the server 120 provides an image search service to multiple clients including the client 110. The question acquisition unit 123 adds questions corresponding to the input queries received from each client to a question list.

[0031] The answer estimation unit 125 receives a question from the question acquisition unit 123. The answer estimation unit 125 estimates an answer to the question for each of a plurality of images stored in the search target storage unit 121 to obtain an answer estimation result. The answer estimation result includes an answer to the question for each image. Typically, the answer estimation unit 125 performs answer estimation using a trained model configured to estimate an answer based on a question about an image. The trained model assumed in this embodiment is a model related to VQA (Visual Question Answering), which derives an answer when an image and a question related to the image are input. Hereinafter, the model related to VQA will also be referred to as a VQA model. The trained model is stored in the model storage unit 126.

[0032] The answer estimation unit 125 sends the answer estimation result to the grouping unit 128 and stores the answer estimation result in the estimation result storage unit 127. As shown in FIG. 3, the estimation result storage unit 127 stores the answer to the question about each image in association with the image ID of the image and the question ID of the question. A record including the image ID, question ID, and answer is associated with the answer ID. The answer ID is identification information for identifying the record (a set of the image ID, question ID, and answer in the example shown in FIG. 3).

[0033] The answer estimation unit 125 may output the confidence level of the answer along with the answer. In this case, the answer estimation result includes the answer to the question for each image and the confidence level of the answer. The confidence level is an index indicating the likelihood of the answer. Furthermore, the answer estimation unit 125 may output multiple answers to the question and their respective confidence levels for each image. For example, if the question can be answered with "yes," "no," or "I don't know," the answer estimation unit 125 outputs the confidence level of the answer "yes," the confidence level of the answer "no," and the confidence level of the answer "I don't know" for each image. In this case, as shown in FIG. 4, each record stored in the estimation result storage unit 127 further includes the confidence level. The confidence level information can be used for grouping by the grouping unit 128 and for selecting a representative image by the display control unit 129, which will be described later.

[0034] The question storage unit 124 and the estimation result storage unit 127 are used to shorten the time required for answer estimation when the question acquisition unit 123 acquires a question that is identical to a question that has been processed in the past. If the question acquired by the question acquisition unit 123 is identical to a question stored in the question storage unit 124, the answer estimation unit 125 acquires an answer estimation result associated with the question ID of that question from the estimation result storage unit 127. This makes it possible to omit re-executing answer estimation.

[0035] The grouping unit 128 groups images using the answer estimation result. Specifically, the grouping unit 128 generates the same number of answer groups as the number of answer types included in the answer estimation result, and classifies images into these answer groups. For example, if the answer to the question "Are you wearing a helmet?" is "Yes," "No," or "I don't know," the grouping unit 128 creates a first answer group corresponding to the answer "Yes," a second group corresponding to the answer "No," and a third answer group corresponding to the answer "I don't know." Images with the answer "Yes" belong to the first answer group, images with the answer "No" belong to the second answer group, and images with the answer "I don't know" belong to the third answer group. For example, if the answer estimation result shown in FIG. 5 is obtained, images 1, 2, and 6 with the answer "Yes" belong to the first answer group, images 3 and 5 with the answer "No" belong to the second answer group, and image 5 with the answer "I don't know" belongs to the third answer group. Here, image i represents the image whose image ID is i.

[0036] When the answer estimation unit 125 outputs multiple answers and their confidence levels for each image, the grouping unit 128 may group the images using the answer with the highest confidence level. The grouping unit 128 may also assign each image to multiple answer groups. In this case, the images in each answer group may be ranked according to their confidence levels. For example, when the answer estimation result shown in FIG. 6 is obtained, each image belongs to both answer groups. In the answer group for the answer "yes," the images are sorted in the order of image 1, image 2, and image 3, and in the answer group for the answer "no," the images are sorted in the order of image 3, image 2, and image 1. The grouping unit 128 may exclude an image whose confidence level is below a predetermined threshold from the answer group corresponding to that answer. For example, if the threshold is set to 0.2, in the example shown in FIG. 6, image 1 would not be included in the answer group for the answer "no."

[0037] The display control unit 129 selects a representative image for each answer group and displays the representative image together with the answer corresponding to the answer group. The representative image is selected from the images belonging to the answer group. For example, the display control unit 129 may select the image with the smallest image ID as the representative image for each group. In the example shown in FIG. 5, image 1 is selected as the representative image for the answer group of "Yes," image 3 is selected as the representative image for the answer group of "No," and image 4 is selected as the representative image for the answer group of "I don't know." Two or more representative images may be selected for each answer group.

[0038] The display control unit 129 generates and outputs a search screen to be displayed to the user. The display control unit 129 generates a search screen showing search results including representative images of each answer group and answers corresponding to each answer group. For example, the display control unit 129 adds the search results to the search screen.

[0039] As described above, if the query is not a question, the question acquiring unit 123 converts the query into a question. In this case, the display control unit 129 may further display the question obtained by converting the query. Furthermore, when displaying the question, the display control unit 129 may also hide the display of the text input form.

[0040] Fig. 7 schematically shows a search screen including search results when the query is a question, and Fig. 8 schematically shows a search screen including search results when the query is a keyword. Fig. 7 displays a query (question) entered in a text input form 701 and search results 702. Fig. 8 displays a search result 802 including the query entered in a text input form 801 and a question obtained by converting the query. Note that when displaying the question obtained by converting the query, the display of text input form 801 may be hidden. In the examples shown in Figs. 7 and 8, the search results displayed include answers, the number of images belonging to the answer groups corresponding to each answer, and representative images of the answer groups.

[0041] When multiple questions are obtained from a query, the search screen may be configured to allow the user to select one of the questions. For example, suppose that the sentence "two people are working" is input as a query, and the question acquisition unit 123 acquires two questions, "how many people are working" and "are two people working?" In this case, as shown in FIG. 9, the search screen includes a dropdown 901 that allows the user to select one of the questions. When the user selects one of the questions in the dropdown 901, search results 902 corresponding to the selected question are displayed.

[0042] The search screen may be configured to allow the user to select one of multiple answers (i.e., multiple answer groups) to narrow down the images. For example, as shown in FIG. 10 , when the user selects the answer "Yes," an additional text input form 1001 is displayed so that the user can input a further query. When a further query is input into the additional text input form 1001, the answer estimation unit 125 performs answer estimation on images belonging to the answer group of the answer "Yes," the grouping unit 128 performs grouping based on the answer estimation result obtained by the answer estimation unit 125, and the display control unit 129 displays the search results obtained by the grouping.

[0043] Display control unit 129 may be realized by any method. For example, display control unit 129 may be implemented in HTML (Hypertext Markup Language) or JavaScript (registered trademark) that is a response of a Web application and is displayed on the browser of client 110. Furthermore, when the image search function is implemented in the form of an application on a local computer terminal, display control unit 129 may be implemented in the source code of the application.

[0044] 11 schematically illustrates an example of the hardware configuration of the server 120. As illustrated in FIG. 11, the server 120 includes a CPU (Central Processing Unit) 1101, a RAM (Random Access Memory) 1102, a storage device 1103, and a communication interface 1104.

[0045] The CPU 1101 is an example of a general-purpose processor capable of executing programs, and controls the RAM 1102, the storage device 1103, and the communication interface 1104, and performs various types of information processing.

[0046] The RAM 1102 includes a volatile memory and is used as a work area for the CPU 1101. The storage device 1103 includes a nonvolatile memory such as a hard disk drive (HDD) or a solid state drive (SSD), and stores programs including an information processing program, data, and the like. The CPU 1101 operates in accordance with a program stored in the storage device 1103. For example, when the information processing program is executed by the CPU 1101, the CPU 1101 performs the process described below regarding the server 120. Specifically, the CPU 1101 functions as a query input unit 122, a question acquisition unit 123, an answer estimation unit 125, a grouping unit 128, and a display control unit 129 in accordance with the information processing program. The storage device 1103 functions as a search target storage unit 121, a question storage unit 124, a model storage unit 126, and an estimation result storage unit 127.

[0047] The communication interface 1104 is an interface for communicating with an external device. The CPU 1101 communicates with the client 110 via the communication interface 1104.

[0048] A program such as an image search program may be provided to the server 120 in a state where it is stored on a computer-readable recording medium. In this case, the server 120 has a drive for reading data from the recording medium and acquires the program from the recording medium. Examples of recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memories. The program may also be distributed via a communication network. Specifically, the program may be stored on a server on the communication network, and the server 120 may download the program from the server.

[0049] Note that a dedicated processor such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used instead of or in addition to a general-purpose processor. The processing circuit refers to a general-purpose processor, a dedicated processor, or a combination of a general-purpose processor and a dedicated processor.

[0050] Next, the operation of the server 120 will be described.

[0051] 12A and 12B schematically show an example of the procedure of the image search method according to the first embodiment. The image search method shown in FIGS.

[0052] 12A, the question acquiring unit 123 acquires a question corresponding to a query input by a user. For example, the question acquiring unit 123 converts the text input by the user into a question in text format.

[0053] In step S1202, the question acquiring unit 123 determines whether or not the same question as the question acquired in step S1201 exists in the question storage unit 124.

[0054] If the question identical to the question obtained in step S1201 exists in the question storage unit 124 (step S1202; Yes), the flow proceeds to step S1203. In step S1203, the question acquiring unit 123 extracts the question ID of the question identical to the question obtained in step S1201 from the question storage unit 124, and notifies the answer estimation unit 125 of the extracted question ID and the question obtained in step S1201.

[0055] On the other hand, if the question identical to the question obtained in step S1201 does not exist in the question storage unit 124 (step S1202; No), the flow proceeds to step S1204. In step S1204, the question acquiring unit 123 assigns a question ID to the question obtained in step S1201, and stores the question obtained in step S1201 in association with the assigned question ID in the question storage unit 124. Furthermore, the question acquiring unit 123 notifies the answer estimation unit 125 of the assigned question ID and the question obtained in step S1201.

[0056] Here, the question ID extracted in step S1203 or the question ID assigned to the question in step S1204 is set to i. Below, a question with question ID i will be referred to as question i.

[0057] For example, when a user inputs the query "helmet" for the first time, the question acquisition unit 123 converts the input query to obtain the question "Are you wearing a helmet?". Since no questions are initially stored in the question storage unit 124, the question acquisition unit 123 generates a new question ID=1 and stores the question in the question storage unit 124 in association with the question ID=1.

[0058] Suppose that another user or the same user subsequently inputs the query "Are you wearing a helmet?" The question acquisition unit 123 converts the input query to obtain two questions, "Are you wearing a helmet?" and "Are you wearing a helmet?" The question acquisition unit 123 individually checks whether a question identical to the two questions exists in the question storage unit 124. Since the question text "Are you wearing a helmet?" is stored in the question storage unit 124, the question acquisition unit 123 recognizes that a question identical to the two questions exists in the question storage unit 124. The question acquisition unit 123 sets the question ID=1 of the question "Are you wearing a helmet?" in the question storage unit 124 as the question ID to be used in subsequent processing.

[0059] When the flow proceeds from step S1203 or step S1204 to step S1205, the answer estimation unit 125 creates an empty answer estimation result. For example, the answer estimation unit 125 creates an answer estimation result that includes an image ID but no answer, as shown in the left part of Fig. 13. Thereafter, the flow proceeds to step S1206 in Fig. 12B.

[0060] In step S1206 of FIG. 12B, the response estimation unit 125 sets the variable j to 1.

[0061] In step S1207, the response estimation unit 125 determines whether the variable j is N I Determine whether variable j is greater than N I The larger the answer, the more likely the answer estimate is N I This refers to the fact that it contains answers for all images.

[0062] Variable j is N I If it is equal to or less than i (step S1207; No), the flow proceeds to step S1208. In step S1208, the answer estimation unit 125 determines whether or not a record having a question ID equal to i and an image ID equal to j exists in the estimation result storage unit 127. If a record having a question ID equal to i and an image ID equal to j exists (step S1208; Yes), the flow proceeds to step S1209, and if a record having a question ID equal to i and an image ID equal to j does not exist (step S1208; No), the flow proceeds to step S1210.

[0063] When the flow proceeds from step S1208 to step S1209, the answer estimation unit 125 extracts an answer from a record whose question ID is equal to i and whose image ID is equal to j from the estimation result storage unit 127. Then, in step S1213, the answer estimation unit 125 adds the extracted answer to the answer estimation result as an answer for image j.

[0064] When the flow proceeds from step S1208 to step S1210, the answer estimation unit 125 retrieves image j from the search target storage unit 121. In step S1211, the answer estimation unit 125 estimates an answer to question i for image j. For example, the answer estimation unit 125 inputs image j and question i to a VQA model and obtains an answer output from the VQA model. In step S1212, the answer estimation unit 125 associates the estimated answer with the image ID and question ID and stores them in the estimation result storage unit 127. Thereafter, in step S1213, the answer estimation unit 125 adds the estimated answer to the answer estimation result as an answer for image j.

[0065] When the process shown in step S1213 is completed, the flow proceeds to step S1214, where the answer estimation unit 125 increments the variable j by 1. After that, the flow returns to step S1207. I A series of processes shown in steps S1207 to S1214 are repeated until the answer estimation result exceeds N. I It will contain answers for all images.

[0066] Variable j is N I If it is greater than N (step S1207; Yes), the flow proceeds to step S1215. In step S1215, the grouping unit 128 uses the answer estimation result to I For example, the grouping unit 128 creates groups equal in number to the number of answer types included in the answer estimation result, and assigns each image to a group corresponding to the answer for that image.

[0067] In step S1216, the display control unit 129 selects a representative image from the images belonging to each answer group, and displays the search results in which the answers are associated with the representative image of the answer group.

[0068] As described above, in the first embodiment, the server 120 receives a query input by a user, acquires a question corresponding to the received query, estimates an answer to the question for each of a plurality of images to generate an answer estimation result, classifies the plurality of images into a plurality of answer groups using the answer estimation result, and generates search results indicating content belonging to each of the plurality of answer groups. In this way, the server 120 treats the input query as a perspective for searching for images. This allows the user to easily find a desired image. As a result, the effort required for the user to find the desired image can be reduced. For example, if a user inputs the keyword "helmet," not only images that match the sentence "wearing a helmet" but also images that do not match the sentence "wearing a helmet" will be presented to the user.

[0069] [Second embodiment] The second embodiment relates to a scene search for searching for a desired scene from a video. In the first embodiment, the search target is a group of images, but in the second embodiment, the search target is a video. The second embodiment corresponds to a case where the multiple images in the first embodiment are multiple frame images (also simply referred to as frames) included in a video. A video refers to a sequence of images accompanied by information indicating time or order, such as a video, animation, or a set of still images captured consecutively at a predetermined time interval. In the second embodiment, the information processing system 100 shown in FIG. 1 will also be referenced, and differences from the first embodiment will be mainly described, with similarities to the first embodiment being omitted where appropriate.

[0070] The server 120 implements a video scene search function for searching for a desired scene from a video and provides a video scene search service to the client 110. In one example, the video scene search function is implemented on the server 120 as a web application, and the client 110 accesses the video scene search service using a web browser. When the client 110 accesses the video scene search service, a search screen including a video selection form for selecting a video and a text input form for inputting a text query is displayed on the web browser. The user selects a video to be searched in the video selection form and inputs a query in text input form. The server 120 receives information indicating the video selected by the user and the input query from the client 110, and performs a video scene search on the video indicated by the received information using the input query indicated by the received information to obtain a video scene search result. The server 120 adds the video scene search results to the search screen, thereby presenting the video scene search results to the user via the client 110.

[0071] The search target storage unit 121 stores a plurality of moving images in association with a moving image ID. The moving image ID is identification information for identifying the moving image.

[0072] The query input unit 122 is a user interface that enables video selection and query input. For example, as shown in FIG. 14 , the query input unit 122 is a combination of a video selection form 1410 for selecting a video and a text input form 1420 for inputting a query in text format. The video selection form 1410 may be configured to select a video using a dropdown. Furthermore, if the search target storage unit 121 is provided in an external device, the video selection form 1410 may be configured to display a dialog for selecting a video file when clicked by the user. In addition to the video selection form 1410 and the text input form 1420, the search screen includes a video display area 1430 that displays a video selected by the user and a search result display area 1440 that displays search results. The video display area 1430 includes a play button 1431, a stop button 1432, and a seek bar 1433.

[0073] The answer estimation unit 125 receives a question from the question acquisition unit 123. The answer estimation unit 125 estimates an answer to the question for each of a plurality of frames included in a video selected by a user, and obtains an answer estimation result. The answer estimation result indicates the answer to the question for each of a plurality of frames, as shown in FIG. 15. In FIG. 15, a frame ID is identification information for identifying a frame. The target for answer estimation may be all frames constituting a video, or may be a plurality of frames extracted from a video by sampling. In the example shown in FIG. 15, the sampling rate is one frame per second.

[0074] The answer estimation unit 125 may perform answer estimation using a VQA model. In the second embodiment, the VQA model is configured to receive an image and a question as input and output an answer and basis information indicating a position on the image that is the basis for the answer. The basis information is also referred to as visual grounding. The basis information may be area information indicating a rectangular area on the image, such as the rectangle 1450 shown in FIG. 14. For example, the basis information includes the coordinates of the upper left position of the rectangular area, as well as the width and height of the rectangular area. The basis information may also include a score value for each pixel of the image, or an image displaying the score values ​​as a heat map. The basis information may be output for all answers. Furthermore, whether or not to output the basis information may be switched depending on the type of answer. For example, the basis information may be output when the answer is "yes" and not output when the answer is "no." As described in the first embodiment, the answer estimation unit 125 may output a confidence level for each answer.

[0075] The answer estimation unit 125 stores answer estimation results, including answers to questions for each frame, in the estimation result storage unit 127. As shown in FIG. 16 , each record in the estimation result storage unit 127 includes an answer ID, a video ID, time information, a question ID, an answer, and evidence information. The video ID is identification information for identifying a video. The time information indicates a time point in the video. The record may further include a frame ID. The frame ID has a one-to-one correspondence with a time point in the video. In the example shown in FIG. 16 , the evidence information is information for identifying a rectangular area on the image. In the notation (x, y, width, height), x and y represent the coordinates of the upper left position of the rectangular area, and width and height represent the width and height of the rectangular area. If the evidence information is a score map, the estimation result storage unit 127 may store the score map itself. If the evidence information is a heat map image of scores, the estimation result storage unit 127 may store the heat map image, for example, encoded using BASE64.

[0076] The display control unit 129 selects a representative image for each answer group, and displays the answers corresponding to the answer group together with the representative image in association with the time of the video.

[0077] The display control unit 129 may select a representative image using time information. Each group will have one or more frame sets, which are sets of consecutive frames. A long frame set indicates that a situation matching the answer is being displayed stably on the video. The display control unit 129 selects the frame set with the longest time length from the frame sets, and selects the frame with the median time in the selected frame set as the representative image.

[0078] Alternatively, the display control unit 129 may select a representative image using the basis information. For example, if the basis information is information for identifying a rectangular area in a frame, the display control unit 129 may select a frame with the largest rectangular area as the representative image.

[0079] Furthermore, the display control unit 129 may select a representative image by using a combination of time information and grounds information. For example, the display control unit 129 calculates a score for each frame by using the time information and grounds information, and selects the frame with the highest score as the representative image. If the time length of a frame set including a frame is L, the size (area) of a rectangular region indicated by the grounds information of the frame is S, and the score is s, the score s can be expressed, for example, by the following formula: s=a×L+b×S Here, a and b are constants that are determined in advance.

[0080] Furthermore, the display control unit 129 may select a representative image using the same method as that described in relation to the first embodiment. For example, the display control unit 129 may select the first frame as the representative image, or may select the frame with the highest degree of certainty as the representative image.

[0081] As shown in FIG. 14, the display control unit 129 colors the corresponding time portion on the seek bar 1433 in the same color as that used to display the answer group.

[0082] The answer estimation may be unstable, and the answers for some frames may vary from the surrounding frames. For example, if the sampling interval is 0.5 seconds, the answer may be "Yes" for frames from 0.0 to 5.0 seconds, "No" for frames from 5.5 seconds, and "Yes" for frames from 6.0 to 10.0 seconds. If the answers for frames in such a short period are different, it is possible that the VQA model is making an estimation error.

[0083] To correct the estimation error described above, the answer estimation unit 125 may smooth the answers included in the answer estimation result. Specifically, the answer estimation unit 125 identifies the most common type of answer among the answers for N consecutive frames and sets the identified answer as the answer for the central frame, while shifting the frame by one frame. For example, assuming N is 7 and referring back to the above example, the answer estimation unit 125 first checks the answers for seven frames from 0.0 to 3.0 seconds. Since all seven answers are "yes," the answer estimation unit 125 leaves the answer for the 1.5-second frame as "yes." Next, the answer estimation unit 125 checks the answers for seven frames from 0.5 to 3.5 seconds. Since all seven answers are "yes," the answer estimation unit 125 leaves the answer for the 2.0-second frame as "yes." Similarly, the answer estimation unit 125 leaves the answer for the 2.5-5.0-second frames as "yes." Next, the answer estimation unit 125 checks the answers for the seven frames from 4.0 to 7.0 seconds. Since there are six answers of "Yes" and one answer of "No," the answer estimation unit 125 corrects the answer for the frame at 5.5 seconds from "No" to "Yes."

[0084] The above-described smoothing method is an example, and any smoothing method may be used.

[0085] The answer estimation result includes the answer as well as the basis information. The display control unit 129 may display the basis information together with the representative image. For example, if the basis information indicates a rectangular area, as shown in FIG. 14, the display control unit 129 displays the rectangle specified by the basis information superimposed on the representative image. If the basis information indicates a score value for each pixel, the display control unit 129 may convert the score into a heat map image and display the heat map image superimposed on the representative image. Furthermore, if the basis information is a heat map image, the display control unit 129 may display the heat map image as is superimposed on the representative image. Alternatively, the display control unit 129 may display the basis information in response to a mouse hover over the representative image. Furthermore, the display control unit 129 may also display the basis information superimposed on the main body of the video.

[0086] FIG. 17 schematically illustrates an example of a search screen when one of the answer groups is selected. Specifically, FIG. 17 illustrates a state in which the user has selected the answer group with the answer "yes" on the search screen illustrated in FIG. 14. As illustrated in FIG. 17, the display control unit 129 leaves the display on the seek bar corresponding to the selected answer group and removes the display on the seek bar corresponding to answer groups other than the selected answer group. Furthermore, the display control unit 129 displays an additional text input form 1810 so that the user can input a further query. When a further query is input, the answer estimation unit 125 performs answer estimation on frames belonging to the answer group with the answer "yes," and the grouping unit 128 performs grouping based on the answer estimation result obtained by the answer estimation unit 125.

[0087] Next, the operation of the server 120 will be described.

[0088] It takes time to estimate answers to questions for all frames obtained by sampling a video. For this reason, when a video is added to the search target storage unit 121, the server 120 may perform a process to estimate answers to questions stored in the question storage unit 124 for this video. This makes it possible to obtain search results in a short time when a question identical to a question stored in the question storage unit 124 is entered.

[0089] FIG. 18 shows an outline of an example of the procedure of a video adding method according to this embodiment, which is executed when a video is added to the search target storage unit 121. The video adding method shown in FIG. 18 is executed by the server 120. Here, the question storage unit 124 stores question IDs 1 to N. Q N Q It is assumed that questions are stored.

[0090] In step S1801, the answer estimation unit 125 extracts the added video from the search target storage unit 121, samples the extracted video to obtain frames, and assigns a frame ID to each frame. Here, the frame IDs are 1 to N. F N F Assume that one frame is obtained.

[0091] In step S1802, the response estimation unit 125 sets the variable i to 1.

[0092] In step S1803, the answer estimation unit 125 extracts a question i (a question whose question ID is i) from the question storage unit .

[0093] In step S1804, the response estimation unit 125 sets the variable j to 1.

[0094] In step S1805, the answer estimation unit 125 estimates an answer to question i for frame j (a frame whose frame ID is j). For example, the answer estimation unit 125 inputs frame j and question i to a VQA model, and obtains an answer output from the VQA model in response to the input of frame j and question i.

[0095] In step S1806, the answer estimation unit 125 stores the answer obtained in step S1805 in the estimation result storage unit 127 in association with the question ID=i and the frame ID=j.

[0096] In step S1807, the response estimation unit 125 determines whether the variable j is N F Determine whether variable j is equal to N F is equal to N F This indicates that answer estimation for question i has been completed for all frames.

[0097] Variable j is N F If it is less than that (step S1807; No), the flow proceeds to step S1808. In step S1808, the answer estimation unit 125 increments the variable j by 1. After that, the flow returns to step S1805.

[0098] Variable j is N F If the variable i is equal to N (step S1807; Yes), the flow proceeds to step S1809. Q Determine whether variable i is equal to N Q is equal to N Q This means that answer estimation has been completed for all questions.

[0099] Variable i is N Q If it is less than that (step S1809; No), the flow proceeds to step S1810. In step S1810, the answer estimation unit 125 increments the variable i by 1. After that, the flow returns to step S1803.

[0100] Variable i is NQ If it is equal to (step S1809; Yes), the flow ends.

[0101] Figures 19A and 19B show an outline of an example of the procedure of the video scene search method according to this embodiment. The video scene search method shown in Figures 19A and 19B is executed by the server 120. The flow shown in Figures 19A and 19B is similar to the flow shown in Figures 12A and 12B, but there are differences in details due to differences in the search targets.

[0102] 19A, the answer estimation unit 125 acquires information indicating a video selected by a user. The answer estimation unit 125 calculates, from the duration of the video selected by the user, the number of frames N obtained by sampling the video. F Here, we calculate N by sampling the video. F frames are obtained, and 1 to N F The frame ID will be assigned.

[0103] In step S1902, the question acquiring unit 123 converts the query input by the user into a question in text format. In step S1903, the question acquiring unit 123 determines whether the same question as the question acquired in step S1902 exists in the question storage unit 124.

[0104] If the question identical to the question obtained in step S1902 exists in the question storage unit 124 (step S1903; Yes), the flow proceeds to step S1904. In step S1904, the question acquiring unit 123 extracts the question ID of the question identical to the question obtained in step S1902 from the question storage unit 124, and sends the extracted question ID and the question obtained in step S1902 to the answer estimation unit 125.

[0105] On the other hand, if the question identical to the question obtained in step S1902 does not exist in the question storage unit 124 (step S1903; No), the flow proceeds to step S1905. In step S1905, the question acquiring unit 123 assigns a question ID to the question obtained in step S1905, and stores the question obtained in step S1902 in association with the assigned question ID in the question storage unit 124. Furthermore, the question acquiring unit 123 sends the assigned question ID and the question obtained in step S1902 to the answer estimation unit 125.

[0106] Here, the question ID extracted in step S1904 or the question ID assigned to the question in step S1905 is set to i.

[0107] When the flow proceeds from step S1904 or step S1905 to step S1906, the answer estimation unit 125 creates an empty answer estimation result. For example, the answer estimation unit 125 creates an answer estimation result that includes a frame ID and time information but does not include an answer. Note that because a frame ID corresponds one-to-one with a time point indicated by the time information, the answer estimation result does not need to include a frame ID. Thereafter, the flow proceeds to step S1907 in FIG. 19B.

[0108] In step S1907 of FIG. 19B, the response estimation unit 125 sets the variable j to 1. F If it is not greater (step S1908; No), the flow proceeds to step S1909.

[0109] If a record with a question ID equal to i and a frame ID equal to j exists in the estimation result storage unit 127 (step S1909; Yes), the flow proceeds to step S1910, where the answer estimation unit 125 extracts the answer and the basis information included in the record with a question ID equal to i and a frame ID equal to j from the estimation result storage unit 127. Then, in step S1912, the answer estimation unit 125 adds the extracted answer and the basis information to the answer estimation result.

[0110] If the estimation result storage unit 127 does not contain a record whose question ID is equal to i and whose frame ID is equal to j (step S1909; No), the flow proceeds to step S1911, where the answer estimation unit 125 extracts frame j (a frame whose frame ID is equal to j) from the search target storage unit 121. In step S1912, the answer estimation unit 125 estimates the answer to question i and the basis information for frame j. For example, the answer estimation unit 125 inputs frame j and question i to a VQA model and obtains the answer and the basis information output from the VQA model.

[0111] In step S1913, the answer estimation unit 125 associates the estimated answer and the basis information with the video ID, frame ID, and question ID, and stores them in the estimation result storage unit 127. Then, in step S1914, the answer estimation unit 125 adds the estimated answer and the basis information to the answer estimation result.

[0112] When the process shown in step S1914 ends, the flow proceeds to step S1915, where the answer estimation unit 125 increments the variable j by 1. After that, the flow returns to step S1908. F A series of processes shown in steps S1908 to S1915 are repeated until the answer estimation result exceeds N F It will contain answers for all frames.

[0113] Variable j is N F If it is greater than N (step S1908; Yes), the flow proceeds to step S1916. In step S1916, the grouping unit 128 uses the answer estimation result to F For example, the grouping unit 128 creates groups in the number equal to the number of answer types included in the answer estimation result, and assigns each image to a group according to the answer for that image.

[0114] In step S1917, the display control unit 129 selects a representative image for each group. In step S1918, the display control unit 129 displays the search results.

[0115] As described above, in the second embodiment, answers to questions based on input queries are displayed in association with time information of the video, and information on the basis of the answers is superimposed on the display, allowing the user to easily grasp where desired scenes are located and how they are distributed.

[0116] [Third embodiment] In the first and second embodiments, the user inputs a query each time. It takes time to think of a query, and the user may not always be able to input an appropriate query for the image group or video to be searched. Furthermore, since answer estimation takes time, inputting an unnecessary query not only results in meaningless search results but also wastes time.

[0117] An information processing device according to the third embodiment stores a question list containing multiple questions and presumes answers to each question in the question list. When presenting search results for a query entered by a user, the information processing device selects questions from the question list that are useful for narrowing down the search results and suggests them to the user. This enables efficient interactive searches.

[0118] Fig. 20 schematically shows an information processing system 2000 according to the third embodiment. In Fig. 20, parts similar to those shown in Fig. 1 are denoted by the same reference numerals, and duplicated explanations will be omitted. As shown in Fig. 20, the information processing system 2000 includes a client 110 and a server 2020 as an information processing device according to the third embodiment. The server 2020 is configured by adding a narrowing-down candidate generation unit 2021 to the server 120 shown in Fig. 1.

[0119] The question storage unit 124 stores a question list including multiple questions. The question list may include multiple questions prepared in advance and additional information related to these questions. The multiple questions prepared in advance may be created manually. Alternatively, the multiple questions prepared in advance may be created automatically by extracting multiple sentences or characteristic keywords from a document such as a safety manual and converting each sentence or keyword into a question using a method similar to that used by the question acquisition unit 123. The question list is used not only for search processing using a query entered by a user but also for generating subsequent narrowed-down candidates.

[0120] 21 shows an example of a question list stored in the question storage unit 124. In the example shown in Fig. 21, the additional information includes time information indicating the date and time of registration, usage information indicating the number of times selected, and category information indicating the category. The additional information does not need to include all three of these items, and it is sufficient if it includes only the items used by the narrowing-down candidate generation unit 2021.

[0121] The registration date and time indicates the date and time when the question was registered in the question list. For a prepared question, the registration date and time may be set to the date and time when the question list was created. For a question added based on a query entered by a user, the registration date and time may be set to the date and time when the user entered the query.

[0122] The number of uses indicates the number of times a question has been used in a search. For example, the number of uses is the sum of the number of times a question has been used in a search using a query entered by a user and the number of times a question has been used in a refined search. For example, if a question identical to the question acquired by the question acquisition unit 123 when the user entered a query is present in the question list, the number of uses is incremented by 1. Also, if a question is presented as a refinement candidate and the user selects that question and then selects one of the answer groups presented thereafter, the number of uses is incremented by 1.

[0123] The category indicates a category of content that the question may be used to search. For a pre-prepared question, the category may be assigned when the question is created. For a question added based on a query entered by a user, the category may be assigned by inferring the category of the keywords entered by the user as the query or keywords extracted from the sentence entered by the user as the query. Any keyword classification method may be used to infer the category. Examples of keyword classification methods include a method using a keyword-category correspondence table or ontology, a method using word2vec to match keyword variations and words with the same meaning but different spellings, and a method building a model to classify keywords into categories.

[0124] The question storage unit 124 may store a question usage history along with the question list. As shown in FIG. 22, the question usage history includes three fields for storing a question ID, a usage date and time, and a user ID. The question ID is identification information for identifying a question, the usage date and time indicates the date and time when the question was used, and the user ID is information for identifying the user who used the question. Each time a question is used in a search, a record including the question ID, the usage date and time, and the user ID is added to the question usage history. The question usage history is an example of additional information.

[0125] The narrowing-down candidate generation unit 2021 generates narrowing-down candidates including a question for a narrowing-down search for content to be searched. For example, the content to be searched may be a plurality of images stored in the search target storage unit 121 as in the first embodiment, or a video selected by the user as in the second embodiment. The narrowing-down candidates may further include an answer to the question for the narrowing-down search. For example, the narrowing-down candidates may include an answer to the question for a narrowing-down search for each of a plurality of images stored in the search target storage unit 121, or may include an answer to the question for a narrowing-down search for each of a plurality of frames included in a video selected by the user.

[0126] Specifically, the narrowing-down candidate generation unit 2021 selects one or more questions from the question list as questions for narrowing down the search, and generates an answer estimation result indicating an answer to the question for narrowing down the search for each of the multiple images. The narrowing-down candidate generation unit 2021 generates the answer estimation result using the answer estimation unit 125. In other words, the narrowing-down candidate generation unit 2021 causes the answer estimation unit 125 to estimate the answer to the question for narrowing down the search. Note that the narrowing-down candidate generation unit 2021 may be configured to perform answer estimation in a manner similar to that of the answer estimation unit 125 in order to generate the answer estimation result.

[0127] The refinement candidate generation unit 2021 may use additional information included in the question list to select a question for the refined search. For example, the refinement candidate generation unit 2021 selects the most recent question that has not been used in the current search session as the question for the refined search. By selecting the most recent question, it is possible to present a question that has been added recently. Alternatively, the refinement candidate generation unit 2021 uses the most frequently selected question that has not been used in the current search session as the question for the refined search. By selecting the most frequently selected question, it is possible to present a question that is likely to be used.

[0128] The refinement candidate generation unit 2021 may use categories as an auxiliary means to select questions for refined search. Categories are used to narrow down selection candidates when the number of questions stored in the question storage unit 124 becomes large. For example, the refinement candidate generation unit 2021 acquires or estimates the category of a video selected by a user as a search target, selects questions in a category matching the video category as selection candidates, and selects a question from the selection candidates based on other items (e.g., registration date and time or number of selections). If the video to be searched is provided by a video sharing service, the video category is often assigned in advance. Existing category classification technology can be used to estimate the category. For example, the refinement candidate generation unit 2021 may classify the video category based on frames included in the video. Alternatively, the refinement candidate generation unit 2021 may convert the audio of the video into text using speech recognition and classify the video category from the text. The refinement candidate generation unit 2021 may estimate the category of a query entered by a user and select a question for refined search from questions in a category matching the estimated category.

[0129] The narrowing-down candidate generation unit 2021 may use the question usage history stored in the question storage unit 124 to select questions for narrowing down the search. The narrowing-down candidate generation unit 2021 may select questions that have been used many times recently (for example, the number of times used in the past week) as questions for narrowing down the search. The narrowing-down candidate generation unit 2021 may select questions selected by users with similar selection tendencies to the user currently performing the search as questions for narrowing down the search. This can be achieved by using collaborative filtering technology. Rather than selecting questions for narrowing down the search based on the number of selections, it becomes possible to present narrowing-down candidates that are more suited to the current situation or user.

[0130] Narrowing candidates can be generated either before or after the user inputs a query. When generating narrowing candidates after the user inputs a query, the narrowing candidate generation unit 2021 may select narrowing questions from a question list based on the input query. When documents related to scene search or image search, such as safety manuals, are prepared in advance, the narrowing candidate generation unit 2021 may extract keywords that co-occur with keywords included in the input query from the documents and select questions containing the extracted keywords as questions for the narrowing search. For example, the narrowing candidate generation unit 2021 calculates the co-occurrence between each keyword in the document and a keyword included in the input query, extracts keywords from the document with the highest calculated co-occurrence or higher than a predetermined value, and selects one or more questions from among the questions containing the extracted keywords as questions for the narrowing search. The co-occurrence between a first keyword and a second keyword indicates, for example, the number of sentences in the document that contain both the first keyword and the second keyword. By selecting narrowing questions based on the input query, questions highly relevant to the input query can be presented.

[0131] Before or after the user inputs a query, the display control unit 129 presents the narrowing-down candidates generated by the narrowing-down candidate generation unit 2021. In this embodiment, the display control unit 129 presents the narrowing-down candidates in response to the user inputting a query.

[0132] FIG. 23 schematically illustrates an example of presenting refinement candidates. In the example illustrated in FIG. 23, a user selects a video and inputs the query "helmet." The server 2020 displays scene search results 2301 in response to the video selection and query input, and presents refinement candidates 2302 and an additional text input form 2304 in response to the user selecting the "Yes" answer group. In the example illustrated in FIG. 23, the server 2020 displays one refinement candidate and allows the user to select the refinement candidate to be displayed using a drop-down list 2303. Alternatively, multiple refinement candidates may be displayed simultaneously. In this manner, by displaying the refinement candidates 2302 and the additional text input form 2304, a search can be interactively advanced.

[0133] Below the video display area 2305, a seek bar 2306 showing the results of a search using the input query and a seek bar 2307 showing the results of a refined search are arranged in parallel. Each of the seek bars 2306, 2307 indicates the time at which a frame belonging to an individual answer group is located. If the user enters an additional query, a seek bar showing the results of a search using the additional query is added below the two seek bars 2306, 2307. Note that multiple seek bars may be displayed translucently in a single layer.

[0134] The search screen is configured to allow a user to select one of the answer groups displayed as a result of a search using an input query, and further configured to allow a user to select one of the answer groups displayed as a result of a refined search.

[0135] The refinement candidate generation unit 2021 updates the refinement search results in response to the user selecting one of the answer groups displayed as a search result using the input query. Before selecting an answer group, the refinement search targets multiple frames included in the video selected by the user. After selecting an answer group, the refinement search targets multiple frames belonging to the group selected by the user. Referring again to the example shown in FIG. 23, the user selects the group with the answer "yes." In this case, the refinement search targets frames belonging to the group with the answer "yes." At least one worker appears in the frame belonging to the group with the answer "yes." Therefore, the refinement candidates do not include the answer "0 people." If the user deselects the group with the answer "yes," the refinement candidate generation unit 2021 updates the refinement candidates for the entire video. In this case, the group with the answer "0 people" will also be displayed in the refinement search results. A representative image is also selected for the group with the answer "0 people."

[0136] Next, a description will be given of the operation of the server 2020. Here, the description will be mainly given of the case where the search target is a video selected by the user as described in the second embodiment.

[0137] When a video is added to the search target storage unit 121, the server 2020 performs the process shown in the video adding method described with reference to FIG.

[0138] When a query is input by the user, the server 2020 performs a search using the input query in the same manner as the video scene search method described with reference to Figures 19A and 19B. Following the search using the input query, the server 2020 performs a process of presenting narrowed-down candidates, which will be described later.

[0139] 24 is a flow chart showing an example of a method for presenting narrowed-down candidates according to the third embodiment. The method for presenting narrowed-down candidates shown in FIG.

[0140] In step S2401 of FIG. 24, the narrowing-down candidate generation unit 2021 selects a question for a narrowed-down search from the question list. For example, the narrowing-down candidate generation unit 2021 uses the input query to select a question for a narrowed-down search from the question list. Specifically, the narrowing-down candidate generation unit 2021 selects a question from the question list that includes a keyword that co-occurs with a keyword included in the input query as a question for a narrowed-down search. For simplicity of explanation, a case will be described here where the narrowing-down candidate generation unit 2021 selects one question.

[0141] In step S2402, the narrowing-down candidate generation unit 2021 generates an answer estimation result indicating the answer to the narrowing-down search question for each of a plurality of frames included in the video. The narrowing-down candidate generation unit 2021 performs answer estimation in the same manner as the series of processes shown in steps S1906 to S1914 in Fig. 19B to obtain an answer estimation result.

[0142] In step S2403, the grouping unit 128 classifies the frames into a plurality of answer groups using the answer estimation results. In step S2404, the display control unit 129 selects, for each answer group, a representative image from the frames included in the answer group. In step S2405, the display control unit 129 displays refinement candidates including the question for refined search and the results of grouping the frames.

[0143] When the narrowing-down candidate generating unit 2021 selects a plurality of narrowing-down search questions, generation and grouping of answer prediction results are performed for each of the narrowing-down search questions.

[0144] In step S2406, the narrowing-down candidate generating unit 2021 waits for selection of one of the answer groups displayed as the search results based on the input query. When an answer group is selected (step S2406; Yes), the flow proceeds to step S2407.

[0145] In step S2407, the narrowing-down candidate generating unit 2021 updates the narrowing-down candidates for the frames included in the selected answer group, and in step S2408, the display control unit 129 displays the updated narrowing-down candidates.

[0146] The first to third embodiments can be combined with existing techniques such as keyword-based image search, etc. For example, the image search technique according to the first embodiment can be applied to search results obtained by keyword-based image search.

[0147] As shown in Figure 25, suppose you perform a regular image search using the keyword "A100," which is the name of a sneaker. The search results include product images of the sneakers specified by the keyword, images of people wearing the sneakers specified by the keyword, and images of them being used in outfits.

[0148] With the search results displayed, the user inputs an additional query such as "Are they wearing sneakers?" When an image search according to the first embodiment is performed for the input question, the images are classified into images in which the person is wearing sneakers A100 and images in which the person is not wearing sneakers A100 (such as product images).

[0149] The image search technology according to the embodiment may also be combined with a search technology for sound events such as laughter. For example, as shown in Fig. 26, the sound event search technology detects portions of a video that contain laughter, and the image search technology according to the embodiment is then applied to the detected portions.

[0150] As described above, in the third embodiment, the server 2020 presents narrowing-down candidates including questions useful for narrowing down search of search results based on an input query. Because the narrowing-down candidates are perspectives for narrowing down, unlike a search using keywords, the search is not narrowed down too much, and the user can easily find a desired scene.

[0151] [Fourth embodiment] In the fourth embodiment, a function of supporting the creation of rules to be included in a safety manual or the like by referring to a specified video will be described. The configuration of the information processing device according to the fourth embodiment is similar to that of the server 2020 according to the third embodiment. In the fourth embodiment, the information processing system 2000 shown in FIG. 20 will also be referenced.

[0152] The query input unit 122 is configured to allow input of a long sentence consisting of multiple sentences. For example, a rule creator inputs a rule to be included in a safety manual into the query input unit 122. The server 2020 divides the sentence input into the query input unit 122 into multiple sentences and converts each sentence into a question. The server 2020 generates an answer group for each question obtained by the conversion.

[0153] The server 2020 generates multiple refinement candidates when a query is input and presents them as recommended additional rules. As an example, the server 2020 identifies keywords that have a high co-occurrence rate with keywords included in a sentence input by the rule creator in another safety manual, extracts sentences including the identified keywords from the other safety manual, and generates refinement candidate questions from the extracted sentences.

[0154] Fig. 27 shows an example of a screen according to the fourth embodiment. As shown in Fig. 27, the screen includes a text input form 2701, a video display area 2702, a search result display area 2703, and a suggestion display area 2704. The search result display area 2703 displays search results including multiple questions based on the query entered in the text input form 2701. The suggestion display area 2704 displays questions as additional rules to be suggested to the user. The search result display area 2703 and the suggestion display area 2704 display a match rate for each answer group. The match rate is the value obtained by dividing the number of frames belonging to the group by the total number of frames obtained by sampling.

[0155] When the user selects one of the answer groups, the corresponding portion is colored on the seek bar in the video display area 2702. When answer groups are selected for multiple questions, multiple seek bars may be displayed in parallel.

[0156] As described above, in the fourth embodiment, when a rule creator wants to create rules based on a video, additional rules can be suggested along with the display of the matching rate. As a result, the rule creator does not need to think up rules while searching the video themselves, which reduces the time and effort required.

[0157] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0158] 100...information processing system, 110...client, 111...input unit, 112...output unit, 120...server, 121...search target storage unit, 122...query input unit, 123...question acquisition unit, 124...question storage unit, 125...answer estimation unit, 126...model storage unit, 127...estimation result storage unit, 128...grouping unit, 129...display control unit, 1101...CPU, 1102...RAM, 1103...storage device, 1104...communication interface, 2000...information processing system, 2020...server, 2021...narrowing candidate generation unit.

Claims

1. Obtaining a first question in a predetermined format according to information input by a user; generating a first answer estimation result including a plurality of answers associated with each of the plurality of contents by estimating an answer to the first question for each of the plurality of contents; generating a first search result indicating content belonging to each of the first groups by classifying the content into a plurality of first groups using the plurality of answers included in the first answer estimation result; wherein the predetermined format is text, image, audio, numeric, or a combination thereof.

2. The information processing method includes: selecting a second question from the plurality of questions included in the question list; generating a second answer estimation result including a plurality of answers to the second question associated with each of the plurality of contents by estimating an answer to the second question for each of the plurality of contents; generating second search results indicating content belonging to each of the second groups by classifying the content into a plurality of second groups using the plurality of answers included in the second answer estimation result; displaying the second search results together with the first search results; The information processing program according to claim 1 , comprising:

3. displaying the second search results comprises displaying the plurality of first groups in a selectable manner; The information processing method includes: In response to a selection of one of the plurality of first groups, using the second search result, classifying content belonging to the selected first group into the plurality of second groups, thereby generating a third search result indicating content belonging to each of the plurality of second groups; displaying the third search result together with the first search result; The information processing program according to claim 2 , further comprising:

4. The information processing program according to claim 2 , wherein the information processing method further comprises adding the first question to the question list.

5. the question list further includes time information indicating a date and time when each of the plurality of questions was registered in the question list or a date and time when each of the plurality of questions was used in a search; The information processing program according to claim 2 , wherein selecting the second question comprises selecting the second question from the plurality of questions based on the time information.

6. the question list further includes usage information indicating the number of times the plurality of questions have been used in searches; The information processing program according to claim 2 , wherein selecting the second question comprises selecting the second question from among the plurality of questions based on the usage information.

7. Selecting the second question includes: extracting second keywords that co-occur with the first keywords included in the input information from documents prepared in advance; selecting a question including the second keyword from the plurality of questions as the second question; The information processing program according to claim 2 , comprising:

8. obtaining a first question in a predetermined format according to the input information comprises converting the information in a first format into the question in the predetermined format; The information processing program according to claim 1 , wherein the first format is text, image, audio, or a combination thereof.

9. the plurality of contents are a plurality of images; The information processing program according to any one of claims 1 to 8.

10. The information processing program according to claim 9 , wherein the plurality of images are frame images included in a moving image.

11. The information processing method includes: selecting a representative image from among the images belonging to each of the plurality of first groups; displaying the answers corresponding to the plurality of first groups and the representative image; The information processing program according to claim 9 , further comprising:

12. the plurality of images are associated with time information; The information processing program according to claim 11 , wherein selecting the representative image comprises selecting the representative image from the images belonging to each of the plurality of first groups based on the time information.

13. the first answer prediction result further includes a confidence level of the answer associated with each of the plurality of contents; The information processing program according to claim 11 , wherein selecting the representative image comprises selecting the representative image from the images belonging to each of the plurality of first groups based on the degree of certainty.

14. the first answer estimation result further includes basis information indicating a basis for the answer associated with each of the plurality of contents; The information processing program according to claim 11 , wherein the information processing method further comprises displaying the basis information corresponding to the representative image in a manner superimposed on the representative image.

15. the grounds information includes area information indicating the grounds area, 15. The information processing program according to claim 14, wherein selecting the representative image comprises selecting the representative image from the images belonging to each of the plurality of first groups based on the size of the area indicated by the area information.

16. 1. A computer-implemented information processing method, comprising: Obtaining a first question in a predetermined format according to information input by a user; generating a first answer estimation result including a plurality of answers associated with each of the plurality of contents by estimating an answer to the first question for each of the plurality of contents; classifying the plurality of contents into a plurality of first groups using the plurality of answers included in the first answer estimation result, thereby generating a first search result indicating contents belonging to each of the plurality of first groups; Equipped with An information processing method, wherein the predetermined format is text, image, audio, numeric, or a combination thereof.

17. a question acquisition unit that acquires a first question in a predetermined format according to information input by a user; an answer estimation unit that estimates an answer to the first question for each of a plurality of pieces of content to generate a first answer estimation result including a plurality of the answers associated with each of the plurality of pieces of content; a grouping unit that classifies the plurality of contents into a plurality of first groups using the plurality of answers included in the first answer estimation result, and generates a first search result indicating contents belonging to each of the plurality of first groups; Equipped with The predetermined format is text, image, audio, numeric value, or a combination thereof.

Images