Information retrieval system

Abstract

To provide an information retrieval system that can improve the accuracy of response sentences generated by AI models. [Solution] In the information retrieval system 100 of this disclosure, the input unit 101 receives input of image data containing at least one candidate object and first text data containing a question from the user. The object detection unit 103 inputs the image data and the first text data into an image segmentation model to detect a target object that is a candidate object related to the question contained in the first text data from among the at least one candidate object contained in the image data. The search unit 104 searches for information on the target object based on the detection result of the target object and the first text data, and generates second text data containing an answer to the question based on the search result. The output unit 105 outputs the second text data.

Description

Technical Field

[0001] The present invention relates to an information retrieval system.

Background Art

[0002] Patent Document 1 discloses an information retrieval system that uses an AI (Artificial Intelligence) model trained to output an answer to a question sentence by taking as input image data and language data that is a question sentence regarding an object included in the image data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology according to Patent Document 1, there is a problem that as the image data contains more candidate objects to be searched, the difficulty of generating an answer sentence that matches the question sentence from the user by the AI model becomes higher.

[0005] In view of such problems, an object of the present disclosure is to provide an information retrieval system capable of improving the accuracy of an answer sentence generated by an AI model.

Means for Solving the Problems

[0006] The information retrieval system of the present disclosure includes: an input unit that receives input of image data including at least one candidate object and first text data including a question from a user; an object detection unit that inputs the image data and the first text data into an image segmentation model to detect a target object that is a candidate object related to the question included in the first text data from among the at least one candidate object included in the image data; a search unit that searches for information on the target object based on the detection result of the target object and the first text data, and generates second text data including an answer to the question based on the search result; and an output unit that outputs the second text data.

[0007] In the information retrieval system of this disclosure, the search unit retrieves information about the target object by inputting the detection result of the target object and the first text data into a visual language model, and generates second text data including an answer to the question based on the search results.

[0008] In the information retrieval system of this disclosure, the search unit retrieves information about the target object by inputting the detection result of the target object, the first text data, and the web search engine, generates second text data including an answer to the question based on the search results, and summarizes the second text data by inputting the detection result of the target object and the second text data into a visual language model.

[0009] In the information retrieval system of this disclosure, the input unit further inputs face image data including the user's face, the object detection unit calculates the user's gaze data from the face image data, and uses the gaze data to detect the target object related to the question sentence contained in the first text data from among at least one candidate object contained in the image data.

[0010] In the information retrieval system disclosed herein, the image segmentation model is trained based on training data in which image data containing a predetermined object is associated with text data related to the predetermined object. [Effects of the Invention]

[0011] This disclosure enables the provision of an information retrieval system that can improve the accuracy of response sentences generated by AI models. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example of the configuration of an information processing system according to the first embodiment. [Figure 2] This is a block diagram showing an example of the hardware configuration of an information retrieval system in an information processing system according to the first embodiment. [Figure 3] This is a schematic diagram showing a first specific example of the configuration of the object detection unit and the search unit in the information retrieval system according to the first embodiment. [Figure 4] This is a schematic diagram showing a second specific example of the configuration of the object detection unit and the search unit in the information retrieval system according to the first embodiment. [Figure 5] This flowchart shows an example of the operation of the information retrieval system in the information processing system according to the first embodiment. [Modes for carrying out the invention]

[0013] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations will be omitted where necessary for clarity.

[0014] (First embodiment) First, the hardware configuration of the information processing system 1 according to the first embodiment will be described using Figures 1 and 2.

[0015] Figure 1 is a block diagram showing an example of the hardware configuration of an information processing system 1 according to the first embodiment. As shown in Figure 1, the information processing system 1 includes an imaging device 10, an audio input device 20, a computing device 30, an external computing server 40, and an output device 50.

[0016] The computing unit 30 communicates with the imaging device 10, the audio input device 20, the external computing server 40, and the output device 50 via wired or wireless means. Together with the external computing server 40, the computing unit 30 constitutes the information retrieval system 100.

[0017] The imaging device 10 is a device that captures image data such as an RGB image. The imaging device 10 is, for example, a camera. However, the imaging device 10 is not limited to a camera. For example, the imaging device 10 may be a camera built into a mobile terminal used by the user.

[0018] The voice input device 20 is a device that receives voice data, including a question, from the user. The voice input device 20 is, for example, a microphone. Alternatively, the voice input device 20 may be a text input device (not shown) that receives text data, including a question, from the user. The text input device is, for example, a touch panel.

[0019] The arithmetic unit 30 is a device used for arithmetic operations to execute algorithms and the like. The arithmetic unit 30 is a computer including a processor and a memory. The processor may be, for example, a microprocessor, a MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor may include a plurality of processors. The memory is composed of a combination of volatile memory and non-volatile memory. The memory may include storage arranged separately from the processor. In this case, the processor may access the memory via an I / O interface not shown in the drawings. The processor executes one or more programs including a group of instructions for causing a computer to perform the algorithms described using the drawings. The program includes a group of instructions (or software code) for causing a computer to perform one or more functions described in the embodiments when read into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0020] The external computing server 40 is an external server that executes computations to assist the computations of the arithmetic unit 30. The output device 50 is a device that displays data to the user. The output device 50 is, for example, a display or a speaker.

[0021] Incidentally, the above-described information processing system 1 may include a user imaging device (not shown). The user imaging device is a device that captures image data such as an RGB image including the user's face. The user imaging device is, for example, a camera. The user imaging device communicates with the arithmetic unit 30 wirelessly or by wire.

[0022] The above-described information processing system 1 can be realized as a system that performs information search during vehicle driving by a user who is a vehicle driver. In that case, the imaging device 10 is an in-vehicle camera mounted on the vehicle in which the user rides. The imaging device 10 is installed on the vehicle so as to be able to capture an image in the traveling direction of the vehicle (that is, an image with a视野 as close as possible to the user's viewpoint). The voice input device 20 is a microphone mounted on the vehicle in which the user rides. The output device 50 is a display or a speaker mounted on the vehicle in which the user rides. Also, although not shown, the text input device is a touch panel mounted on the vehicle in which the user rides. The user imaging device is an in-vehicle camera mounted on the vehicle in which the user rides. The user imaging device is installed on the vehicle so as to be able to capture an image including the face of the user riding in the vehicle.

[0023] Subsequently, with reference to FIG. 2, the software configuration of the information search system 100 in the information processing system 1 according to the first embodiment will be described. FIG. 2 is a block diagram showing an example of the software configuration of the information search system 100 in the information processing system 1 according to the first embodiment. As shown in FIG. 2, the information search system 100 includes an input unit 101, a voice recognition unit 102, an object detection unit 103, a search unit 104, and an output unit 105 (collectively referred to as each component). The functions of each component of the information search system 100 are realized by the arithmetic unit 30 and the external calculation server 40. Incidentally, the functions of each component of the information search system 100 may be realized only by the arithmetic unit 30 without using the external calculation server 40.

[0024] The input unit 101 receives image data from the imaging device 10. The image data includes at least one object (hereinafter referred to as a candidate object). The input unit 101 also receives audio data (hereinafter referred to as first audio data) from the audio input device 20, which includes a question from the user. The question relates to at least one of the candidate objects included in the image data. Alternatively, the input unit 101 may receive text data (hereinafter referred to as first text data) including the question from a text input device (not shown) instead of the first audio data.

[0025] The speech recognition unit 102 generates first text data including a question from the first audio data by performing speech recognition processing to convert the audio data into text data. If the input unit 101 receives input of first text data including a question from a text input device (not shown), the speech recognition unit 102 does not need to perform speech recognition processing.

[0026] The object detection unit 103 detects a target object from among at least one candidate object contained in the image data that is related to the question sentence contained in the first text data, based on the image data and the first text data. The search unit 104 searches for data on the target object based on the detection results of the object detection unit 103 and the first text data. Based on the search results, the search unit 104 generates text data (hereinafter referred to as the second text data) that includes the answer to the question. Here, specific examples of the configurations of the object detection unit 103 and the search unit 104 will be described later with reference to Figures 3 and 4.

[0027] The output unit 105 outputs the second text data via the output device 50. Alternatively, the output unit 105 may convert the second text data into audio data (hereinafter referred to as the second audio data) and output the second audio data via the output device 50.

[0028] Next, using Figure 3, a first specific example of the configuration of the object detection unit 103 and the search unit 104 in the information retrieval system 100 according to the first embodiment will be described. Figure 3 is a schematic diagram showing a first specific example of the configuration of the object detection unit 103 and the search unit 104 in the information retrieval system 100 according to the first embodiment. In the example shown in Figure 3, the information processing system 1 is implemented as a system that allows a user, who is the driver of a vehicle, to retrieve information while driving the vehicle.

[0029] As shown in Figure 3, the object detection unit 103 inputs image data and first text data including a question into the image segmentation model. The image data is an image of the direction of travel of the vehicle the user is riding in, and includes at least one candidate object. The question is asked by the user, who is the driver of the vehicle, and relates to at least one of the candidate objects included in the image data. The question is, "What's that white car?" The image segmentation model is an AI model that performs image segmentation. Image segmentation is a technique that divides image data at the pixel level and classifies which object each pixel belongs to. Examples of image segmentation models include FastSAM (Fast Segment Anything Model) and SAM (Segment Anything Model). The image segmentation model is trained on training data in which image data containing a predetermined object is associated with text data related to that predetermined object. Furthermore, the image segmentation model may be trained based on training data in which bounding boxes or points indicating the location of a target object are further associated with image data containing a given object.

[0030] The object detection unit 103 detects a target object from among at least one candidate object contained in the image data that is related to the question sentence contained in the first text data, based on the input to the image segmentation model. In this example, the question sentence is "What's that white car?". The object detection unit 103 detects the white car intended by the user in the image data. The detection result of the target object includes the image data and a segmentation mask, which is data indicating which pixels in the image data belong to the target object.

[0031] Furthermore, the object detection unit 103 may detect a candidate object related to the question sentence contained in the first text data from among at least one candidate object contained in the image data as a target object using the following method. In particular, the ability to detect target objects can be improved when multiple target objects exist in the image data.

[0032] As a first method, the object detection unit 103 may display image data with numbers assigned to the target object on an in-vehicle display (not shown) as additional instructions to the user (e.g., spatial instructions such as left / right or front / back), and allow the user to select by number. By doing so, the object detection unit 103 can detect the optimal target object. As a second method, the input unit 101 further receives face image data, including the user's face, from a user imaging device (not shown). The object detection unit 103 calculates the user's gaze data from the face image data. The user's gaze data includes, for example, the position of the user's eyes (position on the face image data or on the three-dimensional coordinate system) and the direction of the user's gaze. The object detection unit 103 further uses the gaze data to detect a target object from among at least one candidate object included in the image data that is a candidate object related to the question sentence included in the first text data. Specifically, the object detection unit 103 uses the gaze data to calculate data indicating the point on the image data that the user is fixated on. The object detection unit 103 uses the data indicating the point on the image data that the user is fixated on, along with the image data and the first text data including the question sentence, as input to the image segmentation model. By doing so, the object detection unit 103 detects a target object from among at least one candidate object included in the image data that is a candidate object related to the question sentence included in the first text data and that the user is fixated on.

[0033] Next, the search unit 104 inputs the object detection result of the object detection unit 103 and the first text data including the question to the visual language model. The visual language model is an AI model that understands both image data and language data and performs processes such as text generation, question answering, text summarization, and translation. An example of a visual language model is GPT-4. The visual language model is trained using training data in which image data containing a predetermined object is associated with text data related to that predetermined object.

[0034] The search unit 104 retrieves data on the target object based on input to the visual language model and generates second text data, including an answer to the question, based on the search results. The answer is, for example, "The white car is..."

[0035] Next, using Figure 4, a second specific example of the configuration of the object detection unit 103 and the search unit 104 in the information retrieval system 100 according to the first embodiment will be described. Furthermore, Figure 4 is a schematic diagram showing a second specific example of the configuration of the object detection unit 103 and the search unit 104 in the information retrieval system 100 according to the first embodiment. In the example shown in Figure 4, the information processing system 1 is implemented as a system that allows a user, who is the driver of a vehicle, to retrieve information while driving the vehicle.

[0036] As shown in Figure 4, the object detection unit 103 receives image data and first text data including a question as input to a visual language model. The image data is an image of the direction of travel of the vehicle the user is riding in and includes at least one candidate object. The question is made by the user, who is the driver of the vehicle, and relates to at least one of the at least one candidate object included in the image data. The question is, "What's that white car?". The visual language model is, for example, GPT-4. The object detection unit 103 detects a target object from among at least one candidate object contained in the image data that is related to the question sentence contained in the first text data, based on input to the visual language model. The detection result by the visual language model includes the image data and a rectangular region indicating the position of the target object in the image data.

[0037] Next, the object detection unit 103 inputs the image data, a rectangular region indicating the location of the target object in the image data, and first text data containing the question sentence into the image segmentation model. The image segmentation model is, for example, FastSAM or SAM. By doing so, the object detection unit 103 further detects the target object related to the question sentence contained in the first text data from among at least one candidate object contained in the image data. The detection result by the image segmentation model includes the image data and a segmentation mask, which is data indicating which pixels in the image data belong to the target object.

[0038] Next, the search unit 104 inputs the object detection result of the object detection unit 103 (i.e., the detection result by the image segmentation model) and the first text data into a web search engine. The web search engine is an engine for searching for data on the web related to a predetermined keyword. For example, the web search engine is Microsoft Bing. Based on the input to the web search engine, the search unit 104 searches for data on the target object and generates second text data, which includes an answer to the question, based on the search results.

[0039] The second text data generated by the web search engine is likely to be unnecessarily long. Next, the search unit 104 inputs the second text data, which includes the object detection result of the object detection unit 103 and the response sentence, into the visual language model. The visual language model is, for example, GPT-4. By doing so, the search unit 104 summarizes the second text data, which includes the generated response sentence. The response sentence is, for example, "The white car is..."

[0040] The configuration of the information processing system 1 according to the first embodiment has been described above. Next, the operation of the information retrieval system 100 in the information processing system 1 according to the first embodiment will be described using Figure 5.

[0041] Figure 5 is a flowchart showing an example of the operation of the information retrieval system 100 in the information processing system 1 according to the first embodiment.

[0042] As shown in Figure 5, in step S101, the input unit 101 of the information retrieval system 100 receives image data from the imaging device 10. The image data includes at least one candidate object. The input unit 101 also receives first voice data from the voice input device 20, which includes a question from the user. The question relates to at least one object among the at least one candidate object included in the image data. In step S102, the speech recognition unit 102 converts the first speech data into first text data by performing speech recognition processing.

[0043] In step S103, the object detection unit 103 detects a target object from among at least one candidate object contained in the image data that is related to the question sentence contained in the first text data, based on the image data and the first text data. In step S104, the search unit 104 searches for data on the target object based on the detection result and the first text data, and generates second text data including an answer to the question based on the search result. In step S105, the output unit 105 outputs the second text data to the output device 50.

[0044] As described above, the information retrieval system 100 of the information processing system 1 according to the second embodiment accepts input of image data containing at least one candidate object and first text data containing a question from the user. The information retrieval system 100 then inputs the image data and the first text data into an image segmentation model to narrow down (detect) the target object that is a candidate object related to the question contained in the first text data from among the at least one candidate object contained in the data. In this way, the information retrieval system 100 can generate an answer sentence that matches the question from the user using an AI model, even if the image data contains many candidate objects to be searched. In other words, the information retrieval system 100 can improve the accuracy of the answer sentence generated by the AI ​​model. Furthermore, the information processing system 1 can be implemented as a system that allows the user, who is the driver of the vehicle, to retrieve information while driving the vehicle. The information processing system 1 can utilize the ability of the image segmentation model described above to handle language data such as text data, enabling the retrieval of object information through a more intuitive interface based on the user's language utterances.

[0045] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, although the above-described information processing system 1 is primarily intended for information retrieval by a user while driving a vehicle, it can also be used as a method for intuitively retrieving information about objects within the robot's field of view during robot teleoperation. [Explanation of symbols]

[0046] 1 Information processing system, 10 Imaging device, 20 Voice input device, 30 Calculation unit, 40 External computing server, 50 Output device, 100 Information retrieval system, 101 Input unit, 102 Voice recognition unit, 103 Object detection unit, 104 Search unit, 105 Output unit

Claims

1. An input unit that accepts image data containing at least one candidate object and first text data containing a question from the user, An object detection unit inputs the image data and the first text data into an image segmentation model to detect a target object that is a candidate object related to the question sentence contained in the first text data from among at least one candidate object contained in the image data, A search unit that searches for information about the target object based on the detection result of the target object and the first text data, and generates second text data including an answer to the question based on the search result, The system comprises an output unit that outputs the second text data, Information retrieval system.

2. The aforementioned search unit, The detection results of the target object and the first text data are input into the visual language model to retrieve information about the target object, and second text data including an answer to the question is generated based on the search results. The information retrieval system according to claim 1.

3. The aforementioned search unit, The detection results of the target object and the first text data are entered into a web search engine to search for information about the target object, and based on the search results, second text data including an answer to the question is generated. By inputting the detection results of the target object and the second text data into the visual language model, the second text data is summarized. The information retrieval system according to claim 1.

4. The aforementioned input unit is Further input of facial image data including the user's face, The object detection unit, The user's gaze data is calculated from the facial image data, and the target object related to the question sentence contained in the first text data is detected from among at least one candidate object contained in the image data using the gaze data. The information retrieval system according to claim 1.

5. The aforementioned image segmentation model is The system learns based on training data in which image data containing a predetermined object is linked to text data related to that predetermined object. The information retrieval system according to claim 1.

Images