Input data conversion device, input data conversion method, and input data conversion program

Abstract

This input data conversion device is provided with: an acquisition unit that acquires input data which includes image data and a prompt and which is an input to a multimodal model; a detection unit that detects an object included in the image represented by the image data; a conversion unit that converts the input data using position information indicating the position of the object detected by the detection unit; and an input unit that enters the input data converted by the conversion unit into the multimodal model. Thus the input data conversion device can support decision-making that uses a language model generated by machine learning.

Description

Input data conversion device, input data conversion method, and input data conversion program

[0001] The present disclosure relates to an input data conversion device, an input data conversion method, and an input data conversion program.

[0002] Research is underway into multimodal models (MML) that can simultaneously learn and recognize language and images. Multimodal models allow for flexible changes in recognition tasks by changing the input image and language prompt. An example of a multimodal model technology is Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (BLIP2) technology, described in Non-Patent Document 1.

[0003] Junnan Li, et al., "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", 2023

[0004] The technology described in Non-Patent Document 1 does not have a mechanism for focusing on individual objects in an image, and therefore has the problem of reduced recognition accuracy for tasks that require attention to individual objects.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology that can improve the recognition accuracy of a task using a multimodal model without retraining the multimodal model.

[0006] An input data conversion device according to one exemplary aspect of the present disclosure includes an acquisition means for acquiring input data, which is input to a multimodal model and includes image data and a prompt; a detection means for detecting an object included in an image represented by the image data; a conversion means for converting the input data using position information indicating the position of the object detected by the detection means; and an input means for inputting the input data converted by the conversion means into the multimodal model.

[0007] An input data conversion method according to one exemplary aspect of the present disclosure includes an acquisition process in which at least one processor acquires input data, including image data and a prompt, which is input to a multimodal model; a detection process in which the at least one processor detects an object included in an image represented by the image data; a conversion process in which the at least one processor converts the input data using position information indicating the position of the object detected in the detection process; and an input process in which the at least one processor inputs the input data converted in the conversion process into the multimodal model.

[0008] An input data conversion program according to an exemplary aspect of the present disclosure functions as an acquisition means for acquiring input data, which is input to a multimodal model and includes image data and a prompt; a detection means for detecting an object included in an image represented by the image data; a conversion means for converting the input data using position information indicating the position of the object detected by the detection means; and an input means for inputting the input data converted by the conversion means into the multimodal model.

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that the recognition accuracy of a task using a multimodal model can be improved without retraining the multimodal model.

[0010] FIG. 1 is a block diagram showing a configuration of an input data conversion device according to the present disclosure. FIG. 2 is a flow diagram showing a flow of an input data conversion method according to the present disclosure. FIG. 3 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 4 is a block diagram showing an example of a configuration and processing flow of an application execution unit according to the present disclosure. FIG. 5 is a flow diagram showing an example of a flow of information processing executed by an information processing device according to the present disclosure. FIG. 6 is a block diagram showing an example of a configuration and processing flow of an application execution unit according to the present disclosure. FIG. 7 is a block diagram showing an example of a configuration and processing flow of an application execution unit according to the present disclosure. FIG. 8 is a block diagram showing an example of a configuration and processing flow of an application execution unit according to the present disclosure. FIG. 9 is a block diagram showing a configuration of an application execution unit according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Input Data Conversion Device) The configuration of the input data conversion device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the input data conversion device 1. As shown in Fig. 1, the input data conversion device 1 includes an acquisition unit 11, a detection unit 12, a conversion unit 13, and an input unit 14.

[0014] The acquisition unit 11 acquires input data including image data and prompts, which are input to the multimodal model. The detection unit 12 detects an object included in an image represented by the image data. The conversion unit 13 converts the input data using position information indicating the position of the object detected by the detection unit 12. The input unit 14 inputs the input data converted by the conversion unit 13 to the multimodal model.

[0015] (Effects of the Input Data Conversion Device) As described above, the input data conversion device 1 is configured to include an acquisition unit 11 that acquires input data including image data and a prompt, which is input to a multimodal model, a detection unit 12 that detects an object included in an image represented by the image data, a conversion unit 13 that converts the input data using position information indicating the position of the object detected by the detection unit 12, and an input unit 14 that inputs the input data converted by the conversion unit 13 to the multimodal model. Therefore, the input data conversion device 1 has the effect of improving the recognition accuracy of a task using a multimodal model without re-learning MML.

[0016] (Flow of Input Data Conversion Method) The flow of the input data conversion method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the input data conversion method S1. As shown in Fig. 2, the input data conversion method S1 includes an acquisition process S11, a detection process S12, a conversion process S13, and an input process S14.

[0017] In an acquisition process S11, at least one processor acquires input data including image data and a prompt, which is input to a multimodal model. In a detection process S12, at least one processor detects an object included in an image represented by the image data. In a conversion process S13, at least one processor converts the input data using position information indicating the position of the object detected in the detection process S12. In an input process S14, at least one processor inputs the input data converted in the conversion process S13 into the multimodal model.

[0018] (Effects of the Input Data Conversion Method) As described above, the input data conversion method S1 includes an acquisition process S11 for acquiring input data including image data and a prompt, which is input to a multimodal model, a detection process S12 for detecting an object included in an image represented by the image data, a conversion process S13 for converting the input data using position information indicating the position of the object detected in the detection process S12, and an input process S14 for inputting the input data converted in the conversion process S13 into the multimodal model. Therefore, the input data conversion method S1 has the effect of improving the recognition accuracy of a task using a multimodal model without re-learning multimodality.

[0019] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0020] <Configuration of Information Processing Device> The configuration of the information processing device 1A will be described with reference to FIG. 3. FIG. 3 is a block diagram showing the configuration of the information processing device 1A. The information processing device 1A is an example of an input data conversion device according to the present disclosure. The information processing device 1A is, for example, a general-purpose computer, and more specifically, is, for example, a smartphone or tablet terminal equipped with a camera or capable of having an external camera. Note that the information processing device 1A is not limited to these and may be a device other than those described above.

[0021] The information processing device 1A has a function for executing tasks using a multimodal model (hereinafter referred to as "MML"). Here, a multimodal model is a model capable of handling data in multiple formats, such as numerical values, images, text, and audio. As an example, the MML is stored in a storage device (such as an external server) connected to the information processing device 1A via a communication unit 30A. Here, "the MML is stored in a storage device" means that parameters defining the MML are stored in the storage device. In this case, the information processing device 1A transmits data to be input to the MML and receives data output from the MML via the communication unit 30A. The MML may also be stored in a storage unit 20A of the information processing device 1A.

[0022] The information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30A, an input unit 40A, and an output unit 50A.

[0023] (Communication Unit) The communication unit 30A communicates with devices external to the information processing device 1A via a communication line. While the specific configuration of the communication line does not limit the present exemplary embodiment, examples of the communication line include a wireless local area network (LAN), a wired LAN, a wide area network (WAN), a public line network, a mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices, and supplies data received from other devices to the control unit 10A.

[0024] (Input Unit) The input unit 40A is configured to receive input to the information processing device 1A, and includes, for example, input devices such as a keyboard, a mouse, a touch panel, a camera, a microphone, etc. The input unit 40A may also be configured to receive data from the input devices via an interface such as a USB (Universal Serial Bus).

[0025] (Output Unit) The output unit 50A is a component for performing output from the information processing device 1A, and includes, for example, output devices such as a display, a printer, a touch panel, a speaker, etc. The output unit 50A may be configured to include, for example, an interface such as a USB, and to output data to the output device via the interface.

[0026] (Storage Unit) The storage unit 20A stores various types of information referenced by the control unit 10A. Examples of such information include image data D1 and a prompt D2. The image data D1 and the prompt D2 are data acquired by the information processing device 1A as input to MML.

[0027] (Control Unit) The control unit 10A includes an application execution unit 16A. The application execution unit 16A has the function of executing tasks using MML. The application execution unit 16A includes an image data acquisition unit 11A, a prompt acquisition unit 12A, a detection unit 13A, a similarity calculation unit 14A, and a conversion unit 15A. The image data acquisition unit 11A and the prompt acquisition unit 12A are examples of acquisition means according to the present disclosure. The detection unit 13A is an example of detection means according to the present disclosure. The similarity calculation unit 14A is an example of calculation means according to the present disclosure. The conversion unit 15A is an example of conversion means and input means according to the present disclosure.

[0028] (Application Execution Unit) Fig. 4 is a block diagram showing an example of the configuration and processing flow of the application execution unit 16A. Note that the unidirectional arrows in Fig. 4 simply indicate the direction of flow of a certain signal (data) or the direction of flow of processing, and do not exclude bidirectionality.

[0029] (Image Data Acquisition Unit) In Fig. 4, the image data acquisition unit 11A acquires image data that is input to the MML 17A. As an example, the image data is moving image data composed of a sequence of frames. In other words, as an example, the image data acquisition unit 11A acquires a sequence of frames as image data. However, the image data acquired by the image data acquisition unit 11A is not limited to moving image data, and may also be data representing still images.

[0030] As an example, the image data acquisition unit 11A acquires image data captured by an imaging device included in the input unit 40A from the imaging device. Alternatively, the image data acquisition unit 11A may acquire the image data by reading it from a storage location (which may be within the storage device of the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. Alternatively, the image data acquisition unit 11A may receive image data from another device via the communication unit 30A.

[0031] (Prompt Acquisition Unit) The prompt acquisition unit 12A acquires a prompt, which is input to the MML 17A. Here, a prompt is text, which is input to the MML 17A. Examples of a prompt include text such as "How many people are wearing masks in the image?" or text such as "How many cars are there?"

[0032] For example, the prompt acquisition unit 12A acquires a prompt input by a user via a touch panel included in the input unit 40A. The prompt acquisition unit 12A may also acquire the prompt by reading it from a storage location (which may be in a storage device of the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. The prompt acquisition unit 12A may also receive a prompt from another device via the communication unit 30A. The prompt acquisition unit 12A may also acquire, as a prompt, text obtained by performing speech recognition processing on voice data output by a microphone included in the input unit 40A. The prompt acquisition unit 12A may also acquire, as a prompt, text obtained by performing character recognition processing on an image captured by a camera included in the input unit 40A.

[0033] (Detection Unit) The detection unit 13A acquires image data and detects objects included in the image represented by the acquired image data. Examples of objects detected by the detection unit 13A from an image include, but are not limited to, people, vehicles, animals, buildings, signs, etc. In the present disclosure, an object includes a character or a character string included in an image. As an example, the detection unit 13A detects an object from an image using an object detection model generated by machine learning. Examples of object detection models include R-CNN (Regions with CNN features), Fast R-CNN, Faster R-CNN, SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), etc., but the object detection model is not limited to these.

[0034] As another example, the detection unit 13A may detect an object without using an object detection model. For example, the detection unit 13A may detect a person based on the difference between a reference image captured in a workplace or the like when no people are present and an image represented by the acquired image data.

[0035] The detection unit 13A outputs position information indicating the position of the detected object to the similarity calculation unit 14A and the conversion unit 15A. When the detection unit 13A detects multiple objects, it outputs the position information of each of the detected multiple objects to the similarity calculation unit 14A and the conversion unit 15A. Furthermore, the detection unit 13A may output information identifying the object in addition to the position information of the object to the similarity calculation unit 14A and the conversion unit 15A. The information identifying the object is, for example, text indicating the name of the object.

[0036] (Similarity Calculation Unit) The similarity calculation unit 14A calculates the similarity between the object detected by the detection unit 13A and the prompt acquired by the prompt acquisition unit 12A. In the present disclosure, a larger similarity value indicates a higher degree of similarity. As an example, the similarity calculation unit 14A converts the prompt acquired by the prompt acquisition unit 12A and information identifying the object detected by the detection unit 13A (such as text representing the object name) into vectors using a language model, and calculates the cosine similarity between the two vectors. Examples of language models that can be used to convert text into vectors include, but are not limited to, Bidirectional Encoder Representations from Transformers (BERT), word2vec, and doc2vec. The similarity calculated by the similarity calculation unit 14A is not limited to cosine similarity. For example, the similarity calculation unit 14A may compare the prompt text with the object name text and calculate the similarity based on the degree of agreement between the two.

[0037] When the detection unit 13A detects multiple objects, the similarity calculation unit 14A calculates the similarity between each object and the prompt for each of the multiple objects, and outputs the calculated similarity to the conversion unit 15A.

[0038] The conversion unit 15A converts the prompt using position information indicating the position of an object, of one or more objects detected by the detection unit 13A, whose similarity calculated by the similarity calculation unit 14A satisfies a predetermined condition. Examples of objects whose similarity satisfies the predetermined condition include objects whose similarity is higher than a predetermined threshold, objects whose similarity is higher than the average, and objects whose ranking when sorted by similarity is within a predetermined rank.

[0039] (Specific Example 1 of Conversion Processing) As an example, the conversion unit 15A adds information indicating the name and position information of the object detected by the detection unit 13A to the prompt. For example, if the prompt is text such as "How many people are wearing masks in the image?" and the objects whose similarity is equal to or greater than the threshold are a person at position X1, a person at position X2, and a person at position X3, the conversion unit 15A converts the prompt to text such as "People appear at positions X1, X2, and X3 in the image. How many people are wearing masks in the image?" As another example, if the prompt is text such as "Looking for a restaurant" and the objects whose similarity is equal to or greater than the threshold are signs at positions X1, X2, and X3, the conversion unit 15A converts the prompt to text such as "Looking for a restaurant. Signs appear at positions X1, X2, and X3 in the image." In other words, in this example, the conversion unit 15A can also add text describing the object detected by the detection unit 13A to the prompt.

[0040] (Specific Example 2 of Conversion Processing) As another example, the conversion unit 15A may add only the position information of the object to the prompt. For example, if the prompt is a text such as "How many people are wearing masks in the image?" and positions X1, X2, and X3 are detected as positions of objects whose similarity calculated by the similarity calculation unit 14A is equal to or greater than a threshold, the conversion unit 15A converts the prompt to a text such as "Objects are shown at X1, X2, and X3 in the image. How many people are wearing masks in the image?"

[0041] <Information Processing Flow> Fig. 5 is a flow diagram showing an example of the flow of information processing executed by the information processing device 1 A. The processing shown in Fig. 5 is started, for example, when a user performs an operation to start an application using an input device such as a touch panel.

[0042] In step S21, the image data acquisition unit 11A acquires image data, and the prompt acquisition unit 12A acquires a prompt. The image data acquired by the image data acquisition unit 11A is, for example, data representing an image captured by a camera built into the information processing device 1A, and the prompt is, for example, a predetermined fixed phrase or text input by the user using an input device such as a touch panel.

[0043] In step S22, the detection unit 13A executes an object detection process on the image data acquired by the image data acquisition unit 11 A. In step S23, the similarity calculation unit 14A calculates the similarity between the detected object and the prompt.

[0044] In step S24, the conversion unit 15A converts the input data by adding to the prompt information indicating the name and position of an object detected by the detection unit 13A whose similarity calculated by the similarity calculation unit 14A satisfies a predetermined condition.

[0045] In step S25, the application execution unit 16A inputs the prompt converted by the conversion unit 15A and the image data acquired by the image data acquisition unit 11A to the MML 17A. In step S26, the application execution unit 16A acquires the output of the MML 17A. The output of the MML 17A includes, for example, text indicating the recognition result of the task indicated by the prompt. More specifically, the output of the MML 17A includes the text "There are five people wearing masks." As another example, the output of the MML 17A may include text describing a restaurant included in the image data.

[0046] In step S27, the application execution unit 16A outputs the output of the MML 17A to the output unit 50A. For example, the application execution unit 16A displays text, which is the output of the MML 17A, on a display. The application execution unit 16A may also output the output of the MML 17A as sound from a speaker.

[0047] In step S28, the application execution unit 16A determines whether to terminate the application (app). If the application is to be terminated (YES in step S28), the application execution unit 16A terminates the process. On the other hand, if the application is not to be terminated (NO in step S28), the application execution unit 16A returns to the process of step S21 and continues the task recognition process using the MML 17A.

[0048] <Application Examples> (Application Example 1) The information processing device 1A is used in various situations. For example, when searching for a person or a lost child, a person with a specific characteristic may be searched for in a video. In this case, when the information processing device 1A receives a prompt such as "How many men are wearing sunglasses?", the information processing device 1A extracts people from the image and adds text indicating the position of the extracted people to the prompt. By inputting the prompt with the added text to the MML 17A, it is possible to obtain an output of the MML 17A with higher recognition accuracy.

[0049] (Application Example 2) Another example is traffic monitoring. For example, when checking or monitoring the traffic volume of a specific vehicle model, the information processing device 1A adds text indicating the positions of the vehicles extracted from the image to a prompt such as "How many vehicles of vehicle model 'xyz' are there?" and inputs the added text to the MML 17A. This makes it possible to obtain an output from the MML 17A with higher recognition accuracy.

[0050] (Application Example 3) The information processing device 1A according to the present disclosure can also be used in the medical and healthcare fields. For example, in a video surveillance system aimed at ensuring the safety of people moving around within a hospital, it is conceivable to estimate the behavior of a target person, such as a patient, and output an alert based on the estimation result. In this case, the information processing device 1A, for example, performs an object detection process on video of the hospital captured by a camera, and inputs a prompt with information indicating the location of the detected object and the video captured by the camera into the MML 17A. More specifically, the information processing device 1A may convert the prompt, for example, "Is anyone engaging in dangerous behavior?" by adding the text "People are visible at positions X1, X2, and X3."

[0051] Furthermore, when the information processing device 1A determines, based on the information output from the MML 17A, that a subject is engaging in dangerous behavior, it outputs an alert to hospital staff, etc. Specifically, the information processing device 1A may, for example, transmit an alert to a terminal owned by the hospital staff via the communication unit 30A, or may output the alert to an output device (for example, a display, a speaker, etc.) connected to the output unit 50A.

[0052] In this example, the targets of object detection are, for example, patients, medical staff (including medical professionals), and visitors. The targets of object detection also include, for example, medical equipment such as intravenous drip equipment, trolleys, wheelchairs, and the like.

[0053] The information processing device 1A may perform risk estimation based on video acquired in real time and issue an alert, or may perform risk estimation collectively based on accumulated video and output it as a report.

[0054] <Effects of Information Processing Device> The BLIP2 technology described in Non-Patent Document 1 does not have a mechanism for focusing on individual objects, resulting in reduced recognition accuracy in tasks that require focusing on individual objects. More specifically, for example, it may fail to detect an object in a task requiring counting, or may give an incorrect answer to a question about a specific object's appearance. More specifically, for example, when a user is prompted to count the number of people wearing glasses in an image containing multiple people, the user may fail to recognize the object because they are unable to focus on each individual person. In contrast, with the information processing device 1A, the detection unit 13A identifies the position of an object requiring attention and, based on the identified object's position, shapes the prompt in a way that allows attention to individual objects. By shaping the prompt in this manner, recognition processing can be performed while focusing on individual objects without re-learning the MML 17A. This improves the recognition accuracy of tasks using the MML 17A.

[0055] Furthermore, in the information processing device 1A, the conversion unit 15A is configured to add information indicating the name and position of the object detected by the detection unit 13A to the input data. Therefore, according to the information processing device 1A, by incorporating information indicating the name of the object detected from the image into the prompt, it is possible to perform recognition processing while focusing on each object detected from the image, and it is possible to improve the recognition accuracy of the task using the MML 17A.

[0056] Furthermore, in the information processing device 1A, the conversion unit 15A is configured to add text describing the object detected by the detection unit 13A to the prompt. By adding text describing the object detected from the image to the prompt, it is possible to perform the recognition process while focusing on each individual object detected from the image, and it is possible to improve the recognition accuracy of the task using the MML 17A.

[0057] The information processing device 1A further includes a similarity calculation unit 14A that calculates the similarity between an object detected by the detection unit 13A and a prompt included in the input data, and the conversion unit 15A converts the input data using information indicating the position of an object, of one or more objects detected by the detection unit 13A, whose similarity calculated by the similarity calculation unit 14A satisfies a predetermined condition. This allows the recognition process to focus on objects that are more closely related to the prompt, thereby improving the recognition accuracy of tasks using the MML 17A.

[0058] [Third Exemplary Embodiment] The configuration of the application execution unit according to the present disclosure is not limited to the example shown in Fig. 4. Fig. 6 is a block diagram showing another example of the configuration and processing flow of the application execution unit 16B according to the present disclosure. Note that the unidirectional arrows in Fig. 6 simply indicate the direction of flow of a certain signal (data) or the direction of flow of processing, and do not exclude bidirectionality.

[0059] The application execution unit 16B shown in FIG. 6 includes an image data acquisition unit 11B, a prompt acquisition unit 12B, and a conversion unit 15B in addition to the detection unit 13A and similarity calculation unit 14A shown in FIG. 4 . The image data acquisition unit 11B outputs acquired image data to the detection unit 13A and the conversion unit 15B. The conversion unit 15B converts input data by superimposing an image representing the position of an object detected by the detection unit 13A on an image representing the image data of the object detected by the detection unit 13A. Here, the image representing the position of the object may be, for example, a frame shape (rectangular frame, elliptical frame) surrounding the object, or may be, for example, a shape such as an arrow pointing to the object. The application execution unit 16B inputs the image data converted by the conversion unit 15B and the prompt acquired by the prompt acquisition unit 12B to the MML 17A.

[0060] In this configuration example, the conversion unit 15B is configured to superimpose an image representing the position of an object detected by the detection unit 13A on the image represented by the image data. By including information representing the position of an object detected from the image in the image data, it is possible to perform recognition processing while focusing on each object detected from the image, thereby improving the recognition accuracy of the task using the MML 17A.

[0061] 7 is a block diagram showing an example of the configuration and processing flow of an application execution unit 16C according to the present disclosure. Note that the unidirectional arrows in Fig. 7 simply indicate the direction of a signal (data) flow or the direction of processing flow, and do not exclude bidirectionality.

[0062] 7 includes a detection unit 13C and a conversion unit 15C in addition to the image data acquisition unit 11A, prompt acquisition unit 12A, and similarity calculation unit 14A shown in Fig. 4. The detection unit 13C performs character recognition processing on the image data acquired by the image data acquisition unit 11A to detect text from the image data.

[0063] The conversion unit 15C converts the input data using the text detected by the detection unit 13C and information indicating the position of the text. For example, if the prompt is "Looking for a restaurant," the conversion unit 15C converts the prompt by adding the following text to the prompt: "The image contains the characters 'abc' at position X1. The image contains the characters 'xyz' at position X2. The image contains the characters 'αβγ' at position X3." The application execution unit 16C inputs the prompt converted by the conversion unit 15C and the image data acquired by the image data acquisition unit 11A to the MML 17A.

[0064] For example, a user may take a picture of a street scene with the camera of a mobile device and input a prompt such as "I'm looking for a restaurant." In this case, the application execution unit 16C uses character recognition to read store signs in the image, analyzes the image based on that information, and recommends restaurants or navigates to the destination. The application execution unit 16C adds text indicating the location of signs and other items extracted from the image to the prompt and inputs the text to the MML 17A. This allows the MML 17A to output with higher recognition accuracy.

[0065] In this configuration example, the detection unit 13C performs character recognition processing on the image data to detect text from the image data, and the conversion unit 15C converts the input data using the text detected by the detection unit 13C and information indicating the position of the text. By converting the input data using the text obtained by the character recognition processing, it is possible to perform recognition processing that takes into account the content of the text included in the image, thereby improving the recognition accuracy of tasks using the MML 17A.

[0066] [Fifth Exemplary Embodiment] (Another Configuration Example 3 of the Application Execution Unit) Fig. 8 is a block diagram showing an example of the configuration and processing flow of the application execution unit 16D according to the present disclosure. Note that the unidirectional arrows in Fig. 8 simply indicate the direction of flow of a certain signal (data) or the direction of flow of processing, and do not exclude bidirectionality.

[0067] The application execution unit 16D shown in Fig. 8 includes a detection unit 13D and a conversion unit 15D in addition to the image data acquisition unit 11A, prompt acquisition unit 12A, and similarity calculation unit 14A shown in Fig. 4. In this example, the image data acquisition unit 11A acquires moving image data that is a frame sequence.

[0068] The detection unit 13D detects an area where an image has changed in the frame sequence acquired by the image data acquisition unit 11A. The conversion unit 15D converts the input data using position information indicating the area detected by the detection unit 13D. As an example, the conversion unit 15D may add text such as "A change has been detected in area X" to the prompt. The application execution unit 16D inputs the prompt converted by the conversion unit 15D and the image data acquired by the image data acquisition unit 11A to the MML 17A.

[0069] According to this configuration example, the image data acquisition unit 11A acquires a frame sequence as image data, the detection unit 13D detects an area in the frame sequence where the image has changed, and the conversion unit 15D converts the input data using position information indicating the area detected by the detection unit 13D. By inputting the input data converted using information indicating the area in which a change has been detected in the image to the MML 17A, it is possible to perform recognition processing while focusing on the area in which a change has been detected, thereby improving the recognition accuracy of tasks using the MML 17A.

[0070] [Sixth Exemplary Embodiment] Fig. 9 is a block diagram showing an example of the configuration and processing flow of an application execution unit 16E. Note that the unidirectional arrows in Fig. 9 simply indicate the direction of flow of a certain signal (data) or the direction of flow of processing, and do not exclude bidirectionality.

[0071] The application execution unit 16E shown in FIG. 9 includes a detection unit 13E in addition to the image data acquisition unit 11A, prompt acquisition unit 12A, similarity calculation unit 14A, and conversion unit 15A shown in FIG. 4. The detection unit 13E detects the position of an object from a frame in which an image has changed in the frame sequence. The conversion unit 15A converts at least one of the frame in which the image has changed and the prompt using position information indicating the position of the object detected by the detection unit 13E. In other words, the application execution unit 16E extracts frames in which a change has occurred from multiple frames constituting the video image data, inputs the extracted frames to the MML 17A, and does not input frames in which no change has occurred to the MML. The application execution unit 16E inputs the frames detected by the detection unit 13E and the prompt converted by the conversion unit 15A to the MML 17A.

[0072] As described above, in this configuration example, the image data acquisition unit 11A acquires a frame sequence as image data, the detection unit 13E detects the position of an object from a frame in which an image has changed in the frame sequence, and the conversion unit 15A converts at least one of the frame in which the image has changed and the prompt using position information indicating the position of the object detected by the detection unit 13E. By inputting the frame in which a change has occurred to the MML 17A in this way, the processing load in the task recognition process using the MML 17A can be reduced compared to when all frames are input to the MML 17A.

[0073] [Modification] In the information processing device 1A described above, the conversion unit 15A converts the input data using the position information of an object, of the plurality of objects detected by the detection unit 13A, whose similarity satisfies a predetermined requirement. However, the conversion process performed by the conversion unit 15A is not limited to the above example. As an example, the conversion unit 15A may convert the input data using the position information of all objects detected by the detection unit 13A. In this case, the information processing device 1A may be configured without the similarity calculation unit 14A.

[0074] [Example of implementation by software] Some or all of the functions of the input data conversion device 1 and the information processing device 1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0075] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Figure 10 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0076] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0077] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0078] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0079] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0080] [Appendix 1] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0081] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0082] (Appendix A1) An input data conversion device comprising: an acquisition means for acquiring input data, which is input to a multimodal model and includes image data and a prompt; a detection means for detecting an object included in an image represented by the image data; a conversion means for converting the input data using position information indicating the position of the object detected by the detection means; and an input means for inputting the input data converted by the conversion means into the multimodal model.

[0083] (Appendix A2) The input data conversion device according to Appendix A1, wherein the conversion means adds information indicating a position and information indicating a name of the object detected by the detection means to the input data.

[0084] (Appendix A3) The input data conversion device according to appendix A1 or A2, wherein the conversion means adds text describing the object detected by the detection means to the prompt.

[0085] (Appendix A4) The input data conversion device according to any one of Appendices A1 to A3, wherein the conversion means superimposes an image representing a position of the object detected by the detection means on the image represented by the image data.

[0086] (Appendix A5) The input data conversion device according to any one of Appendices A1 to A4, wherein the detection means performs character recognition processing on the image data to detect text from the image data, and the conversion means converts the input data using the text detected by the detection means and information indicating a position of the text.

[0087] (Appendix A6) An input data conversion device according to any one of Appendices A1 to A5, further comprising a calculation means for calculating a similarity between an object detected by the detection means and the prompt included in the input data, wherein the conversion means converts the input data using information indicating the position of an object, of one or more objects detected by the detection means, whose similarity calculated by the calculation means satisfies a predetermined condition.

[0088] (Appendix A7) The input data conversion device according to any one of Appendices A1 to A6, wherein the acquisition means acquires a frame sequence as the image data, the detection means detects an area in the frame sequence where an image has changed, and the conversion means converts the input data using position information indicating the area detected by the detection means.

[0089] (Appendix A8) The input data conversion device according to any one of Appendices A1 to A7, wherein the acquisition means acquires a frame sequence as the image data, the detection means detects the position of the object from a frame in which an image has changed in the frame sequence, and the conversion means converts at least one of the frame in which the image has changed and the prompt using position information indicating the position of the object detected by the detection means.

[0090] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0091] (Appendix B1) An input data conversion method including: an acquisition process in which at least one processor acquires input data, which is input to a multimodal model and includes image data and a prompt; a detection process in which the at least one processor detects an object included in an image represented by the image data; a conversion process in which the at least one processor converts the input data using position information indicating the position of the object detected in the detection process; and an input process in which the at least one processor inputs the input data converted in the conversion process into the multimodal model.

[0092] (Supplementary Note B2) The input data conversion method according to Supplementary Note B1, wherein in the conversion process, the at least one processor adds information indicating a position and information indicating a name of an object detected in the detection process to the input data.

[0093] (Supplementary Note B3) The input data conversion method according to Supplementary Note B1 or B2, wherein in the conversion process, the at least one processor adds text to the prompt that describes an object detected in the detection process.

[0094] (Supplementary Note B4) The input data conversion method according to any one of Supplementary Notes B1 to B3, wherein in the conversion process, the at least one processor superimposes an image representing a position of an object detected in the detection process on an image represented by the image data.

[0095] (Appendix B5) An input data conversion method according to any one of Appendices B1 to B4, wherein the at least one processor, in the detection process, performs character recognition processing on the image data to detect text from the image data, and in the conversion process, the at least one processor converts the input data using the text detected in the detection process and information indicating the position of the text.

[0096] (Appendix B6) An input data conversion method described in any one of Appendices B1 to B5, further comprising a calculation process in which the at least one processor calculates a similarity between an object detected in the detection process and the prompt included in the input data, and in the conversion process, the at least one processor converts the input data using information indicating the position of an object, of one or more objects detected in the detection process, whose similarity calculated in the calculation process satisfies a predetermined condition.

[0097] (Appendix B7) An input data conversion method according to any one of Appendices B1 to B6, wherein in the acquisition process, the at least one processor acquires a frame sequence as the image data; in the detection process, the at least one processor detects an area in the frame sequence where an image has changed; and in the conversion process, the at least one processor converts the input data using position information indicating the area detected in the detection process.

[0098] (Appendix B8) An input data conversion method according to any one of Appendices B1 to B7, wherein in the acquisition process, the at least one processor acquires a frame sequence as the image data; in the detection process, the at least one processor detects the position of the object from a frame in the frame sequence where an image has changed; and in the conversion process, the at least one processor converts at least one of the frame in which the image has changed and the prompt using position information indicating the position of the object detected in the detection process.

[0099] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0100] (Appendix C1) An input data conversion program that causes a computer to function as an input data conversion device, the input data conversion program causing the computer to function as: an acquisition means that acquires input data including image data and a prompt, which is input to a multimodal model; a detection means that detects an object included in an image represented by the image data; a conversion means that converts the input data using position information that indicates the position of the object detected by the detection means; and an input means that inputs the input data converted by the conversion means into the multimodal model.

[0101] (Supplementary Note C2) The input data conversion program according to Supplementary Note C1, wherein the conversion means adds information indicating a position and information indicating a name of the object detected by the detection means to the input data.

[0102] (Supplementary Note C3) The input data conversion program according to Supplementary Note C1 or C2, wherein the conversion means adds text describing the object detected by the detection means to the prompt.

[0103] (Supplementary Note C4) The input data conversion program according to any one of Supplementary Notes C1 to C3, wherein the conversion means superimposes an image representing a position of the object detected by the detection means on an image represented by the image data.

[0104] (Appendix C5) The input data conversion program according to any one of Appendices C1 to C4, wherein the detection means performs character recognition processing on the image data to detect text from the image data, and the conversion means converts the input data using the text detected by the detection means and information indicating a position of the text.

[0105] (Appendix C6) An input data conversion program according to any one of Appendices C1 to C5, further causing the computer to function as a calculation means for calculating a similarity between an object detected by the detection means and the prompt included in the input data, and the conversion means converts the input data using information indicating the position of an object, of one or more objects detected by the detection means, whose similarity calculated by the calculation means satisfies a predetermined condition.

[0106] (Appendix C7) The input data conversion program according to any one of Appendices C1 to C6, wherein the acquisition means acquires a frame sequence as the image data, the detection means detects an area in the frame sequence where an image has changed, and the conversion means converts the input data using position information indicating the area detected by the detection means.

[0107] (Appendix C8) The input data conversion program according to any one of Appendices C1 to C7, wherein the acquisition means acquires a frame sequence as the image data, the detection means detects the position of the object from a frame in the frame sequence where an image has changed, and the conversion means converts at least one of the frame in which the image has changed and the prompt using position information indicating the position of the object detected by the detection means.

[0108] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0109] (Appendix D1) An input data conversion device comprising at least one processor, the at least one processor executing: an acquisition process for acquiring input data, which is input to a multimodal model and includes image data and a prompt; a detection process for detecting an object included in an image represented by the image data; a conversion process for converting the input data using position information indicating the position of the object detected in the detection process; and an input process for inputting the input data converted in the conversion process into the multimodal model.

[0110] (Supplementary Note D2) The input data conversion device according to Supplementary Note D1, wherein in the conversion process, the at least one processor adds information indicating a position and information indicating a name of an object detected in the detection process to the input data.

[0111] (Appendix D3) The input data conversion device according to appendix D1 or D2, wherein in the conversion process, the at least one processor adds text to the prompt that describes an object detected in the detection process.

[0112] (Supplementary Note D4) The input data conversion device according to any one of Supplementary Notes D1 to D3, wherein in the conversion process, the at least one processor superimposes an image representing a position of an object detected in the detection process on an image represented by the image data.

[0113] (Appendix D5) The input data conversion device according to any one of Appendices D1 to D4, wherein in the detection process, the at least one processor performs character recognition processing on the image data to detect text from the image data, and in the conversion process, the at least one processor converts the input data using the text detected in the detection process and information indicating the position of the text.

[0114] (Appendix D6) An input data conversion device described in any one of Appendices D1 to D5, wherein the at least one processor further executes a calculation process to calculate a similarity between an object detected in the detection process and the prompt included in the input data, and in the conversion process, the at least one processor converts the input data using information indicating the position of an object, of one or more objects detected in the detection process, whose similarity calculated in the calculation process satisfies a predetermined condition.

[0115] (Appendix D7) An input data conversion device described in any one of Appendices D1 to D6, wherein in the acquisition process, the at least one processor acquires a frame sequence as the image data; in the detection process, the at least one processor detects an area where an image has changed in the frame sequence; and in the conversion process, the at least one processor converts the input data using position information indicating the area detected in the detection process.

[0116] (Appendix D8) An input data conversion device described in any one of Appendices D1 to D7, wherein in the acquisition process, the at least one processor acquires a frame sequence as the image data; in the detection process, the at least one processor detects the position of the object from a frame in the frame sequence where an image has changed; and in the conversion process, the at least one processor converts at least one of the frame in which the image has changed and the prompt using position information indicating the position of the object detected in the detection process.

[0117] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0118] (Appendix E1) A non-transitory recording medium having recorded thereon an input data conversion program that causes a computer to function as an input data conversion device, the input data conversion program causing the computer to execute: an acquisition process that acquires input data including image data and a prompt, which is input to a multimodal model; a detection process that detects an object included in an image represented by the image data; a conversion process that converts the input data using position information indicating the position of the object detected in the detection process; and an input process that inputs the input data converted in the conversion process into the multimodal model.

[0119] 1 Input data conversion device 1A Information processing device 11 Acquisition unit 11A, 11B Image data acquisition unit 12, 13A, 13C, 13D, 13E Detection unit 12A, 12B Prompt acquisition unit 13, 15A, 15B, 15C, 15D Conversion unit 14, 40A Input unit 14A Similarity calculation unit 17A MML 16A, 16B, 16C, 16D, 16E Application execution unit 20A Storage unit 30A Communication unit 50A Output unit

Claims

1. An acquisition means for acquiring input data, including image data and prompts, which are inputs to a multimodal model, A detection means for detecting objects included in the image represented by the aforementioned image data, A conversion means that converts the input data using position information indicating the position of an object detected by the detection means, An input means for inputting the input data converted by the conversion means into the multimodal model, An input data conversion device equipped with the following features.

2. The conversion means adds information indicating the location and name of the object detected by the detection means to the input data. The input data conversion device according to claim 1.

3. The conversion means adds text describing the object detected by the detection means to the prompt. The input data conversion device according to claim 1 or 2.

4. The conversion means superimposes an image representing the position of the object detected by the detection means onto the image represented by the image data. The input data conversion device according to claim 1 or 2.

5. The detection means performs character recognition processing on the image data to detect text from the image data, The conversion means converts the input data using the text detected by the detection means and information indicating the location of the text. The input data conversion device according to claim 1 or 2.

6. The system further comprises a calculation means for calculating the similarity between the object detected by the detection means and the prompt included in the input data, The conversion means converts the input data using information indicating the location of one or more objects detected by the detection means, the similarity calculated by the calculation means satisfying predetermined conditions. The input data conversion device according to claim 1 or 2.

7. The acquisition means acquires a frame sequence as the image data, The detection means detects the region in the frame sequence where the image has changed, The conversion means converts the input data using position information indicating the region detected by the detection means. The input data conversion device according to claim 1 or 2.

8. The acquisition means acquires a frame sequence as the image data, The detection means detects the position of the object from the frame in the frame sequence where the image has changed. The conversion means uses position information indicating the position of the object detected by the detection means to convert at least one of the frame in which the image has changed and the prompt. The input data conversion device according to claim 1 or 2.

9. At least one processor performs an acquisition process to obtain input data, including image data and prompts, which are inputs to a multimodal model. The at least one processor performs a detection process to detect objects included in the image represented by the image data, The at least one processor performs a conversion process that converts the input data using position information indicating the position of an object detected in the detection process, The at least one processor performs an input process that inputs the input data converted in the conversion process into the multimodal model, A method for converting input data, including the conversion of input data.

10. A program that makes a computer function as an input data conversion device, The aforementioned computer, An acquisition means for acquiring input data, including image data and prompts, which are inputs to a multimodal model, A detection means for detecting objects included in the image represented by the aforementioned image data, A conversion means that converts the input data using position information indicating the position of an object detected by the detection means, An input means for inputting the input data converted by the conversion means into the multimodal model, An input data conversion program that functions as such.