Information processing device, prompt generation method, and computer program

Abstract

In this technology that enables the use of AI in 3D space, it allows input data to be processed using a model designed for standard 2D images, eliminating the need for 3D-specific training, and reducing data size compared to conventional methods while improving scalability. [Solution] The information processing device comprises an acquisition unit that acquires a plurality of 2D images representing an arbitrary 3D space and a question sentence, a database generation unit, a prompt generation unit, and an output unit. The database generation unit estimates the position and orientation of the camera when each of the plurality of images was taken, calculates a 3D point cloud of the objects captured in the plurality of images, and generates a feature database that associates each image with the camera position and orientation and 3D point cloud data corresponding to each image. The prompt generation unit generates a prompt that includes a prerequisite statement for the LLM, at least a portion of the images in the feature database and the camera position and orientation corresponding to those images, and a question for the LLM.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a prompt generation method, and a computer program.

Background Art

[0002] In recent years, generative AI has been utilized. Generative AI is an AI that generates content such as text, images, and audio according to an input prompt. Generative AI can be said to be a technology specialized in generating new data and content among artificial intelligence (AI). A prompt is data used to ask questions to generative AI. For example, many existing generative AIs such as ChatGPT (registered trademark) are constructed based on large language models (LLMs). A large language model is a language model constructed by a large amount of teacher data and deep learning technology, and is one of the technologies responsible for natural language processing (NLP) in AI.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, there is a desire to use generative AI on 3D space, for example, by "training the generative AI on a 3D space and then asking it questions (providing prompts) about that 3D space." However, 3D data is difficult to handle from a 2D computer, and there are not enough 3D datasets available on the network. Non-patent document 1 describes how to feature a 3D space using voxels to enable asking questions about that 3D space. However, the technology described in non-patent document 1 requires a considerable amount of pre-training using 3D data, resulting in a large data size and poor scalability. It should be noted that this problem is not limited to generative AI, but is also common when using AI built on large-scale language models, such as natural language processing AI.

[0005] The present invention was made to solve at least some of the problems described above, and aims to enable the use of AI in 3D space by allowing input data to be processed with a model designed for ordinary 2D images, eliminating the need for 3D-specific learning, and improving scalability by reducing the data size compared to conventional methods. [Means for solving the problem]

[0006] The present invention has been made to solve at least some of the above-mentioned problems and can be realized in the following forms.

[0007] (1) According to one embodiment of the present invention, an information processing device is provided. This information processing device includes: an acquisition unit that acquires a plurality of two-dimensional images representing an arbitrary three-dimensional space, which are a plurality of different images taken using a camera, and a question sentence made by a user; a database generation unit that generates a feature database using the plurality of images; a prompt generation unit that generates a prompt to be given to a large-scale language model of an AI using the feature database; and an output unit that outputs the generated prompt. The database generation unit estimates the position and orientation of the camera when each of the plurality of images was taken, calculates a three-dimensional point cloud of objects depicted in the plurality of images, and generates the feature database which stores each of the plurality of images, the position and orientation of the camera corresponding to each image, and data relating to the three-dimensional point cloud in association with each image. The prompt generation unit generates the prompt which includes a premise sentence for the large-scale language model, at least a portion of the images stored in the feature database and the position and orientation of the camera corresponding to those images, and a user question for the large-scale language model which is a user question generated from the question sentence.

[0008] With this configuration, a large-scale AI language model can understand any 3D space that is the subject of a user question, using at least some of the multiple images (2D images) stored in the feature database and the camera position and orientation associated with each of those images. Therefore, this configuration makes it possible to teach a model that has already been trained for 2D images to understand 3D space. In other words, it becomes possible to process input data with a model that is normally trained for 2D images, eliminating the need for 3D-specific training. Furthermore, the data size of the input data can be reduced compared to when 3D-specific training is performed, and scalability can be improved as the data size is reduced.

[0009] (2) In the above-described information processing device, the database generation unit may further approximate the calculated three-dimensional point cloud with a Gaussian distribution and use the obtained distribution information as data relating to the three-dimensional point cloud. In this configuration, the database generation unit approximates the 3D point cloud with a Gaussian distribution and uses the resulting distribution information as data related to the 3D point cloud. Since the distribution information has a smaller data size compared to the 3D point cloud, the database generation unit can reduce the capacity of the feature database.

[0010] (3) In the above-described information processing device, the database generation unit may further estimate a depth image for each of the multiple images using a pre-prepared depth estimation model, and estimate the position and orientation of the camera for each of the multiple images using the multiple images and the estimated depth image. In this configuration, the database generation unit uses multiple images and estimated depth images to estimate the camera's position and orientation for each image included in the multiple images. Therefore, the database generation unit can estimate the camera's position and orientation when the images were taken, even if the multiple images acquired by the acquisition unit are not RGBD images. An RGBD image is an image in which RGB pixels and depth are pre-associated.

[0011] (4) In the above-described information processing device, the database generation unit may further extract image language features from each of the multiple images using a pre-prepared image language model, and generate a feature database which stores each of the multiple images, the camera position and orientation corresponding to each image, the data relating to the 3D point cloud, and the image language features in association with each of the images. In this configuration, the database generation unit generates a feature database that stores each image contained in multiple images, the camera position and orientation corresponding to each image, data related to the 3D point cloud, and, in addition, image language features associated with each image. Therefore, the prompt generation unit can use the image language features to select at least some of the multiple images stored in the feature database.

[0012] (5) In the above-described information processing apparatus, if the number of images stored in the feature database exceeds the amount that can be input to the large-scale language model, the prompt generation unit may, prior to generating the prompt, select at least a portion of the images stored in the feature database using at least one of the similarity of each image and the distance between the distributions of each image. With this configuration, the prompt generation unit selects at least a portion of the multiple images stored in the feature database, thus enabling it to handle cases where the number of images in the feature database exceeds the amount that can be input into the large-scale language model. Furthermore, since the prompt generation unit selects images using at least one of the similarity of each image and the distribution distance between each image, it can suppress the waste of multiple similar images being input into the large-scale language model.

[0013] (6) In the above-described information processing apparatus, the acquisition unit may capture a plurality of different still images from a video taken by the user while moving the camera in the three-dimensional space, and the captured plurality of still images may be used as the plurality of images. In this configuration, the acquisition unit captures multiple distinct still images from a video recorded by the user while moving the camera in a 3D space, and combines these captured still images into multiple images. Therefore, the user only needs to record a video while moving the camera in any 3D space they wish to target, making user operation simple.

[0014] (7) In the above-described information processing device, the output unit may output the prompt to the large-scale language model of the AI, and output the execution result of the prompt obtained from the AI ​​to the user terminal. In this configuration, the output unit outputs prompts to the AI's large-scale language model and outputs the execution results of the prompts obtained from the AI ​​to the user terminal. Therefore, the user can obtain the desired result simply by sending a video of a 3D space and a question, thereby improving user convenience.

[0015] Furthermore, the present invention can be realized in various forms, for example, as an information processing device that generates a feature database and generates prompts, an information processing device that generates only a feature database, an information processing device that generates only prompts using a feature database, a home assistant system that includes the functions of at least one of these information processing devices, a system that includes at least one of these information processing devices and other devices, a method for controlling these devices and systems, a computer program executed in these devices and systems, a server device for distributing the computer program, a non-temporary storage medium storing the computer program, and so on. [Brief explanation of the drawing]

[0016] [Figure 1] This is an explanatory diagram illustrating the configuration of an auxiliary system as one embodiment of the present invention. [Figure 2] This flowchart shows an example of the prompt generation process. [Figure 3] This is a data flow diagram of the prompt generation process. [Figure 4] This figure shows an example of multiple images. [Figure 5] This is a diagram illustrating the position and orientation of the camera. [Figure 6] This is a diagram illustrating the method for generating prompts. [Figure 7]It is an explanatory diagram illustrating the configuration of the auxiliary system according to the second embodiment.

Mode for Carrying Out the Invention

[0017] <First Embodiment> FIG. 1 is an explanatory diagram illustrating the configuration of an auxiliary system 1 as one embodiment of the present invention. The auxiliary system 1 is a system for using a generative AI for an arbitrary three-dimensional space. The generative AI is a technology specialized in generating new data and content, particularly among artificial intelligences (AI: Artificial Intelligence). For example, many existing generative AIs such as ChatGPT are constructed based on large language models (LLMs: Large Language Models). A large language model is a language model constructed by a large amount of teacher data and deep learning technology, and is one of the technologies responsible for natural language processing (NLP: Natural Language Processing) among AIs. The auxiliary system 1 may be constructed as a system that uses an AI constructed based on a large language model, such as an AI related to natural language processing, not limited to the generative AI. In the following embodiments, "AI" means an AI that accepts an input of a two-dimensional image and is constructed based on a large language model, including the generative AI.

[0018] The auxiliary system 1 includes an information processing device 100, a user terminal 200, and a server 300. In FIG. 1, the flow of data between the devices 100, 200, 300 is indicated by dashed arrows. The user terminal 200 shown above the information processing device 100 and the user terminal 200 shown below the information processing device 100 are the same device. The user terminal 200 is a computer used by the user of the auxiliary system 1 (information processing device 100). The server 300 is a computer that realizes an AI constructed based on the large language model 310. In FIG. 1, for the user terminal 200 and the server 300, the illustration of the CPU, storage unit, communication unit, etc. included in each device 200, 300 is omitted.

[0019] The information processing device 100 is a computer that generates and outputs a prompt for the user terminal 200 to use the AI of the server 300. A prompt is data used to ask questions to the generative AI. The prompt generated by the information processing device 100 in the present embodiment is data for achieving the following objectives a1 and a2. (a1) To make the large language model 310 of the server 300 understand an arbitrary three-dimensional space intended by the user. (a2) To obtain an execution result (answer) from the large language model 310 of the server 300 for a question or instruction from the user regarding the three-dimensional space understood according to objective a1. That is, for example, by sending a video of an arbitrary room to the information processing device 100, the user can cause the information processing device 100 to generate a prompt for asking a question such as "Where is the wristwatch?" or giving an instruction such as "I would like a proposal for a pattern change to give this room a modern impression" to the AI for the room.

[0020] The information processing device 100 includes a CPU 10, a storage unit 20, a communication unit 30, and a ROM / RAM 40, and each unit is interconnected by a bus not shown in the figure. The information processing device 100 can be configured by, for example, a personal computer. The information processing device 100 may be configured as a home assistant device for realizing an arbitrary function including, for example, the functions described later.

[0021] The storage unit 20 consists of a hard disk, flash memory, memory card, etc. The feature database 21 is stored in the storage unit 20 after a prompt generation process described later. The storage unit 20 also pre-stores a depth estimation model 22 and an image language model 23. The depth estimation model 22 is an existing algorithm for obtaining depth images from RGB images. For example, Depth-Anything can be used as the depth estimation model 22. The image language model 23 is an existing foundational model that simultaneously processes multimodal information of images and language and associates images with text. For example, CLIP (Contrastive Language-Image Pretraining) can be used as the image language model 23. Note that at least one of the depth estimation model 22 and the image language model 23 may not be stored in the storage unit 20 and may be implemented by an external server.

[0022] The communication unit 30 controls communication with other devices via a communication interface (not shown). Other devices may include a user terminal 200, a server 300, and other servers (not shown) that implement the depth estimation model 22 and the image language model 23.

[0023] The CPU 10 controls each part of the information processing device 100 by loading the computer program stored in the ROM 40 into the RAM 40 and executing it. The CPU 10 also functions as an acquisition unit 11, a database generation unit 12, a prompt generation unit 13, and an output unit 14. The acquisition unit 11 acquires video IN1 and question text IN2 from the user terminal 200, and acquires multiple images from video IN1. The database generation unit 12 generates a feature database 21 using the multiple images. The prompt generation unit 13 generates prompts to be given to the AI's large-scale language model 310 using the feature database 21. The output unit 14 outputs the generated prompts. In this embodiment, the output destination of the prompts is the user terminal 200.

[0024] Figure 2 is a flowchart of an example of the prompt generation process. Figure 3 is a data flow diagram of the prompt generation process. The step numbers in Figure 2 and Figure 3 correspond to each other. In Figure 3, the data generated by each step of the prompt generation process is represented by dot hatching. The prompt generation process is executed by the information processing device 100. The prompt generation process may be executed by predetermined triggers. For example, the prompt generation process may be executed when a predetermined application installed on the information processing device 100 is started, or when a user inputs an instruction to the information processing device 100. The instruction may be a text command or a voice command.

[0025] Figure 4 shows an example of multiple images. In step S10, the acquisition unit 11 acquires multiple images, for example, by following the steps b1 and b2 below. (b1) The acquisition unit 11 acquires video IN1 from the user terminal 200. In step b1, the acquisition unit 11 may display guidance on how to shoot video IN1 to the user terminal 200. The guidance may include, for example, "Activate the camera and slowly move the camera within the target space to shoot a video of the space. After shooting the video, press the send button." As a result, the user uses the camera built into the user terminal 200 to shoot a video of the 3D space that is the subject of the question while moving the camera. The user can then send the shot video as video IN1 to the information processing device 100.

[0026] (b2) The acquisition unit 11 captures multiple still images that are different from each other from the acquired video IN1 and acquires the captured multiple still images as "multiple images". In Figure 3 and the following description, the images acquired by the acquisition unit 11 in procedure b2 are also referred to as "RGB image D1". If the target space is a single room, for example, it is preferable for the acquisition unit 11 to capture still images of the room viewed from various different positions and orientations, as shown in Figures 4(A) and (B). The target space can be arbitrarily determined by the user. Multiple spaces may be, for example, a house containing multiple rooms and corridors. The number of still images captured by the acquisition unit 11 (i.e., the number of RGB image D1 in Figure 3) is, for example, about 1000 to 2000. The number of still images captured by the acquisition unit 11 (i.e., the number of RGB image D1 in Figure 3) may change in proportion to the duration of the video IN1. In Figure 4, the first RGB image D1 is represented as D1(1), and the second RGB image D1 is represented as D1(2).

[0027] In step S10, the acquisition unit 11 further acquires question text IN2 from the user terminal 200. Question text IN2 is the content of a question from the user. Question text IN2 may be text or audio. Question text IN2 can be a question such as "Where is my watch?" or an instruction such as "I want to make this room a little more modern." Question text IN2 may also be audio information contained in video IN1. In this case, the user can, for example, say "Where is my watch?" while recording a video. If question text IN2 is audio information contained in video IN1, in step S10, the acquisition unit 11 can extract the audio information from the video and acquire the question text.

[0028] In step S20, the database generation unit 12 estimates a depth image D2 for each image in the RGB image D1 using the depth estimation model 22. The depth image D2 is an image that represents the distance from the camera to each pixel in the RGB image D1. As a result of step S20, as shown in Figure 3, a depth image D2 is obtained for each image in the RGB image D1.

[0029] In step S22, the database generation unit 12 estimates the camera position and orientation at the time each image was captured for each image included in the RGB image D1. Specifically, the database generation unit 12 can estimate the camera position and orientation using Visual SLAM based on the RGB image D1 (multiple images) and the depth image D2. The database generation unit 12 may also use Structure from Motion instead of Visual SLAM. As a result of step S22, as shown in Figure 3, the camera position and orientation D3 is obtained for each image included in the RGB image D1.

[0030] Figure 5 is a diagram illustrating the camera's position and orientation. In this embodiment, "camera position and orientation" is a concept that includes both the camera's position and its orientation. The camera's position can be expressed, for example, by a combination of x, y, and z values ​​representing its relative position from the origin O, where SP is the center of the object's three-dimensional space. Specifically, the camera's position at point P1 can be expressed as x={x1}[m], y={y2}[m], z={z2}[m], where {x1}, {y1}, and {z1} are specific values. The camera's orientation can be expressed, for example, by a combination of yaw, pitch, and roll values ​​representing the camera's orientation angle. Specifically, the camera's orientation at point P1 can be expressed as yaw={yaw1}°, pitch={pitch1}°, and roll={roll1}°, where {yaw1}, {pitch1}, and {roll1} are specific values.

[0031] In step S24, the database generation unit 12 calculates a 3D point cloud of objects captured in the RGB image D1 (multiple images), in other words, a 3D point cloud of objects existing in 3D space. An object refers to a physical object, including living organisms. Specifically, the database generation unit 12 calculates a 3D point cloud representing the objects captured in the RGB image D1 by back-projecting the depth based on the depth image D2 and the camera position and orientation D3. The 3D point cloud can be calculated geometrically if the camera parameters are known. Here, camera parameters refer to a matrix for converting camera coordinates to image coordinates. As a result of step S24, a 3D point cloud of objects captured in the RGB image D1 (multiple images) is obtained, as shown in Figure 3. This 3D point cloud is also called "point cloud information D4".

[0032] In step S26, the database generation unit 12 compresses the point cloud information D4 by approximating it with a Gaussian distribution. The compressed data is also called "distribution information D5". As a result of step S26, distribution information D5 corresponding to the 3D point cloud is obtained, as shown in Figure 3.

[0033] In step S28, the database generation unit 12 extracts image language features from each image in the RGB image D1 using the image language model 23. Image language features are representations obtained when a common representation space (feature space) is learned between images and text, and are used to handle visual information and text information in a unified manner. As a result of step S28, as shown in Figure 3, image language features D6 are obtained for each image in the RGB image D1.

[0034] In step S30, the database generation unit 12 generates a feature database 21 by associating and storing the RGB image D1, camera position and orientation D3, distribution information D5, and image language features D6. At this time, the database generation unit 12 may perform feature reduction. Feature reduction means reducing the total number of RGB images D1 by discarding images with a narrower shooting range among similar RGB images D1 (multiple images) using the similarity of the image language features D6 or the distance between distributions. When performing feature reduction, the database generation unit 12 may use the variance of a Gaussian distribution (distribution information D5) to determine the shooting range.

[0035] Figure 6 is a diagram illustrating the prompt generation method. In step S32, the prompt generation unit 13 generates a prompt, for example, by following the steps c1 to c4 below. (c1) The prompt generation unit 13 generates a premise statement P1. The premise statement P1 is data that allows the large-scale language model 310 to understand the explanation that follows the premise statement P1. The premise statement P1 can be something like, "Now, we will present some images of a certain room taken from a specific camera position and orientation." The premise statement P1 can use the language used in the question statement IN2, or a language that the user has set in advance for the information processing device 100 (for example, Japanese, English, etc.).

[0036] (c2) The prompt generation unit 13 generates spatial information data P2. Spatial information data P2 is data that allows the large-scale language model 310 to understand the three-dimensional space that is the subject of the question. Spatial information data P2 includes N sets (N is a natural number) of image datasets. An "image dataset" refers to a pair of a key image and the position and orientation of the camera from which the key image was acquired. When generating spatial information data P2, the prompt generation unit 13 determines whether the number of RGB images D1 stored in the feature database 21 exceeds the amount that can be input into the large-scale language model 310. If the number of RGB images D1 in the feature database 21 is less than or equal to the amount that can be input into the large-scale language model 310, the prompt generation unit 13 uses all of the RGB images D1 as key images. On the other hand, if the number of RGB images D1 in the feature database 21 exceeds the amount that can be input into the large-scale language model 310, the prompt generation unit 13 selects at least some of the images from the RGB images D1 in the feature database 21 and uses the selected images as key images. The prompt generation unit 13 can select images using the similarity of image language features D6 or inter-distribution distance, similar to step S30 in Figure 2. In this case, it is preferable for the prompt generation unit 13 to select the maximum number of images that can be input into the large-scale language model 310. Finally, the prompt generation unit 13 sets a key image and the camera position and orientation D3 corresponding to that key image as one image dataset, and generates an image dataset for all key images. As a result, spatial information data P2 containing N sets of image datasets can be generated.

[0037] (c3) The prompt generation unit 13 generates a user question P3. User question P3 is data that allows the large-scale language model 310 to understand the content of the user's question. User question P3 is content that conforms to the question statement IN2 sent from the user terminal 200, for example, a question such as "Where is the watch?" or an instruction such as "Please suggest some ideas for redecorating this room to give it a modern feel." User question P3 can use the language used in the question statement IN2, or a language that the user has set in advance for the information processing device 100 (for example, Japanese, English, etc.). The prompt generation unit 13 may also add a specification of the answer format to the large-scale language model 310 before the user's question or instruction. Specifying the answer format is a note such as, for example, "The user does not have information about the image given to you. Therefore, please answer without directly referring to the image." By specifying the answer format, it is possible to suppress the large-scale language model 310 from producing an execution result that directly refers to an image, such as "It's the leftmost part of the 12th image."

[0038] (c4) The prompt generation unit 13 generates a prompt PP which is a set of the premise statement P1, spatial information data P2, and user question P3 generated in steps c1 to c3.

[0039] In step S50, the output unit 14 outputs the generated prompt PP to the user terminal 200, as shown in (1) in Figure 1. As a result, the user terminal 200 sends the prompt PP to the server 300, as shown in (2) in Figure 1, and can obtain the execution result from the server 300. The execution result of the prompt PP is the answer from the generating AI to the user question P3.

[0040] As described above, according to the information processing device 100 of the first embodiment, the AI's large-scale language model 310 can understand any 3D space SP that is the target of the user question P4 by at least a portion of the multiple images (RGB image D1: 2D image) stored in the feature database 21 and the camera position and orientation D3 associated with each of those images (Figure 6: spatial information data P2). Therefore, according to the information processing device 100, it is possible to make the trained model 310 for 2D images understand the 3D space SP. In other words, it is possible to process input data with the model 310 that is normally designed for 2D images, eliminating the need for 3D-specific training. Furthermore, the data size of the input data can be reduced compared to when 3D-specific training is performed, and scalability can be improved as the data size is reduced.

[0041] Furthermore, according to the information processing device 100 of the first embodiment, the database generation unit 12 approximates the 3D point cloud (point cloud information D4) with a Gaussian distribution (distribution information D5), and uses the obtained distribution information D5 as data related to the 3D point cloud. Since the distribution information D5 has a smaller data size compared to the 3D point cloud (point cloud information D4), the database generation unit 12 can reduce the capacity of the feature database 21.

[0042] Furthermore, according to the information processing device 100 of the first embodiment, the database generation unit 12 uses a plurality of images (RGB images D1) and an estimated depth image D2 to estimate the camera position and orientation D3 for each image included in the plurality of images. Therefore, even if the plurality of images acquired by the acquisition unit 11 are not RGBD images, the database generation unit 12 can estimate the camera position and orientation D3 at the time the images were taken. An RGBD image is an image in which RGB pixels and Depth are pre-associated.

[0043] Furthermore, according to the information processing device 100 of the first embodiment, the database generation unit 12 generates a feature database 21 that stores each image included in the plurality of images (RGB images D1), the camera position and orientation D3 corresponding to each image, data related to the 3D point cloud (distribution information D5), and in addition, image language features D6 in association with each image.Therefore, the prompt generation unit 13 can use the image language features D6 of the feature database 21 to select at least a portion of the plurality of images (RGB images D1) stored in the feature database 21.

[0044] Furthermore, according to the information processing device 100 of the first embodiment, the prompt generation unit 13 selects at least a portion of the multiple images (RGB images D1) stored in the feature database 21 (selects a key image). Therefore, it can handle cases where the number of images in the feature database 21 exceeds the amount that can be input into the large-scale language model 310. In addition, the prompt generation unit 13 selects an image (selects a key image) using at least one of the similarity of each image and the distribution distance between each image. Therefore, it can suppress the waste of multiple similar images being input into the large-scale language model 310.

[0045] Furthermore, according to the information processing device 100 of the first embodiment, the acquisition unit 11 captures multiple distinct still images from the video IN1 shot by the user of the information processing device 100 while moving the camera in a three-dimensional space, and combines the captured still images into multiple images (RGB images D1). Therefore, the user only needs to shoot a video while moving the camera in any three-dimensional space they wish to target, thus simplifying user operation.

[0046] <Second Embodiment> Figure 7 is an explanatory diagram illustrating the configuration of the auxiliary system 1A in the second embodiment. In the second embodiment, the effort required for the user terminal 200 to send a prompt PP to the server 300 can be reduced. The auxiliary system 1A is equipped with an information processing device 100A in place of the information processing device 100 in the configuration described in the first embodiment.

[0047] The CPU 10 of the information processing device 100A is equipped with an output unit 14A instead of an output unit 14. In step S50 of the process shown in Figure 2, the output unit 14 sends the generated prompt PP to the server 300 as shown in Figure 7(1), and obtains the execution result of the prompt PP from the server 300 as shown in Figure 7(2). Subsequently, the output unit 14A sends the execution result of the prompt PP to the user terminal 200 as shown in Figure 7(3). As a result, if the user terminal 200 sends the video IN1 and the question IN2 to the information processing device 100A, it can directly obtain an answer to the question IN2 from the information processing device 100A.

[0048] Thus, the output unit 14A of the information processing device 100A may arbitrarily change the output destination of the prompt PP. The information processing device 100A of the second embodiment can also achieve the same effects as the first embodiment described above. Furthermore, according to the information processing device 100A of the second embodiment, if the user terminal 200 transmits the video IN1 and the question IN2 to the information processing device 100A, it can directly obtain an answer to the question IN2 from the information processing device 100A. For this reason, the information processing device 100A can be configured as a home assistant device. As a result, user convenience can be further improved.

[0049] <Modified form of this embodiment> The present invention is not limited to the embodiments described above, and can be implemented in various forms without departing from its spirit. For example, a part of the configuration implemented by hardware may be replaced with software, or conversely, a part of the configuration implemented by software may be replaced with hardware. In addition, the following modifications are also possible, for example.

[0050] [Example 1] The above embodiment shows an example of the configuration of auxiliary systems 1, 1A and information processing devices 100, 100A. However, the configuration of auxiliary systems 1, 1A and information processing devices 100, 100A can be modified in various ways. For example, auxiliary system 1 may include multiple user terminals 200. For example, auxiliary system 1 may include multiple servers 300 that implement different AIs. For example, the depth estimation model 22 of information processing device 100 may be provided by an external server. For example, the image language model 23 of information processing device 100 may be provided by an external server. For example, the functions of the acquisition unit 11, the database generation unit 12, the prompt generation unit 13, and the output unit 14 may be realized by the cooperation of multiple information processing devices.

[0051] For example, the information processing devices 100 and 100A may not have at least some of the functional units described above. For example, the information processing device 100 may not have the database generation unit 12. In this case, the prompt generation unit 13 of the information processing device 100 can generate a prompt PP using a feature database 21 provided from an external server or user terminal 200. For example, the information processing device 100 may not have the prompt generation unit 13. In this case, the information processing device 100 may send the generated feature database 21 to the user terminal 200.

[0052] [Differentiation 2] The above embodiments show an example of the processing content in the information processing devices 100 and 100A (Figures 2 and 3). However, this processing content can be modified in various ways. For example, the execution order of each step may be changed, some steps may be omitted, or other steps not described may be executed.

[0053] For example, if video IN1 is a video acquired by an RGBD camera, the database generation unit 12 may omit step S20. This is because RGBD images contain depth information.

[0054] For example, the database generation unit 12 may omit step S26. In this case, the database generation unit 12 can generate the feature database 21 using point cloud information D4 instead of distribution information D5.

[0055] For example, the database generation unit 12 may omit step S28. In this case, the database generation unit 12 only needs to generate a feature database 21 that does not include the image language feature D6.

[0056] For example, the information processing device 100 may use a different format for representing the camera's orientation than the combination of yaw, pitch, and roll values ​​that represent the camera's orientation angle. This other format could be, for instance, a more discretized format than the combination of yaw, pitch, and roll values. A discretized format would allow the use of words that express the camera's direction, such as "The camera is facing north."

[0057] [Difference 3] The auxiliary systems 1,1A and information processing devices 100,100A of the first and second embodiments described above, and the auxiliary systems 1,1A and information processing devices 100,100A of the modified examples 1 and 2 described above, may be combined as appropriate.

[0058] The embodiments of this specification have been described above based on the embodiments and modifications described above. The embodiments described above are for the purpose of facilitating understanding of this specification and do not limit it. This specification may be modified and improved without departing from its spirit and the scope of the claims, and equivalents thereof are included in this specification. Furthermore, any technical features that are not described as essential in this specification may be deleted as appropriate.

[0059] The present invention can also be realized in the following forms. [Application Example 1] An information processing device, An acquisition unit that acquires multiple 2D images representing an arbitrary 3D space, each being a different image captured using a camera, and a question text from the user. A database generation unit that generates a feature database using the aforementioned multiple images, A prompt generation unit that generates prompts to be given to a large-scale language model of an AI using the aforementioned feature database, An output unit that outputs the generated prompt, Equipped with, The aforementioned database generation unit, The position and orientation of the camera at the time each image included in the plurality of images was taken is estimated. The 3D point cloud of the objects shown in the aforementioned multiple images is calculated, A feature database is generated by associating each image included in the plurality of images, the position and orientation of the camera corresponding to each image, and the data relating to the 3D point cloud. The prompt generation unit comprises the prompt, The premise for the aforementioned large-scale language model, At least a portion of the images stored in the aforementioned feature database and the position and orientation of the camera corresponding to said image, An information processing device that generates a prompt including a user question for the large-scale language model, the user question being generated from the question statement. [Application Example 2] The information processing device described in Application Example 1, The database generation unit further approximates the calculated three-dimensional point cloud with a Gaussian distribution and uses the obtained distribution information as data relating to the three-dimensional point cloud, as an information processing device. [Application Example 3] An information processing device as described in Application Example 1 or Application Example 2, The database generation unit further, For each of the multiple images mentioned above, a depth image is estimated using a pre-prepared depth estimation model. An information processing device that estimates the position and orientation of the camera for each image included in the plurality of images, using the plurality of images and the estimated depth image. [Application Example 4] An information processing device described in any one of Application Examples 1 to 3, The database generation unit further, For each image included in the aforementioned set of images, image language features are extracted using a pre-prepared image language model. An information processing device that generates a feature database which stores each of the multiple images, the position and orientation of the camera corresponding to each image, the data relating to the 3D point cloud, and further associates the image language features with the feature database. [Application Example 5] An information processing device described in any one of Application Examples 1 to 4, The prompt generation unit, If the number of images stored in the feature database exceeds the amount that can be input into the large-scale language model, Prior to generating the prompt, an information processing device selects at least a portion of the plurality of images stored in the feature database using at least one of the similarity of each image and the distance between the distributions of each image. [Application Example 6] An information processing device described in any one of Application Examples 1 to 5, The acquisition unit is an information processing device that captures a plurality of different still images from a video taken by a user while moving the camera in the three-dimensional space, and combines the captured plurality of still images into a plurality of images. [Application Example 7] An information processing device according to any one of Application Examples 1 to 6, The output unit outputs the prompt to the large-scale language model of the AI, and outputs the execution result of the prompt obtained from the AI ​​to the user terminal, which is an information processing device. [Application Example 8] A prompt generation method, wherein an information processing device comprising a memory unit, A process of acquiring multiple 2D images representing an arbitrary 3D space, which are all different images captured using a camera, and a question text provided by the user. A database generation step of generating a feature database using the aforementioned multiple images, A prompt generation step that generates prompts to be given to a large-scale language model of AI using the aforementioned feature database, An output step which outputs the generated prompt, This is what does the following: In the aforementioned database generation process, The position and orientation of the camera at the time each image included in the plurality of images was taken is estimated. The 3D point cloud of the objects shown in the aforementioned multiple images is calculated, A feature database is generated by associating each image included in the plurality of images, the position and orientation of the camera corresponding to each image, and the data relating to the 3D point cloud. In the prompt generation step, the prompt is, The premise for the aforementioned large-scale language model, At least a portion of the images stored in the aforementioned feature database and the position and orientation of the camera corresponding to said image, A prompt generation method for generating a prompt that includes a user question for the large-scale language model, the user question being generated from the question statement. [Application Example 9] A computer program for an information processing device equipped with a memory unit, A function to acquire multiple 2D images representing an arbitrary 3D space, each being a different image captured using a camera, and a question text provided by the user. A database generation function that generates a feature database using the aforementioned multiple images, A prompt generation function that generates prompts to be given to a large-scale language model of AI using the aforementioned feature database, An output function that outputs the generated prompt, Make it run, The aforementioned database generation function is The position and orientation of the camera at the time each image included in the plurality of images was taken is estimated. The 3D point cloud of the objects shown in the aforementioned multiple images is calculated, A feature database is generated by associating each image included in the plurality of images, the position and orientation of the camera corresponding to each image, and the data relating to the 3D point cloud. The prompt generation function is the prompt, The premise for the aforementioned large-scale language model, At least a portion of the images stored in the aforementioned feature database and the position and orientation of the camera corresponding to said image, A computer program that generates a prompt, which includes a user question for the large-scale language model, the user question generated from the question statement. [Explanation of Symbols]

[0060] 1.1A... Auxiliary system 10…CPU 11…Acquisition part 12…Database generation unit 13... Prompt generation unit 14,14A…Output section 20...Storage section 21…Feature Database 22…Depth Estimation Model 23…Image language model 30... Communications Department 40…ROM / RAM 100, 100A… Information Processing Equipment 200... User terminals 300... Server 310...Large-scale language models

Claims

1. An information processing device, An acquisition unit that acquires multiple 2D images representing an arbitrary 3D space, each being a different image captured using a camera, and a question text from the user. A database generation unit that generates a feature database using the aforementioned multiple images, A prompt generation unit generates prompts to be given to a large-scale language model of AI using the aforementioned feature database, An output unit that outputs the generated prompt, Equipped with, The aforementioned database generation unit, The position and orientation of the camera at the time each image included in the plurality of images was taken is estimated. The three-dimensional point cloud of the objects shown in the aforementioned multiple images is calculated, A feature database is generated by associating each image included in the plurality of images, the position and orientation of the camera corresponding to each image, and the data relating to the three-dimensional point cloud. The prompt generation unit comprises the prompt, The premise for the aforementioned large-scale language model, At least a portion of the images stored in the aforementioned feature database and the position and orientation of the camera corresponding to said image, An information processing device that generates a prompt including a user question for the large-scale language model, the user question being generated from the question statement.

2. An information processing apparatus according to claim 1, The database generation unit further approximates the calculated three-dimensional point cloud with a Gaussian distribution and uses the obtained distribution information as data relating to the three-dimensional point cloud, as an information processing device.

3. An information processing apparatus according to claim 2, The database generation unit further, For each of the multiple images mentioned above, a depth image is estimated using a pre-prepared depth estimation model. An information processing device that estimates the position and orientation of the camera for each image included in the plurality of images, using the plurality of images and the estimated depth image.

4. The information processing apparatus according to claim 3, The database generation unit further, For each image included in the aforementioned set of images, image language features are extracted using a pre-prepared image language model. An information processing device that generates a feature database which stores each of the multiple images, the position and orientation of the camera corresponding to each image, the data relating to the three-dimensional point cloud, and further associates the image language features with each image.

5. An information processing device according to any one of claims 1 to 4, The prompt generation unit, If the number of images stored in the feature database exceeds the amount that can be input into the large-scale language model, Prior to generating the prompt, an information processing device selects at least a portion of the plurality of images stored in the feature database using at least one of the similarity of each image and the distance between the distributions of each image.

6. An information processing device according to any one of claims 1 to 4, The acquisition unit is an information processing device that captures a plurality of different still images from a video taken by a user while moving the camera in the three-dimensional space, and combines the captured plurality of still images into a plurality of images.

7. An information processing device according to any one of claims 1 to 4, The output unit outputs the prompt to the large-scale language model of the AI, and outputs the execution result of the prompt obtained from the AI ​​to the user terminal, which is an information processing device.

8. A prompt generation method, wherein an information processing device comprising a memory unit, An acquisition process that acquires multiple 2D images representing an arbitrary 3D space, each being a different image captured using a camera, and a question text provided by the user. A database generation step of generating a feature database using the aforementioned multiple images, A prompt generation step that generates prompts to be given to a large-scale language model of AI using the aforementioned feature database, An output step which outputs the generated prompt, This is what does the following: In the aforementioned database generation process, The position and orientation of the camera at the time each image included in the plurality of images was taken is estimated. The three-dimensional point cloud of the objects shown in the aforementioned multiple images is calculated, A feature database is generated by associating each image included in the plurality of images, the position and orientation of the camera corresponding to each image, and the data relating to the three-dimensional point cloud. In the prompt generation step, the prompt is, The premise for the aforementioned large-scale language model, At least a portion of the images stored in the aforementioned feature database and the position and orientation of the camera corresponding to said image, A prompt generation method for generating a prompt that includes a user question for the large-scale language model, the user question being generated from the question statement.

9. A computer program for an information processing device equipped with a memory unit, A function to acquire multiple 2D images representing an arbitrary 3D space, each being a different image captured using a camera, and a question text provided by the user. A database generation function that generates a feature database using the aforementioned multiple images, A prompt generation function that generates prompts to be given to a large-scale language model of AI using the aforementioned feature database, An output function that outputs the generated prompt, Make it run, The aforementioned database generation function is The position and orientation of the camera at the time each image included in the plurality of images was taken is estimated. The three-dimensional point cloud of the objects shown in the aforementioned multiple images is calculated, A feature database is generated by associating each image included in the plurality of images, the position and orientation of the camera corresponding to each image, and the data relating to the three-dimensional point cloud. The prompt generation function is the prompt, The premise for the aforementioned large-scale language model, At least a portion of the images stored in the aforementioned feature database and the position and orientation of the camera corresponding to said image, A computer program that generates a prompt, which includes a user question for the large-scale language model, the user question generated from the question statement.

Images