Information processing system, information processing method, information processing device, determination method, and determination program

Abstract

This information processing system comprises: an information processing device provided with a first multimodal large-scale language model; and an operating body provided with a second multimodal large-scale language model. The information processing device comprises: a prediction unit that predicts, on the basis of instruction information indicating a human instruction to the operating body, a predicted time required for the operating body to complete an instruction task corresponding to the human instruction; and a determination unit that, on the basis of a comparison between the predicted time and a threshold time until the operating body consumes a storage capacity usable for executing tasks, determines to cause the first multimodal large-scale language model to generate control information for the operating body for executing the instruction task if the predicted time is longer than the threshold time, and determines to cause the second multimodal large-scale language model to generate the control information if the predicted time is equal to or shorter than the threshold time. The operating body includes an operation control unit that controls the operation of the operating body to execute the instruction task on the basis of the control information.

Description

Information processing system, information processing method, information processing device, decision method, and decision program

[0001] The present invention relates to an information processing system, an information processing method, an information processing device, a decision method, and a decision program.

[0002] In recent years, multimodal large-scale language models (VLAs) have become known as machine learning models that can handle multiple types of data (modalities) such as images, audio, and video, in addition to text. For example, the VLA (Vision-Language-Action) model is known as an example of a multimodal large-scale language model. Furthermore, technologies that use multimodal large-scale language models to control the behavior (also called actions) and autonomous driving of robots are known. For example, a technology is known in which images of the environment surrounding the robot and text indicating human instructions to the robot are input into the VLA model to generate robot actions that correspond to the environment surrounding the robot and the human instructions.

[0003] Anthony Brohan, Noah Brown, Justice Carbajal et al., "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control", July 2023, [Retrieved September 17, 2020], Internet <URL: https: / / robotics-transformer2.github.io / assets / rt2.pdf>

[0004] The information processing system according to the present application is an information processing system including an information processing device having a first multimodal large language model and an operating body having a second multimodal large language model. The information processing device includes a prediction unit that predicts a predicted time required for the operating body to complete an instruction task according to a person's instruction based on instruction information indicating the person's instruction to the operating body, and a determination unit that determines to cause the first multimodal large language model to generate control information for the operating body to execute the instruction task when the predicted time is greater than a threshold time until the storage capacity available for the operating body to execute a task is consumed, based on a comparison between the predicted time and the threshold time, and determines to cause the second multimodal large language model to generate the control information when the predicted time is less than or equal to the threshold time. The operating body includes an operation control unit that controls the operation of the operating body to execute the instruction task based on the control information.

[0005] Figure 1 illustrates how the size of the KV cache in a multimodal large-scale language model that generates robot behavior increases rapidly in a short time. Figure 2 illustrates the relationship between the memory capacity available for the operating body to execute a task and the threshold time according to the embodiment. Figure 3 illustrates information processing related to the comparative technique. Figure 4 illustrates information processing related to the comparative technique. Figure 5 is a diagram showing an example configuration of an information processing system according to the embodiment. Figure 6 is a diagram showing an example configuration of an information processing device according to the embodiment. Figure 7 is a diagram showing an example configuration of an operating body according to the embodiment. Figure 8 is a diagram illustrating an example of information processing according to the embodiment. Figure 9 is a diagram illustrating an example of information processing according to the embodiment. Figure 10 is a diagram illustrating an example of information processing according to the embodiment. Figure 11 is a diagram illustrating an example of information processing according to the embodiment. Figure 12 is a diagram illustrating an example of information processing according to the embodiment. Figure 13 is a diagram illustrating an example of information processing according to the embodiment. Figure 14 is a diagram illustrating an example of information processing according to the embodiment. Figure 15 is a diagram illustrating an example of information processing according to the embodiment. Figure 16 is a diagram showing an example of hardware configuration.

[0006] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing system, information processing method, information processing device, decision method, and decision program relating to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing system, information processing method, information processing device, decision method, and decision program relating to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0007] (Embodiments) [1. Introduction] In recent years, a technology has been developed to control the behavior (also called actions) of robots using a multimodal large-scale language model. For example, a technology has been developed in which images of the environment surrounding the robot and text indicating instructions from a person to the robot are input into a multimodal large-scale language model to generate robot actions that correspond to the environment surrounding the robot and the instructions from the person (Non-Patent Literature 1). It should be noted that a robot acting in response to its surrounding environment and instructions from a person can be rephrased as a robot performing a task in response to instructions from a person.

[0008] Furthermore, large-scale language models generally use not only the current token but also all previously generated tokens when creating a new token. For example, when a large-scale language model creates its 100th token, it needs information from the 99 tokens previously created. Similarly, when it creates its 101st token, it needs information from the 100 tokens previously created. In this way, large-scale language models perform the same calculation multiple times to create a single token, making them computationally inefficient. Therefore, a key-value (KV) cache is used in the computation of large-scale language models. Specifically, the large-scale language model caches the intermediate output of the calculation when the same calculation is repeated. More specifically, the large-scale language model caches the values ​​of the Key vector and Value vector in the attention (KV cache). This eliminates the need for the large-scale language model to repeat the same calculation, speeding up the token generation process. On the other hand, since memory is consumed by accumulating KV cache during the computation of large-scale language models, securing the necessary memory capacity for the computation becomes a problem. In particular, in multimodal large-scale language models that generate robot behavior, the size of the KV cache increases rapidly in a short time, making it a problem how to secure the memory capacity required for computation.

[0009] Figure 1 illustrates how the size of the KV cache in a multimodal large-scale language model that generates robot behavior can increase rapidly in a short time. Figure 1 describes an example of the computation of a multimodal large-scale language model that generates robot behavior (hereinafter sometimes simply referred to as the "multimodal large-scale language model"). Figure 1 also describes the case where the multimodal large-scale language model is installed on a robot. For example, the multimodal large-scale language model takes images of the environment surrounding the robot and text indicating human instructions to the robot as input, and generates robot behavior in response to the environment surrounding the robot and the human instructions. For example, the multimodal large-scale language model generates control information that controls the robot's behavior. For example, the control information may include multiple tokens. For example, the control information may be a string relating to a program that controls the robot's behavior. For example, the control information may be a string corresponding to source code that controls the robot's behavior.

[0010] In the upper part of Figure 1, the multimodal large-scale language model generates a first KV cache based on 50 tokens, which are the sum of 1 token corresponding to control information generated in the time step before the current time step ("Previous Time Control" in the upper part of Figure 1), 32 tokens corresponding to images of the environment surrounding the robot ("Image Input" in the upper part of Figure 1), and 17 tokens corresponding to text indicating human instructions to the robot ("Average Character Input" in the upper part of Figure 1). The first KV cache occupies the robot's GPU memory. The robot's GPU memory is consumed by the size of the first KV cache. Here, the size of the KV cache is determined by multiplying the memory size per token of the KV cache by the number of tokens. For example, the size of the first KV cache is determined by multiplying the memory size per token of the KV cache by the number of tokens, which is 50. The multimodal large-scale language model also generates 1 token corresponding to the control information in the current time step based on the first KV cache. In this way, the multimodal large-scale language model uses the KV cache from the time step immediately preceding the current time step (the first KV cache) to generate tokens corresponding to the control information in the current time step.

[0011] In the lower part of Figure 1, the multimodal large-scale language model generates a second KV cache based on 50 tokens, which are the sum of 1 token corresponding to the control information generated in the current time step ("Previous Time Control" in the lower part of Figure 1), 32 tokens corresponding to the image input to the multimodal large-scale language model ("Image Input" in the lower part of Figure 1), and 17 tokens corresponding to the text indicating human instructions to the robot ("Average Character Input" in the lower part of Figure 1). The second KV cache occupies the robot's GPU memory. The robot's GPU memory is consumed by the size of the second KV cache. The size of the second KV cache is determined by multiplying the memory size per token of the KV cache by the number of tokens, which is 50. The multimodal large-scale language model also generates 1 token corresponding to the control information in the next time step, one step after the current time step ("Control" in the lower part of Figure 1), based on the second KV cache. In this way, the multimodal large-scale language model uses the KV cache (second KV cache) in the current timestep to generate tokens corresponding to the control information in the next timestep, which is one step after the current timestep.

[0012] For example, the multimodal large-scale language model that performs the calculations in Figure 1 may be Llama3-8B. Also, the amount of memory available for the KV cache from the robot's GPU memory is 2GB. Furthermore, the time step period (computation period) of the calculation of the multimodal large-scale language model that performs the calculations in Figure 1 is 100ms (10Hz). The rate at which the KV cache size increases is proportional to a value determined by the characteristics of the multimodal large-scale language model (number of layers, etc.), the memory size per token in the KV cache, the number of input tokens (50 tokens in Figure 1), and the computation period of the multimodal large-scale language model (10Hz in Figure 1). For example, in Figure 1, the KV cache size increases at a rate of 0.125 (GB / s). In this case, the time until the available memory size for the KV cache (2GB) is consumed is only 16 seconds. Thus, the KV cache size in the multimodal large-scale language model that generates the robot's actions increases rapidly in a short time.

[0013] Furthermore, in Figure 1, the multimodal large-scale language model consumes the memory size available for the KV cache (2GB) in 16 seconds. Here, the memory size available for the KV cache can be rephrased as the size of memory that the robot can use to perform computational processing of the multimodal large-scale language model. In other words, in Figure 1, the time the robot can perform computational processing of the multimodal large-scale language model is 16 seconds. Moreover, the time the robot can perform computational processing of the multimodal large-scale language model can be rephrased as the time the robot can continuously generate its own control information. In other words, in Figure 1, the time the robot can continuously generate its own control information is 16 seconds. Moreover, the time the robot can continuously generate its own control information can be rephrased as the time the robot can continuously store information related to its own task. In other words, in Figure 1, the time the robot can continuously store information related to its own task is 16 seconds. Moreover, the time the robot can continuously store information related to its own task can be rephrased as the length of time the multimodal large-scale language model installed on the robot can consider the input information. For example, the input information consists of images of the environment surrounding the robot and text indicating human instructions to the robot. In other words, in Figure 1, the length of time that the multimodal large-scale language model can consider input information is 16 seconds. Furthermore, the time that a robot can sustainably remember information about its task can be rephrased as the time that the robot has available to perform the task. That is, in Figure 1, the time that the robot has available to perform the task is 16 seconds. However, the time that the robot actually takes to perform the task may exceed the time that the robot has available to perform the task. Therefore, in a multimodal large-scale language model that generates robot behavior, the problem is how to secure the memory capacity necessary for computation.

[0014] Figure 2 illustrates the relationship between the memory capacity available for the robot to perform a task and the threshold time according to the embodiment. In the graph shown in Figure 2, the horizontal axis represents elapsed time, and the vertical axis represents the memory capacity of the robot's VRAM (Video Random Access Memory). In Figure 2, the size of the multimodal large-scale language model (model size) occupies M1 of the robot's maximum memory capacity M3. That is, the memory size available for the KV cache (maximum KV cache size) explained in Figure 1 corresponds to the remaining amount (M3-M1) obtained by subtracting the model size M1 from the robot's maximum memory capacity M3. Here, the memory size available for the KV cache can be rephrased as the memory capacity available for a robot equipped with a multimodal large-scale language model to perform a task.

[0015] Furthermore, in Figure 2, when the first token is input to the multimodal large-scale language model at time t1, the first KV cache is generated during time D, and the memory size available for the KV cache increases rapidly from M1 to M2. From time t2 onward, after time D has elapsed from time t1, the multimodal large-scale language model generates a KV cache at each time step Δt, and the memory size available for the KV cache is consumed by the size of the KV cache generated at each time step ΔM. The time steps described in Figure 1 correspond to Δt shown in Figure 2. Also, the size of the KV cache at each time step described in Figure 1 (for example, the first KV cache or the second KV cache) corresponds to ΔM shown in Figure 2. Furthermore, at time t3, after time L has elapsed from time t1, the maximum KV cache size is completely consumed. Hereafter, the time L until the maximum KV cache size is consumed may be referred to as the threshold time. Thus, the threshold time L is the time it takes for a robot equipped with a multimodal large-scale language model to consume the memory size available for the KV cache. Here, as mentioned above, the memory size available for the KV cache can be rephrased as the memory capacity available for a robot equipped with a multimodal large-scale language model to perform a task. In other words, the threshold time L is the time it takes for a robot equipped with a multimodal large-scale language model to consume the memory capacity available for performing a task.

[0016] In other words, threshold time L is the time during which a robot equipped with a multimodal large-scale language model can perform computational processing of the multimodal large-scale language model. In other words, threshold time L is the time during which a robot equipped with a multimodal large-scale language model can continuously generate its own control information. In other words, threshold time L is the time during which a robot equipped with a multimodal large-scale language model can continuously store information related to its own task. In other words, threshold time L is the length of time during which a multimodal large-scale language model installed on a robot can consider input information. In other words, threshold time L is the time available for a robot equipped with a multimodal large-scale language model to perform a task in accordance with the input information of the multimodal large-scale language model.

[0017] Figures 3 and 4 are diagrams illustrating information processing related to comparative technologies. The robot 4 and MEC server 3 shown in Figures 3 and 4 are connected to each other wirelessly so that they can communicate with one another. The robot 4 shown in Figures 3 and 4 is equipped with a multimodal large-scale language model (hereinafter sometimes referred to as "robot-mounted LLM").

[0018] In Figures 3 and 4, first, the MEC server 3 receives text indicating a human instruction to the robot 4, such as "Take this bag to the kitchen." When the MEC server 3 receives text indicating a human instruction, it sends the text indicating the human instruction to the robot 4. The robot 4 receives the text indicating the human instruction from the MEC server 3. When the robot 4 receives the text indicating a human instruction, it also acquires images of its surrounding environment. The robot 4 also attempts to perform a task in accordance with the human instruction (for example, the task of taking the bag to the kitchen).

[0019] Figure 3 shows an image acquired by robot 4, which depicts a situation where a vacuum cleaner has fallen over in the direction robot 4 is moving. When robot 4 acquires an image, it inputs the acquired image and text indicating the person's instructions into the robot-mounted LLM to generate actions for robot 4. For example, robot 4 uses the robot-mounted LLM to generate control information for robot 4 to perform actions in response to the environment around robot 4 and the person's instructions. For example, robot 4 generates control information that causes it to act as if it were thinking, "There's a vacuum cleaner lying over in the direction robot 4 is moving, so I could jump over it, but since I'm taking a bag to the kitchen, there might be eggs in the bag, so I should take a different route."

[0020] Figure 4 illustrates the case where the time required for robot 4 to complete the task is longer than the time (threshold time) required for robot 4 to consume the memory capacity available for executing the task. In other words, in Figure 4, because the time required for robot 4 to complete the task is longer than the threshold time, robot 4 cannot continuously store information about its task until the task is completed. In Figure 4, robot 4 consumes the memory capacity available for executing the task before it completes the task. Furthermore, because robot 4 has consumed the memory capacity available for executing the task, it has forgotten the information about its task. Furthermore, because robot 4 has forgotten the information about its task, it is unable to properly generate its own control information. In other words, because robot 4 has consumed the memory capacity available for executing the task, it is unable to perform the calculation processing of the robot-mounted LLM. In other words, robot 4 is unable to continuously generate its own control information. That is, because robot 4 has consumed the memory capacity available for executing the task, it is unable to properly execute the task. For example, robot 4 performs actions as if it were thinking, "Why did I choose this route? The lower route is shorter, so let's go back."

[0021] As mentioned above, the time it takes for a robot to complete a task may exceed the time it takes for the robot to consume the memory capacity available for performing the task (threshold time). In other words, the robot may not be able to continuously store information about its task until it is completed. Therefore, the robot may not be able to perform the task properly. Furthermore, because the robot cannot perform the task properly, it may not be able to complete the task. For this reason, there is a need for technology that enables the robot to perform the task properly even when the time it takes to complete the task exceeds the threshold time.

[0022] In contrast, the information processing system according to this embodiment includes an information processing device equipped with a first multimodal large-scale language model and an operating body (e.g., a robot) equipped with a second multimodal large-scale language model. The information processing device predicts the estimated time required for the operating body to complete an instruction task in response to a human instruction, based on instruction information indicating a human instruction to the operating body. The information processing device also compares the estimated time with a threshold time until the operating body consumes the available memory capacity to perform the task. If the estimated time is greater than the threshold time, the information processing device decides to have the first multimodal large-scale language model generate control information for the operating body to perform the instruction task. If the estimated time is less than or equal to the threshold time, the information processing device decides to have the second multimodal large-scale language model generate control information. The operating body then controls its actions to perform the instruction task based on the control information.

[0023] Thus, in the information processing system according to the embodiment, if the operator can continuously store information about its own task until the task is completed, the system causes the operator's second multimodal large-scale language model to generate control information for the operator. This enables the information processing system according to the embodiment to enable the operator to execute the task appropriately. Furthermore, if the operator cannot continuously store information about its own task until the task is completed, the information processing system according to the embodiment causes the operator's control information to be generated by the first multimodal large-scale language model of an information processing device with a larger memory capacity than the operator. This enables the information processing system according to the embodiment to generate appropriate operation information even if the time required for the operator to complete the task exceeds a threshold time, thereby enabling the operator to execute the task appropriately.

[0024] [2. Configuration of the Information Processing System] Figure 5 is a diagram showing an example configuration of an information processing system according to the embodiment. As shown in Figure 5, the information processing system 1 according to the embodiment includes an information processing device 10 equipped with a first multimodal large-scale language model 121 and an operator 20 equipped with a second multimodal large-scale language model 231. The information processing device 10 and the operator 20 are connected wirelessly via a mobile network. The information processing device 10 and the operator 20 are located within the mobile network. The information processing system 1 may include a plurality of information processing devices 10 and a plurality of operator 20. In Figure 5, the storage unit that stores information related to the first multimodal large-scale language model 121 is referred to as "multimodal LLM 121". Also in Figure 5, the storage unit that stores information related to the second multimodal large-scale language model 231 is referred to as "multimodal LLM 231".

[0025] The information processing device 10 predicts the estimated time required for the operator 20 to complete the instruction task in response to the human instruction, based on instruction information indicating the human instruction to the operator 20. Furthermore, based on a comparison of the predicted time with a threshold time until the operator 20 consumes the available memory capacity to execute the task, the information processing device 10 decides to have the first multimodal large-scale language model 121 generate control information for the operator 20 to execute the instruction task if the predicted time is greater than the threshold time, and decides to have the second multimodal large-scale language model 231 generate the control information if the predicted time is less than or equal to the threshold time.

[0026] The information processing device 10 is a MEC (Multi-access Edge Computing) server. Here, an MEC server is an edge server built in a location closer to the operating device 20, such as a network before it reaches the internet or a local 5G network. This enables the information processing system 1 to achieve low latency in communication between the information processing device 10 and the operating device 20. The following describes the case where the information processing device 10 is a MEC server.

[0027] The operating body 20 is a robotic device that operates based on control information. Based on the control information, the operating body 20 controls its own movement to execute an instructed task. For example, the operating body 20 may be an autonomous mobile robotic device. Alternatively, the operating body 20 may be a multi-joint robotic device (manipulator-type robotic device) equipped with a robotic arm. Furthermore, the operating body 20 may be a robotic device in which a manipulator-type robotic device is mounted on an autonomous mobile robotic device. The following description will focus on the case where the operating body 20 is an autonomous mobile robotic device.

[0028] [3. Configuration of Information Processing Device] Figure 6 is a diagram showing an example of the configuration of an information processing device according to the embodiment. The information processing device 10 according to the embodiment has a communication unit 110, a storage unit 120, and a control unit 130.

[0029] (Communication Unit 110) The communication unit 110 is wirelessly connected to a mobile network (cellular network) and transmits and receives information with the operating unit 20. For example, the communication unit 110 is implemented by a NIC (Network Interface Card) or an antenna. For example, the mobile network may be a 5G mobile network or a 6G mobile network. For example, the mobile network may include network equipment that provides functions for realizing the mobile network. The network equipment may be a base station (BS) or equipment that constitutes a core network (CN). The network equipment may also be an entity called an application function. The core network may be a 5G core network, an EPC (Evolved Packet Core), or a 6th generation or later core network. The base station forms a cell (area cell) and uses this cell to wirelessly communicate with the information processing device 10 and the operating unit 20. The communication unit 110 may include, for example, a Bluetooth® or Wi-Fi® antenna.

[0030] For example, the information processing device 10 may be a server device that implements the functions of a RIC (RAN Intelligent Controller). For example, the information processing device 10 may be a server device that implements the functions of an AI-RAN (Artificial Intelligence Radio Access Network). For example, the information processing device 10 may be a server device located within a mobile network that implements the functions of a base station within the mobile network using software. For example, the communication unit 110 provides functions for realizing a mobile network. For example, the information processing device 10 may form a cell (area cell) and use this cell to communicate wirelessly with the operating unit 20. For example, the communication unit 110 may form a cell (area cell) and use this cell to communicate wirelessly with the operating unit 20.

[0031] (Storage Unit 120) The storage unit 120 is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. Specifically, the storage unit 120 stores information about the first multimodal large-scale language model 121. The storage unit 120 also stores a decision program according to the embodiment. The storage unit 120 also stores environmental map information about the environment surrounding the operating body 20. For example, the environmental map information is a pre-map corresponding to the movement environment of the operating body 20, which is an autonomous mobile robot device, and may be information about a pre-generated pre-map. The storage unit 120 also stores information about a third multimodal large-scale language model that has been trained to output a predicted time when instruction information, sensor information, and environmental map information are input. The storage unit 120 also stores a threshold time until the operating body 20 consumes the available storage capacity to perform a task.

[0032] The multimodal LLM 121 stores information about the first multimodal large-scale language model 121. The first multimodal large-scale language model 121 is a large-scale language model that can handle multiple types of data simultaneously, such as text, images, audio, and video. For example, the first multimodal large-scale language model 121 includes a modality encoder, an input projector, an LLM backbone, an output projector, and a modality generator.

[0033] A modality encoder is an encoder that takes images, audio, and video as input and extracts feature quantities from images, audio, and video. For example, a modality encoder takes instruction information indicating a person's instructions to an operating object 20 and sensor information acquired by the operating object 20 as input and extracts feature quantities from the instruction information and sensor information. For example, the sensor information may include at least one of either image or point cloud information relating to the environment around the operating object 20.

[0034] The input projector converts the feature vectors output from the modality encoder into a data format that large language models (LLMs) can understand. For example, the input projector converts the feature vectors of instruction information and sensor information into a data format that large language models can understand. The input projector generates the feature vectors of instruction information and sensor information converted into a data format that large language models can understand.

[0035] The LLM backbone is a machine learning model that includes Large Language Models (LLMs). The LLM backbone uses data output from an input projector as input to the Large Language Model and generates text from the data output from the input projector. For example, the LLM backbone uses the feature quantities of instruction information and sensor information, which have been converted into a data format that the LLM can understand, as input to the LLM and generates text corresponding to the operation information based on the instruction information and sensor information.

[0036] The output projector converts the text output from the LLM backbone into a data format that the modality generator can understand. The output projector then generates operation information in a data format that the modality generator can understand.

[0037] The modality generator generates modality data other than text. For example, the modality generator generates the actions of agent 20. For example, the modality generator generates the actions of agent 20 by taking action information, which has been converted into a data format that the modality generator can understand, as input to the modality generator.

[0038] The multimodal large-scale language model 121 may be a machine learning model that includes a large-scale vision language model (VLM). For example, the multimodal large-scale language model 121 may be a machine learning model that includes GPT-4, Gemini, CogVLM, AnyGPT, or LLaMA3.

[0039] (Control Unit 130) The control unit 130 is a controller, and is realized by executing various programs (for example, decision programs) stored in the storage device inside the information processing device 10 using RAM or VRAM as the working area, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array).

[0040] The control unit 130 has an acquisition unit 131, a prediction unit 132, a determination unit 133, a securing unit 134, a generation unit 135, and a transmission unit 136 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 6, and other configurations are also possible as long as they perform the information processing described later. Also, each functional unit indicates the function of the control unit 130 and does not necessarily have to be physically separated.

[0041] (Acquisition unit 131) The acquisition unit 131 acquires instruction information indicating instructions given by a person to the operating body 20. For example, the instruction information is in text format. More specifically, the communication unit 110 forms a cell (area cell) and uses this cell to communicate wirelessly with a terminal device used by a person. The acquisition unit 131 also acquires instruction information from the terminal device used by a person via the communication unit 110.

[0042] Furthermore, the acquisition unit 131 acquires sensor information acquired by the operating body 20. More specifically, the acquisition unit 131 acquires sensor information from the operating body 20 via the communication unit 110. For example, the sensor information may be an image. Alternatively, the sensor information may be point cloud information acquired by LIDAR (Light Detection And Ranging).

[0043] (Prediction Unit 132) The prediction unit 132 predicts the estimated time required for the operating body 20 to complete the instruction task in response to the human instruction, based on instruction information indicating human instructions to the operating body 20. More specifically, when instruction information and sensor information are acquired by the acquisition unit 131, the prediction unit 132 refers to the storage unit 120 and acquires a third multimodal large-scale language model. Here, the third multimodal large-scale language model is a multimodal large-scale language model that has been trained to output the estimated time required for the operating body 20 to complete the instruction task in response to the human instruction, given instruction information indicating human instructions to the operating body 20, sensor information acquired by the operating body 20, and environmental map information relating to the environment around the operating body 20 as input. Furthermore, when the prediction unit 132 acquires the third multimodal large-scale language model, it inputs the instruction information, sensor information, and environmental map information acquired by the acquisition unit 131 into the third multimodal large-scale language model to predict the estimated time required for the operating body 20 to complete the instruction task in response to the instruction information.

[0044] (Decision Unit 133) The decision unit 133 determines to cause the first multimodal large language model 121 to generate control information for the operating body 20 to execute the instructed task when the predicted time predicted by the prediction unit 132 is greater than the threshold time until the storage capacity available for the operating body 20 to execute the task is consumed. When the predicted time is less than or equal to the threshold time, it is determined to cause the second multimodal large language model 231 to generate the control information. For example, when the predicted time is predicted by the prediction unit 132, the decision unit 133 acquires information about the threshold time until the storage capacity available for the operating body 20 to execute the task is consumed from the operating body 20. Also, when the decision unit 133 acquires the information about the threshold time, it determines whether the predicted time is greater than the threshold time. Further, when the decision unit 133 determines that the predicted time is greater than the threshold time, it determines to cause the first multimodal large language model 121 to generate the control information for the operating body 20 to execute the instructed task. Also, when the decision unit 133 determines that the predicted time is less than or equal to the threshold time, it determines to cause the second multimodal large language model 231 to generate the control information. Further, when the decision unit 133 determines to cause the second multimodal large language model 231 to generate the control information, it transmits second instruction information instructing the operating body 20 to secure the storage capacity corresponding to the predicted time to the operating body 20.

[0045] (Securing Unit 134) When it is determined by the decision unit 133 to cause the first multimodal large language model 121 to generate the control information, the securing unit 134 secures the storage capacity corresponding to the predicted time. For example, when it is determined by the decision unit 133 to cause the first multimodal large language model 121 to generate the control information, the securing unit 134 secures the storage capacity of the VRAM (GPU memory) of the information processing apparatus 10 corresponding to the predicted time.

[0046] (Generation Unit 135) The generation unit 135 inputs sensor information and instruction information acquired by the operating unit 20 into the first multimodal large-scale language model 121 to generate control information. Specifically, the generation unit 135 generates control information when the storage unit 134 has secured storage capacity corresponding to the predicted time. For example, when the generation unit 135 receives sensor information, instruction information and environmental map information as input, it inputs the sensor information and instruction information acquired by the acquisition unit 131, as well as the environmental map information, into the first multimodal large-scale language model 121, which has been trained to output control information, to generate control information.

[0047] (Transmitting Unit 136) The transmitting unit 136 transmits the control information generated by the generation unit 135 to the operating unit 20. For example, if control information is generated by the generation unit 135, the transmitting unit 136 transmits the control information generated by the generation unit 135 to the operating unit 20.

[0048] Furthermore, if the prediction unit 132 determines that the time has become earlier by the time required to predict the predicted time than the time when the memory capacity reserved by the reservation unit 134 (or reservation unit 243) will be consumed, it predicts the predicted time again. Specifically, the time required to predict the predicted time is known. If the prediction unit 132 determines that the time has become earlier by the time required to predict the predicted time than the time when the memory capacity reserved by the reservation unit 134 (or reservation unit 243) will be consumed, it inputs the instruction information, sensor information, and environmental map information acquired by the acquisition unit 131 within a predetermined time into the third multimodal large-scale language model and predicts the predicted time again. Furthermore, the decision unit 133 may, based on a comparison of the predicted time predicted again by the prediction unit 132 with the threshold time, decide to have the first multimodal large-scale language model 121 generate control information for the operator 20 to execute the instruction task if the predicted time predicted again is greater than the threshold time, and decide to have the second multimodal large-scale language model 231 generate control information if the predicted time predicted again is less than or equal to the threshold time.

[0049] [4. Configuration of the operating body] FIG. 7 is a diagram showing a configuration example of the operating body according to the embodiment. The operating body 20 according to the embodiment includes a sensor unit 210, a communication unit 220, a storage unit 230, a control unit 240, and a drive unit 250.

[0050] (Sensor unit 210) The sensor unit 210 has a function of collecting various sensor information in the environment around the operating body 20. For example, the sensor unit 210 includes an external sensor. The external sensor is a sensor that acquires information such as the shape of an object existing around the operating body 20, the distance and direction to an object existing around the operating body 20 under the control of the control unit 240. For example, the external sensor includes at least one of a camera, LiDAR (Light Detection And Ranging), Sonar, or ToF (Time of Flight). For example, the external sensor includes a 2D-LiDAR that acquires the environment as two-dimensional information or a 3D-LiDAR that acquires the environment as two-dimensional information. Further, the sensor information includes an image generated by a camera. The image includes a still image and a moving image. Further, the sensor information includes point cloud information generated by LiDAR. For example, the sensor information includes point cloud information that is three-dimensional information or two-dimensional information.

[0051] Also, the sensor unit 210 includes an internal sensor. The internal sensor is a sensor for acquiring information such as the moving distance, moving speed, moving direction, and posture of the operating body 20 under the control of the control unit 240. For example, the internal sensor includes an inertial measurement unit (IMU; Inertial Measurement Unit) for detecting the acceleration of the direction and movement of the operating body 20 and an encoder (or potentiometer) for detecting the driving amount of the actuator. Note that the internal sensor may include an acceleration sensor, an angular velocity sensor, etc. in addition to these.

[0052] (Communication Unit 220) The communication unit 220 is wirelessly connected to the mobile network and transmits and receives information with the information processing device 10. For example, the communication unit 210 is implemented by a NIC, an antenna, etc. The communication unit 210 may also include, for example, a Bluetooth® or Wi-Fi® antenna.

[0053] (Storage Unit 230) The storage unit 230 is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 230 stores various programs executed by the operating unit 20. The storage unit 230 also stores environmental map information relating to the environment around the operating unit 20. For example, the environmental map information is a pre-map corresponding to the movement environment of the operating unit 20, which is an autonomous mobile robot device, and may be information relating to a pre-generated pre-map.

[0054] (Control Unit 240) The control unit 240 is a controller and is realized by various programs stored in the storage device inside the operating unit 20, which are executed by a CPU, MPU, GPU, ASIC, FPGA, etc., using RAM or VRAM as the working area.

[0055] The control unit 240 has an acquisition unit 241, an operation control unit 242, a securing unit 243, and a generation unit 244 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 240 is not limited to the configuration shown in Figure 7, and other configurations are also possible as long as they perform the information processing described later. Also, each functional unit represents the function of the control unit 240 and does not necessarily have to be physically separated.

[0056] (Acquisition Unit 241) The acquisition unit 241 acquires sensor information. Specifically, the acquisition unit 241 acquires sensor information from the sensor unit 210. For example, the sensor information may be an image. Alternatively, the sensor information may be point cloud information acquired by LIDAR.

[0057] Furthermore, the acquisition unit 241 acquires instruction information indicating human instructions to the operating body 20. For example, the instruction information is text. More specifically, when the decision unit 133 determines that control information should be generated in the second multimodal large-scale language model 231, the acquisition unit 241 acquires instruction information from the information processing device 10 via the communication unit 110.

[0058] (Operation Control Unit 242) The operation control unit 242 controls the operation of the operating body 20 to execute an instructed task based on the control information. Specifically, the operation control unit 242 controls the operation of the operating body 20 so that it outputs an operation corresponding to the control information. For example, the operation control unit 242 receives control information from the information processing device 10. When the operation control unit 242 receives control information, it controls the operation of the operating body 20 based on the received control information.

[0059] (Reservation Unit 243) When the decision unit 133 determines that control information should be generated in the second multimodal large-scale language model 231, the reservation unit 243 reserves memory capacity corresponding to the predicted time. For example, the reservation unit 243 receives second instruction information from the information processing device 10. Also, when the reservation unit 243 receives the second instruction information, it reserves memory capacity in the VRAM (GPU memory) of the operating unit 20 corresponding to the predicted time according to the second instruction information.

[0060] (Generation Unit 244) The generation unit 244 inputs sensor information and instruction information acquired by the operating unit 20 into the second multimodal large-scale language model 231 to generate control information. Specifically, the generation unit 244 generates control information when the storage unit 243 has secured storage capacity corresponding to the predicted time. For example, when the generation unit 244 receives sensor information, instruction information and environmental map information as input, it inputs the sensor information and instruction information acquired by the acquisition unit 241, as well as the environmental map information, into the second multimodal large-scale language model 231, which has been trained to output control information, to generate control information.

[0061] Figures 8 to 15 are diagrams illustrating an example of information processing according to the embodiment. First, using Figures 8 to 11, an example of information processing when the decision unit 133 decides to generate control information in the first multimodal large-scale language model 121 will be explained. In Figure 8, the acquisition unit 131 of the information processing device 10 acquires instruction information and sensor information acquired by the operating body 20. The prediction unit 132 of the information processing device 10 predicts the prediction time to be 23 seconds based on the instruction information and sensor information acquired by the acquisition unit 131. Also in Figure 8, the threshold time is 10 seconds. The decision unit 133 of the information processing device 10 determines that the prediction time is greater than the threshold time based on a comparison of the prediction time (23 seconds) and the threshold time (10 seconds). Since the prediction time is greater than the threshold time, the decision unit 133 decides to generate control information for the operating body 20 to execute the instruction task in the first multimodal large-scale language model 121.

[0062] Furthermore, the memory capacity of the MEC server, the information processing device 10, is larger than the memory capacity of the operator 20. That is, the amount of memory available to the information processing device 10 for the calculation of the first multimodal large-scale language model 121 is larger than the amount of memory available to the operator 20 for the calculation of the second multimodal large-scale language model 231. Here, the first multimodal large-scale language model 121 and the second multimodal large-scale language model 231 may be exactly the same model. For this reason, the time it takes for the information processing device 10 to consume the amount of memory available for the calculation of the first multimodal large-scale language model 121 is longer than the time it takes for the operator 20 to consume the amount of memory available for the calculation of the second multimodal large-scale language model 231. In Figure 8, the time it takes for the information processing device 10 to consume the amount of memory available for the calculation of the first multimodal large-scale language model 121 is 300 seconds.

[0063] In Figure 9, the information processing device 10's allocation unit 134 allocates the VRAM (GPU memory) storage capacity corresponding to the predicted time (23 seconds) because the decision unit 133 has decided to generate control information in the first multimodal large-scale language model 121. The information processing device 10's generation unit 135 inputs the sensor information and instruction information acquired by the acquisition unit 131 into the first multimodal large-scale language model 121 to generate control information. The information processing device 10's transmission unit 136 transmits the control information generated by the generation unit 135 to the operating unit 20. The operating control unit 242 of the operating unit 20 receives the control information from the information processing device 10.

[0064] In Figure 10, when the motion control unit 242 receives control information from the information processing device 10, it controls the operation of the operating body 20 based on the received control information.

[0065] In Figure 11, the prediction unit 132 of the information processing device 10 determines that the time has become earlier than the time when the memory capacity reserved by the reservation unit 134 will be consumed, by the time required to predict the predicted time, and predicts the predicted time again. In Figure 11, the prediction unit 132 predicts the predicted time again as 35 seconds. Also, if the reservation unit 134 of the information processing device 10 predicts the predicted time again (35 seconds) as predicted by the prediction unit 132, it reserves the memory capacity of the VRAM (GPU memory) of the information processing device 10 corresponding to the predicted time again (35 seconds).

[0066] Next, an example of information processing when the decision unit 133 decides to generate control information in the second multimodal large-scale language model 231 will be explained using Figures 12 to 15. In Figure 12, the acquisition unit 131 of the information processing device 10 acquires instruction information and sensor information acquired by the operating body 20. The prediction unit 132 of the information processing device 10 predicts the prediction time to be 18 seconds based on the instruction information and sensor information acquired by the acquisition unit 131. In Figure 12, the threshold time is 20 seconds. The decision unit 133 of the information processing device 10 determines that the prediction time is less than or equal to the threshold time based on a comparison between the prediction time (18 seconds) and the threshold time (20 seconds). Since the prediction time is less than or equal to the threshold time, the decision unit 133 decides to generate control information in the second multimodal large-scale language model 231. If the decision unit 133 decides to have the second multimodal large-scale language model 231 generate control information, it sends a second instruction to the operator 20 instructing it to secure memory capacity corresponding to the predicted time (18 seconds). Also, in Figure 12, similar to Figure 9, the time until the information processing device 10 consumes the size of memory available for the calculation processing of the first multimodal large-scale language model 121 is 300 seconds.

[0067] In Figure 13, the allocation unit 243 of the operating unit 20 receives the second instruction information. When the allocation unit 243 receives the second instruction information, it allocates the VRAM (GPU memory) storage capacity of the operating unit 20 corresponding to the predicted time (18 seconds) according to the second instruction information. The generation unit 244 of the operating unit 20 inputs the sensor information and instruction information acquired by the operating unit 20 into the second multimodal large-scale language model 231 to generate control information.

[0068] In Figure 14, the motion control unit 242 controls the operation of the operating body 20 based on the control information generated by the generation unit 244.

[0069] In Figure 15, the prediction unit 132 of the information processing device 10 determines that the time has become earlier by the time required to predict the prediction time than the time when the memory capacity allocated by the allocation unit 243 will be consumed, and predicts the prediction time again. In Figure 15, as in Figure 11, the prediction unit 132 predicts the newly predicted time as 35 seconds. The decision unit 133 of the information processing device 10 also determines that the newly predicted time (35 seconds) is greater than the threshold time, based on a comparison between the newly predicted time (35 seconds) and the threshold time (20 seconds). Since the newly predicted time is greater than the threshold time, the decision unit 133 decides to have the first multimodal large-scale language model 121 generate control information for the operator 20 to execute the instruction task. The allocation unit 134 of the information processing device 10, having determined by the decision unit 133 to generate control information in the first multimodal large-scale language model 121, allocates the VRAM (GPU memory) storage capacity of the information processing device 10 corresponding to the again predicted time (35 seconds). The generation unit 244 also obtains information about the KV cache from the operator 20 when the storage capacity corresponding to the again predicted time by the allocation unit 243 has been allocated. For example, the generation unit 244 also sends request information to the operator 20 requesting that it send the KV cache generated within a predetermined time to the information processing device 10. The operator 20 receives the request information from the information processing device 10. Upon receiving the request information, the operator 20 sends the KV cache generated within a predetermined time according to the request information to the information processing device 10. The information processing device 10 obtains information about the KV cache from the operator 20. The generation unit 244 generates new control information based on the KV cache obtained from the operator 20.

[0070] [5. Effects] As described above, the information processing system 1 according to the embodiment includes an information processing device 10 equipped with a first multimodal large-scale language model 121 and an operator 20 equipped with a second multimodal large-scale language model 231. The information processing device 10 includes a prediction unit 132 that predicts the estimated time required for the operator 20 to complete an instruction task in response to a human instruction based on instruction information indicating a human instruction to the operator 20, and a decision unit 133 that, based on a comparison of the predicted time and a threshold time until the operator 20 consumes the available memory capacity to execute the task, decides to have the first multimodal large-scale language model 121 generate control information for the operator 20 to execute the instruction task if the predicted time is greater than the threshold time, and decides to have the second multimodal large-scale language model 231 generate control information if the predicted time is less than or equal to the threshold time. The operator 20 includes an operation control unit 242 that controls the operation of the operator 20 to execute the instruction task based on the control information.

[0071] Thus, if the operator can continuously store information about its own task until the task is completed, the information processing system 1 causes the operator's second multimodal large-scale language model to generate control information for the operator. This enables the information processing system 1 to enable the operator to execute the task appropriately. Furthermore, if the operator cannot continuously store information about its own task until the task is completed, the information processing system 1 causes the first multimodal large-scale language model of an information processing device with a larger memory capacity than the operator to generate control information for the operator. This enables the information processing system 1 to generate appropriate operation information even if the time required for the operator to complete the task exceeds a threshold time, thereby enabling the operator to execute the task appropriately. In addition, because the information processing system 1 enables the operator to execute the task appropriately, it can contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0072] Furthermore, the information processing device 10 includes, when the decision unit 133 decides to generate control information in the first multimodal large-scale language model 121, a reservation unit 134 that reserves memory capacity corresponding to the predicted time, a generation unit 135 that inputs sensor information and instruction information acquired by the operating body 20 into the first multimodal large-scale language model 121 to generate control information, and a transmission unit 136 that transmits the control information generated by the generation unit 135 to the operating body 20.

[0073] As a result, if the operating entity cannot continuously store information about its own task until the task is completed, the information processing system 1 can appropriately generate control information using the first multimodal large-scale language model provided by the information processing device, which has a larger memory capacity than the operating entity.

[0074] Furthermore, the operating unit 20 includes a storage unit 243 that secures memory capacity corresponding to the predicted time when the decision unit 133 decides to generate control information in the second multimodal large-scale language model 231, and a generation unit 244 that inputs sensor information and instruction information acquired by the operating unit 20 into the second multimodal large-scale language model 231 to generate control information.

[0075] As a result, if the operating entity can continuously store information about its own task until the task is completed, the information processing system 1 can appropriately generate control information using the second multimodal large-scale language model provided by the operating entity.

[0076] Furthermore, if the prediction unit 132 determines that the time is earlier than the time when the memory capacity reserved by the reservation unit 134 (or reservation unit 243) will be consumed, by the time required to predict the predicted time, it predicts the predicted time again.

[0077] This enables the information processing system 1 to appropriately generate control information for the next operative before the memory capacity reserved for the computation of the multimodal large-scale language model is consumed.

[0078] Furthermore, the information processing device 10 is a server device located within the mobile network that implements the functions of a base station within the mobile network using software.

[0079] This allows the information processing system 1 to improve the communication speed between the information processing device 10 and the operating device 20. Therefore, the information processing system 1 can enable the operating device 20 to perform appropriate operations quickly.

[0080] Furthermore, the information processing device 10 is a MEC (Multi-access Edge Computing) server.

[0081] This allows the information processing system 1 to improve the communication speed between the information processing device 10 and the operating device 20. Therefore, the information processing system 1 can enable the operating device 20 to perform appropriate operations quickly.

[0082] [6. Hardware Configuration] The information processing device 10 and the operating unit 20 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 16. The following explanation will use the information processing device 10 as an example. Figure 16 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0083] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0084] The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various calculations, such as RAM (Random Access Memory) or VRAM (Video Random Access Memory). The secondary storage device 1050 is a storage device in which data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), a file server, etc.

[0085] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010 that output various types of information, such as displays, projectors, and printers. It is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners. It is implemented using, for example, USB.

[0086] Furthermore, the output interface 1060 and the input interface 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. In other words, the output device 1010 and the input device 1020 may be wireless devices.

[0087] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0088] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0089] The network interface 1080 receives data from other devices via the network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via the network N.

[0090] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0091] For example, when computer 1000 functions as an information processing device 10, the arithmetic unit 1030 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Alternatively, computer 1000's arithmetic unit 1030 may load a program obtained from another device via the network interface 1080 onto the primary storage device 1040 and execute the loaded program. Furthermore, computer 1000's arithmetic unit 1030 may cooperate with other devices via the network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0092] [7. Others] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0093] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0094] For example, the information processing device 10 described above may be implemented using multiple server computers, and depending on the function, it may be implemented by calling external platforms, etc., via API (Application Programming Interface) or network computing, so the configuration can be flexibly changed.

[0095] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0096] 1 Information Processing System 10 Information Processing Device 110 Communication Unit 120 Storage Unit 121 Multimodal LLM 130 Control Unit 131 Acquisition Unit 132 Prediction Unit 133 Decision Unit 134 Reservation Unit 135 Generation Unit 136 Transmission Unit 20 Operating Unit 230 Storage Unit 231 Multimodal LLM 240 Control Unit 241 Acquisition Unit 242 Operation Control Unit 243 Reservation Unit 244 Generation Unit

Claims

1. An information processing system comprising: an information processing device having a first multimodal large-scale language model; and an operator having a second multimodal large-scale language model, wherein the information processing device includes: a prediction unit that predicts the time required for the operator to complete an instruction task in response to a person's instructions, based on instruction information indicating instructions from a person to the operator; a decision unit that, based on a comparison of the prediction time with a threshold time until the operator consumes the available memory capacity to perform the task, decides to have the first multimodal large-scale language model generate control information for the operator to perform the instruction task if the prediction time is greater than the threshold time, and decides to have the second multimodal large-scale language model generate the control information if the prediction time is less than or equal to the threshold time; and the operator includes an operation control unit that controls the operation of the operator to perform the instruction task based on the control information.

2. The information processing system according to claim 1, comprising: a securing unit that secures a memory capacity corresponding to the predicted time when the decision unit has decided to generate the control information in the first multimodal large-scale language model; a generating unit that inputs sensor information and instruction information acquired by the operating body into the first multimodal large-scale language model to generate the control information; and a transmitting unit that transmits the control information generated by the generating unit to the operating body.

3. The information processing system according to claim 1, comprising: a securing unit that secures a memory capacity corresponding to the predicted time when the decision unit determines that the control information should be generated in the second multimodal large-scale language model; and a generating unit that inputs sensor information and instruction information acquired by the operating unit into the second multimodal large-scale language model to generate the control information.

4. The information processing system according to claim 2 or 3, wherein the prediction unit determines that the time is earlier than the time when the memory capacity reserved by the reservation unit will be consumed by the time required to predict the prediction time, and then predicts the prediction time again.

5. The information processing system according to claim 1, wherein the information processing device is a server device located within a mobile network and implements the functions of a base station within the mobile network using software.

6. The information processing system according to claim 1, wherein the information processing device is a MEC (Multi-access Edge Computing) server.

7. An information processing method to be performed by an information processing system including an information processing device having a first multimodal large-scale language model and an operator having a second multimodal large-scale language model, wherein the information processing device predicts the time required for the operator to complete an instruction task in response to a person's instruction, based on instruction information indicating instructions from a person to the operator; determines, based on a comparison of the predicted time with a threshold time until the operator consumes the memory capacity available to perform the task, to have the first multimodal large-scale language model generate control information for the operator to perform the instruction task if the predicted time is greater than the threshold time; determines, to have the second multimodal large-scale language model generate the control information if the predicted time is less than or equal to the threshold time; and controls the operation of the operator to perform the instruction task based on the control information.

8. An information processing device comprising: a prediction unit that predicts the estimated time required for an operating body to complete an instruction task in response to a person's instructions, based on instruction information indicating instructions given by a person to the operating body; and a decision unit that, based on a comparison of the predicted time with a threshold time until the operating body consumes the available memory capacity to perform the task, determines, if the predicted time is greater than the threshold time, to have a first multimodal large-scale language model provided by the information processing device generate control information for the operating body to perform the instruction task, and determines, if the predicted time is less than or equal to the threshold time, to have a second multimodal large-scale language model provided by the operating body generate the control information.

9. A decision method to be executed by an information processing device, comprising: a prediction step of predicting the time required for an operator to complete an instruction task in response to a person's instructions, based on instruction information indicating instructions from a person to the operator; and a decision step of comparing the predicted time with a threshold time until the operator consumes the available memory capacity to perform the task, determining, if the predicted time is greater than the threshold time, to have a first multimodal large-scale language model provided by the information processing device generate control information for the operator to perform the instruction task, and if the predicted time is less than or equal to the threshold time, to have a second multimodal large-scale language model provided by the operator generate the control information.

10. A decision program that causes a computer to execute a prediction procedure which predicts the time required for an operating body to complete an instruction task in response to a human instruction, based on instruction information indicating human instructions to the operating body; and a decision procedure which, based on a comparison of the prediction time with a threshold time until the operating body consumes the memory capacity available to perform the task, determines, if the prediction time is greater than the threshold time, to have a first multimodal large-scale language model provided by the information processing device generate control information for the operating body to perform the instruction task, and if the prediction time is less than or equal to the threshold time, to have a second multimodal large-scale language model provided by the operating body generate the control information.

Images