Information processing system, information processing method, information processing device, generation method, and generation program

Abstract

This information processing system includes an information processing device and an operation body. The information processing device includes: an acquisition unit that acquires sensor information acquired by the operation body operating according to a predetermined rule; a generation unit that inputs the sensor information to a multimodal large-scale language model to generate a temporary rule different from the predetermined rule, the temporary rule being a rule for determining an operation according to an environmental condition indicated by the sensor information; and a transmission unit that transmits the temporary rule to the operation body. The operation body includes: a reception unit that receives the temporary rule; and an operation control unit that controls the operation of the operation body according to rules including the temporary rule and the predetermined rule.

Description

Information Processing System, Information Processing Method, Information Processing Apparatus, Generation Method, and Generation Program

[0001] The present invention relates to an information processing system, an information processing method, an information processing apparatus, a generation method, and a generation program.

[0002] In recent years, a multimodal large language model, which is a machine learning model capable of handling multiple types of data (modalities) such as images, voices, and videos in addition to text, is known. Also, techniques for controlling the behavior (also referred to as operation) of a robot or autonomous driving using a multimodal large language model are known. For example, a technique in which the multimodal large language model is caused to generate source code for controlling the operation of a robot, and the operation of the robot is controlled by the generated source code is known.

[0003] Wenlong Huang, et al., "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models", November 2023, [Searched on September 11, 2024], Internet <URL: https: / / arxiv.org / pdf / 2307.05973>

[0004] The information processing system according to the present application is an information processing system including an information processing apparatus and an operating body, wherein the information processing apparatus includes an acquisition unit that acquires sensor information acquired by the operating body operating according to a predetermined rule, a generation unit that inputs the sensor information into a multimodal large language model to generate a temporary rule that is a temporary rule different from the predetermined rule and is a rule for determining the operation according to the environmental conditions indicated by the sensor information, and a transmission unit that transmits the temporary rule to the operating body, and the operating body includes a reception unit that receives the temporary rule and an operation control unit that controls the operation of the operating body according to a rule including the temporary rule and the predetermined rule.

[0005] Figure 1 is a diagram illustrating information processing related to comparative technology. Figure 2 is a diagram showing an example of information processing according to the embodiment. Figure 3 is a diagram showing an example configuration of an information processing system according to the embodiment. Figure 4 is a diagram showing an example configuration of an information processing device according to the embodiment. Figure 5 is a diagram showing an example configuration of an operating body according to the embodiment. Figure 6 is a diagram showing an example of an operating body and operation by the operating body according to the embodiment. Figure 7 is a diagram showing an example of a temporary rule according to the embodiment. Figure 8 is a diagram showing an example of a temporary rule according to the embodiment. Figure 9 is a diagram showing an example of a method for deleting a temporary rule according to the embodiment. Figure 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device.

[0006] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing system, information processing method, information processing device, generation method, and generation program relating to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing system, information processing method, information processing device, generation method, and generation program relating to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0007] (Embodiment) [1. Introduction] Figure 1 is a diagram for explaining information processing related to comparative technology. The robot 4 and MEC server 3 shown in Figure 1 are connected to each other wirelessly so that they can communicate with one another. In Figure 1, communication between the robot 4 and the MEC server 3 requires a large delay time compared to when processing is done using if / then rules inside the robot. First, the robot 4 acquires sensor information about the surrounding environment. For example, the sensor information may be an image. For example, the image acquired by the robot 4 shows a situation where a toy placed on a shelf falls onto a child's head. The robot 4 recognizes the environmental conditions (also called the situation) around the robot 4 from the image it has acquired and performs actions according to the recognized environmental conditions. For example, the robot 4 performs a hazard avoidance action to avoid a dangerous situation according to the base rules that determine the basic actions of the robot 4.

[0008] Here, the base rules are source code created primarily by humans. Therefore, it is impossible for humans to consider the nearly infinite number of environmental conditions for robot 4 and determine rules that correspond to all of them. Consequently, the flexibility of robot 4's movements is low with only the base rules. In other words, robot 4 has difficulty performing flexible movements in response to diverse environmental conditions with only the base rules. For example, in Figure 1, the base rules do not include a rule that determines the actions of robot 4 in a situation where a toy placed on a shelf falls on a child's head (an example of a dangerous situation). Therefore, robot 4 cannot perform a danger avoidance action to prevent the toy from falling on the child's head with only the base rules. To supplement the movements of robot 4, a MEC server 3 equipped with a multimodal large-scale language model is used.

[0009] Specifically, robot 4 transmits the acquired image to MEC server 3, which is equipped with a multimodal large-scale language model. MEC server 3 retrieves the image from robot 4. When MEC server 3 retrieves an image from robot 4, it inputs the acquired image into the multimodal large-scale language model to generate a new rule that determines the actions of robot 4 according to the situation in the environment surrounding robot 4. For example, the new rule may be a rule that determines the actions of robot 4 in a situation where a toy placed on a shelf falls on a child's head. For example, the new rule may be a rule that associates the situation where a toy placed on a shelf falls on a child's head with a danger avoidance action to avoid that situation. When MEC server 3 generates a new rule, it transmits the new rule to robot 4. Robot 4 operates based on the new rule obtained from MEC server 3. As a result, robot 4 can perform a danger avoidance action to avoid the situation where a toy placed on a shelf falls on a child's head, according to the new rule generated by MEC server 3.

[0010] As shown in Figure 1, by using the MEC server 3 equipped with a multimodal large-scale language model, rules can be generated to determine flexible actions in response to diverse environmental conditions. This allows the robot 4 to perform flexible actions in response to diverse environmental conditions. However, when the robot 4 transmits images acquired by the robot to the MEC server 3, it requires a much larger delay compared to processing with if / then rules within the robot. Similarly, when the MEC server 3 transmits new rules it has generated to the robot 4, it also requires a much larger delay compared to processing with if / then rules within the robot. Therefore, when the robot 4 operates based on new rules acquired from the MEC server 3, it takes approximately 40 milliseconds to begin operation, resulting in a slower response from the robot 4 compared to when it operates according to the base rules. In this way, a slow response from the robot 4 means that by the time the robot 4 begins hazard avoidance actions, it may already be in a dangerous situation. Therefore, there are cases where the robot 4 is unable to avoid dangerous situations. Consequently, when using the MEC server 3 equipped with a multimodal large-scale language model, the robot 4 (an example of an operating body) was not always able to perform appropriate actions in response to diverse environmental conditions.

[0011] Figure 2 is a diagram showing an example of information processing according to the embodiment. The information processing according to the embodiment is performed by the information processing system 1 according to the embodiment. The information processing system 1 shown in Figure 2 includes an information processing device 10 and an operating device 20. The information processing device 10 shown in Figure 2 corresponds to the MEC server 3 shown in Figure 1. The operating device 20 shown in Figure 2 corresponds to the robot 4 shown in Figure 1. The information processing device 10 is a MEC (Multi-access Edge Computing) server. Here, the MEC server is an edge server built in a location closer to the operating device 20, such as a network before it goes out to the internet or a local 5G. As a result, the information processing system 1 can achieve low latency in communication between the information processing device 10 and the operating device 20. The following describes the case where the information processing device 10 is a MEC server.

[0012] In Figure 2, the operating body 20 operates according to a base rule (hereinafter sometimes referred to as "predetermined rule") that determines the basic operation of the operating body 20. The operating body 20 acquires sensor information about the surrounding environment. For example, the sensor information may be an image. For example, the image acquired by the operating body 20 shows a situation in which a toy placed on a shelf falls onto a child's head, similar to Figure 1. The information processing device 10 acquires the sensor information acquired by the operating body 20. In Figure 2, the information processing device 10 acquires an image as an example of sensor information. Furthermore, when the information processing device 10 acquires sensor information, it inputs the sensor information into a multimodal large-scale language model to generate a temporary rule, which is a temporary rule different from the base rule and determines the operation according to the environmental conditions shown in the image. In Figure 2, when the information processing device 10 acquires an image, it inputs the image into a multimodal large-scale language model to generate a temporary rule. For example, the temporary rule is a rule that determines the operation of the operating body 20 in response to a situation in which a toy placed on a shelf falls onto a child's head. For example, a temporary rule could be one that associates a situation where a toy placed on a shelf falls on a child's head (an example of a dangerous situation) with an action to take to prevent the toy from falling on a child's head. For example, an action to prevent the toy from falling on a child's head could be to move the toy so that it does not fall, or to maintain a distance at which the arm can protect the child at any time.

[0013] Furthermore, if the information processing device 10 generates a temporary rule, it transmits the temporary rule to the operator 20. The operator 20 receives the temporary rule from the information processing device 10. The operator 20 also controls its operation according to the rule which includes the temporary rule and the base rule. In this way, the information processing system 1 pre-transmits the temporary rule generated by the information processing device 10, which is equipped with a multimodal large-scale language model, to the operator 20. This allows the information processing system 1 to control the operation of the operator 20 according to the rule which includes the temporary rule and the base rule. This prevents the information processing system 1 from experiencing delays in the operator 20's response. This enables the information processing system 1 to enable the operator 20 to avoid dangerous situations. Therefore, the information processing system 1 enables the operator 20 to perform operations according to a variety of environmental conditions.

[0014] [2. Configuration of the Information Processing System] Figure 3 is a diagram showing an example configuration of an information processing system according to the embodiment. As shown in Figure 3, the information processing system 1 according to the embodiment includes an information processing device 10 and an operating unit 20. The information processing device 10 and the operating unit 20 are connected wirelessly via a mobile network. The information processing device 10 and the operating unit 20 are located within the mobile network. Note that the information processing system 1 may include a plurality of information processing devices 10 and a plurality of operating units 20.

[0015] The information processing device 10 acquires sensor information obtained by the operating device 20, inputs the acquired sensor information into a multimodal large-scale language model (hereinafter sometimes referred to as "multimodal LLM") 121, and generates temporary rules, which are temporary rules different from the base rules and determine the operation according to the environmental conditions indicated by the sensor information. When the information processing device 10 generates temporary rules, it transmits the temporary rules to the operating device 20.

[0016] The operating body 20 is a robotic device that operates according to a set of rules, including base rules and temporary rules. For example, the operating body 20 may be an autonomous robotic device that performs autonomous actions according to the surrounding environmental conditions, in accordance with the rules, including base rules and temporary rules. For example, the operating body 20 may be an autonomous mobile robotic device that moves autonomously within space and performs various actions. For example, the operating body 20 may be a robotic device that performs hazard avoidance actions to avoid dangerous situations. The operating body 20 acquires sensor information about the environment around it. The operating body 20 selects a rule according to the environmental conditions indicated by the sensor information acquired within a predetermined time, and controls the operation of the operating body 20 based on the selected rule.

[0017] [3. Configuration of the Information Processing Device] Figure 4 is a diagram showing an example of the configuration of the information processing device according to the embodiment. The information processing device 10 according to the embodiment has a communication unit 110, a storage unit 120, and a control unit 130.

[0018] (Communication Unit 110) The communication unit 110 is wirelessly connected to a mobile network (cellular network) and transmits and receives information with the operating unit 20. For example, the communication unit 110 is implemented by a NIC (Network Interface Card) or an antenna. For example, the mobile network may be a 5G mobile network or a 6G mobile network. For example, the mobile network may include network equipment that provides functions for realizing the mobile network. The network equipment may be a base station (BS) or equipment that constitutes a core network (CN). The network equipment may also be an entity called an application function. The core network may be a 5G core network, an EPC (Evolved Packet Core), or a 6th generation or later core network. The base station forms a cell (area cell) and uses this cell to wirelessly communicate with the information processing device 10 and the operating unit 20. The communication unit 110 may include, for example, a Bluetooth® or Wi-Fi® antenna.

[0019] For example, the information processing device 10 may be a server device that implements the functions of a RIC (RAN Intelligent Controller). For example, the information processing device 10 may be a server device that implements the functions of an AI-RAN (Artificial Intelligence Radio Access Network). For example, the information processing device 10 may be a server device located within a mobile network that implements the functions of a base station within the mobile network using software. For example, the communication unit 110 provides functions for realizing a mobile network. For example, the information processing device 10 may form a cell (area cell) and use this cell to communicate wirelessly with the operating unit 20. For example, the communication unit 110 may form a cell (area cell) and use this cell to communicate wirelessly with the operating unit 20.

[0020] (Storage Unit 120) The storage unit 120 is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. Specifically, the storage unit 120 stores information related to the multimodal LLM 121. The storage unit 120 also stores the generation program according to the embodiment.

[0021] The multimodal LLM121 is a multimodal large-scale language model. A multimodal large-scale language model is a large-scale language model that can handle multiple types of data simultaneously, such as text, images, audio, and video. For example, a multimodal large-scale language model includes a modality encoder, an input projector, and an LLM backbone.

[0022] A modality encoder is an encoder that takes images, audio, and video as input and extracts feature quantities from images, audio, and video. For example, a modality encoder takes sensor information acquired by an operating device 20 as input and extracts feature quantities from the sensor information.

[0023] The input projector converts the feature vectors output from the modality encoder into a data format that large language models (LLMs) can understand. For example, the input projector converts the feature vectors of sensor information into a data format that large language models can understand. The input projector generates the feature vectors of sensor information converted into a data format that large language models can understand.

[0024] The LLM backbone is a machine learning model that includes Large Language Models (LLMs). The LLM backbone uses data output from an input projector as input to the Large Language Model and generates text from the data output from the input projector. For example, the LLM backbone uses feature quantities of sensor information converted into a data format that the LLM can understand as input to the LLM and generates source code or parameter values ​​of functions included in the source code that determine the operation of the operating body 20 according to the environmental conditions indicated by the sensor information.

[0025] The multimodal large-scale language model 121 may be a machine learning model that includes a large-scale vision language model (VLM). For example, the multimodal large-scale language model 121 may be a machine learning model that includes GPT-4, Gemini, CogVLM, AnyGPT, or LLaMA3.

[0026] (Control Unit 130) The control unit 130 is a controller, and is realized by executing various programs stored in the memory device inside the information processing device 10 using RAM as the working area, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit). The control unit 130 is a controller, and is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0027] The control unit 130 has an acquisition unit 131, a generation unit 132, a transmission unit 133, and a selection unit 134 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 4, and other configurations are also acceptable as long as they perform the information processing described later. Furthermore, each functional unit represents the function of the control unit 130 and does not necessarily have to be physically separated.

[0028] (Acquisition Unit 131) The acquisition unit 131 acquires various types of information. Specifically, the acquisition unit 131 acquires sensor information acquired by the operating body 20, which operates according to a base rule. For example, the sensor information may be an image. Alternatively, the sensor information may be point cloud information acquired by LIDAR (Light Detection And Ranging). In this way, the acquisition unit 131 acquires sensor information acquired by the operating body 20, which operates according to a predetermined rule (for example, a base rule). More specifically, the acquisition unit 131 acquires sensor information from the operating body 20.

[0029] (Generation Unit 132) The generation unit 132 generates various types of information. Specifically, the generation unit 132 inputs the sensor information acquired by the acquisition unit 131 into the multimodal large-scale language model 121 and generates temporary rules, which are temporary rules different from the base rules and determine the operation of the operating body 20 according to the environmental conditions indicated by the sensor information. More specifically, the generation unit 132 inputs the sensor information into the multimodal large-scale language model 121, which has been trained to output temporary rules when sensor information is input into the multimodal large-scale language model 121, and generates temporary rules. For example, when sensor information is acquired by the acquisition unit 131, the generation unit 132 refers to the storage unit 120 and acquires the multimodal large-scale language model 121 (hereinafter sometimes referred to as "trained multimodal LLM 121"), which has been trained to output temporary rules when sensor information is input into the multimodal large-scale language model 121. When the generation unit 132 acquires a trained multimodal LLM 121, it inputs sensor information into the trained multimodal LLM 121 and generates temporary rules corresponding to the sensor information. Alternatively, the generation unit 132 may input a prompt to the multimodal large-scale language model 121 along with the sensor information, instructing it to generate temporary rules, and generate temporary rules. The prompt includes a base rule. The prompt may also include an instruction to generate temporary rules that do not contradict the base rule. Furthermore, the prompt may include an instruction to generate a second temporary rule that does not contradict the first temporary rule that was generated earlier.

[0030] Specifically, the temporary rule is control information for causing the operating body 20 to perform an action in accordance with the environmental conditions indicated by the sensor information. The temporary rule includes information that associates condition information indicating the environmental conditions indicated by the sensor information with action information indicating an action in accordance with the environmental conditions indicated by the sensor information. More specifically, the temporary rule is information about a program for causing the operating body 20 to perform an action in accordance with the environmental conditions indicated by the sensor information. For example, the temporary rule includes, as action information, source code containing a function for causing the operating body 20 to perform an action in accordance with the environmental conditions indicated by the sensor information, or the parameter values ​​of a function included in the source code. Note that source code refers to a string of characters (code) that represents a program written in a predetermined programming language. The base rule also includes information that associates information indicating predetermined conditions with information indicating an action in accordance with the predetermined conditions. The base rule is information about a program for causing the operating body 20 to perform an action in accordance with predetermined conditions. For example, the base rule includes source code containing a function for causing the operating body 20 to perform an action in accordance with predetermined conditions, or the parameter values ​​of a function included in the source code.

[0031] Figure 6 shows an example of an operating body and operation by the operating body according to an embodiment. In Figure 6, the case where the operating body 20 is an autonomous mobile robot device is explained. In Figure 6, the basic operation of the operating body 20, which is an autonomous mobile robot device, is explained. In Figure 6, the operating body 20 operates according to the base rules. In Figure 6, the basic operation of the operating body 20 according to the base rules is explained. Specifically, the operating body 20 performs autonomous movement using self-position estimation by map matching. The operating body 20 estimates its own position (robot position) by referring to a map (environmental map information) based on the environment detected using sensors, and achieves autonomous movement by controlling the movement speed, movement direction, etc. according to the estimated self-position. More specifically, the operating body 20 estimates its own position based on sensor information and environmental map information. The operating body 20 may use technologies such as SLAM to generate a surrounding map and estimate its own position based on sensor information. The operating body 20 also calculates a cost map based on sensor information and environmental map information. For example, a cost map is a two-dimensional grid map where each cell (grid) has a cost based on its distance from an obstacle. In a cost map, cells closer to obstacles have higher costs. Also, in a cost map, occupied cells are expanded (i.e., given a bulge) to the size of the operating body 20. For example, a cost map includes information indicating the location of obstacles and the size of the bulge around obstacles, which are located in the environment surrounding the operating body 20, in order to avoid them. The operating body 20 also plans its path based on the cost map. For example, the operating body 20 calculates a movement path (global path) that moves from the starting point to the goal point while avoiding obstacles. The operating body 20 also calculates more detailed movement path candidates (local path candidates) based on the global path. The operating body 20 also follows the path and controls its motors based on the local path candidates.

[0032] Although Figure 6 illustrates the case where the operating body 20 is an autonomous mobile robot device, the operating body 20 is not limited to an autonomous mobile device. For example, the operating body 20 may be a multi-joint robot device equipped with a robotic arm (manipulator-type robot device). Alternatively, the operating body 20 may be a robot device in which a manipulator-type robot device is mounted on an autonomous mobile robot device.

[0033] Figure 7 shows an example of a temporary rule according to the embodiment. The right side of Figure 7 shows the basic operation of the operating body 20 according to the base rule described in Figure 6. The left side of Figure 7 shows the temporary rule. In Figure 7, the base rule includes a function that indicates the size of the bulge around XX (an example of an obstacle) in the coordinates of the environment map around the operating body 20, and a rule that the parameter value of the function that indicates the size of the bulge around XX (an example of an obstacle) is 20 mm. The parameter value that indicates the size of the bulge around the obstacle can be changed in the range of 15 mm to 100 mm. The acquisition unit 131 acquires an image as sensor information showing the situation where a cat is in the environment around the operating body 20. The generation unit 132 inputs the image showing the situation where a cat is in the environment around the operating body 20 to the learned multimodal LLM 121 and generates a temporary rule that sets the bulge radius of the area of ​​XX (an example of an obstacle) in the map coordinates where the cat is located to 100 mm. More specifically, the generation unit 132 generates a temporary rule that sets the parameter value of a function indicating the size of the bulge around XX (an example of an obstacle) in the coordinates of the environmental map surrounding the operating body 20 to 100 mm. When the generation unit 132 generates a temporary rule, it determines whether the generated temporary rule is within the range of changeable parameter values. If the generation unit 132 determines that the generated temporary rule is within the range of changeable parameter values, it decides to adopt the temporary rule. In this way, the generation unit 132 generates temporary rules that determine the parameter values ​​of a function included in the source code for causing the operating body 20 to perform actions according to the environmental conditions indicated by the sensor information.

[0034] The base rule also includes a function indicating the movement speed of the moving body 20 and a rule that the parameter value of the function indicating the movement speed of the moving body 20 is 0.5 (m / s). The value of the movement speed of the moving body 20 can be changed within the range of -1.0 to 1.5 (m / s). The acquisition unit 131 acquires an image as sensor information that shows a situation in which the work (e.g., movement) of the moving body 20 must be done quickly. The generation unit 132 inputs the image showing a situation in which the movement of the moving body 20 must be done quickly into the learned multimodal LLM 121 and generates a temporary rule that sets the value of the movement speed of the moving body 20 in a situation in which the movement of the moving body 20 must be done quickly to 10 (m / s). When the generation unit 132 generates a temporary rule, it determines whether the generated temporary rule is within the range of the changeable parameter value. If the generation unit 132 determines that the generated temporary rule is not within the range of the changeable parameter value, it decides not to adopt the temporary rule.

[0035] Figure 8 is a diagram showing an example of a temporary rule according to the embodiment. The right side of Figure 8 is the same as the right side of Figure 7. The left side of Figure 8 shows the temporary rule. In Figure 8, the acquisition unit 131 acquires point cloud information as sensor information, indicating a situation in which a point cloud cluster of a certain size or larger has occurred within the YY coordinate range of the environmental map around the operating body 20. The generation unit 132 inputs the point cloud information acquired by the acquisition unit 131 into the learned multimodal LLM 121 and generates source code that includes a function indicating that the operating body 20 moves ZZm to the left and stops when a point cloud cluster of a certain size or larger has occurred within the YY coordinate range of the environmental map around the operating body 20. For example, the generation unit 132 adds the generated source code to line XX of the source code showing the path plan. In this way, the generation unit 132 generates a temporary rule, which is source code that includes a function for causing the operating body 20 to perform an action according to the environmental conditions indicated by the sensor information.

[0036] (Transmitting Unit 133) The transmitting unit 133 transmits various types of information. Specifically, the transmitting unit 133 transmits temporary rules generated by the generation unit 132 to the operating unit 20. More specifically, when temporary rules are generated by the generation unit 132, the transmitting unit 133 transmits the temporary rules to the operating unit 20. For example, the transmitting unit 133 transmits the temporary rules to the operating unit 20 via the communication unit 110.

[0037] (Selection Unit 134) The selection unit 134 selects unnecessary rules from among the temporary rules according to the environmental conditions indicated by the sensor information. For example, the selection unit 134 selects temporary rules other than those corresponding to the environmental conditions indicated by the sensor information from among the temporary rules already transmitted to the operating unit 20 by the transmission unit 133 as unnecessary rules. The selection unit 134 also transmits instruction information to the operating unit 20 instructing it to delete the unnecessary rules. For example, if the selection unit 134 has selected unnecessary rules, it generates instruction information instructing it to delete the selected unnecessary rules. The selection unit 134 also transmits instruction information to the operating unit 20 when it has generated such information.

[0038] Figure 9 shows an example of a method for deleting temporary rules according to the embodiment. In Figure 9, the selection unit 134 selects temporary rules (1) and (3), other than temporary rule (2) which corresponds to the environmental conditions indicated by the sensor information, as unnecessary rules from among the three temporary rules (1) to (3) that have already been transmitted to the operator 20 by the transmission unit 133. When the selection unit 134 has selected temporary rules (1) and (3) as unnecessary rules, it transmits instruction information to the operator 20 instructing it to delete temporary rules (1) and (3). When the operator 20 receives the instruction information, it deletes temporary rules (1) and (3) according to the instruction information.

[0039] [4. Configuration of the Operating Body] Figure 5 is a diagram showing an example of the configuration of the operating body according to the embodiment. The operating body 20 according to the embodiment has a sensor unit 210, a communication unit 220, a storage unit 230, a control unit 240, and a drive unit 250.

[0040] (Sensor Unit 210) The sensor unit 210 has the function of collecting various sensor information in the environment surrounding the moving body 20. For example, the sensor unit 210 includes an external sensor. The external sensor is a sensor that, under the control of the control unit 240, acquires information such as the shape of objects present around the moving body 20 and the distance and direction to objects present around the moving body 20. For example, the external sensor includes at least one of the following: a camera, LiDAR (Light Detection And Ranging), Sonar, or ToF (Time of Flight). For example, the external sensor includes a 2D-LiDAR that acquires the environment as two-dimensional information or a 3D-LiDAR that acquires the environment as two-dimensional information. The sensor information also includes images generated by the camera. The images include still images and moving images. The sensor information also includes point cloud information generated by the LiDAR. For example, the sensor information includes point cloud information that is three-dimensional or two-dimensional information.

[0041] Furthermore, the sensor unit 210 includes an internal sensor. The internal sensor is a sensor that acquires information such as the distance traveled, speed, direction of movement, and posture of the moving body 20 under the control of the control unit 240. For example, the internal sensor includes an inertial measurement unit (IMU) for detecting the orientation and acceleration of the movement of the moving body 20, and an encoder (or potentiometer) for detecting the amount of drive of the actuator. In addition to these, the internal sensor may also include an acceleration sensor, an angular velocity sensor, etc.

[0042] (Communication Unit 220) The communication unit 220 is wirelessly connected to the mobile network and transmits and receives information with the information processing device 10. For example, the communication unit 220 is implemented by a NIC, an antenna, etc. The communication unit 220 may also include, for example, a Bluetooth® or Wi-Fi® antenna.

[0043] (Memory Unit 230) The memory unit 230 is realized by, for example, a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The memory unit 230 stores various programs executed by the operating body 20. Specifically, the memory unit 230 stores a program corresponding to the base rule. Also, the memory unit 230 stores a program corresponding to the temporary rule. Further, the memory unit 230 stores environmental map information regarding the environment around the operating body 20. For example, the environmental map information may be a pre-map corresponding to the moving environment of the operating body 20 and information regarding a pre-generated pre-map.

[0044] (Control Unit 240) The control unit 240 is a controller and is realized, for example, by a CPU, an MPU, etc., by executing various programs stored in the storage device inside the operating body 20 using the RAM as a work area. Also, the control unit 240 is a controller and is realized by an integrated circuit such as an ASIC or an FPGA.

[0045] The control unit 240 has an acquisition unit 241, a reception unit 242, and an operation control unit 243 as functional units and may realize or execute the operations of information processing described below. Note that the internal configuration of the control unit 240 is not limited to the configuration shown in FIG. 5, and other configurations may be used as long as they perform the information processing described later. Also, each functional unit indicates the function of the control unit 240 and does not necessarily have to be physically distinguished.

[0046] (Acquisition Unit 241) The acquisition unit 241 acquires sensor information. Specifically, the acquisition unit 241 acquires sensor information from the sensor unit 210. Also, when the acquisition unit 241 acquires sensor information, it transmits the sensor information to the information processing device 10.

[0047] (Reception Unit 242) The reception unit 242 receives a temporary rule from the information processing device 10. When the reception unit 242 receives a temporary rule, it stores information regarding the temporary rule in the memory unit 120. Also, the reception unit 242 receives instruction information from the information processing device 10. When the reception unit 242 receives instruction information, it deletes unnecessary rules according to the instruction information.

[0048] (Operation Control Unit 243) The operation control unit 243 controls the operation of the operating body 20 according to rules including temporary rules and base rules. Specifically, the operation control unit 243 selects a rule corresponding to the environmental conditions indicated by the sensor information acquired within a predetermined time from the rules including temporary rules and base rules, and controls the operation of the operating body 20 based on the selected rule. For example, the operation control unit 243 recognizes the environmental conditions (situation) indicated by the sensor information acquired within a predetermined time based on the sensor information acquired within a predetermined time. For example, the operation control unit 243 recognizes the environmental conditions indicated by the sensor information acquired within a predetermined time by performing object recognition, motion recognition, obstacle recognition, etc. using known techniques based on the sensor information acquired within a predetermined time. Also, the operation control unit 243 may comprehensively estimate the situation using the pre-stored environmental map information.

[0049] Also, the operation control unit 243 selects a rule corresponding to the environmental conditions indicated by the sensor information acquired within a predetermined time from the rules including temporary rules and base rules, and controls the operation of the operating body 20 based on the selected rule. For example, the operation control unit 243 selects the value of the parameter of the function included in the source code or the function included in the source code that includes the function for causing the operating body 20 to perform an operation corresponding to the environmental conditions indicated by the sensor information acquired within a predetermined time from the source code corresponding to the rules including the source code corresponding to the temporary rule and the source code corresponding to the base rule. Subsequently, the operation control unit 243 controls the operation of the operating body 20 based on the value of the parameter of the function included in the source code or the function included in the source code that includes the function for causing the operating body 20 to perform an operation corresponding to the environmental conditions indicated by the sensor information. For example, the operation corresponding to the environmental conditions indicated by the sensor information may be a danger avoidance operation for avoiding a dangerous situation.

[0050] (Drive Unit 250) The drive unit 250 has the function of driving the physical components of the operating body 20. The drive unit 250 has the function of performing the operation of the operating body 20. For example, the drive unit 250 has the function of moving the position of the operating body 20. For example, the drive unit 250 controls the movement of the position of the operating body 20 according to the control of the motion control unit 243. The drive unit 250 is, for example, an actuator or a motor driver. The drive unit 250 may have any configuration as long as the operating body 20 can perform the desired operation. For example, the drive unit 250 moves the operating body 20 and changes the position of the operating body 20 by driving the movement mechanism of the operating body 20 in response to instructions from the motion control unit 243.

[0051] [5. Modifications] The processing according to the above embodiment may be carried out in various other forms besides those described above.

[0052] In the embodiment described above, the selection unit 134 selects an unnecessary rule from the temporary rules according to the environmental conditions indicated by the sensor information and transmits instruction information to the operator 20 to delete the unnecessary rule, and the receiving unit 242 deletes the unnecessary rule according to the instruction information when it receives the instruction information. In a modified example, the receiving unit 242 may delete the temporary rule after a predetermined time has elapsed since receiving the temporary rule.

[0053] In another modified version, the information processing device 10 acquires the operation results of the operating body 20 that operated based on the temporary rules (for example, control information indicating whether or not it was able to pass through obstacles) from the operating body 20. The information processing device 10 also generates information that associates sensor information, temporary rules corresponding to the environmental conditions indicated by the sensor information, and the operation results of the operating body 20 that operated based on the temporary rules. The information processing device 10 also inputs the information that associates sensor information, temporary rules, and operation results into a large-scale language model for generating RAG (Retrieval Augmented Generation) data to generate RAG data. For example, the RAG data may include information indicating the percentage of operation results of the operating body 20 that operated based on the temporary rules that indicate success. For example, a result indicating success may be a result indicating that the operating body 20 completed its operation within a predetermined time. For example, a result indicating success may be a result indicating that the operating body 20 moved from the starting point to the goal point while avoiding obstacles. The person who created the base rules refers to information indicating the percentage of successful outcomes for the operative 20 that operated based on the temporary rules. For example, if the percentage of successful outcomes for the operative 20 that operated based on the temporary rules exceeds a predetermined threshold, the temporary rule is added to the base rules. In this way, the person who created the base rules updates the base rules based on the temporary rules.

[0054] [6. Effects] As described above, the information processing system 1 according to the embodiment includes an information processing device 10 and an operating body 20. The information processing device 10 includes an acquisition unit 131 that acquires sensor information acquired by the operating body 20 which operates according to predetermined rules, a generation unit 132 that inputs the sensor information into a multimodal large-scale language model and generates temporary rules which are temporary rules different from the predetermined rules and which determine the operation according to the environmental conditions indicated by the sensor information, and a transmission unit 133 that transmits the temporary rules to the operating body 20. The operating body 20 includes a receiving unit 242 that receives the temporary rules and an operation control unit 243 that controls the operation of the operating body 20 according to rules which include the temporary rules and the predetermined rules.

[0055] As a result, the information processing system 1 can control the operation of the operator 20 according to rules that include temporary rules and base rules, by pre-passing temporary rules generated by the information processing device 10 equipped with a multimodal large-scale language model to the operator 20. Therefore, the information processing system 1 can enable the operator 20 to operate in accordance with a variety of environmental conditions. Furthermore, because the information processing system 1 can enable the operator 20 to operate in accordance with a variety of environmental conditions, it can contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0056] Furthermore, the motion control unit 243 selects a rule from among the rules, including temporary rules and predetermined rules, that corresponds to the environmental conditions indicated by the sensor information acquired within a predetermined time, and controls the operation of the operating body 20 based on the selected rule.

[0057] This enables the information processing system 1 to allow the operating body 20 to perform actions in accordance with the environmental conditions indicated by sensor information acquired within a predetermined time.

[0058] Furthermore, the information processing device 10 is further equipped with a selection unit 134 that selects unnecessary rules from among the temporary rules according to the environmental conditions indicated by the sensor information. The selection unit 134 transmits instruction information to the operator 20 instructing it to delete the unnecessary rules. The receiving unit 242 receives the instruction information and deletes the unnecessary rules according to the instruction information.

[0059] This prevents the information processing system 1 from generating temporary rules for each environmental condition indicated by the sensor information, which would make it difficult to generate new temporary rules that do not contradict the rapidly increasing number of temporary rules.

[0060] Furthermore, the actions taken in response to the environmental conditions indicated by the sensor information are hazard avoidance actions designed to prevent dangerous situations.

[0061] This enables the information processing system 1 to perform hazard avoidance actions to prevent the operating body 20 from encountering dangerous situations.

[0062] Furthermore, the information processing device 10 is a server device located within the mobile network that implements the functions of a base station within the mobile network using software.

[0063] This allows the information processing system 1 to improve the communication speed between the information processing device 10 and the operating device 20. Therefore, the information processing system 1 can enable the operating device 20 to perform operations quickly in response to various environmental conditions.

[0064] Furthermore, the information processing device 10 is a MEC (Multi-access Edge Computing) server.

[0065] This allows the information processing system 1 to improve the communication speed between the information processing device 10 and the operating device 20. Therefore, the information processing system 1 can enable the operating device 20 to perform operations quickly in response to various environmental conditions.

[0066] [7. Hardware Configuration] The information processing device 10 and the operating unit 20 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 10. The information processing device 10 according to the embodiment will be described below. Figure 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 10. The computer 1000 includes a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0067] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0068] The HDD 1400 stores programs executed by the CPU 1100, and data used by such programs. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0069] The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0070] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0071] For example, when the computer 1000 functions as an information processing device 10 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0072] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0073] [8. Others] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0074] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0075] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0076] 1 Information Processing System 10 Information Processing Device 110 Communication Unit 120 Storage Unit 121 Multimodal LLM 130 Control Unit 131 Acquisition Unit 132 Generation Unit 133 Transmission Unit 134 Selection Unit 20 Operating Unit 240 Control Unit 241 Acquisition Unit 242 Receiving Unit 243 Operation Control Unit

Claims

1. An information processing system comprising an information processing device and an operating body, wherein the information processing device comprises: an acquisition unit that acquires sensor information acquired by the operating body operating according to predetermined rules; a generation unit that inputs the sensor information into a multimodal large-scale language model and generates temporary rules which are temporary rules different from the predetermined rules and which determine the operation according to the environmental conditions indicated by the sensor information; and a transmission unit that transmits the temporary rules to the operating body, wherein the operating body comprises: a receiving unit that receives the temporary rules; and an operation control unit that controls the operation of the operating body according to rules which include the temporary rules and the predetermined rules.

2. The information processing system according to claim 1, wherein the operation control unit selects a rule from among the rules, including the temporary rule and the predetermined rule, that corresponds to the environmental conditions indicated by the sensor information acquired within a predetermined time, and controls the operation of the operating body based on the selected rule.

3. The information processing system according to claim 1, further comprising a selection unit that selects unnecessary rules from the temporary rules according to the environmental conditions indicated by the sensor information, the selection unit transmits instruction information to the operating unit instructing it to delete the unnecessary rules, and the receiving unit receives the instruction information and deletes the unnecessary rules according to the instruction information.

4. The information processing system according to claim 1, wherein the operation in response to the environmental conditions indicated by the sensor information is a hazard avoidance operation to avoid a dangerous situation.

5. The information processing system according to claim 1, wherein the information processing device is a server device located within a mobile network and implements the functions of a base station within the mobile network using software.

6. The information processing system according to claim 1, wherein the information processing device is a MEC (Multi-access Edge Computing) server.

7. An information processing method performed by an information processing system including an information processing device and an operating body, wherein the information processing device acquires sensor information obtained by the operating body which operates according to predetermined rules, inputs the sensor information into a multimodal large-scale language model to generate temporary rules which are temporary rules different from the predetermined rules and which determine the operation according to the environmental conditions indicated by the sensor information, transmits the temporary rules to the operating body, and the operating body receives the temporary rules and controls the operation of the operating body according to rules which include the temporary rules and the predetermined rules.

8. An information processing device comprising: an acquisition unit that acquires sensor information obtained by an operating body that operates according to predetermined rules; a generation unit that inputs the sensor information into a multimodal large-scale language model and generates temporary rules which are temporary rules different from the predetermined rules and which determine the operation according to the environmental conditions indicated by the sensor information; and a transmission unit that transmits the temporary rules to the operating body.

9. A generation method executed by an information processing device, comprising: an acquisition step of acquiring sensor information obtained by an operating body that operates according to predetermined rules; a generation step of inputting the sensor information into a multimodal large-scale language model to generate temporary rules which are temporary rules different from the predetermined rules and which determine the operation according to the environmental conditions indicated by the sensor information; and a transmission step of transmitting the temporary rules to the operating body.

10. A generation program comprising: an acquisition procedure for acquiring sensor information obtained by an operating body that operates according to predetermined rules; a generation procedure for inputting the sensor information into a multimodal large-scale language model to generate temporary rules that are different from the predetermined rules and determine the operation according to the environmental conditions indicated by the sensor information; and a transmission procedure for transmitting the temporary rules to the operating body.

Images