Information processing device, information processing system, terminal device, and information processing method

Abstract

This facilitates the acquisition of behavioral models that the subject of the action uses when performing a task. [Solution] The information processing device 10 has a control unit 13. The control unit 13 acquires trial information used to learn an action model used when the target of action performs a task from the first terminal device 20 of the first user, and builds an action model by repeatedly performing trials based on the acquired trial information to learn the actions of the target of action when performing a task.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, a terminal device, and an information processing method.

Background Art

[0002] Conventionally, there has been known a technique of obtaining a behavior model by deep reinforcement learning or the like that learns appropriate behavior from a large number of trials and causing a behavior target such as a device to execute a task. For example, in Patent Document 1, the current state of a device is acquired, a first behavior of the device is determined based on the current state and a reinforcement learning model, a second behavior of the device is determined based on the current state and a first rule, and depending on the progress of learning of the reinforcement learning model, either one of the first behavior and the second behavior is selected as a third behavior to be output to the device. A learning device is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technique described in Patent Document 1, there is room for improvement in terms of facilitating the acquisition of a behavior model.

[0005] An object of the present disclosure is to provide an information processing apparatus, an information processing system, a terminal device, and an information processing method that facilitate the acquisition of a behavior model used when a behavior target executes a task.

Means for Solving the Problems

[0006] An information processing apparatus according to an embodiment of the present disclosure is A control unit acquires trial information from the first terminal device of the first user, which is used to learn the behavior model used when the target of the action performs the task, and learns the behavior of the target of the action when performing the task by repeatedly performing trials based on the acquired trial information, and constructs the behavior model. It is equipped with.

[0007] An information processing system according to one embodiment of this disclosure is The above-mentioned information processing device, The first terminal device provides the aforementioned trial information to the information processing device, It is equipped with.

[0008] A terminal device that embodies one embodiment of the present disclosure is A control unit provides trial information used to learn an action model used when an action target performs a task, based on user input, and causes the information processing unit to perform an operation to learn the actions of the action target when performing the task and construct the action model by repeatedly conducting trials based on the acquired trial information. It is equipped with.

[0009] An information processing method according to one embodiment of this disclosure is: This involves obtaining trial information from the user's terminal device that is used to train the behavioral model used when the target of the action performs a task, and By repeatedly conducting trials based on the acquired trial information, the behavior of the target when performing the task is learned, and the behavioral model is constructed. Includes. [Effects of the Invention]

[0010] According to an information processing device, information processing system, terminal device, and information processing method according to one embodiment of the present disclosure, it is possible to facilitate the acquisition of a behavioral model used by an action target when performing a task. [Brief explanation of the drawing]

[0011] [Figure 1]This is a configuration diagram showing the configuration of an information processing system according to one embodiment of this disclosure. [Figure 2] This block diagram shows an example of the configuration of the information processing device, the first terminal device, and the second terminal device, as shown in Figure 1. [Figure 3] This is a sequence diagram illustrating an example of an information processing method performed by the information processing system shown in Figure 1. [Figure 4] Figure 2 is the first schematic diagram illustrating an example of the operation of the information processing device. [Figure 5] This is a second schematic diagram illustrating an example of the operation of the information processing device shown in Figure 2. [Figure 6] This is a third schematic diagram illustrating an example of the operation of the information processing device shown in Figure 2. [Modes for carrying out the invention]

[0012] This section provides a more detailed explanation of the background and problems of conventional technologies.

[0013] In conventional commercial robot systems, motion generation has primarily been achieved using models formulated based on various theories, such as control engineering. For example, autonomous mobile robots typically generate a detailed environmental map in advance based on static structures and feature points in the area they want to travel, and then achieve autonomous driving by matching sensor data with the environmental map during travel to estimate their own position and plan their path. Similarly, robotic arms used in factories have also primarily been operated based on sophisticated mechanical models and theories of control engineering.

[0014] However, in recent years, instead of deductive methods based on such solid theories, inductive methods using behavioral models acquired from a large number of trials through deep reinforcement learning are beginning to spread. In the case of mobile robots, it becomes possible to move by directly outputting robot action commands from input information from external sensors such as camera images, without having to perform self-localization and path planning using a precise environmental map.

[0015] Not limited to mobile robots, methods based on similar learning are also expected to be effective in controlling arm robots and the like. Even for complex tasks that were difficult to achieve with conventional concepts, it has become possible to easily achieve them by training an action model through learning. However, the combinations of conditions such as the hardware configuration of the robot, the operating environment, and the execution tasks are enormous. A large number of trials are required to achieve comprehensive learning of the action model. That is, the acquisition method of the action model has become a major issue.

[0016] An object of the present disclosure is to provide an information processing apparatus, an information processing system, a terminal device, and an information processing method that facilitate the acquisition of an action model used when an action target executes a task. For example, the present disclosure relates to a technology that provides a platform on which an action model of an action target including a robot or the like can be learned, stored, or sold, and facilitates the acquisition of an action model on the platform.

[0017] Hereinafter, an embodiment of the present disclosure will be mainly described with reference to the accompanying drawings.

[0018] FIG. 1 is a configuration diagram showing the configuration of an information processing system 1 according to an embodiment of the present disclosure. With reference to FIG. 1, the outline of the information processing system 1 according to an embodiment of the present disclosure will be mainly described. The information processing system 1 includes an information processing apparatus 10, a first terminal device 20, and a second terminal device 30. In FIG. 1, for simplicity of explanation, one information processing apparatus 10, one first terminal device 20, and one second terminal device 30 are illustrated, but the number of information processing apparatuses 10, first terminal devices 20, and second terminal devices 30 included in the information processing system 1 may each be two or more.

[0019] Each of the information processing apparatus 10, the first terminal device 20, and the second terminal device 30 is communicably connected to a network 40 including, for example, a mobile communication network or the Internet. The information processing apparatus 10, the first terminal device 20, and the second terminal device 30 are communicably connected to each other via the network 40.

[0020] The information processing device 10 includes, for example, one or a plurality of server devices capable of communicating with each other. The information processing device 10 is not limited to these and may include any general-purpose electronic device such as a PC (Personal Computer), tablet PC, smartphone, or wearable device such as a smartwatch, or it may include other electronic devices dedicated to the information processing system 1. The information processing device 10 is managed, for example, by a provider that provides a platform for efficiently acquiring behavioral models used by the target of action when performing a task, via the first user of the first terminal device 20.

[0021] In this disclosure, “behavioral model” includes, for example, a computational model that enables an object to perform a task by determining an action from a predetermined state. For example, the predetermined state changes moment by moment in response to the action of the object. The task is completed by the object repeatedly determining an action from the predetermined state moment by moment. For example, the behavioral model includes a model built on reinforcement learning such as deep reinforcement learning. “Object” includes, for example, any object that performs a predetermined task based on the behavioral model. For example, the object includes robots such as mobile robots, arm robots, or power supply robots, power control equipment used for power management, household appliances such as air conditioners, or any other equipment. For example, a robot as an object may include multiple robots used for tasks such as moving, which cooperate to perform tasks such as transportation.

[0022] A "task" includes, for example, a mobile robot moving from its current position to a predetermined destination, a robotic arm grasping an object with appropriate force without damaging it, a robotic arm grasping a doorknob to open and close a door, a power control device controlling power, or an air conditioner maintaining a comfortable indoor environment. A "first user" may be an individual, such as a child, student, or adult such as an engineer or researcher; a corporation that wants to be directly involved in development by performing the learning work of behavioral models; a university research laboratory; or any other organization. A "provider" may be an individual, a corporation, or any other organization other than a corporation. The provider operates or maintains the platform.

[0023] The first terminal device 20 includes any general-purpose electronic device such as a PC, tablet PC, smartphone, or wearable device such as a smartwatch. The first terminal device 20 includes, for example, an electronic device used by a first user that provides trial information used for learning an action model to the information processing device 10 using a platform provided by the information processing device 10. The first terminal device 20 is not limited to these and may include one or more server devices that can communicate with each other used by the first user, or other electronic devices dedicated to the information processing system 1.

[0024] In this disclosure, “trial information” includes, for example, input information to be input to the behavior model, output information output from the behavior model, reward design for learning the behavior model according to the task, hardware configuration including the structure and arrangement of sensors or actuators of the target of action, network structure of the behavior model, curriculum for advancing the learning of the behavior model in stages, or virtual environment used for simulation to make the target of action act. The trial information may, for example, relate to a single target of action acting independently in reinforcement learning, or it may relate to multiple targets of action acting cooperatively to perform a task.

[0025] In this disclosure, “input information” includes, for example, information acquired by sensors possessed by the object being acted upon. For example, input information includes information from images captured by a camera, location information acquired by GPS (Global-Positioning System), power information acquired by a power sensor, temperature information acquired by a temperature sensor, or humidity information acquired by a humidity sensor. “Output information” includes, for example, information about actions performed by the object being acted upon when executing a task. For example, output information includes the robot’s movement speed or operating speed, the rotation speed of the robot’s wheels, the rotation speed of the robot when the robot changes direction of movement, the angle or angular velocity of each joint of the arm of an arm robot, control information for performing power control, or setting information such as the temperature, airflow, or airflow direction of an air conditioner.

[0026] In this disclosure, “reward design” includes, for example, values ​​that enable the learning of the optimal behavior for the target behavior. Specific examples of reward design are shown in Table 1 below. “Virtual environment” includes, for example, a two-dimensional or three-dimensional environment. Virtual environments include, for example, urban environments, alley environments, construction site environments, or indoor environments such as buildings.

[0027] The second terminal device 30 includes, for example, any general-purpose electronic device such as a PC, tablet PC, smartphone, or wearable device such as a smartwatch. The second terminal device 30 includes, for example, an electronic device used by a second user that provides the information processing device 10 with configuration information of an action target for simulating the action target's actions using a platform provided by the information processing device 10. The second terminal device 30 is not limited to these and may include one or more server devices that can communicate with each other used by the second user, or other electronic devices dedicated to the information processing system 1.

[0028] In this disclosure, “Configuration Information” includes, for example, information such as a simulated model generated by reflecting the basic configuration of the actual device of the target being operated by the second user. For example, the configuration information includes information such as the location, number, or type of sensors, the presence or absence of tires, or the location, number, or type of tires, and other basic hardware configuration information of the actual device. The “Second User” may be, for example, an individual, a legal entity, or any other organization other than a legal entity that develops the target being operated as an actual device.

[0029] In one embodiment, the information processing device 10 acquires trial information from the first user's first terminal device 20, which is used to learn the behavioral model used when the target of action performs a task. The information processing device 10 learns the behavior of the target of action when performing a task by repeatedly conducting trials based on the acquired trial information, and constructs a behavioral model.

[0030] Figure 2 is a block diagram showing an example of the configuration of the information processing device 10, the first terminal device 20, and the second terminal device 30 as shown in Figure 1. The configuration of each device included in the information processing system 1 will be mainly explained with reference to Figure 2.

[0031] As shown in Figure 2, the information processing device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0032] The communication unit 11 includes one or more communication interfaces connected to the network 40. These communication interfaces support, but are not limited to, mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), wired LAN (Local Area Network) standards, or wireless LAN standards. For example, the communication interface may also support short-range wireless communication standards. In one embodiment, the information processing device 10 is communicably connected to the network 40 via the communication unit 11. The communication unit 11 transmits and receives various types of information via the network 40.

[0033] The storage unit 12 includes one or more memories. In this disclosure, “memory” includes, but is not limited to, semiconductor memory, magnetic memory, or optical memory. Each memory included in the storage unit 12 may function as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 12 stores information necessary to realize the operation of the information processing device 10. For example, the storage unit 12 may store a system program, an application program, or embedded software.

[0034] The storage unit 12 stores information obtained through the operation of the information processing device 10. The storage unit 12 stores, for example, a database. This database includes, for example, multiple behavioral models constructed by the information processing device 10 in cooperation with the first terminal device 20 based on trial information acquired from the first terminal device 20. The information stored in the storage unit 12 may be updateable with information acquired from the network 40 via, for example, the communication unit 11.

[0035] The control unit 13 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. In this disclosure, "processor" refers to, but is not limited to, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. "Programmable circuit" refers to, but is not limited to, an FPGA (Field-Programmable Gate Array). "Dedicated circuit" refers to, but is not limited to, an ASIC (Application Specific Integrated Circuit). The control unit 13 is communicatively connected to each component constituting the information processing device 10 and controls the operation of the entire information processing device 10.

[0036] This section will primarily describe the configuration of the first terminal device 20 included in the information processing system 1. As shown in Figure 2, the first terminal device 20 includes a communication unit 21, a storage unit 22, an input unit 23, an output unit 24, and a control unit 25.

[0037] The communication unit 21 includes one or more communication interfaces connected to the network 40. These communication interfaces support, but are not limited to, mobile communication standards such as LTE, 4G, or 5G, wired LAN standards, or wireless LAN standards, and may support any communication standard. For example, the communication interface may also support short-range wireless communication standards. In one embodiment, the first terminal device 20 is communicably connected to the network 40 via the communication unit 21. The communication unit 21 transmits and receives various types of information via the network 40.

[0038] The storage unit 22 includes one or more memories. Each memory included in the storage unit 22 may function as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 22 stores information necessary to realize the operation of the first terminal device 20. For example, the storage unit 22 may store system programs, application programs, or embedded software. The storage unit 22 also stores information obtained through the operation of the first terminal device 20. The information stored in the storage unit 22 may be updateable with information obtained from the network 40 via, for example, the communication unit 21.

[0039] The input unit 23 includes one or more input interfaces that detect user input and acquire input information based on user operations. These input interfaces include physical keys, capacitive keys, pointing devices such as mice, touchscreens integrated with the display of the output unit 24, imaging modules such as cameras, or microphones that accept audio input. The input unit 23 is not limited to these and may include any other input interfaces. Instead of being provided in the first terminal device 20, the input unit 23 may be connected to the first terminal device 20 as an external input device. Any connection method can be used, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or Bluetooth (Bluetooth).

[0040] The output unit 24 includes one or more output interfaces for outputting information to a user of the first terminal device 20. These output interfaces include a display for visually outputting information as an image, a speaker for audibly outputting information as sound, or a vibrator for tactilely outputting information as vibration. In this disclosure, "display" includes, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescent) display. Instead of being provided in the first terminal device 20, the output unit 24 may be connected to the first terminal device 20 as an external output device. Any connection method can be used, such as USB, HDMI®, or Bluetooth®.

[0041] The control unit 25 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The control unit 25 is communicatively connected to each component constituting the first terminal device 20 and controls the operation of the entire first terminal device 20.

[0042] The configuration of the second terminal device 30 included in the information processing system 1 will be mainly described. As shown in Figure 2, the second terminal device 30 has a communication unit 31, a storage unit 32, an input unit 33, an output unit 34, and a control unit 35.

[0043] The communication unit 31 includes one or more communication interfaces connected to the network 40. These communication interfaces support, but are not limited to, mobile communication standards such as LTE, 4G, or 5G, wired LAN standards, or wireless LAN standards, and may support any communication standard. For example, the communication interface may also support short-range wireless communication standards. In one embodiment, the second terminal device 30 is communicably connected to the network 40 via the communication unit 31. The communication unit 31 transmits and receives various types of information via the network 40.

[0044] The storage unit 32 includes one or more memories. Each memory included in the storage unit 32 may function as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 32 stores information necessary to realize the operation of the second terminal device 30. For example, the storage unit 32 may store system programs, application programs, or embedded software. The storage unit 32 also stores information obtained through the operation of the second terminal device 30. The information stored in the storage unit 32 may be updateable with information obtained from the network 40 via, for example, the communication unit 31.

[0045] The input unit 33 includes one or more input interfaces that detect user input and acquire input information based on the provider's operation. These input interfaces include physical keys, capacitive keys, pointing devices such as mice, touchscreens integrated with the display of the output unit 34, imaging modules such as cameras, or microphones that accept voice input. The input unit 33 is not limited to these and may include any other input interfaces. Instead of being provided in the second terminal device 30, the input unit 33 may be connected to the second terminal device 30 as an external input device. Any connection method can be used, such as USB, HDMI®, or Bluetooth®.

[0046] The output unit 34 includes one or more output interfaces for outputting information to a provider using the second terminal device 30. These output interfaces include a display for visually outputting information as an image, a speaker for audibly outputting information as sound, or a vibrator for tactilely outputting information as vibration. Instead of being integrated into the second terminal device 30, the output unit 34 may be connected to the second terminal device 30 as an external output device. Any connection method can be used, such as USB, HDMI®, or Bluetooth®.

[0047] The control unit 35 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The control unit 35 is communicatively connected to each component constituting the second terminal device 30 and controls the operation of the entire second terminal device 30.

[0048] Figure 3 is a sequence diagram illustrating an example of an information processing method performed by the information processing system 1 shown in Figure 1. Referring to Figure 3, the operation of the information processing device 10 according to one embodiment will be explained, and an overview of the operation of the entire information processing system 1 will be provided. The sequence diagram in Figure 3 illustrates the flow of processing performed by the information processing device 10.

[0049] In step S101, the control unit 35 of the second terminal device 30 receives input operations from a second user to input information, for example, via the input unit 33. This information includes, for example, configuration information for simulating the actions of the target, tasks to be performed by the target, and evaluation criteria for evaluating the action model constructed by the information processing device 10. This information is not limited to these and may further include a set of environmental information for performing the tasks of the target.

[0050] In this disclosure, “evaluation criteria” include, for example, criteria for determining the quality of an action model, which are determined by the second user depending on the task. The evaluation criteria include, for example, criteria for characteristics that the second user considers important depending on the task, such as the time to complete, accuracy, or safety when the action subject performs the task using the action model being evaluated.

[0051] More specifically, the evaluation criteria correspond to the degree of emphasis given to each performance indicator, for example, which of the multiple performance indicators of the behavioral model are given more weight. "Performance indicators" include, for example, completion time, accuracy, or energy required. For example, the evaluation criteria include the weights when completion time, accuracy, and energy required are given weights a:b:c, respectively. Performance indicators are expressed by predetermined values ​​such as completion time x[s], accuracy y[m], or energy required z[W].

[0052] For example, the control unit 35 displays a simulation settings screen provided by the information processing device 10 for learning an action model via the output unit 34. The control unit 35 receives input operations from a second user on the settings screen, inputting configuration information, tasks, and evaluation criteria via the input unit 33.

[0053] In step S102, the control unit 35 of the second terminal device 30 provides the information received in step S101 to the information processing device 10. For example, the control unit 35 transmits the information to the information processing device 10 via the communication unit 31 and the network 40. The control unit 13 of the information processing device 10 receives the information from the second terminal device 30 via the network 40 and the communication unit 11. As a result, the control unit 13 acquires the information received by the second terminal device 30 in step S101 from the second terminal device 30.

[0054] In step S103, the control unit 13 of the information processing device 10, having acquired configuration information for simulating the actions of the target of action from the second user's second terminal device 30 in step S102, provides said configuration information to the first terminal device 20. The control unit 13, having acquired information about a task defined by the second user from the second user's second terminal device 30 in step S102, provides information about said task to the first terminal device 20. The control unit 13, having acquired information about evaluation criteria defined by the second user from the second user's second terminal device 30 in step S102, provides information about said evaluation criteria to the first terminal device 20. Not limited to the above, the control unit 13 may also acquire information about the environment for executing the tasks of the target of action from the second user's second terminal device 30 in step S102, and may further provide information about said environment information to the first terminal device 20.

[0055] For example, when the control unit 13 receives access from the first terminal device 20, it causes the output unit 24 of the first terminal device 20 to display a screen for simulating the actions of the target device. The control unit 13 then displays the configuration information, tasks, and evaluation criteria information provided by the second terminal device 30 on the screen.

[0056] In step S104, the control unit 25 of the first terminal device 20 receives input operations from the first user to input trial information, for example, via the input unit 23. For example, the control unit 25 displays a simulation screen provided by the information processing device 10 for learning the behavior model via the output unit 24. The control unit 25 receives input operations from the first user to input trial information on this screen via the input unit 23. For example, the control unit 25 receives input operations from the first user to repeatedly set the values ​​of each parameter included in the trial information through trial and error via the input unit 23.

[0057] For example, the first user of the first terminal device 20 may not immediately perform repeated settings in the environment provided by the information processing system 1, but rather break it down into a curriculum and gradually improve the performance of the behavioral model, ultimately achieving the task with predetermined environmental information from the aforementioned group of environmental information. The group of environmental information allows the task to be executed in multiple scenarios.

[0058] In step S105, the control unit 25 of the first terminal device 20 provides the trial information received in step S104 to the information processing device 10. For example, the control unit 25 transmits the trial information to the information processing device 10 via the communication unit 21 and the network 40. The control unit 13 of the information processing device 10 receives the trial information from the first terminal device 20 via the network 40 and the communication unit 11. As a result, the control unit 13 acquires the trial information that was repeatedly received as input by the first terminal device 20 in step S104 from the first terminal device 20.

[0059] The control unit 13 acquires trial information from the first user's first terminal device 20 that is used to learn the behavioral model used when the target of action performs a task. For example, the control unit 13 acquires the virtual environment in which the target of action acts in the simulation as trial information from the first terminal device 20. For example, the control unit 13 acquires the reward design for learning according to the task, which was provided to the first terminal device 20 as information in step S103, as trial information from the first terminal device 20. If the task involves, for example, moving an autonomous mobile robot to a predetermined destination, the reward design may give a large positive reward for reaching the destination and a large negative reward for colliding with a wall. Specific examples of reward designs are shown in Table 1.

[0060] [Table 1]

[0061] In step S106, the control unit 13 of the information processing device 10 learns the behavior of the target when performing a task by repeatedly conducting trials based on the trial information acquired in step S105, and constructs a behavior model. For example, the control unit 13 may learn the behavior of the target by gradually changing from simple behavior to complex behavior based on the curriculum included in the trial information acquired in step S105. For example, the control unit 13 constructs multiple behavior models based on the trial information acquired from multiple first terminal devices 20 in step S105. The control unit 13 includes the constructed multiple behavior models as data in a database and stores the database in the storage unit 12.

[0062] In step S107, the control unit 13 of the information processing device 10, having obtained evaluation criteria for evaluating the behavioral model constructed in step S106 from the second terminal device 30 as information in step S102, evaluates the behavioral model based on the obtained evaluation criteria. As an example of the evaluation method, the control unit 13 ranks the multiple behavioral models constructed in step S106 based on the evaluation criteria obtained in step S102.

[0063] For example, if a second user sets evaluation criteria that prioritize short completion time, the control unit 13 of the information processing device 10 will rank higher the behavioral models that result in shorter completion times when the target of the action performs the task. If a second user sets evaluation criteria that prioritize accuracy even if it means longer completion times, the control unit 13 will rank higher the behavioral models that result in higher accuracy when the target of the action performs the task. If a second user sets evaluation criteria that prioritize safety over completion time and accuracy, the control unit 13 will rank higher the behavioral models that result in higher safety when the target of the action performs the task.

[0064] In step S108, the control unit 13 of the information processing device 10 determines the selling price when selling the behavioral model from the first user to the second user, according to the ranking of the behavioral model determined in step S107. For example, the control unit 13 determines a higher selling price for the higher the ranking of the behavioral model.

[0065] In step S109, the control unit 35 of the second terminal device 30 receives input operations from the second user, for example, via the input unit 33, to input application information for the second user to adopt the first user. For example, the control unit 35 displays a setting screen provided by the information processing device 10 to notify the first user of the first terminal device 20 of the application information via the output unit 34. The control unit 35 receives input operations from the second user, for inputting application information on the setting screen, via the input unit 33.

[0066] In step S110, the control unit 35 of the second terminal device 30 provides the application information received in step S109 to the information processing device 10. For example, the control unit 35 transmits the application information to the information processing device 10 via the communication unit 31 and the network 40. The control unit 13 of the information processing device 10 receives the application information from the second terminal device 30 via the network 40 and the communication unit 11. As a result, the control unit 13 acquires the application information received by the second terminal device 30 in step S109 from the second terminal device 30.

[0067] In step S111, when the control unit 13 of the information processing device 10 obtains the offer information for the second user to accept the first user from the second user's second terminal device 30 in step S110, it provides the offer information to the first terminal device 20. For example, when the control unit 13 receives access from the first terminal device 20, it causes the output unit 24 of the first terminal device 20 to display a screen for notifying the offer information, and displays the offer information provided by the second terminal device 30 on that screen.

[0068] Subsequently, for example, the control unit 25 of the first terminal device 20 may receive input operations from the first user, for example, via the input unit 23, to input response information to accept or reject the offer information received from the second user. For example, the control unit 25 may display a setting screen provided by the information processing device 10 to notify the second user of the second terminal device 30 of the response information via the output unit 24. The control unit 25 may receive input operations from the first user, for inputting response information on the setting screen, via the input unit 23.

[0069] The control unit 25 of the first terminal device 20 may provide the received response information to the information processing device 10. For example, the control unit 25 may transmit the response information to the information processing device 10 via the communication unit 21 and the network 40. The control unit 13 of the information processing device 10 may receive the response information from the first terminal device 20 via the network 40 and the communication unit 11. In this way, the control unit 13 may acquire the response information received as input by the first terminal device 20 from the first terminal device 20.

[0070] The control unit 13 of the information processing device 10 may, upon acquiring response information from the first user's first terminal device 20, provide said response information to the second terminal device 30. For example, upon receiving access from the second terminal device 30, the control unit 13 may cause the output unit 34 of the second terminal device 30 to display a screen for notifying the response information, and display the response information provided by the first terminal device 20 on that screen.

[0071] Figure 4 is a first schematic diagram illustrating an example of the operation of the information processing device 10 shown in Figure 2. Referring to Figure 4, we will mainly describe an example of a database stored in the storage unit 12 by the control unit 13 of the information processing device 10. In Figure 4, a behavioral model including "A" and "B" is shown as a representative example, but the number of behavioral models included in the database is not limited to two.

[0072] The database stored in the memory unit 12 has a data structure in which information is associated with behavioral models constructed by the information processing device 10 based on trial information from the first terminal device 20. The information associated with the behavioral models includes, for example, identification information of the first user using the first terminal device 20. This information includes, for example, the ranking of the behavioral models determined by the information processing device 10 and the selling price when the behavioral models are sold. This information also includes, for example, configuration information, task information, and information on values ​​for each performance indicator related to evaluation criteria provided by the second terminal device 30.

[0073] In this disclosure, “identification information” includes, for example, account information such as a username for using the platform provided by the information processing device 10, which is pre-registered by a first user through an application on the first terminal device 20. The control unit 13 of the information processing device 10 includes in the database the same number of accounts as identification information as the number of first users who are registered through an application on the first terminal device 20 and who have contributed to the construction of at least one behavioral model.

[0074] The control unit 13 associates the identification information of the first user of the first terminal device 20, which provided trial information for constructing the behavior model "A", with the behavior model "A". The control unit 13 associates the ranking and selling price determined by the information processing device 10 with the behavior model "A". The control unit 13 associates the configuration information, task information, and performance indicator values ​​provided by the second terminal device 30, which are necessary for the learning of the behavior model on the first terminal device 20 side, with the behavior model "A".

[0075] The control unit 13 associates the identification information of the first user of the first terminal device 20, which provided trial information for constructing the behavior model "B", with the behavior model "B". The control unit 13 associates the ranking and selling price determined by the information processing device 10 with the behavior model "B". The control unit 13 associates the configuration information, task information, and performance indicator values ​​provided by the second terminal device 30, which are necessary for the learning of the behavior model on the first terminal device 20 side, with the behavior model "B".

[0076] The control unit 13 performs the same processing on other behavioral models as it does on behavioral model "A" or "B".

[0077] Figure 5 is a second schematic diagram illustrating an example of the operation of the information processing device 10 in Figure 2. Figure 5 shows a first example of the screen displayed on the output unit 24 of the first terminal device 20 when the input unit 23 receives input from the first user to input trial information in step S104 of Figure 3. The first example of the simulation screen displayed on the output unit 24 of the first terminal device 20 will be explained with reference to Figure 5.

[0078] The control unit 25 of the first terminal device 20, for example, when it accesses the information processing device 10 via the network 40, displays a screen for simulating the actions of the target on the output unit 24. For example, the control unit 25 displays a two-dimensional virtual environment on the simulation screen of the output unit 24, which is a virtual environment that simulates the indoor environment in which the target acts. For example, the control unit 25 may display the results of the actions of the target in the virtual environment based on the trial information on the simulation screen each time the values ​​of each parameter included in the trial information are repeatedly set by the first user using the input unit 23.

[0079] Figure 6 is a third schematic diagram illustrating an example of the operation of the information processing device 10 in Figure 2. Figure 6 shows a second example of the screen displayed on the output unit 24 of the first terminal device 20 when the input unit 23 receives input from the first user to input trial information in step S104 of Figure 3. The second example of the simulation screen displayed on the output unit 24 of the first terminal device 20 will be explained with reference to Figure 6.

[0080] The control unit 25 of the first terminal device 20, for example, when it accesses the information processing device 10 via the network 40, displays a screen for simulating the actions of the target on the output unit 24. For example, the control unit 25 displays a three-dimensional virtual environment on the simulation screen of the output unit 24, which is a virtual environment that simulates the outdoor environment in which the target acts. For example, each time the values ​​of each parameter included in the trial information are repeatedly set by the first user using the input unit 23, the control unit 25 may display the results of the actions of the target in the virtual environment based on the trial information on the simulation screen.

[0081] According to the embodiment described above, it is possible to easily acquire an action model used when the target of action performs a task. The information processing device 10 acquires trial information used to learn the action model used when the target of action performs a task from the first terminal device 20 of the first user and constructs the action model. As a result, the information processing device 10 can provide a platform for efficiently acquiring action models via the first terminal device 20. For example, the information processing device 10 can construct a platform that makes learning action models easy and enjoyable, and can involve not only professional engineers or researchers, but also ordinary adults, elementary, junior high, and high school students, and even university students as first users in the action model learning process, and can construct and accumulate a large number of high-quality action models.

[0082] For example, elementary, middle, and high school students or university students, acting as first users, can receive robotics education through the platform described above, provided the target of their actions is a robot. These first users can also enjoy incentives not available in existing robotics classrooms. These incentives include, for example, free education, revenue based on the selling price of action models, the ability to participate online from home without the need for transportation to and from a classroom, accessibility regardless of gender, and avoidance of interpersonal problems. As a result, the information processing device 10 can contribute to the development of many future robotics engineers.

[0083] For first-time users, learning behavioral models using deep reinforcement learning requires less advanced expertise compared to conventional robot control; rather, "wisdom is more important than knowledge." First-time users can also determine step-by-step learning by breaking down trial information into input information, output information, reward design, hardware configuration, behavioral model network structure (expressiveness), curriculum, and virtual environment, making it intuitively easy to understand. A first-time user with good intuition, even a primary school student, can create a high-quality behavioral model.

[0084] Even top-tier engineers don't necessarily have a significant advantage when it comes to training behavioral models using deep reinforcement learning. On the other hand, it's not a simple task that can be outsourced like annotation work in the field of image recognition. The first user needs to be thoroughly familiar with the know-how. Given these circumstances, it can be said that there are strong game-like elements to this process, where unique know-how is acquired through repeated practice. Therefore, given the characteristics of this learning process, the first user can easily and enjoyably perform the behavioral model training in a way that feels like playing a simulation game.

[0085] The information processing device 10 acquires configuration information for simulating the actions of an action target from the second user's second terminal device 30 and provides it to the first terminal device 20. This allows the second user to request the first user to train an action model for operating an action target that is part of their own product or service, via the platform provided by the information processing device 10. For example, the second user can register configuration information as part of planning a competition or tournament and request the first user to train the action model as part of that plan, in addition to the normal game mode.

[0086] The information processing device 10 acquires evaluation criteria for evaluating the constructed behavioral model as information from the second terminal device 30, and evaluates the behavioral model based on the acquired evaluation criteria. This allows the first user and the second user to objectively determine the quality of one behavioral model ultimately constructed by the information processing device 10 through the first user's learning process, as a relative evaluation to other behavioral models. The information processing device 10 can also use the evaluation obtained by automatically assessing the quality of the behavioral model as the basis for determining the selling price of the behavioral model.

[0087] The information processing device 10 ranks the constructed behavioral models based on evaluation criteria. This allows the first and second users to easily determine the quality of one behavioral model relative to others through a relative ranking. The information processing device 10 can also use the ranking obtained by automatically assessing the quality of the behavioral models as the basis for determining the selling price of the behavioral models.

[0088] The information processing device 10 determines the selling price when selling a behavioral model from the first user to the second user, according to the ranking of the behavioral models. This allows the information processing device 10 to appropriately determine the value of the behavioral models. The second user can also purchase high-quality behavioral models on the platform provided by the information processing device 10 by paying a fee equivalent to the determined selling price.

[0089] This allows a second user, for example, to receive support for service deployment through the first user's training of behavioral models, even if they cannot develop the behavioral control of the target robot in-house. The information processing device 10 can also contribute to solving the problem of a shortage of robot engineers or machine learning engineers caused by a declining birthrate, an aging population, or labor shortages. Even if the second user can develop the system in-house, they can reduce the costs associated with research and development. The second user can also significantly reduce research and development costs by purchasing behavioral models that meet their required specifications at a low cost from those displayed on the platform provided by the information processing device 10.

[0090] For example, with robot control based on learning-based behavioral models expected to become mainstream in the future, there is a need for development into diverse motion control, such as object manipulation or movement on uneven terrain. The information processing device 10 can contribute to the advancement of development of actionable robots in various applications, such as smart agriculture, cooking robots, household robots, weeding robots, rescue robots, delivery robots, toilet cleaning robots, or moving robots.

[0091] The provider of the information processing device 10 can, for example, provide the platform free of charge and allow the accumulated behavioral models to be sold to a second user, thereby earning commission income from the sale price. The provider of the information processing device 10 can also, for example, allow the first or second user to view the past transaction history of the behavioral models. The transaction history includes, for example, the price at which behavioral models of a certain quality were sold. The first user who performed the training work on the behavioral models can also earn a profit from the sale of the behavioral models to the second user, minus the commission income paid to the provider, as supplementary income.

[0092] The information processing device 10 acquires the task defined by the second user as information from the second terminal device 30 and provides it to the first terminal device 20, and acquires a learning reward design corresponding to the task as trial information from the first terminal device 20. As a result, the first user can accurately grasp the target task requested by the second user and appropriately determine the reward design to suit that task. The information processing device 10 can easily acquire a highly accurate behavioral model that is better suited to the second user's requirements.

[0093] The information processing device 10 acquires the virtual environment in which the target of action acts in the simulation as trial information from the first terminal device 20. This allows the first user to easily and enjoyably create the virtual environment itself in which the target of action acts in the simulation, in a game-like manner. The information processing device 10 can also increase the motivation of the first user to use the platform and easily increase the number of first users who perform the learning work of the behavioral model.

[0094] The information processing device 10 obtains information from the second terminal device 30 regarding an offer from the second user to recruit the first user and provides it to the first terminal device 20. This allows the information processing device 10 to provide the second user with an opportunity to headhunt the first user. If the second user successfully recruits the first user, they can easily secure personnel to be involved in the research and development of their target project.

[0095] The information processing device 10 learns the actions of the target by gradually changing from simple actions to complex actions based on the curriculum included in the trial information. This allows the first user to progressively train the action model so that the target can perform increasingly complex actions in stages. For example, the first user can train the robot to only move in a straight line, and then continue to train the robot to move in a curved direction. The first user can also accumulate such curricula as their own know-how.

[0096] The information processing device 10 described above can be applied to learning behavioral models for autonomous mobile robots that do not require prior map creation. The information processing device 10 can be applied to learning behavioral models for arm robots that perform various handling tasks such as pick-and-place, opening and closing doors, picking fruit, and other agricultural or household chores. The information processing device 10 can be applied to learning behavioral models for legged robots that move over uneven terrain to perform rescue operations and the like.

[0097] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions included in each configuration or step can be rearranged in a logically consistent manner, and multiple configurations or steps can be combined into one or divided into two.

[0098] For example, it is also possible to configure a general-purpose electronic device such as a smartphone or computer to function as the information processing device 10 according to the above-described embodiment. Specifically, a program describing the processing content that realizes each function of the information processing device 10 according to the embodiment is stored in the memory of the electronic device, and the processor of the electronic device reads and executes the program. Therefore, this disclosure can also be implemented as a program that can be executed by a processor.

[0099] Alternatively, the disclosure may also be realized as a non-temporary computer-readable medium storing a program executable by one or more processors for causing an information processing device 10 or the like to perform each function according to one embodiment. It should be understood that these are also included within the scope of the disclosure.

[0100] For example, at least some of the processing operations performed in the information processing device 10 in the above-described embodiment may be performed in at least one of the first terminal device 20 and the second terminal device 30. At least some of the processing operations performed in at least one of the first terminal device 20 and the second terminal device 30 may be performed in the information processing device 10.

[0101] In the above embodiment, it was explained that the information processing device 10 obtains configuration information for simulating the actions of the target of action from the second user's second terminal device 30 and provides it to the first terminal device 20, but this is not limited to this. The information processing device 10 does not have to obtain configuration information from the second user's second terminal device 30, and even if it does obtain configuration information from the second terminal device 30, it does not have to provide it to the first terminal device 20.

[0102] In the above embodiment, the information processing device 10 was described as acquiring evaluation criteria for evaluating the constructed behavioral model as information from the second terminal device 30 and evaluating the behavioral model based on the acquired evaluation criteria, but it is not limited to this. The information processing device 10 does not have to acquire evaluation criteria as information from the second terminal device 30, and even if it does acquire evaluation criteria as information from the second terminal device 30, it does not have to evaluate the behavioral model based on said evaluation criteria.

[0103] In the above embodiment, the information processing device 10 was described as ranking the constructed behavioral models based on evaluation criteria, but it is not limited to this. Even if multiple behavioral models are constructed, the information processing device 10 does not have to rank them based on evaluation criteria.

[0104] In the above embodiment, the information processing device 10 was described as determining the selling price of the behavioral models according to their ranking, but it is not limited to this. The information processing device 10 may determine the selling price of the behavioral models based on any other factor different from the ranking, in addition to the ranking, or it may not perform the process of determining the selling price of the behavioral models at all.

[0105] For example, instead of the information processing device 10, the control unit 25 of the first terminal device 20 may determine the selling price of the behavioral model based on the evaluation information acquired by the information processing device 10 in step S107 of Figure 3. For example, the control unit 25 may receive an input operation from the first user who has confirmed the evaluation information via the input unit 23 and determine the selling price of the behavioral model. In other words, instead of the information processing device 10, the control unit 25 of the first terminal device 20 may execute the process in step S108 of Figure 3.

[0106] At this time, the control unit 13 of the information processing device 10 may provide the evaluation information acquired in step S107 to the first terminal device 20. When the control unit 25 of the first terminal device 20 determines the selling price of the behavioral model in step S108, it may provide the information of the determined selling price to the information processing device 10.

[0107] For example, in a situation where multiple first users are participating in a competition as creators of behavioral models, the information processing device 10 may disclose the information shown in Figure 4, including information on the evaluation of the behavioral models and their selling prices, to both the first and second users. The information processing device 10 may configure a platform that allows behavioral models with appraisal results to be bought and sold on an application. For example, the first user can freely set the selling price of the behavioral model on the application of the first terminal device 20, and may lower the price if no second user appears as a buyer. In this case, the information processing device 10 may implement a chat function on the above platform to connect the first and second users for price negotiation.

[0108] The second user can also purchase a behavioral model of their choice based on the information disclosed by the information processing device 10. In addition, the information processing device 10 may, instead of only selling behavioral models, or in addition to selling trial information used to construct the behavioral models, also be subject to sale. The second user can purchase this trial information and customize it themselves. Therefore, trial information has high value as an item for sale. The selling price of the trial information may be determined, for example, by the information processing device 10, or by the first terminal device 20 by receiving input operations from the first user via the input unit 23.

[0109] In the above embodiment, the information processing device 10 was described as acquiring a task defined by the second user as information from the second terminal device 30 and providing it to the first terminal device 20, but it is not limited to this. The information processing device 10 does not have to acquire a task as information from the second terminal device 30, and even if it does acquire a task as information from the second terminal device 30, it does not have to provide it to the first terminal device 20.

[0110] In the embodiments described above, trial information includes, but is not limited to, information such as input information, output information, reward design, hardware configuration, behavioral model network structure, curriculum, or virtual environment. Trial information may include any other information in place of or in addition to each of these pieces of information.

[0111] In the above embodiment, it was explained that the information processing device 10 obtains offer information from the second terminal device 30 for the second user to adopt the first user and provides it to the first terminal device 20, but it is not limited to this. The information processing device 10 does not have to obtain offer information from the second terminal device 30, and even if it does obtain offer information from the second terminal device 30, it does not have to provide it to the first terminal device 20.

[0112] In the above embodiment, the information processing device 10 was described as learning the target behavior by gradually changing from simple behavior to complex behavior based on the curriculum included in the trial information, but it is not limited to this. The information processing device 10 does not have to perform such a stepwise learning process.

[0113] In the above embodiment, the information processing system 1 was described as having a second terminal device 30 in addition to the information processing device 10 and the first terminal device 20, but it is not limited to this. The information processing system 1 may have only the information processing device 10 and the first terminal device 20, without the second terminal device 30.

[0114] In the above embodiment, the information processing device 10 was described as providing a platform on which behavioral models of targets can be learned, stored, and sold, but it is not limited thereto. The information processing device 10 does not have to perform processing for at least one of the storage and sale of behavioral models. For example, the information processing device 10 may store behavioral models using any storage medium of an external device accessible in any way from the information processing device 10, instead of or in addition to the storage unit 12 of the information processing device 10 itself. In this case, the information processing system 1 may further include, in addition to the information processing device 10, the first terminal device 20, and the second terminal device 30, such an external device that is communicably connected to a network 40, for example. For example, the information processing device 10 may provide a platform on which behavioral models of targets cannot be sold.

[0115] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] A control unit acquires trial information from the first terminal device of the first user, which is used to learn the behavior model used when the target of the action performs the task, and learns the behavior of the target of the action when performing the task by repeatedly performing trials based on the acquired trial information, and constructs the behavior model. Equipped with, Information processing device. [Note 2] The information processing device described in Appendix 1, The control unit acquires configuration information for simulating the actions of the target of action from the second user's second terminal device and provides it to the first terminal device. Information processing device. [Note 3] The information processing device described in Appendix 2, The control unit acquires evaluation criteria for evaluating the constructed behavioral model as information from the second terminal device, and evaluates the behavioral model based on the acquired evaluation criteria. Information processing device. [Note 4] The information processing device described in Appendix 3, The control unit ranks the constructed plurality of behavioral models based on the evaluation criteria. Information processing device. [Note 5] The information processing device described in Appendix 4, The control unit determines the selling price when selling the behavior model from the first user to the second user, according to the ranking of the behavior model. Information processing device. [Note 6] An information processing device described in any one of the appendices 2 to 5, The control unit acquires the task defined by the second user as information from the second terminal device and provides it to the first terminal device, and acquires the learning reward design corresponding to the task as trial information from the first terminal device. Information processing device. [Note 7] An information processing device described in any one of the appendices 2 to 6, The control unit acquires the virtual environment in which the target of action acts during the simulation as trial information from the first terminal device. Information processing device. [Note 8] An information processing device described in any one of the appendices 2 to 7, The control unit obtains information from the second terminal device regarding the second user's application to adopt the first user and provides it to the first terminal device. Information processing device. [Note 9] An information processing device described in any one of the appendices 1 to 8, The control unit learns the behavior of the target behavior by gradually changing from simple behaviors to complex behaviors based on the curriculum included in the trial information. Information processing device. [Note 10] An information processing device described in any one of the appendices 1 to 9, The first terminal device provides the aforementioned trial information to the information processing device, Equipped with, Information processing system. [Note 11] A control unit provides trial information used to learn an action model used when an action target performs a task, based on user input, and causes the information processing unit to perform an operation to learn the actions of the action target when performing the task and construct the action model by repeatedly conducting trials based on the acquired trial information. Equipped with, Terminal device. [Note 12] This involves obtaining trial information from the user's terminal device that is used to train the behavioral model used when the target of the action performs a task, and By repeatedly conducting trials based on the acquired trial information, the behavior of the target when performing the task is learned, and the behavioral model is constructed. including, Information processing methods. [Explanation of symbols]

[0116] 1. Information Processing System 10 Information Processing Devices 11 Communications Department 12 Storage section 13 Control Unit 20. First Terminal Device (Terminal Device) 21 Communications Department 22 Memory section 23 Input section 24 Output section 25 Control Unit 30 Second Terminal Device 31 Communications Department 32 Storage section 33 Input section 34 Output section 35 Control Unit 40 Networks

Claims

1. A control unit acquires trial information from the first terminal device of the first user, which is used to learn the behavior model used when the target of the action performs the task, and learns the behavior of the target of the action when performing the task by repeatedly performing trials based on the acquired trial information, and constructs the behavior model. Equipped with, Information processing device.

2. An information processing apparatus according to claim 1, The control unit acquires configuration information for simulating the actions of the target of action from the second user's second terminal device and provides it to the first terminal device. Information processing device.

3. An information processing apparatus according to claim 2, The control unit acquires evaluation criteria for evaluating the constructed behavioral model as information from the second terminal device, and evaluates the behavioral model based on the acquired evaluation criteria. Information processing device.

4. An information processing apparatus according to claim 3, The control unit ranks the constructed plurality of behavioral models based on the evaluation criteria. Information processing device.

5. An information processing apparatus according to claim 4, The control unit determines the selling price when selling the behavior model from the first user to the second user, according to the ranking of the behavior model. Information processing device.

6. An information processing apparatus according to any one of claims 2 to 5, The control unit acquires the task defined by the second user as information from the second terminal device and provides it to the first terminal device, and acquires the learning reward design corresponding to the task as trial information from the first terminal device. Information processing device.

7. An information processing apparatus according to any one of claims 2 to 5, The control unit acquires the virtual environment in which the target of action acts during the simulation as trial information from the first terminal device. Information processing device.

8. An information processing apparatus according to any one of claims 2 to 5, The control unit obtains information from the second terminal device regarding the second user's application to adopt the first user and provides it to the first terminal device. Information processing device.

9. An information processing apparatus according to any one of claims 1 to 5, The control unit learns the behavior of the target behavior by gradually changing from simple behaviors to complex behaviors based on the curriculum included in the trial information. Information processing device.

10. An information processing device according to any one of claims 1 to 5, The first terminal device provides the aforementioned trial information to the information processing device, Equipped with, Information processing system.

11. A control unit provides trial information used to learn an action model used when an action target performs a task, based on user input, and causes the information processing unit to perform an operation to learn the actions of the action target when performing the task and construct the action model by repeatedly conducting trials based on the acquired trial information. Equipped with, Terminal device.

12. This involves obtaining trial information from the user's terminal device that is used to train the behavioral model used when the target of the action performs a task, and By repeatedly conducting trials based on the acquired trial information, the behavior of the target when performing the task is learned, and the behavioral model is constructed. including, Information processing methods.

Images