Information processing system, robot, robot control method, and robot control program

Abstract

The system uses information to enable tasks and actions tailored to the specific characteristics of each robot. [Solution] The information processing device stores user authentication information used to verify the user, which includes normal attribute information including basic user setting information and special attribute information. The robot has a trained model that generates drive instruction information that instructs the drive mechanism to be driven using external information detected by sensors, based on the user authentication information, and performs drive control of the drive mechanism using external information detected by sensors, based on the drive instruction information.

Description

Technical Field

[0001] The present invention relates to an information processing system, a robot, a robot control method, and a robot control program. Background Art

[0002] With recent technological development, robotic technology has been developing rapidly in diverse fields, and is being effectively utilized in various scenarios not limited to the industrial field, including medical care, welfare, service industry, agriculture and other fields. Conventional robots have mainly been focused on performing regular work that repeatedly carries out predetermined motion patterns. Currently, however, there are many scenarios that require autonomous judgment based on recognition of the surrounding environment, and work and motions based on such judgment.

[0003] By way of example only, industrial robots have mainly been used for performing work such as workpiece conveyance, assembly, welding and painting at high speed and with high accuracy on production lines. While conventional industrial robots generally operated within safety fences to avoid contact with humans, in recent years collaborative robots (also referred to as "cobots") that cooperate to perform work in the same work space as humans have emerged.

[0004] Furthermore, in the field of mobile robots, AGVs (Automated Guided Vehicles) and AMRs (Autonomous Mobile Robots) have been introduced for improving logistics efficiency in warehouses and factories, and more advanced autonomous traveling technology and obstacle avoidance technology have been developed. In addition, in various service fields, the practical application of robots for customer service and guidance is accelerating. In addition, in the medical and welfare field, surgical support robots, rehabilitation support robots, nursing care support robots and the like have been developed, and both high-precision motion and safety are required. Prior Art Documents Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2025-57204 [Patent Document 2] Japanese Patent Publication No. 2025-62515 [Overview of the project] [Problems that the invention aims to solve]

[0006] Reference 1 describes an invention relating to a system including an AI chatbot for providing after-sales support. More specifically, it describes how the AI ​​chatbot interacts with users in a conversational format, accepts instructions from users and procedures in a conversational format, and automatically links with the system to reflect those instructions.

[0007] Furthermore, Reference 2 describes an invention that improves the maneuverability of a robot while reducing the amount of data traffic transmitted during remote control of the robot. Specifically, it describes how efficient remote control is possible by identifying a region of interest on the robot's video based on operation data and setting the video quality of that region higher than that of the region of no interest.

[0008] These various robots employ specific control methods tailored to their respective applications, but in these conventional technologies, the main principle is that each robot performs appropriate tasks according to its chosen control method.

[0009] The present invention aims to solve the problems of the above-mentioned prior art by sharing necessary information with a robot and using that information to perform tasks and actions according to the characteristics of the robot. [Means for solving the problem]

[0010] To solve the above problems, the information processing system of the present invention is A robot comprising: a storage unit; a processor that performs information processing based on the information stored in the storage unit; the storage unit includes an information processing device that stores user authentication information used to verify the user, which includes user attribute information; a robot storage unit that stores at least robot identification information; a robot communication unit; a sensor capable of detecting external information; a drive mechanism including an actuator; a trained model that generates drive instruction information that instructs the drive mechanism to be driven using external information detected by the sensor, based on the user authentication information; and a drive control unit that performs drive control of the drive mechanism using external information detected by the sensor, based on the drive instruction information; the attribute information includes normal attribute information including the user's basic settings information and special attribute information including the user's physical information; the robot storage unit further stores the drive instruction information and drive control information obtained by the drive control unit as training data; the trained model uses the user authentication information including the attribute information as input information to the trained model and outputs the drive instruction information which includes drive information obtained by the drive control unit using the training data. .

[0011] Furthermore, the robot of the present invention, The robot communication unit receives user authentication information from an information processing device which includes a storage unit that stores at least user authentication information used to verify the user, including user attribute information, and a processor that performs information processing based on the information stored in the storage unit; a robot storage unit that stores at least robot identification information; a sensor capable of detecting external information; a drive mechanism including an actuator; a trained model that generates drive instruction information that instructs the drive mechanism to perform drive control using the external information detected by the sensor based on the user authentication information; and a drive control unit that performs drive control of the drive mechanism using the external information detected by the sensor based on the drive instruction information. The attribute information includes normal attribute information including the user's basic settings information and special attribute information including the user's physical information. The robot storage unit further stores the drive instruction information and drive control information obtained by the drive control unit as training data. The trained model uses user authentication information including the attribute information as input information to the trained model, and output information including drive instruction information obtained by the drive control unit using the training data. . [Effects of the Invention]

[0012] The present invention has the effect of enabling optimal motion control tailored to the characteristics of each robot. Furthermore, the present invention offers the advantage of enabling the application of an integrated control architecture to various types of robots and to different robots of the same type. Furthermore, the present invention has the effect of enabling the appropriate sharing of technical knowledge and necessary information among robots. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a diagram showing the configuration of an information processing system in an embodiment of the present invention. [Figure 2] Figure 2 shows an example of user authentication information in an embodiment of the present invention. [Figure 3] Figure 3 is a functional block diagram showing the user terminal and detailed functions of the robot that constitute the information processing system in an embodiment of the present invention. [Figure 4] Figure 4 is a functional block diagram showing the functions of the robot that constitutes the information processing system in an embodiment of the present invention. [Figure 5] Figure 5 shows the configuration of a computer. [Figure 6] Figure 6 is a diagram illustrating an overview of the information processing system in an embodiment of the present invention. [Figure 7] Figure 7 is a table diagram showing an example of a robot ID management table used in an information processing system according to an embodiment of the present invention. [Figure 8] Figure 8 is a table diagram showing an example of a service usage status management table used in an information processing system according to an embodiment of the present invention. [Figure 9] Figure 9 is a table diagram showing an example of a robot management table used in an information processing system according to an embodiment of the present invention. [Figure 10]FIG. 10 is a table diagram showing an example of a position information management table used in the information processing system according to an embodiment of the present invention. [Figure 11] FIG. 11 is a table diagram showing an example of a maintenance history table used in the information processing system according to an embodiment of the present invention. [Figure 12] FIG. 12 is a table diagram showing an example of a permission information table used in the information processing system according to an embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing an example of a screen displayed on a user terminal that constitutes the information processing system according to an embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing an example of a screen displayed on a user terminal that constitutes the information processing system according to an embodiment of the present invention. [Figure 15] FIG. 15 is a sequence diagram showing the flow of processing in the information processing system according to an embodiment of the present invention. [Figure 16] FIG. 16 is a diagram showing the detailed flow of robot control processing performed by a robot that is a target device according to an embodiment of the present invention. DESCRIPTION OF EMBODIMENTS

[0014] An embodiment of the present invention will be described with reference to the drawings. Note that the following embodiments can apply each other's techniques to each other, including modifications. Furthermore, the following embodiments do not limit the content of the present invention, and modifications can be made without departing from the scope of the claims.

[0015] FIG. 1 is a diagram showing the configuration of an information processing system according to an embodiment of the present invention. The information system shown in FIG. 1 includes a user terminal 10, an AI passport 20, an external server 30, an external DB 40, a robot 50, a management device 60 provided in an external organization or the like, and a management device DB 70, all of which are in a state capable of communicating with each other via a network communication network. Furthermore, although Figure 1 shows only one of each of these devices, this is merely for illustrative purposes, and it is naturally possible to have multiple instances of each device. Moreover, this information processing system does not have to exist independently; it may also be part of another system. In other words, it may be implemented as a function within another system.

[0016] The user terminal 10 is an electronic device (computer) used by users (also referred to as "users") who receive services provided by the operation and driving of the robot 50, and is also referred to as an "information processing device." This user terminal 10 has the configuration shown in Figure 5 and includes a storage unit, a processor, an interface (including a display unit, an operation unit, etc.), a communication unit, etc. Various processes are performed by the processor executing an executable program stored in the storage unit.

[0017] The memory unit of this user terminal 10 stores user authentication information. This user authentication information is used to authenticate the user and is also called user certificate information, user authentication information, user electronic certificate information, or AI passport. It includes first user information, which includes user attribute information, and second user information, which consists of information about the robot 50. This user attribute information includes a profile such as name, age, gender, family structure, hobbies and preferences, operation history (device usage history, etc.), location information, address, biometric information, behavioral history information, affiliated company information (affiliated organization information), and role (position).

[0018] Furthermore, the user terminal 10 provides necessary instructions and information for the user to receive services provided by the robot 50 by operating the control unit, and also displays information related to the services provided by the robot 50 on the display unit. In particular, when using the service, the user terminal 10 transmits necessary information, such as the AI ​​passport 20 and the instruction information necessary to receive the service provided by the robot 50, to the robot 50 and the external server 30 via the communication unit.

[0019] Upon receiving this information, the robot 50 and the external server 30 perform processing to provide predetermined services based on the necessary instructions, using the AI ​​passport 20 as the basis for the required information. In particular, the robot 50 provides predetermined services by performing predetermined actions according to these instructions. Furthermore, the user terminal 10 receives information from the robot 50 and the external server 30.

[0020] The AI ​​Passport 20 in this case is user authentication information consisting of a configuration such as that shown in Figure 2.

[0021] Figure 2 shows an example of user authentication information in an embodiment of the present invention. In Figure 2, the AI ​​Passport 20, which is user authentication information, has an AI Passport Identification Information (PassID) set to identify the AI ​​Passport. This PassID is used by the robot 50 that registered the AI ​​Passport to identify the AI ​​Passport when customizing the service menu for the services provided by that robot. In other words, if robot 50 receives multiple AI Passports and provides a predetermined service based on each AI Passport, the service to be provided is identified by the PassID of the AI ​​Passport. Furthermore, the AI ​​Passport may include, at a minimum, first user information consisting of special attribute information 21 and normal attribute information 22 as user attribute information, and may also include second user information consisting of information about the robot. The first user information, consisting of special attribute information 21 and normal attribute information 22, is preferably stored in the user terminal 10. However, the special attribute information 21 may be stored in the user terminal 10, and the normal attribute information 22 may be stored separately in an external storage medium. In particular, special attribute information 21 is highly confidential information, such as occupational information and personal physical information. Ordinary attribute information 22 is information used on a daily basis to identify an individual.

[0022] This standard attribute information 22 includes information that can be used as basic data in RAG (Retrieval-Augmented Generation). This information is used as basic data (explanatory variables in the trained model) in the trained model (also called the "trained model" or "inference model") installed on the robot 50. For example, it includes information contained in the user's attribute information, prescribed information (rules), various manuals (including manuals corresponding to the robot type), literature, papers, etc. This standard attribute information 22 further includes user basic setting information, which can be set for each target device receiving the service, such as the robot 50, and is basic setting information applicable to the target device. Of course, user basic setting information may also be set for each type of robot. This user basic setting information includes, for example, the basic movement speed desired by the user for the robot 50, preferred colors, and prohibited information regarding actions (drives) that the user wants to prohibit from the robot 50. Furthermore, the attribute information 22 can include pre-trained features such as preference patterns extracted from the user's past robot usage history, frequently used services, and preferred driving speed and operation patterns. These features can serve as important explanatory variables in the trained model for generating more accurate driving instruction information.

[0023] In addition, the regular attribute information 22 includes information on the items to be inherited, and this information includes information on goods, real estate, money, etc. to be inherited in advance or through a will. This information on the items to be inherited is used in inheritance and succession processing. The information subject to this transfer includes conditional information (including PINs, encryption keys, etc.) that enables the disclosure (output) of the information subject to this transfer. On the user terminal 10, when a predetermined operation is performed, a read processing program is executed to read the information to be inherited and to receive condition information. The program determines whether the information entered as this condition information matches (satisfies) or does not match (does not satisfy) the first predetermined inheritance confirmation information (such as a password related to inheritance consent). If they match, the information to be inherited is disclosed (output) to the interface. If it matches the second predetermined inheritance confirmation information (such as a password related to inheritance waiver), the information to be inherited is deleted in a way that makes it impossible to recover.

[0024] Next, the second type of user information, which concerns the robot, includes instruction information (user instruction information), which is the content of the instructions given by the user to the robot 50, and at least a portion of the drive control information, which controls the drive mechanism. This information can also be used as training data for the learning process in the trained model of the robot 50 (reinforcement learning). In addition, although not shown in the diagram, a second set of user information may be provided, which may include robots designated for each facility as registered robots.

[0025] This instruction information concerns instructions for the robot 50 to perform processing, operation, and driving. For example, it primarily provides instructions for industrial robots that perform tasks such as workpiece transport, assembly, welding, and painting on manufacturing lines, specifying the type of work, operation speed, and number of operations (executions). It also provides instructions for mobile robots (delivery robots) specifying the object to be moved, the specific operation content, movement speed, and delivery route. Furthermore, for medical and welfare robots, it provides instructions regarding surgical procedures, rehabilitation procedures, and caregiving procedures.

[0026] Furthermore, the drive control information, which represents the drive control of the drive mechanism, is the result of the drive control performed by the robot 50 based on the drive instruction information, using external information detected by sensors capable of detecting external information. Furthermore, while it is possible to store all of this result information as drive control information, at least some of it (a portion of the drive control information) will be included. In other words, a portion of the drive control information is held as second user information. Examples of some of this drive control information include the details of the drive processing performed based on the instruction information, and also the drive result, which indicates whether or not the drive processing was performed successfully in response to the drive instruction information.

[0027] This AI Passport 20 can also be described as user-specific basic information (profile information) that is commonly used across various target devices and multiple robots of different types. The robot 50 and the external server 30 use the AI ​​passport 20 as necessary information, and based on the AI ​​passport 20, they perform processing to provide predetermined services according to the necessary instruction information.

[0028] Next, we will explain with reference to Figure 1. The external server 30 shown in Figure 1 appropriately connects the user terminal 10 with the target device such as the robot 50, enables communication, and manages information related to the specified service appropriately, in order for a specified service to be provided to the target device such as the robot 50 by providing instructions and necessary information from the user terminal 10.

[0029] This external server 30 performs initial communication support processing when communication between the user terminal 10 and target devices such as the robot 50 begins. This initial communication support processing corresponds to processes S1502, S1505, etc., in Figure 15, which will be described later. Furthermore, the external server 30 manages the status of the devices, communication status, and service usage, as well as managing necessary historical data, when processing between the user terminal 10 and target devices such as the robot 50. This data management includes processes such as S1512, S1517, and S1523 in Figure 15, which will be described later.

[0030] The external DB 40 is a storage medium that stores information managed by the external server 30, and stores tables that manage the usage status of communication between the user terminal 10 and the target device (for example, a service usage status management table), and various information when the user terminal 10 and the target device are in a communication state, and for a certain period of time after the communication state has been disconnected (information related to the robot, including operating status, location information, maintenance history, and communication history between the user terminal 10 and the target device (history associated with robot ID, user ID, PassID (described later), date, etc.)).

[0031] Next, robot 50 shows an example of target equipment that provides a predetermined service. As shown in Figure 1, this target equipment includes industrial robots, humanoid robots, vehicles, aircraft, small aircraft, mobile communication devices, and electrical equipment (personal computers, computers). In particular, Robot 50 includes industrial robots such as manufacturing robots like assembly and welding robots used in automobile manufacturing lines, AGV (Automated Guided Vehicle) transport robots that transport parts and products within factories, and inspection robots such as image recognition robots that perform quality inspections of products. It also includes service robots such as robots that provide guidance and customer service in stores and reception areas, delivery robots that deliver goods to designated locations, and security robots that patrol and monitor facilities. In addition, there are medical robots, agricultural robots, construction robots, and educational robots. In this context, a robot refers to an industrial machine that possesses automatic control for manipulation and movement, and can perform various tasks (driving) based on predetermined instructions (driving instruction information) using a program or a pre-trained model.

[0032] These robots 50 include at least unique identification information (robot identification information, robot ID) to identify the robot 50, a robot memory unit that stores a control program for controlling the robot's drive, a robot communication unit that communicates with the outside world, sensors capable of detecting external information, a drive unit (drive mechanism) including various actuators, a learned model, and a drive control unit that controls the drive of the drive unit (drive mechanism). This trained model, also called a learning model or inference model, is capable of generating information (such as drive instruction information) used for drive control of the robot 50 based on the AI ​​Passport (user authentication information). Specifically, it uses the AI ​​Passport (user authentication information) as an explanatory variable (input information) and training data (trained data) to generate drive instruction information that instructs drive control using external information detected by sensors as the target variable (output information). In addition, this trained model can also generate drive signals (as explanatory variables) that instruct the robot's drive mechanism (actuators, etc.) based on the generated drive instruction information (as the target variable).

[0033] Then, in robot 50, the robot memory unit stores the drive instruction information and the drive control information obtained by the drive control unit driving the drive mechanism as learning data, in association with each other. The trained model uses this training data to learn (reinforcement learning), and after training, the trained model repeatedly generates information (such as drive instruction information) used for driving control of the robot 50 based on the AI ​​passport (user authentication information). This enables the trained model to generate drive instruction information in accordance with the AI ​​Passport and user instruction information, and allows the provision of a predetermined service in which the robot 50 performs appropriate drive control according to the AI ​​Passport and user instruction information.

[0034] (Knowledge sharing among multiple robots) The information processing system of the present invention has a mechanism for sharing learning data among multiple robots. Learning data stored in the robot memory unit of the first robot (the sharing source robot) can be transmitted to the second robot (the sharing destination robot) via the robot communication unit. This transmission is achieved either through direct robot-to-robot communication or indirect communication via an external server.

[0035] In the case of knowledge sharing via an external server, the external server 30 stores the learning data collected from multiple robots in the external DB 40 (external database). The first robot converts the learning data into an intermediate representation format and then sends it to the external server 30 along with robot identification information. The external server 30 stores the intermediate representation format in the external database in association with the robot identification information.

[0036] When the second robot performs transfer learning, it requests an intermediate representation format from the external server 30 of another robot with a drive mechanism similar to its own robot type. Based on the robot type of the second robot, the external server 30 selects an intermediate representation format that can be used for transfer learning and provides it to the second robot. This selection is made based on criteria such as the similarity of robot types, the commonality of tasks, and the similarity of environmental conditions.

[0037] Next, the management device 60 is a management device installed in an external organization that manages unique identification information for target equipment such as the robot 50. The management device DB70 connected to this management device 60 manages a unique identification number (serial number) for all target equipment. An example of this is shown in Figure 7 and will be described later. This management device 60 performs at least two processes: acquiring an identification number (robot identification information, robot ID) (numbering process) and associating the acquired identification number with the equipment identification information of the target equipment (referred to as equipment ID, robot ID) and storing it in the management device DB 70. The process of obtaining identification numbers involves sequentially obtaining identification numbers according to the numbering system, as well as randomly obtaining identification numbers different from those already obtained. In other words, when connecting to the user terminal 10 and using a new target device such as a robot, the manufacturer or user of the target device will inquire with the management device 60 to obtain a unique identification number and register it in the target device. This management device 60 may also manage information such as the location of robots to which identification numbers have been assigned, and whether or not the robots are alive.

[0038] This system configuration enables the provision of desired services by realizing optimal drive control (motion control) according to the characteristics of each robot 50, based on user attribute information and instruction information from the AI ​​passport of the user terminal 10.

[0039] Figure 3 is a functional block diagram showing the detailed functions of the user terminal 10 and robot 50 that constitute the information processing system in an embodiment of the present invention. In Figure 3, the user terminal 10 is comprised of a communication unit 101 (also referred to as a transmission unit, reception unit, etc.), a robot information processing unit 102, a storage unit 104, a location information acquisition unit 105, a display unit 106, an operation unit 107, and an AI passport update unit 108 (also referred to as the "information update unit"). The display unit 106 and the operation unit 107 may be integrated into a single touch panel. The robot 50 shown in Figure 3 represents the functions of the information processing unit 501 of the robot 50 shown in Figure 4. The robot 50 (main control unit 500) is capable of communicating with a storage unit 580 that stores the learned model 51 equipped by the robot 50, a communication unit 570, and is equipped with a learned model 51, a state management unit 52, an information management unit 53, a robot control processing unit 54, and a service menu creation unit 55.

[0040] This invention refers to a wide variety of robots used in target equipment, including industrial robots that manufacture, transport, and inspect products; service robots that perform customer service, guidance, cleaning, delivery, and security; medical robots that perform surgical assistance, rehabilitation support, and nursing care support; and agricultural and construction robots that perform agricultural support, construction support, and surveying support. The invention does not limit the type of robot, but for the sake of explanation, the following description will focus on service robots that provide customer service and delivery as services. Providing customer service and delivery as services is merely one example, and of course, robots that provide only customer service or only delivery as standalone services are also acceptable. By performing predetermined actions and drive control in the service robot, the robot 50 will provide customer service and delivery services to users.

[0041] The storage unit 104 in the user terminal 10 stores user attribute information, including user authentication information (AI Passport) used to verify the user's identity. In addition, the storage unit 104 stores service programs. When a user receives a service provided by the robot 50, for example, by executing a service program and referring to the screen displayed on the display unit 106, the user operates the operation unit 107 of the user terminal 10, and the robot information processing unit 102 specifies the type of service specified by the user and specific instruction information for the service. For example, in customer service, the user can specify information such as the type of product, the name of the product, the quantity, and the color, and then specify instructions (orders) based on this information as instruction information. In delivery service, the user can specify information such as the delivery destination, the delivery route, the estimated completion time for delivery, and the items to be delivered.

[0042] Furthermore, the robot information processing unit 102 acquires location information using the location information acquisition unit 105. This location information acquisition unit 105 is equipped with a GNSS receiver that receives GNSS radio waves from multiple artificial satellites and an inertial measuring device that measures the acceleration and angular velocity of the robot 50, allowing it to grasp the robot's location information and movement status (whether it is moving, the movement path, etc.). Based on the location information received by this GNSS receiver, the user's current location is identified, and the location where the robot is deployed is determined. This inertial measurement device is equipped with a 3-axis gyro sensor (angular velocity meter) and a 3-axis acceleration sensor (accelerometer), and measures the three-dimensional angular velocity and acceleration of the robot 50. The robot information processing unit 102 can grasp the current position information and movement status (delivery status). For example, by using the three-dimensional angular velocity and acceleration measured by the inertial measurement device, it is possible to determine whether the robot 50 has fallen over, stopped, or been involved in an accident.

[0043] When the robot information processing unit 102 receives instruction information given by the user to the robot, or location information based on the user's location received by the GNSS receiver, it receives the serial number of the robot 50 in that location information from the external server 30 via the communication unit 101, and displays the robots 50 available in that location information (list display, etc.). In displaying this list of robots, the robot information processing unit 102 requests the external server 30 for the user's identification information (user ID) and information about robots available at that location (robot ID, etc.). In response, the external server 30 uses tables as shown in Figures 9 and 12 to provide the user terminal 10 with information about robots available to the user (user's identification information).

[0044] As a result, when a robot 50 is specified by the user, the robot information processing unit 102 of the user terminal 10 records the robot ID of the specified robot 50 and sends the user ID to that robot 50.

[0045] At this time, via the communication unit 570 of the robot 50, the information management unit 53 records the user ID in the storage unit 580 and requests an AI passport from the user terminal associated with that user ID. The robot information processing unit 102 of the user terminal 10 then transmits the AI ​​passport 20 to the robot 50.

[0046] Figure 6 shows the user, the user terminal 10 that the user operates, the AI ​​passport 20 stored in the memory unit of the user terminal 10, the external server 30 that can communicate with the user terminal 10, location information where robots can be used (restaurant RT1, shop SP1, office OF1), and the robots installed at each location. In the robot information processing unit 102, when any of these location information is specified or identified, the robots belonging to that location information are displayed. For example, if "Restaurant RT1" is identified as the location information, three robots will be displayed: Robot 1 (RB_1), Robot 2 (RB_2), and Robot 3 (RB_3).

[0047] Next, I will explain using Figure 3. The information management unit 53 of the robot 50 stores the AI ​​passport 20 in the storage unit 580. The service menu creation unit 55 then creates a service menu related to the services provided by the robot 50 based on the AI ​​passport. In other words, it creates an individual service menu corresponding to the AI ​​passport and sends it to the user terminal. For example, in customer service, the service menu would show the services provided, such as seating arrangements and a list of available products. Similarly, in delivery services, the service menu would show the delivery details, such as delivery destinations, specified items, and delivery fees.

[0048] Then, the robot information processing unit 102 specifies instruction information (user instruction information) for the robot based on the service menu provided by the robot. The service menu provided by the robot at this time is a service menu created by the robot based on the contents of the AI ​​Passport and provided individually to the user of the AI ​​Passport.

[0049] The robot information processing unit 102 displays this service menu on the display unit 106, and when it receives a service specification operation from the service menu via the operation unit 107, it transmits instruction information based on the specified operation to the robot 50.

[0050] As a result, the robot control processing unit 54 of the robot 50 performs robot control processing and provides services. The detailed flow of this robot control processing is shown in Figure 16 and will be described later. In this robot control process, based on user instruction information and the AI ​​passport stored in the memory unit, the trained model 51 uses training data to generate and output drive instruction information, which includes drive information for controlling the robot's drive mechanism. Then, based on this drive instruction information, the drive mechanism, including the actuators, is controlled to provide the service. Finally, drive control information is created as a result of this drive control of the drive mechanism.

[0051] The robot control processing unit 54 then transmits the drive instruction information and drive control information for the robot control processing to the user terminal 10, and further stores the learning data, which includes the drive instruction information and drive control information, in the storage unit. This training data is used in a training process (reinforcement learning) using the pre-trained model 51.

[0052] The trained model 51, as described above, is also called a learning model or inference model, and performs a learning process and an information generation process, which is an example of an inference process. The learning process involves reinforcement learning using the learning data described above, and the information generation inference process generates information (such as drive instruction information) used for driving control of the robot 50 based on the AI ​​passport (user authentication information). Specifically, the AI ​​passport (user authentication information) is used as an explanatory variable (input information), and using the learning data (trained data), drive instruction information that instructs driving control using external information detected by the sensor is generated as the target variable (output information).

[0053] Furthermore, the training process in the pre-trained model 51 employs transfer learning, allowing the knowledge of drive control learned on any robot to be applied to robots of the same robot type or to robots of different robot types with similar drive mechanisms. In other words, the pre-trained model has a transfer learning function that learns drive control for its own robot through transfer learning using training data generated on other robots different from its own.

[0054] This transfer learning method will be explained in detail. First, the correspondence (operation pattern) between the drive instruction information generated by the trained model 51 in the source robot and the resulting drive control information is converted into a general-purpose intermediate representation format. This intermediate representation format is an abstract format that does not depend on the robot's specific physical characteristics (number of actuators, range of motion, type of sensors, etc.), and is expressed as an operation pattern broken down into basic operation units such as "grasping operation," "movement operation," and "rotation operation." In other words, the trained model includes a conversion unit that converts the correspondence between drive instruction information and drive control information generated by other robots (whether it is transfer learning between robots of the same type or transfer learning between robots of different types, as described later) into an intermediate representation format that does not depend on the robot's specific physical characteristics.

[0055] In transfer learning between robots of the same type, at least the motion patterns converted to the intermediate representation format described above are used as common motion patterns. Furthermore, in this transfer learning, the trained parameters (also referred to as "weight parameters" or "weight coefficients") of the trained model may be reused. Specifically, in delivery robot A operating at restaurant RT1, the actual drive control information is converted into an intermediate representation of motion patterns such as "tray holding operation: stability level X, vibration suppression level Y", "movement between tables: path type = shortest, speed K, obstacle avoidance mode = on", and "stop position adjustment: distance N from the edge of the table, angle M degrees". Then, this motion pattern and the trained parameters of the corresponding trained model are saved as a set. When introducing delivery robot B, which is the same type as robot A, the operation pattern in this intermediate representation format is interpreted according to the robot's own environmental characteristics (e.g., table arrangement, aisle width, floor material), and the corresponding trained parameters are set as the initial values ​​of the robot's trained model. For example, if robot A's "tray holding operation: stability level X" corresponds to controlling the tilt angle when a drink is placed within a predetermined angle, robot B, which can achieve the same stability, will apply the learned parameters corresponding to this operation pattern and perform fine-tuning to match the slope and steps of the restaurant RT2 floor.

[0056] On the other hand, in transfer learning between robots of different types but with similar drive mechanisms, knowledge conversion and adaptation processes are performed via the motion patterns of the intermediate representation format described above. Specifically, the drive control of a multi-layer tray type delivery robot is broken down into a series of motion patterns in an intermediate representation format, such as "food placement operation: tray number N, placement position center," "rotation operation: rotation in place, 180 degrees," "aisle movement: straight movement, congestion level G," and "height adjustment operation: adjust to table height H." The target robot (a single-layer cart-type delivery robot) performs a process to convert these motion patterns into its own executable motion space. For example, the motion pattern "height adjustment operation: H" of a multi-layer tray-type robot is converted in the adaptive layer to an alternative motion pattern "stop position adjustment: shorten the approach distance to the table side by MM" because the single-layer cart-type robot does not have a height adjustment mechanism. This conversion rule is learned as an equivalence mapping table between motion patterns (e.g., "tray height adjustment + front positioning" → "side-positioning stop + angle adjustment"), and adaptive processing is performed to match the drive control of its own robot (a destination robot different from the source robot). In other words, (in transfer learning between different types of robots), the trained model has an adaptive unit that converts the intermediate representation format to suit the drive mechanism of its own robot, and transfer learning is performed using the information converted by this adaptive unit.

[0057] The robot information processing unit 102 then receives, via the communication unit 101, at least the drive control information from the robot 50, which includes drive control information indicating the results of drive control performed by the robot 50, and drive instruction information specifying the content of the drive control based on the AI ​​passport.

[0058] The robot information processing unit 102 then stores this information in the memory unit and transmits this drive control information to the AI ​​passport update unit 108 to instruct it to perform the AI ​​passport update process. The AI ​​passport update unit 108 performs an update process to update the user's AI passport stored in the memory unit 104 using the drive control information. Specifically, it performs a process to record this drive control information in addition to the previous drive control information. As a result, the AI ​​passport will have all past drive control information recorded in it.

[0059] With this configuration, the user terminal 10 (information processing device) can receive predetermined services based on the AI ​​Passport from the robot 50. Furthermore, this configuration enables the robot 50 to provide user-specific services by using the trained model and training data to create user-inputted instruction information and driving instruction information for the AI ​​Passport.

[0060] Next, we will explain the drive control in robot 50 using Figure 4. Figure 4 is a functional block diagram showing the functions of the robot that constitutes the information processing system in an embodiment of the present invention.

[0061] In Figure 4, the robot 50 comprises a main control unit 500, a user interface 510, a sensor unit 520, a position control unit 530, a drive control unit 540, a communication unit 570, and a storage unit 580. The control unit 120 includes an environment recognition unit 121, an action planning unit 122, a safety monitoring unit 123, and an action execution unit 124. The definitions and processing contents of each component are described below.

[0062] The user interface 510 comprises a display screen, an operation panel, voice guidance, an input device, and an imaging device, and interacts with the user. The display screen is an output device that displays the robot's status and service menu, enabling visual interaction with the user. The control panel consists of a touchscreen and physical buttons, receiving user input and transmitting it to the processing system as electrical signals. The voice guide consists of a speaker and voice software, providing information via voice. The input device can substitute for the control panel and enables voice input. The imaging device consists of a camera and image processing unit, capable of capturing images of the robot 50's surrounding environment, people, objects, etc., and acquiring them as image data.

[0063] The sensor unit 520 is a sensor that measures (senss) the internal state of the robot and the external environment, and includes external information sensors, distance measurement sensors, state detection sensors, etc. Of course, it may also include position / attitude sensors that detect the robot's position and orientation, torque sensors, pressure sensors, capacitance sensors, etc. The sensor unit 520 collects external data related to the robot's physical state and surrounding environment and outputs it to the main control unit 500. Furthermore, external information sensors detect information about the surrounding environment as external information, and collect external information using, for example, infrared sensors and sound sensors. Distance measurement sensors measure the distance to an object and also perform spatial recognition in which the robot 50 operates. In addition, state detection sensors are sensors that detect the state of the robot 50, and perform functions such as measuring 3-axis acceleration and detecting abnormal depths.

[0064] The position control unit 530 in a mobile robot comprises a SLAM processing unit, a position designation unit, and a map management unit, and performs processing to control movement to the location where the service will be provided. For example, the SLAM processing unit (Simultaneous Localization and Mapping) constructs map data of the external environment based on data from various sensors and cameras. The position designation unit then calculates the delivery route to the target location (delivery location, delivery source location). The map management unit stores the constructed external environment information and map data.

[0065] The drive control unit 540 performs drive control processing on the drive mechanism based on the drive signal (drive instruction information) received from the main control unit 500 and the information detected by the sensor unit 520. This drive mechanism can include robot joints, servo motors, DC motors, and actuators. The drive control in this drive control unit 540 enables the robot 50 to provide a predetermined service to the user.

[0066] The communication unit 570 controls internal robot communication using communication protocols such as ROS (Robot Operating System), as well as external communication control. This communication unit 140 can receive information from the external server 30, receive instruction information and AI passport information from the user terminal 10, and transmit it to the main control unit 500. The communication unit 570 can also transmit information to the user terminal 10, the external server 30, and other locations.

[0067] The main control unit 500 performs robot control processing based on the information received from the communication unit 570 and the information received from the sensor unit 520. The detailed flow of the information processing program (information processing method) executed by the information processing system as this robot control processing is shown in the flowchart of Figure 16. When the main control unit 500 receives user instruction information, it processes the user instruction information, along with the AI ​​passport stored in the memory unit 580, for input to the trained model. Then, it transmits the drive instruction information resulting from the processing in the trained model (described later) to the drive control unit 540. As described above, the drive control unit 540 performs drive control based on this drive instruction information.

[0068] The memory unit 580 also functions as a temporary memory area, storing control programs (robot control programs), parameters, map information, and other data used for robot control. Furthermore, the memory unit 580 stores trained models (such as machine learning models like neural networks).

[0069] This trained model is based on deep learning, and outputs drive instruction information, which is the target variable of the drive control unit 540, by generating information using training data. In order to output this target variable, the trained model accepts user instructions for the robot 50 and input from the AI ​​Passport as explanatory variables, which are input information to the trained model. Furthermore, it is possible to use features obtained by processing data acquired from the sensor unit 520 or data acquired from an external system via the communication unit 140, in addition to the training data, to perform information generation processing on the trained model. In the training process for this pre-trained model, the weight parameters of the neural network are updated using training data that associates drive instruction information with drive control information. Specifically, a result signal indicating the success or failure of drive control is set, and the pre-trained model is optimized (enhanced) using a reinforcement learning algorithm to generate more effective drive instruction information.

[0070] More specifically, the trained model is configured as a deep neural network. This deep neural network has a hierarchical structure consisting of an input layer that accepts features extracted from the AI ​​Passport (vectorized data of user attribute information) and external information obtained from the sensor unit, an encoder layer that maps the input information to a high-dimensional feature space and extracts the user's latent preference patterns, a context integration layer that comprehensively processes the user's past usage history, the current environmental state, and the robot's functional constraints, a decoder layer that generates specific drive parameters (velocity, acceleration, trajectory, etc.) from the integrated information, and an output layer that outputs control signals to each actuator as drive instruction information.

[0071] This section explains the processing flow in a pre-trained model. First, the sensor unit 520 acquires external information about the robot's surroundings and inputs it into the trained model as explanatory variables via the communication unit 570 from the user terminal 10 or external server 30. At this time, the trained model integrates the sensor data and explanatory variables to generate features. Of course, this feature generation process may also be performed in the main control unit 500. Then, the trained model uses the training data to perform information generation processing through deep learning, processes these features, and outputs the target variable as described above.

[0072] Furthermore, the robot 50 controls the drive mechanism based on the drive instruction information, which is the objective variable, to provide a predetermined service. Of the drive control information, which is the result of the processing in this drive control, and the drive instruction information, at least the drive control information is transmitted to the user terminal 10 via the communication unit 570. Then, the user terminal 10 adds (updates) the drive control information, or at least the drive instruction information, to the AI ​​Passport. In addition to this drive control information, the drive instruction information can be used as new training data. Based on the above, by providing (transmitting) the AI ​​Passport to Robot 50, the trained model will use the training data contained in this AI Passport, which will contribute to improving the accuracy of subsequent motion control and drive control according to the user's attributes.

[0073] Figure 5 illustrates an example of a hardware configuration when the user terminal 10, external server 30, and management device 60 are each implemented using computers. Of course, it is also possible to implement the functions of each device using multiple devices.

[0074] Furthermore, since the user terminal 10, external server 30, and management device 60 perform various types of information transmission, processing, and storage, they can each be described as "information processing devices" or "information processing terminals." As shown in Figure 4, the user terminal 10, external server 30, and management device 60 are each equipped with a processor 901, a storage device 902, an I / F 903, and a display device 904.

[0075] The processor 901 controls processing in the computer (user terminal 10, external server 30, management device 60) by executing programs stored in the storage device 902. For example, each functional unit and processing unit of the user terminal 10, external server 30, and management device 60 are realized by the processor 901 executing programs stored in the storage device 902. These programs are executed by the computer (user terminal 10, external server 30, management device 60).

[0076] The storage device 902 temporarily stores the source code of programs executed by the processor 901 and data necessary for program execution. It is a volatile or non-volatile storage medium such as RAM, ROM, a hard disk drive (HDD), or flash memory. This storage device 902 also serves as a database, storing the operating system (OS), various programs, configuration information, tables, etc., necessary to implement the above configurations. The processor 901 reads and executes these programs, configuration information, table information, etc., to realize various functions.

[0077] The I / F903 is an input / output interface (also called the "communication unit") for receiving user input from keyboards, mice, touch panels, various sensors, wearable devices, etc. It is an interface for inputting and outputting data from external devices to a computer, and a communication interface for communicating data between computers or with external devices to a computer via a communication network, either wired or wirelessly.

[0078] The display device 904 is a device that displays various types of information. Specific examples include liquid crystal displays, organic EL (Electro-Lumince) displays, and it may also be a wearable device. It may also be a touch panel or touch input device that combines an input / output interface with this display device 904.

[0079] Figure 7 is a table diagram showing an example of a robot ID management table used in an information processing system according to an embodiment of the present invention.

[0080] The robot ID management table shown in Figure 7 is stored in the management device DB70 and associates robot ID, location information (equipment information), robot type, and validity period with each other. The robot ID indicates that the robot is located in the place, facility, company, group, organization, etc., as shown in the location information (equipment information). The type of robot indicated by this robot ID is shown in the robot type section, and the period during which this robot can be used is shown in the validity period section. This ensures that robot IDs are objectively managed by an external organization, guaranteeing their uniqueness.

[0081] Figure 8 is a table diagram showing an example of a service usage status management table used in an information processing system according to an embodiment of the present invention.

[0082] The service usage management table shown in Figure 8 is stored in the management device DB70 and the external DB40, and associates service information, which is information about the services provided by the robot, user IDs that identify the users, robot IDs that are the target of the service, and robot drive status information. This service usage management table is provided for each user and displays robot usage history information, such as robots that the user has used in the past and robots that they are currently using. This allows the external server 30 to understand the status of the robot and its users. Furthermore, it becomes possible to understand the services that the robot has processed.

[0083] Figure 9 is a table diagram showing an example of a robot management table used in an information processing system according to an embodiment of the present invention.

[0084] The robot management table shown in Figure 9 is stored in the management device DB70 and the external DB40, and associates robot ID, location information history ID, maintenance history ID, and permission information ID with each other. For robots identified by a robot ID, the movement status is shown in the movement information history table, indicated by the location information history ID; the maintenance history is shown in the maintenance history table, indicated by the maintenance history ID; and the permitted service information is shown in the permitted information table, indicated by the permitted information ID. This makes it possible to manage various historical and configuration information for the robot.

[0085] Figure 10 is a table diagram showing an example of a location information management table used in an information processing system according to an embodiment of the present invention.

[0086] The location information management table shown in Figure 10 is the information referenced by the location information history ID in the robot management table shown in Figure 9. One or more location information entries are associated with each location information history ID in chronological order. This allows us to understand the movement path of each robot and easily grasp its work status (operating status).

[0087] Figure 11 is a table diagram showing an example of a maintenance history table used in an information processing system according to an embodiment of the present invention.

[0088] The maintenance history table shown in Figure 11 is the information referenced by the maintenance history ID in the robot management table shown in Figure 9. One or more maintenance history entries are associated with each maintenance history ID as history information in chronological order. This allows us to access maintenance history information for robots, which can then be used for appropriate robot operation and selection based on working time.

[0089] Figure 12 is a table diagram showing an example of a permission information table used in an information processing system according to an embodiment of the present invention.

[0090] The authorization information table shown in Figure 12 is the information referenced by the authorization information ID in the robot management table shown in Figure 9, and one or more authorization information entries are associated with each authorization information ID. This authorization information consists of information that specifies the users who are permitted to provide services that the robot can perform, and information that specifies the services (operation, driving, work) that the robot is permitted to perform. This makes it possible to appropriately control whether or not robots provide services.

[0091] Figures 13 and 14 show examples of screens displayed on a user terminal that constitutes an information processing system according to an embodiment of the present invention. Figure 13 shows the screen transitions in a process that uses location information from a user terminal 10 to search for robots belonging to that location and then allows the user to select one or more robots from the searched robots. On the other hand, Figure 14 shows the screen transitions in the process of searching for a robot based on the services provided by the robot, and then selecting one or more robots from the searched robots.

[0092] In Figure 13, Figure 13(a1) is the initial screen of the robot search screen, which indicates that the robot will be searched for based on location information. When the "OK" button is pressed on this screen, the screen shown in Figure 13(a2) is displayed, which indicates that location information such as a facility or store where a robot is installed has been detected based on the current location information of the user terminal 10.

[0093] The screen shown in Figure 13(a2) illustrates an example where three locations are detected based on location information. Selecting one of the facilities at these locations will display either the screen shown in Figure 13(a3) or Figure 13(a4). Figure 13(a3) shows the screen displayed when a single service is provided by robot drive control at the selected facility. Without specifying the service, a list of robots capable of providing that service is displayed. If the user is a natural person, a list of robots for natural people is displayed; if the user is a legal entity, a list of robots labeled "for legal entities" is displayed. While Figure 13(a3) shows both robot lists simultaneously, displaying only one of them is sufficient.

[0094] Figure 13(a4) is a screen accessed from Figure 13(a2), where, if the robot located at the searched location can provide multiple services, the user can specify which service to receive. Figure 13(a4) shows an example where "Task B" is selected as the service to receive from the robot. Figure 13(a5) shows a list of robots capable of performing task B, given that task B is selected in Figure 13(a4).

[0095] In this way, when Figure 13(a3) or Figure 13(a5) is displayed, if one or more robots are selected by the user, the screen shown in Figure 13(a6) will be displayed. The screen shown in Figure 13(a6) is a screen that requests permission from the user terminal 10 to provide the AI ​​Passport to the selected robot. If multiple robots are specified, the screen will display all of the specified robots. In the screen shown in Figure 13(a6), when the user performs an operation related to permission, the user terminal 10 and the robot 50 perform a cooperative process that establishes a communication channel and enables the transmission and reception of data.

[0096] Then, a screen for operating the robot, as shown in Figure 13(a7), and a screen for specifying the services provided by the robot, as shown in Figure 13(a8), are displayed.

[0097] Next, in Figure 14, the screen shown in Figure 14(b1) is where the user specifies the service they wish to receive from the robot, and a service menu is displayed. When the user selects a service using the screen shown in Figure 14(b1), a screen for specifying the service category, as shown in Figure 14(b2), is displayed. As shown in Figure 14(b2), when a service category is specified, the following screen, shown in Figure 14(b3), displays facilities and stores that match that category on a map based on the current location information of the user terminal 10. In this map, as an example, facilities within a radius of 500m are displayed based on the current location information.

[0098] Next, on the screen shown in Figure 14(b3), once a user has specified a facility, the user terminal 10 sends the facility ID, the terminal ID of the user terminal 10, or the user ID (user ID) to the external server 30, and displays a list of robots received from the external server 30. Figure 14(b4) shows a list of robots that the external server 30 has searched for and output as a result of the search for available robots for the user ID.

[0099] Then, when the user selects the robot detected in Figure 14(b4), the system checks the communication status between the user terminal 10 and the robot 50. If communication is possible, the user terminal 10 and the robot 50 establish a communication path and cooperate with each other. Then, in Figure 14(b5), the system confirms whether to register the linked robot 50 in the AI ​​Passport. If registration is selected, the linked robot 50 is added to the AI ​​Passport (added to the record). Subsequently, the screen shown in Figure 14(b6) confirms that the learning data described above will be registered in the AI ​​Passport as a result of learning by the specified robot 50, and a screen like the one shown in Figure 14(b7) displays information about the services provided by the linked robot.

[0100] Figure 15 is a sequence diagram showing the processing flow in an information processing system according to an embodiment of the present invention. Figure 15 shows the processing flow in the user terminal 10, the external server 30, and the robot 50. This Figure 15 shows the detailed flow in the information processing program executed in the information processing system.

[0101] In Figure 15, when a user operates the screen displayed on the user terminal 10 (for example, the screen shown in Figure 13(a1)) to search for a robot, the user terminal 10 receives the current location information with a GNSS receiver and transmits location information based on that location information (information about the location (facility) where the robot will be used) and instruction information for the robot search to the external server 30 (S1501). The instruction information at this time may also include information about the service to be used, as specified on the screen shown in Figure 13(a4). When the external server 30 receives this information from the user terminal 10, it refers to a robot management table as shown in Figure 9 (more specifically, a permission information table as shown in Figure 12) to search for robots 50 that the user can use in the location information (facility) based on the location information, and responds to the user terminal 10 (1502). The robot information (robot ID) at the location information (facility) based on the location information at this time is the information that the external server 30 queries the management device 60 for, and the management device 60 replies by referring to the robot ID management table shown in Figure 7.

[0102] Then, the user terminal 10 displays a list of robots received from the external server 30 (S1503). When the user selects one or more robots and authorizes the provision of AI Passport information to those robots (S1504), the user terminal 10 sends the selected robot ID, the terminal ID of the user terminal 10 (user ID), and the PassID that identifies the AI ​​Passport to the external server 30.

[0103] As a result, the external server 30 sends a notification to the target robot to begin use based on the robot ID (S1505). This notification includes the terminal ID (user ID). The example shown in Figure 15 illustrates how, after three robots were specified, a notification of commencement of use was sent to these three robots (Robot 1 (Robot ID: RB_1), Robot 2 (Robot ID: RB_2), and Robot 3 (Robot ID: RB_3)).

[0104] In the following example, for the sake of explanation, we will show an example of how robot 2 (robot ID: RB_2) controls its actions in response to the notification of service commencement. Robot 2 receives the terminal ID and PassID from the external server 30 and registers the terminal ID and PassID (S1506-1). As a result, robot 2 enters a state of readiness, waiting for notifications from user terminal 10 based on the received terminal ID (the setting is changed from the initial state to the readiness state). This prevents the robot from being misused or experiencing excessive communication due to notifications from an unspecified number of user terminals 10, thus enabling the robot to operate safely.

[0105] Next, the user terminal 10 determines whether the conditions for performing the collaboration process are met, and if these conditions are met, the collaboration process is performed (S1506-2). More specifically, when the user terminal 10 is within a certain distance of the designated robot 2 (a distance less than or equal to the communication distance), the collaboration process is performed between the user terminal 10 and the robot 2 (S1506-2). In this collaboration process, the user terminal 10 sends its terminal ID to the robot 2, and the robot 2 compares this received terminal ID with the terminal ID received in the user activation notification to confirm that they are the same. Once they are confirmed to be the same, a communication channel is established between the user terminal 10 and the robot 2. After establishing a communication channel with the user terminal, the robot 2 notifies the external server 30 of this. Upon receiving this notification, the external server 30 sends a notification to other robots (in this example, robots 1 and 3) that have sent the same user activation notification (identification information that identifies the user activation notification) as robot 2 to cancel their ready state. Upon receiving this cancellation notification, the robot cancels its set ready state and changes (sets) to a state (initial state) where it can receive other user activation notifications.

[0106] Next, robot 2 requests an AI passport from the user terminal 10 (S1507), and the user terminal 10, upon receiving this request, determines within a certain time whether or not it has accepted the request for the AI ​​passport (S1508). If an AI passport request is received within a certain time, the user terminal 10 performs an AI passport update process according to the robot type of the robot 2 (S1509). This AI passport update process is not mandatory, but is performed when it is necessary (more effective) to provide special information depending on the robot type. For example, in addition to the AI ​​passport shown in Figure 2, it is possible to add other necessary information.

[0107] Then, the user terminal 10 decrypts the AI ​​passport and sends it to the requesting robot (robot 2) (S1510). The robot then receives the AI ​​passport from the user terminal 10 and checks whether it has the same PassID as the registered PassID through the above process. If the PassIDs are the same, the robot stores (registers) the AI ​​passport in its memory. Of course, if the PassIDs are different, the communication channel may be cut off to prevent misuse. Declassification refers to the process of making the AI ​​Passport, which is stored in a specific memory area of ​​the user terminal 10, accessible only when certain conditions (for example, when a request for the AI ​​Passport has been received from a robot) are met.

[0108] (Real-time knowledge updates in robots) Multiple robots can update their training data each time they perform drive control and share this updated information in real time. Specifically, they extract only the difference data from the updated training data that has changed since the last sharing, convert this difference data into an intermediate representation format, and share it with other robots.

[0109] The second robot's trained model undergoes continuous transfer learning using the received differential data. This continuous transfer learning is achieved through an online learning method that uses existing trained parameters as initial values ​​and performs additional learning using the differential data. This allows for the real-time sharing of the latest insights from each robot in environments where multiple robots are operating in parallel, continuously improving the overall system performance.

[0110] Furthermore, the user terminal 10 sends the terminal ID and the robot ID of the robot that sent the AI ​​Passport to the external server 30, and the external server 30 updates the "Service Usage Status Management Table" (S1512).

[0111] Next, robot 2, which has registered the AI ​​passport, uses the initial service menu based on this AI passport to create a service menu that corresponds to the contents of the AI ​​passport and sends it to the user terminal 10 (S1513). This service menu creation process is not mandatory, but is performed when it is necessary (more effective) to provide a special service menu.

[0112] The user terminal 10 receives and displays the service menu (S1514), and based on this service menu, sends instruction information (user instruction information) regarding the desired service specified by the user to the robot to request the service (S1515).

[0113] When a robot receives instruction information regarding a desired service (user instruction information), it performs robot control processing (also called "drive control processing") (S1516). Details of this robot control processing are shown in Figure 16. In this robot control process, a trained model generates drive instruction information that instructs the drive mechanism to be controlled using external information detected by sensors installed on the robot, based on the AI ​​Passport. Based on this drive instruction information, the drive mechanism is controlled using the external information detected by the sensors. Furthermore, this trained model generates drive instruction information using training data consisting of past drive instruction information and drive control information obtained from the drive control process.

[0114] Then, the drive control information and drive instruction information, which are the results of the drive control processing performed by this robot control process (S1516), are transmitted to the user terminal 10 as learning data (S1518).

[0115] The user terminal 10 adds this learning data to the AI ​​passport (updates the AI ​​passport) (S1519). Then, upon completion of the requested service, the user terminal 10 sends a command to robot 2 to delete the AI ​​passport (S1520). More specifically, it instructs the deletion of the AI ​​passport identified by the PassID (sends a deletion command). Robot 2 performs the deletion process of the AI ​​passport identified by the PassID and sends a message to the user terminal 10 indicating the completion of the deletion (S1521). The robot also deletes the PassID and disconnects the communication channel (S1522). This ensures that the AI ​​passport is provided to the robot immediately before service use (specifically during processing S1510) and deleted immediately upon completion of the service, minimizing the possibility of information leakage from the AI ​​passport. Furthermore, this will lead to more effective use of the memory area used to store the AI ​​passport in robots.

[0116] The robot then transmits information about its usage status and its movement path to the external server 30, which then updates the service usage status management table and the robot management table (more specifically, the location information history table) (S1523).

[0117] Through the processes described above, the information processing system provides the AI ​​Passport from the user terminal 10 to the robot, and the robot's trained model generates drive instruction information that drives and controls the robot's drive mechanism based on this AI Passport. The robot then drives and controls the drive mechanism based on this drive instruction information, thereby enabling it to provide services based on the AI ​​Passport. The results of this drive control are output as drive control information, and this drive control information, along with the drive instruction information, is repeatedly used as training data for the trained model to learn (deep learning). This process allows the robot to provide the most suitable service for each user based on the AI ​​Passport.

[0118] Figure 16 is a diagram showing the detailed flow of robot control processing performed by the robot, which is the target device in an embodiment of the present invention. In Figure 16, the target device, a robot, receives a request (service request) for processing (services) to be implemented (executed) by the robot from a user terminal, which is an information processing device (S1601). As described above, robots acquire external information using sensors, etc., and then the drive control unit uses that external information to drive the drive mechanism (functional part) to provide a predetermined service. In other words, robots perform predetermined actions by driving the drive mechanism.

[0119] Next, this robot performs pre-request processing in response to the request (S1602). This pre-request processing is as follows: As a pre-processing step for requests, the robot's environment recognition unit (not shown) processes sensor information and performs recognition of the environment and objects based on pre-processing instructions from the main processing unit. Specifically, it uses techniques such as image processing, point cloud processing, and signal processing to perform object detection and tracking, environmental map generation, and situational understanding. The environment recognition unit 121 is responsible for converting raw data input from sensors into meaningful information. Next, as a pre-processing step for the request, the robot's motion planning unit (not shown) performs a process based on the recognition results of the environment recognition unit, following instructions from the main processing unit. Specifically, this includes trajectory planning (calculating the robot's movement path and joint trajectories), task decomposition (breaking down complex tasks into a sequence of basic movements), and scheduling (optimizing the execution order of multiple tasks). The motion planning unit is responsible for determining "what to do" and "how to do it." This process includes inverse kinematics calculations, pathfinding algorithms, and optimization calculations. Furthermore, as a pre-processing step for requests, the robot's safety monitoring unit (not shown) performs a process to monitor the robot's safety during operation, based on pre-processing instructions from the main processing unit. Specifically, it performs collision detection (prediction of contact with obstacles or people), anomaly detection (checking the acceptable range of operation parameters), and emergency response (dealing with dangerous situations). The safety monitoring unit constantly monitors information from sensors and the plan contents of the operation planning unit, and if a safety problem is detected, it instructs the operation execution unit to immediately correct or stop the operation. This process is executed with the highest priority in parallel with other processes.

[0120] Once this pre-processing for the request is complete, the robot determines whether the pre-processing is complete (S1603). Until it is complete (NO in S1603), it waits for the pre-processing to finish. Once the pre-processing is complete (YES in S1603), it performs the process of generating drive instruction information (S1604).

[0121] This drive instruction information is information generated by a trained model to fulfill the received request, and it is the target variable that instructs the drive control unit to perform drive control using external information detected by the sensor, with the AI ​​passport (user authentication information) as the explanatory variable.

[0122] Next, the robot performs pre-drive processing before driving the drive mechanism (drive unit) (S1605). This pre-drive processing includes adjusting the initial position of the drive mechanism (drive unit), starting the actuator, measuring the robot's initial position information, selecting the movement (delivery) route, and confirming the movement priority through communication with other robots.

[0123] Then, once the pre-drive processing is complete, the robot performs drive control processing based on the drive instruction information and by referring to the result information of the pre-drive processing to drive the drive mechanism (S1606). Through this drive control processing, the predetermined service in response to the request is provided. For example, in the case of industrial robots, the process of "sensor input → position detection → trajectory calculation → motor control → execution of motion" represents the provision of an electronic component assembly service. Similarly, in the case of mobile robots, the process of "environmental recognition → self-position estimation → path planning → motion determination → travel control → obstacle avoidance" represents the provision of a service that moves an object from one predetermined location to another.

[0124] The robot then determines whether this drive control process is complete (S1607), and continues the drive control process until it is complete (NO in S1607). Once this drive control process is complete (YES in S1607), it then determines whether the entire request is complete (S1608).

[0125] If the entire request is completed (YES in S1608), the robot creates drive control information (S1609) as a result of controlling the drive mechanisms. This drive control information shows the specific drive details for each drive mechanism and indicates the drive details in the service provided by the robot.

[0126] Next, the robot performs post-drive processing to return to its pre-drive state (S1610). In other words, post-drive processing is the process of returning to the state before the pre-drive processing described above was performed. This post-drive processing performs the reverse processing of the pre-drive processing, and includes adjusting the drive unit position, starting the actuator, setting the arrival position, selecting the movement route, and confirming the movement priority.

[0127] Once this post-drive processing is complete, the robot then stores the AI ​​passport used to create the drive instruction information used in the drive control processing, and the robot control results (drive control result information) that show the drive results performed by the drive control processing, in association with each other (S1611). By having the robot perform the robot control processing described above, users will be able to receive services provided by the robot based on their AI passport.

[0128] The above describes a control device and control method applicable to various types of robots as embodiments of the present invention. By using the control architecture of the present invention, optimal control according to the type of robot becomes possible, improving environmental awareness, safety, cooperation, and adaptability. Furthermore, it is expected that the sharing of technical knowledge among different robot types will be promoted, contributing to the overall development of robot technology.

[0129] As used in the embodiments and claims described above, the terms “part,” “means,” “apparatus,” and “system” do not merely refer to physical means, but also include cases where the functions of these are realized by software or software services.

[0130] <Note 1> The information processing system has the following technical characteristics: An information processing device comprising a storage unit and a processor that performs information processing based on the information stored in the storage unit, wherein the storage unit stores user authentication information that includes user attribute information and is used to verify the user, and the processor comprises a transmission unit that transmits the user authentication information, The present invention comprises a plurality of robots each having a robot memory unit that stores at least robot identification information, a robot communication unit, a sensor capable of detecting external information, a drive mechanism including an actuator, a trained model that generates drive instruction information that instructs the drive mechanism to be driven using the external information detected by the sensor based on the user authentication information, and a drive control unit that performs drive control of the drive mechanism using the external information detected by the sensor based on the drive instruction information, wherein the robot memory unit of the first robot among the plurality of robots stores the drive instruction information and the drive control information obtained by the drive control unit driving the drive mechanism as learning data in association, and the The trained model of robot 1 has a conversion unit that converts the correspondence between the drive instruction information and the drive control information into an intermediate representation format that does not depend on the physical characteristics specific to the robot. The trained model of a second robot among the plurality of robots has an adaptation unit that receives the intermediate representation format from the first robot via the robot communication unit and converts the intermediate representation format to suit the drive mechanism of its own robot. The trained model of the second robot learns drive control in the second robot using the training data generated by the first robot by performing transfer learning using the information converted by the adaptation unit.

[0131] <Note 2> The information processing system, with respect to <Appendix 1>, further possesses the following technical characteristics in transfer learning. If the first robot and the second robot are of the same robot type, the trained model of the second robot is characterized by reusing the trained parameters of the trained model of the first robot. <Note 3> The information processing system has the following technical characteristics in relation to <Appendix 1> or <Appendix 2>, and further in transfer learning. Furthermore, if the first robot and the second robot are of different robot types, the adaptation unit is characterized by using an equivalence mapping table between motion patterns to convert the intermediate representation format into an alternative motion pattern that is compatible with the drive mechanism of the second robot.

[0132] <Note 4> The information processing system has the following technical features related to one of the following: <Appendix 1>, <Appendix 2>, or <Appendix 3>, and also has a function for sharing learning data. The system further comprises an external server capable of communicating with the plurality of robots, the external server having an external database for storing the learning data collected from the plurality of robots, the first robot converts the learning data into the intermediate representation format and then transmits it to the external server, the external server stores the intermediate representation format in the external database in association with the robot identification information, the second robot requests the external server for the intermediate representation format of other robots having a drive mechanism similar to its own robot type, and the external server selects and provides to the second robot the intermediate representation format that can be transferred and learned based on the robot type of the second robot.

[0133] <Note 5> The robot has the following technical characteristics: The robot comprises: a robot communication unit that receives user authentication information, including user attribute information, used to verify the user, from an information processing device; a robot memory unit that stores at least robot identification information; a sensor capable of detecting external information; a drive mechanism including an actuator; a trained model that generates drive instruction information that instructs the drive mechanism to be driven using the external information detected by the sensor, based on the user authentication information; and a drive control unit that performs drive control of the drive mechanism using the external information detected by the sensor, based on the drive instruction information, wherein the robot memory unit stores the drive instruction information and drive control information obtained by the drive control unit in the process of driving the drive mechanism. The system stores the information in association with the training data, and the trained model has a conversion unit that converts the correspondence between the drive instruction information and the drive control information into an intermediate representation format that does not depend on the physical characteristics specific to the robot, and an adaptation unit that converts the intermediate representation format received from another robot to suit its own drive mechanism, and the robot communication unit transmits the intermediate representation format generated by the conversion unit to another robot and receives the intermediate representation format from the other robot, and the trained model learns its own drive control using the training data generated by the other robot by performing transfer learning using the information converted by the adaptation unit.

[0134] Furthermore, the functions of a single "part," "means," "apparatus," or "system" may not only be realized by a single physical means, software, software module, or apparatus, but may also be realized by multiple physical means, software, software modules, apparatus, or combinations thereof.

[0135] The terms used in the embodiments and claims described above should be interpreted as non-limiting terms. For example, the term "includes" should be interpreted as "not limited to those described as including." The term "contains" should be interpreted as "not limited to those described as containing." The term "equips" should be interpreted as "not limited to those described as equipped." The term "possesses" should be interpreted as "not limited to those described as possessing." The term "complements" should be interpreted as "not limited to those described as possessing." [Industrial applicability]

[0136] This invention is applicable to robots in various fields, including industrial robots, mobile robots, collaborative robots, service robots, medical and welfare robots, and agricultural and construction robots, and can be used in a wide range of industrial sectors such as manufacturing, logistics, service industries, medical and welfare fields, agriculture, and construction. [Explanation of symbols]

[0137] 10. User terminals 20 AI Passport 30 External Servers 40 External DB 50 Robots (Target Equipment) 60 Management device 70 Management device DB

Claims

1. Memory unit and, A processor that performs information processing based on the information stored in the aforementioned memory unit. It is equipped with, The aforementioned storage unit is An information processing device that stores user authentication information, which includes user attribute information and is used to authenticate the user, A robot memory unit that stores at least robot identification information, Robot communications department, A sensor capable of detecting external information, A drive mechanism including an actuator, A trained model that generates drive instruction information that instructs the drive control of the drive mechanism using external information detected by the sensor, based on the user authentication information, Based on the aforementioned drive instruction information, a drive control unit performs drive control of the drive mechanism using external information detected by the sensor. A robot equipped with Equipped with, The aforementioned attribute information is, This includes normal attribute information including the user's basic settings information, and special attribute information including the user's physical information, The robot memory unit further, The drive instruction information and the drive control information obtained by the drive control unit in controlling the drive mechanism are stored as learning data. The aforementioned trained model is An information processing system that uses user authentication information including the attribute information as input information to the trained model, and uses the trained data to output drive instruction information including drive information for controlling the drive mechanism in the drive control unit.

2. The aforementioned trained model is a neural network, The information processing system according to claim 1, comprising an input layer that receives feature quantities extracted from the user authentication information and external information acquired from the sensor, and an output layer that outputs a control signal to the actuator as drive instruction information.

3. A storage unit that stores user attribute information and user authentication information used to verify the user, A processor that performs information processing based on the information stored in the aforementioned storage unit. A robot communication unit that receives the user authentication information from an information processing device equipped with the following: A robot memory unit that stores at least robot identification information, A sensor capable of detecting external information, A drive mechanism including an actuator, A trained model that generates drive instruction information that instructs drive control in the drive mechanism using external information detected by the sensor, based on the user authentication information, Based on the aforementioned drive instruction information, a drive control unit performs drive control of the drive mechanism using external information detected by the sensor. It is equipped with, The aforementioned attribute information is, This includes normal attribute information including the user's basic settings information, and special attribute information including the user's physical information, The robot memory unit further, The drive instruction information and the drive control information obtained by the drive control unit in controlling the drive mechanism are stored as learning data. The aforementioned trained model is A robot in which user authentication information including the attribute information is used as input information to the trained model, and drive instruction information including drive information used by the drive control unit to control the drive mechanism using the trained data is used as output information.

4. A robot communication unit that receives user authentication information from an information processing device comprising a storage unit that stores user authentication information including user attribute information and used to verify the user, and a processor that performs information processing based on the information stored in the storage unit, A robot memory unit that stores at least robot identification information, A sensor capable of detecting external information, A drive mechanism including an actuator, A trained model that generates drive instruction information that instructs drive control in the drive mechanism using external information detected by the sensor, based on the user authentication information, Based on the aforementioned drive instruction information, a drive control unit performs drive control of the drive mechanism using external information detected by the sensor. A robot control method performed by a robot computer equipped with the following: The aforementioned attribute information is, This includes normal attribute information including the user's basic settings information, and special attribute information including the user's physical information, The robot memory unit is The drive instruction information and the drive control information obtained by the drive control unit in controlling the drive mechanism are stored as learning data. The aforementioned trained model is A robot control method comprising using user authentication information including the attribute information as input information for the trained model, and outputting drive instruction information including drive information for controlling the drive mechanism in the drive control unit using the trained data.

5. A robot communication unit that receives user authentication information from an information processing device comprising a storage unit that stores user authentication information including user attribute information and used to verify the user, and a processor that performs information processing based on the information stored in the storage unit, A robot memory unit that stores at least robot identification information, A sensor capable of detecting external information, A drive mechanism including an actuator, A trained model that generates drive instruction information that instructs drive control in the drive mechanism using external information detected by the sensor, based on the user authentication information, Based on the aforementioned drive instruction information, a drive control unit performs drive control of the drive mechanism using external information detected by the sensor. A robot control program to be executed by the computer of a robot equipped with the following: The aforementioned attribute information is, This includes normal attribute information including the user's basic settings information, and special attribute information including the user's physical information, The robot memory unit is The drive instruction information and the drive control information obtained by the drive control unit in controlling the drive mechanism are stored as learning data. The aforementioned trained model is A robot control program that uses user authentication information including the attribute information as input information for the trained model, and output information including drive information for controlling the drive mechanism in the drive control unit using the trained data.

Images