GENERATION DEVICE, ROBOT, GENERATION METHOD, AND PROGRAM

Abstract

To provide a device capable of easily generating the program for controlling a robot motion.SOLUTION: A generation device acquires information from a user, generates input information to a generation module based on the robot information related to a robot and on the acquired information, and acquires the program generated by inputting input information into the generation module and for controlling the robot motions. The robot information includes information about at least one of the robot operation environment or the software component to acquire the operation environment information.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a generation device, a robot, a generation method, and a program. [Background technology]

[0002] There are known robots that can recognize external situations through sensors and communications and execute actions accordingly. Some of these robots operate according to pre-created programs. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor, "ChatGPT for Robotics: Design Principles and Model Abilities", [online], [Retrieved August 28, 2020], Internet<URL: https: / / www.microsoft.com / en-us / research / uploads / prod / 2023 / 02 / ChatGPT___Robotics.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an apparatus that allows for easy creation of a program for controlling the operation of a robot. [Means for solving the problem]

[0005] A generation device according to an embodiment of the present disclosure includes at least one memory and at least one processor, and the at least one processor acquires information from a user, generates input information for a generation module based on robot information related to a robot and the acquired information, and acquires a program for controlling the operation of the robot, the program being generated by inputting the input information to the generation module. The robot information is ,B Bot operating environment About Software components for obtaining information To Includes information on The generative module is a generative model. . [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing an example of an overall configuration of a control system. [Diagram 2] FIG. 1 is a diagram illustrating an example of an external configuration of an autonomous vehicle. [Diagram 3] FIG. 13 is a diagram showing the autonomous vehicle docking with a shelf that is the transportation target. [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of a generating device. [Diagram 5] 13 is a flowchart illustrating an example of a generation process. [Figure 6] 13 is a flowchart illustrating an example of an editing process. [Figure 7] 13 is a flowchart illustrating an example of an execution process. [Figure 8] FIG. 2 is a block diagram showing an example of a hardware configuration of a generating device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0008] [Embodiment] One embodiment of the present disclosure is a control system for controlling the movement of a robot. The control system creates a program for controlling the movement of the robot in order to realize an instruction from a user to the robot. The program for controlling the movement of the robot is stored in a storage unit such as a memory provided in the robot, and is executed by a control device provided in the robot in response to a predetermined trigger.

[0009] In this embodiment, the robot has, for example, a sensor and a movement mechanism. This robot has a function of recognizing an external situation through the sensor and communication, and performing an action according to the external situation. The robot may also have a function of actively moving or exercising in order to obtain information about its operating environment. For example, a robot operating in a user's residence can move to the vicinity of the entrance and point the camera's angle of view toward the door in order to detect that the user has returned home.

[0010] However, creating a program to control the operation of a robot requires specialized knowledge of software engineering, making it extremely difficult for users who are not familiar with software engineering to write a program that represents the robot's behavioral principles.

[0011] For example, simply lining up the actions expected of a robot in order is often easy, since it is equivalent to automatically pressing buttons in order to instruct the robot to perform the actions. On the other hand, planning and instructing actions that involve situational judgment and conditional branching in advance requires advanced knowledge and skills, such as knowledge about robots, programming skills, and the selection of information necessary to judge the situation. For example, when considering a robot that operates in the home, it is necessary to make it possible for even users without such specialized knowledge to easily create a program.

[0012] There are three main challenges to having a robot perform the actions expected by the user. The first challenge is to incorporate the expected situation into the robot's actions. The second challenge is to define the actions to be performed by the robot using tasks, conditional branching, etc. The third challenge is to implement tasks, conditional branching, etc., and exception handling in the program.

[0013] The following methods allow a user to give a new principle of behavior to a robot. The methods are listed in order of increasing difficulty for the user.

[0014] The first method is to write the program in a programming language, the second method is to create the program using simplified tools such as visual programming, and the third method is to write a flowchart that represents the desired operation.

[0015] The fourth method is to describe a sentence in natural language that specifies the desired action. For example, "Bring a towel to me in the bath. When I get out, bring me a beer from the fridge." The fifth method is to declare the desired situation in natural language. For example, "When I get out of the bath, I want to feel refreshed and drink a beer."

[0016] An embodiment of the present disclosure realizes creating a program for controlling the operation of a robot by the above fourth method and fifth method. Also, an embodiment of the present disclosure simultaneously realizes creating a program by the third method. In one aspect, according to this embodiment, even a user without specialized knowledge can easily create a program for controlling the operation of a robot.

[0017] <Overall configuration of the control system> The overall configuration of the control system in this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of the control system in this embodiment.

[0018] 1, a control system 1000 in this embodiment includes a terminal device 100, a robot 200, and a generating server 300. The terminal device 100, the robot 200, and the generating server 300 are connected to each other so as to be able to communicate data with each other via a communication network such as a LAN (Local Area Network) or the Internet.

[0019] The terminal device 100 is an information processing terminal such as a personal computer, a smartphone, or a tablet terminal operated by a user of the robot 200. The terminal device 100 accepts input of instruction information indicating an instruction given by the user to the robot, and transmits the instruction information to the robot 200. The terminal device 100 also displays to the user a programming screen for editing a generated program, and updates the program in response to the user's operation on the programming screen. The terminal device 100 may be equipped with a display device for displaying the programming screen.

[0020] The robot 200 is capable of operating autonomously according to a program. The robot 200 includes a generating device 210 and a control device 220.

[0021] Based on the instruction information, the generating device 210 transmits input information for generating a program to the generating server 300. The generating device 210 receives the program from the generating server 300 and stores it in a storage unit such as a memory.

[0022] The control device 220 reads out the program stored in the storage unit by the generation device 210, and executes the program in response to a predetermined trigger. The control device 220 may execute the program generated by the generation device 210 offline. The control device 220 may execute the program generated by the generation device 210 online.

[0023] Here, offline means that the program is generated in advance and is executed automatically in response to a predefined trigger, whereas online means that the program is generated and executed in real time each time a user gives an instruction.

[0024] Programs executed offline need to be created in advance assuming various conditions, which makes programming more difficult. The generation device 210 generates a program based on various information including information about the robot 200 itself, information about the operating environment of the robot 200, and information about the user, and therefore can create complex programs that can be executed offline.

[0025] The robot 200 is equipped with at least a sensor and a movement mechanism, and performs operations including movement according to an external situation recognized by the sensor or the like. In this embodiment, the robot 200 is, for example, an autonomous vehicle that further includes a docking mechanism and transports an object designated by a user to a designated location by docking with a transport object. For example, the transport object is a piece of furniture such as a shelf with casters.

[0026] As other examples, the robot 200 may be a household robot that performs household chores such as cleaning, a service robot that provides various services using an arm with multiple joints, a transport robot that transports luggage, equipment, etc. in a factory or warehouse, an autonomous vehicle that runs on public or private roads, a transport drone that flies while holding items, etc., etc.

[0027] An autonomous vehicle, which is an example of the robot 200, identifies a shelf on which an object designated by a user is placed and detects the position of the shelf. The autonomous vehicle also uses a docking mechanism to dock with the shelf and moves to a location designated by the user while docked. In this way, the autonomous vehicle realizes transporting an object designated by a user to the designated location.

[0028] The generating server 300 is an information processing device such as a personal computer, a workstation, or a server that generates a program in response to a request from the generating device 210. The generating server 300 includes a generating module 310. The generating server 300 receives input information from the generating device 210 and generates a program by inputting the input information to the generating module 310. The generating server 300 transmits the generated program to the generating device 210.

[0029] The generation module 310 is configured to generate a program for controlling a robot. The generation module 310 may be, for example, a generative model trained to receive an instruction written in a natural language, interpret the meaning of the instruction, and output a program in accordance with the meaning of the instruction. The generation model may be, for example, a large-scale language model. As another example, the generation module 310 may be a behavioral planning system. The generation module 310 learns common common sense, interprets the instruction based on the common sense, and generates a program.

[0030] The program in this embodiment is written in a Turing-complete programming language. The program can be executed using an API (Application Programming Interface) provided by the robot 200 as a built-in function. The program can also call other programs during execution.

[0031] The APIs of the robot 200 can be classified into a number of types. The first category is APIs that cause the robot 200 to execute a predetermined task (hereinafter, also referred to as "execution-related APIs"). The second category is APIs that acquire sensor data from a sensor equipped in the robot 200 (hereinafter, also referred to as "acquisition-related APIs"). The third category is APIs that calculate predetermined information based on sensor data (hereinafter, also referred to as "calculation-related APIs"). The calculation-related APIs may calculate the predetermined information using a predefined program provided by the manufacturer of the robot 200, for example.

[0032] The overall configuration of the control system 1000 shown in Fig. 1 is just an example, and various system configuration examples are possible depending on the application and purpose. The control system 1000 may be composed of one or more devices. Each function included in the control system 1000 may be realized by any device that constitutes the system. Each component included in the control system 1000 may be included in any device that constitutes the system.

[0033] For example, the generating module 310 may be built in the generating device 210. Also, for example, the generating module 310 may be built in the terminal device 100 or the robot 200. Furthermore, for example, the generating module 310 may be distributed and held in an external information processing system consisting of a plurality of devices. In this case, the control system 1000 may not include the generating server 300. Also, for example, the generating device 210 may not be built in the robot 200, may be built in the terminal device 100, or may be included in the control system 1000 as another independent device or system. Furthermore, for example, the instruction information may be input directly to the robot 200 without going through the terminal device 100. In this case, the control system 1000 may not include the terminal device 100.

[0034] The control system 1000 may include a plurality of terminal devices 100, robots 200, and generating servers 300. For example, the generating device 210, the control device 220, or the generating server 300 may be realized by a plurality of computers, or may be realized as a cloud computing service. The classification of devices such as the terminal device 100, the robot 200, the generating device 210, the control device 220, and the generating server 300 shown in FIG. 1 is an example.

[0035] <External configuration of autonomous vehicle> The following describes the external configuration of an autonomous vehicle 200, which is an example of the robot 200. Fig. 2 is a diagram showing an example of the external configuration of an autonomous vehicle.

[0036] 2, autonomous vehicle 200 has an overall rectangular parallelepiped shape, and its dimensions in the height direction (z-axis direction) and width direction (x-axis direction) are specified so that autonomous vehicle 200 can enter under the lowest level of the shelf to be transported. Note that the shape of autonomous vehicle 200 is not limited to a rectangular parallelepiped.

[0037] Lock device 511, which is a component of a docking mechanism for docking with a shelf to be transported, is installed on top surface 510 of autonomous vehicle 200. Also installed on top surface 510 of autonomous vehicle 200 is LiDAR (Laser Imaging Detection and Ranging) 512. LiDAR 512 has a measurement range in the front-rear direction (y-axis direction) and width direction (x-axis direction) at the height position of top surface 510 of autonomous vehicle 200, and can detect obstacles and the like within the measurement range by using the measurement results of LiDAR 512.

[0038] A front RGB camera 521 and a ToF (Time of Flight) camera (ToF camera 522) are installed on front surface 520 of autonomous vehicle 200. Note that, although front RGB camera 521 in this embodiment is installed above ToF camera 522, the installation position of front RGB camera 521 is not limited to this position.

[0039] When the autonomous vehicle 200 moves forward, the front RGB camera 521 captures images of, for example, the shelf to be transported, a user near the destination, objects installed near the destination, obstacles on the transport route, etc., and outputs color images.

[0040] ToF camera 522 is an example of a distance image sensor, and is installed facing upward on front surface 520 of autonomous vehicle 200 to such an extent that the surface on which autonomous vehicle 200 travels is not included in the measurement range in order to avoid a multipath problem. One example of the multipath problem is a deterioration in measurement accuracy caused by light emitted from a light source passing through a floor surface and being reflected by another object, and the reflected light being received by ToF camera 522. In this embodiment, the upward installation angle of ToF camera 522 on front surface 520 of autonomous vehicle 200 is set to about 50 degrees with respect to the floor surface.

[0041] Furthermore, when autonomous vehicle 200 moves forward, ToF camera 522 captures images of obstacles and the like within a measurement range that is at least an area through which the docked shelf passes (an area equivalent to the height of the docked shelf x the width of the docked shelf), and outputs a distance image (depth image). In this embodiment, the vertical angle of view of ToF camera 522 is 70 degrees, and the horizontal angle of view is 90 degrees.

[0042] Drive wheels 531 and driven wheels 532 are provided on bottom surface 530 of autonomous vehicle 200 to support autonomous vehicle 200. Drive wheels 531 and driven wheels 532 are an example of a movement mechanism.

[0043] Drive wheels 531 are provided one on each side in the width direction (x-axis direction) (a total of two in the width direction), and each is independently motor-driven to move autonomous vehicle 200 in the forward / reverse directions (y-axis directions). Drive wheels 531 can also turn autonomous vehicle 200 around the z-axis.

[0044] Idler wheels 532 are provided one on each side in the width direction (x-axis direction) (a total of two in the width direction). Idler wheels 532 are provided so as to be able to turn around the z-axis with respect to autonomous vehicle 200. The locations and number of Idler wheels 532 may be other than those described above.

[0045] <Docking Overview> An overview of the operation of autonomous vehicle 200 docking with a transport target will be described. Fig. 3 is a diagram showing the state in which autonomous vehicle 200 docks with a shelf that is a transport target. Fig. 3(a) shows the state immediately before autonomous vehicle 200 docks with shelf 130 that is a transport target and is waiting at the position of anchor 170.

[0046] 3(a), shelf 130 has three stages, and frame guides 610, 620 are attached to the underside of bottom stage 600, approximately in parallel, at intervals corresponding to the width of autonomous vehicle 200. This defines the approach direction when autonomous vehicle 200 approaches the underside of bottom stage 600 of shelf 130, which is the target of transport. Frame guides 610, 620 also function as guides in the width direction when autonomous vehicle 200 transports shelf 130, which is the target of transport, and prevent shelf 130 from shifting in the width direction relative to autonomous vehicle 200.

[0047] The approach direction is an approach direction for autonomous vehicle 200 to bring the transport target into a transportable state. Here, the transportable state includes a state in which autonomous vehicle 200 is docked with the transport target. The transportable state includes a state in which autonomous vehicle 200 can transport the transport target by pushing, lifting, grabbing, placing, towing, or the like. Transporting includes moving the transport target to another location by autonomous vehicle 200. In this embodiment, autonomous vehicle 200 enters below bottom shelf 600 to dock with shelf 130, so the term "approach direction" is used; however, if autonomous vehicle 200 does not enter the transport target and brings the transportable state, the term "travel direction" may be used.

[0048] Furthermore, casters 631-634 are rotatably attached to the base of shelf 130. This allows autonomous vehicle 200 to easily transport shelf 130 when docked therewith.

[0049] On the other hand, FIG. 3(b) shows a state after autonomous vehicle 200 is docked with shelf 130 to be transported. In FIG. 3(b), front surface 520 of autonomous vehicle 200 protrudes in the forward direction beyond each step of shelf 130 and is not covered by the steps of shelf 130. However, depending on the shape or dimensions of shelf 130, at least the measurement range of front RGB camera 521, ToF camera 522, or LiDAR 512 may be blocked by components of shelf 130 (shelf board, pillar, caster, etc.). Therefore, in this embodiment, the shape and dimensions of shelf 130 may be specified, and an invalid area may be set in the measurement range of at least front RGB camera 521, ToF camera 522, or LiDAR 512.

[0050] It should be noted that when autonomous vehicle 200 is docked to shelf 130, the measurement range in the width direction of LiDAR 512 may be blocked by frame guides 610, 620 or components of shelf 130. For this reason, frame guides 610, 620 of shelf 130 are provided with openings 611, 621 to reduce the proportion of the measurement range of LiDAR 512 that is blocked in the width direction.

[0051] <Functional configuration of the generation device> The functional configuration of the generating device in this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the functional configuration of the generating device.

[0052] 4, the generating device 210 includes a robot information storage unit 401, a user information storage unit 402, an instruction receiving unit 403, an input information generating unit 404, a program conversion unit 405, a program storage unit 406, a programming unit 407, and a feedback acquisition unit 408. A generating program is installed in the generating device 210, and the generating device 210 functions as the instruction receiving unit 403, the input information generating unit 404, the program conversion unit 405, the programming unit 407, and the feedback acquisition unit 408 by executing the program.

[0053] The robot information storage unit 401 stores information about the robot 200 (hereinafter also referred to as "robot information"). The robot information may include information about the self-recognition of the robot 200 (hereinafter also referred to as "self-recognition information"), information about software components used to generate a program (hereinafter also referred to as "component information"), and information about the operating environment of the robot 200 (hereinafter also referred to as "environment information").

[0054] The self-awareness information includes, for example, the hardware configuration of the robot 200, a basic policy, and operational constraints. The hardware configuration is the specification of hardware that the robot 200 can use for its operation. An example of the hardware configuration is that the robot 200 is provided with two wheels capable of moving within a plane, one arm capable of gripping any object, two microphones, one speaker, and one camera. Another example of the hardware configuration may be that the robot 200 is provided with sensors such as an inertial sensor, a 3D sensor, a LiDAR, and an ultrasonic sensor, legs capable of walking on four legs or two legs, actuators such as an omnidirectional mechanism such as a Mecanum wheel, and output units such as LEDs, displays, and facial expressions.

[0055] The basic policy is the role or concept of the robot 200. One example of the basic policy is to support the user's life in the home. Other examples of the basic policy may be to support serving food at a restaurant, to help people in an office, to select the shortest route to a given goal, to avoid obstacles if they are present, and so on.

[0056] The operation constraints are constraint conditions that must be satisfied in order for the robot 200 to operate. An example of an operation constraint is that the robot 200 must be charged with a charger when not performing a task. Other examples of operation constraints may be that the robot must not go where there are steps, must not take pictures of places that involve people's privacy, must not make sounds at night, must not emit light at night, must not go near steps, must not enter areas that are inaccessible, and must not transport other shelves while transporting a certain shelf.

[0057] The component information includes, for example, information related to the API provided by the robot 200 (hereinafter also referred to as "API information"), function information indicating functions necessary for generating a program, module information indicating modules necessary for generating a program, and the like.

[0058] The API information differs depending on the type of API. In the case of an execution-type API, the API information includes, for example, an effect obtained by executing the API, arguments for executing the API, and conditions under which the API can be executed. In the case of an acquisition-type API, the API information includes, for example, information obtained by executing the API and conditions under which the API can be executed. For example, in the case of an acquisition-type API that acquires a barcode for identifying furniture, which is the object to be transported, an example of a condition under which the API can be executed is that the robot 200 is present within a range of 0.2 m to 1.0 m from the front of the furniture.

[0059] Examples of APIs may include those that obtain a map of the environment, those that recognize objects captured by a camera and display a list along with their locations, those that allow the robot to move to a desired location, those that obtain the results of voice recognition, those that display a list of existing transport targets, those that obtain a schedule set by the user, those that allow the robot to speak, etc.

[0060] The environmental information includes map information (such as a layout of a room) indicating the range in which the robot 200 can operate, sensor information measured in the operating environment of the robot 200, and information on the target object of the robot 200's operation. The map information includes coordinates of installed objects present in the range in which the robot 200 can operate, and an explanation of each coordinate (such as a name or meaning). The sensor information may be sensor data measured by various sensors equipped in the robot 200, or sensor data measured by various sensors installed in the operating environment of the robot 200. The target object may be, for example, furniture (such as a shelf with casters) which is an example of a transported object, and a charger capable of charging the robot 200. The map information may include areas in which the robot 200 is permitted to enter, areas in which the robot 200 is prohibited from entering, and the like.

[0061] Information about an object includes coordinates indicating the object's position, information about the object's name (full name, abbreviation, name used for voice recognition, etc.), associations between the object and other objects, and descriptions of each object (size, usual location, etc.). Examples of associations between an object and other objects include a pen being placed on a bookshelf, or a plastic bottle of green tea being placed on a drinks shelf. The description of each object includes the object's external dimensions (size or shape, etc.), rules for handling the object (when the bookshelf is not in use, the object should be moved to a designated location on the bookshelf, etc.), etc.

[0062] The environmental information may be information about the operating environment at the current time, or may be information about the operating environment in the past. The information about the operating environment in the past may be history information including information about the operating environment at a plurality of different times.

[0063] When the robot 200 is an autonomous vehicle, the robot information may include, for example, characteristics of the furniture itself, a destination according to the characteristics of the furniture, operation constraints on a map, characteristics of the robot itself, and the like.

[0064] Characteristics of the furniture itself may include, for example, which items are on which shelves, the interior size of the furniture (volume, floor area, etc.), the compatibility of the furniture with the destination, the number of items that can fit in the furniture, combinations that should not be placed together (food and trash, apples and vegetables, etc.), the weight of the furniture, etc.

[0065] Destinations based on the characteristics of furniture may include, for example, which furniture the furniture is highly associated with (a kitchen shelf is highly associated with a kitchen, a spice rack is less associated with a bedroom, etc.), the clearance required for the furniture's placement (distance from other objects), reference landmarks (walls, furniture, appliances, etc.), spatial extent (a bedroom refers to the entire room, a desk refers to the space beside the desk, etc.), the performance to prioritize when transporting the furniture to the destination (whether to prioritize speed and reduce location accuracy, or to prioritize accuracy and increase location accuracy, etc.), and the content to be spoken for each destination.

[0066] Operational constraints on the map may include, for example, the trajectory that the autonomous vehicle will follow during transportation, dangerous areas (such as steps), areas that are not passable according to map information but are actually passable, areas where certain sensors are ignored, etc.

[0067] The characteristics of the robot itself may include, for example, the operating time per unit of remaining charge (because, for example, the operating time per unit of remaining charge changes as the battery deteriorates), the average time it takes to complete charging, the rate at which the remaining charge decreases, the speed or acceleration at which the robot can run, the angular velocity or angular acceleration at which the robot can turn, the range or distance at which the sensor can detect furniture or obstacles, etc.

[0068] The user information storage unit 402 stores information about the user of the robot 200 (hereinafter, also referred to as "user information"). The user information may include a history of instructions by the user, information about the habits of the user, information about the preferences of the user, the frequency of instructions by the user, and the like.

[0069] In the case where the robot 200 is the autonomous vehicle described above, the user information may include furniture usage trends, destinations according to the user's habits, user location trends, and the like.

[0070] Furniture usage trends may include, for example, frequency of use per piece of furniture, frequency of use per user (if multiple users use the same autonomous vehicle), trends in items contained in the furniture, how often the furniture is manually moved by users, the likelihood of it being replaced with other furniture (if there are multiple pieces of furniture with the same role), etc.

[0071] The destination according to the user's habits may include, for example, the frequency of use as a delivery destination for each piece of furniture, statistical information on occupancy rate, etc. The occupancy rate is the probability that other objects, installations, etc. are present at the destination.

[0072] The user's location tendency may include, for example, the movement patterns of people living in the room, the user's usual location, and the like.

[0073] The robot information and the user information may be written in a natural language sentence, for example. That is, the robot information may be a natural language sentence that describes each piece of information related to the robot 200, and the user information may be a natural language sentence that describes each piece of information related to the user. The robot information written in a natural language sentence may be, for example, a sentence such as "In this house, there is a drink shelf and a storage shelf."

[0074] As another example, the robot information and the user information may be stored in a database in which items and values ​​are associated with each other. As a method for associating the items and values ​​with each other, for example, the items and values ​​may be stored as a set, information that allows one piece of information to be obtained from the other piece of information may be stored, or one piece of information and identification information for the other piece of information may be stored as a set.

[0075] The instruction receiving unit 403 receives input of instruction information indicating an instruction from a user to the robot 200. The instruction to the robot 200 may be an instruction for a specific action desired by the user, or may be a declaration of a situation expected by the user.

[0076] The instruction information may be written in a natural language sentence, for example. The natural language sentence indicating the instruction by the user may be, for example, a text input by the user, or may be generated by recognizing the user's speech. The instruction receiving unit 403 may store the received instruction information in the user information storage unit 402 as user information.

[0077] The instruction receiving unit 403 may generate a natural language sentence indicating an instruction by the user by collecting a voice uttered by the user with a microphone provided in the robot 200 and recognizing the voice signal. The instruction receiving unit 403 may receive a natural language sentence indicating an instruction input as a text by the user to the terminal device 100 from the terminal device 100. The instruction receiving unit 403 may receive from the terminal device 100 a recognition result in which the terminal device 100 recognizes the voice uttered by the user.

[0078] The instruction information may include audio information or image information. The audio information may include speech by the user. The image information may include a still image or video of the user. As an example, the image information may be a still image or video of a gesture made by the user. The instruction information may include a result of recognizing the audio information or image information. The result of recognizing the image information may include a result of recognizing the captured gesture.

[0079] The input information generation unit 404 generates input information to be input to the generation module 310 in order to generate a program. The input information generation unit 404 generates the input information based on the instruction information accepted by the instruction acceptance unit 403, the robot information read from the robot information storage unit 401, and the user information read from the user information storage unit 402.

[0080] The input information may be written in a natural language sentence, for example. The input information may be audio information or image information. The input information may be sensor information. The input information may be any combination of these pieces of information.

[0081] When the instruction information, robot information, and user information are written in natural language sentences, the input information generation unit 404 can generate a natural language sentence indicating the input information by connecting those natural language sentences, for example. When the robot information and user information are stored in a database, the input information generation unit 404 can generate a natural language sentence indicating the input information by connecting a natural language sentence indicating an instruction to a natural language sentence generated by embedding those pieces of information in a template, for example. The input information generation unit 404 may generate a natural language sentence indicating the input information based on a trained sentence generation model, for example.

[0082] An example of a natural language sentence indicating input information may be a sentence such as, "The kitchen shelf contains seasonings, plastic wrap, kitchen paper, spoons, forks, and chopsticks. The kitchen shelf has a base that is 40 cm by 60 cm and is about 1.2 m high. It can be stored under Mr. A's desk. It is basically kept in the pantry, but when cooking, it is often placed next to the kitchen island. The top shelf is empty, so it is also used to transport dishes to and from the dining table. From a hygienic point of view, it is not advisable to put garbage in it. This month, Mr. A has used it from Monday to Friday mornings and nights, and Mr. B has used it from Saturday evening to nights. Supermarket bags are sometimes placed in the pantry, and so far the space has been occupied 18% of the time."

[0083] In the above sentence, "The kitchen shelf contains seasonings, plastic wrap, kitchen paper, spoons, forks, and chopsticks. From a hygienic point of view, it is not advisable to put trash in it." is a sentence generated by embedding robot information in the template. Also, in the above sentence, "This month, Person A has been using it from Monday to Friday mornings and nights, and Person B has been using it from Saturday evening to nights. The pantry where supermarket bags are stored is sometimes used, and so far the storage space has been occupied 18% of the time." is a sentence generated by embedding user information in the template.

[0084] The input information may include, for example, robot information such as: To recognize people, the camera, the recognition module, and the distance sensor are activated, and the recognition result is obtained by integrating the bounding box output by the person recognition module and the output from the distance sensor. - To determine the robot's destination, the autonomous mobile component uses map information to select locations that are reachable from the charger as candidates. When the API DetectPerson is launched, it is possible to detect the location of a person on a map expressed by x and y coordinates. -Robot environmental information Information about the software components required to obtain information about the robot's environment

[0085] The input information may include, for example, self-recognition information included in the robot information.

[0086] The input information may include, for example, a natural language sentence indicating specific instruction information from a user. The input information may include the instruction information itself. For example, the instruction information may be a sentence such as the following: Throw out the trash Bring me a spoon Bring shelf A to the couch every Monday at 7pm

[0087] The program conversion unit 405 generates a program based on the input information generated by the input information generation unit 404. The program conversion unit 405 transmits a generation request including the input information to the generation server 300. Upon receiving the program from the generation server 300, the program conversion unit 405 stores the program in the program storage unit 406.

[0088] The program storage unit 406 stores the program generated by the program conversion unit 405. The program storage unit 406 may store a plurality of programs.

[0089] The programming unit 407 updates the program stored in the program storage unit 406 in response to an editing operation by the user. The programming unit 407 transmits screen data for displaying a programming screen that enables visual programming to the terminal device 100. The contents of the program selected by the user are displayed on the programming screen. The programming unit 407 edits the program in response to the user's operation on the programming screen, and updates the program stored in the program storage unit 406 based on the edited program.

[0090] Visual programming is a method for creating a program by arranging program parts having predetermined functions and connecting each program part. The program parts include, for example, those having a function for executing an API provided by the robot 200, those having functions related to execution control such as conditional branching and repetition, and those having a function for performing calculations on variables that can be handled within the program.

[0091] The feedback acquisition unit 408 updates the robot information or the user information based on the feedback information input by the user. The feedback information is a natural language sentence indicating the user's feedback on the result of the operation of the robot 200 according to the program. As with the instruction reception unit 403, the feedback acquisition unit 408 may, for example, acquire feedback information input as text by the user, or may acquire feedback information by recognizing the speech of the user.

[0092] The feedback acquisition unit 408 may update the robot information based on the feedback information. An example of the feedback information for updating the robot information may be "Even if a person comes, you don't have to come to pick them up at night." When such feedback information is acquired, the feedback acquisition unit 408 may add an operation constraint to the robot information indicating that even if a person comes, the person will not be picked up at night. Another example of the feedback information for updating the robot information may be "Don't enter that room," "Move more slowly and carefully," and the like. When such feedback information is acquired, the feedback acquisition unit 408 may update the maximum speed or no-entry areas of the robot included in the robot information.

[0093] The feedback acquisition unit 408 may update the user information based on the feedback information. An example of the feedback information for updating the user information may be "That person does not call the candy shelf." When such feedback information is acquired, the feedback acquisition unit 408 may add information indicating the user's preference regarding candy to the user information.

[0094] The robot information or user information updated by the feedback acquisition unit 408 may be used to generate input information the next time a program is generated.

[0095] The generated program can perform not only task execution, but also subtask execution for acquiring sensor data and satisfying task execution conditions, by combining various APIs provided by the robot 200. In other words, the generated program can accurately execute complex logic based on various information.

[0096] However, since the robot 200 operates in a real space where the state changes from moment to moment, the generated program is not necessarily perfect, and the robot may fail to operate or perform an inappropriate operation. For example, a delay in operation time may occur due to state acquisition that is not actually necessary. Also, for example, the conditions required for state acquisition may not be satisfied, and the correct state may not be acquired.

[0097] Furthermore, for example, the operation of the robot 200 may end up with a result that does not meet the user's expectations. One example of a result that does not meet the user's expectations is when, in response to an operation instruction of "bring me a pen," the robot transports a shelf where the pen is supposed to be placed, but the pen is actually placed on a different shelf. In this case, if the user gives feedback information such as "the pen is placed on the bookshelf," the information on the object of the operation is updated to the correct content, and the subsequent operation will be an operation that meets the user's expectations.

[0098] <Generation process flow> The generation process executed by the generation device 210 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the generation process. The generation process is a process for generating a program for controlling the operation of the robot 200.

[0099] In step S1, a user of the robot 200 inputs an instruction for the robot 200 to the terminal device 100 or the robot 200. The user may speak a natural language sentence indicating an instruction for the robot 200 to the terminal device 100 or the robot 200. In this case, the terminal device 100 or the robot 200 generates instruction information by recognizing the speech of the user. The user may input a natural language sentence indicating an instruction for the robot 200 as text to the terminal device 100. In this case, the terminal device 100 acquires the instruction information input as text by the user.

[0100] In the generating device 210, the instruction receiving unit 403 receives input of instruction information indicating an instruction by a user. Next, the instruction receiving unit 403 sends the received instruction information to the input information generating unit 404.

[0101] In step S2, the input information generation unit 404 of the generation device 210 receives instruction information from the instruction reception unit 403. Next, the input information generation unit 404 reads out robot information from the robot information storage unit 401. The input information generation unit 404 may read out all of the robot information stored in the robot information storage unit 401, or may read out only a portion of the robot information. The robot information read out by the input information generation unit 404 may be selected based on the instruction information. The input information generation unit 404 may be configured to set multiple options based on the robot information as input information, and the generation module 310 may select one of the options.

[0102] In step S3, the input information generation unit 404 of the generation device 210 reads out user information from the user information storage unit 402. The input information generation unit 404 may read out all the user information stored in the user information storage unit 402, or may read out only a part of the user information. The user information read out by the input information generation unit 404 may be selected based on instruction information. For example, the input information generation unit 404 may be configured to set a plurality of options based on the user information as input information, and the generation module 310 may select one of the options.

[0103] In step S4, the input information generation unit 404 of the generation device 210 generates input information based on the instruction information acquired in step S1, the robot information read in step S2, and the user information read in step S3. Next, the input information generation unit 404 sends the generated input information to the program conversion unit 405.

[0104] In step S5, the program conversion unit 405 of the generation device 210 receives input information from the input information generation unit 404. Next, the program conversion unit 405 generates a generation request to request generation of a program. The generation request includes the input information. Then, the program conversion unit 405 transmits the generation request to the generation server 300.

[0105] The generating server 300 receives a generating request from the generating device 210. Next, the generating server 300 extracts input information from the received generating request. Next, the generating server 300 inputs the input information to the generating module 310. The generating module 310 generates and outputs a program based on the input information. The generating server 300 transmits the program output from the generating module 310 to the generating device 210.

[0106] In step S6, the program conversion unit 405 of the generation device 210 receives the program from the generation server 300. The program conversion unit 405 stores the received program in the program storage unit 406.

[0107] <Editing process flow> The editing process executed by the generating device 210 in this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the editing process. The editing process is a process for editing a program generated by the generating device 210.

[0108] In step S11, the user of the robot 200 performs an operation to display a programming screen on the terminal device 100. The operation to display the programming screen is performed by selecting a generated program. For example, the terminal device 100 outputs a list of generated programs to a display device, and the user performs an operation to select a generated program from the list, thereby performing an operation to display the programming screen. The terminal device 100 transmits an editing request to the generating device 210 to request editing of the program. The editing request includes information indicating the program selected by the user.

[0109] In the generating device 210, the programming unit 407 receives the editing request. Next, the programming unit 407 extracts information indicating a program from the received editing request. Next, the programming unit 407 reads out the program selected by the user from the program storage unit 406 based on the information indicating the program.

[0110] In step S12, programming unit 407 generates screen data for displaying a programming screen. The contents of the read program are embedded in the screen data.

[0111] The programming screen is a screen that allows visual programming. In visual programming, a program is created by arranging program parts having specific functions and connecting each program part, so the program displayed on the programming screen is a graphic form in which program parts corresponding to each function are connected according to the order of processing or conditional branching, etc.

[0112] Next, the programming unit 407 transmits screen data for displaying a programming screen to the terminal device 100. The terminal device 100 receives the screen data from the generating device 210. Then, based on the received screen data, the terminal device 100 outputs to the display device a programming screen on which the contents of the generated program are graphically displayed.

[0113] In step S13, the user of the robot 200 performs an operation to edit the program on the programming screen output on the display device of the terminal device 100. Examples of the operation to edit the program include adding or deleting a program part, changing the connection between the program parts, changing the properties of the program parts, etc. The terminal device 100 edits the program in response to the user's operation on the programming screen. Next, the terminal device 100 transmits the edited program to the generation device 210.

[0114] In the generating device 210, the programming unit 407 receives the edited program from the terminal device 100. Next, the programming unit 407 generates a new program based on the edited program. Then, the programming unit 407 stores the generated program in the program storage unit 406. The programming unit 407 may overwrite an existing program with the new program, or may store the new program separately from the existing program.

[0115] It should be noted that the editing process does not necessarily have to be performed. Whether or not the editing process is performed depends on whether or not the user checks the generated program and requests editing. In other words, if the user does not perform an operation to display the programming screen, the editing process will not be performed.

[0116] <Execution process flow> The execution process executed by the control device 220 in this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the execution process. The execution process is a process for controlling the movement of the robot 200 in accordance with the program generated by the generating device 210.

[0117] In step S21, the control device 220 reads out the programs stored in the program storage unit 406 of the generation device 210. The control device 220 may read out all of the programs stored in the program storage unit 406, or may read out some of the programs in response to a predetermined trigger. Examples of the predetermined trigger include when a condition for starting execution of the program is satisfied, or when the program is called from another program.

[0118] Conditions for starting program execution may, for example, be a GPS detecting a person approaching the house, a specific command being input via voice, the doorbell being rung, or a specific person being recognized from an image.

[0119] In step S22, the control device 220 executes the program read out in step S21. In other words, the control device 220 controls the operation of the robot 200 in accordance with the program read out.

[0120] When the control device 220 executes a program online, the control device 220 executes the program acquired from the generation module 310 in the generation process. In this case, the control device 220 may hold the program acquired in step S6 of the generation process in memory and execute the program.

[0121] In step S23, the feedback acquisition unit 408 of the generation device 210 accepts input of feedback information by the user. Next, the feedback acquisition unit 408 updates the robot information stored in the robot information storage unit 401 or the user information stored in the user information storage unit 402 based on the input feedback information.

[0122] <Summary> As is clear from the above description, the generating device 210 according to the embodiment acquires instruction information for the robot 200 from a user, generates input information for the generating module 310 based on the robot information and the instruction information, and acquires a program for controlling the operation of the robot 200, which is generated by inputting the input information to the generating module 310. The robot information includes at least information related to either the operating environment of the robot 200 or a software component for acquiring information about the operating environment.

[0123] The robot information may include information regarding the self-awareness of the robot 200.

[0124] The information about the software components may include execution conditions of the software components. The information about the operating environment may include map information of the environment in which the robot 200 operates. The information about the operating environment may include information about an object of the operation of the robot 200. The information about the object may include at least a position or a name of the object.

[0125] The generating device 210 may generate the input information based on the user information. The user information may include at least information regarding any one of a history of instructions by the user, a habit of the user, or a preference of the user.

[0126] The generating device 210 may generate a program by inputting input information to the generating module 310 .

[0127] The generation device 210 may obtain feedback information from a user regarding the operation result of the robot 200 based on the execution of the program, and generate second input information for the generation module 310 based on the feedback information.

[0128] The generation of the program may be executed by one or more information processing devices capable of communicating with the generating device 210 via a network.

[0129] The generating device 210 may obtain the instruction information by recognizing the user's voice.

[0130] The generating device 210 may output a screen for editing the program to a display device and edit the program based on an instruction from a user. The screen may display the program by arranging program parts having predetermined functions and connecting the program parts.

[0131] The robot 200 may include a generating device 210 and execute a program. The robot 200 may have a sensor and a moving mechanism. The robot 200 may start executing a program in response to a predetermined trigger. The robot 200 may include one or more sensors and acquire desired sensor information by executing a program. The robot 200 may be an autonomous vehicle that docks with a transport object and transports the object.

[0132] As a result, according to one embodiment of the present disclosure, it is possible to provide an apparatus that can easily create a program for controlling the operation of a robot.

[0133] [Other embodiments] A part or the whole of the generating device 210 in the above-mentioned embodiment may be configured with hardware, or may be configured with information processing of software (program) executed by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like. In the case of being configured with information processing of software, software for realizing at least a part of the functions of the generating device 210 in the above-mentioned embodiment may be stored in a non-transitory storage medium such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by reading the software into at least one processor. The software may also be downloaded via a communication network. Furthermore, the information processing by the software may be executed by hardware by implementing all or a part of the processing of the software in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0134] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed storage medium such as a hard disk, memory, etc. The storage medium may be provided inside the generating device 210 (such as a main storage device or an auxiliary storage device), or may be provided in the generating device 210.

[0135] 8 is a block diagram showing an example of a hardware configuration of the generating device 210. As an example, the generating device 210 may be realized as a computer including a processor 701, a main storage device 702 (memory), an auxiliary storage device 703 (memory), a network interface 704, and a device interface 705, which are connected via a bus 706.

[0136] Although the generating device 210 in FIG. 8 includes one of each component, it may include a plurality of the same components.

[0137] Various calculations of the generating device 210 may be executed in parallel using one or more processors, or using multiple computers via a network. Also, various calculations may be distributed to multiple computing cores in the processor 701 and executed in parallel. Also, a part or all of the processes, means, etc. of the present disclosure may be realized by at least one of a processor and a storage device provided on a cloud that can communicate with the generating device 210 via the network interface 704.

[0138] The processor 701 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls or performs calculations on the generating device 210. The processor 701 may be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device including both a general-purpose processor and a dedicated processing circuit. The processor 701 may include an optical circuit, or may include a calculation function based on quantum computing.

[0139] The processor 701 may perform arithmetic processing based on data or software input from each device or the like in the internal configuration of the generating device 210, and may output arithmetic results or control signals to each device or the like. The processor 701 may control each component constituting the generating device 210 by executing an OS (Operating System), an application, or the like of the generating device 210.

[0140] The generating device 210 may be realized by one or more processors 701. Here, the processor 701 may refer to one or more electronic circuits arranged on one chip, or to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the respective electronic circuits may communicate with each other by wire or wirelessly.

[0141] The main memory 702 may store instructions executed by the processor 701 and various data, and information stored in the main memory 702 may be read by the processor 701. The auxiliary memory 703 is a memory device other than the main memory 702. Note that these memory devices refer to any electronic components capable of storing electronic information, and may be semiconductor memories. The semiconductor memories may be either volatile or nonvolatile memories. The memory device for saving various data in the generating device 210 in the above-mentioned embodiment may be realized by the main memory 702 or the auxiliary memory 703, or may be realized by an internal memory built into the processor 701. For example, the robot information storage unit 401, the user information storage unit 402, or the program storage unit 406 in the above-mentioned embodiment may be realized by the main memory 702 or the auxiliary memory 703.

[0142] When the generating device 210 is configured with at least one storage device (memory) and at least one processor connected (coupled) to the at least one storage device, at least one processor may be connected to one storage device. At least one storage device may be connected to one processor. A configuration in which at least one processor among the multiple processors is connected to at least one storage device among the multiple storage devices may be included. A configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

[0143] The network interface 704 is an interface for connecting to the communication network 740 wirelessly or by wire. The network interface 704 may be an appropriate interface, such as one conforming to an existing communication standard. The network interface 704 may exchange information with an external device 730 connected via the communication network 740. The communication network 740 may be any one of a wide area network (WAN), a local area network (LAN), a personal area network (PAN), etc., or a combination thereof, as long as information is exchanged between the generating device 210 and the external device 730. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or Near Field Communication (NFC).

[0144] The device interface 705 is an interface such as a USB that directly connects to an external device 750 .

[0145] The external device 730 is a device connected to the terminal device 100 and the generating device 210 via a network. The external device 730 may be, for example, an external server that stores various information used by the terminal device 100, the generating device 210, or the control device 220. The external device 750 is a device directly connected to the generating device 210.

[0146] The external device 730 or the external device 750 may be, for example, an input device. The input device is, for example, a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, a touch panel, or the like, and provides acquired information to the generating device 210. Alternatively, the input device may be a device including an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0147] In this embodiment, the input device is, for example, an electronic device such as a camera, a microphone, or various sensors, and provides acquired information to the generation device 210.

[0148] Furthermore, the external device 730 or the external device 750 may be a storage device (memory). For example, the external device 730 may be a network storage or the like, and the external device 750 may be a storage device such as an HDD.

[0149] Furthermore, the external device 730 or the external device 750 may be a device having some of the functions of the components of the generation device 210. In other words, the generation device 210 may transmit some or all of the processing results to the external device 730 or the external device 750, or may receive some or all of the processing results from the external device 730 or the external device 750.

[0150] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0151] In this specification (including claims), when expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes the use of data itself or data that has been processed in some way (e.g., data with noise added, normalized, feature values ​​extracted from data, intermediate representations of data, etc.). In addition, when it is stated that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), unless otherwise specified, this includes the use of the result based only on the data, or the use of the result influenced by other data, factors, conditions, and / or states other than the data. In addition, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes the use of data itself as output, or data that has been processed in some way (e.g., data with noise added, normalized, feature values ​​extracted from data, intermediate representations of various data, etc.) as output.

[0152] In this specification (including the claims), the terms "connected" and "coupled" are intended as open-ended terms including any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, physically connection / coupling, etc. The terms should be interpreted appropriately depending on the context in which the terms are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in the terms without any restriction.

[0153] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured / set to actually perform operation B. For example, when element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, when element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure, etc. of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0154] In this specification (including the claims), when terms implying containing or possessing (e.g., "comprising / including," "having," etc.) are used, they are intended to be open-ended terms that include cases in which the term contains or possesses things other than the object indicated by the object of the term. When the object of such terms implying containing or possessing does not specify a quantity or suggests a singular number (such as an expression using the article "a" or "an"), the expression should be construed as not being limited to a specific number.

[0155] In this specification (including the claims), even if expressions such as "one or more" and "at least one" are used in some places and expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0156] In this specification, when a particular advantage / result is described as being obtained from a particular configuration of an embodiment, it should be understood that the same effect can also be obtained from one or more other embodiments having the same configuration, unless there is a reason to the contrary. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states are satisfied, and the effect is not necessarily obtained in the invention according to the claim that specifies the configuration or a similar configuration.

[0157] In this specification (including claims), when a plurality of pieces of hardware perform a predetermined process, each piece of hardware may cooperate to perform the predetermined process, or a portion of the hardware may perform all of the predetermined process. Also, a portion of the hardware may perform a portion of the predetermined process, and another piece of hardware may perform the remainder of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. The hardware may include an electronic circuit, a device including an electronic circuit, and the like.

[0158] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0159] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the above-mentioned individual embodiments. Various additions, modifications, replacements, partial deletions, etc. are possible within the scope of the conceptual idea and intent of the present invention derived from the contents defined in the claims and their equivalents. For example, in the above-mentioned embodiments, when numerical values ​​or formulas are used for explanation, these are shown for illustrative purposes and do not limit the scope of the present disclosure. In addition, the order of each operation shown in the embodiment is also illustrative and does not limit the scope of the present disclosure. [Explanation of symbols]

[0160] 100 Terminal Equipment 200 Robots (autonomous vehicles) 210 Generator 220 Control device 300 Generating Server 310 Generation Module 401 Robot information storage unit 402 User information storage unit 403 Instruction Reception Department 404 Input Information Generation Unit 405 Program Conversion Department 406 Program Memory Unit 407 Programming Department 408 Feedback Acquisition Department 1000 Control System

Claims

1. At least one memory; At least one processor; The at least one processor Obtain information from the user, generating input information for a generation module based on robot information about the robot and the obtained information; acquiring a program for controlling an operation of the robot, the program being generated by inputting the input information into the generation module; the robot information includes information about a software component for acquiring information about an operating environment of the robot; The generation module is a generative model. generator.

2. The robot information includes information regarding a self-awareness of the robot. The generating device of claim 1 .

3. The information about the software component includes an execution condition of the software component. The generating device of claim 1 .

4. The information about the operating environment includes any one of map information of the environment in which the robot operates, sensor information measured in the operating environment of the robot, or information about an object of the robot's operation. The generating device of claim 1 .

5. The information about the operating environment includes at least information about the object, The information about the object includes at least information about the object's location, name, relationship to other objects, size, usual location, or handling rules; The generating device according to claim 4 .

6. the at least one processor generating the input information based on information about the user. The generating device of claim 1 .

7. The information about the user includes at least information about a history of the acquired information, a habit of the user, or a preference of the user. The generating device according to claim 6.

8. The information regarding the software component includes at least information regarding an API that acquires data from a sensor equipped in the robot, The generating device of claim 1 .

9. The generating module, The at least one processor generates the program by inputting the input information to the generation module. A generating device according to any one of claims 1 to 8.

10. The at least one processor obtaining feedback information from the user regarding a result of an operation of the robot based on the execution of the program; generating second input information to the generating module based on the feedback information; A generating device according to any one of claims 1 to 8.

11. The generation of the program is executed by one or more information processing devices capable of communicating with the generation device via a network. A generating device according to any one of claims 1 to 8.

12. the at least one processor obtaining information from the user by recognizing speech of the user; A generating device according to any one of claims 1 to 8.

13. The acquired information includes instruction information from the user to the robot. A generating device according to any one of claims 1 to 8.

14. The instruction information includes information regarding a situation expected by the user. The generating device according to claim 13.

15. The at least one processor outputting a screen for editing the program to a display device; Editing the program based on instructions from the user; A generating device according to any one of claims 1 to 8.

16. the screen displays the program by arranging program parts having predetermined functions and connecting the program parts; 16. The generating device of claim 15.

17. A robot comprising the generating device according to claim 1 and executing the program.

18. In response to a predetermined trigger, execution of the program is started. The robot of claim 17.

19. One or more sensors; Executing the program to obtain desired sensor information; The robot of claim 17.

20. It is an autonomous vehicle that docks with and transports objects. The robot of claim 17.

21. At least one processor Obtain information from the user, generating input information for a generation module based on robot information about the robot and the obtained information; acquiring a program for controlling an operation of the robot, the program being generated by inputting the input information into the generation module; the robot information includes information about a software component for acquiring information about an operating environment of the robot; The generation module is a generative model. Generation method.

22. At least one processor, Obtain information from the user, generating input information for a generation module based on robot information about the robot and the obtained information; execute a process of acquiring a program for controlling an operation of the robot, the program being generated by inputting the input information to the generation module; the robot information includes information about a software component for acquiring information about an operating environment of the robot; The generation module is a generative model. program.

Images