Method and apparatus for generating learning model for controlling autonomous driving of robot trained to reflect preference information through input of preference information

Abstract

Provided is a method, performed by a computer system, for generating a learning model for autonomous driving of a robot. The computer system obtains driving data from the driving of the robot or a driving simulation of a robot agent, determines a plurality of candidate paths for the state of the robot or the robot agent at a first time point, ranks the candidate paths on the basis of preference information, and trains a compensation model that associates candidate paths having relatively high ranks, among the candidate paths, with higher compensation scores, thereby generating a learning model for autonomous driving of the robot.

Description

Method and device for generating a learning model for controlling autonomous driving of a robot that is trained to reflect preference information by injecting preference information

[0001] The following description relates to a method and device for generating a learning model for controlling autonomous driving of a robot, and to a method and device for generating a learning model that is trained to reflect preference information by injecting preference information from a user or the like.

[0002] Autonomous robots are robots that can independently navigate their surroundings, detect obstacles, and use wheels or legs to find the optimal route to their destination. They are being developed and utilized in various fields, including autonomous vehicles, logistics, hotel services, and robot vacuum cleaners.

[0003] The autonomous driving of a robot can be controlled based on various driving algorithms. For example, these driving algorithms may be based on reinforcement learning.

[0004] Incorporating user preference information into the driving algorithms used to control autonomous robots requires appropriately designing reward functions that reflect this preference information. However, designing reward functions that ensure robot control according to the intent indicated by the preference information requires rigorous engineering effort. Furthermore, even when controlling a robot using a driving algorithm generated using the designed reward function, the robot frequently experiences behavior that deviates from the intent indicated by the preference information.

[0005] Korean Patent Publication No. 10-2005-0024840 is a technology regarding a path planning method for an autonomous mobile robot, and discloses a method for planning an optimal path for a mobile robot moving autonomously at home or in the office to safely and quickly reach a target point while avoiding obstacles.

[0006] The information described above is for the purpose of understanding only and may contain matters that do not form part of the prior art and may not contain what the prior art would suggest to a person skilled in the art.

[0007] A method for generating a learning model for autonomous driving of a robot can be provided by acquiring driving data from a driving simulation of a robot or a robot agent, determining a plurality of candidate paths for the state of the robot or robot agent at a first point in time, ranking them based on preference information, and training a reward model such that a candidate path with a relatively high rank among the candidate paths is associated with a greater number of reward points.

[0008] A method can be provided to directly inject preference information for a desired section of a robot's driving or a robot agent's driving simulation, and thus control the robot by generating a learning model trained to reflect the preference information.

[0009] In one aspect, a method for generating a learning model for autonomous driving of a robot is provided, which is performed by a computer system, comprising: obtaining driving data from a simulation of driving of the robot in a space or a driving of a robot agent for the space; determining, based on the driving data, a plurality of candidate paths for a state of the robot or the robot agent at a first point in time during the driving of the robot or the driving simulation of the robot agent; ranking the plurality of candidate paths based on a first preference input from a user or a predefined second preference; training a reward model such that a candidate path with a relatively high ranking among the candidate paths is associated with a greater number of reward points; and generating a learning model for autonomous driving of the robot using the reward model.

[0010] The driving data may include at least one of sensor data, global path information, and speed information obtained from the driving of the robot or the driving simulation of the robot agent.

[0011] Each candidate path of the above candidate paths may represent a subsequent position or sequence of positions of the robot or the robot agent with respect to the state of the robot or the robot agent at the first point in time.

[0012] The above ranking step may include: ranking the plurality of candidate routes based on a first preference input from the user; receiving, from the user, a preference label for each of the candidate routes as the first preference; and ranking the candidate routes based on the received preference label.

[0013] The above ranking step may include: a step of ranking the plurality of candidate paths based on the determined second preference, obtaining the second preference defined in a predetermined driving algorithm for autonomous driving of the robot; and a step of ranking the candidate paths by labeling each of the candidate paths based on the obtained second preference.

[0014] The above reward model may be configured to input the state of the robot or the robot agent at the first point in time and one of the candidate paths, and output a reward score associated with the candidate path.

[0015] The step of training the above reward model may include: selecting two candidate paths from among the candidate paths; and training the reward model using a loss function that inputs the two candidate paths and the state of the robot or the robot agent at the first point in time, and causes a candidate path with a relatively higher rank among the two candidate paths to receive a higher reward score.

[0016] The above loss function is defined to learn the reward model based on calculating the expected value for the values ​​of Equation 1,

[0017] [Mathematical Formula 1]

[0018]

[0019] Above Is is the reward function of the above compensation model with parameters, and D is the x, y w , and the above y l represents a data set including, x is the state of the robot or the robot agent at the first point in time selected from D, and y w is a candidate path with a relatively higher rank among the two candidate paths selected from the above D, and y l is a candidate path with a relatively lower rank among the two candidate paths selected from the above D, and can represent the sigmoid function.

[0020] Among the above driving data, sensor data is periodically captured during driving of the robot or driving simulation of the robot agent, and the first point in time can be determined based on the point in time at which the sensor data is captured.

[0021] The first point in time may represent a point in time at which a long-term path plan of the robot or the robot agent is established based on the captured sensor data, and a short-term path plan of the robot or the robot agent is established based on the captured sensor data and the long-term path plan.

[0022] The above-determining step, the ranking step, and the learning step may be performed for each point in time during the driving simulation of the robot or the robot agent.

[0023] The robot drives the space based on a reinforcement learning algorithm, a driving simulation of the robot agent is performed based on the reinforcement learning algorithm, and the step of determining the plurality of candidate paths can determine the plurality of candidate paths for the state of the robot or the robot agent at a first point in time based on the reinforcement learning algorithm.

[0024] The method for generating a learning model for autonomous driving of the robot further includes the steps of: obtaining updated driving data by controlling driving of the robot within the space or driving simulation of the robot agent for the space using the generated learning model; and determining a plurality of updated candidate paths for a state of the robot or the robot agent at a second point in time during the driving of the robot or the driving simulation of the robot agent based on the updated driving data, wherein the updated candidate paths may reflect the first preference or the second preference more than the candidate paths.

[0025] The method for generating a learning model for autonomous driving of the robot further includes the steps of: providing a user terminal of the user with a visual representation of each of the plurality of candidate paths; and providing the user terminal with a function for designating a ranking of each candidate path provided in the visual representation, wherein the ranking step ranks the plurality of candidate paths based on the first preference input by the user, and the plurality of candidate paths can be ranked according to a ranking designated by the function as the first preference.

[0026] In another aspect, a computer system for generating a learning model for autonomous driving of a robot is provided, comprising at least one processor configured to execute computer-readable instructions, wherein the at least one processor obtains driving data from a simulation of driving of the robot in a space or a driving of a robot agent for the space, determines a plurality of candidate paths for a state of the robot or the robot agent at a first point in time during the driving of the robot or the driving simulation of the robot agent based on the driving data, ranks the plurality of candidate paths based on a first preference input from a user or a predefined second preference, trains a reward model such that a candidate path with a relatively high ranking among the candidate paths is associated with a greater number of reward points, and generates a learning model for autonomous driving of the robot using the reward model.

[0027] By directly injecting preference information into a desired section of a robot's driving or a robot agent's driving simulation, a learning model can be created to control the driving of the robot that has been trained to reflect the preference information.

[0028] By determining a plurality of candidate paths representing future positions or position sequences for the driving of the robot or the current state of the robot agent, and labeling preference information for each of the determined candidate paths, a reward model that directly reflects the preference information can be trained, and a learning model for controlling the driving of the robot generated using the reward model can be used to determine a behavioral sequence of the robot that reflects the preference information. The learning model can be updated to determine candidate paths that further reflect the preference information for the driving of the robot or the current state of the robot agent.

[0029] FIG. 1 illustrates a method for generating a learning model that is trained to reflect preference information by injecting preference information from a user or the like, according to one embodiment.

[0030] FIG. 2 is a block diagram illustrating a computer system for generating a learning model for autonomous driving of a robot, according to one embodiment.

[0031] FIG. 3 is a block diagram illustrating a robot moving within a space according to one embodiment.

[0032] FIGS. 4 and 5 are block diagrams showing a robot control system for controlling a robot according to one embodiment.

[0033] FIG. 6 is a flowchart illustrating a method for generating a learning model that is trained to reflect preference information by injecting preference information from a user or the like, according to one embodiment.

[0034] FIG. 7 is a flowchart illustrating a method for ranking candidate paths for a state of a robot or robot agent based on preference information from a user, according to an example.

[0035] FIG. 8 is a flowchart illustrating a method for ranking candidate paths for a state of a robot or robot agent according to preference information defined in a predetermined driving algorithm, according to an example.

[0036] Figure 9 is a flowchart illustrating a method for learning a reward model for ranked candidate paths, according to an example.

[0037] FIG. 10 illustrates a method for determining candidate paths that further reflect preference information about the state of a robot or robot agent as a reward model is updated, according to an example.

[0038] FIG. 11 is a flowchart illustrating a method for visualizing candidate paths for a state of a robot or robot agent and labeling each candidate path, according to an example.

[0039] Figure 12 illustrates a visualization of candidate paths for a state of a robot or robot agent, and labeling of each candidate path, according to an example.

[0040] Hereinafter, the detailed description will be given with reference to the attached drawings.

[0041]

[0042] FIG. 1 illustrates a method for generating a learning model that is trained to reflect preference information by injecting preference information from a user or the like, according to one embodiment.

[0043] In Fig. 1, a method is illustrated in which a robot (100) moves toward a destination while avoiding an obstacle (30) by executing a driving algorithm configured to control the movement of the robot (100) toward the destination within a space (10).

[0044] A driving algorithm configured to control movement of the robot (100) to a destination may be executed within the robot (100) or within a robot control system (120). This driving algorithm may be implemented by a pre-learned learning model to control autonomous driving of the robot (100). For example, this learning model may be a reinforcement learning-based model, a machine learning model, a deep learning model, etc.

[0045] The space (10) where the robot (100) moves (or drives) is a place where the robot (100) provides a service, and may represent, for example, an indoor and / or outdoor space.

[0046] A robot (100) driving within a space (10) may be a service robot used to provide a service within the space (10). For example, the robot (100) may be configured to provide a service at a predetermined location within the space (10) or to a predetermined user through autonomous driving, and the (respective) movement of the robot (100) and provision of the service may be controlled by a robot control system (120). The movement of the robot (100) to a destination controlled through the aforementioned driving algorithm may be movement of the robot (100) to a predetermined location for providing such a service.

[0047] The structure of the robot (100) and the robot control system (120) will be described in more detail with reference to FIGS. 3 to 5, which will be described later.

[0048] The robot (100) can acquire sensor data about the surrounding environment within a space, and can be controlled to avoid obstacles (30) and drive within the space (10) based on such sensor data and long-term and short-term path planning from the robot control system (120). The driving algorithm described above may be for short-term path planning of the robot (100).

[0049] Meanwhile, an obstacle (30) is an object that exists temporarily or non-temporarily within space (10), for example, a moving object such as a person or another robot, or a fixed object that does not move within space (10).

[0050] The robot (100) illustrated in FIG. 1 may be a robot that actually drives in space (10), or a robot agent for virtually simulating the driving of the robot (100). That is, the example illustrated in FIG. 1 may represent the actual driving of the robot (100) or a driving simulation of a robot agent.

[0051] In the detailed description to be described later, embodiments are described based on a robot (100), but the term 'robot (100)' in the present disclosure may be used to encompass a robot agent, and overlapping descriptions related to the robot agent may be omitted.

[0052] In an embodiment, the computer system (200) can obtain driving data from a driving simulation of the robot (100) or a robot agent in space (10) from the robot (100) or the robot control system (120), learn a model for autonomous driving of the robot (100), and provide a driving algorithm for driving of the robot (100) implemented using the learning model to the robot (100) or the robot control system (120).

[0053] For example, the computer system (200) can determine a plurality of candidate paths (40) for the state of the robot (100) or the robot agent at a specific point in time (i.e., a specific point in time of a short path) during the driving simulation of the robot (100) or the robot agent based on the driving data. The computer system (200) can rank the candidate paths (40) based on preference labels from a user or automatic preference labels by the system for the plurality of candidate paths (40), and can learn a reward model to reflect the preference indicated by the ranking of the candidate paths (40), thereby generating a learning model for autonomous driving of the robot (100).

[0054] Accordingly, in the embodiment, preference information can be directly injected into a desired section of a driving simulation of a robot (100) or a robot agent, and thus, the computer system (200) can generate a learning model trained to reflect such injected preference information. Accordingly, the robot (100) can be controlled in accordance with the intent indicated by the preference information.

[0055] The computer system (200) can control the robot (100) by transmitting a driving algorithm implemented according to the generated learning model to the robot control system (120).

[0056] The computer system (200) may be a server or a computer system constituting a server. The computer system (200) may be an electronic device that obtains the driving data while communicating with the robot (100) and / or the robot control system (120) and generates a learning model for autonomous driving of the robot (100).

[0057] Alternatively, the computer system (200) may be a computer system included within the robot (100). In this case, the robot (100) may be configured to directly generate an embodiment learning model for autonomous driving based on acquired driving data.

[0058] Alternatively, the computer system (200) may be a computer system included in a robot control system (120) for controlling the robot (100). In this case, the computer system (200) may obtain the driving data while communicating with the robot (100) and generate a learning model for autonomous driving of the robot (100).

[0059] A specific method for generating a learning model for autonomous driving of a robot based on learning that reflects labeled preference information by a computer system (200) is described in more detail with reference to FIGS. 2 to 12, which will be described later.

[0060]

[0061] FIG. 2 is a block diagram illustrating a computer system for generating a learning model for autonomous driving of a robot, according to one embodiment.

[0062] The computer system (200) may be an electronic device configured to perform a method for generating a learning model for autonomous driving of the robot (100) of the embodiment. The computer system (200) may execute an application, software, or program for performing such a data processing method.

[0063] As described above, the computer system (200) may be a server or a computer system constituting a server, or may be a separate computer system for performing the method of generating the above-described learning model. For example, the computer system (200) may be a computer system included in a robot (100) or a computer system included in a robot control system (120) for controlling the robot (100).

[0064] As illustrated in FIG. 2, the computer system (200) may include, as components, a memory (210), a processor (220), a communication interface (230), and an input / output interface (240). The input / output interface (240) may communicate with an input / output device (250) within the computer system (200) or separate from the computer system (200).

[0065] The memory (210) is a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as the disk drive may be included in the computer system (200) as a separate permanent storage device distinct from the memory (210). In addition, the memory (210) may store an operating system and at least one program code. These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (210) through a communication interface (230) rather than a computer-readable recording medium. For example, software components may be loaded into the memory (210) of a computer system (200) based on a computer program that is installed by files received over a network (260).

[0066] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) via the memory (210) or the communication interface (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

[0067] The communication interface (230) may provide a function for the computer system (200) to communicate with other devices via a network (260). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer system (200) according to program codes stored in a recording device such as a memory (210) may be transmitted to other devices via the network (260) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer system (200) via the communication interface (230) of the computer system (200) via the network (260). Signals, commands, data, etc. received via the communication interface (230) may be transmitted to the processor (220) or the memory (210), and files, etc. may be stored in a storage medium (the aforementioned permanent storage device) that the computer system (200) may further include.

[0068] The communication method through the communication interface (230) is not limited, and may include not only a communication method utilizing a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network (260) may include, but also a short-range wired / wireless communication between devices. For example, the network (260) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. In addition, the network (260) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0069] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, keyboard, camera, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (240) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (250) may also be configured as a single device with the computer system (200).

[0070] Additionally, in other embodiments, the computer system (200) may include fewer or more components than those illustrated in FIG. 2. However, most conventional components need not be explicitly depicted. For example, the computer system (200) may be implemented to include at least some of the input / output devices (250) described above, or may further include other components such as a transceiver, a camera, various sensors, a database, and the like.

[0071] The processor (220) of the computer system (200) may be configured to perform steps for performing a method of generating a learning model for autonomous driving of the robot (100) by obtaining driving data from a driving simulation of the robot (100) or a robot agent, determining candidate paths (40) for a state of the robot (100) or the robot agent at a specific point in time, ranking them according to preference information, and training a reward model such that a relatively high-ranking candidate path among the candidate paths (40) is associated with more reward points, as described below.

[0072] In the detailed description to be provided below, for convenience of explanation, the above steps are described as being performed by the computer system (200), and operations performed by the processor (220) or other components of the computer system (200), or operations performed by an application / program executed by the processor (220), may be described as operations performed by the computer system (200) for convenience of explanation.

[0073] Above, the technical features described above with reference to Fig. 1 can also be applied to Fig. 2, so redundant descriptions are omitted.

[0074]

[0075] FIG. 3 is a block diagram illustrating a robot moving within a space according to one embodiment.

[0076] As described above, the robot (100) may be a service robot used to provide a service within a space (10). The robot (100) may autonomously drive within the space (10) to provide a service at a predetermined location within the space (10) or to a predetermined user.

[0077] The robot (100) may be a physical device and may include a control unit (104), a drive unit (108), a sensor unit (106), and a communication unit (102), as illustrated. Alternatively, the robot (100) described in the embodiment may represent a robot agent, in which case the robot agent may not include the components (102 to 108), or the control unit (104), the drive unit (108), the sensor unit (106), and the communication unit (102) may be components virtually included in such a robot agent.

[0078] The control unit (104) may be a physical processor built into the robot (100), and although not separately illustrated, may include a path planning processing module, a mapping processing module, a driving control module, a localization processing module, a data processing module, and a service processing module. In this case, the path planning processing module, the mapping processing module, and the localization processing module may be selectively included in the control unit (104) according to an embodiment to enable indoor autonomous driving of the robot (100) even when communication with the robot control system (120) is not established.

[0079] The communication unit (102) may be a configuration for the robot (100) to communicate with other devices (such as other robots, a computer system (200), or a robot control system (120)). In other words, the communication unit (102) may be a hardware module, such as an antenna, a data bus, a network interface card, a network interface chip, and a networking interface port of the robot (100), or a software module, such as a network device driver or a networking program, that transmits / receives data and / or information to / from other devices.

[0080] The driving unit (108) controls the movement of the robot (100) and may include equipment for performing the movement as a configuration that enables the movement.

[0081] The sensor unit (106) may be configured to collect data required for autonomous driving and service provision of the robot (100). The sensor unit (106) may not include expensive sensing equipment, but may include only sensors such as low-cost ultrasonic sensors and / or low-cost cameras. The sensor unit (106) may include sensors for identifying other robots or people in front and / or behind. For example, other robots, people, and other objects may be identified as obstacles (30) through the camera of the sensor unit (106). Alternatively, the sensor unit (106) may include an infrared sensor (or infrared camera). In addition to the camera, the sensor unit (106) may further include sensors for recognizing / identifying users, other robots, or objects in the vicinity. In this way, the sensor unit (106) may be configured to identify obstacles (30).

[0082] For example, in a case where the robot (100) is controlled according to a long-term path plan, the data processing module of the control unit (104) can transmit sensing data including output values ​​of sensors of the sensor unit (106) to the robot control system (120) through the communication unit (102). The robot control system (120) can transmit path data generated using an indoor map within the space (10) to the robot (100). The path data can be transmitted to the data processing module through the communication unit (102). The data processing module can directly transmit the path data to the driving control module, and the driving control module can control the driving unit (108) according to the path data to control indoor autonomous driving of the robot (100).

[0083] Meanwhile, in a case where the robot (100) is controlled according to a long-term path plan (e.g., the driving algorithm described above), the robot control system (120) can generate a control signal (e.g., a speed and / or direction control signal) for controlling the robot (120) according to the driving algorithm based on sensing data received from the robot (100), and can control the robot (100) based on the generated control signal. Accordingly, the autonomous driving of the robot (100) can be controlled.

[0084] Alternatively, if the robot (100) and the robot control system (120) cannot communicate or an algorithm for autonomous driving is executed within the robot (100), the data processing module may transmit sensing data to the localization processing module and generate path data through the path planning processing module and the mapping processing module to directly process indoor autonomous driving of the robot (100).

[0085] The robot (100) may be distinct from a mapping robot used to create an indoor map within a space (10). Since the robot (100) does not include expensive sensing equipment, it can process indoor autonomous driving using the output values ​​of sensors such as low-cost ultrasonic sensors and / or low-cost cameras. Meanwhile, if the robot (100) has previously processed indoor autonomous driving through communication with a robot control system (120), more accurate indoor autonomous driving may be possible using low-cost sensors by further utilizing mapping data, etc. included in the route data previously received from the robot control system (120).

[0086] However, depending on the embodiment, the robot (100) may also serve as the mapping robot.

[0087] The service processing module can receive commands received through the robot control system (120) through the communication unit (102) or through the communication unit (102) and the data processing module. The drive unit (108) can further include equipment related to services provided by the robot (100) as well as equipment for moving the robot (100). The service processing module can transmit a drive command for a service to be provided to the drive control module, and the drive control module can control the configuration included in the robot (100) or the drive unit (108) according to the drive command so that the service can be provided.

[0088] As described above, if the robot (100) only provides sensing data for controlling the robot (100) to the robot control system (120), and the control of the robot (100) is performed through the robot control system (120), the robot (100) may correspond to a brainless robot.

[0089] Meanwhile, each robot (100) may have a different size and shape depending on the model or service provided.

[0090] The configuration and operation of the robot control system (120) that controls the robot (100) will be described in more detail with reference to FIGS. 4 and 5, which will be described later.

[0091] The description of the technical features described above with reference to FIGS. 1 and 2 can also be applied to FIG. 3, so any redundant description will be omitted.

[0092]

[0093] FIGS. 4 and 5 are block diagrams showing a robot control system for controlling a robot according to one embodiment.

[0094] The robot control system (120) may be a device that controls movement (i.e., driving) of the aforementioned robot (100) within the space (10) and provision of services by the robot (100) within the space (10).

[0095] The robot control system (120) may include at least one computing device and may be implemented as a server (i.e., a cloud server) located within the space (10) or outside the space (10).

[0096] The robot control system (120) may include a memory (330), a processor (320), a communication unit (310), and an input / output interface (340), as illustrated.

[0097] The description of the memory (210), processor (220), communication interface (230), and input / output interface (240) described above with reference to FIG. 2 can be similarly applied to the memory (330), processor (320), communication unit (310), and input / output interface (340), and thus, redundant descriptions are omitted.

[0098] In other embodiments, the robot control system (120) may include more components than those illustrated.

[0099] Referring to FIG. 4, the components (410 to 440) of the processor (320) will be described in more detail. The processor (320) may include a map generation module (410), a localization processing module (420), a path planning processing module (430), and a service operation module (440), as illustrated. The components included in the processor (320) may be representations of different functions performed by at least one processor included in the processor (320) according to control instructions based on the code of an operating system or the code of at least one computer program.

[0100] The map generation module (410) may be a component for generating an indoor map of a target facility using sensing data generated by a mapping robot (not shown) autonomously driving within the space (10) about the target facility (e.g., the interior of the space (10).

[0101] At this time, the localization processing module (420) can determine the location of the robot (100) inside the target facility by using sensing data received from the robot (100) through the network and the indoor map of the target facility generated through the map generation module (410).

[0102] The path planning processing module (430) can generate a control signal for controlling indoor autonomous driving of the robot (100) using the sensing data received from the robot (100) and the generated indoor map. For example, the path planning processing module (430) can generate a path (i.e., path data) of the robot (100). The generated path (path data) can be set for the robot (100) for driving the robot (100) along the path. The robot control system (120) can transmit information about the generated path to the robot (100) via a network. For example, the path information can include information indicating the current location of the robot (100), information for mapping the current location with the indoor map, and path planning information. The path information can include information about a path that the robot (100) should drive to reach a predetermined location within the space (10) or to provide a service to a predetermined user. The path planning processing module (430) can set a path (i.e., path data) for the robot (100). The robot control system (120) can control the movement of the robot (100) so that the robot (100) moves according to the set path (i.e., along the set path).

[0103] The service operation module (440) may include a function for controlling the service provided by the robot (100) within the space (10). For example, the robot control system (120) or the service provider operating the space (10) may provide an IDE (Integrated Development Environment) for the service (e.g., cloud service) provided by the robot control system (120) to the user or manufacturer of the robot (100). In this case, the user or manufacturer of the robot (100) may create software for controlling the service provided by the robot (100) within the space (10) through the IDE and register the software in the robot control system (120). In this case, the service operation module (440) may control the service provided by the robot (100) using the software registered in connection with the robot (100).

[0104] The description of the technical features described above with reference to FIGS. 1 to 3 can also be applied to FIGS. 4 and 5, so redundant descriptions are omitted.

[0105]

[0106] In the detailed description to be provided below, for convenience of explanation, steps and / or operations performed by the processor (220) and the like of the method for generating a learning model of the embodiment are described as operations performed by the computer system (200) including the processor (220).

[0107]

[0108] FIG. 6 is a flowchart illustrating a method for generating a learning model that is trained to reflect preference information by injecting preference information from a user or the like, according to one embodiment.

[0109] In step (610), the computer system (200) may obtain driving data from a driving simulation of the robot (100) within the space (10) or the robot agent relative to the space (10). Such driving data may include at least one of sensor data obtained by the robot (100) or the robot agent while driving, global path information along which the robot (100) or the robot agent is driving, and speed information during driving of the robot (100) or the robot agent.

[0110] In step (620), the computer system (200) may determine a plurality of candidate paths (40) for the state of the robot (100) or the robot agent at a first point in time during the driving of the robot (100) or the driving simulation of the robot agent, based on the acquired driving data. The first point in time may be a time indicating a section during the driving of the robot (100) or the simulation of the robot agent into which preference information is to be injected. The section corresponding to the first point in time may be selected by a user (i.e., a user of the computer system (200)).

[0111] This first point in time may correspond to a time step selected from among the time steps that constitute the driving of the robot (100) or the simulation of the robot agent. Each time step may represent the section into which preference information is to be injected.

[0112] Sensor data among driving data can be captured periodically during driving simulation of the robot (100) or the robot agent.

[0113] The first point in time may be determined based on the time at which the sensor data is captured. In other words, the first point in time may correspond to each of the time points at which the sensor data is captured, or may correspond to a selected time point among the time points at which the sensor data is captured. Each of these time points at which the sensor data is captured may correspond to a time point at which a snapshot of the sensor data is generated, or the data may correspond to each frame of the driving simulation of the robot (100) or the driving simulation of the robot agent. For example, the sensor data may be acquired at intervals of 0.1 or 0.15 seconds, and thus, the first point in time may exist or be selected at intervals of 0.1 or 0.15 seconds.

[0114] Specifically, the first point in time corresponding to each time step may represent a point in time at which a long-term path plan (global path plan) of the robot (100) or robot agent is established (by a long-term path planner) based on captured sensor data, and a short-term path plan (local path plan) of the robot (100) or robot agent is established (by a short-term path planner) based on captured sensor data and the long-term path plan.

[0115] In this way, each time step can represent the last time section of the loop from when the long-term path planner receives sensor data until the short-term path planner produces the computational results for establishing a short-term path plan. In other words, each time step (first point in time) can represent a point in time at which the sensor data, which are input values ​​for producing the results of the short-term path plan of the robot (100) or the robot agent, and the input data such as the long-term path plan can be specified.

[0116] The state of the robot (100) or the robot agent at a first point in time may include the position of the robot (100) or the robot agent at that point in time. Alternatively, the state may include at least one of position, velocity, and direction as a displacement of the robot (100) or the robot agent.

[0117] Each of the plurality of candidate paths (40) for the above state may represent a subsequent state of the robot (100) or the robot agent. For example, each candidate path of the candidate paths (40) may represent a subsequent (i.e., subsequent) position or position sequence of the robot (100) or the robot agent with respect to the state of the robot (100) or the robot agent at the first point in time.

[0118] Meanwhile, the robot (100) may drive in space (10) based on a reinforcement learning algorithm, and the driving simulation of the robot agent may be performed based on the reinforcement learning algorithm. Reinforcement learning is a type of machine learning, and is a learning method that selects the optimal action for a given situation (or state), and a computer program that is the target of reinforcement learning may be called an agent (here, a robot agent). The agent establishes a policy indicating the action it will take for a given situation, and a model may be trained to establish a policy that can obtain the maximum reward.

[0119] In the step (620), the computer system (200) can determine a plurality of candidate paths (40) for the state of the robot (100) or the robot agent at the first point in time based on a reinforcement learning algorithm. That is, the computer system (200) can determine the candidate paths (40) in order of increasing reward (reward score) for the state of the robot (100) or the robot agent and select them as the candidate paths (40) to be labeled.

[0120] In step (630), the computer system (200) may rank candidate routes based on preference information. For example, the computer system (200) may rank a plurality of candidate routes (40) based on a first preference input by a user (who wishes to input preference information) or a predefined second preference.

[0121] A method for labeling preference information for each of the candidate paths (40) is described in more detail with reference to FIGS. 7 and 8.

[0122] In relation to this, first, FIG. 7 is a flowchart illustrating a method for ranking candidate paths for states of a robot or robot agent based on preference information from a user, according to an example.

[0123] The computer system (200) can rank a plurality of candidate paths (400) based on a first preference input from a user (labeler).

[0124] Specifically, at step (710), the computer system (200) may receive a preference label for each of the candidate paths (40) as a first preference from the user.

[0125] At step (720), the computer system (200) may rank the candidate paths (40) according to the received preference labels.

[0126] For example, a user may designate the rank of each of the candidate routes (40) as a preference label through a user terminal provided with the candidate routes (40), and the computer system (200) may rank the candidate routes (40) according to the rank designated by the user. Meanwhile, at least two of the candidate routes (40) may be designated with the same rank.

[0127] Therefore, a compensation model that simulates the first preference information input by the user can be obtained by the subsequent step (640).

[0128] Meanwhile, FIG. 8 is a flowchart illustrating a method for ranking candidate paths for a state of a robot or robot agent according to preference information defined in a predetermined driving algorithm according to an example.

[0129] The computer system (200) can rank a plurality of candidate paths (40) based on a predetermined second preference. Accordingly, the computer system (200) can automatically label the plurality of candidate paths (40) with preference information even if the user does not label the preference information for each candidate path.

[0130] Specifically, in step (810), the computer system (200) can obtain a second preference defined in a predetermined driving algorithm for autonomous driving of the robot (100).

[0131] At step (820), the computer system (200) can rank the candidate paths (40) by labeling each of the candidate paths (40) based on the acquired second preference.

[0132] For example, the computer system (200) can automatically determine a score for each of the candidate paths (40) by referring to a short-term path planning evaluation function of a specific short-term path planning algorithm (e.g., a Dynamic Window Approach (DWA) algorithm), and label the rank of each candidate path based on the score.

[0133] Therefore, a compensation model simulating the predetermined driving algorithm (i.e., the DWA algorithm as a short-term path planning algorithm) can be obtained by the subsequent step (640).

[0134] Meanwhile, the method for ranking candidate routes (40) described with reference to FIG. 7 and the method for ranking candidate routes (40) described with reference to FIG. 8 may be used in combination. For example, some of the candidate routes (40) (e.g., N candidate routes with higher or lower rankings) may be ranked based on labeling by the user, and the rest of the candidate routes (40) may be automatically labeled and ranked based on preference information defined in a predetermined driving algorithm.

[0135] In step (640), the computer system (200) may learn a reward model for the candidate paths (40) based on the ranking of the candidate paths (40) described above. For example, the computer system (200) may learn a reward model that associates a relatively higher-ranked candidate path among the candidate paths (40) with a greater number of reward points based on the ranking of the candidate paths (40) described above. This reward model may be a short-term path planning reward model for the robot (100).

[0136] The learned reward model may be configured to input the state of the robot (100) or the robot agent at the first point in time (or a specific point in time) and one of the candidate paths (40) (i.e., a future short-term path), and output a reward score associated with the input candidate path. The reward model may output a higher reward score for a candidate path with a relatively higher rank among the candidate paths (40). A method for learning such a reward model will be described in more detail with reference to FIG. 9, which will be described later. In step (650), the computer system (200) may generate a learning model for autonomous driving of the robot (100) using the learned reward model. For example, the computer system (200) may generate a learning model for short-term path planning of the robot (100), and the robot (100) may be controlled by a driving algorithm implemented through such a learning model. The driving algorithm implemented through the above learning model may reflect the preference information injected in step (630), and the robot (100) may drive in space (10) while reflecting this preference information. For example, the robot (100) may drive in space (10) while reflecting the injected preference information when driving at a point in time corresponding to the first point in time described above.

[0137] In an embodiment, after a computer system (200) collects driving data of a robot (100) or a robot agent, for a predetermined interval section (i.e., a time step) in which preference information is to be injected, a reward model can be learned by labeling preference information such as rankings for possible candidate paths (e.g., ranking each candidate path from the most preferred path to the least preferred path), and thus, a learning model in which preference information is directly reflected can be generated as a model for controlling autonomous driving of the robot (100).

[0138] Meanwhile, the above-described steps (620 to 640) (the determining step, the ranking step, and the learning step) can be performed for each point in time during the driving simulation of the robot (100) or the driving simulation of the robot agent. In other words, the steps (620 to 640) can be performed for each distinct point in time or each time step during the driving simulation of the robot (100) or the driving simulation of the robot agent, and thus, a learning model in which preference information is directly reflected throughout the driving of the robot (100) can be generated as a model for controlling the autonomous driving of the robot (100).

[0139] The description of the technical features described above with reference to FIGS. 1 to 5 can also be applied to FIGS. 6 to 8, so redundant descriptions are omitted.

[0140]

[0141] Figure 9 is a flowchart illustrating a method for learning a reward model for ranked candidate paths, according to an example.

[0142] Referring to FIG. 9, a method of learning a reward model for ranked candidate paths (40) is described in more detail, in which a reward model is learned so that a relatively high-ranking candidate path among the candidate paths (40) is associated with more reward scores.

[0143] This compensation model may be a short-term path planning compensation model for a robot (100).

[0144] At step (910), the computer system (200) may select two candidate paths from among the ranked candidate paths (40).

[0145] In step (920), the computer system (200) can input two selected candidate paths and the state of the robot (100) or robot agent at the first point in time, and train a reward model using a loss function that allows a candidate path with a relatively higher rank among the two selected candidate paths to receive a higher reward score.

[0146] The loss function used for learning the reward model can be defined, for example, by the following mathematical expression 1.

[0147] [Mathematical Formula 1]

[0148]

[0149] In the above mathematical formula, K may be the number of candidate paths (40). can represent a combination (K combination 2) that selects 2 out of K candidate paths. E can be the expected value. "~" can indicate that the left-hand side term follows the distribution of the right-hand side term.

[0150] Above Is It can be a reward function of the above compensation model with as a parameter. Such Is The deep learning model of the above reward function with x as a parameter can be represented. The above x can be the state of the robot (100) or the robot agent at the first point in time. x can be a vector representing the state of the robot (100). The above y w may be a candidate path with a relatively higher rank (i.e., a preferred candidate path) among the two selected candidate paths. The above y l may be a candidate path with a relatively lower rank (i.e., a non-preferred candidate path) among the two candidate paths above. That is, y w is y l It may be the future path of a more preferred robot (100) compared to y l Silver y wmay be a future path of a robot (100) that is less preferred than the path. The path may be composed of multiple points. The above D may represent a data set. The data set D is (x, y w , y l ) may be a data set consisting of a plurality of values ​​(pairs of values). States x exist at several time steps obtained during the movement of the robot (100), and a path candidate group including a plurality of candidate paths can be generated based on x. Two path candidates are randomly selected from among the path candidates and y w , y l can be decided.

[0151] So to speak, D is the above x, the above y w , and the above y l can represent a data set including x, y w , and the above y l Each of them may be (randomly) selected from the above D. (x, y w , y l )~D is an arbitrary (x, y) from D w , y l ) may mean that it is extracted.

[0152] Above can represent a sigmoid function and can be defined as in mathematical expression 3 below.

[0153] The loss function according to the above mathematical expression 1 can be defined to learn the reward model based on calculating the expected value (average) for the values ​​of the mathematical expression 2 below.

[0154] [Equation 2]

[0155]

[0156] [Equation 3]

[0157]

[0158] In other words, the loss function of the above mathematical expression 1 is a multiple (x, yw , y l ) randomly extracted from a data set D consisting of values ​​(pairs of values) (x, y w , y l ) can be defined as a loss function for learning the reward model based on calculating the expected value (average) for the result values ​​calculated by substituting the values ​​(pairs of values) into the above mathematical expression 2 (i.e., variables below the log of mathematical expression 2).

[0159] For example, if the loss function is described by taking the case where each of the two selected candidate paths is to avoid the aforementioned obstacle (30) to the right (No. 1) and to avoid it to the left (No. 2), the loss function can be configured to learn the preference (i.e., the preference relationship) between the candidate paths No. 1 and No. 2. That is, if the candidate path No. 1 is ranked higher than the candidate path No. 2, the loss function is configured to learn the reward ( ) is the reward associated with candidate path 2 ( ) can be trained to always be higher than candidate path 1. Therefore, the reward model can be trained so that candidate path 1 (higher ranked candidate path) always receives a higher reward score than candidate path 2 (lower ranked candidate path).

[0160] The learning model generated using the reward model learned in this way can control the robot (100) by reflecting the injected preference information.

[0161] For example, if there is no bias toward left-right traffic, and people generally prefer right-hand traffic, the robot (100) may want to evade to the right (i.e., the user has a preference for the robot (100) to evade to the right). The computer system (200) may acquire driving data when there is a person or an obstacle on the straight path (at the corresponding point in time, time step), and may label the right-hand evasion path with a higher priority than the left-hand evasion path among candidate paths for the state of the robot (100) (or robot agent), and may train a reward model for the candidate paths using the labeling results. The training model generated using the trained reward model in this way may control the robot (100) by reflecting the injected preference information, that is, generating the right-hand evasion path with a high probability.

[0162] The description of the technical features described above with reference to FIGS. 1 to 8 can also be applied to FIG. 9, so any duplicate description will be omitted.

[0163]

[0164] FIG. 10 illustrates a method for determining candidate paths that further reflect preference information about the state of a robot or robot agent as a reward model is updated, according to an example.

[0165] The methodology of directly injecting preference information and separately training a reward model in the example allows preference information training to be independent of reinforcement learning model training. Furthermore, after training the reward model, even if candidate paths are subsequently generated using any algorithm (other than reinforcement learning-based algorithms), these candidate paths can be updated to reflect the preference information injected for training the aforementioned reward model.

[0166] That is, through the embodiment, a reward model that directly reflects user or predefined preference information can be learned, and by using a learning model for autonomous driving control of a robot (100) based on this reward model, a state (sequence of actions, or candidate path(s)) after reflecting the preference information on the state of the robot (100) can be generated.

[0167] In this way, through the embodiment, not only can the user or predefined preference information be directly reflected in the robot (100), but also the short-term future actions and states of the robot (100) in which the preference information is reflected can be presented to the user as candidate path(s).

[0168] In this regard, referring to steps (1010 to 1030), a method of updating candidate paths to reflect injected preference information is described in more detail.

[0169] In step (1010), the computer system (200) can obtain updated driving data by controlling the driving of the robot (100) within the space (10) or the driving simulation of the robot agent for the space (10) using the learning model generated by steps (610 to 650) of FIG. 6 described above. The description of step (610) described above can be similarly applied to step (1010), and thus, redundant descriptions are omitted. The updated driving data can further reflect the preference information injected by steps (620 to 640) compared to the driving data obtained in step (610), as the robot (100) drives according to the driving algorithm implemented by the generated learning model.

[0170] In step (1020), the computer system (200) can determine a plurality of update candidate paths for the state of the robot (100) or the robot agent at a second point in time during the driving simulation of the robot (100) or the robot agent based on the updated driving data. The description of step (610) described above can be similarly applied to step (1020), and thus, a redundant description will be omitted. The second point in time may be the same as the first point in time (or a time step corresponding to the first point in time), or may be a point in time different from the first point in time.

[0171] These update candidate paths may further reflect the preference information (i.e., the first preference or the second preference) injected by steps (620 to 640) compared to the candidate paths (640) determined in step (620).

[0172] Meanwhile, the aforementioned steps (630 to 650) can be similarly performed for the renewal candidate paths, so that the learning model can be further updated.

[0173] Accordingly, as updates are repeated, the learning model can generate update candidate paths that increasingly reflect the injected preference information, and the driving of the robot (100) controlled using this learning model can also increasingly reflect the injected preference information.

[0174] Meanwhile, the computer system (200) can update the compensation model through a mathematical method to produce more suitable results.

[0175] When training a reward model, the reward model can be trained as a model capable of partial differentiation of variables. For example, the computer system (200) can determine how rewards change (i.e., increase or decrease in reward) depending on changes in variables corresponding to candidate paths, for example, through partial derivatives (slope values). The reward model can be updated by adjusting candidate paths and / or other variables to increase rewards.

[0176] The description of the technical features described above with reference to FIGS. 1 to 9 can also be applied to FIG. 10, so redundant descriptions are omitted.

[0177]

[0178] Figure 11 is a flowchart illustrating a method for visualizing candidate paths for a state of a robot or a robot agent and labeling each candidate path, according to an example. Meanwhile, Figure 12 illustrates a method for visualizing candidate paths for a state of a robot or a robot agent and labeling each candidate path, according to an example.

[0179] In step (1110), the computer system (200) can provide a visual representation of each of a plurality of candidate paths to a user terminal of a user (labeler).

[0180] In step (1120), the computer system (200) may provide the user terminal with a function for ranking each candidate path provided in a visualized manner.

[0181] For example, the computer system (200) can rank a plurality of candidate routes (40) based on the first preference input by the user through the user terminal using the above function, and can rank the candidate routes (40) according to the rank specified by the function as the first preference.

[0182] As shown in FIG. 12, a user interface (UI) (1200) may be provided to the user terminal. The UI (1200) may include a first UI (1210) for visualizing a plurality of candidate paths (1-9), a second UI (1220) for visualizing the current (or, at the first point in time) state of the robot (100) or the robot agent, and a third UI (1230) for designating the rank of each of the plurality of candidate paths (1-9).

[0183] The first UI (1210) may display a future position sequence of the robot (100) or the robot agent corresponding to each candidate path for the state of the robot (100) or the robot agent. For example, each of these candidate paths may represent a candidate path that the robot or the robot agent (R) displayed in the second UI (1220) can select to avoid an obstacle (O). As in the third UI (1230), the user may designate a ranking for each candidate path. For example, as in the illustrated example, the candidate paths may be labeled with rankings such as 4>2>1>3>5>6=7=8=9. The user may identify the state of the robot or the robot agent (R) through the second UI (1220), confirm the candidate paths corresponding to the state through the second UI (1220), and then label the ranking for each candidate path as in the third UI (1230) by reflecting his / her preference information. Next, as previously mentioned, learning of a reward model that reflects preference information can be performed.

[0184] As described, in the embodiment, a reward function can be learned by directly reflecting preference information, by allowing a user to intuitively identify candidate paths and directly rank each candidate path.

[0185] The description of the technical features described above with reference to FIGS. 1 to 10 can also be applied to FIGS. 11 and 12, so redundant descriptions are omitted.

[0186] The systems or devices described above may be implemented as hardware components, software components, or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0187] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0188] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0189] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0190] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for generating a learning model for autonomous driving of a robot, performed by a computer system, A step of obtaining driving data from a driving simulation of the robot within a space or a driving simulation of a robot agent for the space; A step of determining a plurality of candidate paths for the state of the robot or the robot agent at a first point in time during the driving simulation of the robot or the robot agent based on the driving data; A step of ranking the plurality of candidate paths based on a first preference input by a user or a predefined second preference; A step of training a reward model so that a relatively high-ranking candidate path among the above candidate paths is associated with more reward scores; and A step of creating a learning model for autonomous driving of the robot using the above reward model. A method for generating a learning model for autonomous driving of a robot, comprising:

2. In paragraph 1, The above driving data is, A method for generating a learning model for autonomous driving of a robot, the learning model including at least one of sensor data, global path information, and speed information obtained from driving of the robot or driving simulation of the robot agent.

3. In paragraph 1, A method for generating a learning model for autonomous driving of a robot, wherein each candidate path of the above candidate paths represents a subsequent position or position sequence of the robot or the robot agent with respect to the state of the robot or the robot agent at the first point in time.

4. In paragraph 1, The above ranking step ranks the plurality of candidate paths based on the first preference input from the user, A step of receiving, from the user, a preference label for each of the candidate paths as the first preference; and A step of ranking the candidate paths according to the received preference labels. A method for generating a learning model for autonomous driving of a robot, comprising:

5. In paragraph 1, The above ranking step ranks the plurality of candidate paths based on the second preference determined above, A step of acquiring the second preference defined in a predetermined driving algorithm for autonomous driving of the robot; and A step of ranking the candidate paths by labeling each of the candidate paths based on the second preference obtained above. A method for generating a learning model for autonomous driving of a robot, comprising:

6. In paragraph 1, A method for generating a learning model for autonomous driving of a robot, wherein the reward model is configured to input the state of the robot or the robot agent at the first point in time and one of the candidate paths, and output a reward score associated with the candidate path.

7. In paragraph 1, The step of training the above compensation model is: a step of selecting two candidate paths among the above candidate paths; and A step of training the reward model by using the two candidate paths and the state of the robot or the robot agent at the first point in time as inputs, and using a loss function that allows a candidate path with a relatively higher rank among the two candidate paths to receive a higher reward score. A method for generating a learning model for autonomous driving of a robot, comprising:

8. In paragraph 7, The above loss function is defined to learn the reward model based on calculating the expected value for the values of Equation 1, [Mathematical Formula 1] Above Is is the reward function of the above reward model with as a parameter, The above D is the above x, the above y w , and the above y l Represents a data set containing , The above x is the state of the robot or the robot agent at the first point in time selected from the above D, Above y w is a candidate path with a relatively higher rank among the two candidate paths selected from the above D, Above y l is a candidate path with a relatively lower rank among the two candidate paths selected from the above D, Above A method for creating a learning model for autonomous driving of a robot, which represents a sigmoid function.

9. In paragraph 1, Among the above driving data, sensor data is periodically captured during the driving of the robot or the driving simulation of the robot agent, A method for generating a learning model for autonomous driving of a robot, wherein the first point in time is determined based on the point in time at which the sensor data is captured.

10. In paragraph 9, A method for generating a learning model for autonomous driving of a robot, wherein the first point in time represents a point in time at which a long-term path plan of the robot or the robot agent is established based on the captured sensor data, and a short-term path plan of the robot or the robot agent is established based on the captured sensor data and the long-term path plan.

11. In paragraph 1, A method for generating a learning model for autonomous driving of a robot, wherein the determining step, the ranking step, and the learning step are performed for each point in time during driving of the robot or driving simulation of the robot agent.

12. In paragraph 1, The above robot drives in the space based on a reinforcement learning algorithm, and the driving simulation of the robot agent is performed based on the reinforcement learning algorithm. The step of determining the above multiple candidate paths is: A method for generating a learning model for autonomous driving of a robot, based on the above reinforcement learning algorithm, for determining a plurality of candidate paths for the state of the robot or the robot agent at a first point in time.

13. In paragraph 1, A step of obtaining updated driving data by controlling the driving of the robot within the space or the driving simulation of the robot agent for the space using the generated learning model; and A step of determining a plurality of update candidate paths for the state of the robot or the robot agent at a second point in time during the driving simulation of the robot or the robot agent based on the updated driving data. Including more, A method for generating a learning model for autonomous driving of a robot, wherein the above-mentioned update candidate paths more reflect the first preference or the second preference than the above-mentioned candidate paths.

14. In paragraph 1, A step of providing each of the plurality of candidate paths in a visualized manner to the user terminal of the user; and A step of providing the user terminal with a function for ranking each candidate path provided by visualization. Including more, A method for generating a learning model for autonomous driving of a robot, wherein the ranking step ranks the plurality of candidate paths based on the first preference input from the user, and ranks the plurality of candidate paths according to a rank specified by the function as the first preference.

15. A non-transitory computer-readable recording medium storing a computer program for executing the method of paragraph 1 on a computer system.

16. In a computer system that generates a learning model for autonomous driving of a robot, At least one processor implemented to execute computer-readable instructions Including, At least one processor, Obtaining driving data from a driving simulation of the robot within the space or a robot agent driving in the space, Based on the above driving data, a plurality of candidate paths are determined for the state of the robot or the robot agent at a first point in time during the driving simulation of the robot or the robot agent, Ranking the plurality of candidate paths based on a first preference input by the user or a predefined second preference, Train a reward model so that among the above candidate paths, relatively high-ranking candidate paths are associated with more reward scores, A computer system that creates a learning model for autonomous driving of the robot using the above reward model.

Images