Information processing device, information processing method, and program

Abstract

Provided is an information processing device including: an acquisition unit that acquires information indicating the position and orientation of a device and information indicating the position of an obstacle; an inference unit that infers control data sequences for controlling the movement of the device on the basis of the information acquired by the acquisition unit and a trained model generated on the basis of reinforcement learning; a selection unit that selects, from among the plurality of control data sequences inferred by the inference unit, a control data sequence that is predicted not to result in collision with the obstacle; and a control unit that moves the device on the basis of the control data sequence selected by the selection unit.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] Patent Literature 1 discloses a technology for recognizing obstacles based on sensor information, planning a travel route for a mobile device using a behavior planning algorithm that plans a travel route, and switching the behavior planning algorithm based on the travel environment of the mobile device. It is disclosed that the behavior planning algorithm includes commonly known algorithms, such as the Dynamic Window Approach (DWA), reinforcement learning, and Local Trajectory Planner (TLP).

[0003] International Publication No. 2023 / 063094

[0004] However, in the technology described in Patent Document 1, for example, when a reinforcement learning method is used, although it is possible to determine a more appropriate route that is less prone to meandering than the DWA method, there is a problem in that there is a possibility of colliding with an obstacle.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide a technology that can perform more appropriate control for the movement of a device.

[0006] In a first aspect of the present disclosure, there is provided an information processing device having: an acquisition unit that acquires information indicating a position and orientation of an appliance and information indicating a position of an obstacle; an inference unit that infers a control data sequence for controlling movement of the appliance based on the information acquired by the acquisition unit and a trained model generated based on reinforcement learning; a selection unit that selects, from a plurality of control data sequences inferred by the inference unit, a control data sequence that is predicted not to collide with the obstacle; and a control unit that moves the appliance based on the control data sequence selected by the selection unit.

[0007] In addition, a second aspect of the present disclosure provides an information processing method that acquires information indicating the position and orientation of a device and information indicating the position of an obstacle, infers a control data series that controls movement of the device based on the acquired information and a trained model generated based on reinforcement learning, selects a control data series from the inferred plurality of control data series that is predicted not to collide with the obstacle, and moves the device based on the selected control data series.

[0008] In addition, a third aspect of the present disclosure provides a program for causing a computer to execute the following process: acquiring information indicating the position and orientation of a device and information indicating the position of an obstacle; inferring a control data sequence for controlling movement of the device based on the acquired information and a trained model generated based on reinforcement learning; selecting, from the inferred control data sequences, a control data sequence that is predicted not to collide with the obstacle; and moving the device based on the selected control data sequence.

[0009] In addition, a fourth aspect of the present disclosure provides an information processing device having an acquisition unit that acquires obstacle placement information, and a generation unit that generates a plurality of trained models by reinforcement learning based on the information acquired by the acquisition unit to infer a sequence of control data at each point in time during a specific period, and generates a trained model by supervised learning using each control data sequence inferred by each of the plurality of trained models as correct answer data and information indicating the position and orientation of an equipment and information indicating the position of the obstacle as explanatory variables.

[0010] According to one aspect, more appropriate control can be performed for the movement of the device.

[0011] 1 is a diagram illustrating an example of the configuration of an information processing device that performs processing of a learning phase according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an information processing device that performs processing of an inference phase according to an embodiment. FIG. 3 is a diagram illustrating an example of the hardware configuration of an information processing device according to an embodiment. FIG. 4 is a flowchart illustrating an example of learning processing by reinforcement learning of an information processing device according to an embodiment. FIG. 5 is a flowchart illustrating an example of processing of an information processing device according to an embodiment for generating learning data for supervised learning based on an inference result by reinforcement learning. FIG. 6 is a diagram illustrating an example of a learning data DB (Data Base) according to an embodiment. FIG. 7 is a flowchart illustrating an example of learning processing by supervised learning for inferring an inference result by reinforcement learning of an information processing device according to an embodiment. FIG. 8 is a flowchart illustrating an example of processing of an inference phase of an information processing device according to an embodiment. FIG. 9 is a diagram illustrating an example of candidates for a plurality of control data sequences according to an embodiment. FIG. 10 is a diagram illustrating an example of a movement route according to an embodiment and a movement route calculated only by DWA.

[0012] The principles of the present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0013] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each drawing is merely an example for describing one or more embodiments. Each drawing is not related to only one particular embodiment, but may also be related to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessarily required to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0015] <Configuration> <<Configuration of information processing device 10 that performs processing in the learning phase>> The configuration of the information processing device 10 that performs processing in the learning phase according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 10 that performs processing in the learning phase according to the embodiment. The information processing device 10 has an acquisition unit 11 and a generation unit 12. These units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and memory of the information processing device 10. The information processing device 10 may be, for example, an information processing device such as a server or a cloud server.

[0016] The acquisition unit 11 acquires obstacle placement information. The generation unit 12 generates, by reinforcement learning, a plurality of trained models for inferring a sequence of control data at each time point in a specific period, based on the information acquired by the acquisition unit 11. Furthermore, the generation unit 12 may generate trained models by supervised learning, for example, using each control data sequence inferred by each of the plurality of trained models as correct answer data, and using information indicating the position and orientation of the device and information indicating the position of the obstacle as explanatory variables.

[0017] <<Configuration of Information Processing Device 20 Performing Processing in Inference Phase>> With reference to FIG. 2 , the configuration of the information processing device 20 performing processing in the inference phase according to the embodiment will be described. FIG. 2 is a diagram illustrating an example of the configuration of the information processing device 20 performing processing in the inference phase according to the embodiment. The information processing device 20 includes an acquisition unit 21, an inference unit 22, a selection unit 23, and a control unit 24. These units may be implemented by cooperation between one or more programs installed in the information processing device 20 and hardware such as a processor and memory of the information processing device 20. The information processing device 20 may be, for example, an information processing device such as a microcomputer, a personal computer, a server, or a cloud server. The information processing device 20 controls the movement of a device and moves the device autonomously. The device may be, for example, a robot that moves on wheels or legs, a vehicle, an unmanned aerial vehicle (e.g., a drone), or the like. The information processing device 20 may be mounted on the device. Alternatively, the information processing device 20 may be provided external to the device and control the device via wireless communication.

[0018] The acquisition unit 21 acquires information indicating the position and orientation of the device and information indicating the position of an obstacle. The inference unit 22 infers a control data sequence for controlling the movement of the device based on the information acquired by the acquisition unit 21 and a trained model generated based on reinforcement learning.

[0019] The selection unit 23 selects a control data series that is predicted not to collide with an obstacle from among the plurality of control data series inferred by the inference unit 22. The control unit 24 moves the equipment based on the control data series selected by the selection unit 23. This makes it possible to perform, for example, more appropriate control for the movement of the equipment.

[0020] <Hardware Configuration> Fig. 3 is a diagram showing an example of the hardware configuration of the information processing device 10 and the information processing device 20 according to the embodiment. In the example of Fig. 3, the information processing device 10 and the information processing device 20 (computer 100) include a processor 101, a memory 102, and a communication interface 103. These components may be connected via a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0021] When the program 104 is executed by the processor 101, memory 102, and other components in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type. As a non-limiting example, the memory 102 may be a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may be present in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and, as a non-limiting example, a processor based on a multi-core processor architecture. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0022] Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

[0023] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor or device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0024] The program code for executing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions / acts in the flowcharts and / or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0025] The program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disk media include, for example, Blu-ray discs, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), and CD-RW (Rewritable). Semiconductor memory includes, for example, solid-state drives, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0026] <Processing> <<Learning Phase>> Next, an example of a learning process by reinforcement learning of the information processing device 10 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of a learning process by reinforcement learning of the information processing device 10 according to the embodiment.

[0027] The information processing device 10 repeats the process of FIG. 4 N times (N is an integer equal to or greater than 2) to generate N trained models through reinforcement learning.

[0028] In step S101, the acquisition unit 11 acquires obstacle location information (course information). The obstacle location information may include, for example, map information including information on the position and range of one or more obstacles on a horizontal plane, information indicating the initial position and orientation of the device, and information indicating the destination of the device. The initial position and orientation of the device are the position and orientation of the device at a time before the device starts moving based on the control data. The destination of the device is the location to which the device should move.

[0029] The obstacle placement information may be registered in advance in the information processing device 10 by, for example, an operator (administrator), etc. Alternatively, the obstacle placement information may be randomly generated by the acquisition unit 11.

[0030] Next, the acquisition unit 11 acquires reward information for reinforcement learning (step S102). The reward information may include, for example, information on a Q function for calculating a Q value, which is a numerical value indicating the value of an action a in a specific state s. The reward based on the Q function may be given, for example, as +1 when the robot reaches the destination and -1 when the robot collides with an obstacle. Note that the reward information may be registered in advance in the information processing device 10 by, for example, an operator.

[0031] Next, the generation unit 12 generates a trained model for inferring a sequence of control data at each time point during a specific period by reinforcement learning (step S103). For example, if the specific period is 7.5 seconds and the time points are at 0.5-second intervals, the trained model calculates 15 (=7.5 / 0.5) steps of control data.

[0032] Here, the generation unit 12 may perform, for example, reinforcement learning to approximate the Q function using a neural network. This allows the generation of a trained model that infers (calculates, estimates) the action a that maximizes the Q value in state s. The generation unit 12 may use, for example, at least one of DQN (Deep Q-Network) and PPO (Proximal Policy Optimization) as the reinforcement learning method. Note that the generation unit 12 may use, for example, any method that is currently or will be publicly known as the reinforcement learning method. Furthermore, the generation unit 12 may generate each of the N trained models using a different reinforcement learning method. This, for example, further diversifies the inferred control data sequence. In this case, the generation unit 12 may generate, for example, the first trained model using DQN and the second trained model using PPO. Furthermore, the generation unit 12 may, for example, generate the first trained model using DQN, generate the second trained model using PPO, and generate the third trained model using another reinforcement learning method.

[0033] For example, the generation unit 12 may complete reinforcement learning and output one trained model after performing learning a specific number of times (for example, 100,000 times).

[0034] (Example of supervised learning based on inference results by reinforcement learning) Next, an example of processing in the learning phase of the information processing device 10 according to the embodiment will be described with reference to Figs. 5 to 7 . Fig. 5 is a flowchart showing an example of processing in the information processing device 10 according to the embodiment to generate learning data for supervised learning based on inference results by reinforcement learning. Fig. 6 is a diagram showing an example of a learning data DB (Data Base) 601 according to the embodiment. Fig. 7 is a flowchart showing an example of learning processing by supervised learning for inferring an inference result by reinforcement learning of the information processing device 10 according to the embodiment.

[0035] Note that by performing supervised learning based on the inference results of reinforcement learning described below, it is possible to complete calculations more quickly than when inferring a control data sequence using N trained models generated by the process in Figure 4. Furthermore, the processing load can be reduced, leading to power savings.

[0036] (Process for generating learning data for supervised learning based on inference results from reinforcement learning) In step S201, the acquisition unit 11 acquires obstacle placement information. The process of step S201 may be the same as the process of step S101 in FIG. 4.

[0037] Next, the acquisition unit 11 acquires, as status data, information indicating the position and orientation of the device and information indicating the position of an obstacle (step S202). The position and orientation of the device are the position and orientation of the device on a horizontal plane or in three-dimensional space. The information indicating the position of the obstacle may include, for example, information on the distance and direction from the device to the obstacle. The acquisition unit 11 may calculate the distance and direction from the device to the obstacle based on the obstacle placement information and the position and orientation of the device. Note that the acquisition unit 11 may determine the position and orientation of the device based on, for example, initial values ​​of the position and orientation of the device and an arbitrary control data sequence. Note that the control data sequence of the present disclosure may include information indicating a combination of the moving speed and angular velocity of the device at each time point.

[0038] Next, the generation unit 12 infers N control data sequences for controlling the movement of the device (step S203). Here, the generation unit 12 may calculate each control data sequence using, for example, the N trained models generated by the process of FIG.

[0039] Next, the generation unit 12 records a combination of the state data and the inference result as training data in the training data DB 601 (step S204). In the example of Fig. 6, the training data DB 601 records a first control data sequence, a second control data sequence, ..., and an Nth control data sequence in association with the state data. Each of the first control data sequence, the second control data sequence, ..., and the Nth control data sequence is an inference result by each of the N trained models generated by the process of Fig. 4.

[0040] Next, the generating unit 12 determines whether to end the generation of the learning data (step S205). Here, the generating unit 12 may determine to end the generation of the learning data when, for example, a predetermined number of records (enrolled data) are recorded in the learning data DB 601.

[0041] If it is determined that the generation of the learning data is to be ended (YES in step S205), the generation unit 12 ends the processing in Fig. 5. On the other hand, if it is determined that the generation of the learning data is not to be ended (NO in step S205), the generation unit 12 determines whether to reset the course (step S206). Here, the generation unit 12 may determine to reset the course, for example, when the device collides with an obstacle or when the device is located near the end (destination) of the course (map) based on the obstacle location information.

[0042] If it is determined that the course should be reset (YES in step S206), the process proceeds to step S201 to acquire new obstacle placement information. On the other hand, if it is determined that the course should not be reset (NO in step S206), the process proceeds to step S202.

[0043] (Learning process by supervised learning for inferring inference results by reinforcement learning) In step S301, the acquisition unit 11 acquires learning data recorded in the learning data DB 601. Here, the acquisition unit 11 acquires a data set of combinations of state data and a first control data sequence, a second control data sequence, ..., and an Nth control data sequence.

[0044] Next, the generator 12 generates a trained model through supervised learning based on the training data (step S302). Here, the generator 12 performs supervised learning using the state data as explanatory variables and the first control data sequence, the second control data sequence, ..., and the Nth control data sequence as correct answer data (objective variables). This allows the generation of a trained model that infers (calculates) N control data sequences for a given state data. The generator 12 may use, for example, a neural network (deep learning) as a method of supervised learning.

[0045] <<Inference Phase>> Next, an example of processing in the inference phase of the information processing device 20 according to the embodiment will be described with reference to Figs. 8 to 10. Fig. 8 is a flowchart showing an example of processing in the inference phase of the information processing device 20 according to the embodiment. Fig. 9 is a diagram showing an example of candidates for a plurality of control data sequences according to the embodiment. Fig. 10 is a diagram showing an example of a travel route according to the embodiment and a travel route calculated only by DWA.

[0046] In step S401, the acquisition unit 21 acquires status data including information indicating the position and orientation of the device and information indicating the position of an obstacle. Here, the acquisition unit 21 may acquire the information indicating the position and orientation of the device using, for example, a satellite positioning system such as a Global Positioning System (GPS) and an attitude sensor. The acquisition unit 21 may also calculate the position and orientation of the device based on, for example, the initial position of the device and one or more control data sequences that have controlled the device since the device was located at the initial position until the present. The acquisition unit 21 may also calculate the position and orientation of the device based on, for example, an image captured by a camera mounted on the device and preset three-dimensional map data.

[0047] The acquisition unit 21 may also acquire information indicating the position of an obstacle using a sensor such as a stereo camera or a LiDAR (Light Detection and Ranging) sensor mounted on the device.

[0048] Next, the inference unit 22 infers N control data sequences for controlling the movement of the device based on the state data acquired by the acquisition unit 11 and the trained model generated based on reinforcement learning (step S402). Here, the inference unit 22 may infer the N control data sequences using, for example, each of the N trained models generated by N rounds of the process of Figure 4. In this case, the inference unit 22 repeats the process of inferring the ith control data sequence based on the information acquired by the acquisition unit 21 and the ith trained model generated by reinforcement learning, where i is 1 to N.

[0049] The inference unit 22 may infer N control data sequences using, for example, one trained model generated by the processing of Fig. 7. The trained model is a trained model generated by supervised learning using the first control data sequence, the second control data sequence, ..., and the Nth control data sequence calculated by reinforcement learning as correct answer data, and information indicating the position and orientation of the device and information indicating the position of an obstacle as explanatory variables.

[0050] Next, the selection unit 23 selects control data sequences other than the control data sequence predicted to result in a collision with an obstacle from among the plurality of control data sequences inferred by the inference unit 22 (step S403). Here, the selection unit 23 may predict whether or not a collision with an obstacle will occur for each of the plurality of control data sequences inferred by the inference unit 22 based on a motion model of the device (e.g., a two-wheel model, etc.).

[0051] Furthermore, the selection unit 23 may calculate a value of the evaluation function for each of the plurality of control data sequences inferred by the inference unit 22. Then, the selection unit 23 may select, from the plurality of control data sequences inferred by the inference unit 22, a control data sequence that is predicted not to collide with an obstacle based on the motion model of the appliance and has the highest value of the evaluation function. In this case, the selection unit 23 may, for example, calculate a higher value of the evaluation function the smaller the difference between the movement direction of the appliance according to the control data sequence and the direction from the appliance to the destination. Furthermore, the selection unit 23 may, for example, calculate a higher value of the evaluation function the greater the distance between the movement position of the appliance according to the control data sequence and the nearest obstacle. Furthermore, the selection unit 23 may, for example, calculate a higher value of the evaluation function the greater the movement speed of the appliance according to the control data sequence.

[0052] When calculating a control data sequence, which is a sequence of control data at each point in a specific period (prediction period) using DWA, the specific period needs to be set long to avoid falling into a local solution, but if the specific period is too long, efficiency (e.g., average movement speed) decreases. On the other hand, when calculating (inferring) a control data sequence using reinforcement learning, although the average movement speed can be improved compared to when using DWA, the inference process is black-box-like and there is no mechanism to logically prevent the device from colliding with an obstacle. According to the present disclosure, for example, a control data sequence is calculated using each of multiple reinforcement learning results, and for each of one or more control data sequences, it is predicted whether the device will collide with an obstacle. Then, of the one or more control data sequences, the control data sequence that is predicted not to collide with an obstacle is used to move the device. This allows, for example, more appropriate control of the device's movement.

[0053] If the selection unit 23 predicts that a collision with an obstacle will occur for all of the multiple control data sequences inferred by the inference unit 22, the selection unit 23 may generate a control data sequence using DWA. DWA is a method for calculating control inputs conforming to a motion model of a device such as a robot, based on the current device position and movement speed, obstacle information, and destination position information. DWA first determines the range of control inputs that the device can implement at the next time (e.g., the current movement speed, and the upper and lower limits of the movement speed and angular velocity specified in the device's specifications). Then, it determines the allowable control inputs that will prevent a collision with an obstacle (the range of control inputs that will allow movement without colliding with an obstacle, based on the observation results from an obstacle sensor, etc., and the device's maximum deceleration). This allows the selection unit 23 to identify a control data sequence that will allow movement without colliding with an obstacle.

[0054] Then, the selection unit 23 may output the control data series as the selected control data series to the control unit 24. As a result, even if all of the multiple control data series resulting from the reinforcement learning collide with an obstacle, for example, it is possible to move the equipment using a control data series that does not logically collide.

[0055] Next, the control unit 24 moves the device based on the control data series selected by the selection unit 23 (step S404).

[0056] The example of Figure 9 shows an example of a plurality of control data sequence candidates according to the embodiment. The example of Figure 9 shows an example of a bird's-eye view of a course along which a device moves, with obstacles 901-904 and the like arranged along the course. Also shown is an example of a movement route 912 from the device's initial position 910 to a destination 911. When the device is located at point 921 on the movement route 912, the inference unit 22 infers control data sequences corresponding to each of route candidates 931-933 and the like. Then, the selection unit 23 selects the control data sequence that results in the shortest movement route, and the device moves along the movement route 912.

[0057] The example of Fig. 10 shows an example of a travel route according to the embodiment and a travel route calculated only by DWA. The example of Fig. 10 shows an example of a bird's-eye view of a course along which a device travels, with obstacles 1001-1003 and the like arranged along the course. Also shown are examples of travel routes 1011-1013 along which the device can travel from an initial position 1010 within a specific time (e.g., 50 seconds).

[0058] Movement path 1011 is an example of a movement path calculated only by DWA when the prediction period is relatively long (e.g., 7.5 seconds). Movement path 1012 is an example of a movement path calculated only by DWA when the prediction period is relatively short (e.g., 2.5 seconds). Movement path 1013 is an example of a movement path based on a control data sequence selected from a group of control data sequences (e.g., a sequence of control data controlled 15 times at 0.5 second intervals) inferred by reinforcement learning using the technology of the present disclosure. It can be seen that the movement path 1013 based on the technology of the present disclosure reaches a position closest to the destination compared to movement paths 1011 to 1012.

[0059] <Modifications> The information processing device 10 and the information processing device 20 may each be devices contained in a single housing, but the information processing device of the present disclosure is not limited to this. Each unit of the information processing device 10 and each unit of the information processing device 20 may be realized by cloud computing configured with one or more computers, for example. Furthermore, the information processing device 10 and the information processing device 20 may be the same information processing device. Such an information processing device 10 is also included as an example of the "information processing device" of the present disclosure.

[0060] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0061] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0062] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. Note that some or all of the elements (e.g., configurations and functions) described in each supplementary note dependent on supplementary note 1 may also be dependent on independent supplementary notes in other categories in a similar dependency relationship. Some or all of the elements described in any supplementary note may be applied to various hardware, software, and recording means, systems, and methods for recording software. (Supplementary note 1) An information processing device comprising: an acquisition unit that acquires information indicating a position and orientation of a device and information indicating a position of an obstacle; an inference unit that infers a control data sequence for controlling movement of the device based on the information acquired by the acquisition unit and a trained model generated based on reinforcement learning; a selection unit that selects, from a plurality of control data sequences inferred by the inference unit, a control data sequence that is predicted not to collide with the obstacle; and a control unit that moves the device based on the control data sequence selected by the selection unit. (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the selection unit predicts whether or not the device will collide with the obstacle for each of a plurality of control data sequences inferred by the inference unit based on a motion model of the device. (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the control data sequences selected by the selection unit include information indicating a combination of a moving speed and an angular velocity of the device at each time point. (Supplementary Note 4) The information processing device according to Supplementary Note 1 or 2, wherein the inference unit infers a first control data sequence based on the information acquired by the acquisition unit and a first trained model generated by reinforcement learning, and infers a second control data sequence based on the information acquired by the acquisition unit and a second trained model generated by reinforcement learning. (Supplementary Note 5) The information processing device according to Supplementary Note 4, wherein the first trained model and the second trained model are each generated by a different reinforcement learning method.(Supplementary Note 6) The information processing device according to Supplementary Note 1 or 2, wherein the trained model is a trained model generated by supervised learning using a first control data sequence and a second control data sequence calculated by reinforcement learning as correct answer data, and information indicating the position and orientation of the device and information indicating the position of the obstacle as explanatory variables. (Supplementary Note 7) The information processing device according to Supplementary Note 2, wherein the selection unit selects a control data sequence generated using a dynamic window approach (DWA) when a motion model of the device predicts that all of the control data sequences inferred by the inference unit will result in a collision with an obstacle. (Supplementary Note 8) An information processing method comprising: acquiring information indicating the position and orientation of the device and information indicating the position of an obstacle; inferring a control data sequence for controlling movement of the device based on the acquired information and a trained model generated based on reinforcement learning; selecting a control data sequence predicted not to result in a collision with the obstacle from the inferred control data sequences; and moving the device based on the selected control data sequence. (Supplementary Note 9) A program that causes a computer to execute the following processes: acquire information indicating the position and orientation of a device and information indicating the position of an obstacle, infer a control data sequence for controlling movement of the device based on the acquired information and a trained model generated based on reinforcement learning, select a control data sequence that is predicted not to collide with the obstacle from among the inferred control data sequences, and move the device based on the selected control data sequence. (Supplementary Note 10) An information processing device comprising: an acquisition unit that acquires obstacle placement information, and a generation unit that generates a plurality of trained models by reinforcement learning based on the information acquired by the acquisition unit to infer a control data sequence at each time point in a specific period, using each control data sequence inferred by each of the plurality of trained models as correct answer data, and generate a trained model by supervised learning using information indicating the position and orientation of the device and information indicating the position of the obstacle as explanatory variables.(Supplementary Note 11) An information processing method comprising: acquiring obstacle placement information; generating a plurality of trained models by reinforcement learning based on the acquired information to infer a sequence of control data at each time point in a specific period; using each control data sequence inferred by each of the plurality of trained models as correct answer data, and information indicating the position and orientation of an appliance and information indicating the position of the obstacle as explanatory variables to generate a trained model by supervised learning. (Supplementary Note 12) A program that causes a computer to execute processes of acquiring obstacle placement information; generating a plurality of trained models by reinforcement learning based on the acquired information to infer a sequence of control data at each time point in a specific period; using each control data sequence inferred by each of the plurality of trained models as correct answer data, and information indicating the position and orientation of an appliance and information indicating the position of the obstacle as explanatory variables to generate a trained model by supervised learning.

[0063] This application claims priority based on Japanese Patent Application No. 2024-028359, filed February 28, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0064] REFERENCE SIGNS LIST 10 Information processing device 11 Acquisition unit 12 Generation unit 20 Information processing device 21 Acquisition unit 22 Inference unit 23 Selection unit 24 Control unit

Claims

1. An information processing device having: an acquisition unit that acquires information indicating the position and orientation of a device and information indicating the position of an obstacle; an inference unit that infers a control data sequence that controls movement of the device based on the information acquired by the acquisition unit and a trained model generated based on reinforcement learning; a selection unit that selects, from multiple control data sequences inferred by the inference unit, a control data sequence that is predicted not to collide with the obstacle; and a control unit that moves the device based on the control data sequence selected by the selection unit.

2. The information processing device according to claim 1, wherein the selection unit predicts whether or not a collision with the obstacle will occur for each of a plurality of control data sequences inferred by the inference unit based on a motion model of the device.

3. The information processing device according to claim 1 or 2, wherein the control data sequence selected by the selection unit includes information indicating a combination of the moving speed and angular velocity of the device at each point in time.

4. An information processing device as described in claim 1 or 2, wherein the inference unit infers a first control data sequence based on the information acquired by the acquisition unit and a first learned model generated by reinforcement learning, and infers a second control data sequence based on the information acquired by the acquisition unit and a second learned model generated by reinforcement learning.

5. The information processing device according to claim 4, wherein the first trained model and the second trained model are generated using different reinforcement learning methods.

6. The information processing device according to claim 1 or 2, wherein the trained model is a trained model generated by supervised learning using the first control data series and the second control data series calculated by reinforcement learning as correct answer data, and information indicating the position and orientation of the device and information indicating the position of the obstacle as explanatory variables.

7. The information processing device according to claim 2, wherein the selection unit selects a control data sequence generated using a dynamic window approach (DWA) when it predicts, based on a motion model of the device, that a collision with an obstacle will occur for all of the multiple control data sequences inferred by the inference unit.

8. An information processing method comprising: acquiring information indicating the position and orientation of a device and information indicating the position of an obstacle; inferring a control data sequence for controlling movement of the device based on the acquired information and a trained model generated based on reinforcement learning; selecting, from the inferred control data sequences, a control data sequence that is predicted not to result in a collision with the obstacle; and moving the device based on the selected control data sequence.

9. A program that causes a computer to execute the following process: acquire information indicating the position and orientation of a device and information indicating the position of an obstacle; infer a control data sequence that controls the movement of the device based on the acquired information and a trained model generated based on reinforcement learning; select, from the inferred multiple control data sequences, a control data sequence that is predicted not to collide with the obstacle; and move the device based on the selected control data sequence.

10. An information processing device having: an acquisition unit that acquires obstacle placement information; and a generation unit that generates a plurality of trained models for inferring a sequence of control data at each point in time during a specific period by reinforcement learning based on the information acquired by the acquisition unit, and generates a trained model by supervised learning using each control data sequence inferred by each of the plurality of trained models as correct answer data and information indicating the position and orientation of a device and information indicating the position of the obstacle as explanatory variables.

11. An information processing method comprising: acquiring information on the placement of obstacles; generating, based on the acquired information, a plurality of trained models by reinforcement learning for inferring a sequence of control data at each point in time during a specific period; using each control data sequence inferred by each of the plurality of trained models as correct answer data; and generating a trained model by supervised learning using information indicating the position and orientation of equipment and information indicating the position of the obstacle as explanatory variables.

12. A program that causes a computer to execute the following process: acquire information about the placement of obstacles; generate, based on the acquired information, a plurality of trained models through reinforcement learning to infer a sequence of control data at each point in time during a specific period; use each control data sequence inferred by each of the plurality of trained models as correct answer data; and generate a trained model through supervised learning using information indicating the position and orientation of the equipment and information indicating the position of the obstacle as explanatory variables.

Images