Driving model generating device, driving control device, driving device, driving model generating method, driving control method, and program

Abstract

To provide a technology for improving the accuracy of automatic travel from an initial position to a target position. [Solution] Reinforcement learning is performed using the initial position and target position of a moving object in a virtual three-dimensional space, the state and action of the moving object for each step of movement from the initial position to the target position, and rewards corresponding to the actions, in order to increase the reward, and a moving model is generated that outputs the speed and angular velocity of the moving object in the movement direction when a state including an image of the moving direction of the moving object at its current position is input. At this time, reinforcement learning is repeated using state information including a virtual captured image in which the viewpoint position of the moving object is changed and each pixel shows a value either on the road or off the road, to generate the moving model.

Description

[Technical Field]

[0001] The present disclosure relates to a driving model generation device, a driving control device, a driving device, a driving model generation method, a driving control method, and a program. [Background technology]

[0002] Much research is being conducted to improve technology related to the travel control of moving vehicles. For example, Patent Document 1 discloses a technology for creating a learning model by processing images taken by multiple cameras attached to a vehicle traveling on rails. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-8345 Summary of the Invention [Problem to be solved by the invention]

[0004] In the field of travel control technology for moving objects, there is a demand for technology that improves the accuracy of automatic travel from an initial position to a target position.

[0005] An object of the present disclosure is to provide a driving model generation device, a driving control device, a driving device, a driving model generation method, a driving control method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0006] A driving model generation device according to one aspect of the present disclosure includes a driving model generation means that performs reinforcement learning using an initial position and a target position of a moving body in a virtual three-dimensional space, a state and action of the moving body for each step of movement from the initial position to the target position, and a reward corresponding to the action, in order to increase the reward, and generates a driving model that outputs the speed of the moving body and the angular velocity in the movement direction when the state including an image of the moving direction of the moving body at its current position is input, and the driving model generation means generates the driving model by repeating the reinforcement learning using information about the state including a virtual captured image in which the viewpoint position of the moving body is changed, and in which each pixel shows a value either of the driving path or a value other than the driving path.

[0007] A traveling model generation device according to one aspect of the present disclosure performs reinforcement learning in which a reward increases using a virtual initial position and a virtual target position of a virtual traveling body in a virtual three-dimensional space, a state and an action of the virtual traveling body for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual traveling body and an angular velocity in the movement direction when a state including a virtual current position and a virtual photographed image of the movement direction of the virtual traveling body at the virtual current position is input, and the traveling model generation device includes a virtual photographed image in which each pixel of the virtual photographed image indicates a value of either a traveling path or a value other than a traveling path. a storage means for storing the traveling model generated by repeating the reinforcement learning using the state information including the actual initial position and actual target position of the traveling device traveling in real space; a processed image obtained by segmenting a photographed image of the traveling direction of the current position of the traveling device in the movement from the actual initial position to the real target position into a traveling path and a non-traveling path; input information including the state of the traveling device based on the current position and the actual target position; and a traveling control means for outputting output information including the velocity and angular velocity at the current position using the traveling model, and for controlling the traveling of the traveling device using the output information.

[0008] A traveling device according to one aspect of the present disclosure is a traveling model that performs reinforcement learning in which the reward increases using a virtual initial position and a virtual target position of a virtual traveling body in a virtual three-dimensional space, a state and an action of the virtual traveling body for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual traveling body and an angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual traveling body at the virtual current position is input, and the state information including a virtual photographed image in which each pixel of the virtual photographed image shows either a value corresponding to a traveling path or a value other than a traveling path is input. and a travel control means for outputting, using the travel model, output information including a speed and an angular velocity at the current position, and controlling the travel of the travel device using the output information.

[0009] A driving model generation method according to one aspect of the present disclosure performs reinforcement learning using an initial position and a target position of a moving body in a virtual three-dimensional space, a state and action of the moving body for each step of movement from the initial position to the target position, and a reward corresponding to the action, in order to increase the reward, and generates a driving model that outputs the speed of the moving body and the angular velocity in the movement direction when the state including an image of the moving direction of the moving body at its current position is input, and in generating the driving model, the reinforcement learning is repeated using state information including a virtual captured image in which the viewpoint position of the moving body is changed and each pixel shows a value either of the driving path or a value other than the driving path, to generate the driving model.

[0010] A travel control method according to one aspect of the present disclosure is a travel model that performs reinforcement learning in which a reward increases using a virtual initial position and a virtual target position of a virtual travel object in a virtual three-dimensional space, a state and an action of the virtual travel object for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual travel object and an angular velocity in the movement direction when a state including a virtual current position and a virtual photographed image of the movement direction of the virtual travel object at the virtual current position is input, and each pixel of the virtual photographed image indicates a value of either a travel path or a value other than a travel path. The driving model generated by repeating the reinforcement learning using the state information including the virtual photographed images is stored, and output information including the actual initial position and actual target position of the driving device driving in real space, processed images obtained by segmenting photographed images of the direction of movement of the current position of the driving device as it moves from the actual initial position to the actual target position into a driving path and a non-driving path, input information including the state of the driving device based on the current position and the actual target position, and the velocity and angular velocity at the current position using the driving model is output, and driving control of the driving device is performed using the output information.

[0011] A program according to one aspect of the present disclosure causes a computer of a driving model generation device to function as driving model generation means that performs reinforcement learning using an initial position and a target position of a moving body in a virtual three-dimensional space, the state and action of the moving body for each step of movement from the initial position to the target position, and a reward corresponding to the action, in order to increase the reward, and generates a driving model that outputs the speed of the moving body and the angular velocity in the movement direction when the state including an image of the moving direction of the moving body at its current position is input, and the driving model generation means generates the driving model by repeating the reinforcement learning using information about the state including a virtual captured image in which the viewpoint position of the moving body has been changed, and in which each pixel shows a value either of the driving path or a value other than the driving path.

[0012] A program according to one aspect of the present disclosure is a traveling model that performs reinforcement learning in which a reward increases using a virtual initial position and a virtual target position of a virtual traveling body in a virtual three-dimensional space, a state and an action of the virtual traveling body for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual traveling body and an angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual traveling body at the virtual current position is input, and the program uses information on the state including a virtual photographed image in which each pixel of the virtual photographed image shows a value of either a traveling path or a value other than a traveling path. The computer of the driving control device, which is equipped with a storage means for storing the driving model generated by repeating the reinforcement learning, functions as a driving control means that outputs input information including the actual initial position and actual target position of the driving device driving in real space, processed images obtained by segmenting photographed images of the direction of movement of the current position of the driving device as it moves from the actual initial position to the actual target position into a driving path and an area other than the driving path, and the state of the driving device based on the current position and the actual target position, and output information including the velocity and angular velocity at the current position using the driving model, and controls the driving of the driving device using the output information. [Effects of the Invention]

[0013] According to the above aspect, it is possible to improve the accuracy of automatic traveling of the traveling object from the initial position to the target position. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic block diagram of a cruise control system according to one embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating a hardware configuration of a driving model generation device according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a functional block diagram of a driving model generation device according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a diagram illustrating an example of spatial data of a virtual three-dimensional space according to an embodiment of the present disclosure. [Figure 5]FIG. 10 is a diagram illustrating an example of a virtually captured image according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a diagram illustrating a processing flow of a driving model generation device according to an embodiment of the present disclosure. [Figure 7] 1 is a functional block diagram of a cruise control device according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a diagram illustrating an outline of processing performed by a cruise control device according to an embodiment of the present disclosure. [Figure 9] FIG. 3 is a diagram illustrating a processing flow of a driving control device according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a first diagram illustrating an example result of travel control of a travel device according to an embodiment of the present disclosure. [Figure 11] FIG. 10 is a second diagram illustrating an example result of the travel control of the travel device according to the embodiment of the present disclosure. [Figure 12] FIG. 1 is a diagram illustrating a minimum configuration of a driving model generation device according to an embodiment of the present disclosure. [Figure 13] FIG. 1 is a diagram illustrating a minimum configuration of a cruise control device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Each embodiment will be described below with reference to the drawings. FIG. 1 is a schematic block diagram of a cruise control system according to one embodiment of the present disclosure. As an example, a driving control system 100 of the present disclosure includes a driving model generation device 1 and a driving device 2, as shown in FIG. 1 . The driving model generation device 1 generates a driving model used for automatic driving control of the driving device 2. The driving device 2 includes a driving control device 20 and a photographing device 3. The driving device 2 also includes a drive mechanism for traveling from an initial position to a target position. The drive mechanism may include wheels, a motor, a steering mechanism that determines the direction of travel, etc. The driving control device 20 controls the automatic driving of the driving device 2 using the state of the driving device 2 (information including the distance to the target position and the angular difference between the direction to the target position and the direction of travel), images acquired from the photographing device 3, and the driving model generated by the driving model generation device 1. The photographing device 3 mounted on the driving device 2 is, for example, a camera. The photographing device 3 may also be a smartphone equipped with a camera.

[0016] FIG. 2 is a diagram illustrating a hardware configuration of a driving model generation device according to an embodiment of the present disclosure. 2, the driving model generation device 1 is a computer equipped with hardware such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage device 104, a communication module 105, and an input device 106. The driving control device 20 is also a computer equipped with similar hardware.

[0017] FIG. 3 is a functional block diagram of a driving model generation device according to an embodiment of the present disclosure. The driving model generating device 1 executes a pre-stored driving model generating program, and is thus equipped with a control unit 11, a spatial data generating unit 12, an image acquiring unit 13, and a driving model generating unit . The control unit 11 controls the other functional units. The space data generator 12 generates data of a virtual three-dimensional space. The image acquisition unit 13 acquires a virtual captured image according to the position and moving direction of the virtual traveling object in the virtual three-dimensional space. The traveling model generation unit 14 performs reinforcement learning in which the reward increases using the initial position and target position of the virtual traveling body in the virtual three-dimensional space, the state and action of the virtual traveling body for each movement step from the initial position to the target position, and the reward corresponding to the action. By performing this reinforcement learning, the traveling model generation unit 14 generates a traveling model that outputs the speed of the traveling body and the angular velocity for rotating in the movement direction when a state including a virtual captured image of the traveling body's movement direction at its current position is input.

[0018] FIG. 4 is a diagram illustrating an example of space data of a virtual three-dimensional space according to an embodiment of the present disclosure. The spatial data generation unit 12 generates spatial data of a virtual three-dimensional space based on a user's input operation. For example, the spatial data generation unit 12 receives input such as coordinates of three-dimensional space, such as the floor, walls, and ceiling of the space, and generates spatial data of the virtual three-dimensional space, which is spatial data of the virtual three-dimensional space and indicates, using a binary value, whether the surface represents the floor (road surface) in the virtual three-dimensional space or a surface other than the floor (road surface). The spatial data generation unit 12 also receives input of multiple virtual positions in the virtual three-dimensional space, each specifying a position where a virtual vehicle will travel, and generates spatial data of the virtual three-dimensional space including the virtual positions. The multiple virtual positions may be set at various dispersed positions on the floor (roadway) of the virtual space, as indicated by circles in FIG. 4. The spatial data generation unit 14 can be implemented, for example, by using a program called Unity (registered trademark) that can generate spatial data of a virtual three-dimensional space. The spatial data generation unit 14 may also generate spatial data using other programs.

[0019] FIG. 5 is a diagram illustrating an example of a virtually captured image according to an embodiment of the present disclosure. The image acquisition unit 13 acquires from the space data generation unit 12 a virtual captured image that virtually captures a moving direction based on the position and moving direction of a virtual traveling object in a virtual three-dimensional space. The image acquisition unit 13 outputs the virtual captured image to the traveling model generation unit 14. Each pixel of this virtual captured image contains information that indicates, in binary form, whether the pixel is a surface that indicates a floor (road surface) or a surface that indicates something other than a floor (road surface). The image acquisition unit 13 generates a virtual captured image of each movement step when the virtual traveling object travels in a virtual three-dimensional space based on instructions from the traveling model generation unit 14, and outputs the generated image to the traveling model generation unit 14. The traveling model generation unit 14 uses the virtual captured image to perform machine learning on a traveling model.

[0020] FIG. 6 is a diagram illustrating a processing flow of the driving model generation device according to an embodiment of the present disclosure. The space data generation unit 12 of the driving model generation device 1 generates space data of the virtual three-dimensional space in advance and stores it in the storage device 104. The driving model generation unit 14 of the driving model generation device 1 starts generating a driving model when it receives an instruction to generate a driving model from a user.

[0021] The traveling model generation unit 14 initializes both the speed of the virtual traveling body and the angular velocity at which the virtual traveling body rotates so as to face the traveling direction to 0 (step S101). The traveling model generation unit 14 randomly determines the traveling direction of the virtual traveling body at its current position within a 360° surrounding area (step S102).

[0022] The traveling model generation unit 14 determines whether the virtual traveling object has reached a destination position in an episode of the traveling simulation of the virtual traveling object (step S103). If the virtual traveling object has reached the destination position in the traveling simulation, a random initial position and a random target position are newly specified from among a plurality of virtual positions included in the spatial data (step S104). If the virtual traveling object has not yet reached the destination position in the traveling simulation, the initial position and target position specified in the immediately preceding episode of the traveling simulation are specified (step S105).

[0023] An episode is a processing unit of a traveling simulation in which the travel steps of a traveling object are repeated multiple times until the traveling object reaches a predetermined state, such as reaching a destination position or stopping. The initial position may be specified by a user. The traveling model generation unit 14 also randomly determines the viewpoint position of the virtual traveling object from multiple heights (step S106). As an example, the traveling model generation unit 14 may randomly determine the viewpoint position from two heights of the viewpoint position of the virtual traveling object. As another example, the traveling model generation unit 14 may randomly determine the viewpoint position from multiple height positions of the viewpoint position of the virtual traveling object or from multiple positions shifted laterally. The traveling model generation unit 14 may also specify the viewpoint position of the virtual traveling object by narrowing it down to one.

[0024] The traveling model generation unit 14 outputs a virtual photographed image corresponding to the determined moving direction and viewpoint position at the current position of the virtual traveling body to the image acquisition unit 13. The image acquisition unit 13 acquires a virtual photographed image in virtual three-dimensional space data corresponding to the determined moving direction and viewpoint position at the current position of the virtual traveling body (step S107). The image acquisition unit 13 outputs the virtual photographed image to the traveling model generation unit 14.

[0025] When the travel model generation unit 14 receives the initial position, the target position, the moving direction, the viewpoint position, and the virtual captured image of the current position, it calculates the distance from the current position to the target position and the angular difference between the moving direction and the first direction connecting the current position and the target position (step S108).

[0026] The traveling model generation unit 14 defines, as status information, information including at least the initial position, the target position, the current position, the moving direction, the viewpoint position, the distance between the current position and the target position, the angular difference between the first direction connecting the current position and the target position and the moving direction of the virtual traveling body, and a virtual captured image of the moving direction at the current position. The status information may include at least the distance between the current position and the target position, the angular difference between the first direction connecting the current position and the target position and the moving direction of the virtual traveling body, and a virtual captured image of the moving direction at the current position.

[0027] The traveling model generation unit 14 determines the speed and angular velocity for each movement step, and identifies the speed and angular velocity as behavior information. The traveling model generation unit 14 performs a traveling simulation based on the state information and behavior information (step S109). The traveling model generation unit 14 determines whether processing of one episode of the simulation has ended (step S110). The traveling model generation unit 14 repeats the movement steps and sequentially performs the traveling simulation. The traveling model generation unit 14 determines that processing of one episode of the simulation has ended based on whether the traveling simulation in which the movement steps have been repeated by the traveling model generation unit 14 has caused the virtual traveling object to hit an obstacle such as a wall and stop, reached a target position, or whether the movement steps have been repeated a predetermined number of times, such as 1000 steps.

[0028] The traveling model generation unit 14 determines the end of one episode when, as a result of the traveling simulation in which the travel steps are repeated, it is determined that the virtual traveling object has hit an obstacle such as a wall and stopped, when the current position of the virtual traveling object has reached the target position, or when the travel steps have been repeated a predetermined number of times, such as 1,000 steps. The traveling model generation unit 14 calculates the total reward for each repeated travel step in one episode. The traveling model generation unit 14 learns the relationship between the state, action, and reward in one episode through reinforcement learning and generates a traveling model for outputting behavior information corresponding to each state that maximizes the reward (step S111). The traveling model generation unit 14 determines whether the reinforcement learning for one episode has been repeated a predetermined number of times, such as 1.5 million times (step S112), and generates a traveling model by repeating the predetermined number of times. The traveling model may be generated using a known technique. In other words, the traveling model is a machine learning model that, when the above state information is input, outputs behavior information (the speed of the traveling object and the angular velocity when the traveling object rotates in the direction of travel) that maximizes the reward.

[0029] The process of generating a running model by the running model generation unit 14 is an example of a process of performing reinforcement learning in which the reward increases using the initial position and target position of the running body in a virtual three-dimensional space, the state and action of the running body for each step of movement from the initial position to the target position, and the reward corresponding to the action, and generating a running model that outputs the speed v (meters per second) of the running body and the angular velocity ω (radians per second) of rotation in the movement direction when a state including an image of the movement direction of the running body at its current position is input.

[0030] Furthermore, the process of generating a traveling model by the traveling model generation unit 14 is an example of a process of generating a traveling model by repeating reinforcement learning using information including virtual captured images in which the viewpoint position of the traveling body is changed and each pixel indicates information about the traveling road and other than the traveling road. By performing reinforcement learning using virtual captured images from such multiple viewpoint positions, it is possible to easily generate a traveling model that can cause the traveling device 2 to travel with high traveling accuracy.

[0031] In this disclosure, there are seven rewards for each movement step, from the first reward to the seventh reward. The first reward is given when the virtual traveling object reaches the target position, and as an example, the reward value may be +100. The second reward is a reward given when the virtual traveling object hits an obstacle, and the reward value may be set to -50, for example. The third reward is a reward given for each movement step of the virtual traveling object, and may be set to -0.1, for example. By setting the third reward, the fewer movement steps required to reach the destination, the higher the total reward value for the entire episode.

[0032] The fourth reward is a reward given when the virtual traveling object approaches the target position, and may be +0.1, for example. The closer the virtual traveling object approaches the target position, the higher the total reward value for the entire episode. The fifth reward is a reward that is given when the angle (angle difference) between the first direction connecting the current position of the virtual traveling object and the target position and the moving direction is greater than 90° and less than 270°, and may be -0.1 as an example. The more the moving direction of the virtual traveling object matches the direction of the target position, the higher the total reward value for the entire episode.

[0033] The sixth reward is a reward that is given when the angle (angle difference) between the first direction connecting the current position of the virtual traveling object and the target position and the moving direction is greater than 150° and less than 210°, and may be −0.5 as an example. If the moving direction of the virtual traveling object does not match the direction of the target position to a greater extent, the total reward value for the entire episode will be lower. The seventh reward is given when the virtual object's speed is 0 or its angular velocity is slow, between -0.3 and 0.3; for example, it may be -0.1. The slower the virtual object moves, the lower the total reward for the entire episode.

[0034] In generating the above-described traveling model, the traveling model generation unit 14 may perform reinforcement learning of the traveling model using other rewards. Furthermore, in generating the above-described traveling model, the traveling model generation unit 14 may process each movement step in one episode at a short interval, such as 10 milliseconds.

[0035] The driving model generation unit 14 records the generated driving model in the storage device 104. The driving model generation unit 14 may transmit the generated driving model to a driving control device 20 mounted on the driving device 2. The driving control device 20 stores data of the driving model generated by the driving model generation unit 1. The driving model data may be manually registered in the driving control device 20. The driving control device 20 then controls the driving of the driving device 2 using the stored driving model.

[0036] According to the processing of the above-described traveling model generating unit 14, a traveling model is generated using virtual captured images corresponding to a plurality of viewpoint positions. This makes it possible to improve the traveling accuracy when the traveling device 2 performs traveling control using the traveling model generated by such processing.

[0037] FIG. 7 is a functional block diagram of a cruise control device according to one embodiment of the present disclosure. When the driving control device 20 starts the driving control program, the driving control device 20 is provided with the functions of a driving control unit 21, a processed image generating unit 22, and a position detecting unit 23. The traveling control unit 21 outputs input information including the actual initial position and actual target position of the traveling device 2 traveling in real space, processed images obtained by segmenting captured images of the traveling direction of the current position as the traveling device 2 travels from the actual initial position to the real target position into a traveling path and an area other than the traveling path, and the state of the traveling device based on the current position and the actual target position, and output information including the speed and angular velocity at the current position using a traveling model, and controls the traveling of the traveling device 2 using the output information. The position detection unit 23 detects the position of the traveling device 2 using a function such as GNSS (Global Navigation Satellite System).

[0038] FIG. 8 is a diagram illustrating an outline of processing performed by the driving control device according to an embodiment of the present disclosure. FIG. 9 is a diagram showing a processing flow of the driving control device according to one embodiment of the present disclosure. First, the user sets an initial position and a target position in the driving control device 20. The initial position may be the current position. The driving control device 20 may specify the current position detected by the position detection unit 23 as the initial position. The driving control device 20 may also store map data and detect the target position specified by a user operation in the map data. The driving control unit 21 acquires the initial position and the target position (step S201). The driving control unit 21 calculates a driving route based on the initial position and the target position using the map data (step S202). The calculation of the driving route from the initial position to the target position may use known techniques. The initial position and the target position may be latitude and longitude specified based on the map data. The driving control unit 21 calculates multiple different positions on the specified route between the initial position and the target position, and stores each of the different positions as an intermediate target position (latitude, longitude) (step S203). The driving control device 20 may not store map data or calculate a driving route, but may instead accept and store input of multiple intermediate target positions between the initial position and the target position through user operation. The driving control device 20 may also calculate multiple intermediate target positions between the initial position and the target position using another method.

[0039] Based on a user operation input instructing the start of driving, the driving control device 20 starts driving control (step S204). The processed image generation unit 22 of the driving control device 20 instructs the imaging device 3 to start capturing images. The processed image generation unit 22 of the driving control device 20 sequentially acquires captured images constituting the video captured by the imaging device 3 (step S205). The processed image generation unit 22 sequentially performs segmentation processing on each captured image at the time interval of the travel step of the driving device 2 to generate processed images corresponding to the captured images, and outputs the processed images to the driving control unit 21 (step S206). Note that the processed image generation unit 22 may input the captured images to a neural network configured using a semantic segmentation model, thereby acquiring processed images showing the driving path and non-road areas. The processed images are images after segmentation processing generated for each travel step. The processed images may be generated in the same manner as the segmentation processing performed by the driving model generation device 1. The processed images in the present disclosure are images segmented into two values: the driving path and non-road areas. In other disclosures, the processed image may be an image in which at least the travel path is identified by segmentation processing in the captured image. In the travel control using a travel model, the travel device 2 and travel control system 100 of the present disclosure can accurately control the travel of the travel device 2 by generating a processed image in this manner by segmenting the captured image as information to be input to the travel model.

[0040] The traveling control unit 21 detects the initial position, the target position, and the current position, and when the processed image is input, calculates the distance from the current position to the target position and the angular difference between the first direction connecting the current position and the target position and the direction of movement (step S207).

[0041] The traveling control unit 21 identifies as status information information including at least the initial position, the target position, the current position, the movement direction, the viewpoint position, the distance from the current position to the target position, the angular difference between the first direction connecting the current position and the target position and the movement direction of the virtual traveling body, and a processed image after segmentation processing corresponding to the virtual captured image in the movement direction at the current position.The traveling control unit 21 may identify as status information information including at least the distance from the current position to the target position, the angular difference between the first direction connecting the current position and the target position and the movement direction of the virtual traveling body, and a processed image after segmentation processing corresponding to the virtual captured image in the movement direction at the current position.

[0042] The traveling control unit 21 inputs the state information into a neural network configured by a traveling model, and calculates the speed and angular velocity, which are behavior information (step S208). The traveling control unit 21 controls the drive mechanism (motor and steering angle) based on the calculated speed and angular velocity (step S209). This causes the traveling device 2 to travel toward the target position.

[0043] The driving control unit 21 calculates the speed and angular velocity, which are behavior information, for each movement step and controls the drive mechanism until the current position coincides with the target position. The driving control unit 21 compares the current position with the target position and determines whether the current position coincides with the target position (step S210). The driving control unit 21 compares the current position with the target position and, if the current position coincides with the target position, stops driving control. When calculating the behavior information, the driving control unit 21 may use the next intermediate target position as the target position to calculate the distance between the current position and the intermediate target position and the angular difference between the first direction connecting the current position and the intermediate target position and the movement direction, and input this information into a neural network using a driving model to calculate the behavior information. The driving control unit 21 may then repeat traveling from the current position to the intermediate target position to travel to the final target position.

[0044] According to the above-described processing, the traveling device 2 performs segmentation processing on the captured image to generate a processed image that is classified into the traveling path and non-traveling path, similar to the virtual captured image used when generating the traveling model, and calculates the speed and angular velocity for each movement step using this processed image and the traveling model. With this processing, even if there is a shadow on the traveling path in the real space in which the traveling device 2 actually travels, or even if the real space is different from the virtual three-dimensional space in which the traveling model was generated, the traveling path is clearly identified in the processed image, and the traveling model is also generated using virtual captured images that indicate the traveling path and non-traveling path in binary, so it is possible to provide a traveling device 2 that automatically controls traveling to a target position with high accuracy using such a traveling model.

[0045] Furthermore, the traveling device 2 generates the speed and angular velocity using a traveling model generated using processed images corresponding to multiple viewpoint positions, which can improve the traveling accuracy when the traveling device 2 performs traveling control using the traveling model generated by such processing.

[0046] The embodiments of the present disclosure have been described above. The driving model generation device 1 and the driving device 2 described above can provide a technology for improving the accuracy of automatic driving from an initial position to a target position.

[0047] FIG. 10 is a first diagram illustrating an example of the results of travel control of a travel device according to an embodiment of the present disclosure. 10 shows the results of the arrival rate to the target position when the traveling control device 20 of the traveling device 2 performs traveling control using a traveling model (random model) generated using each virtual captured image in which the height of the viewpoint position of the traveling device is changed, and a traveling model (fixed model) generated using virtual captured images in which the height of the viewpoint position of the traveling device is fixed. Compared to traveling control using the traveling model (fixed model), the traveling control using the traveling model (random model) resulted in higher probabilities of the arrival rate to the target position and speed stability when the traveling path is straight, and higher probabilities of the arrival rate to the target position and speed stability when the traveling path is curved.

[0048] FIG. 11 is a second diagram illustrating an example of the results of the travel control of the travel device according to the embodiment of the present disclosure. FIG. 11 shows the results of the translational velocity change when the driving control device 20 of the driving device 2 performs driving control using a driving model (random model) generated using virtual captured images in which the height of the viewpoint position of the driving device was changed, and a driving model (fixed model) generated using virtual captured images in which the height of the viewpoint position of the driving device was fixed. The translational velocity indicates the velocity (meters / second) in the tangential direction of the arc of an object moving in a circular motion at angular velocity. In FIG. 11, the left graph (A) shows the translational velocity change over time when the driving control device 20 performs driving control using the driving model (random model). In FIG. 11, the right graph (B) shows the translational velocity change over time when the driving control device 20 performs driving control using the driving model (fixed model). Compared to driving control using the driving model (fixed model), driving control using the driving model (random model) resulted in a reduction in the occurrence of a decrease in translational velocity during the journey to the target position.

[0049] The driving model disclosed herein includes a case where a driving model is generated using virtual captured images when the height of the viewpoint position of the traveling object is fixed, as described above, and the driving control device 20 performs driving control using that driving model. Even when using a driving model generated using virtual captured images when the height of the viewpoint position of the traveling object is fixed, the traveling device 2 uses processed images that have been segmented to clarify the driving path during driving control. As a result, both the process of generating the driving model and the process of performing driving control using the driving model use images that show the driving path and non-driving path in binary, which can improve driving accuracy (such as arrival rate and speed stability).

[0050] FIG. 12 is a diagram illustrating a minimum configuration of a driving model generation device according to an embodiment of the present disclosure. The driving model generating device 1 includes at least a driving model generating means 81. The traveling model generation means 81 performs reinforcement learning in which the reward increases using the initial position and target position of the traveling body in virtual three-dimensional space, the state and action of the traveling body for each step of movement from the initial position to the target position, and a reward corresponding to the action, to generate a traveling model that outputs the speed of the traveling body and the angular velocity in the movement direction when a state including an image of the traveling direction of the traveling body at its current position is input. In this process, the traveling model generation means 81 generates a traveling model by repeating reinforcement learning using state information including a virtual captured image in which each pixel shows a value either of the traveling path or a value other than the traveling path.

[0051] FIG. 13 is a diagram illustrating a minimum configuration of a driving control device according to an embodiment of the present disclosure. The driving control device 20 includes at least a storage means 91 and a driving control means 92 . The storage means 91 performs reinforcement learning in which the reward increases using a virtual initial position and a virtual target position of the virtual traveling body in a virtual three-dimensional space, the state and action of the virtual traveling body for each step of movement from the virtual initial position to the virtual target position, and a reward corresponding to the action, and stores a traveling model that outputs the speed and angular velocity of the virtual traveling body in the movement direction when a state including a virtual current position and a virtual photographed image of the movement direction of the virtual traveling body at the virtual current position is input, and that is generated by repeating reinforcement learning using information on a state including a virtual photographed image in which each pixel of the virtual photographed image shows a value either of the traveling path or a value other than the traveling path. The traveling control means 92 outputs output information including the speed and angular velocity at the current position using a traveling model, and input information including the actual initial position and actual target position of the traveling device 2 traveling in real space, processed images obtained by segmenting photographed images of the traveling direction of the current position as the traveling device 2 moves from the actual initial position to the real target position into a traveling path and an area other than the traveling path, and the state of the traveling device based on the current position and the actual target position, and controls the traveling of the traveling device 2 using the output information.

[0052] In the above disclosure, the case where the driving control device 20 is included in the traveling device 2 has been described. However, the driving control device 20 may be provided in a computer that is connected to and communicates with the traveling device 2, such as a cloud server, and the driving control device 20 of the computer may calculate the speed and angular velocity of the traveling device 2 for each step to control the traveling of the traveling device 2.

[0053] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments.

[0054] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0055] (Appendix 1) a running model generation means for performing reinforcement learning in which a reward increases using an initial position and a target position of a running body in a virtual three-dimensional space, a state and an action of the running body for each moving step from the initial position to the target position, and a reward corresponding to the action, and for generating a running model that outputs a speed of the running body and an angular velocity in the moving direction when the state including an image of the moving direction of the running body at a current position is input; The travel model generating means generates the travel model by repeating the reinforcement learning using the state information including a virtual photographed image obtained by changing the viewpoint position of the travel object, in which each pixel indicates a value corresponding to either a travel path or a value other than a travel path. Driving model generation device.

[0056] (Appendix 2) The running model generating means generates the running model by repeating the reinforcement learning using the state information including the virtual captured image in which the height of the viewpoint position of the running object is changed. The driving model generating device according to claim 1.

[0057] (Appendix 3) The running model generation means performs the reinforcement learning using the state information including the initial position randomly selected from a plurality of positions set in the virtual three-dimensional space, the target position randomly selected, the moving direction randomly selected at the current position, the distance from the current position to the target position, the angular difference between the moving direction and a first direction connecting the current position and the target position, the speed at the current position, and the angular velocity of the running body when turning to the moving direction. 3. A driving model generation device according to claim 2.

[0058] (Appendix 4) The travel model generation means generates the travel model by repeatedly changing the combination of the current position, the target position, and the travel direction through reinforcement learning, by giving a predetermined positive reward when the target position is reached and a predetermined negative reward when the target position is not reached, as a reward according to the state and action in the travel step. 4. A driving model generation device according to claim 3.

[0059] (Appendix 5) The running model generation means performs the reinforcement learning by giving, as the negative reward, at least a negative reward when the running object collides with a wall and a negative reward for each repeated action in which the running object runs, stops, and runs again. 5. A driving model generation device according to claim 4.

[0060] (Appendix 6) a travel model that performs reinforcement learning to increase the reward using a virtual initial position and a virtual target position of a virtual travel body in a virtual three-dimensional space, a state and action of the virtual travel body for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual travel body and an angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual travel body at the virtual current position is input, and that stores the travel model generated by repeating the reinforcement learning using information on the state including a virtual photographed image in which each pixel of the virtual photographed image shows a value corresponding to either a travel path or a value other than a travel path; a travel control means for outputting, using the travel model, output information including a real initial position and a real target position of a travel device traveling in a real space, processed images obtained by segmenting captured images of a moving direction of a current position of the travel device during movement from the real initial position to the real target position into a travel path and an area other than the travel path, input information including a state of the travel device based on the current position and the real target position, and output information including a velocity and an angular velocity at the current position; and for controlling the travel of the travel device using the output information; A driving control device comprising:

[0061] (Appendix 7) The storage means stores the traveling model generated by repeating the reinforcement learning using the state information including each processed image obtained by segmenting each of the virtual captured images obtained by changing the viewpoint position of the virtual traveling body into a traveling path and a non-traveling path. 7. The cruise control device according to claim 6.

[0062] (Appendix 8) The storage means stores the traveling model generated by repeating the reinforcement learning using the state information including each processed image obtained by segmenting each of the virtual captured images obtained by changing the height of the viewpoint position of the virtual traveling body into a traveling path and a non-traveling path. 8. The driving control device according to claim 7.

[0063] (Appendix 9) The travel control means A plurality of different positions on the path from the actual initial position to the actual target position are calculated, and the travel control is performed using each of the different positions as an intermediate target position. 9. The cruise control device of claim 8.

[0064] (Appendix 10) A travel device comprising the travel control device according to any one of Supplementary Note 6 to Supplementary Note 8.

[0065] (Appendix 11) performing reinforcement learning using an initial position and a target position of a running body in a virtual three-dimensional space, a state and an action of the running body for each movement step from the initial position to the target position, and a reward corresponding to the action, so that the reward increases; and generating a running model that outputs a speed of the running body and an angular velocity in the movement direction when the state including an image of the movement direction of the running body at its current position is input; In generating the traveling model, the reinforcement learning is repeated using the state information including a virtual photographed image in which the viewpoint position of the traveling body is changed and each pixel indicates a value of either the traveling road or a value other than the traveling road, to generate the traveling model. Driving model generation method.

[0066] (Appendix 12) Repeating the reinforcement learning using the state information including the virtual captured image in which the height of the viewpoint position of the traveling object is changed to generate a traveling model. 12. A driving model generation method according to claim 11.

[0067] (Appendix 13) The reinforcement learning is performed using the state information including the initial position randomly selected from a plurality of positions set in the virtual three-dimensional space, the target position randomly selected, the moving direction randomly selected at the current position, the distance from the current position to the target position, the angular difference between the moving direction and a first direction connecting the current position and the target position, the speed at the current position, and the angular velocity of the moving body when turning to the moving direction, to generate a moving model of the moving body. 13. A driving model generation method according to claim 12.

[0068] (Appendix 14) As a reward according to the state and action in the movement step, a predetermined positive reward is given when the target position is reached, and a predetermined negative reward is given when the target position is not reached, and the travel model is generated by repeating reinforcement learning of changing the combination of the current position, the target position, and the movement direction. 14. A driving model generation method according to claim 13.

[0069] (Appendix 15) As the negative reward, a negative reward when the moving object collides with a wall and a negative reward for each repeated action of the moving object moving, stopping, and moving again are given to perform the reinforcement learning. 15. A driving model generation method according to claim 14.

[0070] (Appendix 16) a travel model that performs reinforcement learning to increase the reward using a virtual initial position and a virtual target position of a virtual travel body in a virtual three-dimensional space, a state and action of the virtual travel body for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs the speed of the virtual travel body and the angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual travel body at the virtual current position is input, and stores the travel model generated by repeating the reinforcement learning using information on the state including a virtual photographed image in which each pixel of the virtual photographed image shows a value of either a travel path or a value other than a travel path; outputting output information including a velocity and an angular velocity at the current position using the travel model, and performing travel control of the travel device using the input information including an actual initial position and an actual target position of the travel device traveling in a real space, a processed image obtained by segmenting a photographed image of the travel direction of the current position of the travel device in the movement from the actual initial position to the real target position into a travel path and a non-travel path, and a state of the travel device based on the current position and the actual target position; Driving control method.

[0071] (Appendix 17) The traveling model generated by repeating the reinforcement learning using the state information including each processed image obtained by segmenting each of the virtual captured images obtained by changing the viewpoint position of the virtual traveling body into a traveling path and a non-traveling path is stored. 17. The cruise control method of claim 16.

[0072] (Appendix 18) The traveling model generated by repeating the reinforcement learning using the state information including each processed image obtained by segmenting each of the virtual captured images obtained by changing the height of the viewpoint position of the virtual traveling body into a traveling path and a non-traveling path is stored. 18. The cruise control method of claim 17.

[0073] (Appendix 19) A plurality of different positions on the path from the actual initial position to the actual target position are calculated, and the travel control is performed using each of the different positions as an intermediate target position. 19. The cruise control method of claim 18.

[0074] (Appendix 20) A computer of the driving model generation device, and performing reinforcement learning using an initial position and a target position of a running body in a virtual three-dimensional space, a state and an action of the running body for each movement step from the initial position to the target position, and a reward corresponding to the action, so that the reward increases, and functioning as running model generation means for generating a running model that outputs the speed of the running body and the angular velocity in the movement direction when the state including an image of the movement direction of the running body at its current position is input; The travel model generating means generates the travel model by repeating the reinforcement learning using the state information including a virtual photographed image obtained by changing the viewpoint position of the travel object, in which each pixel indicates a value corresponding to either a travel path or a value other than a travel path. program.

[0075] (Appendix 21) The running model generating means generates the running model by repeating the reinforcement learning using the state information including the virtual captured image in which the height of the viewpoint position of the running object is changed. 20. The program described in Appendix 20.

[0076] (Appendix 22) The running model generation means performs the reinforcement learning using the state information including the initial position randomly selected from a plurality of positions set in the virtual three-dimensional space, the target position randomly selected, the moving direction randomly selected at the current position, the distance from the current position to the target position, the angular difference between the moving direction and a first direction connecting the current position and the target position, the speed at the current position, and the angular velocity of the running body when turning to the moving direction. 21. The program described in Appendix 21.

[0077] (Appendix 23) The travel model generation means generates the travel model by repeatedly changing the combination of the current position, the target position, and the travel direction through reinforcement learning, by giving a predetermined positive reward when the target position is reached and a predetermined negative reward when the target position is not reached, as a reward according to the state and action in the travel step. 22. The program of claim 1.

[0078] (Appendix 24) The running model generation means performs the reinforcement learning by giving, as the negative reward, at least a negative reward when the running object collides with a wall and a negative reward for each repeated action in which the running object runs, stops, and runs again. 23. The program described in Appendix 23.

[0079] (Appendix 25) a computer for a travel control device, which performs reinforcement learning to increase the reward using a virtual initial position and a virtual target position of a virtual travel object in a virtual three-dimensional space, a state and action of the virtual travel object for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual travel object and an angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual travel object at the virtual current position is input, and which includes storage means for storing the travel model generated by repeating the reinforcement learning using information on the state including a virtual photographed image in which each pixel of the virtual photographed image shows a value of either a travel path or a value other than a travel path; a travel control means for outputting output information including a velocity and an angular velocity at the current position using input information including a real initial position and a real target position of a travel device traveling in a real space, processed images obtained by segmenting a photographed image of a moving direction of a current position of the travel device during movement from the real initial position to the real target position into a travel path and an area other than the travel path, a state of the travel device based on the current position and the real target position, and the travel model, and for controlling the travel of the travel device using the output information; A program that functions as a

[0080] (Appendix 26) The travel control means A plurality of different positions on the path from the actual initial position to the actual target position are calculated, and the travel control is performed using each of the different positions as an intermediate target position. 2. The program described in Appendix 25. [Explanation of symbols]

[0081] 1. Driving model generation device 2. Running gear 3. Imaging device 11 Control section 12. Spatial data generation unit 13. Image acquisition unit 14. Driving model generation unit 21. Travel control unit 22. Processing image generation unit 23 Position detection unit

Claims

1. a running model generation means for performing reinforcement learning in which a reward increases using an initial position and a target position of a running body in a virtual three-dimensional space, a state and an action of the running body for each moving step from the initial position to the target position, and a reward corresponding to the action, and for generating a running model that outputs a speed of the running body and an angular velocity in the moving direction when the state including an image of the moving direction of the running body at a current position is input; The travel model generating means generates the travel model by repeating the reinforcement learning using the state information including a virtual photographed image obtained by changing the viewpoint position of the travel object, in which each pixel indicates a value corresponding to either a travel path or a value other than a travel path. Driving model generation device.

2. The running model generating means generates the running model by repeating the reinforcement learning using the state information including the virtual captured image in which the height of the viewpoint position of the running object is changed. The driving model generating device according to claim 1 .

3. The running model generation means performs the reinforcement learning using the state information including the initial position randomly selected from a plurality of positions set in the virtual three-dimensional space, the target position randomly selected, the moving direction randomly selected at the current position, the distance from the current position to the target position, the angular difference between the moving direction and a first direction connecting the current position and the target position, the speed at the current position, and the angular velocity of the running body when turning to the moving direction. The driving model generating device according to claim 2 .

4. The travel model generation means generates the travel model by repeatedly changing the combination of the current position, the target position, and the travel direction through reinforcement learning, by giving a predetermined positive reward when the target position is reached and a predetermined negative reward when the target position is not reached, as a reward according to the state and action in the travel step. The driving model generating device according to claim 3 .

5. The running model generation means performs the reinforcement learning by giving, as the negative reward, at least a negative reward when the running object collides with a wall and a negative reward for each repeated action in which the running object runs, stops, and runs again. The driving model generating device according to claim 4 .

6. a travel model that performs reinforcement learning to increase the reward using a virtual initial position and a virtual target position of a virtual travel body in a virtual three-dimensional space, a state and action of the virtual travel body for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual travel body and an angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual travel body at the virtual current position is input, and that stores the travel model generated by repeating the reinforcement learning using information on the state including a virtual photographed image in which each pixel of the virtual photographed image shows a value corresponding to either a travel path or a value other than a travel path; a travel control means for outputting, using the travel model, output information including a real initial position and a real target position of a travel device traveling in a real space, processed images obtained by segmenting captured images of a moving direction of a current position of the travel device during movement from the real initial position to the real target position into a travel path and an area other than the travel path, input information including a state of the travel device based on the current position and the real target position, and output information including a velocity and an angular velocity at the current position; and for controlling the travel of the travel device using the output information; A driving control device comprising:

7. The storage means stores the traveling model generated by repeating the reinforcement learning using the state information including the virtual captured image in which the viewpoint position of the virtual traveling body is changed. The cruise control device according to claim 6.

8. The storage means stores the traveling model generated by repeating the reinforcement learning using the state information including the virtual captured image in which the height of the viewpoint position of the virtual traveling body is changed. The cruise control device according to claim 7.

9. The travel control means A plurality of different positions on the path from the actual initial position to the actual target position are calculated, and the travel control is performed using each of the different positions as an intermediate target position. The cruise control device according to claim 8.

10. A traveling device comprising the traveling control device according to any one of claims 6 to 9.

11. performing reinforcement learning using an initial position and a target position of a running body in a virtual three-dimensional space, a state and an action of the running body for each movement step from the initial position to the target position, and a reward corresponding to the action, so that the reward increases; and generating a running model that outputs a speed of the running body and an angular velocity in the movement direction when the state including an image of the movement direction of the running body at its current position is input; In generating the traveling model, the reinforcement learning is repeated using the state information including a virtual photographed image in which the viewpoint position of the traveling body is changed and each pixel indicates a value of either the traveling road or a value other than the traveling road, to generate the traveling model. Driving model generation method.

12. a travel model that performs reinforcement learning to increase the reward using a virtual initial position and a virtual target position of a virtual travel body in a virtual three-dimensional space, a state and action of the virtual travel body for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs the speed of the virtual travel body and the angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual travel body at the virtual current position is input, and stores the travel model generated by repeating the reinforcement learning using information on the state including a virtual photographed image in which each pixel of the virtual photographed image shows a value of either a travel path or a value other than a travel path; outputting output information including a velocity and an angular velocity at the current position using the travel model, and performing travel control of the travel device using the input information including an actual initial position and an actual target position of the travel device traveling in a real space, a processed image obtained by segmenting a photographed image of the travel direction of the current position of the travel device in the movement from the actual initial position to the real target position into a travel path and a non-travel path, and a state of the travel device based on the current position and the actual target position; Driving control method.

13. A computer of the driving model generation device, and functioning as a running model generation means for generating a running model that outputs the speed of the running body and the angular velocity in the moving direction when the state including an image of the moving direction of the running body at its current position is input, by performing reinforcement learning in which the reward increases using an initial position and a target position of the running body in a virtual three-dimensional space, a state and an action of the running body for each moving step from the initial position to the target position, and a reward corresponding to the action; The travel model generating means generates the travel model by repeating the reinforcement learning using the state information including a virtual photographed image obtained by changing the viewpoint position of the travel object, in which each pixel indicates a value corresponding to either a travel path or a value other than a travel path. program.

14. a computer for a travel control device, which performs reinforcement learning to increase the reward using a virtual initial position and a virtual target position of a virtual travel object in a virtual three-dimensional space, a state and action of the virtual travel object for each movement step from the virtual initial position to the virtual target position, and a reward corresponding to the action, and outputs a speed of the virtual travel object and an angular velocity in the movement direction when the state including a virtual current position and a virtual photographed image of the movement direction of the virtual travel object at the virtual current position is input, and which includes storage means for storing the travel model generated by repeating the reinforcement learning using information on the state including a virtual photographed image in which each pixel of the virtual photographed image shows a value of either a travel path or a value other than a travel path; a travel control means for outputting output information including a velocity and an angular velocity at the current position using input information including a real initial position and a real target position of a travel device traveling in a real space, processed images obtained by segmenting a photographed image of a moving direction of a current position of the travel device during movement from the real initial position to the real target position into a travel path and an area other than the travel path, a state of the travel device based on the current position and the real target position, and the travel model, and for controlling the travel of the travel device using the output information; A program that functions as a

Images