Remote control device, remote control system, and risk determination method

Abstract

The present disclosure relates to a remote control device that controls at least one moving body via a transmission path including at least a network, the remote control device comprising: a transmission delay distribution estimation unit that estimates transmission delay distribution information including a probability distribution of transmission delays in the transmission path; and an action planning unit that plans a target action of the at least one moving body on the basis of the transmission delay distribution information, and the transmission delay distribution estimation unit estimates a probability distribution of the transmission delay using a transmission delay model based on a transmission delay pattern.

Description

Technical Field

[0001] The present disclosure relates to a remote control device for a mobile object, and more particularly to a remote control device that takes transmission delay into consideration. Background Art

[0002] In recent years, a remote control device has been developed that implements automatic driving and automatic transportation by, for example, providing an action instruction or executing a control amount calculation to a mobile object at a remote location. Patent Documents 1 and 2 are examples of such a remote control device.

[0003] Generally, in such a remote control device, wireless communication using radio waves or the like and a network composed of wireless communication are used as a means of realizing data transmission and reception between the remote control device and the mobile body. In addition, a control technology for a mobile body is proposed that can reduce the discomfort of the occupants by adjusting the control gain of the mobile body according to the risk of the surroundings.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 6940036

[0007] Patent Document 2: Japanese Patent Application Publication No. 2020-50342 Summary of the invention

[0008] When using a network, the mobile body needs to take safe actions that take into account transmission delays, but the technology disclosed in Patent Document 1 cannot plan appropriate actions corresponding to the state of transmission delays. In addition, the technology disclosed in Patent Document 2 does not take into account transmission delays in remote control of mobile bodies.

[0009] The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide a remote control device that takes transmission delay into consideration in remote control of a mobile object.

[0010] The remote control device involved in the present disclosure is a remote control device that controls at least one mobile body via a transmission path that includes at least a network, and the remote control device includes: a transmission delay distribution estimation unit that estimates transmission delay distribution information including the probability distribution of transmission delays in the transmission path; and an action planning unit that plans a target action of the at least one mobile body based on the transmission delay distribution information, and the transmission delay distribution estimation unit uses a transmission delay model based on the pattern of the transmission delay to estimate the probability distribution of the transmission delay.

[0011] According to the remote control device according to the present disclosure, by taking transmission delay into consideration in remote control of a mobile object, remote control with further improved safety can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing the configuration of a remote control device and a remote control system according to Embodiment 1 of the present disclosure.

[0013] Figure 2 It is a block diagram showing the structure of the first moving body control unit.

[0014] Figure 3 This is a diagram showing an example of a target path when the first moving object is a vehicle.

[0015] Figure 4 It is a block diagram showing the structure of the first moving object.

[0016] Figure 5 This is a diagram showing an example of a configuration in which the first moving object is a vehicle.

[0017] Figure 6 It is a diagram showing an example of the configuration of the object information acquisition unit.

[0018] Figure 7 It is a diagram showing a target path for the first moving body to avoid a stationary object.

[0019] Figure 8 It is a diagram showing an example of the arrangement of the object information acquisition unit and the environment information acquisition unit.

[0020] Fig. 9 This is a diagram showing an example of a target speed generated in the remote control device according to Embodiment 1 of the present disclosure.

[0021] Fig.10 This is a block diagram showing an example of the configuration of a transmission delay distribution estimation unit.

[0022] Fig.11 It is a diagram showing an example of a time series of transmission delay.

[0023] Fig.12 This is a diagram showing an example of modeling using HMM.

[0024] Fig.13 This is a block diagram showing an example of the structure of the mobile object behavior planning unit.

[0025] Fig.14 This is a diagram schematically showing the operation of changing the target behavior in the mobile body behavior planning unit.

[0026] Fig.15 This is a block diagram showing an example of the structure of the mobile object behavior planning unit.

[0027] Fig.16 This is a diagram schematically showing the operation of changing the target behavior due to the mobile body risk.

[0028] Fig.17 This is a block diagram showing an example of a control system in which the remote control device according to the first embodiment of the present disclosure controls a first moving object in a transmission delay environment.

[0029] Fig.18 This is a block diagram showing the configuration of a remote control device and a remote control system according to Embodiment 2 of the present disclosure.

[0030] Fig.19 This is a block diagram showing an example of the structure of the mobile object behavior planning unit.

[0031] Fig. 20 This is a diagram for explaining an action plan when the first moving object and the second moving object intersect with each other.

[0032] Fig.21 This is a block diagram showing the configuration of a remote control device and a remote control system according to Embodiment 2 of the present disclosure.

[0033] Fig. 22 This is a block diagram showing an example of the configuration of a transmission delay distribution estimation unit.

[0034] Fig.23 This is a block diagram showing an example of the configuration of a transmission delay distribution estimation unit of a remote control device according to Embodiment 4 of the present disclosure.

[0035] Fig.24 : is a diagram showing a hardware configuration for realizing a remote control device.

[0036] Fig.25 : is a diagram showing a hardware configuration for realizing a remote control device. DETAILED DESCRIPTION

[0037] <Implementation Method 1>

[0038] <Overall Structure>

[0039] Figure 1 This is a block diagram showing an example of the configuration of the remote control device 1000 according to the first embodiment of the present disclosure and the configuration of a remote control system RCS1 of a mobile object MV which is remotely controlled via a network NW.

[0040] like Figure 1As shown, the remote control system RCS1 has a configuration in which a mobile object MV, a remote control device 1000 , an object information acquisition unit 200 , and an environment information acquisition unit 300 are connected to a network NW.

[0041] The network NW can connect multiple components to each other using cables and radio waves to send and receive data. The network NW is composed of LAN (Local Area Network), WAN (Wide Area Network), Internet, telephone lines, wireless communications, etc., but the network NW is not limited to these, and any medium can be used as long as it is a medium that can send and receive data between the remote control device and the mobile body at the remote end.

[0042] Regarding the moving body MV, in the present first embodiment, a single moving body is used as a control object, but in order to distinguish it from the case where a plurality of moving bodies are used as control objects, it is referred to as the first moving body 100. The first moving body 100 moves according to the control amount sent from the sending unit 1004 of the remote control device 1000, and the state amount of the moving body detected by the internal sensor (described later) composed of the mounted speed sensor, etc. is output as the state information of the first moving body 100, that is, the moving body 1 information. The structure of the first moving body 100 will be described later. Figure 4 Detailed description.

[0043] The object information acquisition unit 200 is composed of one or more sensors mounted around the first moving body 100 or on the first moving body 100. When the moving body is a car and is traveling on a road, the object information acquisition unit 200 is installed at, for example, a traffic light, a utility pole, or a light at an intersection. In addition, there are also cases where it is installed on the roadside. In the case of other moving bodies, such as a moving body moving indoors, the object information acquisition unit is sometimes installed on a ceiling or a wall. The object information acquisition unit 200 acquires the position and speed of obstacles such as other vehicles, bicycles, and pedestrians around the first moving body 100 as object information. In addition, the object information acquisition unit 200 can acquire the position and speed of the first moving body 100 itself as moving body information. In this case, the moving body information is a part of the object information. The object information acquisition unit 200 sends the moving body information to the receiving unit 1012 in the remote control device 1000 via the network NW. In addition, when an internal sensor is installed on the first moving body 100, the moving body information can also be acquired from the internal sensor. In this case, the moving body information corresponds to the moving body 1 information. Therefore, the moving body information can be acquired from the object information acquisition unit 200 or the first moving body 100.

[0044] The object information acquisition unit 200 includes a time synchronization unit 201. The time synchronization unit 201 has a function of synchronizing the timing of data transmission and reception in cooperation with a time synchronization unit (not shown) in the first moving body 100, a time synchronization unit 310 in the environment information acquisition unit 300, and a time synchronization unit 1011 in the remote control device 1000.

[0045] Each of the above-mentioned time synchronization units can be synchronized by using a GNSS (Global Navigation Satellite System) sensor when outdoors. GNSS is a global time synchronization system, and since it is a well-known technology, it can be used to easily synchronize time. On the other hand, when indoors, time synchronization can be performed by accessing an NTP (Network Time Protocol) server set on the network NW.

[0046] The environmental information acquisition unit 300 is composed of one or more sensors arranged around the first moving body 100, similarly to the object information acquisition unit 200. The environmental information acquisition unit 300 is also arranged inside or outside the house. The environmental information acquisition unit 300 acquires environmental information such as traffic lights and stop lines. The environmental information acquisition unit 300 sends the environmental information to the receiving unit 1012 in the remote control device 1000 via the network NW. In addition, environmental information can sometimes be acquired through the object information acquisition unit 200. In all embodiments, object information and environmental information are combined as surrounding information. However, for example, in the case where the moving body is a robot, the surrounding information may not include environmental information, but only object information. In addition, the sensor used in the environmental information acquisition unit 300 can also be mounted on the first moving body 100.

[0047] The environment information acquisition unit 300 includes a time synchronization unit 301. The time synchronization unit 301 has a function of synchronizing the timing of data transmission and reception in cooperation with a time synchronization unit (not shown) in the first moving body 100, the time synchronization unit 201 in the object information acquisition unit 200, and the time synchronization unit 1011 in the remote control device 1000.

[0048] Examples of sensors used in the object information acquisition unit 200 and the environment information acquisition unit 300 include cameras, LiDAR (Light Detection And Ranging), radars, and the like.

[0049] The camera is installed at a position capable of capturing images of the front, sides, and rear, and acquires, from the captured images, for example, the demarcation lines around the first moving body 100 and the position and speed of obstacles.

[0050] LiDAR detects the position of an object by irradiating the surrounding area with laser light and detecting the time difference between the light being reflected by surrounding objects and returning.

[0051] The radar measures the relative distance and relative speed of obstacles in the surrounding area with respect to the radar by irradiating the surrounding area with radar and detecting the reflected waves, and outputs the measurement results.

[0052] In addition, when a GNSS sensor capable of detecting the absolute position of obstacles around the first mobile body 100 is mounted on each obstacle and on the first mobile body 100, and the GNSS sensor is capable of sending absolute position information to the remote control device 1000 via the network NW, object information can be detected by GNSS, so the object information acquisition unit 200 can be omitted.

[0053] The map database 500 stores map data of the surroundings of the first moving object 100. Figure 1 In the embodiment, the mobile body action planning unit 1002 and the mobile body control unit are connected to the map database 500, but the present invention is not limited thereto, and each component in the remote control device 1000 can access the map database 500. When the first mobile body 100 is a vehicle, the map database 500 often includes data related to driving, such as the center coordinate information of the road, the information of the stop line, the information of the white line, and the drivable area.

[0054] Next, the components of the remote control device 1000 are described. Figure 1 As shown, the remote control device 1000 includes a transmission delay distribution estimation unit 1001 , a mobile behavior planning unit 1002 , a mobile control unit 1003 , a transmission unit 1004 , a time synchronization unit 1011 , a reception unit 1012 , and a transmission delay measurement unit 1013 .

[0055] The transmission delay measuring unit 1013 measures the transmission delay generated between the first mobile body 100 and the remote control device 1000 using the time synchronized by the time synchronization unit 1011, and outputs the transmission delay as the transmission delay information of the first mobile body 100, that is, the mobile body 1 transmission delay information, to the transmission delay distribution estimating unit 1001. The transmission delay can be obtained based on the difference between the transmission time included in the mobile body 1 information output from the first mobile body 100 and the reception time when the mobile body 1 information is received by the remote control device 1000.

[0056] In addition, in the case where the time synchronization unit is not provided for the remote control device 1000 and the first mobile body 100, the transmission delay can be measured as described below. That is, first, a packet is sent from the remote control device 1000 to the first mobile body 100, and its time is recorded at the same time. The first mobile body 100 sends the packet to the remote control device 1000 at the same time when it receives the packet, and the transmission delay can be calculated based on the difference between the time received by the remote control device 1000 and the time sent. The transmission delay calculated in this way can be called RTT (Round Trip Time). Similarly, if the time is recorded on the first mobile body 100 side, the RTT from the perspective of the first mobile body 100 can also be calculated.

[0057] The transmission delay distribution estimation unit 1001 uses the transmission delay information from the transmission delay measurement unit 1013 to output transmission delay distribution information related to the first moving body 100, that is, the transmission delay distribution information of the moving body 1. The transmission delay distribution information is information estimated based on a transmission delay model such as a transmission delay pattern in addition to the probability distribution of the transmission delay. The structure and operation of the transmission delay distribution estimation unit 1001 will be described later. Fig.10 To explain.

[0058] The mobile body action planning unit 1002 uses the map data of the surroundings of the first mobile body 100 acquired from the map database 500, the object information acquired via the network NW and output from the object information acquisition unit 200, the environment information output from the environment information acquisition unit 300, the mobile body 1 information output from the first mobile body 100, and the mobile body 1 transmission delay distribution information output from the transmission delay distribution estimation unit 1001 to plan the action of the first mobile body 100, that is, actions such as deceleration, obstacle avoidance, stoppage, lane change, road cliff avoidance, and emergency avoidance, and outputs it as the target action of the first mobile body 100, that is, the mobile body 1 target action. In addition, a typical action plan is known from Japanese Patent No. 6908211, which discloses an action planning device. In addition, in Japanese Patent No. 6908211, it is a technology that uses the scene information where the mobile body is located to determine the action, but in the present disclosure, it is characterized by using the transmission delay information to plan the action so that it can be applied to the remote control of the mobile body.

[0059] The mobile body control unit 1003 includes a first mobile body control unit 1031. The first mobile body control unit 1031 calculates a control amount for making the first mobile body 100 track a target track based on the mobile body information obtained from the network NW via the receiving unit 1012 and the mobile body 1 target behavior obtained from the mobile body behavior planning unit 1002. When the first mobile body 100 is a vehicle, the control amount is, for example, a target steering amount and a target acceleration / deceleration amount, and is output to the network NW via the sending unit 1004 as the mobile body 1 control amount.

[0060] By providing the moving object control unit 1003 , it is possible to control the first moving object 100 according to the action plan in the remote control device 1000 .

[0061] <Mobile body control unit>

[0062] Below, use Figure 2 The first moving body control unit 1031 of the moving body control unit 1003 will be described. Figure 2 It is a block diagram showing the structure of the first moving body control unit 1031.

[0063] like Figure 2 As shown, the first mobile body control unit 1031 includes a target trajectory generation unit 311 and a control amount calculation unit 312 .

[0064] The target trajectory generating unit 311 generates a target trajectory for the first moving body 100 to achieve the target behavior based on the map data around the first moving body 100 obtained from the map database 500 , the moving body 1 information obtained from the network NW via the receiving unit 1012 , and the moving body 1 target behavior obtained from the moving body behavior planning unit 1002 .

[0065] For example, if the target action is "stop at a position 5m ahead", a trajectory such as stopping at a position 5m ahead is generated, that is, a sequence of a target path and a target speed. Figure 6 to Figure 9 To give an example.

[0066] The control amount calculation unit 312 calculates a control amount such that the first moving body 100 tracks the target trajectory output from the target trajectory generation unit 311 .

[0067] Figure 3 1 is a diagram showing an example of a target path TR, that is, a sequence of positions in a target track, when the first moving body 100 is a vehicle. The target path TR is represented by an absolute coordinate system having an X-axis and a Y-axis. The lateral offset and the deflection angle of the first moving body 100 relative to the target path TR are represented by e y and e θ express.

[0068] The control amount calculation unit 312 is provided with Figure 3 In the case of a target path TR, the control amount of the first mobile body 100, which is the steering angle in this case, is calculated so as to track the target path TR. In the case of a sequence of target speeds included in the target trajectory, the control amount of controlling the accelerator and the brake so as to achieve the target speed is calculated.

[0069] The target orbit generation unit 311 and the control amount calculation unit 312 can also be executed simultaneously using model predictive control or the like. In model predictive control, the target orbit and the control amount can be calculated simultaneously through successive optimization using a model (state equation) that predicts the behavior of the moving body and an evaluation function. In successive optimization, the optimization is performed while predicting the state of the moving body within a certain time interval (horizons) in the future, such as position and speed, so it is called model predictive control. In model predictive control, the control amount within the horizon is optimized, but if the control amount is used, the optimal orbit of the moving body within the horizon can be calculated, and it can be used as the target orbit. That is, the target orbit and the control amount calculation can be performed simultaneously.

[0070] Furthermore, the moving body control unit 1003 can also calculate the control amount using the transmission delay profile information of the moving body 1 outputted from the transmission delay profile estimation unit 1001. This method is known from Japanese Patent No. 6940036.

[0071] In addition, the mobile body control unit can also be provided in the first mobile body 100. That is, it is also possible to send only the target action to the first mobile body 100 from the remote control device 1000, generate a target trajectory in the first mobile body 100, calculate the control amount, such as the target steering angle, based on the target trajectory, and control the first mobile body 100 based on the control amount. In this way, the first mobile body 100 can be controlled by the first mobile body 100 itself according to the action plan.

[0072] <Mobile body>

[0073] Next, use Figure 4 as well as Figure 5 The structure of the first moving body 100 will be described. Figure 4 1 is a block diagram showing the structure of the first moving body 100. Figure 4 As shown, the first moving body 100 includes an internal sensor 401 , a command value calculation unit 402 , an actuator 403 , a receiving unit 404 , a transmitting unit 405 , and a time synchronization unit 406 .

[0074] The internal sensor 401 is an IMU (Inertial Measurement Unit) sensor, a speed sensor, an acceleration sensor, a steering angle sensor, and a steering torque sensor, etc., which detects internal information of the first moving body 100, outputs it as moving body 1 information, and inputs it into the network NW via the sending unit 405.

[0075] The command value calculation unit 402 obtains the control amount of the mobile body 1 calculated by the mobile body control unit 1003 of the remote control device 1000 via the receiving unit 404, and performs calculation to convert it into an actuator command value that can be input to the actuator 403. For example, if it is a target steering angle, it is converted into a control current value of the electric power steering (EPS), etc. The actuator 403 is composed of a motor that actually moves the first mobile body 100.

[0076] The time synchronization unit 406 has a function of synchronizing the timing of data transmission and reception in cooperation with the time synchronization unit 201 in the object information acquisition unit 200 , the time synchronization unit 310 in the environment information acquisition unit 300 , and the time synchronization unit 1011 in the remote control device 1000 .

[0077] Figure 5 1 is a diagram showing an example of a structure in which the first mobile body 100 is a vehicle. A steering wheel 1 provided for a driver, i.e., a driver, to operate the vehicle is engaged with a steering shaft 2. The steering shaft 2 is engaged with a pinion shaft 13 of a rack and pinion mechanism 4. The rack shaft 14 of the rack and pinion mechanism 4 freely moves back and forth according to the rotation of the pinion shaft 13, and a front steering knuckle 6 is connected to the left and right ends thereof via a tie rod 5. The front steering knuckle 6 rotatably supports a front wheel 15 as a steering wheel, and is supported on a vehicle body frame so that the steering wheel can be freely turned.

[0078] The torque generated by the driver's operation of the steering wheel 1 rotates the steering shaft 2, and the rack and pinion mechanism 4 moves the rack shaft 14 to the left and right according to the rotation of the steering shaft 2. The front knuckle 6 rotates around a steering pin shaft (not shown) by the movement of the rack shaft 14, thereby turning the front wheel 15 to the left and right. Therefore, the driver can change the lateral movement amount of the vehicle by operating the steering wheel 1 when the vehicle moves forward and backward.

[0079] Furthermore, in the case of non-rider mobile objects such as fully automated vehicles and drones, there is no need for a driver-operated component such as a steering wheel.

[0080] The first moving body 100 is provided with a vehicle speed sensor 20 , an IMU sensor 21 , a steering angle sensor 22 , and a steering torque sensor 23 as internal sensors 401 for identifying the moving state of the first moving body 100 , and the detection values ​​are input to a command value calculation unit 402 .

[0081] The command value calculation unit 402 uses Figure 4 As described above, calculation is performed to convert the control amount of the mobile body 1 into an actuator command value that can be input to the actuator 403 , and the actuator command value is input to the acceleration / deceleration control device 9 and the steering control device 12 .

[0082] The first moving body 100 is provided with actuators such as an electric motor 3 for realizing lateral movement of the first moving body 100 , a vehicle driving device 7 for controlling movement of the first moving body 100 in the front-rear direction, and a brake control device 10 .

[0083] The acceleration / deceleration control device 9 controls the vehicle drive device 7 and the brake control device 10 , and the steering control device 12 controls the electric motor 3 .

[0084] The electric motor 3 is generally composed of a motor and a gear, and can freely rotate the steering shaft 2 by providing torque to the steering shaft 2. That is, the electric motor 3 can freely steer the front wheels 15 independently of the driver's steering wheel operation.

[0085] The vehicle drive device 7 is an actuator for driving the first mobile body 100 in the front-rear direction. The vehicle drive device 7 provides a driving force obtained by a driving source such as an engine or a motor via a transmission device and a shaft (not shown), so as to rotate the front wheels 15 and the rear wheels 16. Thus, the vehicle drive device 7 can freely control the driving force of the first mobile body 100.

[0086] On the other hand, the brake control device 10 is an actuator for braking the first mobile body 100, and controls the braking amount of the brakes 11 provided on the front wheels 15 and the rear wheels 16 of the first mobile body 100. A general brake generates a braking force by pressing a brake disc against a disc rotor that rotates together with the front wheels 15 and the rear wheels 16 using hydraulic pressure.

[0087] The internal sensors and other devices form a network using a CAN (Controller Area Network) or a LAN (Local Area Network) in the first moving body 100 . Figure 5 Each device in the first mobile body 100 shown can obtain its own information via the network. In addition, the internal sensors can send and receive data to each other via the network. In addition, even if the first mobile body is other than a vehicle, it has the same structure as the actuator, internal sensor, command value calculation unit, etc.

[0088] <Object Information Acquisition Unit and Target Trajectory Generation Unit>

[0089] Next, use Figure 6 as well as Figure 7 An example of the configuration of the object information acquisition unit 200 and an example of the target trajectory generated in the remote control device 1000 are described below. Figure 6 , as an example of the arrangement of the object information acquisition unit 200, the case where the external sensors 42 and 43 are arranged beside the road on which the first moving body 100 moves is shown, and a stationary object OB exists in front of the first moving body 100. The detection ranges of the external sensors 42 and 43 are ranges R42 and R43, respectively.

[0090] The external sensors 42 and 43 are composed of cameras, LiDAR, radar, sonar, infrared cameras, etc., and detect the position and speed of the first moving body 100 and other objects. Figure 6 It is arranged beside the road, but can also be carried on the first moving body 100.

[0091] Figure 7 1 is a diagram showing a target path TR for generating a target trajectory for the first moving body 100 to avoid the stationary object OB when there is a stationary object OB in front of the first moving body 100 .

[0092] use Figure 6 The external sensor 42 in the first moving body 100 detects the relative position and relative speed of the first moving body 100 with respect to the external sensor 42. In addition, the external sensors 42 and 43 are used to detect the relative position and relative speed of the stationary object OB with respect to the external sensors 42 and 43. The object information acquisition unit 200 transforms the information of the relative position and speed of the moving body, the stationary object OB, and the external sensors 42 and 43 into the information of the relative position and speed of the stationary object OB observed from the first moving body 100. Alternatively, the information of the relative position and speed of the first moving body 100, the stationary object OB, and the external sensors 42 and 43 is transformed into a coordinate system unified in the first moving body 100 and the stationary object OB, such as a geographic coordinate system used in GNSS, etc., thereby calculating the relative position and speed of the first moving body 100 and the stationary object OB.

[0093] When the target action of the mobile body 1 such as "avoiding an object" is input from the mobile body action planning unit 1002 of the remote control device 1000, the target trajectory generating unit 311 in the first mobile body control unit 1031 generates the following based on the information: Figure 7 The target path TR is a path such that the first moving body 100 avoids the stationary object OB and moves within the movable area RR. Although not shown here, the target trajectory generation unit 311 also generates a target speed of the first moving body 100, which is combined with the target path TR as a target trajectory.

[0094] Figure 8 is a diagram showing an example of the configuration of the object information acquisition unit 200 and the environment information acquisition unit 300. Fig. 9 1 is a diagram showing an example of a target speed generated in the remote control device 1000 .

[0095] exist Figure 8 In the figure, as an example of the arrangement of the object information acquisition unit 200 when there is a stop line STL and a traffic light TL in front of the first moving body 100, the case where the external sensor 42 is arranged beside the road on which the first moving body 100 moves is shown, and as an example of the arrangement of the environment information acquisition unit 300, the case where the external sensor 52 is arranged at a position where the light emission colors of the stop line STL and the traffic light TL can be detected is shown. The detection ranges of the external sensors 42 and 52 are ranges R42 and R52, respectively.

[0096] use Figure 8 The external sensor 42 detects the relative position and relative speed of the first moving body 100 with respect to the external sensor 42 , and the external sensor 52 detects the relative positions of the stop line STL and the traffic light TL with respect to the external sensor 52 .

[0097] use Figure 8 The external sensor 42 detects the relative position and relative speed of the first moving body 100 with respect to the external sensor 42, and the external sensor 52 detects the light emission colors of the stop line STL and the signal light TL.

[0098] exist Fig. 9 In FIG. 1 , the horizontal axis represents the moving distance when the first movable body 100 moves toward the stop line STL, and the vertical axis represents the speed of the first movable body 100 .

[0099] When the external sensor 52 detects that the traffic light TL is red, and when the target action of the mobile body 1 such as "the traffic light is red, please stop at the stop line" is input from the mobile body action planning unit 1002 of the remote control device 1000, the target trajectory generating unit 311 in the first mobile body control unit 1031 generates the target trajectory according to the information. Figure 8 The target path TR shown by the dashed line in the figure stops at the stop line STL, and is set as in Fig. 9 The target speed TV is shown by the dashed line in FIG. The target speed TV is a speed such that the speed of the first moving body 100 is gradually reduced and becomes zero at the stop line STL. The target trajectory generation unit 311 generates a target trajectory (stop trajectory) that combines the target path and the target speed.

[0100] <Transmission Delay Distribution Estimation Unit>

[0101] Fig.101001 is a block diagram showing an example of the configuration of the transmission delay distribution estimation unit 1001. In this example, the transmission delay distribution estimation unit 1001 is composed of a transmission delay preprocessing unit 111 and a transmission delay model unit 112.

[0102] The transmission delay preprocessing unit 111 has a function of converting the transmission delay information from the transmission delay measuring unit 1013 into a transmission delay feature quantity referenced by the transmission delay model unit 112. The transmission delay feature quantity can be, for example, the average value, variance, or higher-order moment of the transmission delay distribution within a predetermined time interval. Alternatively, the maximum value or minimum value of the transmission delay within the time interval can also be used as the transmission delay feature quantity.

[0103] The transmission delay model unit 112 has a function of estimating at least the probability distribution of the current transmission delay using the transmission delay model constructed in advance by using the transmission delay feature calculated by the transmission delay preprocessing unit 111, and outputting it as transmission delay distribution information. In addition, the transmission delay distribution information can also include information other than the above-mentioned probability distribution, such as the current or past mode in the hidden Markov model described later. Various models can be used in the transmission delay model unit 112, but in this disclosure, a hidden Markov model ("Hidden Markov Model", hereinafter abbreviated as "HMM") is described as an example of a transmission delay model.

[0104] HMM is a probability model that is constructed to output a sequence of modes (states) that follow a discrete or continuous probability distribution, and transitions according to transition probabilities determined between each mode. Hereinafter, the probability distribution corresponding to each mode in HMM is referred to as output distribution.

[0105] Explain the output of HMM and the transition of modes. For example, when the HMM is in mode A at a certain moment, the output follows the probability distribution of mode A. On the other hand, there are also cases where the mode is transferred to another mode according to a certain transition probability, and the probability distribution of the output changes. For example, when it is transferred from mode A to mode B, in the time interval of mode B, the output follows the probability distribution of mode B. In HMM, it is impossible to directly observe which mode is currently in, but only its output sequence is observed, so it is called "hidden".

[0106] Next, use Fig.11 , explaining why transmission delay can be modeled using HMM. Fig.11 is a diagram showing an example of a time series of transmission delays. Fig.11 In the figure, the horizontal axis represents time and the vertical axis represents the transmission delay amount, which can be easily obtained using the output of the transmission delay measurement unit 1013.

[0107] Regarding transmission delay, when a dedicated line is not used, it is generally not a constant value, but always has a varying value. Fig.11 As shown, it can be seen that the deviation pattern changes for each certain time interval. Fig.11 The time interval 1, the time interval 3, or the time interval 4 and the time interval 6 show the same degree of deviation, and the time interval 2 and the time interval 5 are the intervals where a large transmission delay is obviously likely to occur.

[0108] Here, regarding the transmission delay, when considering the HMM output sequence, the time intervals showing the same tendency of deviation can be regarded as the same pattern in the HMM. That is, it is considered that time interval 1 and time interval 3, time interval 2 and time interval 4 are the same pattern and can be set as pattern 1 and pattern 2 respectively. Similarly, time interval 2 and time interval 5 are considered as pattern 3 and pattern 4 respectively.

[0109] When these are represented by HMM, Fig.12 Modeling shown. Fig.12 This is an HMM having 4 patterns in total, and each delay pattern can be expressed as follows.

[0110] Mode 1: Mode with small variance

[0111] Mode 2: Mode with large variance

[0112] Mode 3: Mode that generates small bursts

[0113] Mode 4: Mode that generates large bursts

[0114] However, these distributions are only hypothetical, and note that actual communication delays will not take negative values ​​as in a normal distribution.

[0115] exist Fig.12 In the ij (i=1, 2, 3, 4, j=1, 2, 3, 4) As the transition probability from mode i of the transition source to mode j of the transition target, for example, when the mode of the transition source is set to 1, p11, p12, and p13 respectively represent the transition probability from mode 1 to mode 1, the transition probability from mode 1 to mode 2, and the transition probability from mode 1 to mode 3. The same is true for mode 2, mode 3, and mode 4.

[0116] In this way, as an example of a transmission delay model, the transmission delay can be modeled using HMM.

[0117] In addition, as to why the transmission delay is modeled as such, the inventors' view is described. In the case of a general network, in order to efficiently and reliably send packets, which are units of data transmission and reception, to the correct transmission destination, path control is performed. Due to this path control, the transmission path of the packet sometimes changes, which can be interpreted as a mode transition to express the switching of the transmission path. Regarding such path control, the frequency is small in a small-scale network, but in the case of a large-scale network, the switching frequency of the transmission path and the change in the deviation mode of the transmission delay become large. Such switching of the transmission path can be interpreted as switching of the mode in the HMM.

[0118] If the above-mentioned model is used, it becomes clear to which mode the current mode is likely to shift, so the accuracy of the action plan is improved. In addition, the modes 1 to 4 are sometimes referred to as delay modes 1 to 4.

[0119] For example, in the case of mode 1, when p11 = 80%, p12 = 19%, p13 = 1%, etc., it is difficult to shift to mode 3 where a burst occurs, so the risk of transmission delay is considered to be small.

[0120] The HMM described above is a non-hierarchical hidden Markov model, but a hierarchical hidden Markov model obtained by hierarchizing the HMM can be used as a more precise transmission delay model. Both the non-hierarchical and hierarchical hidden Markov models can accurately predict transmission delays.

[0121] <Method for Creating HMM Using Prior Information>

[0122] In the above description, the transmission delay distribution estimation unit 1001 obtains the transmission delay information from the transmission delay measurement unit 1013 online and generates the HMM using the transmission delay model unit 112. However, the HMM can also be generated based on the transmission delay information obtained in advance by the transmission delay measurement unit 1013.

[0123] That is, a set of sequence data is obtained periodically, for example, at intervals of 0.01 seconds, or aperiodically, with a certain time width, for example, one hour, and a plurality of such sets are obtained. The plurality of obtained data sets are defined as "prior information".

[0124] For the obtained data set, a certain time interval, such as about 1 second, is determined, and the average, variance, maximum value, minimum value, etc., which can be used to estimate the mode of the transmission delay according to the time interval, are set as the transmission delay feature. In addition, the transmission delay feature can also be defined as "prior information". For each of the transmission delay feature, the pattern is classified by clustering and other techniques. This is performed on multiple data sets to obtain the transition probability between patterns and the output distribution of each pattern, so that it can be used as the final HMM.

[0125] In addition, the transmission delay information obtained online from the transmission delay measurement unit 1013 when actually controlling the first moving body 100 is distinguished from the advance information and defined as "post-event information". Post-event information means the transmission delay information obtained when the remote control device 1000 performs remote control of the first moving body 100. By creating an HMM based on the advance information, the time required for HMM creation can be shortened, and by creating an HMM based on the post-event information, an HMM that is more in line with reality can be created. In Embodiment 3, a method for creating an HMM based on post-event information is described.

[0126] <Mobile Action Planning Department>

[0127] Fig.13 1 is a block diagram showing an example of the structure of the mobile body behavior planning unit 1002. In this example, the mobile body behavior planning unit 1002 is composed of a mobile body 1 behavior planning unit 1021, and the map data of the surroundings of the first mobile body 100 obtained from the map database 500, the mobile body 1 information obtained from the network NW via the receiving unit 1012, and the surrounding information are input to the mobile body 1 behavior planning unit 1021.

[0128] The mobile body 1 action planning unit 1021 determines the action plan for the normal movement of the first mobile body 100 based on this information, that is, target actions such as normal movement, emergency stop and speed limit, but has the function of changing the target action according to the mobile body 1 transmission delay distribution information output from the transmission delay distribution estimation unit 1001.

[0129] Fig.14 This is a diagram schematically showing the operation of changing the target behavior based on the transmission delay distribution information of the mobile body 1 in the mobile body 1 behavior planning unit 1021.

[0130] exist Fig.14 In the figure, it is shown that the first moving body 100 is Fig.12 In the case where the delay mode 1 shown in the figure changes to the delay mode 2 during normal movement, the transmission delay has a large deviation, so the speed is limited. In the case where the delay mode 3 or the delay mode 4 is shown in the figure, the possibility of sudden delay is high, so the first moving body 100 is stopped urgently.

[0131] In this way, by changing the target behavior based on the transmission delay distribution information of the mobile body 1, it is possible to plan the behavior in consideration of the situation where transmission delay is likely to occur.

[0132] In addition, the target behavior can be expressed by the target position, posture, path, etc. of the first moving body 100 in addition to the above-described behavior.

[0133] <Mobile Action Planning Unit including Risk Assessment Unit>

[0134] Fig.15 1 is a block diagram showing an example of the configuration of the moving body behavior planning unit 1002. In this example, the moving body behavior planning unit 1002 is composed of a risk determination unit 1022 and a moving body 1 behavior planning unit 1021, and the risk determination unit 1022 is composed of a first moving body risk determination unit 10221.

[0135] The mobile body 1 behavior planning unit 1021 plans behavior using the risk calculated by the first mobile body risk determination unit 10221 .

[0136] In the first moving body risk determination unit 10221, map data, moving body 1 information, surrounding information and moving body 1 transmission delay distribution information from the transmission delay distribution estimation unit 1001 are used to numerically calculate the risk of dangers such as lane departure, collision with obstacles, collision with other moving bodies, collision with walls, etc. of the first moving body 100, which is determined to be dangerous, and output as the first moving body risk.

[0137] In the calculation of risk, a state equation of the moving object or a dynamic model or dynamics can be used. A state equation expressing a relative relationship between the position and velocity of the moving object and surrounding objects can be used.

[0138] Risks such as transmission delay and relative positional relationship with surrounding objects can be determined, and actions can be planned based on the risks, thereby improving the accuracy of action planning.

[0139] The moving body 1 behavior planning unit 1021 has a function of changing a target behavior with respect to the behavior plan for normal movement of the first moving body 100 using the risk output from the first moving body risk determination unit 10221 .

[0140] Fig.16 This is a diagram schematically showing an operation of changing the target behavior in the mobile body 1 behavior planning unit 1021 according to the first mobile body risk from the first mobile body risk determination unit 10221 .

[0141] exist Fig.16In the figure, it is shown that when the risk output from the first moving body risk determination unit 10221 becomes large (large risk) during the normal movement of the first moving body 100, the first moving body 100 is made to stop urgently, and when the risk becomes medium (medium risk), the first moving body 100 is made to automatically retreat to a road cliff, etc.

[0142] In this way, the target action is determined based on the actual risk rather than the transmission delay pattern, so the accuracy of the action plan is improved. For example, when performing remote control on a plain without any obstacles, there is no risk of collision, so even if a large delay occurs, it is judged as no risk, and a high-precision action plan can be formulated.

[0143] <Calculation method of transmission delay risk>

[0144] use Fig.12 The delay pattern of the HMM shown illustrates the method of calculating the transmission delay risk.

[0145] Set the current time to k and the current mode to M k , set the next moment mode to M k+1 At this moment, at the current time k, the mode of HMM is 1, set to M k = 1. Use the following formula (1) to calculate the transmission delay h at the next time k+1 k+1 becomes greater than the risk threshold h t The event represented by this probability is referred to as event A1.

[0146] [Formula 1]

[0147] P(h k+1 >h t |M k =1)

[0148] =P(h k+1 >h t |M k+1 =1)P(M k+1 =1|M k =1)+P(h k+1 >h t |M k+1 =2)P(M k+1 =2M k =1)+P(h k+1 >h t |M k+1 =3)P(M k+1 =3|M k =1)

[0149] =P(h k+1 >ht |M k+1 =1)p 11 +P(h k+1 >h t |M k+1 =2)p 12 +P(h k+1 >h t ||M k+1 =3)p 13

[0150] …(1)

[0151] Here, when the cumulative distribution function of the output distribution of mode 1 is set to F1, P(h k+1 >h t |M k+1 =1). The same is true for other modes.

[0152] [Formula 2]

[0153] P(h k+1 >h t |M k+1 =1) = 1-F1(h t )…(2)

[0154] The probability of occurrence of event A1 is captured as risk, and when the risk becomes a predetermined % or more, the risk is considered high and an action plan such as emergency stop is formulated.

[0155] In addition, although the transmission delay risk at time k+1 is calculated here, if the same calculation can be performed, the transmission delay risks after time k+1, such as k+2, k+3, etc., can also be calculated.

[0156] In the above description, the case of using the transmission delay model of the HMM is described, but other probability models can also be used for calculation in the same manner.

[0157] <Calculation method of risk based on dynamic model>

[0158] Risks other than the transmission delay risk can also be calculated using a dynamic model, for example, and the risk of lane departure, collision with an obstacle, collision with another moving object, and collision with a wall can be calculated as described above. Figure 3 , illustrating a method for calculating the risk of lane departure using a dynamic model.

[0159] like Figure 3 As shown in FIG. 1 , the lateral deviation and the deflection angle of the first moving body 100 relative to the target path TR represented by the absolute coordinate system having the X axis and the Y axis are respectively represented by e y and eθ When , the lateral continuous-time state equation of the first moving body 100 can be expressed by the following formula (3).

[0160] [Formula 3]

[0161]

[0162] v x :Vehicle speed [m / s]

[0163] δ: Rudder angle [rad]

[0164] m: mass [kg]

[0165] L f : Distance between center of gravity and front wheel axle [m]

[0166] L r : Distance between center of gravity and rear wheel axle [m]

[0167] I z : Moment of inertia about the yaw axis [kg·m 2 ]

[0168] C f : Front wheel cornering stiffness [N / rad]

[0169] C r : Rear wheel cornering stiffness [N / rad]

[0170] e y : Lateral deviation of the vehicle's center of gravity to the target path [m]

[0171] e θ : The deviation angle from the center of gravity of the vehicle to the target path [rad]

[0172] The time differential associated with the lateral displacement of the moving body

[0173] Time derivative related to the deflection angle of the moving body

[0174] Cornering stiffness is a proportionality coefficient indicating the relationship between the lateral force generated in the moving body and the lateral slip angle, and is a value that changes depending on the state of the contact surface between the moving body and the road surface, such as a dry surface, a wet surface, or an icy surface.

[0175] By using such a dynamics model, it is possible to evaluate lateral risks such as how much the moving object will deviate from the target path at a future time and the degree of the risk of lane departure.

[0176] Next, a case where it is desired to evaluate the longitudinal risk of the first mobile body 100 will be described. The longitudinal risk is the risk of colliding with an obstacle or a wall ahead, etc. The longitudinal continuous-time state equation of the vehicle of the following formula (4) can be used.

[0177] [Formula 4]

[0178]

[0179] Formula (4) uses the longitudinal acceleration α x , will be increased from the target acceleration u a To vehicle speed v x The state equation is modeled as the time constant T a The case of a single-delay system.

[0180] By using this formula, the risk of how much the first moving body 100 will move forward at a future moment can be evaluated. In addition, a continuous-time state equation including the relative distance to the obstacle ahead and the relative speed can be formed. In this case, the risk of colliding with the obstacle ahead can be evaluated. However, although this example is a primary delay system, a more detailed model can also be used, such as a secondary delay system, and a model of the characteristics of an actuator used in the control of the front and rear directions of the moving body.

[0181] In this way, various risks can be evaluated by using a continuous-time state equation that fits the risk that one wishes to evaluate.

[0182] Fig.17 1 is a block diagram showing an example of a control system in which the remote control device 1000 controls the first moving object 100 which is a control target in a transmission delay environment.

[0183] exist Fig.17 In the figure, the solid line represents the input and output of the signal represented by continuous values, and the dotted line represents the input and output of the signal represented by discrete values. c and u c are the states and inputs in continuous time.

[0184] The moving body information of the first moving body 100 obtained by various sensors is a discrete value, so the moving body information corresponds to the output value of the sampler S. The moving body information is transmitted to the remote control device 1000 via the network NW, so a transmission delay occurs at this time, and an upload transmission delay D occurs here. UP The mobile information delay is the upload transmission delay D upThe controller ψ outputs the control amount calculated using the control gain based on the mobile body information. This control amount is equivalent to the control amount output by the control amount calculation unit 312 of the first mobile body control unit 1031. The control amount is sent to the mobile body via the network NW, so a transmission delay occurs at this time, and a download transmission delay D occurs here. dw The control amount input to the first moving body 100 at a certain time becomes a constant value by the holder H until the next input. That is, the holder H has a 0th order holding function. The 0th order held control amount is input to the first moving body 100 as the control object Pc.

[0185] By using Figure 3 An absolute coordinate system with X-axis and Y-axis and Fig.17 The control system shown takes into account the transmission delay to include u in the state vector k-1 In the form of, the discrete time state equation of the expanded system is constructed. Here, the current time is set to k, and the control input at the current time is set to u k , set the previous moment to k-1, and the control input to u k-1 .

[0186] When the probability variable representing the transmission delay is set to h k When X is the state x of the next moment k+1 It is expressed by the following formula (5).

[0187] [Formula 5]

[0188] x k+1 =A(h k )x k +B(h k ) k …(5)

[0189] In addition, y, which is the evaluation quantity for evaluating the risk at the next moment, k+1 It is expressed by the following formula (6).

[0190] [Formula 6]

[0191] y k+1 =Cx k+1 …(6)

[0192] Here, x k It is the state at time k plus the input u at time k-1. k-1 The state vector obtained, in the case of formula (3), is k It is expressed by the following formula (7).

[0193] [Formula 7]

[0194]

[0195] In the above formula, C is used to generate y according to the state vector k+1 In addition, let the matrix A at time k be k , Matrix B k When , it is expressed by the following formula (8) and formula (9) respectively.

[0196] [Formula 8]

[0197]

[0198] [Formula 9]

[0199]

[0200] Here, A in formula (8) c And B in formula (9) c is the coefficient matrix of the continuous-time state equation.

[0201] Furthermore, matrices A and B of formula (5) are expressed by the following formulas (10) and (11), respectively.

[0202] [Formula 10]

[0203]

[0204] [Formula 11]

[0205]

[0206] For example, when the lateral position deviation e from the target path is set as the evaluation amount y，k+1 In the case of , the matrix e of formula (6) can be expressed by the following formula (12).

[0207] [Formula 12]

[0208] C=[1 0 0 0 0]…(12)

[0209] Use as the transmission delay h k The function of g k+1 (h k ), the evaluation value y obtained in this way is expressed by the following formula (13): k+1 .

[0210] [Formula 13]

[0211] y k+1 :=g k+1 (h k )=C{A(h k )xk +B(h k ) k}…(13)

[0212] At this time, you can t (>0), count back to the initial y k+1 =y t The transmission delay h k The result of this back calculation is recorded as g k -1 (y t ).

[0213] The evaluation amount y of the first moving body 100 at time k+1 is k+1 becomes greater than y t (>0) is defined as event A2, and the probability of event A2 occurring as a risk is considered. If it is determined at time k that the state is in mode 1 at time k-1, the evaluation value y k+1 Initially, it becomes the evaluation quantity y t The transmission delay h k =g k+1 -1 (y t ) is set to h A2，k When the probability of event A2 occurs, that is, the evaluation value y k+1 becomes greater than the evaluation value y t The probability of is expressed by the following formula (14), and if it is calculated by the method described in the calculation method of transmission delay risk, the risk can be calculated.

[0214] [Formula 14]

[0215] P(y k+1 >y t |M k-1 =1) = P(h k >h A2，k |M k-1 =1)…(14)

[0216] In addition, the transmission delay h can be calculated analytically or numerically. A2,k and greater than the transmission delay h A2，k probability.

[0217] In the above, the evaluation method of the risk that the deviation increases toward the positive side is described, but the risk that the deviation decreases toward the negative side can also be evaluated by the same consideration method.

[0218] In another example, the risk can also be set as the evaluation amount y k+1 The expected value y e，k+1When it is determined at time k that the state is in mode 1 at time k-1, the expected value y can be calculated using the following formula (15): e，k+1 .

[0219] [Formula 15]

[0220]

[0221] Here, F(|) is the conditional cumulative distribution function. In addition to the expected value, y e，k+1 Variance and higher order moments etc. can also be used as risks.

[0222] In the above, the risk at time k+1 is calculated. However, by performing the same calculation as above, the risks at time k+2, k+3, ... and after time k+1 can also be calculated, similarly to the case of the transmission delay risk.

[0223] If the configuration of the first mobile body control unit 1031 is known, the information can be used to predict the future control input u at time k+1 or later. k+1 wait.

[0224] In addition, since the above-mentioned formula (3) depends on the dynamic model of the first moving body 100, when the technology of the present disclosure is applied to other moving bodies, the formula (3) is changed and calculated in accordance with the moving body of interest.

[0225] In addition, although risk assessment in the form of a continuous-time state equation has been described above, risk can also be similarly assessed by using a formula that expresses a dynamic model and dynamics, such as an equation of motion, instead of a continuous-time state equation.

[0226] When the dynamics model of the moving object is complex, it may be difficult to easily find P(y k+1 >y t |M k-1 =1), y e,k+1 In this case, it is also possible to use numerical calculation techniques such as the Monte Carlo method for calculation.

[0227] <Implementation Method 2>

[0228] <Overall Structure>

[0229] Fig.18 This is a block diagram showing an example of the configuration of the remote control device 2000 according to the second embodiment of the present disclosure and the configuration of a remote control system RCS2 of a mobile object MV which is remotely controlled via a network NW.

[0230] like Fig.18As shown, the remote control system RCS2 has a configuration in which a mobile object MV, a remote control device 2000 , an object information acquisition unit 200 , and an environment information acquisition unit 300 are connected to a network NW.

[0231] The remote control device 2000 of the second embodiment is different from the remote control device 1000 in that it controls a plurality of moving objects. The moving object control unit 1003 includes a first moving object control unit 1031 and a second moving object control unit 1032. The moving objects MV to be controlled are the first moving object 100 and the second moving object 101. Fig.18 About and use Figure 1 The same components as those of the remote control device 1000 described above are denoted by the same reference numerals, and duplicate descriptions thereof will be omitted.

[0232] The first moving body 100 and the second moving body 101 move respectively according to the moving body 1 control amount and the moving body 2 control amount sent from the sending unit 1004 of the remote control device 2000, and output the state amount of the moving body detected by the internal sensors composed of the speed sensors etc. respectively carried as the moving body 1 information and the moving body 2 information.

[0233] The object information acquisition unit 200 is composed of one or more sensors mounted around the first moving body 100 and the second moving body 101 or the first moving body 100 and the second moving body 101. The object information acquisition unit 200 acquires the position and speed of obstacles such as other vehicles, bicycles and pedestrians around the first moving body 100 and the second moving body 101 as environmental information. In addition, the object information acquisition unit 200 can acquire the position and speed of the first moving body 100 itself as moving body 1 information, and can acquire the position and speed of the second moving body 101 itself as moving body 2 information. In this case, the moving body information is a part of the object information. The object information acquisition unit 200 sends the moving body information to the receiving unit 1012 in the remote control device 2000 via the network NW. In addition, when the first moving body 100 is provided with an internal sensor, the moving body information can also be acquired from the internal sensor, and when the second moving body 101 is provided with an internal sensor, the moving body information can also be acquired from the internal sensor. Therefore, the moving object information can be acquired from the object information acquisition unit 200 as well as from the first moving object 100 and the second moving object 101 .

[0234] The environment information acquisition unit 300 is composed of one or more sensors installed around the first moving body 100 and one or more sensors installed around the second moving body 101, similarly to the object information acquisition unit 200. The environment information acquisition unit 300 acquires environment information such as traffic lights and stop lines. The environment information acquisition unit 300 transmits the environment information to the receiving unit 1012 in the remote control device 2000 via the network NW.

[0235] Next, the components of the remote control device 2000 are described. Fig.18 As shown, the remote control device 2000 includes a transmission delay distribution estimation unit 1001 , a mobile behavior planning unit 1002 , a mobile control unit 1003 , a transmission unit 1004 , a time synchronization unit 1011 , a reception unit 1012 , and a transmission delay measurement unit 1013 .

[0236] These have Figure 1 The remote control device 1000 shown in the figure has the same functions, but the transmission delay measuring unit 1013 uses the time synchronized by the time synchronization unit 1011 to measure the various transmission delays generated between the first mobile body 100 and the second mobile body 101 and the remote control device 2000, and outputs them to the transmission delay distribution estimation unit 1001 as the mobile body 1 transmission delay information of the first mobile body 100 and the mobile body 2 transmission delay information of the second mobile body 101.

[0237] The transmission delay profile estimation unit 1001 uses the transmission delay information of the mobile body 1 and the transmission delay information of the mobile body 2 from the transmission delay measurement unit 1013 to output the transmission delay profile information of the mobile body 1 related to the first mobile body 100 and the transmission delay profile information of the mobile body 2 related to the second mobile body 101 .

[0238] In addition, when the network environment and surrounding conditions of the plurality of mobile bodies are roughly the same and the tendency of transmission delay is considered to be of the same degree, the transmission delay measurement unit 1013 and the transmission delay distribution estimation unit 1001 regard the transmission delay information of the first mobile body 100 and the transmission delay information of the second mobile body 101 as the same, group the first mobile body 100 and the second mobile body 101, use the transmission delay information of either party to estimate the common transmission delay distribution information and output it to the mobile body action planning unit 1002.

[0239] This makes it possible to reduce the calculation load by only calculating the estimated transmission delay distribution once. The same is true when there are three or more moving objects.

[0240] The mobile body action planning unit 1002 uses the map data of the surroundings of the first mobile body 100 obtained from the map database 500, the object information obtained via the network NW and output from the object information acquisition unit 200, the environmental information output from the environmental information acquisition unit 300, the mobile body 1 information output from the first mobile body 100, the mobile body 2 information output from the second mobile body 101, the mobile body 1 transmission delay distribution information and the mobile body 2 transmission delay distribution information output from the transmission delay distribution estimation unit 1001, and plans the actions of the first mobile body 100 and the second mobile body 101, namely, actions such as slowing down, avoiding obstacles, stopping, changing lanes, retreating to road cliffs, and emergency avoidance, as outputs of the mobile body 1 target action of the first mobile body 100 and the mobile body 2 target action of the second mobile body 101.

[0241] The first moving body control unit 1031 of the moving body control unit 1003 calculates the moving body 1 control amount for causing the first moving body 100 to track the target trajectory based on the moving body 1 information obtained from the network NW via the receiving unit 1012 and the moving body 1 target behavior obtained from the moving body behavior planning unit 1002 .

[0242] The second moving body control unit 1032 of the moving body control unit 1003 calculates the moving body 2 control amount for causing the second moving body 101 to track the target trajectory based on the moving body 2 information obtained from the network NW via the receiving unit 1012 and the moving body 2 target behavior obtained from the moving body behavior planning unit 1002 .

[0243] <Mobile Action Planning Department>

[0244] Fig.19 1 is a block diagram showing an example of the structure of the mobile body action planning unit 1002. In this example, the mobile body action planning unit 1002 is composed of a risk determination unit 1022 and a multi-unit action planning unit 1023, and the risk determination unit 1022 is composed of a first mobile body risk determination unit 10221 and a second mobile body risk determination unit 10222.

[0245] The multiple-unit action planning unit 1023 plans actions using the first moving object risk calculated by the first moving object risk determining unit 10221 and the second moving object risk calculated by the second moving object risk determining unit 10222 .

[0246] In the first moving body risk judgment unit 10221, map data, moving body 1 information, surrounding information and moving body 1 transmission delay distribution information from the transmission delay distribution estimation unit 1001 are used to numerically calculate the risk of dangers such as lane departure, collision with obstacles, collision with other moving bodies, collision with walls, etc. of the first moving body 100, which is judged as the first moving body risk output.

[0247] In the second moving body risk determination unit 10222, map data, moving body 2 information, surrounding information and moving body 2 transmission delay distribution information from the transmission delay distribution estimation unit 1001 are used to numerically calculate the risk of dangers such as lane departure, collision with obstacles, collision with other moving bodies, collision with walls, etc. of the second moving body 101, and output it to the multiple action planning units 1023 as the second moving body risk.

[0248] Risk calculation and Fig.15 The risk determination unit 1022 of the mobile body behavior planning unit 1002 shown is the same as that shown in the figure, and the description thereof is omitted.

[0249] <Multi-unit Action Plan>

[0250] use Fig. 20 Next, a description will be given of a plurality of moving objects' behavior plans created by the plurality of moving object behavior planning unit 1023 of the moving object behavior planning unit 1002 when the remote control device 2000 controls a plurality of moving objects.

[0251] Fig. 20 This is a diagram illustrating an action plan when the first moving body 100 and the second moving body 101 intersect, and two external sensors 42 are arranged next to the road on which the first moving body 100 moves. The two external sensors 42 are arranged at an interval such that a portion of the detection range indicated by the dotted line overlaps, covering the first moving body 100 and the second moving body 101 in separated positions.

[0252] Here, it is assumed that the transmission delay of the second moving body 101 is in a pattern that is prone to sudden changes, and the first moving body 100 is in a pattern with a small variance. In this case, the multi-unit action planning unit 1023 determines that the risk of the second moving body 101 is high, and generates a Fig. 20 On the other hand, if it is determined that the risk of the first moving body 100 is small, it is made to move straight as it is, but if it is determined that there is still a risk, a target path TR1 shown by the single-dot chain line is generated as shown in Fig. 20 The target path TR indicated by the dashed line in the middle instructs the first moving body 100 to behave so as to slightly move away from the second moving body 101 .

[0253] In this way, the multiple-unit action planning unit 1023 instructs each moving body to take actions that reduce the overall risk by taking into account the relative positions, speeds, and transmission delay distributions of the moving bodies. This allows the multiple moving bodies to operate efficiently.

[0254] Furthermore, the target trajectory of each moving object can be used to predict the future position of the moving object and calculate the risk.

[0255] In the above description, the behaviors of the plurality of moving bodies are simultaneously planned regarding the multi-machine behavior plan, but the behaviors of the plurality of moving bodies can also be planned independently.

[0256] <Implementation Method 3>

[0257] <Overall Structure>

[0258] Fig.21 This is a block diagram showing an example of the configuration of the remote control device 3000 according to the third embodiment of the present disclosure and the configuration of a remote control system RCS3 of a mobile object MV which is remotely controlled via a network NW.

[0259] like Fig.21 As shown, the remote control system RCS3 has a configuration in which a mobile object MV, a remote control device 3000 , an object information acquisition unit 200 , and an environment information acquisition unit 300 are connected to a network NW.

[0260] The remote control device 3000 of the third embodiment is Figure 1 The remote control device 1000 shown in the figure is different in that the transmission delay distribution estimation unit 1001 receives not only the transmission delay information from the transmission delay measurement unit 1013 but also the map data from the map database 500, the surrounding information obtained via the network NW, and the moving body information. Fig.21 About and use Figure 1 The same components as those of the remote control device 1000 described above are denoted by the same reference numerals, and duplicate descriptions thereof will be omitted.

[0261] In addition, Fig.21 In the example, for ease of explanation, the moving body MV is only the first moving body 100, but multiple moving bodies can also be combined. Fig.18 The remote control device 2000 shown is similarly controlled. In this case, the transmission delay distribution estimation unit 1001 receives information of a plurality of moving objects from the reception unit 1012 and outputs transmission delay distribution information of each of the plurality of moving objects to the moving object behavior planning unit 1002 .

[0262] Furthermore, when the network environment and surrounding conditions of the plurality of mobile bodies are substantially the same and the tendency of the transmission delay is considered to be of the same degree, the transmission delay measurement unit 1013 and the transmission delay distribution estimation unit 1001 consider the transmission delay information of the first mobile body 100 and the transmission delay information of the second mobile body 101 to be the same, group the first mobile body 100 and the second mobile body 101, estimate the common transmission delay distribution information using the transmission delay information of either party, and output it to the mobile body action planning unit 1002. Thus, the calculation of estimating the transmission delay distribution is sufficient only once, and the calculation load can be reduced.

[0263] <Transmission Delay Distribution Estimation Unit>

[0264] Fig. 22 1 is a block diagram showing an example of the configuration of the transmission delay distribution estimation unit 1001. In this example, the transmission delay distribution estimation unit 1001 is composed of a transmission delay preprocessing unit 111, a transmission delay model unit 112, and an environment preprocessing unit 113.

[0265] The transmission delay preprocessing unit 111 has a function of converting the transmission delay information from the transmission delay measuring unit 1013 into a transmission delay feature amount referenced by the transmission delay model unit 112 .

[0266] The transmission delay model unit 112 preliminarily models the transmission delay feature amount calculated by the transmission delay preprocessing unit 111, and calculates transmission delay distribution information by referring to the transmission delay feature amount and the environment feature amount.

[0267] Transmission delay is affected by the position of the moving object, the time period, and the surrounding environment, that is, the relative position of the surrounding structures and the moving object, etc. These information can be obtained from map data and moving object information.

[0268] The environment preprocessing unit 113 calculates the characteristic amount of the environment information other than the transmission delay information, that is, it calculates the environment characteristic amount that characterizes the time period, the position of the moving object, and the surrounding environment based on the moving object information and the map data.

[0269] The transmission delay feature is a feature directly obtained from the sequence of transmission delays, while the environmental feature is a feature representing the surrounding conditions of the mobile body, and is calculated based on physically measurable values ​​such as the current time, the condition of radio waves around the mobile body, the presence or absence of structures around the mobile body, the distance from the mobile body, the conductors around the mobile body, obstacles, radio wave strength, and traffic volume. In the environmental preprocessing unit 113, the environmental feature is calculated using map data, mobile body information, and surrounding information.

[0270] For example, if there is a building near a moving object, the location of the building can be detected from map data, and the location of the moving object can be detected from GNSS, so the relative distance can be digitized as an environmental feature. In addition, radio wave strength can also be detected by antennas and receivers, so it can be used as an environmental feature.

[0271] As for the transmission delay, as described above, the deviation mode changes due to the switching of the transmission path, but in addition, the transmission delay may also occur due to the load conditions of the network NW such as the conditions of line users, the traffic volume, and the characteristics of routers. In addition, since the moving body moves, the transmission delay may also occur due to the presence or absence of obstacles and interference on the radio wave propagation path, the presence or absence of conductors around the moving body, etc. It is believed that these factors affect each other and become the final transmission delay.

[0272] By inputting environmental feature quantities expressing such situations into the transmission delay model, it is possible to create the transmission delay model and estimate the transmission delay distribution with higher accuracy, which results in an effect of improving the accuracy of risk estimation and action planning.

[0273] Specifically, the transmission delay model in the third embodiment is constructed as follows: Fig.12 Use p in ij The transition probability between each mode represented by (i=1, 2, 3, 4, j=1, 2, 3, 4) changes according to the environmental feature quantity. For example, when moving to a place where there are many structures such as buildings that refract or shield radio waves, the transition probability is increased so that it is easy to transition to a mode with a large transmission delay. Alternatively, since the communication volume decreases at night, it is considered to model the transition probability so that it is not easy to transition to a mode with a large transmission delay.

[0274] In the transmission delay model unit 112 of implementation mode 3, the environmental characteristic quantity calculated by the environmental preprocessing unit 113 is also referenced. However, for example, when the radio wave strength is selected as the environmental characteristic quantity, the current mode of the HMM can be inferred by referring to the radio wave strength, so the probability distribution under the mode is output as the transmission delay distribution information.

[0275] <Implementation Method 4>

[0276] <Transmission Delay Distribution Estimation Unit>

[0277] Fig.23 1 is a block diagram showing an example of the structure of the transmission delay distribution estimation unit 1001 of the remote control device 4000 according to the fourth embodiment of the present disclosure. In this example, the transmission delay distribution estimation unit 1001 is composed of the model unit 115. In addition, other structures except the transmission delay distribution estimation unit 1001 are similar to those of FIG. Fig.21 The remote control device 3000 of the third embodiment shown in FIG. 1 is the same as that of the remote control device 3000 of the third embodiment shown in FIG. Fig.21 The same description is omitted.

[0278] The model unit 115 inputs the transmission delay information from the transmission delay measurement unit 1013, the map data from the map database 500, and the surrounding information and moving body information acquired via the network NW, and calculates the transmission delay distribution information through machine learning.

[0279] In recent years, machine learning technologies using AI (Artificial Intelligence) have developed significantly, led by deep learning technology.

[0280] In the present embodiment, by learning a transmission delay model using a machine learning technique and using the obtained learned model, it is possible to obtain transmission delay distribution information with high accuracy.

[0281] Fig.23 The transmission delay distribution estimation unit 1001 is a structure that uses a learned model learned by machine learning to output transmission delay distribution information online. When learning the transmission delay model, it is first necessary to obtain learning data. In order to obtain learning data, the first moving body 100 is moved, and the transmission delay information of the moving body 1 from the transmission delay measurement unit 1013, the surrounding information obtained via the network NW, and the moving body information are obtained and stored as a data group. The model unit 115 can perform learning using the stored data group and the map database 500.

[0282] The learning method of setting the transmission delay model as the HMM model is often studied in the field of speech recognition. By using this method, the HMM model can be learned. When it is desired to learn a more general transmission delay model, the machine learning method using LSTM (Long Short Time Memory) of the learning time series can be used for learning.

[0283] The transmission delay may include at least one of a transmission delay amount, an average value of the transmission delay in a predetermined time interval, a variance of the transmission delay, a maximum value, and a minimum value of the transmission delay.

[0284] The surrounding information may include at least the time, the state of radio waves around the moving body, the presence or absence of structures around the moving body, the distance from the structures to the moving body, conductors and obstacles around the moving body, radio wave strength, weather, and communication volume.

[0285] The map data may include at least the shape of roads around the moving object, and the positions and shapes of surrounding structures.

[0286] In machine learning, learning can be achieved if there is a correlation between the input and the output. By inputting the transmission delay feature amount, the environmental feature amount, and the map data into the model unit 115, the transmission delay distribution information is output. For example, in "Speech Recognition System Made with Free Software (Japanese name: フリーソフトでつくる音声認識システム) (Second Edition)" written by Masahiro Araki and published by Morikita Publishing Co., Ltd., a learning method of HMM based on deep learning is disclosed.

[0287] <Hardware Structure>

[0288] In addition, each component of the remote control devices 1000 to 4000 described in the above embodiments 1 to 4 can be configured using a computer and implemented by a computer executing a program. That is, the remote control devices 1000 to 4000 are implemented, for example, by Fig.24 the processing circuit 60 shown. In the processing circuit 60, processors such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor) are applied, and the functions of each part are implemented by executing the program stored in the storage device.

[0289] In addition, in the processing circuit 60, dedicated hardware can also be applied. When the processing circuit 60 is dedicated hardware, the processing circuit 60 conforms to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a circuit formed by combining these.

[0290] The remote control devices 1000 to 4000 can implement the functions of each component either with separate processing circuits or by concentrating these functions and implementing them with one processing circuit.

[0291] In addition, in Fig.25 , the hardware structure in the case of configuring the processing circuit 60 using a processor is shown. In this case, the functions of each part of the remote control devices 1000 to 4000 are implemented through a combination with software, etc. (software, firmware, or software and firmware). The software, etc. is described as a program and stored in the memory 62. The processor 61 that functions as the processing circuit 60 realizes the functions of each part by reading out and executing the program stored in the memory 62 (storage device). That is, this program can be called a program that causes a computer to execute the operations and methods of the components of the remote control devices 1000 to 4000.

[0292] Here, the memory 62 can be set to, for example, RAM, ROM, flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory) and other non-volatile or volatile semiconductor memories, HDD (Hard Disk Drive), magnetic disk, floppy disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, etc., or all storage media used in the future.

[0293] The above describes a structure in which the functions of the components of the remote control devices 1000 to 4000 are implemented by either hardware or software. However, the present invention is not limited thereto, and some of the components of the remote control devices 1000 to 4000 can be implemented by dedicated hardware, and other components can be implemented by software. For example, the functions of some components can be implemented by the processing circuit 60 as dedicated hardware, and the functions of other components can be implemented by the processing circuit 60 as the processor 61 reading and executing the program stored in the memory 62.

[0294] As described above, the remote control devices 1000 to 4000 can realize the above-mentioned functions through hardware, software, etc., or a combination of these.

[0295] Although the present disclosure has been described in detail, the above description is in all aspects illustrative and the present disclosure is not limited thereto. It should be understood that numerous modifications not shown here can be conceived without departing from the scope of the present disclosure.

[0296] In addition, the present disclosure can freely combine the various embodiments, or appropriately modify or omit the various embodiments within the scope of the disclosure.

Claims

1. A remote control device for controlling at least one moving object via a transmission path including at least a network, wherein: The remote control device comprises: a transmission delay distribution estimation unit that estimates transmission delay distribution information including a probability distribution of transmission delays in the transmission path; and an action planning unit, which plans a target action of the at least one mobile body based on the transmission delay distribution information, The transmission delay distribution estimation unit estimates the probability distribution of the transmission delay using a transmission delay model based on the pattern of the transmission delay.

2. The remote control device according to claim 1, wherein: The transmission delay distribution estimation unit estimates the transmission delay distribution information based on transmission delay information acquired in advance or the transmission delay information acquired online.

3. The remote control device according to claim 1 or 2, wherein: The remote control device comprises: A control unit calculates a control amount for controlling the at least one mobile body so that the at least one mobile body executes the target behavior planned by the behavior planning unit.

4. The remote control device according to any one of claims 1 to 3, wherein: The action planning section plans the target action in a manner corresponding to the pattern of the transmission delay.

5. The remote control device according to claim 2, wherein: The transmission delay distribution estimation unit estimates the transmission delay distribution information based on the transmission delay information and an environmental feature quantity that characterizes the environment around the at least one moving object.

6. The remote control device according to claim 1, wherein: The transmission delay distribution estimation unit learns and constructs the transmission delay model using machine learning.

7. The remote control device according to any one of claims 1 to 6, wherein: The transmission delay distribution estimation unit uses a hierarchical or non-hierarchical hidden Markov model as the transmission delay model.

8. The remote control device according to any one of claims 1 to 7, wherein: The at least one moving object is a plurality of moving objects, The action planning unit plans the target action for each of the plurality of moving objects.

9. The remote control device according to claim 8, wherein: When the transmission delay distribution information of each of the plurality of moving objects is considered to be the same, the transmission delay distribution estimation unit groups the plurality of moving objects and estimates the common transmission delay distribution information using any of the transmission delay distribution information.

10. The remote control device according to any one of claims 1 to 9, wherein: In addition to using the transmission delay distribution information, the action planning unit also uses the state information of the at least one moving body and the surrounding information of the at least one moving body, and uses a state equation representing the relative relationship between the position and speed of the at least one moving body and the surrounding objects to determine the risk of the at least one moving body, and changes the target action based on the determined risk.

11. A remote control system comprising: The remote control device according to claim 1 or 2; the network; and The at least one moving object, The at least one mobile body includes a control unit that executes the target action planned by the action planning unit of the remote control device. The remote control device transmits the target action planned by the action planning unit to the at least one moving object via the network, and causes the at least one moving object to execute the target action.

12. A remote control system comprising: The remote control device according to claim 3; the network; and The at least one moving object, The remote control device transmits the control amount calculated by the control unit to the at least one mobile object via the network, so as to cause the at least one mobile object to execute the target action planned by the action planning unit.

13. A risk determination method for determining the risk of at least one moving object when the at least one moving object is remotely controlled, wherein: Using transmission delay distribution information including at least a probability distribution of transmission delays in a transmission path of a network, state information of the at least one moving body, and surrounding information of the at least one moving body, the risk of the at least one moving body is determined using a state equation representing the relative relationship between the position and speed of the at least one moving body and surrounding objects.

Images