Controller

Abstract

A controller for controlling an operation of a movable object traveling in automatic driving includes: a detection unit configured to detect an occurrence of an abnormal state, including an abnormal stop of the movable object, when the movable object travels in automatic driving; and a control unit. When (i) the occurrence of the abnormal state is detected and (ii) a predetermined first condition is satisfied, the control unit is configured to execute control to resume automatic driving of the movable object. When (i) the occurrence of the abnormal state is detected and (ii) the first condition is not satisfied, the control unit is configured not to execute the control of resuming the automatic driving.

Description

Technical Field

[0001] The present invention relates to a controller. Background Art

[0002] Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2017-538619 (JP 2017-538619A) describes an autonomous driving vehicle with remote control. Summary of the invention

[0003] When a movable object (e.g., a vehicle traveling in automatic driving) fails and the movable object stops, the automatic driving is stopped and an operator is allowed to remotely perform a driving operation of the movable object by operating an input device, thereby causing the movable object to travel. However, there are cases where it is desired to restore the automatic driving of the movable object. A technology for determining whether to select one of a remote driving operation performed by an operator and restoration of the automatic driving is desired.

[0004] The present invention can be implemented as the following aspects.

[0005] One aspect of the present invention provides a controller that controls the operation of a movable object capable of traveling in automatic driving. The controller includes: a detection unit configured to detect the occurrence of an abnormal state when the movable object is traveling in automatic driving, the abnormal state including an abnormal stop of the movable object; and a control unit. When (i) the occurrence of the abnormal state is detected and (ii) a predetermined first condition is met, the control unit is configured to execute control to restore the automatic driving of the movable object. When (i) the occurrence of the abnormal state is detected and (ii) the first condition is not met, the control unit is configured not to execute control to restore the automatic driving. Using a controller according to this aspect, it is easy to determine whether to allow the selection of the resumption of automatic driving.

[0006] In the controller according to the above aspect of the present invention, the movable object may have a manned driving mode, a remote manual driving mode, and a remote automatic driving mode. The manned driving mode may be a driving mode in which the movable object travels when a driver in the movable object operates a driver input device set in the movable object. The remote manual driving mode may be a driving mode in which the movable object travels by receiving a travel condition without a driver in the movable object. The travel condition may be a condition generated when an operator operates an operator input device set at a location different from the location where the movable object is located, and the travel condition may define the travel behavior of the movable object. The remote automatic driving mode may be a driving mode in which the movable object travels by receiving a travel condition generated by the controller without operating any one of the driver input device and the operator input device. When (i) the occurrence of an abnormal state is detected and (ii) a first condition is satisfied, the control unit may be configured to perform control to restore automatic driving while the driving mode of the movable object is maintained in the remote automatic driving mode. When (i) the occurrence of an abnormal state is detected and (ii) the first condition is not satisfied, the control unit may be configured not to perform control to restore automatic driving in the remote automatic driving mode. With the controller according to this aspect, it is easy to determine whether to allow resumption of the automatic driving in the remote automatic driving mode to be selected.

[0007] In the controller according to the above aspect of the present invention, when (i) the occurrence of an abnormal state is detected, (ii) the first condition is not satisfied, and (iii) a predetermined second condition is satisfied, the control unit can be configured to switch the driving mode of the movable object from the remote automatic driving mode to the remote manual driving mode. With the controller according to this aspect, it is easy to determine whether to allow switching from the remote automatic driving mode to the remote manual driving mode.

[0008] In the controller according to the above aspect of the present invention, the first condition may include a condition that a time period required to resume the automatic driving is shorter than or equal to a predetermined time limit. With the controller according to this aspect, it is possible to easily determine whether to allow the resumption of the automatic driving in the remote automatic driving mode to be selected based on the time period required to resume the automatic driving.

[0009] In the controller according to the above aspect of the present invention, the first condition may include the following condition: the total number of the movable object and other movable objects in which the abnormal state occurs is less than or equal to a value M (M is an integer greater than or equal to 2). Using the controller according to this aspect, it is easy to determine whether to allow the resumption of automatic driving to be selected based on the number of movable objects in which the abnormal state occurs.

[0010] In the controller according to the above aspect of the present invention, the first condition may include the following condition: the total number of the movable object and other movable objects traveling in automatic driving is less than or equal to a value N (N is an integer greater than or equal to 2). With the controller according to this aspect, it is possible to easily determine whether to allow the resumption of automatic driving to be selected based on the number of movable objects traveling in automatic driving.

[0011] In the controller according to the above aspect of the present invention, the first condition may include the following condition: the current position of the movable object is in a predetermined area. With the controller according to this aspect, it is possible to easily determine whether to allow the resumption of the automatic driving to be selected according to the current position of the movable object.

[0012] In the controller according to the above aspect of the present invention, the predetermined area may be any one of a first area, a second area, and a third area. The first area may be an area where the frequency of other movable objects traveling in automatic driving stops is low. The second area may be an area where the distance from the predetermined route to the movable object is shorter than or equal to the predetermined distance when the movable object deviates from the predetermined route. The predetermined route may be set for the movable object. The third area may be an area where the operator can identify the movable object. The operator can take a position and posture so that the operator can operate the operator input device in the third area. With the controller according to this aspect, it is easy to determine whether to allow the resumption of automatic driving, taking into account the impact on the travel of other movable objects and the ease of resumption of automatic driving.

[0013] In the controller according to the above aspect of the present invention, the first condition may include a condition that a yaw rate sensor mounted on the movable object is not damaged. With the controller according to this aspect, when the yaw rate sensor required for automatic driving has been damaged, restoration of automatic driving is not selected. Whether to allow restoration of automatic driving to be selected may be determined according to the state of the yaw rate sensor.

[0014] In the controller according to the above aspect of the present invention, the first condition may include the following condition: the movable object does not need to deviate from a predetermined route set for the movable object. With the controller according to this aspect, when the movable object needs to deviate from the predetermined route, in order to ensure the normal travel of the movable object after the resumption of the autonomous driving, the resumption of the autonomous driving is not selected. Whether the resumption of the autonomous driving is allowed to be selected can be easily determined according to the situation of the movable object.

[0015] In the controller according to the above aspect of the present invention, the first condition may include the following condition: the weather around the movable object is not bad weather. With the controller according to this aspect, when the travel of the movable object after the resumption of autonomous driving may be difficult, the resumption of autonomous driving is not selected. Whether the resumption of autonomous driving is allowed to be selected can be easily determined according to the weather.

[0016] In the controller according to the above aspect of the present invention, the first condition may include a condition that a line for communication between the movable object and the controller is established. With the controller according to this aspect, since it is difficult to communicate with the movable object, the resumption of the automatic driving is not selected. Whether the resumption of the automatic driving is allowed to be selected can be easily determined according to the situation of the movable object.

[0017] In the controller according to the above aspect of the present invention, the second condition may include a condition that a time period required to resume the automatic driving exceeds a time limit. With the controller according to this aspect, when a time period is required to resume the automatic driving, the resumption of the automatic driving is not selected. Whether to allow switching from the automatic driving to the remote manual control can be easily determined according to the state of the movable object.

[0018] In the controller according to the above aspect of the present invention, the second condition may include a condition that the total number of the movable object and other movable objects in which the abnormal state occurs exceeds a value M. With the controller according to this aspect, when the number of movable objects in which the abnormal state occurs exceeds a specific number, it is difficult to resume the autonomous driving in the remote autonomous driving mode for all the expected movable objects, and thus the resumption of the autonomous driving is not selected. Whether to allow switching from the autonomous driving to the remote manual control can be easily determined according to the number of movable objects in which the abnormal state occurs.

[0019] In the controller according to the above aspect of the present invention, the second condition may include the following condition: the total number of the movable object and other movable objects traveling in automatic driving exceeds the value N. With the controller according to this aspect, it is possible to easily determine whether to allow switching from automatic driving to remote manual control according to the number of movable objects traveling in the remote automatic driving mode.

[0020] In the controller according to the above aspect of the present invention, the second condition may include the following condition: the current position of the movable object is not in the predetermined area. Using the controller according to this aspect, it is easy to determine whether to allow switching from automatic driving to remote manual control based on the current position of the movable object.

[0021] In the controller according to the above aspect of the present invention, the second condition may include the following condition: the yaw rate sensor installed on the movable object has been destroyed. With the controller according to this aspect, it is possible to easily determine whether to allow switching from automatic driving to remote manual control according to the state of the yaw rate sensor.

[0022] In the controller according to the above aspect of the present invention, the second condition may include the following condition: the movable object needs to deviate from the predetermined route set for the movable object. With the controller according to this aspect, when the movable object needs to travel along the predetermined route, after resuming the automatic driving, the driving mode is switched to the remote manual driving mode to ensure the normal travel of the movable object. Whether to allow switching from automatic driving to remote manual control can be easily determined according to the situation of the movable object.

[0023] In the controller according to the above aspect of the present invention, the second condition may include the following condition: the weather around the movable object is bad weather. With the controller according to this aspect, in the case of bad weather, after resuming the automatic driving, the movable object may be difficult to travel, so the driving mode is switched to the remote manual driving mode. Whether to allow switching from the automatic driving to the remote manual driving control can be easily determined according to the weather.

[0024] In the controller according to the above aspect of the present invention, the second condition may include a condition that the duration of a state in which communication between the movable object and the controller is stopped is longer than or equal to a predetermined time period. With the controller according to this aspect, since it is difficult to perform autonomous driving in the remote autonomous driving mode, the driving mode is switched to the remote manual driving mode. Whether to allow switching from the remote autonomous driving mode to the remote manual driving mode can be easily determined according to the situation of the movable object.

[0025] In the controller according to the above aspect of the invention, the second condition includes the condition that the operator takes a position and a posture in which the operator can operate the operator input device. With the controller according to this aspect, when the operator can perform remote operation of the movable object, the driving mode is switched to the remote manual driving mode. Whether to allow switching from automatic driving to remote manual control can be easily determined according to the situation of the operator.

[0026] The controller according to the above aspects of the present invention may further include a determination unit. The determination unit may be configured to determine whether a first condition is satisfied. The determination unit may be configured to further determine whether a second condition is satisfied when the first condition is not satisfied. In the controller according to the above aspects, initially, it is determined whether resumption of autonomous driving is allowed, and then it is determined whether the driving mode is switched to a remote manual driving mode. The number of operators who remotely operate movable objects is limited; however, when priority is given to determining whether to resume autonomous driving, human resources are efficiently managed.

[0027] The controller according to the above aspect of the present invention may further include a notification unit configured to provide a notification of information that the first state is established when the determination unit determines that the first state that does not satisfy the first condition and the second condition is established. In the controller according to the above aspect, when the driving in the automatic driving or remote manual driving mode cannot be resumed, a notification of this situation is provided, so that processing can be quickly continued in another method.

[0028] The controller according to the above aspect of the present invention may further include a notification unit configured to provide a notification of information that the second state has been established when the second state is established. The second state may be a state in which the duration of the state in which communication between the movable object and the controller is stopped is longer than or equal to a predetermined time period. In the controller according to the above aspect, when it is difficult to resume the autonomous driving in the remote autonomous driving mode or the driving in the remote manual driving mode, a notification of this situation is provided so that processing can be quickly continued in another method.

[0029] The present invention can be implemented in various modes, and can be implemented in modes such as a remote operation system, a remote controller, a movable object controller, a remote automatic driving method, and a manufacturing method of a movable object. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described below with reference to the accompanying drawings, in which like numerals represent like elements, and in which: Figure 1 is a diagram showing a configuration of a remote operation system according to a first embodiment; Figure 2 is a schematic block diagram showing an overview of a movable object; Figure 3 is a schematic block diagram showing an overview of a remote operation device; Figure 4A is a flow chart showing a vehicle travel method implemented by a remote controller; Figure 4B is a flowchart showing a determination process regarding switching of a driving mode; Figure 5 is a diagram showing a configuration of a remote operation system according to a second embodiment; Figure 6 is a flowchart showing a determination process regarding switching of a driving mode according to a second embodiment; Figure 7 is a block diagram showing a functional configuration of a movable object capable of traveling in an autonomous driving mode; and Figure 8 is a flowchart showing a method of traveling of a movable object in which an autonomous driving mode is set. DETAILED DESCRIPTION A. First Embodiment A1. Remote operating system configuration

[0031] Figure 1 is a diagram showing a configuration of a remote operation system 1 according to the first embodiment. Figure 2 2 is a schematic block diagram showing an outline of the movable object 20 . Figure 1 The remote operation system 1 shown is used in a factory for producing a movable object 20. The target controlled by the remote operation system 1 is a movable object 20 that can move by remote operation. The remote operation system 1 is used in the manufacturing process of the movable object, including the assembly steps, inspection steps, etc. of the movable object 20. The remote operation system 1 transports the movable object 20 in a predetermined transport section during the manufacturing process. The movable object 20 in transport is processed by an operator or equipment, such as assembly and inspection.

[0032] In the present embodiment, the movable object 20 is a battery electric vehicle (BEV) capable of traveling in automatic driving. The term "unmanned driving" means driving that is not performed by the driving operation of the driver in the movable object 20. Driving operation means an operation related to at least any one of "traveling", "turning" and "stopping" of the vehicle. Automatic driving is achieved by automatic or manual remote control using a device set outside the movable object 20 or autonomous control of the movable object 20. A person who does not perform a driving operation may be in the movable object 20 that travels in automatic driving. Examples of people who do not perform a driving operation include a person who simply sits on a seat of the movable object 20, and a person who performs work different from the driving operation (i.e., assembling, inspecting, or operating a switch) while in the movable object 20. Driving performed by the driving operation of the driver in the movable object 20 may be referred to as "manned driving".

[0033] The remote operation system 1 includes at least one movable object 20 , a plurality of detection devices 50 , and a remote operation apparatus 100 .

[0034] The movable object 20 travels in any one of a manned driving mode, a remote manual driving mode, and a remote automatic driving mode.

[0035] The manned driving mode is a driving mode in which the movable object 20 travels according to the driving operation of the driver in the movable object 20. When the manned driving mode is set, the driver in the movable object 20 operates the driver input device 210 (see Figure 2 ) to determine the traveling condition of the movable object 20. The driver input device 210 includes a steering wheel, an accelerator pedal, a brake pedal, etc. (not shown). The movable object 20 adopts a wire-controlled steering system, a wire-controlled throttle system, and a wire-controlled brake system. The movable object 20 travels according to the determined traveling condition. The traveling condition defines the traveling behavior of the movable object 20. The traveling condition includes, for example, a traveling speed, an acceleration, and a steering angle of a wheel.

[0036] The remote manual driving mode is a driving mode in which the travel of the movable object 20 is remotely controlled by the driving operation of an operator who is not in the movable object 20. When the remote manual driving mode is set, a travel condition is generated by a remote operating device 100 (described later) according to the operation of an operator input device (described later) by the operator. The remote operating device 100 is set at a location different from the location where the movable object 20 is located. The travel of the movable object 20 is controlled according to the travel condition generated by the remote operating device 100. In the remote manual driving mode, the driver in the movable object 20 does not need to operate the driving input device set in the movable object 20 to make the movable object 20 travel.

[0037] The remote automatic driving mode is a driving mode that controls the travel of the movable object 20 according to the travel conditions generated by the remote operation device 100. When the remote automatic driving mode is set, the travel conditions are generated by the remote operation device 100 without the operation of the operator input device by the operator. In the remote automatic driving mode, neither the operation of the driving input device by the driver nor the operation of the operator input device by the operator who is not in the movable object 20 is required. The driving of the movable object 20 in the remote automatic driving mode is automatic driving. The term "automatic driving" means that the system, rather than the driver, performs all recognition, prediction, judgment, and operation related to the driving operation, and autonomously causes the movable object 20 to travel.

[0038] like Figure 2As shown, the movable object 20 includes a driver input device 210, a driving device 220 for driving the movable object 20, a steering device 230 for changing the traveling direction of the movable object 20, a braking device 240 for braking the movable object 20, a communication device 260, an external sensor 270, an internal sensor 280, and a movable object controller 290 for controlling the devices of the movable object 20. In the present embodiment, the driving device 220 includes a battery, a driving motor driven by the power of the battery, and a driving wheel rotated by the driving motor.

[0039] The communication device 260 is used to communicate with the remote operation device 100 (see Figure 1 The communication device 260 communicates with a device outside the movable object 20 through, for example, a controller area network (CAN) communication. CAN communication is a communication standard with which transmission or reception can be performed in many directions. The communication device 260 of the movable object 20 communicates with, for example, a remote operation device 100 connected to the network NT via an access point (not shown) in a factory.

[0040] Figure 2 The external sensor 270 shown includes a plurality of sensors for detecting the situation around the movable object 20. The external sensor 270 includes a camera, a millimeter wave radar, a laser radar (Light Detection and Ranging, LiDAR), an ultrasonic sensor, etc., which are mounted on the movable object 20. The on-board camera captures an image of at least a portion of the area around the movable object 20. The radar detects the distance, relative angle, relative speed, etc. relative to an object existing in the area around the movable object 20. The laser radar detects the distance and relative angle, the shape of the object, etc. relative to the object in the area around the movable object 20. Each sensor outputs the detection value to the movable object controller 290.

[0041] The internal sensor 280 includes a plurality of sensors for detecting the internal state of the movable object 20. For example, the internal sensor 280 includes a current sensor, a voltage sensor, a battery temperature sensor, a vehicle speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, a Global Navigation Satellite System (GNSS) receiver, etc., which are installed on the movable object 20. The current sensor detects the output current value of the battery of the movable object 20. The voltage sensor detects the output voltage value of the battery of the movable object 20. The battery temperature sensor detects the temperature of the battery of the movable object 20. The vehicle speed sensor detects the speed of the movable object 20. The acceleration sensor detects the acceleration of the movable object 20. The yaw rate sensor detects the yaw axis angular velocity (yaw rate) of the movable object 20. The steering angle sensor detects the steering angle of each wheel of the movable object 20. The GNSS receiver receives GNSS information. Each sensor outputs the detection value to the movable object controller 290.

[0042] The movable object controller 290 is a computer including a central processing unit (CPU) 291, a memory 292, an input / output interface 293, and an internal bus 294. The movable object controller 290 is composed of, for example, one or two or more electronic control units (ECUs).

[0043] The CPU 291, the memory 292, and the input / output interface 293 are connected via an internal bus 294, so that two-way communication is possible. The driver input device 210, the drive device 220, the steering device 230, the brake device 240, the communication device 260, the external sensor 270, and the internal sensor 280 are connected to the input / output interface 293. Various programs and data are stored in the memory 292. The CPU 291 controls the devices of the movable object 20 by running the programs stored in the memory 292.

[0044] In the present embodiment, when the manned driving mode is set, the movable object controller 290 can cause the movable object 20 to travel by controlling the device of the movable object 20 in response to the driving operation of the driver in the movable object 20 using the driver input device 210. When the remote manual driving mode or the remote automatic driving mode is set, the movable object controller 290 causes the movable object 20 to travel by controlling the device of the movable object 20 in response to the control command received from the remote operation device 100.

[0045] The movable object controller 290 transmits information detected by the external sensor 270 as movable object sensor information to the remote operation apparatus 100 via the communication device 260. The movable object controller 290 transmits information detected by the internal sensor 280 to the remote operation apparatus 100 via the communication device 260 as movable object state information.

[0046] Figure 1 The detection device 50 shown is used to monitor the movable object 20 moving in the area of ​​the factory. The detection device 50 is, for example, a camera installed in the factory area. The position and orientation of the detection device 50 are adjusted in advance. For example, each detection device 50 communicates with a remote operation device 100 connected to a network via an access point (not shown) in the factory. The detection device 50 sends external image capture information obtained by capturing an image of the image capture range to the remote operation device 100. The external image capture information is, for example, video data obtained by the detection device 50.

[0047] Figure 3 2 is a schematic block diagram showing an outline of the remote operation device 100. The remote operation device 100 is a device for remotely operating the travel of the movable object 20.

[0048] The remote operation apparatus 100 includes a communication device 105, an operator input device 110, a display device 115, a speaker 117, and a remote controller 120. The remote controller 120 is also referred to as a controller.

[0049] The communication device 105 connects the remote controller 120 with the movable object 20 and the detection device 50, making communication possible. The communication device 105 is, for example, a wireless communication device.

[0050] The operator input device 110 remotely operates the movable object 20 for which the remote manual driving mode is set, and thus the operator input device 110 is a device operated by the operator. The operator input device 110 is provided at a location different from the location where the movable object 20 is located, and is connected to the remote controller 120. The operator input device 110 includes a steering wheel 111 as an input device for steering the movable object 20 by remote operation, an accelerator pedal 112 as an input device for accelerating the movable object 20 by remote operation, and a brake pedal 113 as an input device for decelerating the movable object 20 by remote operation.

[0051] The display device 115 displays an image representing various information according to the control of the remote controller 120. The display device 115 is installed at a visually recognizable position where the operator takes a position and a posture where the operator can operate the operator input device 110. For example, the display device 115 displays internal image capture information acquired by the vehicle-mounted camera and external image acquisition information acquired by the detection device 50.

[0052] The speaker 117 reproduces voice data according to the control of the remote controller 120. For example, the speaker 117 reproduces voice data acquired by a microphone (not shown) attached to a vehicle-mounted camera installed on the movable object 20. The voice data is included in the internal image capture information acquired by the vehicle-mounted camera. The speaker 117 reproduces voice data acquired by a microphone (not shown) attached to a camera as the detection device 50. The voice data is included in the external image capture information acquired by the detection device 50.

[0053] The remote controller 120 is a device for remotely controlling the travel of the movable object 20 .

[0054] When the remote manual driving mode is set in the movable object 20, the operator operates the operator input device 110 to make the movable object 20 travel. The remote controller 120 generates a control command according to the driving operation of the operator and sends the control command to the movable object 20. The movable object 20 travels according to the control command received from the remote controller 120.

[0055] When the remote automatic driving mode is set, the remote controller 120 generates a control instruction by using detection values ​​from various sensors, etc. The movable object 20 travels according to the control instruction received from the remote controller 120 .

[0056] The remote controller 120 is a computer including a CPU 11 , a memory 12 , an input / output interface 13 , and an internal bus 14 .

[0057] The CPU 11, the memory 12, and the input / output interface 13 are connected via an internal bus 14, so that two-way communication is possible. A communication device 105, an operator input device 110, a display device 115, and a speaker 117 are connected to the input / output interface 13. Various programs and data are stored in the memory 12. In the present embodiment, information about a predetermined route of the movable object 20 is stored in the memory 12. The predetermined route is a preset route along which the movable object 20 is to travel. The CPU 11 implements various functions, namely, an acquisition unit 121, a calculation unit 122, an abnormality detection unit 123, a determination unit 124, and a control value creation unit 125, by running the programs stored in the memory 12.

[0058] The acquisition unit 121 acquires movable object sensor information and movable object state information from the movable object 20. The acquisition unit 121 acquires external image capturing information from the detection device 50.

[0059] The calculation unit 122 calculates the position and orientation of the movable object 20. For example, the position of the movable object 20 and the orientation of the movable object 20 can be estimated by using the external image capture information. For example, the position of the movable object 20 can be acquired by calculating the coordinates of the measurement point of the movable object 20 in the image coordinate system by using the outer shape of the movable object 20 detected from the external image capture information, and converting the calculated coordinates into coordinates in the global coordinate system.

[0060] The orientation of the movable object 20 can be estimated based on, for example, the orientation of the motion vector of the movable object 20 calculated by using the optical flow technique. In the optical flow technique, the orientation of the motion vector of the movable object 20 is calculated according to the change in the position of the feature point of the movable object between frames of the captured image. Alternatively, the orientation of the movable object 20 can be calculated by using the detection value of the yaw rate sensor installed on the movable object 20.

[0061] The outer shape of the movable object 20 can be detected by, for example, inputting the captured image into the detection model using artificial intelligence. Examples of the detection model include a trained machine learning model trained to implement any one of semantic segmentation and instance segmentation. For example, a convolutional neural network (Convolutional Neural Network, hereinafter referred to as CNN) trained by supervised learning using a training data set can be used as a machine learning model. The training data set has: a plurality of training images, each of which includes a movable object; and a correct label, each indicating whether the area in the training image represents an area of ​​a movable object or an area of ​​an object other than the movable object. During the training of the CNN, the parameters of the CNN are preferably updated so that the error between the output result from the detection model and the correct label is reduced by back propagation.

[0062] The abnormality detection unit 123 detects the occurrence of an abnormal state in the movable object 20. The abnormality detection unit 123 is also referred to as a detection unit. In the present embodiment, when the movable object 20 to which the remote autonomous driving mode is set is traveling, the abnormality detection unit 123 detects the occurrence of an abnormal state in the movable object 20 by using movable object sensor information and movable object state information of the movable object 20, external image capture information, information indicating the communication state between the remote operation device 100 and the movable object 20, and the like. The abnormal state in the movable object 20 means that the movable object 20 to which the remote autonomous driving mode is set stops abnormally. The abnormal stop means an unexpected stop of the movable object 20 remotely operated in the remote autonomous driving mode.

[0063] When the movable object 20 to which the remote automatic driving mode is set stops abnormally, the determination unit 124 determines whether to resume the automatic driving in the remote automatic driving mode or to switch from the remote automatic driving mode to the remote manual driving mode.

[0064] When the movable object 20 in which the remote automatic driving mode is set stops abnormally, any one of the following processes is performed. The processes to be performed include (i) switching to the remote manual driving mode and remotely operating the driving of the movable object 20 by the driving operation of the operator, (ii) resuming the automatic driving in the remote automatic driving mode, and (iii) an operator entering the movable object 20 and the operator driving the movable object 20.

[0065] In the present embodiment, when the first condition is satisfied, the determination unit 124 determines to resume the autonomous driving of the movable object 20 in the remote autonomous driving mode. The first condition is a condition determined regarding the resumption of the autonomous driving in the remote autonomous driving mode. When the determination unit 124 determines to resume the autonomous driving of the movable object 20 in the remote autonomous driving mode, the switching of the driving mode is not performed. On the other hand, when the first condition is not satisfied, the determination unit 124 determines to switch from the remote autonomous driving mode to the remote manual driving mode. The details of the first condition related to the resumption of the autonomous driving in the remote autonomous driving mode will be described later.

[0066] The determination unit 124 transmits a notification about the switching of the driving mode to the control value creation unit 125 according to the determination result about whether to resume the automatic driving in the remote automatic driving mode.

[0067] The control value creation unit 125 generates a control instruction for remotely controlling the movable object 20 to which the remote automatic driving mode is set. The control value creation unit 125 is also referred to as a control unit. The control value creation unit 125 generates a control instruction regarding the movable object 20 by using the position and orientation of the movable object 20 calculated by the calculation unit 122, and sends the control instruction to the movable object 20. For example, the control instruction is generated as a control value indicating acceleration and a steering angle. The control value creation unit 125 sends the control instruction according to the driving operation of the operator to the movable object 20 so as to control the movable object 20 to which the remote manual driving mode is set.

[0068] Figure 4A 1 is a flowchart showing a method in which the remote controller 120 causes the movable object 20 to which the remote automatic driving mode is set to travel. The remote controller 120 acquires the position of the movable object 20 by using the detection result of the detection device 50 (step S11). Specifically, the calculation unit 122 calculates the position of the movable object 20 by using the external image capture information acquired by the acquisition unit 121 from the detection device 50. The control value creation unit 125 acquires information about the position of the movable object 20 calculated by the calculation unit 122. The information about the position of the movable object 20 includes X, Y, and Z coordinates in the global coordinate system of the plant.

[0069] The control value creation unit 125 determines the target position to which the movable object 20 is to go next (step S12). In the present embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system of the plant. Information about a predetermined route is pre-stored in the memory 12 of the remote controller 120, and the predetermined route is the route along which the movable object 20 is to travel. The predetermined route is represented by a node indicating a departure point, a node indicating a passing point, a node indicating a destination, and road sections connecting these nodes respectively. The control value creation unit 125 determines the target position to which the movable object 20 is to go next by using the position of the movable object 20 and the predetermined route. The control value creation unit 125 determines the target position on the predetermined route in front of the current position of the movable object 20.

[0070] The control value creation unit 125 generates a control instruction for causing the movable object 20 to travel toward the determined target position (step S13). In the present embodiment, the control instruction includes the acceleration and the steering angle of the movable object 20 as parameters. The control value creation unit 125 calculates the travel speed of the movable object 20 according to the change in the position of the movable object 20, and compares the calculated travel speed with the predetermined target speed of the movable object 20. When the travel speed is lower than the target speed, the control value creation unit 125 determines the acceleration to accelerate the movable object 20. When the travel speed is higher than the target speed, the control value creation unit 125 determines the acceleration to decelerate the movable object 20. When the movable object 20 is located on the predetermined route, the control value creation unit 125 determines the steering angle so that the movable object 20 does not deviate from the predetermined route. When the movable object 20 is not located on the predetermined route, in other words, when the movable object 20 deviates from the predetermined route, the control value creation unit 125 determines the steering angle so that the movable object 20 returns to the predetermined route.

[0071] The control value creation unit 125 sends the generated control instruction to the movable object 20 (step S14). The control value creation unit 125 repeats acquiring the position of the movable object 20, determining the target position, generating the control instruction, and sending the control instruction at predetermined intervals.

[0072] The movable object controller 290 installed on the movable object 20 receives the control instruction from the remote controller 120 (step S15), and controls the driving device 220, the steering device 230, and the braking device 240 by using the received control instruction so that the movable object 20 moves at the acceleration and the steering angle indicated by the control instruction (step S16). The movable object controller 290 repeats the reception and control of the control instruction related to the movement of the movable object 20 at predetermined intervals. When the movable object 20 is moved by remote control, the movable object 20 can be moved without using a conveying device such as a crane and a conveyor.

[0073] In the present embodiment, when the control value creation unit 125 receives a notification from the determination unit 124 that the driving mode is not switched, the control value creation unit 125 performs control to restore the automatic driving of the movable object 20 in the remote automatic driving mode in the factory area while the driving mode is maintained in the remote automatic driving mode. On the other hand, when the control value creation unit 125 receives a notification from the determination unit 124 that the driving mode is switched to the remote manual driving mode, the control value creation unit 125 performs control on the movable object 20 to switch from the remote automatic driving mode to the remote manual driving mode. In this case, the control to restore the automatic driving of the movable object 20 in the remote automatic driving mode in the factory area is not performed. Therefore, the automatic driving of the movable object 20 in the remote manual driving mode is started. A2. First condition

[0074] At least one of the conditions listed below may be adopted as the first condition. Two or more of the conditions listed below may be selected as the first condition.

[0075] <Condition 1> The time period required to resume autonomous driving is shorter than or equal to a predetermined time limit. <Condition 2> The total number of the movable object 20 and other movable objects in which the abnormal state occurs is less than or equal to a predetermined value M (M is an integer greater than or equal to two). <Condition 3> The total number of the movable object 20 and other movable objects traveling in automatic driving is less than or equal to a predetermined value N (N is an integer greater than or equal to two). <Condition 4> The current position of the movable object 20 (ie, the position where the movable object 20 stops) is in an area where other movable objects traveling in automatic driving stop with low frequency. <Condition 5> When the movable object 20 deviates from the predetermined route, the distance from the predetermined route to the movable object 20 is shorter than or equal to the predetermined distance. <Condition 6> The current position of the movable object 20 (ie, the position where the movable object 20 stops) is in a region where the operator can recognize the movable object 20 in a position and posture where the operator can operate the operator input device. <Condition 7> The yaw rate sensor mounted on the movable object 20 is not damaged. <Condition 8> The movable object 20 does not need to deviate from the set predetermined route. <Condition 9> The weather around the movable object 20 is not severe weather. <Condition 10> A line through which the movable object 20 and the remote operation device 100 communicate is established. A3.Condition 1

[0076] When the time period required to resume the autonomous driving in the remote autonomous driving mode of the movable object 20 is shorter than or equal to the predetermined time limit, it is efficient to select the resumption of the autonomous driving in the remote autonomous driving mode (not the remote manual driving mode). In order to switch the driving mode to the remote manual driving mode and remotely operate the movable object 20 by the operator's operation, the operator needs to move to the location where the operator input device is installed and take a posture in which the operator can operate the operator input device. On the other hand, when the time period required to resume the autonomous driving in the remote autonomous driving mode of the movable object 20 is longer than the time limit, the autonomous driving in the remote manual driving mode or driving by the operator in the movable object 20 is efficient. Therefore, the remote controller 120 can easily determine whether to allow the resumption of the autonomous driving in the remote autonomous driving mode to be selected based on whether the condition 1 is satisfied.

[0077] When the condition 1 is used, the remote controller 120 predicts a time period required to resume the autonomous driving in the remote autonomous driving mode of the movable object 20 by using the movable object sensor information and the movable object state information acquired by the movable object 20 . A4. Condition 2

[0078] When the movable object 20 stops abnormally, there may be a situation where other movable objects traveling in the remote automatic driving mode in the factory area also stop abnormally. In this case, when the remote controller 120 sequentially performs the process of restoring the automatic driving of the movable object 20 in the remote automatic driving mode and the process of restoring the automatic driving of other movable objects in the remote automatic driving mode, any one of the movable objects stops at the current position for a period of time. Therefore, this may, for example, affect the travel of other movable objects that are not in an abnormal stop. By using condition 2, it is determined whether to select the resumption of automatic driving in the remote automatic driving mode in consideration of the impact on the travel of other movable objects. Therefore, the remote controller 120 can easily determine whether to allow the selection of the resumption of automatic driving in the remote automatic driving mode based on whether condition 2 is satisfied.

[0079] When condition 2 is used, the remote controller 120 predicts the total number of movable objects 20 and other movable objects in which abnormal conditions occur by using external image capture information, movable object sensor information and movable object status information acquired by the movable object 20, and movable object sensor information and movable object status information acquired by other movable objects. A5. Condition 3

[0080] Since it takes a certain period of time to resume the autonomous driving of the movable object 20 in the remote autonomous driving mode, the movable object 20 stops at the current position for a period of time. When other movable objects other than the movable object 20 are traveling in the remote autonomous driving mode in the factory area, the stop of the movable object 20 may affect the travel of the other movable objects. By using condition 3, it is determined whether to allow the resumption of the autonomous driving in the remote autonomous driving mode to be selected in consideration of the influence on the travel of other movable objects. Therefore, the remote controller 120 can easily determine whether to allow the resumption of the autonomous driving in the remote autonomous driving mode to be selected based on whether condition 3 is satisfied.

[0081] When condition 3 is used, the remote controller 120 predicts the total number of movable objects 20 and other movable objects traveling in the remote autonomous driving mode by using external image capture information, movable object sensor information and movable object status information acquired by the movable object 20, and movable object sensor information and movable object status information acquired by other movable objects. A6. Condition 4

[0082] When the position where the movable object 20 stops is in an area where other movable objects traveling in the remote automatic driving mode stop at a low frequency, even when it takes a certain period of time to resume the automatic driving of the movable object 20 in the remote automatic driving mode, the influence on the other movable objects is small. On the other hand, when the position where the movable object 20 stops is in an area where other movable objects traveling in the remote automatic driving mode stop at a high frequency, it is expected to move the movable object 20 quickly from the current position. The area where other movable objects traveling in the remote automatic driving mode stop at a high frequency (hereinafter, referred to as the avoidance area) is a specific area including, for example, an intersection. This is because the movable object 20 and other movable objects temporarily stop before passing through the intersection. By using condition 4, it is determined whether to allow the resumption of the automatic driving in the remote automatic driving mode to be selected in consideration of the influence on the travel of other movable objects. Therefore, the remote controller 120 can easily determine whether to allow the resumption of the automatic driving in the remote automatic driving mode to be selected based on whether condition 4 is satisfied.

[0083] When condition 4 is used, the remote controller 120 determines whether the current position of the movable object 20 is in an area where other movable objects stop at a low frequency by using movable object sensor information and movable object state information acquired by the movable object 20, external image capture information, and information about a preset avoidance area. The information about the avoidance area is pre-stored in the memory 12. A7. Condition 5

[0084] Although the movable object 20 has been traveling along the predetermined route set in advance, the movable object 20 may leave the predetermined route before the abnormal stop due to some reasons. When the distance from the current position of the movable object 20 to the predetermined route is longer than the predetermined distance, it is difficult for the movable object 20 to return to the predetermined route again even when the automatic driving in the driving mode is restored. On the other hand, when the distance from the current position of the movable object 20 to the predetermined route is shorter than or equal to the predetermined distance, it is relatively easy for the movable object 20 to return to the predetermined route again by restoring the automatic driving in the driving mode. By using condition 5, whether to select the restoration of the automatic driving in the remote automatic driving mode is determined in consideration of the ease of restoration of the automatic driving. Therefore, the remote controller 120 can easily determine whether to allow the selection of the restoration of the automatic driving in the remote automatic driving mode based on whether condition 5 is satisfied.

[0085] When Condition 5 is used, the remote controller 120 determines whether the current position of the movable object 20 deviates from the predetermined route by using information about the predetermined route set for the movable object 20, external image capture information, movable object sensor information, and movable object state information. When the current position of the movable object 20 deviates from the predetermined route, the remote controller 120 further determines whether the distance from the predetermined route to the movable object 20 is shorter than or equal to a predetermined distance. A8. Condition 6

[0086] When the current position of the movable object 20 is in an area where the operator can identify the movable object 20, when the movable object 20 resumes the autonomous driving in the remote autonomous driving mode, the operator can visually recognize the traveling state of the movable object 20. In addition, after the autonomous driving in the remote autonomous driving mode is resumed, even when the driving mode is switched to the remote manual driving mode for some reason, the operator can easily handle the situation. The situation where the movable object 20 is in an area where the operator can recognize the movable object 20 by taking a position and posture in which the operator can operate the operator input device means that the movable object 20 is within the image capture range of the detection device 50. When the external image capture information captured by the detection device 50 is displayed on the display device 115, the operator can visually recognize the movable object 20.

[0087] Whether to select resumption of autonomous driving in the remote autonomous driving mode is determined in consideration of the ease of resumption of autonomous driving by using condition 6. Therefore, the remote controller 120 can easily determine whether to allow the resumption of autonomous driving in the remote autonomous driving mode to be selected based on whether condition 6 is satisfied.

[0088] When condition 6 is used, the remote controller 120 determines whether the current position of the movable object 20 is in an area where the operator can recognize the movable object 20 by using the movable object sensor information and the movable object state information acquired by the movable object 20 and the information about the installation position of the detection device 50. The information about the installation position of the detection device 50 is pre-stored in the memory 12. A9. Condition 7

[0089] When the yaw rate sensor mounted on the movable object 20 has been destroyed, the orientation of the movable object 20 cannot be estimated, and thus it is impossible to perform automatic driving of the movable object 20. Therefore, the remote controller 120 can easily determine whether to allow the resumption of automatic driving in the remote automatic driving mode to be selected based on whether the condition 7 is satisfied.

[0090] When Condition 7 is used, the remote controller 120 determines whether the yaw rate sensor mounted on the movable object 20 has been broken based on the movable object state information acquired by the movable object 20. For example, when a detection value is not received from the yaw rate sensor within a certain period of time, the remote controller 120 determines that the yaw rate sensor has been broken. A10. Condition 8

[0091] For example, in order for other movable objects to stop on the predetermined route set for the movable object 20, the movable object 20 needs to avoid collision with other movable objects. Moreover, because the operator performs some work on the predetermined route, the movable object 20 needs to avoid collision with the operator. This situation corresponds to a situation where the movable object 20 needs to deviate from the set predetermined route. When the movable object 20 needs to deviate from the set predetermined route, even when the movable object 20 resumes the autonomous driving in the remote autonomous driving mode, it can be expected that the smooth travel of the movable object 20 becomes difficult.

[0092] Whether to select resumption of the autonomous driving in the remote autonomous driving mode is determined in consideration of the surrounding situation by using Condition 8. Therefore, the remote controller 120 can easily determine whether to allow the resumption of the autonomous driving in the remote autonomous driving mode to be selected based on whether Condition 8 is satisfied.

[0093] When Condition 8 is used, the remote controller 120 determines whether the movable object 20 needs to travel deviating from the predetermined route by using information on the predetermined route set for the movable object 20, external image capturing information, and movable object sensor information. A11. Condition 9

[0094] The movable object 20 may travel outside the building in the factory area. In this case, it may be difficult for the movable object 20 to travel after resuming the autonomous driving in the remote autonomous driving mode, depending on the weather around the movable object 20. By using condition 9, whether to select the resumption of the autonomous driving in the remote autonomous driving mode is determined in consideration of the difficulty of traveling in the resumed autonomous driving. Therefore, the remote controller 120 can easily determine whether to allow the resumption of the autonomous driving in the remote autonomous driving mode to be selected based on whether condition 9 is satisfied.

[0095] When condition 9 is used, the remote controller 120 obtains information about the weather at the current location of the movable object 20 from an external server that distributes information about the weather at many locations. The remote controller 120 determines whether it is bad weather based on the information about the weather. Bad weather is weather that makes it difficult to automatically drive the movable object 20 in the remote automatic driving mode. For example, bad weather includes rainy or snowy weather and foggy weather. A12. Condition 10

[0096] A communication failure may occur on a line through which the movable object 20 communicates with the remote controller 120. In this case, the movable object 20 cannot communicate with the remote controller 120, and thus the autonomous driving of the movable object 20 in the remote autonomous driving mode is impossible, with the result that the movable object 20 may stop abnormally. Therefore, the remote controller 120 can easily determine whether to allow the resumption of the autonomous driving in the remote autonomous driving mode to be selected based on whether the condition 10 is satisfied. A13. About the determination process of driving mode switching

[0097] Figure 4B 1 is a flowchart showing a determination process regarding the switching of the driving mode. When the abnormality detection unit 123 detects the occurrence of an abnormal state of the movable object 20 based on the movable object sensor information, the movable object state information, and the external image capture information regarding the movable object 20 acquired by the acquisition unit 121, the abnormality detection unit 123 starts Figure 4B Hereinafter, an example in which the above-mentioned condition 1 is adopted as the first condition will be described.

[0098] In step S101 , the determination unit 124 determines whether condition 1, which is the first condition, is satisfied.

[0099] Specifically, the determination unit 124 predicts a time period required to restore the autonomous driving of the movable object 20 in the remote autonomous driving mode by using the movable object sensor information and the movable object state information acquired by the movable object 20. When the predicted time period required to restore the autonomous driving is shorter than or equal to the predetermined time limit, the determination unit 124 determines that the first condition is satisfied. On the other hand, when the predicted time period required to restore the autonomous driving is longer than the predetermined time limit, the determination unit 124 determines that the first condition is not satisfied.

[0100] When the first condition is satisfied (Yes in step S101), the determination unit 124 performs the process of step S102. On the other hand, when the first condition is not satisfied (No in step S101), the determination unit 124 performs the process of step S104.

[0101] In step S102, the determination unit 124 determines to continue the remote automatic driving mode of the movable object 20. In step S103, the determination unit 124 outputs a notification of executing the resumption of the automatic driving in the remote automatic driving mode to the control value creation unit 125. When the control value creation unit 125 receives the notification of executing the resumption of the automatic driving in the remote automatic driving mode, the control value creation unit 125 sends a control instruction to resume the automatic driving to the movable object 20. Then, Figure 4B The process shown ends.

[0102] Thereafter, the control value creation unit 125 remotely controls the movable object 20 in the remote automatic driving mode by sending a control instruction to the movable object 20 as needed.

[0103] In step S104, the determination unit 124 determines to switch from the remote automatic driving mode to the remote manual driving mode. In step S105, the determination unit 124 outputs a notification that the driving mode is switched from the remote automatic driving mode to the remote manual driving mode to the control value creation unit 125. When the control value creation unit 125 receives the notification that the driving mode is switched from the remote automatic driving mode to the remote manual driving mode, the control value creation unit 125 outputs a notification that the driving mode is switched to the remote manual driving mode to the display device 115. Then, Figure 4B The process shown ends.

[0104] Thereafter, each time the control value creating unit 125 receives an input from the operator input device 110, the control value creating unit 125 sends a control instruction according to the operator's driving operation to the movable object 20. In this way, remote control of the movable object 20 in the remote manual driving mode is performed.

[0105] As described above, in the present embodiment, the remote controller 120 can easily determine whether to allow the resumption of the automatic driving in the remote automatic driving mode to be selected according to whether the predetermined first condition is satisfied. B. Second Embodiment

[0106] Figure 5 2 is a diagram showing a configuration of a remote operation system 1 according to a second embodiment. Hereinafter, components different from those of the first embodiment will be described in detail. Descriptions of components similar to those of the first embodiment are omitted.

[0107] In the second embodiment, in order to determine whether to switch the driving mode of the movable object 20 from the remote automatic driving mode to the remote manual driving mode, determination regarding the second condition is performed in addition to determination regarding the first condition.

[0108] When the occurrence of an abnormal state of the movable object 20 is detected, the determination unit 124 of the remote controller 120 determines whether a predetermined first condition and a predetermined second condition are satisfied. The second condition is a condition determined regarding switching to the remote manual driving mode. When the determination unit 124 determines that the first condition is not satisfied and the second condition is satisfied, the determination unit 124 switches the driving mode of the movable object 20 from the remote automatic driving mode to the remote manual driving mode. At least one of the conditions listed in the first embodiment may be adopted as the first condition.

[0109] When the determination unit 124 determines that the first condition is not satisfied and the second condition is not satisfied, the determination unit 124 determines to switch the driving mode of the movable object 20 from the remote automatic driving mode to the manned driving mode. In this case, the determination unit 124 provides an instruction notification to at least one operator WO in the factory area via the terminal device T1 held by the operator WO to recycle the movable object 20. The determination unit 124 is also referred to as a notification unit. The determination unit 124 is capable of providing an instruction notification to the operator WO at the waiting position OF via the terminal device T2 installed at the waiting position OF to recycle the movable object 20. The waiting position OF is a position where the operator WO waits or rests.

[0110] At least one of the conditions listed below may be adopted as the second condition. Two or more of the conditions listed below may be selected as the second condition.

[0111] <Condition 11> The operator takes a position and a posture in which the operator can operate the operator input device 110 . <Condition 12> The period of time required to resume autonomous driving is longer than the pre-determined time limit. <Condition 13> The total number of the movable object 20 and other movable objects in which the abnormal state occurs is greater than a predetermined value M (M is an integer greater than or equal to two). <Condition 14> The total number of the movable object 20 and other movable objects traveling in automatic driving is greater than a predetermined value N (N is an integer greater than or equal to two). <Condition 15> The current position of the movable object 20 (ie, the position where the movable object 20 stops) is not within the predetermined area. <Condition 16> The yaw rate sensor mounted on the movable object 20 has been destroyed. <Condition 17> The movable object 20 needs to deviate from the set predetermined route. <Condition 18> The weather around the movable object 20 is bad weather. <Condition 19> The duration of the state in which the communication between the movable object 20 and the remote operation device 100 is interrupted is longer than or equal to the predetermined period of time.

[0112] When the driving mode is switched from the remote automatic driving mode to the remote manual driving mode, the operator needs to remotely operate the driving of the movable object 20. Therefore, by using the condition 11, it is determined whether the operator takes a position and posture at which the operator can operate the operator input device 110. The remote controller 120 can easily determine whether to allow the selection of the control of the movable object 20 in the remote manual driving mode based on whether the condition 11 is satisfied.

[0113] When condition 11 is used, when the remote controller 120 receives an input from the operator input device 110 within a predetermined period of time after the notification that the driving mode is switched to the remote manual driving mode is output to the display device 115, the remote controller 120 determines that condition 11 is satisfied. When there is no input from the operator input device 110 within a predetermined period of time after the notification that the driving mode is switched to the remote manual driving mode is output to the display device 115, the remote controller 120 determines that condition 11 is not satisfied.

[0114] Conditions 12 to 14 are conditions that negate conditions 1 to 3 described in the first embodiment, respectively. Conditions 16 to 19 are conditions that negate conditions 7 to 10, respectively.

[0115] The predetermined area of ​​condition 15 is, for example, an area where other movable objects traveling in the remote automatic driving mode stop at a low frequency, as shown in condition 4. The predetermined area of ​​condition 15 may be an area where the distance from the predetermined route to the movable object 20 is shorter than or equal to the predetermined distance when the movable object 20 deviates from the predetermined route, as shown in condition 5. The predetermined area of ​​condition 15 may be an area where the current position of the movable object 20 (i.e., the position where the movable object 20 stops) is in an area where the operator who takes the position and posture where the operator can operate the operator input device can recognize the movable object 20, as shown in condition 6.

[0116] In the present embodiment, whether to allow the driving mode of the movable object 20 to be switched from the remote automatic driving mode to the remote manual driving mode is determined based on the fact that the first condition is not satisfied but the second condition is satisfied. Therefore, it is assumed that a condition that negates the set first condition is not selected as the second condition. For example, when condition 1 is set as the first condition, condition 12 is not selected as the second condition.

[0117] Figure 6 1 is a flowchart showing a determination process regarding the switching of the driving mode. When the abnormality detection unit 123 detects the occurrence of an abnormal state of the movable object 20 based on the movable object sensor information, the movable object state information, and the external image capture information regarding the movable object 20 acquired by the acquisition unit 121, the abnormality detection unit 123 starts Figure 6 Hereinafter, an example in which the condition 7 described in the first embodiment is adopted as the first condition and the condition 11 is adopted as the second condition will be described.

[0118] In step S201 , the determination unit 124 determines whether condition 7, which is the first condition, is satisfied.

[0119] The determination unit 124 determines whether the yaw rate sensor mounted on the movable object 20 has been destroyed based on the movable object state information acquired by the movable object 20. Specifically, when the determination unit 124 does not receive a detection value from the yaw rate sensor within a certain period of time, the determination unit 124 determines that the yaw rate sensor has been destroyed. In this case, the determination unit 124 determines that the first condition is not satisfied. On the other hand, when the determination unit 124 has received a detection value from the yaw rate sensor, the determination unit 124 determines that the yaw rate sensor is not destroyed. In this case, the determination unit 124 determines that the first condition is satisfied.

[0120] When the first condition is satisfied (Yes in step S201), the determination unit 124 performs the process of step S202. On the other hand, when the first condition is not satisfied (No in step S201), the determination unit 124 performs the process of step S204.

[0121] In step S202, the determination unit 124 determines to continue the remote automatic driving mode of the movable object 20. In step S203, the determination unit 124 outputs a notification of executing the resumption of the automatic driving in the remote automatic driving mode to the control value creation unit 125. When the control value creation unit 125 receives the notification of executing the resumption of the automatic driving in the remote automatic driving mode, the control value creation unit 125 sends a control instruction to resume the automatic driving to the movable object 20. Then, Figure 6 The process shown ends.

[0122] Thereafter, the control value creation unit 125 remotely controls the movable object 20 in the remote automatic driving mode by sending a control instruction to the movable object 20 as needed.

[0123] In step S204 , the determination unit 124 determines whether condition 11 as the second condition is satisfied.

[0124] Specifically, the determination unit 124 outputs a notification that the driving mode is switched from the remote automatic driving mode to the remote manual driving mode to the control value creation unit 125. When the control value creation unit 125 receives the notification that the driving mode is switched from the remote automatic driving mode to the remote manual driving mode, the control value creation unit 125 outputs an instruction for driving in the remote manual driving mode to the display device 115.

[0125] When there is an input from the operator input device 110 within a predetermined period of time after the determination unit 124 provides the control value creation unit 125 with a notification related to the switching of the driving mode, the determination unit 124 determines that the operator takes a position and posture at which the operator can operate the operator input device 110. In this case, the determination unit 124 determines that the second condition is satisfied. When there is no input from the operator input device 110 within a predetermined period of time after the determination unit 124 provides the control value creation unit 125 with a notification related to the switching of the driving mode, the determination unit 124 determines that the operator does not take a position and posture at which the operator can operate the operator input device 110. In this case, the determination unit 124 determines that the second condition is not satisfied.

[0126] When the second condition is satisfied (Yes in step S204), the determination unit 124 performs the process of step S205. On the other hand, when the second condition is not satisfied (No in step S204), the determination unit 124 performs the process of step S207.

[0127] In step S205, the determination unit 124 determines to switch from the remote automatic driving mode to the remote manual driving mode. In step S206, the determination unit 124 outputs a notification that the driving mode is switched from the remote automatic driving mode to the remote manual driving mode to the control value creation unit 125. Then, Figure 6 The process shown ends.

[0128] Thereafter, each time the control value creating unit 125 receives an input from the operator input device 110, the control value creating unit 125 sends a control instruction according to the operator's driving operation to the movable object 20. In this way, remote control of the movable object 20 in the remote manual driving mode is performed.

[0129] In step S207, the determination unit 124 determines to switch from the remote automatic driving mode to the manned driving mode. In step S208, the determination unit 124 outputs a notification that the driving mode is switched to the manned driving mode to the control value creation unit 125. When the control value creation unit 125 receives the notification that the driving mode is switched to the manned driving mode, the control value creation unit 125 outputs the notification that the driving mode is switched to the manned driving mode to the display device 115.

[0130] In step S209, the determination unit 124 provides an instruction notification to at least one worker WO via the terminal device T1 held by the worker WO or the terminal device T2 installed at the waiting position OF to recover the movable object 20. The worker WO who received the notification moves to the position of the movable object 20 and drives to move the movable object 20. In this way, the movable object 20 is recovered.

[0131] In step S210, the determination unit 124 waits until the determination unit 124 receives a notification from the operator WO via the terminal device T1 that the recovery of the movable object 20 has been completed (No in step S210). When the determination unit 124 receives a notification from the operator WO via the terminal device T1 that the recovery of the movable object 20 has been completed (Yes in step S210), the determination unit 124 ends. Figure 6 The process shown.

[0132] As described above, in the present embodiment, as in the case of the first embodiment, the remote controller 120 can easily determine whether to allow the resumption of the automatic driving in the remote automatic driving mode to be selected according to whether the predetermined first condition is satisfied.

[0133] Under the condition that the first condition is not satisfied but the second condition is satisfied, the remote controller 120 can easily determine whether to allow the driving mode to be switched from the remote automatic driving mode to the remote manual driving mode.

[0134] In this embodiment, first, it is determined whether to allow resumption of the autonomous driving in the remote autonomous driving mode, and then it is determined whether to switch to driving in the remote manual driving mode. The number of operators who remotely operate the movable object 20 is limited; however, when priority is given to determining whether to resume the autonomous driving, human resources are efficiently managed.

[0135] In the present embodiment, when neither the automatic driving in the remote automatic driving mode nor the driving in the remote manual driving mode is allowed, the movable object 20 can be quickly recovered by providing a notification to the worker WO. C. Other Embodiments

[0136] (C1) In the second embodiment, the remote controller 120 determines whether the first condition is satisfied, and then determines whether the second condition is satisfied. However, the remote controller 120 may perform determination regarding the first condition and determination regarding the second condition at the same time.

[0137] (C2) In the second embodiment, in condition 19 as the second condition, it is determined whether the duration of the state in which the communication between the movable object 20 and the remote controller 120 is interrupted is longer than or equal to the predetermined time period. Even when condition 19 is not adopted as the second condition, the remote controller 120 can provide a notification of the fact via the terminal device T1 and the terminal device T2 when the duration of the state in which the communication between the movable object 20 and the remote controller 120 is interrupted is longer than or equal to the predetermined time period.

[0138] (C3) In the first embodiment, the abnormal stop of the movable object 20 in which the remote automatic driving mode is set has been described as an example of the abnormal state of the movable object 20. However, the abnormal state of the movable object 20 is not limited to the abnormal stop of the movable object 20. For example, a case where a failure occurs in the communication line between the remote controller 120 and the movable object 20 or a case where a failure occurs in the remote operation device 100 itself may be determined as the occurrence of the abnormal state.

[0139] (C4) In the first embodiment, the process from the acquisition of the position information of the movable object 20 to the generation of the control command is performed by the remote controller 120. In contrast, at least a part of the process from the acquisition of the position information of the movable object 20 to the generation of the control command can be performed by the movable object 20. For example, the following modes (1) to (3) can be applied.

[0140] (1) The remote controller 120 may acquire the position information of the movable object 20, determine the target position to which the movable object 20 is to go next, and generate a route from the current position of the movable object 20 indicated by the acquired position information to the target position. The remote controller 120 may generate a route to the target position between the current position and the destination, or may generate a route to the destination. The remote controller 120 may send the generated route to the movable object 20. The movable object 20 may generate a control instruction so that the movable object 20 travels along the route received from the remote controller 120, and control the driving device 220, the steering device 230, and the braking device 240 by using the generated control instruction.

[0141] (2) The remote controller 120 may acquire the position information of the movable object 20 and transmit the acquired position information to the movable object 20. The movable object 20 may determine the target position to which the movable object 20 is to go next, generate a route from the current position of the movable object 20 indicated by the received position information to the target position, generate a travel control signal so that the movable object 20 travels along the generated route, and control the driving device 220, the steering device 230, and the braking device 240 by using the generated travel control signal.

[0142] (3) In the above-mentioned modes (1) and (2), the detection result outputted from at least one of the external sensor 270 and the internal sensor 280 installed on the movable object 20 may be used for at least one of the generation of the route and the generation of the control instruction. For example, in the above-mentioned mode (1), the remote controller 120 may obtain the detection result of at least one of the external sensor 270 and the internal sensor 280, and reflect the detection result in the route when generating the route. In the above-mentioned mode (1), the movable object 20 may obtain the detection result of at least one of the external sensor 270 and the internal sensor 280, and reflect the detection result in the control instruction when generating the control instruction. In the above-mentioned mode (2), the movable object 20 may obtain the detection result of at least one of the external sensor 270 and the internal sensor 280, and reflect the detection result in the route when generating the route. In the above-mentioned mode (2), the movable object 20 may obtain the detection result of at least one of the external sensor 270 and the internal sensor 280, and reflect the detection result in the control instruction when generating the control instruction.

[0143] (C5) In the first embodiment, an example is described in which the movable object 20 travels in any one of the manned driving mode, the remote manual driving mode, and the remote automatic driving mode. The movable object 20 can travel in an autonomous driving mode in addition to these driving modes. The autonomous driving mode is a driving mode in which the travel of the movable object 20 is controlled by the autonomous control of the movable object 20. In the remote automatic driving mode, the driving of the movable object 20 is automatic driving.

[0144] Figure 7 2 is a block diagram showing a functional configuration of a movable object 20 capable of traveling in an autonomous driving mode. This embodiment is different from the first embodiment in that a movable object controller 290 provided in the movable object 20 has a function as a controller. Hereinafter, components different from those of the first embodiment will be described in detail. Descriptions of components similar to those of the first embodiment are omitted.

[0145] In the memory 292, a program for realizing not only the functions of the CPU 291 shown in the first embodiment but also the functions of the acquisition unit 121, the calculation unit 122, and the control value creation unit 125 of the remote controller 120 is stored. The CPU 291c functions as the acquisition unit 221, the calculation unit 222, and the control value creation unit 225 corresponding to these functions of the remote controller 120 in the first embodiment by running the program stored in the memory 292. Information on the scheduled route of the movable object 20 is stored in the memory 292. The scheduled route is represented by a node indicating a departure point, a node indicating a passing point, a node indicating a destination, and sections connecting these nodes respectively.

[0146] The acquisition unit 221 acquires the movable object sensor information and the movable object state information. The acquisition unit 221 acquires the external image capturing information from the detection device 50.

[0147] The calculation unit 222 calculates the position and orientation of the movable object 20. For example, the position of the movable object 20 and the orientation of the movable object 20 can be estimated by using the external image capture information. For example, the position of the movable object 20 can be acquired by calculating the coordinates of the measurement point of the movable object 20 in the image coordinate system by using the outer shape of the movable object 20 detected from the external image capture information, and converting the calculated coordinates into coordinates in the global coordinate system.

[0148] The orientation of the movable object 20 may be estimated based on, for example, the orientation of the motion vector of the movable object 20 calculated by using an optical flow technique. Alternatively, the orientation of the movable object 20 may be calculated by using a detection value of a yaw rate sensor mounted on the movable object 20. For example, the shape of the movable object 20 may be detected by inputting a captured image into a detection model using artificial intelligence.

[0149] The control value creating unit 225 determines the acceleration and the steering angle of the movable object 20 by using the position and the orientation of the movable object 20 calculated by the calculating unit 222, and causes the movable object 20 to travel at the determined acceleration and steering angle.

[0150] Figure 8 2 is a flowchart showing a traveling method of the movable object 20 in which the autonomous driving mode is set. The calculation unit 222 acquires the position of the movable object 20 by using the detection result of the detection device 50 (step S21). Specifically, the calculation unit 222 calculates the position of the movable object 20 by using the external image capture information acquired by the acquisition unit 221 from the detection device 50. The information about the position of the movable object 20 includes X, Y, and Z coordinates in the global coordinate system of the plant.

[0151] The control value creation unit 225 determines the target position to which the movable object 20 will go next (step S22). In the present embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system of the plant. The control value creation unit 225 determines the target position to which the movable object 20 will go next by using the information about the position of the movable object 20 calculated by the calculation unit 222 and the predetermined route. The control value creation unit 225 determines the target position on the predetermined route ahead of the current position of the movable object 20.

[0152] The control value creation unit 225 generates a control instruction for causing the movable object 20 to travel toward the determined target position (step S23). In the present embodiment, the control instruction includes the acceleration and steering angle of the movable object 20 as parameters. The control value creation unit 225 calculates the travel speed of the movable object 20 according to the change in the position of the movable object 20, and compares the calculated travel speed with the predetermined target speed of the movable object 20. When the travel speed is lower than the target speed, the control value creation unit 225 determines the acceleration to accelerate the movable object 20. When the travel speed is higher than the target speed, the control value creation unit 225 determines the acceleration to decelerate the movable object 20. When the movable object 20 is located on the predetermined route, the control value creation unit 225 determines the steering angle so that the movable object 20 does not deviate from the predetermined route. When the movable object 20 is not located on the predetermined route, in other words, when the movable object 20 deviates from the predetermined route, the control value creation unit 225 determines the steering angle so that the movable object 20 returns to the predetermined route.

[0153] The control value creation unit 225 controls the driving device 220, the steering device 230, and the braking device 240 by the acceleration and the steering angle indicated by the generated control command, so that the movable object 20 moves (step S24). The control value creation unit 225 repeatedly acquires the position of the movable object 20, determines the target position, generates the control command, and controls the driving at predetermined intervals.

[0154] When the movable object 20 travels by autonomous control, the movable object 20 can be caused to travel without control of the remote controller 120 .

[0155] The abnormality detection unit 123 of the remote controller 120 detects the occurrence of an abnormal state in the movable object 20, as in the case of the first embodiment. In the present embodiment, when the movable object 20 to which the remote automatic driving mode or the autonomous driving mode is set is traveling, the abnormality detection unit 123 detects the occurrence of an abnormal state in the movable object 20 by using movable object sensor information and movable object state information of the movable object 20, external image capture information, information indicating the communication state between the remote operation device 100 and the movable object 20, and the like. The abnormal state in the movable object 20 means that the movable object 20 to which the remote automatic driving mode or the autonomous driving mode is set stops abnormally. The abnormal stop means an unexpected stop of the movable object 20 traveling in the remote automatic driving mode or the autonomous driving mode.

[0156] When the movable object 20 set in the remote automatic driving mode or the autonomous driving mode stops abnormally, the determination unit 124 determines whether to restore the automatic driving in the remote automatic driving mode or the autonomous driving mode, or to switch from the remote automatic driving mode or the autonomous driving mode to the remote manual driving mode.

[0157] When the movable object 20 in which the remote automatic driving mode or the autonomous driving mode is set stops abnormally, any one of the following processes is performed. The processes to be performed include (i) switching to the remote manual driving mode and remotely operating the driving of the movable object 20 by the driving operation of the operator, (ii) resuming the automatic driving in the remote automatic driving mode or the autonomous driving mode, and (iii) an operator entering the movable object 20 and the operator driving the movable object 20.

[0158] In the present embodiment, when the first condition is satisfied, the determination unit 124 determines to restore the automatic driving of the movable object 20 in the remote automatic driving mode. The first condition is a condition determined regarding the restoration of the automatic driving in the remote automatic driving mode or the autonomous driving mode. When the determination unit 124 determines to restore the automatic driving of the movable object 20 in the remote automatic driving mode or the autonomous driving mode, the switching of the driving mode is not performed. On the other hand, when the first condition is not satisfied, the determination unit 124 determines to switch from the remote automatic driving mode or the autonomous driving mode to the remote manual driving mode. The details of the first condition related to the restoration of the automatic driving in the remote automatic driving mode or the autonomous driving mode are similar to those of the first embodiment.

[0159] The determination unit 124 transmits a notification about the switching of the driving mode to the control value creation unit 125 according to the determination result about whether to resume the automatic driving in the remote automatic driving mode or the autonomous driving mode.

[0160] When the control value creation unit 125 receives a notification from the determination unit 124 that the driving mode is not switched, the control value creation unit 125 performs control for restoring the automatic driving of the movable object 20 in the factory area while the driving mode is maintained in the remote automatic driving mode or the autonomous driving mode. On the other hand, when the control value creation unit 125 receives a notification from the determination unit 124 that the driving mode is switched to the remote manual driving mode, the control value creation unit 125 performs control on the movable object 20 to switch from the remote automatic driving mode to the remote manual driving mode. Thus, the automatic driving of the movable object 20 in the remote manual driving mode is started.

[0161] The determination process of switching the driving mode is similar to that of the first embodiment (see Figure 4B ). Assume that Figure 4BIn the text, the term "remote automated driving mode" is read as "remote automated driving mode or autonomous driving mode".

[0162] As described above, also in the present embodiment, the remote controller 120 can easily determine whether to permit selection of the remote automatic driving mode or resumption of the automatic driving in the autonomous driving mode according to whether the predetermined first condition is satisfied.

[0163] (C6) In the above mode (C5), the movable object 20 may not be able to travel in the remote automatic driving mode. In this case, the movable object 20 can travel in the manned driving mode, the remote manual driving mode, and the autonomous driving mode.

[0164] (C7) In the above-mentioned mode (C5), the detection result output from the internal sensor 280 of the movable object 20 may be used for at least one of the generation of the route and the generation of the travel control signal. For example, the movable object 20 may obtain the detection result of the internal sensor 280, and reflect the detection result of the internal sensor 280 in the route when generating the route. The movable object 20 may obtain the detection result of the internal sensor 280, and reflect the detection result of the internal sensor 280 in the control instruction when generating the control instruction.

[0165] (C8) In the above-mentioned mode (C5), in order to determine whether to switch the driving mode of the movable object 20 from the remote automatic driving mode or the autonomous driving mode to the remote manual driving mode, determination regarding the first condition is performed. Alternatively, as in the case of the second embodiment, in order to determine whether to switch the driving mode of the movable object 20 from the remote automatic driving mode or the autonomous driving mode to the remote manual driving mode, not only determination regarding the first condition but also determination regarding the second condition may be performed.

[0166] (C9) In the first embodiment, an example in which the movable object 20 is a battery electric vehicle has been described. However, the movable object 20 is not limited to a battery electric vehicle. The term "movable object" means an object that is capable of moving, and is, for example, a vehicle or an electric vertical take-off and landing machine (a so-called aircraft). The vehicle may be a vehicle that travels by wheels, such as a passenger car, a truck, a bus, a two-wheeled vehicle, and a four-wheeled vehicle. Alternatively, the vehicle may be a vehicle that travels by tracks, such as a tank and an engineering vehicle. Vehicles include battery electric vehicles, gasoline vehicles, hybrid electric vehicles, and fuel cell electric vehicles. When the movable object 20 is an object other than a vehicle, the expression "vehicle" in the aspects of the present invention may be replaced with "movable object" as needed, and the expression "travel" may be replaced with "move" as needed.

[0167] (C10) In the first embodiment, an example has been described in which the movable object 20 adopts a steer-by-wire system, a throttle-by-wire system, and a brake-by-wire system. However, in the movable object 20, the steering wheel and the wheel may be connected by a mechanical structure. In the movable object 20, the accelerator pedal and the throttle valve may be connected by a mechanical structure. In the movable object 20, the brake pedal and the brake caliper may be connected by a mechanical structure.

[0168] (C11) In the above-mentioned first embodiment, an example has been described in which the remote controller 120 remotely operates the movable object 20 in the remote automatic driving mode and the remote manual driving mode. Alternatively, a remote automatic driving device that remotely operates the movable object 20 in the remote automatic driving mode and a remote manual driving device that remotely operates the movable object 20 in the remote manual driving mode may be prepared separately. In this case, another device different from the remote automatic driving device or the remote manual driving device may determine whether to switch the driving mode. Alternatively, either the remote automatic driving device or the remote manual driving device may determine whether to switch the driving mode.

[0169] (C12) The means for realizing the functions of the remote controller 120 are not limited to software, and part or all of the functions may be realized by dedicated hardware. For example, a circuit typically a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC) may be used as dedicated hardware.

[0170] (C13) In the above-mentioned various embodiments, the movable object 20 only needs to have a configuration that is movable by automatic driving, and for example, it can be a mode of a platform including the following configuration. Specifically, in order to exercise three functions (i.e., "travel", "turn" and "stop") by automatic driving, the movable object 20 only needs to include at least a movable object controller, a drive device, a steering device and a braking device. When the movable object 20 obtains information from an external source for automatic driving, the movable object 20 only needs to further include a communication device. In other words, the movable object 20 that can be moved by automatic driving does not need to be equipped with at least one or some internal components (such as a driver's seat and an instrument panel), does not need to be equipped with at least one or some external components (such as a bumper and a fender), and does not need to be equipped with a body shell. In this case, the remaining components such as the body shell can be attached to the movable object 20 before the movable object 20 is shipped from the factory, or the movable object 20 can be shipped from the factory in a state where the remaining components such as the body shell are not attached to the movable object 20, and then the remaining components such as the body shell can be attached to the movable object 20. The components can be attached in a selected direction, i.e., the upper side, lower side, front side, rear side, right side or left side of the movable object 20 can be attached in the same direction, and can be attached in different directions. The pattern of the platform can also be positioned similarly to the movable object 20 in the first embodiment.

[0171] (C14) The movable object 20 can be manufactured by a selected manufacturing method. For example, the movable object 20 can be manufactured by assembling a plurality of modules. A module means a unit composed of a plurality of components concentrated according to the position or function of the movable object 20. For example, the platform of the movable object 20 can be manufactured by assembling a front module forming the front of the platform, a center module forming the center of the platform, and a rear module forming the rear of the platform. The number of modules constituting the platform is not limited to three, and can be two or less or four or more. In addition to or instead of the components forming the platform, the components forming the parts of the movable object 20 that are different from the platform can be modularized. Various modules can include selected external components (such as bumpers and grilles) or selected internal components (such as seats and consoles). Not limited to the movable object 20, a movable object according to a selected mode can be manufactured by assembling a plurality of modules. Such a module can be manufactured by, for example, joining a plurality of components using welding or fixings, or can be manufactured by integrally molding at least one component or some of the plurality of components constituting the module as one component. The molding technology of molding a component, especially a relatively large component, in one piece is also called giga casting or mega casting. For example, the front module, the center module and the rear module can be manufactured by using giga casting.

[0172] (C15) Transporting a vehicle using the movement of the vehicle caused by automatic driving is also called "self-propelled transportation". The configuration for realizing self-propelled transportation is called "vehicle remote control autonomous travel transportation system". A production system that produces vehicles by using self-propelled transportation is called "self-propelled production". For example, in self-propelled production, at least a part of the transportation of the vehicle is realized by self-propelled transportation in the factory where the vehicle is manufactured.

[0173] The present invention is not limited to the above-described embodiments, and can be implemented in various modes without departing from the gist of the present invention. For example, the technical features in the embodiments corresponding to the technical features in the aspects described in the summary of the invention can be replaced or combined as needed to solve the above-mentioned inconveniences or achieve one, some or all of the above-mentioned advantageous effects. Technical features that are not described as indispensable in the specification can be deleted as needed.

Claims

1. A controller for controlling the operation of a movable object capable of traveling in an automated driving mode, characterized in that: The controller comprises: a detection unit configured to detect occurrence of an abnormal state when the movable object is traveling in automatic driving, the abnormal state including an abnormal stop of the movable object; and A control unit, wherein: The control unit is configured to perform control to restore the automatic driving of the movable object when (i) the occurrence of the abnormal state is detected and (ii) a predetermined first condition is satisfied; and When (i) the occurrence of the abnormal state is detected and (ii) the first condition is not satisfied, the control unit is configured not to perform control to resume the automatic driving.

2. The controller according to claim 1, characterized in that: The movable object has a manned driving mode, a remote manual driving mode and a remote automatic driving mode; The manned driving mode is a driving mode in which the movable object travels when a driver in the movable object operates a driver input device provided in the movable object; The remote manual driving mode is a driving mode in which the movable object travels by receiving a traveling condition without the driver in the movable object; The travel condition is a condition generated when an operator operates an operator input device provided at a location different from a location where the movable object is located, and the travel condition defines a travel behavior of the movable object; the remote automatic driving mode is a driving mode in which the movable object travels by receiving the travel condition generated by the controller without operating any one of the driver input device and the operator input device; When (i) the occurrence of the abnormal state is detected and (ii) the first condition is satisfied, the control unit is configured to perform control to restore the autonomous driving while the driving mode of the movable object is maintained in the remote autonomous driving mode; and When (i) occurrence of the abnormal state is detected and (ii) the first condition is not satisfied, the control unit is configured not to perform control of resuming the autonomous driving in the remote autonomous driving mode.

3. The controller according to claim 2, characterized in that: The control unit is configured to switch the driving mode of the movable object from the remote automatic driving mode to the remote manual driving mode when (i) the occurrence of the abnormal state is detected, (ii) the first condition is not satisfied, and (iii) a predetermined second condition is satisfied.

4. The controller according to claim 3, characterized in that: The first condition includes a condition that a time period required to resume the autonomous driving is shorter than or equal to a predetermined time limit.

5. The controller according to claim 3, characterized in that: The first condition includes a condition that the total number of the movable object and other movable objects in which the abnormal state occurs is less than or equal to a value M (M is an integer greater than or equal to two).

6. The controller according to claim 3, characterized in that: The first condition includes a condition that a total number of the movable object and other movable objects traveling in the autonomous driving is less than or equal to a value N (N is an integer greater than or equal to two).

7. The controller according to claim 3, characterized in that: The first condition includes a condition that a current position of the movable object is within a predetermined area.

8. The controller according to claim 7, characterized in that: The predetermined area is any one of the first area, the second area and the third area; The first area is an area where other movable objects traveling in the autonomous driving stop at a low frequency; The second area is an area where the distance from the predetermined route to the movable object is shorter than or equal to a predetermined distance when the movable object deviates from the predetermined route; The predetermined route is set for the movable object; The third area is an area where the operator can identify the movable object; and The operator assumes a position and posture that enables the operator to operate the operator input device in the third area.

9. The controller according to claim 3, characterized in that: The first condition includes the following condition: the yaw rate sensor installed on the movable object is not damaged.

10. The controller according to claim 3, characterized in that: The first condition includes a condition that the movable object does not need to travel deviating from a predetermined route set for the movable object.

11. The controller according to claim 3, characterized in that: The first condition includes the following condition: the weather around the movable object is not bad weather.

12. The controller according to claim 3, characterized in that: The first condition includes a condition that a line for communication between the movable object and the controller is established.

13. The controller according to claim 4, characterized in that: The second condition includes the following condition: a time period required to resume the automatic driving exceeds the time limit.

14. The controller according to claim 5, characterized in that: The second condition includes a condition that the total number of the movable object and other movable objects in which the abnormal state occurs exceeds the value M.

15. The controller according to claim 6, characterized in that: The second condition includes a condition that the total number of the movable object and other movable objects traveling in the automatic driving exceeds the value N.

16. The controller according to claim 7, characterized in that: The second condition includes a condition that the current position of the movable object is not within the predetermined area.

17. The controller according to claim 9, characterized in that: The second condition includes a condition that the yaw rate sensor mounted on the movable object has been destroyed.

18. The controller according to claim 10, characterized in that: The second condition includes a condition that the movable object needs to travel deviating from the predetermined route set for the movable object.

19. The controller according to claim 11, characterized in that: The second condition includes the following condition: the weather around the movable object is the bad weather.

20. The controller according to claim 12, characterized in that The second condition includes a condition that a duration of a state in which communication between the movable object and the controller is stopped is longer than or equal to a predetermined period of time.

21. The controller according to claim 4, characterized in that The second condition includes a condition that the operator takes a position and a posture at which the operator can operate the operator input device.

22. A controller according to any one of claims 3 to 21, characterized in that: Also included is a determination unit, wherein: The determining unit is configured to determine whether the first condition is satisfied; and The determining unit is configured to further determine whether the second condition is satisfied when the first condition is not satisfied.

23. The controller according to claim 22, characterized in that A notification unit is further included, the notification unit being configured to provide notification of information that the first state is established when the determination unit determines that the first state that does not satisfy the first condition and the second condition is established.

24. A controller according to any one of claims 3 to 21, characterized in that It also includes a notification unit configured to provide a notification of information that a second state has been established when the second state is established, wherein the second state is a state in which the duration of the state in which communication between the movable object and the controller is stopped is longer than or equal to a predetermined time period.

Images