Control device and control method

Abstract

A control device includes a processor. The processor is configured to: when an abnormal state occurs in a plurality of mobile bodies that perform automatic driving autonomously or by remote control, designate at least one mobile body to be processed preferentially from among the plurality of mobile bodies as a target mobile body based on the states of the plurality of mobile bodies; and perform control to take necessary measures on the target mobile body.

Description

Technical Field

[0001] The present disclosure relates to a control device and a control method. Background Art

[0002] Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2017-538619 (JP 2017-538619A) describes a vehicle that is autonomously driven by remote control. Summary of the invention

[0003] When a plurality of vehicles traveling by automatic driving stop due to some kind of failure, it is unclear how to handle the plurality of stopped vehicles in what order of priority. A technology for determining how to handle the plurality of stopped vehicles in what order of priority is desired.

[0004] The present disclosure may be implemented as follows.

[0005] (1) A first aspect of the present disclosure provides a control device. The control device includes a processor. The processor is configured to: when an abnormal state occurs in a plurality of mobile bodies that perform autonomous driving autonomously or through remote control, designate at least one mobile body to be processed preferentially from among the plurality of mobile bodies as a target mobile body based on the states of the plurality of mobile bodies; and perform control to take necessary measures on the target mobile body. By using the control device according to the present invention, it is possible to easily determine the target to be processed first among multiple moving bodies in abnormal states, and take necessary measures for the designated moving bodies. Therefore, it is easy to determine how to process the moving bodies that are already in abnormal states in what priority order.

[0006] (2) In the control device according to the above aspect, the automatic driving performed by remote control may include: automatic driving in which a plurality of mobile bodies drive by receiving driving conditions generated by the control device rather than based on remote operation of an operator located at a remote location; and the processor may be configured to: designate at least one mobile body that preferentially performs automatic driving autonomously or by remote control as a target mobile body from among a plurality of mobile bodies detected as being in an abnormal state, and perform control to enable the target mobile body to perform automatic driving autonomously or by remote control. With the control device according to the present aspect, the automatic driving performed autonomously or by remote control can be immediately resumed for the moving body designated as the target to be processed preferentially.

[0007] (3) In the control device according to the above aspect, the target moving body may include a moving body whose distance from a current position to a position where the downstream process is performed is equal to or smaller than a predetermined distance. When a mobile body located closer to the position where the downstream process is to be performed than other mobile bodies among the mobile bodies in an abnormal state is preferentially moved, the mobile body can reach the position where the downstream process is to be performed earlier than when a mobile body located farther from the position where the downstream process is to be performed than other mobile bodies is preferentially moved. With the control device according to the above aspect, the occurrence of idle time in the downstream process is reduced.

[0008] (4) In the control device according to the above aspect, the target moving body may include a moving body whose distance from a current position to a position where the upstream process is performed is equal to or smaller than a predetermined distance. For example, when an abnormal state occurs in a moving body, while an upstream process is being executed, or after an upstream process has just been executed, other moving bodies cannot enter the position where the upstream process is executed within a predetermined range from the position where the upstream process is executed, resulting in stagnation of the flow of the upstream process. With the control device according to the above aspect, in the case where an abnormal state occurs in a moving body, stagnation of the flow of the upstream process is reduced.

[0009] (5) In the control device according to the above aspect, the target moving body may include a moving body whose time from the current time until the set expected shipping date is equal to or less than a predetermined time. With the control device according to this aspect, occurrence of delay in shipment is suppressed.

[0010] (6) In the control device according to the above aspect, the target moving object may include a moving object on which no stationary object exists on the expected travel route. With the control device according to the present aspect, when there is no object on the expected travel route, the mobile body can be easily driven autonomously by remote control.

[0011] (7) In the control device according to the above aspect, the target mobile bodies may include mobile bodies of a model planned to be produced preferentially. With the control device according to the present aspect, it is easy to designate a mobile body to be processed with priority. In addition, the occurrence of delay in the production plan is suppressed.

[0012] (8) In the control device according to the above aspect, the target moving object may include a moving object currently located outdoors. With the control device according to this aspect, it is possible to prevent a moving object in an abnormal state from being left outdoors for a long period of time.

[0013] (9) In the control device according to the above aspect, the target moving object may include a moving object currently located indoors. According to the control device of this aspect, stagnation of the flow of movement of the moving object is suppressed in a limited indoor space.

[0014] (10) In the control device according to the above aspect, the processor can be configured to: when the target mobile body is a mobile body that enters an abnormal state while performing autonomous driving, execute control to start autonomous driving on the target mobile body through remote control; and when the target mobile body is a mobile body that enters an abnormal state while performing autonomous driving through remote control, execute control to start autonomous driving on the target mobile body. With the control device according to the present invention, switching to a driving mode different from the driving mode set when an abnormal state occurs is performed. As a result, automatic driving of the moving object can be started immediately, either autonomously or by remote control.

[0015] (11) In the control device according to the above aspect, the automatic driving performed by remote control may include: automatic driving in which a plurality of moving bodies travel based on a remote operation performed by an operator using an operator input device set at a position different from the moving bodies, and automatic driving in which a plurality of moving bodies travel by receiving driving conditions generated by the control device rather than based on the remote operation of the operator; and the processor may be configured to: designate at least one moving body for which automatic driving based on the remote operation of the operator is to be started preferentially as a target moving body from among a plurality of moving bodies detected as being in an abnormal state, and execute control of starting automatic driving based on the remote operation of the operator on the target moving body. With the control device according to the present aspect, automatic driving can be immediately started based on the remote operation of the operator for the moving object designated as a target to be processed with priority.

[0016] (12) In the control device according to the above aspect, the target moving body may include a moving body whose distance from a current position to a position where the downstream process is performed is equal to or smaller than a predetermined distance. When a mobile body located closer to the position where the downstream process is to be performed than other mobile bodies among the mobile bodies in an abnormal state is preferentially moved, the mobile body can reach the position where the downstream process is to be performed earlier than when a mobile body located farther from the position where the downstream process is to be performed than other mobile bodies is preferentially moved. With the control device according to the present aspect, the occurrence of idle time in the downstream process can be reduced.

[0017] (13) In the control device according to the above aspect, the target moving body may include a moving body whose distance from a current position to a position where the upstream process is performed is equal to or smaller than a predetermined distance. For example, when an abnormal state occurs in a moving body, while an upstream process is being executed, or after an upstream process has just been executed, other moving bodies cannot enter the position where the upstream process is executed within a predetermined range from the position where the upstream process is executed, resulting in stagnation of the flow of the upstream process. With the control device according to the present aspect, when an abnormal state occurs in a moving body, stagnation of the flow of the upstream process can be reduced.

[0018] (14) In the control device according to the above aspect, the target moving body may include a moving body whose time from the current time until the set expected shipping date time is equal to or less than a predetermined time. The control device according to this aspect can suppress the occurrence of delays in shipment.

[0019] (15) In the control device according to the above aspect, the target mobile object may include a mobile object capable of providing visual information to be used by an operator located at a remote location to remotely operate the mobile object. With the control device according to the present aspect, it is possible to provide the operator with visual information required for remote operation, thereby enabling the operator to smoothly drive the moving object by remote control.

[0020] (16) In the control device according to the above aspect, the target mobile object may include a mobile object located on a path along which an operator at a remote location has caused the mobile object to travel based on remote operation. With the control device according to the present aspect, the designated moving body is located on the path along which the operator causes the moving body to travel based on the remote operation, so the operator can smoothly drive the moving body by remote control.

[0021] (17) In the control device according to the above aspect, the target moving object may include a moving object around which no object that hinders the moving of the moving object exists when the moving of the moving object based on the remote operation of the operator at the remote location is started. With the control device according to the present aspect, there is no object that prevents the moving body from starting to travel, so that the operator can smoothly drive the moving body by remote control.

[0022] (18) In the control device according to the above aspect, the target mobile bodies may include mobile bodies of a model planned to be produced preferentially. With the control device according to this aspect, a moving body matching a set priority is preferentially moved, thereby preventing the progress of the plan from being affected.

[0023] (19) In the control device according to the above aspect, the target moving object may include a moving object currently located outdoors. According to the control device of this aspect, it is possible to prevent a moving object in an abnormal state from being left outdoors for a long time.

[0024] (20) In the control device according to the above aspect, the target moving object may include a moving object currently located indoors. With the control device according to the present aspect, it is possible to reduce the stagnation of the flow of movement of a moving object in a limited indoor space.

[0025] (21) In the control device according to the above aspect, the processor can be configured to: designate at least one moving body to be collected preferentially through the driving of an operator riding on the moving body as a target moving body from among multiple moving bodies detected as being in an abnormal state; and notify the operator of an instruction to collect the target moving body together with the information required for collection. With the control device according to the present aspect, at least one of the stopped moving bodies is designated as a moving body to be manually driven and collected by the operator. It is easy to determine how to deal with the stopped moving bodies in what order of priority. Further, the work of driving and collecting the designated moving bodies by the operator can be started immediately.

[0026] (22) In the control device according to the above aspect, the target moving body may include a moving body whose distance from a current position to a position where the downstream process is performed is equal to or smaller than a predetermined distance. When a mobile body located closer to the position where the downstream process is to be performed than other mobile bodies among the mobile bodies in an abnormal state is preferentially moved, the mobile body can reach the position where the downstream process is to be performed earlier than when a mobile body located farther from the position where the downstream process is to be performed than other mobile bodies is preferentially moved. With the control device according to the present aspect, the occurrence of idle time in the downstream process can be reduced.

[0027] (23) In the control device according to the above aspect, the target moving body may include a moving body whose distance from a current position to a position where the upstream process is performed is equal to or smaller than a predetermined distance. For example, when an abnormal state occurs in a moving body, while an upstream process is being executed, or after an upstream process has just been executed, other moving bodies cannot enter the position where the upstream process is executed within a predetermined range from the position where the upstream process is executed, resulting in stagnation of the flow of the upstream process. With the control device according to the present aspect, when an abnormal state occurs in a moving body, stagnation of the flow of the upstream process can be reduced.

[0028] (24) In the control device according to the above aspect, the target moving body may include a moving body whose time from the current time until the set expected shipping date time is equal to or less than a predetermined time. The control device according to this aspect can suppress the occurrence of delays in shipment.

[0029] (25) In the control device according to the above aspect, the target moving object may include a moving object in which there is no object that hinders the moving of the moving object when the moving object starts to be driven by the operator riding on the moving object. With the control device according to the present aspect, there is no object that prevents the moving body from starting to travel, so that the operator can smoothly drive the moving body.

[0030] (26) In the control device according to the above aspect, the target mobile bodies may include mobile bodies of a model planned to be produced preferentially. With the control device according to this aspect, a moving body matching a set priority is preferentially moved, thereby preventing the progress of the plan from being affected.

[0031] (27) In the control device according to the above aspect, the target moving object may include a moving object currently located outdoors. According to the control device of this aspect, it is possible to prevent a moving object in an abnormal state from being left outdoors for a long time.

[0032] (28) In the control device according to the above aspect, the target moving object may include a moving object currently located indoors. With the control device according to the present aspect, it is possible to reduce the stagnation of the flow of movement of a moving object in a limited indoor space.

[0033] (29) A second aspect of the present disclosure provides a control method. The control method is a method for controlling a moving object by a computer to perform automatic driving autonomously or by remote control without remote operation by an operator at a remote location, comprising: in the event that an abnormal state occurs in a plurality of moving objects that perform automatic driving autonomously or by remote control, designating at least one moving object that is to be preferentially performed autonomously or by remote control as a target moving object from among the plurality of moving objects; and performing control to cause the target moving object to perform automatic driving autonomously or by remote control. With the control method according to the above aspect, for the moving body designated as a target to be processed preferentially, the automatic driving performed autonomously or by remote control can be immediately resumed.

[0034] (30) A third aspect of the present disclosure provides a control method. The control method is a method for controlling a moving object by a computer to perform automatic driving autonomously or by remote control without remote operation by an operator at a remote location, comprising: when an abnormal state occurs in a plurality of moving objects that perform automatic driving autonomously or by remote control, designating at least one moving object that is to be prioritized to start automatic driving based on the remote operation of the operator from among the plurality of moving objects as a target moving object; and executing control to start automatic driving based on the remote operation by the operator on the target moving object. With the control method according to the above aspect, for a moving body designated as a target to be processed preferentially, automatic driving based on remote operation by an operator can be started immediately.

[0035] (31) A fourth aspect of the present disclosure provides a control method. The control method is a method for controlling a moving body by a computer to perform automatic driving autonomously or by remote control without relying on remote operation by an operator at a remote location, comprising: when an abnormal state occurs in a plurality of moving bodies that perform automatic driving autonomously or by remote control, designating at least one moving body to be collected by the operator riding the moving body as a target moving body from among the plurality of moving bodies; and notifying the operator of an instruction to collect the target moving body together with information required for collection. With the control method according to the above aspect, it is possible to immediately start the work of the operator driving and collecting the moving body designated as a target to be processed with priority.

[0036] The present disclosure can be implemented as various aspects, and for example, can be implemented as aspects such as a remote operation system, a remote operation device, a control method, a method of manufacturing a mobile body, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described below with reference to the accompanying drawings, wherein like numerals represent like elements, and wherein: Figure 1 shows the configuration of the remote operating system; Figure 2 is a schematic diagram showing an overview of a moving body; Figure 3 is a schematic diagram showing an overview of a remote operation device; Figure 4 An operator input device is shown; Figure 5 is a flow chart showing a vehicle driving method implemented by a remote control device; Figure 6 is a flow chart showing a control method according to a first embodiment; Figure 7is a flow chart showing a control method according to a second embodiment; Figure 8 is a flowchart showing a control method according to a third embodiment; Fig. 9 is a block diagram showing a functional configuration of a mobile body capable of traveling in an autonomous driving mode; and Fig.10 is a flowchart showing a method of driving a mobile object set in an autonomous driving mode. DETAILED DESCRIPTION A. First Embodiment A1. Configuration of remote control system

[0038] Figure 1 The configuration of the remote operation system 1 according to the first embodiment is shown. Figure 2 2 is a schematic diagram showing an overview of the moving object 20 . Figure 1 The remote operation system 1 shown in FIG. 1 is used in a factory that produces a moving body 20. The remote operation system 1 controls the moving body 20 that can be moved based on remote operation. The remote operation system 1 is used in a process of manufacturing the moving body, such as an assembly process and an inspection process of the moving body 20. The remote operation system 1 transports the moving body 20 over a predetermined transport section in the manufacturing process. The assembly, inspection, and other processes of the transported moving body 20 are performed by an operator or a facility.

[0039] In the present embodiment, the mobile bodies 20 are all pure electric vehicles (Battery Electric Vehicle, BEV) that can travel by automatic driving. The term "unattended operation" refers to an operation that is not based on the driving operation of the driver riding on the mobile body 20. Driving operation refers to an operation related to at least one of "driving", "turning" and "stopping" of the vehicle. Automatic driving is achieved by automatic or manual remote control performed by a device set outside the vehicle or by autonomous control of the mobile body 20. People who do not perform driving operations can be on the mobile body 20 that travels by automatic driving. Examples of people who do not perform driving operations include people who simply sit on the seat of the mobile body 20 and people who perform work different from driving operations (such as assembly, inspection and switch operations) while riding on the mobile body 20. Operations based on the driving operation of the driver riding on the mobile body 20 are sometimes also called "attended operations."

[0040] The remote operation system 1 includes a plurality of mobile bodies 20 , a plurality of detection devices 50 , and a remote operation device 100 .

[0041] The mobile object 20 travels in any one of the attended driving mode, the remote manual driving mode, and the remote automatic driving mode.

[0042] The attended driving mode is a driving mode in which the vehicle 20 is driven according to the driving operation of the driver riding on the vehicle 20. When the attended driving mode is set, the driving conditions of the vehicle 20 are set by the driver riding on the vehicle 20 operating the driver input device 210 provided in the vehicle 20 (see Figure 2 ) is determined. The driver input device 210 includes a steering wheel, an accelerator pedal, a brake pedal, etc. (not shown). The mobile body 20 adopts a wire-controlled steering system, a wire-controlled throttle system, and a wire-controlled brake system. The mobile body 20 travels according to the determined driving conditions. The driving conditions are conditions that specify the driving operation of the mobile body 20. Examples of driving conditions include driving speed, acceleration, and wheel steering angle.

[0043] The remote manual driving mode is a driving mode for remotely controlling the travel of the mobile body 20 based on the driving operation of a driver who is not on the mobile body 20. When the remote manual driving mode is set, a remote operating device 100 to be discussed later generates a travel condition based on the operator's operation of an operator input device to be discussed later. The remote operating device 100 is provided at a position different from the mobile body 20. The travel of the mobile body 20 is controlled based on the travel condition generated by the remote operating device 100. The driving of the mobile body 20 in the remote manual driving mode is also referred to as remote manual driving. In the remote manual driving mode, the driver does not need to ride on the mobile body 20 and operate the driving input device provided in the mobile body 20 to make the mobile body 20 travel.

[0044] The remote automatic driving mode is a driving mode in which the driving of the mobile body 20 is controlled according to the driving conditions generated by the remote operating device 100. When the remote automatic driving mode is set, the driving conditions are generated by the remote operating device 100 without the operator operating the operator input device. The driving of the mobile body 20 is controlled according to the driving conditions generated by the remote operating device 100. The driving of the mobile body 20 in the remote automatic driving mode is also called remote automatic driving. In the remote automatic driving mode, there is no need for the driver to operate the driving input device or for the operator who is not on the mobile body 20 to operate the operator input device. The driving of the mobile body 20 in the remote automatic driving mode is automatic driving. The term "automatic driving" refers to driving in which the mobile body 20 is driven autonomously by using a system instead of a driver to perform all recognition, prediction, determination, and operations related to driving operations.

[0045] like Figure 2As shown in FIG. 1 , the mobile body 20 includes a driver input device 210, a driving device 220 that drives the mobile body 20, a steering device 230 that changes the forward direction of the mobile body 20, a braking device 240 that brakes the mobile body 20, a communication device 260, an external sensor 270, an internal sensor 280, and a mobile body control device 290 that controls various parts of the mobile body 20. In the present embodiment, the driving device 220 includes a battery, a travel motor driven by power from the battery, and a drive wheel rotated by the travel motor.

[0046] The communication device 260 communicates with the remote control device 100 (see Figure 1 ) communication device. For example, the communication device 260 communicates with a device outside the mobile body 20 through a controller area network (CAN) communication. CAN communication is a communication standard that allows multi-directional transmission and reception. For example, the communication device 260 communicates with the remote operation device 100 connected to the network NT via an access point (not shown) in the factory.

[0047] The external sensor 270 includes a plurality of sensors for detecting the state of the surroundings of the mobile body 20. The external sensor 270 may include a vehicle-mounted camera, a millimeter wave radar, a light detection and ranging (LiDAR), an ultrasonic sensor, etc., mounted on the mobile body 20. The vehicle-mounted camera captures at least a portion of the area around the mobile body 20. The radar detects the distance, relative angle, relative speed, etc. relative to objects present in the area around the mobile body 20. The laser radar detects the distance and relative angle, the shape of the objects, etc. relative to objects present in the area around the mobile body 20. The sensor outputs the detection value to the mobile body control device 290.

[0048] The internal sensor 280 includes a plurality of sensors for detecting the internal state of the mobile body 20. The internal sensor 280 may include a current sensor, a voltage sensor, a battery temperature sensor, a vehicle speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, a Global Navigation Satellite System (GNSS) receiver, etc., which are mounted on the mobile body 20. The current sensor detects the value of the output current from the battery of the mobile body 20. The voltage sensor detects the value of the output voltage from the battery of the mobile body 20. The battery temperature sensor detects the temperature of the battery of the mobile body 20. The vehicle speed sensor detects the speed of the mobile body 20. The acceleration sensor detects the acceleration of the mobile body 20. The yaw rate sensor detects the yaw axis angular velocity (yaw rate) of the mobile body 20. The steering angle sensor detects the steering angle of each wheel of the mobile body 20. The GNSS receiver receives GNSS information. The sensor outputs the detection value to the mobile body control device 290.

[0049] The mobile body control device 290 is a computer including a central processing unit (CPU) 291, a memory 292, an input / output interface 293, and an internal bus 294. The mobile body control device 290 is composed of, for example, one or more electronic control units (ECUs).

[0050] The CPU 291, the memory 292, and the input / output interface 293 are connected in a bidirectionally communicable manner via the internal bus 294. The driver input device 210, the drive device 220, the steering device 230, the brake device 240, the communication device 260, the external sensor 270, and the internal sensor 280 are connected to the input / output interface 293. The memory 292 stores various programs and data. The CPU 291 controls various parts of the mobile body 20 by executing the programs stored in the memory 292.

[0051] In the present embodiment, when the attended driving mode is set, the mobile body control device 290 can control various parts of the mobile body 20 to make the mobile body 20 travel according to the driving operation performed by the driver riding on the mobile body 20 using the driver input device 210. When the remote manual driving mode or the remote automatic driving mode is set, the mobile body control device 290 controls various parts of the mobile body 20 according to the control command received from the remote operation device 100 to make the mobile body 20 travel.

[0052] Furthermore, the mobile control device 290 transmits information detected by the external sensor 270 as mobile sensor information to the remote operation device 100 via the communication device 260. The mobile control device 290 transmits information detected by the internal sensor 280 as mobile state information to the remote operation device 100 via the communication device 260.

[0053] Figure 1 The detection device 50 shown in is used to monitor the mobile body 20. The detection device 50 includes a plurality of cameras (external cameras) installed at a site of a factory. The position and orientation of the detection device 50 are pre-adjusted. For example, the detection device 50 communicates with the remote operation device 100 connected to the network NT via an access point (not shown) in the factory. The detection device 50 transmits external captured image information obtained by capturing a capture range to the remote operation device 100. The external captured image information may be, for example, image data captured by the detection device 50.

[0054] Figure 3 1 is a schematic diagram showing an overview of the remote control device 100. The remote control device 100 is a device for remotely controlling the travel of the mobile object 20.

[0055] The remote operation device 100 includes a communication device 105, at least one operator input device 110, at least one display device 115, at least one speaker 117, and a remote control device 120. The remote control device 120 is also referred to as a control device.

[0056] The communication device 105 connects the remote control device 100 to the mobile object 20 and the detection device 50 so as to be communicable. The communication device 105 may be, for example, a wireless communication device.

[0057] Figure 4 1 shows an operator input device 110. The operator input device 110 is a device operated by the operator OP to remotely control the mobile body 20 set to the remote manual driving mode. Figure 3 As shown in , the operator input device 110 is connected to the remote control device 120. Figure 4 As shown in , the operator input device 110 is provided in the cab (cockpit) CP together with the display device 115 and the speaker 117. The cab CP is provided at a position different from the mobile body 20. The operator input device 110 includes a steering wheel 111 as an input device for steering the mobile body 20 based on the remote operation, an accelerator pedal 112 as an input device for accelerating the mobile body 20 based on the remote operation, and a brake pedal 113 as an input device for decelerating the mobile body 20 based on the remote operation. Figure 4, the operator OP is shown as being viewed from the side, and thus the accelerator pedal 112 is not shown. The operator OP sits on a driving seat ST provided in the cab CP to operate the operator input device 110.

[0058] The display device 115 displays images showing various information according to the control of the remote control device 120. The display device 115 is installed at a position where the operator OP, who is in a position and posture where the operator input device 110 can be operated, can visually recognize the display device 115. When the operator OP is in a position and posture where the operator input device 110 can be operated, it means, for example, that the operator OP is sitting in the driving seat ST in the cab CP. The display device 115 displays, for example, the internal captured image information acquired by the vehicle-mounted camera and the external captured image information acquired by the detection device 50 according to the control of the remote control device 120.

[0059] The speaker 117 reproduces audio data according to the control of the remote control device 120. For example, the speaker 117 reproduces audio data acquired by a microphone (not shown) attached to a vehicle-mounted camera installed on the mobile body 20. The audio data is included in the internal captured image information acquired by the vehicle-mounted camera. The speaker 117 also reproduces audio data acquired by a microphone (not shown) attached to a camera as the detection device 50. The audio data is included in the external captured image information acquired by the detection device 50.

[0060] Figure 3 The remote control device 120 shown in FIG. 1 is a device for remotely controlling the travel of the mobile body 20 .

[0061] When the mobile body 20 is set to the remote manual driving mode, the operator operates the operator input device 110 to drive the mobile body 20. The remote control device 120 generates a control command that matches the operator's driving operation and transmits the control command to the mobile body 20. The mobile body 20 drives according to the control command received from the remote control device 120.

[0062] When the moving object 20 is set to the remote automatic driving mode, the remote control device 120 generates a control command using moving object sensor information, moving object state information, etc. The moving object 20 travels according to the control command received from the remote control device 120 .

[0063] The remote control device 120 is constituted by a computer including a CPU 11 , a memory 12 , an input / output interface 13 , and an internal bus 14 .

[0064] The CPU 11, the memory 12, and the input-output interface 13 are connected in a bidirectionally communicable manner via the internal bus 14. The communication device 105, the operator input device 110, the display device 115, and the speaker 117 are connected to the input-output interface 13. The memory 12 stores various programs and data. For example, the memory 12 stores information about the planned route of the moving body 20. The planned route is a path that is set in advance and along which the moving body 20 should travel. The respective functions of the acquisition unit 121, the calculation unit 122, the abnormality detection unit 123, the determination unit 124, and the control value preparation unit 125 are realized by the CPU 11 executing the program stored in the memory 12.

[0065] The acquisition unit 121 acquires moving body sensor information and moving body state information from the moving body 20. The acquisition unit 121 also acquires external captured image information from the detection device 50.

[0066] The calculation unit 122 calculates the position and orientation of the mobile body 20. For example, the position and orientation of the mobile body 20 can be estimated using external captured image information. The position of the mobile body 20 can be acquired by, for example, calculating the coordinates of the measurement point of the mobile body 20 in the image coordinate system using the outer shape of the mobile body 20 detected from the external captured image information, and converting the calculated coordinates into coordinates in the global coordinate system.

[0067] The orientation of the mobile object 20 can be estimated based on the orientation of a vector related to the motion of the mobile object 20 calculated using, for example, an optical flow method. In the optical flow method, the orientation of a vector related to the motion of the mobile object 20 is calculated based on the change in position of a feature point of the mobile object between frames of captured images. Alternatively, the orientation of the mobile object 20 can be calculated using a detection value from a yaw rate sensor mounted on the mobile object 20.

[0068] For example, the shape of the mobile object 20 can be detected by inputting the captured image into a detection model using artificial intelligence. Examples of detection models include, for example, a trained machine learning model that has been trained to achieve one of semantic segmentation and instance segmentation. As a machine learning model, for example, a convolutional neural network (hereinafter referred to as "CNN") trained by supervised learning using a learning data set can be used. The learning data set includes a plurality of training images containing a moving object, and a correct answer label indicating whether each area in the training image is an area indicating a moving object or an area indicating an area other than the moving object. During the training of the CNN, the parameters of the CNN are preferably updated by back propagation (error back propagation method) to reduce the error between the result output from the detection model and the correct answer label.

[0069] The abnormality detection unit 123 detects the occurrence of an abnormal state of the mobile body 20. In the present embodiment, when a plurality of mobile bodies 20 set to the remote automatic driving mode are traveling, the abnormality detection unit 123 detects the occurrence of an abnormal state of the mobile body 20 using mobile body sensor information and mobile body state information of the mobile body 20, external captured image information, information indicating the communication state between the remote operating device 100 and the mobile body 20, and the like. The abnormal state of the mobile body 20 refers to a state in which the mobile body 20 set to the remote automatic driving mode has been abnormally stopped. The abnormal stop refers to an unintentional stop of the mobile body 20 remotely operated in the remote automatic driving mode.

[0070] When multiple mobile bodies 20 are in an abnormal state, the determination unit 124 determines whether each mobile body 20 is a target to be processed with priority. The processing of restoring the automatic driving in the remote automatic driving mode is prioritized for the mobile body 20 that is the target. In the present embodiment, the target that prioritizes the start of automatic driving in the remote automatic driving mode is called the target mobile body. The determination unit 124 determines the mobile body 20 that satisfies a predetermined first condition as the target mobile body. The first condition is a condition determined in relation to the priority of restoring the automatic driving in the remote automatic driving mode. The target mobile body, that is, the mobile body 20 determined to restore the automatic driving in the remote automatic driving mode, is subject to control for restoring the automatic driving while remaining in the remote automatic driving mode. The first condition will be discussed in detail later. The determination unit 124 is also referred to as a designation unit.

[0071] On the other hand, the determination unit 124 determines the mobile body 20 that does not meet the first condition as a mobile body that does not prioritize the restoration of the automatic driving in the remote automatic driving mode (hereinafter referred to as a non-target mobile body). For the non-target mobile body, the remote automatic driving mode is switched to a different driving mode. For example, the mobile body 20 as a non-target mobile body is switched to the remote manual driving mode to travel based on the remote operation of the operator. Alternatively, the mobile body 20 as a non-target mobile body is switched to the manned driving mode and is driven and collected by the operator WO.

[0072] The determination unit 124 transmits to the control value preparation unit 125 a notification of switching of the driving mode according to a determination result as to whether the mobile body 20 is a target for resuming the autonomous driving in the remote autonomous driving mode, for each of the plurality of mobile bodies 20 that have been abnormally stopped.

[0073] The control value preparation unit 125 generates a control instruction for remotely controlling the mobile body 20 set to the remote automatic driving mode. The control value preparation unit 125 is also referred to as a control unit. The control value preparation unit 125 uses the mobile body sensor information, the mobile body state information, the position and orientation of the mobile body 20 calculated by the calculation unit 122, etc. to generate a control instruction for the mobile body 20, and transmits the control instruction to the mobile body 20. For example, the control instruction is generated as a control value representing the acceleration and the steering angle. In addition, the control value preparation unit 125 generates a control instruction that matches the operator's driving operation on the operator input device 110, and transmits the control instruction to the mobile body 20 to remotely control the mobile body 20 set to the remote manual driving mode.

[0074] Figure 5 1 is a flowchart showing a method in which the remote control device 120 causes the mobile body 20 set to the remote automatic driving mode to travel. The remote control device 120 uses the detection result from the detection device 50 to acquire the position of the mobile body 20 (step S11). Specifically, the calculation unit 122 calculates the position of the mobile body 20 using the external captured image information acquired by the acquisition unit 121 from the detection device 50. The control value preparation unit 125 acquires the information about the position of the mobile body 20 calculated by the calculation unit 122. The information about the position of the mobile body 20 includes X, Y, and Z coordinates in the global coordinate system of the factory.

[0075] The control value preparation unit 125 determines the target position to which the mobile body 20 should go next (step S12). In the present embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system of the plant. The memory 12 of the remote control device 120 pre-stores information about the planned route as the path along which the mobile body 20 should travel. The planned route is represented by a node indicating a departure position, a node indicating a passing point, a node indicating a destination position, and links connecting the nodes. The control value preparation unit 125 uses the position of the mobile body 20 and the planned route to determine the target position to which the mobile body 20 should go next. The control value preparation unit 125 determines a target position on the planned route that is further ahead of the current position of the mobile body 20.

[0076] The control value preparation unit 125 generates a control instruction for causing the moving body 20 to travel toward the determined target position (step S13). In the present embodiment, the control instruction includes the acceleration and steering angle of the moving body 20 as parameters. The control value preparation unit 125 calculates the travel speed of the moving body 20 according to the transition of the position of the moving body 20, and compares the calculated travel speed with the predetermined target speed of the moving body 20. The control value preparation unit 125 determines the acceleration to accelerate the moving body 20 when the travel speed is lower than the target speed, and determines the acceleration to decelerate the moving body 20 when the travel speed is higher than the target speed. The control value preparation unit 125 determines the steering angle to prevent the moving body 20 from deviating from the planned route when the moving body 20 is on the planned route; when the moving body 20 is not on the planned route, or in other words, when the moving body 20 deviates from the planned route, the steering angle is determined to return the moving body 20 to the planned route.

[0077] The control value preparation unit 125 transmits the generated control command to the moving body 20 (step S14). The control value preparation unit 125 repeatedly acquires the position of the moving body 20, determines the target position, generates the control command, and transmits the control command at a predetermined cycle.

[0078] The mobile body control device 290 installed on the mobile body 20 receives the control command from the remote control device 120 (step S15), and uses the received control command to control the driving device 220, the steering device 230, and the braking device 240 so that the mobile body 20 travels according to the acceleration and steering angle indicated by the control command (step S16). The mobile body control device 290 receives the control command at a predetermined cycle and performs control related to the travel of the mobile body 20. By moving the mobile body 20 through remote control, the mobile body 20 can be moved without using a transportation facility such as a crane or a conveyor.

[0079] A configuration unique to the present embodiment will be described below. In the present embodiment, the control value preparation unit 125, upon receiving a notification from the determination unit 124 that the switching of the driving mode has not been made for the moving body 20 that has been subjected to an abnormal stop, performs control for resuming the automatic driving of the moving body 20 while maintaining the remote automatic driving mode. The control for resuming the automatic driving of the moving body 20 includes not only resuming the automatic driving but also preparing for resuming the automatic driving.

[0080] In addition, the control value preparation unit 125 performs control for switching the driving mode of the mobile body 20 to the remote manual driving mode and starting the driving of the mobile body 20 in the remote manual driving mode, when receiving a notification that the switching to the remote manual driving mode is made for the mobile body 20 that has been abnormally stopped from the determination unit 124. The control for starting the driving of the mobile body 20 in the remote manual driving mode includes not only starting the driving in the remote manual driving mode but also preparing for starting the driving in the remote manual driving mode.

[0081] In addition, control value preparation unit 125 performs control for switching mobile body 20 from remote automatic driving mode to attended driving mode control when receiving notification from determination unit 124 that switching to attended driving mode has been made for mobile body 20 that has been abnormally stopped. A2. First condition

[0082] At least one of the conditions listed below may be adopted as the first condition. Two or more of the conditions listed below may be selected as the first condition.

[0083] Condition 1-1 The distance from the current position of the moving body 20 to the position where the downstream process is performed is equal to or smaller than a predetermined distance. Condition 1-2 The distance from the current position of the mobile body 20 to the position where the upstream process is performed is equal to or smaller than a predetermined distance. Conditions 1-3 The time from the current time until the expected shipping date and time set for the mobile body 20 is equal to or less than the predetermined time. Conditions 1-4 The model of the mobile body 20 corresponds to the model planned to be produced preferentially. Conditions 1-5 The current position of the mobile object 20 is outdoors. Conditions 1-6 The current position of the moving object 20 is indoors. Conditions 1-7 There are no stationary objects, including people and other moving objects, on the expected travel route of the moving object 20. A3. Conditions that can be adopted as the first condition Condition 1-1

[0084] For example, suppose that an abnormal state occurs in two moving bodies 20, one of which is located closer to the position where the downstream process is performed than the other moving body 20. In this case, when the moving body 20 located closer to the position where the downstream process is performed is preferentially moved to the position where the downstream process is performed, one of the two moving bodies 20 can reach the position where the downstream process is performed earlier than when the moving body 20 located farther from the position where the downstream process is performed is preferentially moved to the position where the downstream process is performed. Therefore, the occurrence of idle time in the downstream process can be effectively reduced.

[0085] The remote control device 120 determines whether the distance from the current position of the mobile body 20 to the position where the downstream process is executed is equal to or less than a distance predetermined using the mobile body sensor information and the mobile body state information acquired by the mobile body 20 and the information about the position where the downstream process is executed. The memory 12 stores therein information about the distance to the position where the downstream process is executed and information about the predetermined distance in advance.

[0086] The remote control device 120 can easily determine the mobile body 20 that should be processed first among the mobile bodies 20 in abnormal states according to whether the condition 1-1 is satisfied. In addition, by giving priority to abnormal states of the mobile bodies 20 whose distance from the current position to the position where the downstream process is executed is equal to or less than a predetermined distance, the occurrence of idle time in the downstream process is reduced. Condition 1-2

[0087] For example, it is assumed that the moving body 20 experiences an abnormal stop while executing an upstream process, or after having just executed an upstream process and then within a predetermined range from the position where the upstream process is executed. In this case, for example, other moving bodies 20 cannot enter the position where the upstream process is executed, resulting in a stagnation of the flow of the upstream process. When, among the moving bodies 20 in an abnormal state, a moving body 20 located closer to the position where the upstream process is executed is moved in priority to other moving bodies 20, the stagnation of the flow of the upstream process can be effectively reduced.

[0088] The remote control device 120 determines whether the distance from the current position of the mobile body 20 to the position where the upstream process is performed is equal to or less than a distance predetermined using the mobile body sensor information and the mobile body state information acquired by the mobile body 20 and the information about the position where the upstream process is performed. The memory 12 stores therein information about the distance to the position where the upstream process is performed and information about the predetermined distance in advance.

[0089] The remote control device 120 can easily determine the mobile body 20 that should be processed first among the mobile bodies 20 in abnormal states according to whether the conditions 1-2 are satisfied. In addition, by giving priority to the abnormal state of the mobile body 20 whose distance from the current position to the position where the upstream process is executed is equal to or less than a predetermined distance, the stagnation of the flow of the upstream process is reduced. Conditions 1-3

[0090] Among the moving bodies 20 in the abnormal state, it is desirable that the moving body 20 whose time is close to the expected delivery date should be moved with priority over the other moving bodies 20 so as to meet the expected delivery date.

[0091] The remote control device 120 determines whether the time before the expected shipping date is equal to or less than the predetermined time based on the current time and the expected shipping date set for the mobile body 20. It is assumed that the remote control device 120 can obtain information about the current time from a Network Time Protocol (NTP) server. The memory 12 pre-stores information about the expected shipping date set for the mobile body 20 and information about the predetermined time.

[0092] The remote control device 120 can easily determine the mobile body 20 to be processed first among the mobile bodies 20 in abnormal states according to whether conditions 1 to 3 are satisfied. Furthermore, by prioritizing the abnormal states of the mobile bodies 20 close to the expected shipping date, the occurrence of shipping delays is suppressed. Conditions 1-4

[0093] Occasionally, it is planned to give priority to the production of the mobile body 20 corresponding to a specific model. For example, a vehicle having a special specification may be selected as the mobile body 20 corresponding to the specific model. Alternatively, a vehicle of a popular model may be selected as the mobile body 20 of the specific model. It is expected that the mobile body 20 corresponding to the specific model should be shipped more immediately than the mobile body 20 not corresponding to the specific model.

[0094] The remote control device 120 determines whether the model of the mobile body 20 corresponds to the model planned to be produced preferentially based on the model of the mobile body 20. Information about a specific model planned to be produced preferentially is pre-stored in the memory 12. Information about the model of each mobile body 20 is also pre-stored in the memory 12.

[0095] The remote control device 120 can easily determine the mobile body 20 to be processed first among the mobile bodies 20 in abnormal states according to whether conditions 1 to 4 are satisfied. In addition, by processing the abnormal states of the mobile bodies 20 corresponding to specific models first, the occurrence of production plan delays is suppressed. Conditions 1-5

[0096] In order to prevent the moving bodies 20 from being exposed to the weather, it is desirable that the moving bodies 20 located outdoors among the moving bodies 20 in an abnormal state should be moved immediately.

[0097] The remote control device 120 uses the mobile body sensor information and the mobile body state information, the external captured image information and the factory map information to determine whether the mobile body 20 is currently located outdoors. The factory map information is pre-stored in the memory 12. It can be determined that the conditions 1-5 are satisfied when the mobile body 20 is located within a predetermined range outdoors, rather than being located in the entire outdoors of the site of the factory.

[0098] The remote control device 120 can easily determine the mobile body 20 that should be processed first among the mobile bodies 20 in abnormal states according to whether conditions 1 to 5 are satisfied. In addition, by processing the abnormal state of the mobile body 20 located outdoors first, the mobile body 20 can be prevented from being exposed to the weather for a long time. Conditions 1-6

[0099] When the moving body 20 is subjected to abnormal stop in the limited indoor space, the movement of other moving bodies 20 may be hindered. As a result, the flow of the movement of the moving body 20 may be stagnant.

[0100] The remote control device 120 uses the mobile body sensor information and the mobile body state information, the external captured image information and the factory map information to determine whether the mobile body 20 is currently located indoors. It can be determined that conditions 1-6 are satisfied when the mobile body 20 is located within a predetermined range indoors, rather than in the entire indoors of the site of the factory.

[0101] The remote control device 120 can easily determine the mobile body 20 to be processed first among the mobile bodies 20 in abnormal states according to whether conditions 1 to 6 are satisfied. In addition, by processing the abnormal state of the mobile body 20 located indoors first, the movement of the mobile body 20 can be prevented from being stagnant. Conditions 1-7

[0102] When an object is stationary on the expected travel route of the moving body 20, the moving body 20 needs to avoid collision with the object, and it is not expected to start the travel of the moving body 20. The state of the object being stationary includes not only the state in which the object is completely stationary, but also the state in which the object can be considered to be substantially stationary. The object includes not only other moving bodies 20, but also people. The situation in which a stationary object exists on the expected travel route of the moving body 20 includes the situation in which another moving body 20 undergoes an abnormal stop on the expected travel route, the situation in which an operator is doing some work on the expected travel route, etc.

[0103] The remote control device 120 determines whether there is an object on the expected travel route of the moving body 20 using information on the expected travel route of the moving body 20 and external captured image information acquired from the detection device 50. The information on the expected travel route of the moving body 20 is stored in the memory 12 in advance.

[0104] The remote control device 120 can easily determine the mobile body 20 that should be processed first among the mobile bodies 20 in an abnormal state according to whether the conditions 1 to 7 are satisfied. When there is no object on the expected travel route, it is easy to autonomously drive the mobile body 20 by remote control. In addition, the mobile body 20 can be moved immediately by giving priority to the mobile body 20 that satisfies the condition that there is no stationary object on the expected travel route. A4. Determination process for switching driving modes

[0105] Figure 6 When the abnormality detection unit 123 detects the occurrence of abnormal conditions of two or more moving bodies 20 based on the external captured image information and the moving body sensor information and moving body internal information of each moving body 20 acquired by the acquisition unit 121, the abnormality detection unit 123 starts Figure 6 The processing shown in . An example in which the conditions 1 to 3 are adopted as the first condition will be described below. Here, an example in which N (N is an integer greater than or equal to two) moving bodies 20 have been subjected to abnormal stoppage is described.

[0106] In step S101, the determination unit 124 determines whether one of the N moving bodies 20 that have been subjected to the abnormal stop satisfies the conditions 1-3 as the first condition. Specifically, the determination unit 124 determines whether the time before the expected shipping date time set for the moving body 20 is equal to or less than a predetermined time. The determination unit 124 determines that the first condition is satisfied when the time before the expected shipping date time set for the moving body 20 is equal to or less than a predetermined time. When it is determined that the first condition is satisfied (step S101: Yes), the determination unit 124 performs the processing in step S102.

[0107] When the time before the expected shipping date set for the mobile body 20 is not equal to or less than the predetermined time (step S101 : No), on the other hand, the determination unit 124 determines that the first condition is not satisfied, and performs the processing in step S103 .

[0108] In step S102 , the determination unit 124 adds information specifying the moving object 20 (ie, the target moving object) determined in step S101 to satisfy the first condition to the list of the target moving objects.

[0109] When the determination for the first condition is not made for all the N moving bodies 20 in the abnormal state in step S103 (step S103: No), the determination unit 124 performs the processing in step S101 again. When step S101 is performed again, the determination unit 124 performs the processing in step S101 for one moving body 20 in the abnormal state for which the determination for the first condition has not been made.

[0110] On the other hand, when the determination for the first condition is made for all the N mobile bodies 20 in the abnormal state in step S103 (step S103: Yes), the determination unit 124 performs the processing in step S104. When the determination for the first condition for the N mobile bodies 20 that are already in the abnormal state is completed, the list of target mobile bodies includes information specifying the mobile bodies 20 that satisfy the first condition among the N mobile bodies 20 that are already in the abnormal state. However, when there is no mobile body 20 that satisfies the first condition among the N mobile bodies 20 that are already in the abnormal state, the list of target mobile bodies is empty.

[0111] In step S104, the determination unit 124 outputs a notification that the automatic driving in the remote automatic control mode is to be resumed together with the list of target moving bodies to the control value preparation unit 125. For example, when two moving bodies 20 among the N moving bodies 20 that have been subjected to the abnormal stop state satisfy the first condition, the automatic driving of the two moving bodies 20 satisfying the first condition is resumed while the remote automatic control mode is maintained.

[0112] In step S105, the control value preparation unit 125 transmits a control instruction for resuming the automatic driving in the remote automatic control mode to the mobile body 20 which is a target mobile body included in the list. However, when the list of target mobile bodies is empty, there is no mobile body 20 to resume the automatic driving in the remote automatic control mode, and therefore the control instruction for resuming the automatic driving in the remote automatic control mode is not transmitted. Then, Figure 6 The processing shown in is completed.

[0113] Thereafter, the control value preparation unit 125 remotely controls the mobile body 20 as the target mobile body by transmitting the control command to the mobile body 20 as the target mobile body.

[0114] The moving object 20 (non-target moving object) determined as not satisfying the first condition in step S101 is not a target to be processed preferentially. Figure 6After starting the remote control of the mobile body 20 as the target mobile body by the processing of step S105 in, the following processing is performed. The determination unit 124 determines to switch to the remote manual driving mode for the non-target mobile body. The determination unit 124 transmits the notification of switching to the remote manual driving mode and the information specifying the mobile body 20 as the non-target mobile body to the control value preparation unit 125. When receiving the notification of switching from the remote automatic driving mode to the remote manual driving mode, the control value preparation unit 125 outputs the notification of switching to the remote manual driving mode to the display device 115. Thereafter, each time the control value preparation unit 125 receives an input from the operator input device 110, it transmits a control instruction matching the driving operation of the operator to the mobile body 20. In this way, remote control in the remote manual driving mode is performed for the mobile body 20 (non-target mobile body) determined not to satisfy the first condition.

[0115] When only one of the moving bodies 20 is in an abnormal state, the determination unit 124 may adopt a predetermined processing method for the moving body 20 in the abnormal state instead of the method discussed above. Examples of the predetermined processing method include the operator WO driving and collecting the moving body 20.

[0116] As described above, the remote control device 120 can easily determine the target to be processed with priority among the multiple mobile bodies that have been in an abnormal state according to whether the predetermined first condition is satisfied. When the multiple mobile bodies 20 that are traveling by automatic driving stop due to a certain failure, it is necessary to use the available range of physical resources and human resources to handle the stopped vehicles. With the configuration according to the present embodiment, it is easy to determine how to handle the mobile bodies that have been in an abnormal state in what priority order. In addition, for the mobile body designated as the target to be processed with priority, the automatic driving can be immediately restored by remote control. B. Second Embodiment B1 Remote Control System Configuration

[0117] The remote control system 1 according to the second embodiment is used to move the mobile body 20 by remote control in a factory producing the mobile body 20, as in the first embodiment (see Figure 1 ). The moving body 20 includes similar components to those according to the first embodiment (see Figure 2 ). The hardware configuration of the remote operation device 100 is similar to that according to the first embodiment (see Figure 3 ). The operator input device 110 includes similar components to the first embodiment (see Figure 3 and Figure 4 ).

[0118] As in the first embodiment, the remote control device 120 is a device for remotely controlling the travel of the mobile body 20. The respective functions of the acquisition unit 121, the calculation unit 122, the abnormality detection unit 123, the determination unit 124, and the control value preparation unit 125 are realized by the CPU 11 of the remote control device 120 executing the program stored in the memory 12. Figure 3 The functions of the acquisition unit 121, the calculation unit 122, the abnormality detection unit 123, and the control value preparation unit 125 shown in FIG. 1 are the same as those according to the first embodiment. In addition, the remote control device 120 controls the travel of the mobile body 20 set to the remote automatic driving mode, as in the first embodiment (see Figure 5 ) like that.

[0119] The following will mainly describe the components different from those according to the first embodiment. The components similar to those according to the first embodiment will not be repeated. In the present embodiment, when a plurality of mobile bodies 20 have been abnormally stopped, the determination unit 124 determines whether each mobile body 20 is a target to be processed with priority. The mobile body 20 as the target is processed with priority by starting driving in the remote manual driving mode. In the present embodiment, the target for which the automatic driving in the remote manual driving mode is started with priority is referred to as the target mobile body. The determination unit 124 determines the mobile body 20 that satisfies a predetermined second condition as the target mobile body. The second condition is a condition determined in relation to the priority of starting the automatic driving in the remote manual driving mode. The target mobile body, i.e., the mobile body 20 for which the automatic driving in the remote manual driving mode is determined to be started, is remotely controlled after being switched to the remote manual driving mode. The second condition will be discussed in detail later.

[0120] On the other hand, the determining unit 124 determines the mobile body 20 that does not satisfy the second condition as a mobile body that is not a mobile body for which the automatic driving in the remote manual driving mode is started with priority (hereinafter referred to as a non-target mobile body). For the non-target mobile body, the remote automatic driving mode is switched to a different driving mode. For example, the mobile body 20 as a non-target mobile body is switched to the manned driving mode and driven and collected by the operator WO.

[0121] The determination unit 124 transmits to the control value preparation unit 125 a notification of switching of the driving mode according to a determination result as to whether the mobile body 20 is a target for starting autonomous driving in the remote autonomous driving mode, for each of the plurality of mobile bodies 20 that have been abnormally stopped. B2. Second Condition

[0122] At least one of the conditions listed below may be adopted as the second condition. Two or more of the conditions listed below may be selected as the second condition.

[0123] Condition 2-1 The distance from the current position of the moving body 20 to the position where the downstream process is performed is equal to or smaller than a predetermined distance. Condition 2-2 The distance from the current position of the mobile body 20 to the position where the upstream process is performed is equal to or smaller than a predetermined distance. Conditions 2-3 The time from the current time until the expected shipping date and time set for the mobile body 20 is equal to or less than the predetermined time. Conditions 2-4 The model of the mobile body 20 corresponds to the model planned to be produced preferentially. Conditions 2-5 The current position of the mobile object 20 is outdoors. Conditions 2-6 The current position of the moving object 20 is indoors. Conditions 2-7 The mobile body 20 can provide the operator with visual information required for remote operation. Condition 2-8 The moving body 20 is located on a path along which the operator causes the moving body 20 to travel based on remote operation. Condition 2-9 When the travel of the moving body 20 starts based on the remote operation performed by the operator, there is no object that hinders the travel around the moving body 20 .

[0124] B3. Conditions that can be adopted as the second condition The reasons for setting conditions 2-1 to 2-6, the method of determining whether these conditions are satisfied, and the effects when these conditions are adopted are the same as those according to conditions 1-1 to 1-6 in the first embodiment, and will not be described in detail. Conditions 2-7

[0125] When an operator located at a remote location remotely operates the mobile body 20, it is desirable that the mobile body 20 should be able to provide the internal captured image information acquired by the onboard camera to the remote control device 120. The operator is occasionally able to remotely operate the mobile body 20 using only the external captured image information acquired by the detection device 50. However, the operator can remotely operate the mobile body 20 by using the external captured image information acquired from the detection device 50 and the internal captured image information acquired by the onboard camera of the mobile body 20, thereby driving the mobile body 20 more smoothly.

[0126] The remote control device 120 uses the mobile body sensor information and the mobile body state information to determine whether the mobile body 20 can provide the operator with the visual information required for the remote operation. For example, when the remote control device 120 does not receive the internal shooting image information obtained by the vehicle-mounted camera for a period of time, it determines that the mobile body 20 cannot provide the operator with the visual information required for the remote operation.

[0127] The remote control device 120 can easily determine the mobile body 20 that should be processed preferentially among the mobile bodies 20 in the abnormal state according to whether the conditions 2-7 are satisfied. When the visual information required for the remote operation is provided from the mobile body 20, in addition, the operator can remotely operate the mobile body 20 while checking the internal captured image information displayed on the display device 115. Therefore, the operator can smoothly drive the mobile body 20 based on the remote operation. In addition, the mobile body 20 can move immediately. Condition 2-8

[0128] It is considered that the mobile body 20 can be driven based on remote operation more easily when the mobile body 20 is caused to travel on a path along which the operator has caused the mobile body 20 to move than when the mobile body 20 is caused to travel on a path along which the operator has not caused the mobile body 20 to move.

[0129] The remote control device 120 uses the mobile body sensor information and the mobile body state information and the external captured image information to determine the current position of the mobile body 20. Further, the remote control device 120 uses the historical information about the operator's remote operation to determine whether the mobile body 20 is located on the path along which the operator causes the mobile body 20 to travel based on the remote operation. The historical information about the operator's remote operation is pre-stored in the memory 12.

[0130] The remote control device 120 can easily determine the mobile body 20 that should be processed preferentially among the mobile bodies 20 in the abnormal state according to whether the conditions 2-8 are satisfied. When the mobile body 20 is located on the path along which the operator causes the mobile body to travel based on the remote operation, the operator can smoothly drive the mobile body 20 by remote control. In addition, the mobile body 20 can move immediately. Condition 2-9

[0131] There are cases where objects that hinder the travel exist around the moving body 20, including a case where another moving body 20 undergoes an abnormal stop in front of the moving body 20, a case where an operator is doing some work in front of the moving body 20, etc. In such a case, it is necessary to avoid collision with the object, and it is not desirable to start the travel of the moving body 20. Here, the object includes not only another moving body 20 but also a person.

[0132] The remote control device 120 uses the moving body sensor information and the moving body state information as well as the external captured image information to determine whether there is an object that hinders the travel around the moving body 20 .

[0133] The remote control device 120 can easily determine the mobile body 20 that should be processed first among the mobile bodies 20 in an abnormal state according to whether the conditions 2-9 are satisfied. When there is no object that hinders the travel around the mobile body 20, the operator can smoothly drive the mobile body 20 based on the remote operation. In addition, the mobile body 20 can move immediately. B4. Determination Processing for Switching Driving Modes

[0134] Figure 7 When the abnormality detection unit 123 detects the occurrence of abnormal conditions of two or more moving bodies 20 based on the external captured image information and the moving body sensor information and moving body internal information of each moving body 20 acquired by the acquisition unit 121, the abnormality detection unit 123 starts Figure 7 The processing shown in . An example in which condition 2-7 is adopted as the second condition will be described below. Here, an example in which N (N is an integer greater than or equal to two) moving bodies 20 have been subjected to abnormal stoppage is described.

[0135] In step S201, the determination unit 124 determines whether one of the N moving bodies 20 that have been subjected to abnormal stop satisfies the condition 2-7 as the second condition. Specifically, the determination unit 124 determines whether the interior captured image information acquired by the vehicle-mounted camera has been successfully received. When the interior captured image information is successfully received and it is determined that the second condition is satisfied (step S201: Yes), the determination unit 124 performs the processing in step S202.

[0136] On the other hand, when the interior captured image information acquired by the vehicle-mounted camera has not been received for a while, the determination unit 124 determines that the second condition is not satisfied (step S201 : No), and performs the processing in step S203 .

[0137] In step S202 , the determination unit 124 adds information specifying the moving object 20 (ie, the target moving object) determined in step S201 to satisfy the second condition to the list of the target moving objects.

[0138] When the determination for the second condition is not made for all the N moving bodies 20 in the abnormal state in step S203 (step S203: No), the determination unit 124 performs the processing in step S201 again. When step S201 is performed again, the determination unit 124 performs the processing in step S201 for one moving body 20 in the N moving bodies 20 in the abnormal state for which the determination for the second condition has not been made.

[0139] On the other hand, when the determination for the second condition is made for all the N mobile bodies 20 in the abnormal state in step S203 (step S203: Yes), the determination unit 124 performs the processing in step S204. When the determination for the second condition for the N mobile bodies 20 that have been in the abnormal state is completed, the list of target mobile bodies includes information specifying the mobile bodies 20 that satisfy the second condition among the N mobile bodies 20 that have been in the abnormal state. However, when there is no mobile body 20 that satisfies the second condition among the N mobile bodies 20 that have been in the abnormal state, the list of target mobile bodies is empty.

[0140] In step S204, the determination unit 124 outputs a notification that driving in the remote manual control mode is to be started together with the list of target moving bodies to the control value preparation unit 125. For example, when one moving body 20 among the N moving bodies 20 that have been subjected to the abnormal stop state satisfies the second condition, the one moving body satisfying the second condition is switched to the remote manual control mode.

[0141] In step S205, the control value preparation unit 125 outputs a notification that the switch to the remote manual driving mode is made to the display device 115. In addition, the control value preparation unit 125 transmits a control instruction to start driving in the remote manual control mode to the mobile body 20 as the target mobile body included in the list. However, when the list of target mobile bodies is empty, there is no mobile body 20 to start driving in the remote manual control mode, and therefore the control instruction to start driving in the remote manual control mode is not transmitted. Then, Figure 7 The processing shown in is completed.

[0142] Thereafter, the control value preparation unit 125 transmits a control instruction matching the operator's driving operation to the moving body 20 each time receiving an input from the operator input device 110. In this way, remote control of the moving body 20 in the remote manual driving mode is performed.

[0143] The moving object 20 (non-target moving object) determined as not satisfying the second condition in step S201 is not a target to be processed preferentially. Figure 7 After starting remote control of the mobile object 20 as the target mobile object by the process of step S205 in the above, the following process is performed. The determination unit 124 determines to switch to the attended driving mode for the non-target mobile object. The determination unit 124 sends a notification for instructing the collection of the mobile object 20 together with the information specifying the mobile object 20 as the non-target mobile object to at least one worker WO located at a site of the factory (see Figure 1). The determination unit 124 sends a notification to the worker WO via the terminal device T1 of the operator WO and the terminal device T2 installed at the standby position OF. The standby position OF is a position where the operator WO is on standby or resting. In this way, for the moving body 20 (non-target moving body) determined not to satisfy the second condition, driving in the manned driving mode is performed.

[0144] As described above, the remote control device 120 can easily determine the target to be processed with priority among the multiple mobile bodies that are already in an abnormal state, depending on whether the predetermined second condition is satisfied. When the multiple mobile bodies 20 that are traveling by automatic driving stop due to a certain failure, it is necessary to use the available range of physical resources and human resources to handle the stopped vehicles. With the configuration according to the present embodiment, it is easy to determine how to handle the mobile bodies that are already in an abnormal state in what priority order. In addition, for the mobile body designated as the target to be processed with priority, driving can be started immediately based on the remote operation of the operator. C. Third Embodiment C1. Configuration of remote control system

[0145] The remote control system 1 according to the second embodiment is used to move the mobile body 20 by remote control in a factory producing the mobile body 20, as in the first embodiment (see Figure 1 ). The moving body 20 includes similar components to those according to the first embodiment (see Figure 2 ). The hardware configuration of the remote operation device 100 is similar to that according to the first embodiment (see Figure 3 ). The operator input device 110 includes similar components to the first embodiment (see Figure 3 and Figure 4 ).

[0146] As in the first embodiment, the remote control device 120 is a device for remotely controlling the travel of the mobile body 20. The respective functions of the acquisition unit 121, the calculation unit 122, the abnormality detection unit 123, the determination unit 124, and the control value preparation unit 125 are realized by the CPU 11 of the remote control device 120 executing the program stored in the memory 12. Figure 3 The functions of the acquisition unit 121, the calculation unit 122, the abnormality detection unit 123, and the control value preparation unit 125 shown in FIG. 1 are the same as those according to the first embodiment. In addition, the remote control device 120 controls the travel of the mobile body 20 set to the remote automatic driving mode, as in the first embodiment (see Figure 5 ) like that.

[0147] The following will mainly describe the components different from those according to the first embodiment. The components similar to those according to the first embodiment will not be repeated. In the present embodiment, when a plurality of mobile bodies 20 are in an abnormal state, the determination unit 124 determines whether each mobile body 20 is a target to be processed with priority. The mobile body 20 as the target is processed with priority by the operator WO driving and collecting the mobile body 20 in the manned driving mode. In the present embodiment, the target to be driven and collected with priority by the operator WO in the manned driving mode is called the target mobile body. The determination unit 124 determines the mobile body 20 that satisfies the third condition determined in advance as the target mobile body. The third condition is a condition determined in relation to the priority of the operator WO collecting the mobile body 20 in the manned driving mode. The target mobile body, that is, the mobile body 20 determined to be collected by the operator WO, is driven and collected by the operator WO after switching to the manned driving mode. The third condition will be discussed in detail later.

[0148] On the other hand, the determination unit 124 determines the mobile body 20 that does not satisfy the third condition as a mobile body that is not driven and collected by the operator in the manned driving mode (hereinafter referred to as a non-target mobile body). For the non-target mobile body, the remote automatic driving mode is switched to a different driving mode. For example, the mobile body 20 as the non-target mobile body is switched to the remote manual driving mode to start being driven based on the driving operation of the operator.

[0149] The determination unit 124 transmits, to the control value preparation unit 125, a notification of switching of the driving mode according to the determination result as to whether the mobile body 20 is a target to be driven and collected by the operator WO in the manned driving mode, for each of the plurality of mobile bodies 20 that have been subjected to the abnormal stop.

[0150] C2. The third condition At least one of the conditions listed below may be adopted as the third condition. Two or more of the conditions listed below may be selected as the third condition.

[0151] Condition 3-1 The distance from the current position of the moving body 20 to the position where the downstream process is performed is equal to or smaller than a predetermined distance. Condition 3-2 The distance from the current position of the mobile body 20 to the position where the upstream process is performed is equal to or smaller than a predetermined distance. Condition 3-3 The time from the current time until the expected shipping date and time set for the mobile body 20 is equal to or less than the predetermined time. Conditions 3-4 The model of the mobile body 20 corresponds to the model planned to be produced preferentially. Conditions 3-5 The current position of the mobile object 20 is outdoors. Conditions 3-6 The current position of the moving object 20 is indoors. Conditions 3-7 When the moving body 20 starts traveling based on the driving of the operator riding on the moving body 20 , there is no object that hinders the traveling around the moving body 20 .

[0152] C3. Conditions that can be adopted as the third condition The reasons for setting conditions 3-1 to 3-6, the method of determining whether these conditions are satisfied, and the effects when these conditions are adopted are the same as those according to conditions 1-1 to 1-6 in the first embodiment, and will not be described in detail. Conditions 3-7

[0153] Examples of the case where there are objects that hinder the travel around the moving body 20 include a case where another moving body 20 undergoes an abnormal stop in front of the moving body 20, and a case where an operator is doing some work in front of the moving body 20. In this case, it is necessary to avoid collision with the object, and it is not desirable to start the travel of the moving body 20. Here, the object includes not only another moving body 20 but also a person.

[0154] The remote control device 120 uses the moving body sensor information and the moving body state information as well as the external captured image information to determine whether there is an object that hinders the travel around the moving body 20 .

[0155] The remote control device 120 can easily determine the moving body 20 that should be processed first among the moving bodies 20 in an abnormal state according to whether the conditions 3 to 7 are satisfied. When there is no object that hinders the movement around the moving body 20, the operator can smoothly drive the moving body 20. In addition, the moving body 20 can move immediately. C4. Determination process for switching driving mode

[0156] Figure 8 When the abnormality detection unit 123 detects the occurrence of abnormal conditions of two or more moving bodies 20 based on the external captured image information and the moving body sensor information and moving body internal information of each moving body 20 acquired by the acquisition unit 121, the abnormality detection unit 123 starts Figure 8 The processing shown in . An example in which the condition 3-5 is adopted as the third condition will be described below. Here, an example in which N (N is an integer greater than or equal to two) moving bodies 20 have been subjected to abnormal stoppage is described.

[0157] In step S301, the determination unit 124 determines whether one of the N moving bodies 20 that have been subjected to abnormal stop satisfies the conditions 3-5 as the second condition. Specifically, the determination unit 124 uses the moving body sensor information and the moving body state information and the external captured image information to determine the current position of the moving body 20. Further, the determination unit 124 uses the plant map information to determine whether the moving body 20 is currently located outdoors. When the moving body 20 is currently located outdoors and it is determined that the third condition is satisfied (step S301: Yes), the determination unit 124 performs the processing in step S302.

[0158] On the other hand, when the moving body 20 is currently located indoors, the determination unit 124 determines that the third condition is not satisfied (step S301 : No), and performs the processing in step S303 .

[0159] In step S302 , the determination unit 124 adds information specifying the moving object 20 (ie, the target moving object) determined in S301 to satisfy the first condition to the list of the target moving objects.

[0160] When the determination for the third condition is not made for all the N moving bodies 20 in the abnormal state in step S303 (step S303: No), the determination unit 124 performs the processing in step S301 again. When step S301 is performed again, the determination unit 124 performs the processing in step S301 for one moving body 20 in the abnormal state for which the determination for the third condition has not been made.

[0161] On the other hand, when the determination for the third condition is made for all the N mobile bodies 20 in the abnormal state in step S303 (step S303: Yes), the determination unit 124 performs the processing in step S304. When the determination for the third condition for the N mobile bodies 20 that have been in the abnormal state is completed, the list of target mobile bodies includes information specifying the mobile bodies 20 that satisfy the third condition among the N mobile bodies 20 that have been in the abnormal state. However, when there is no mobile body 20 that satisfies the third condition among the N mobile bodies 20 that have been in the abnormal state, the list of target mobile bodies is empty.

[0162] In step S304, the determination unit 124 outputs a notification that driving in the manned driving mode is to be started together with the list of target moving bodies to the control value preparation unit 125. For example, when two moving bodies 20 among the N moving bodies 20 that have been abnormally stopped satisfy the third condition, the two moving bodies satisfying the third condition are switched to the manned driving mode to be driven and collected by the operator WO.

[0163] In step S305, the determination unit 124 sends a notification for instructing to collect the moving body 20 together with information specifying the moving body 20 as the target moving body to at least one worker WO located in a station of the factory (see Figure 1 1). The determination unit 124 sends a notification to the worker WO via the terminal device T1 of the operator WO and the terminal device T2 installed at the standby position OF. The standby position OF is the position where the operator WO is on standby or resting. Then, Figure 8 The processing shown in is completed.

[0164] Thereafter, the operator WO moves to a position where the mobile body 20 as the target mobile body is stopped, and drives the mobile body 20. In this way, the mobile body 20 as the target mobile body is collected.

[0165] The moving body 20 (non-target moving body) determined not to satisfy the third condition in step S301 is not a target to be processed with priority. After an instruction to collect the moving body 20 as the target moving body is given, the following processing is performed: the determination unit 124 determines to switch to the manned driving mode for the non-target moving body. The determination unit 124 sends a notification to instruct to collect the moving body 20 together with the information specifying the moving body 20 as the non-target moving body to at least one worker WO located in a station of the factory (see Figure 1 ). The determination unit 124 sends a notification to the worker WO via the terminal device T1 of the operator WO and the terminal device T2 installed at the standby position OF. The standby position OF is a position where the operator WO is on standby or resting. In this way, for the moving body 20 (non-target moving body) determined not to satisfy the second condition, driving in the manned driving mode is performed.

[0166] As described above, the remote control device 120 can easily determine the target to be processed with priority among the multiple mobile bodies that have been in an abnormal state, depending on whether the predetermined third condition is satisfied. When the multiple mobile bodies 20 that are traveling by automatic driving stop due to a certain failure, it is necessary to use the available range of physical resources and human resources to handle the stopped vehicles. With the configuration according to the present embodiment, it is easy to determine how to handle the mobile bodies that have been in an abnormal state in what priority order. In addition, the work of the operator WO driving and collecting the mobile bodies 20 designated as the targets to be processed with priority can be started immediately. D. Fourth Embodiment

[0167] In relation to the first to third embodiments, the mobile body 20 is described as traveling in one of the manned driving mode, the remote manual driving mode, and the remote automatic driving mode. In addition to these driving modes, the mobile body 20 may also be able to travel in the autonomous driving mode. The autonomous driving mode is a driving mode in which the driving of the mobile body 20 is controlled by the mobile body 20 through autonomous control. The driving of the mobile body 20 in the autonomous driving mode is automatic driving.

[0168] The remote control system 1 according to the fourth embodiment is used to move the mobile body 20 by remote control in a factory producing the mobile body 20, as in the first embodiment (see Figure 1 The hardware configuration of the remote operation device 100 is similar to that according to the first embodiment (see Figure 3 ). The operator input device 110 includes similar components to those according to the first embodiment (see Figure 3 and Figure 4 ).

[0169] As in the first embodiment, the remote control device 120 is a device for remotely operating the travel of the mobile body 20 set to the remote automatic driving mode. The respective functions of the acquisition unit 121, the calculation unit 122, the abnormality detection unit 123, the determination unit 124, and the control value preparation unit 125 are implemented by the CPU 11 of the remote control device 120 executing the program stored in the memory 12. Figure 3 The functions of the acquisition unit 121, the calculation unit 122, the abnormality detection unit 123, and the control value preparation unit 125 shown in FIG. 1 are the same as those according to the first embodiment. In addition, the remote control device 120 controls the travel of the mobile body 20 set to the remote automatic driving mode, as in the first embodiment (see Figure 5 ) like that.

[0170] Fig. 9 2 is a block diagram showing the functional configuration of a mobile body 20 capable of traveling in an autonomous driving mode. Figure 2 ) is that the mobile body control device 290c provided in the mobile body 20 includes a function as a control device. The following will mainly describe the components different from those according to the first embodiment. The components similar to those according to the first embodiment will not be repeated.

[0171] The memory 292 stores programs for realizing the functions of the acquisition unit 121, the calculation unit 122, and the control value preparation unit 125 provided in the remote control device 120, and the functions of the CPU 291 indicated in the first embodiment. The CPU 291c operates the acquisition unit 221, the calculation unit 222, and the control value preparation unit 225 as corresponding to the functions provided in the remote control device 120 according to the first embodiment by executing the programs stored in the memory 292. The memory 292 also stores information on the planned route of the mobile body 20. The planned route is represented by a node indicating a departure position, a node indicating a passing point, a node indicating a destination position, and links connecting between the nodes.

[0172] The acquisition unit 221 acquires the moving body sensor information and the moving body state information. The acquisition unit 221 also acquires the external captured image information from the detection device 50.

[0173] The calculation unit 222 calculates the position and orientation of the mobile body 20. For example, the external captured image information can be used to estimate the position and orientation of the mobile body 20. The position of the mobile body 20 can be acquired by, for example, calculating the coordinates of the measurement point of the mobile body 20 in the image coordinate system using the outer shape of the mobile body 20 detected from the external captured image information, and converting the calculated coordinates into coordinates in the global coordinate system.

[0174] The orientation of the mobile body 20 may be estimated based on the orientation of a vector related to the motion of the mobile body 20 calculated using, for example, an optical flow method. Alternatively, the orientation of the mobile body 20 may be calculated using a detection value from a yaw rate sensor mounted on the mobile body 20. For example, the outer shape of the moving object 20 can be detected by inputting the captured image into a detection model using artificial intelligence.

[0175] The control value preparation unit 225 determines the acceleration and steering angle of the moving body 20 using the position and orientation of the moving body 20 calculated by the calculation unit 222 , and causes the moving body 20 to travel at the determined acceleration and steering angle.

[0176] Fig.10 2 is a flowchart showing a method of driving the mobile body 20 set to the autonomous driving mode. The calculation unit 222 uses the detection result from the detection device 50 to acquire the position of the mobile body 20 (step S21). Specifically, the calculation unit 222 uses the external captured image information acquired by the acquisition unit 221 from the detection device 50 to calculate the position of the mobile body 20. The information about the position of the mobile body 20 includes X, Y, and Z coordinates in the global coordinate system of the plant.

[0177] The control value preparation unit 225 determines the target position to which the moving body 20 should go next (step S22). In the present embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system of the plant. The control value preparation unit 225 uses the information on the position of the moving body 20 calculated by the calculation unit 222 and the planned route to determine the target position to which the moving body 20 should go next. The control value preparation unit 225 determines the target position on the planned route that is further ahead of the current position of the moving body 20.

[0178] The control value preparation unit 225 generates a control instruction for causing the moving body 20 to travel toward the determined target position (step S23). In the present embodiment, the control instruction includes the acceleration and steering angle of the moving body 20 as parameters. The control value preparation unit 225 calculates the travel speed of the moving body 20 according to the transition of the position of the moving body 20, and compares the calculated travel speed with the predetermined target speed of the moving body 20. The control value preparation unit 225 determines the acceleration to accelerate the moving body 20 when the travel speed is lower than the target speed, and determines the acceleration to decelerate the moving body 20 when the travel speed is higher than the target speed. The control value preparation unit 225 determines the steering angle to prevent the moving body 20 from deviating from the planned route when the moving body 20 is on the planned route; when the moving body 20 is not on the planned route, or in other words, when the moving body 20 deviates from the planned route, the steering angle is determined to return the moving body 20 to the planned route.

[0179] The control value preparation unit 225 controls the driving device 220, the steering device 230, and the braking device 240 according to the acceleration and the steering angle indicated by the generated control command, so as to make the moving body 20 travel (step S24). The control value preparation unit 225 repeatedly acquires the position of the moving body 20, determines the target position, generates the control command, and controls the travel in a predetermined cycle.

[0180] Since the mobile body 20 moves by autonomous control, the mobile body 20 can move without being controlled by the remote control device 120 .

[0181] As in the first embodiment, the abnormality detection unit 123 of the remote control device 120 detects the occurrence of an abnormal state of the mobile body 20. In the present embodiment, when the mobile body 20 set to the remote automatic driving mode or the autonomous driving mode is traveling, the abnormality detection unit 123 detects the occurrence of an abnormal state of the mobile body 20 using the mobile body sensor information and mobile body state information of the mobile body 20, the external captured image information, the information indicating the communication state between the remote control device 100 and the mobile body 20, and the like. The abnormal state of the mobile body 20 refers to a state in which the mobile body 20 set to the remote automatic driving mode or the autonomous driving mode has been abnormally stopped. The abnormal stop refers to an unintentional stop of the mobile body 20 that is traveling in the remote automatic driving mode or the autonomous driving mode.

[0182] The determination unit 124 determines whether to resume the automatic driving in the remote automatic driving mode or the autonomous driving mode when the moving body 20 set to the remote automatic driving mode or the autonomous driving mode undergoes an abnormal stop. This determination method is similar to that according to the first embodiment (see Figure 6 )same.

[0183] When the moving body 20 set to the remote automatic driving mode or the autonomous driving mode experiences an abnormal stop, one of the following measures is taken. The measures to be taken include: (i) switching the moving body 20 to the remote manual driving mode and remotely operating the driving of the moving body 20 based on the driving operation of the operator, (ii) resuming the automatic driving in the remote automatic driving mode or the autonomous driving mode, and (iii) the operator getting on the moving body 20 and driving the moving body 20.

[0184] Specifically, the determination unit 124 determines the mobile body 20 that satisfies the first condition as the target mobile body. The details of the first condition are the same as those according to the first embodiment. The target mobile body, that is, the mobile body 20 for which the automatic driving in the remote automatic driving mode or the autonomous driving mode is determined to be restored, is controlled to restore the automatic driving while remaining in the remote automatic driving mode. When it is determined that the automatic driving of the mobile body 20 in the remote automatic driving mode or the autonomous driving mode is to be restored, the driving mode is not switched.

[0185] On the other hand, the determination unit 124 determines the mobile body 20 that does not meet the first condition as a mobile body (non-target mobile body) that does not give priority to restoring the automatic driving in the remote automatic driving mode. For the non-target mobile body, the remote automatic driving mode or the autonomous driving mode is switched to a different driving mode. For example, the mobile body 20 as a non-target mobile body is switched to the remote manual driving mode to travel based on the remote operation of the operator. Alternatively, the mobile body 20 as a non-target mobile body is switched to the manned driving mode and is driven and collected by the operator WO.

[0186] The determination unit 124 transmits to the control value preparation unit 125 a notification of switching of the driving mode according to the determination result as to whether to resume the automatic driving in the remote automatic driving mode or the autonomous driving mode for each of the plurality of moving bodies 20 that have been abnormally stopped.

[0187] The control value preparation unit 125 performs control for restoring the automatic driving of the mobile body 20 at the site of the factory while maintaining the remote automatic driving mode or the autonomous driving mode when receiving a notification that the switching to the driving mode is not made from the determination unit 124. On the other hand, the control value preparation unit 125 performs control for switching the mobile body 20 from the remote automatic driving mode to the remote manual driving mode control when receiving a notification that the switching to the remote manual driving mode is made from the determination unit 124. As a result, the automatic driving of the mobile body 20 in the remote manual driving mode is started.

[0188] As described above, the remote control device 120 can easily determine a target to be processed preferentially among a plurality of moving objects that have been in an abnormal state, according to whether the predetermined first condition is satisfied. E. Other Embodiments

[0189] (E1) In the above-mentioned fourth embodiment, when the mobile body 20 set to the remote automatic driving mode or the autonomous driving mode experiences an abnormal stop, it is determined whether to restore the automatic driving in the remote automatic driving mode or the autonomous driving mode, or to switch from the remote automatic driving mode or the autonomous driving mode to other driving modes.

[0190] Alternatively, when a moving body 20 set to the remote automatic driving mode or the autonomous driving mode undergoes an abnormal stop, as in the second embodiment, the determination unit 124 may determine whether each moving body 20 is a target to preferentially start driving in the remote manual driving mode. This determination method is similar to that according to the second embodiment (see Figure 7 )same.

[0191] Specifically, the determination unit 124 determines the mobile body 20 satisfying the second condition as the target mobile body. The details of the second condition are the same as those according to the second embodiment. The target mobile body, that is, the mobile body 20 determined to start driving in the remote manual driving mode, is remotely controlled after being switched to the remote manual driving mode.

[0192] On the other hand, the determining unit 124 determines the mobile body 20 that does not satisfy the second condition as a mobile body (non-target mobile body) for which the automatic driving in the remote manual driving mode is not started preferentially. For the non-target mobile body, the remote automatic driving mode is switched to a different driving mode. For example, the mobile body 20 as the non-target mobile body is switched to the attended driving mode and driven and collected by the operator WO.

[0193] The determination unit 124 transmits, to the control value preparation unit 125 , notification of switching of the driving mode according to the determination result as to whether driving in the remote manual driving mode is to be started, for each of the plurality of moving bodies 20 that have been subjected to the abnormal stop.

[0194] The control value preparation unit 125 performs control for switching the mobile body 20 from the remote automatic driving mode to the remote manual driving mode control when receiving the notification of the switching to the remote manual driving mode from the determination unit 124. As a result, the driving of the mobile body 20 in the remote manual driving mode is started.

[0195] As described above, the remote control device 120 can easily determine a target to be processed preferentially among a plurality of moving objects that have been in an abnormal state, according to whether the predetermined second condition is satisfied.

[0196] (E2) In the above-mentioned fourth embodiment, when the mobile body 20 set to the remote automatic driving mode or the autonomous driving mode experiences an abnormal stop, it is determined whether to restore the automatic driving in the remote automatic driving mode or the autonomous driving mode, or to switch from the remote automatic driving mode or the autonomous driving mode to other driving modes.

[0197] Alternatively, when the moving body 20 set to the remote automatic driving mode or the autonomous driving mode undergoes an abnormal stop, as in the third embodiment, the determination unit 124 may determine whether each moving body 20 is a target to be driven and collected preferentially by the operator WO in the unattended driving mode. The determination processing regarding the switching of the driving mode is the same as that according to the third embodiment (see Figure 8 )same.

[0198] Specifically, the determination unit 124 determines the mobile object 20 satisfying the third condition as the target mobile object. The details of the third condition are the same as those according to the third embodiment. The target mobile object, that is, the mobile object 20 determined to be collected by the operator WO, is driven and collected by the operator WO after switching to the manned driving mode.

[0199] On the other hand, the determination unit 124 determines the mobile body 20 that does not satisfy the third condition as a mobile body (non-target mobile body) that is not driven and collected by the operator in the manned driving mode with priority. For the non-target mobile body, the remote automatic driving mode is switched to a different driving mode. For example, the mobile body 20 as the non-target mobile body is switched to the remote manual driving mode to start being driven based on the driving operation of the operator.

[0200] The determination unit 124 transmits, for each of the plurality of moving bodies 20 that have been subjected to the abnormal stop, a notification of switching of the driving mode according to the result of the determination as to whether the moving body 20 is a target to be driven and collected in the manned driving mode by the operator WO, to the control value preparation unit 125. Further, the determination unit 124 transmits the notification to instruct to collect the moving body 20 together with the information specifying the moving body 20 as the target moving body to the operator WO.

[0201] As described above, also in the present embodiment, the remote control device 120 can easily determine a target to be processed preferentially among a plurality of moving objects that have been in an abnormal state, according to whether the predetermined third condition is satisfied.

[0202] (E3) In the fourth embodiment and other embodiments (E1) and (E2) described above, when the mobile body 20 travels in the autonomous driving mode, the detection result output from the internal sensor 280 of the mobile body 20 may be used in at least one of the generation of the path and the generation of the driving control signal. For example, the mobile body 20 may obtain the detection result from the internal sensor 280 and reflect the detection result from the internal sensor 280 in the path when the path is generated. The mobile body 20 may obtain the detection result from the internal sensor 280 and reflect the detection result from the internal sensor 280 in the control instruction when the control instruction is generated.

[0203] (E4) In the first to fourth embodiments, etc., when a plurality of mobile bodies 20 set to the remote automatic driving mode are traveling, the abnormality detection unit 123 detects the occurrence of an abnormal state of the mobile body 20. Alternatively, the abnormality detection unit 123 may detect the occurrence of an abnormal state of the mobile body 20 that is traveling in the remote automatic driving mode or is traveling by autonomous control. The determination unit 124 is capable of determining the mobile body 20 that should be processed with priority among the mobile bodies 20 in the abnormal state. In this case as well, as in the first to fourth embodiments, it is possible to easily determine the target to be processed with priority among the plurality of mobile bodies in the abnormal state, and take necessary measures for the designated mobile body.

[0204] (E5) The mobile object 20 may be capable of traveling in two or three driving modes among the attended driving mode, the remote manual driving mode, the remote automatic driving mode, and the autonomous driving mode.

[0205] (E6) When the mobile body 20 is capable of traveling in one of the attended driving mode, the remote manual driving mode, and the remote automatic driving mode, and is also capable of traveling by autonomous control, the remote control device 120 may perform the following control.

[0206] When the target mobile body is a mobile body that causes an abnormal state while driving in the autonomous driving mode, the remote control device 120 performs control to start the automatic driving of the target mobile body in the remote control mode. When the target mobile body is the mobile body 20 that causes an abnormal state while being subjected to automatic driving by remote control, the remote control device 120 performs control to start the driving of the target mobile body in the autonomous driving mode. The target mobile body is specified based on the condition 1 described in relation to the first embodiment.

[0207] When an abnormal state is caused in the mobile body 20 that is subjected to automatic driving performed autonomously or by remote control, it is assumed that it is generally difficult to resume driving in the same driving mode. Therefore, in another embodiment 2, when an abnormal state is caused, a driving mode different from the set driving mode is switched. As a result, the automatic driving of the mobile body 20 performed autonomously or by remote control can be started immediately.

[0208] (E7) In the first to fourth embodiments, etc., different processes are performed on the target mobile body and the non-target mobile body. However, the target mobile body and the non-target mobile body may be processed in the same manner. In this case, first, the target mobile body is processed preferentially. After the processing of all target mobile bodies is completed, the processing of the non-target mobile body is started.

[0209] (E8) In the first to fourth embodiments, as an example of an abnormal state of the mobile body 20, the mobile body 20 set to the remote automatic driving mode undergoes an abnormal stop. However, the abnormal state of the mobile body 20 is not limited to the abnormal stop of the mobile body 20. For example, a failure in the communication line between the remote control device 120 and the mobile body 20, or a failure in the remote control device 100 itself, can be determined as the occurrence of an abnormal state.

[0210] (E9) In the first embodiment, the remote control device 120 performs the processing from acquiring the position information of the mobile body 20 to generating the control command. In contrast, the mobile body 20 may perform at least a part of the processing from acquiring the position information of the mobile body 20 to generating the control command. For example, the following aspects (1) to (3) may be used:

[0211] (1) The remote control device 120 can obtain the position information of the mobile body 20, determine the target position to which the mobile body 20 should go next, and generate a path from the current position of the mobile body 20 represented by the obtained position information to the target position. The remote control device 120 can generate a path to the target position between the current position and the destination position, or can also generate a path to the destination position. The remote control device 120 can transmit the generated path to the mobile body 20. The mobile body 20 can generate a control instruction to make the mobile body 20 travel on the path received from the remote control device 120, and use the generated control instruction to control the driving device 220, the steering device 230, and the braking device 240.

[0212] (2) The remote control device 120 can obtain the position information of the mobile body 20 and transmit the obtained position information to the mobile body 20. The mobile body 20 can determine the target position to which the mobile body 20 should go next, generate a path from the current position of the mobile body 20 indicated by the received position information to the target position, generate a travel control signal for causing the mobile body 20 to travel on the generated path, and control the driving device 220, the steering device 230, and the braking device 240 using the generated travel control signal.

[0213] (3) In the above aspects (1) and (2), the detection result output from at least one of the external sensor 270 or the internal sensor 280 installed on the mobile body 20 may be used for at least one of the generation of the path and the generation of the control command. For example, in the above aspect (1), the remote control device 120 may obtain the detection result from at least one of the external sensor 270 or the internal sensor 280, and reflect the detection result in the path when generating the path. In the above aspect (1), the mobile body 20 may obtain the detection result from at least one of the external sensor 270 or the internal sensor 280, and reflect the detection result in the control command when generating the control command. In the above aspect (2), the mobile body 20 may obtain the detection result from at least one of the external sensor 270 or the internal sensor 280, and reflect the detection result in the path when generating the path. In the above aspect (2), the mobile body 20 may obtain the detection result from at least one of the external sensor 270 or the internal sensor 280, and reflect the detection result in the control command when generating the control command.

[0214] (E10) In the first to fourth embodiments, the moving body 20 is a pure electric vehicle. However, the moving body 20 is not limited to a pure electric vehicle. The term "moving body" refers to a movable object, such as a vehicle or an electric vertical lift (a so-called flying vehicle). The vehicle may be a vehicle that runs on wheels, such as a passenger car, a truck, a bus, a two-wheeled vehicle, and a four-wheeled vehicle. Alternatively, the vehicle may be a vehicle that runs on an endless track, such as a tank and a construction vehicle. Vehicles include pure electric vehicles, gasoline vehicles, hybrid vehicles, and fuel cell electric vehicles. When the moving body 20 is not a vehicle, the expressions "vehicle" and "car" used herein may be replaced with "moving body" as appropriate, and the expression "running" may be replaced with "moving" as appropriate.

[0215] (E11) In the first embodiment and the like, the mobile body 20 adopts a steer-by-wire system, a throttle-by-wire system, and a brake-by-wire system. However, the steering wheel and the wheels may be connected to each other through a mechanical structure in the mobile body 20. The accelerator pedal and the throttle valve may be connected to each other through a mechanical structure in the mobile body 20. The brake pedal and the brake caliper may be connected to each other through a mechanical structure in the mobile body 20.

[0216] (E12) In the above-described embodiment, the remote control device 120 is used to remotely operate the mobile body 20 in the remote automatic driving mode and the remote manual driving mode. Alternatively, a remote automatic driving device for remotely operating the mobile body 20 in the remote automatic driving mode and a remote manual driving device for remotely operating the mobile body 20 in the remote manual driving mode may be prepared separately. In this case, another device different from the remote automatic driving device and the remote manual driving device may determine the switching of the driving mode. Alternatively, one of the remote automatic driving device and the remote manual driving device may determine the switching of the driving mode.

[0217] (E13) In the above-mentioned first embodiment, the calculation unit 122 uses the external captured image information to calculate the position and orientation of the mobile body 20. Alternatively, the calculation unit 122 may calculate the position and orientation of the mobile body using information acquired by a sensor of a different type from the external camera as the detection device 50. The calculation unit 122 may calculate the position and orientation of the mobile body using, for example, three-dimensional point cloud information acquired by an external laser radar. The external laser radar is a laser radar set at a position different from the mobile body 20. The external laser radar includes the mobile body 20 as a calculation target within the detection range.

[0218] In this case, the calculation unit 122 calculates the position and orientation of the mobile body 20, for example, by matching between the measurement point cloud data and the reference point cloud data. The measurement point cloud data is three-dimensional point cloud data obtained by detecting the mobile body 20 as the calculation target by an external laser radar. The reference point cloud data is a reference data to be used as a template in the process of matching with the measurement point cloud data. The reference point cloud data is, for example, virtual three-dimensional point cloud data generated based on three-dimensional computer-aided design (CAD) data representing the shape of the mobile body 20. For example, one of iterative closest point (ICP) and normal distribution transform (NDT) is used as an algorithm for matching by the calculation unit 122. The calculation unit 122 can calculate the position and orientation of the mobile body using the measurement point cloud data obtained by detecting the mobile body 20 as the calculation target by a vehicle-mounted laser radar installed on another mobile body different from the mobile body 20 as the calculation target.

[0219] (E14) The functions of the remote control device 120 are not limited to being implemented by software, but may be partially or entirely implemented by dedicated hardware. For example, a circuit represented by a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC) may be used as the dedicated hardware.

[0220] (E15) In each of the above embodiments, the mobile body 20 only needs to include components that enable it to move by automatic driving, and the mobile body 20 can be in the form of a platform including the following components, for example. Specifically, in order to realize the three functions of "driving", "turning" and "stopping" by automatic driving, the mobile body 20 only needs to include at least a mobile body control device, a drive device, a steering device and a brake device. In order for the mobile body 20 to obtain information for automatic driving from the outside, the mobile body 20 only needs to include a communication device. That is, at least a part of the internal components such as the driver's seat or the dashboard may not be installed to the mobile body 20 that can be moved by automatic driving, at least a part of the external components such as the bumper or the fender may not be installed to the mobile body 20, or the body shell may not be installed to the mobile body 20. In this case, the remaining components such as the body shell may be installed to the mobile body 20 before the mobile body 20 is shipped from the factory, or the remaining components such as the body shell may be installed to the mobile body 20 after the mobile body 20 is shipped from the factory in a state where the remaining components such as the body shell are not installed on the mobile body 20. These components can be mounted to the moving body 20 from any direction (including the upper side, the lower side, the front side, the rear side, the right side and the left side), and can be mounted from the same direction or from different directions. In addition, when the moving body 20 is a platform form, the position is determined in the same manner as the moving body 20 according to the first embodiment.

[0221] (E16) The mobile body 20 can be manufactured by any manufacturing method. For example, the mobile body 20 can be manufactured by combining a plurality of modules. Each module refers to a unit composed of a plurality of components grouped according to the area or function of the mobile body 20. For example, the platform of the mobile body 20 can be manufactured by combining a front module constituting the front part of the platform, a central module constituting the central part of the platform, and a rear module constituting the rear part of the platform. The number of modules constituting the platform is not limited to 3, but can be less than 2 or more than 4. In addition to or instead of the components constituting the platform, the module may include components constituting a portion of the mobile body 20 that is different from the platform. Various modules may include any external component (such as a bumper or grille) or any internal component (such as a seat or console). Not only the mobile body 20, but also any aspect of the mobile body can be manufactured by combining a plurality of modules. For example, such a module can be manufactured by welding, joining a plurality of components using a clamp, etc., or can be manufactured by casting at least a portion of the components constituting the module as a single component. The method of integrally molding a single component (especially a relatively large component) is called gigacasting or megacasting. For example, the front module, the center module, and the rear module described above can be manufactured by integral casting.

[0222] (E17) The transportation of a moving body by driving the moving body automatically is called "self-propelled transportation". The configuration for realizing self-propelled transportation is called "vehicle remote control autonomous driving transportation system". The method of producing a moving body by using self-propelled transportation is called "self-propelled production". In self-propelled production, at least a part of the transportation of the moving body is realized by, for example, self-propelled transportation in a factory that manufactures the moving body.

[0223] The present disclosure is not limited to the above-mentioned embodiments, and can be implemented by various configurations without departing from the spirit of the present disclosure. For example, the technical features in the various embodiments corresponding to the technical features of the various aspects described in the summary of the invention can be appropriately replaced or combined to solve the above-mentioned problems, or to achieve some or all of the above-mentioned effects. When a technical feature is not described as being necessary in this article, such a feature can be omitted as appropriate.

Claims

1. A control device for controlling the operation of a moving object, characterized in that comprising a processor, wherein the processor is configured to: In a case where an abnormal state occurs in a plurality of mobile bodies that perform autonomous driving autonomously or by remote control, designating at least one mobile body to be processed preferentially from among the plurality of mobile bodies as a target mobile body based on the states of the plurality of mobile bodies; and A control is executed to take necessary measures on the target moving object.

2. The control device according to claim 1, characterized in that: The autonomous driving performed by the remote control includes autonomous driving in which the plurality of moving bodies travel by receiving the travel conditions generated by the control device without being based on remote operation by an operator at a remote location; and The processor is configured as follows: designating at least one moving object that preferentially performs autonomous driving autonomously or by remote control as the target moving object from among the plurality of moving objects detected as being in the abnormal state, and A control is performed to cause the target moving object to perform the automatic driving autonomously or through the remote control.

3. The control device according to claim 2, characterized in that: The target moving body includes a moving body whose distance from a current position to a position where a downstream process is performed is equal to or smaller than a predetermined distance.

4. The control device according to claim 2, characterized in that: The target moving body includes a moving body whose distance from a current position to a position where an upstream process is performed is equal to or smaller than a predetermined distance.

5. The control device according to claim 2, characterized in that: The target moving bodies include moving bodies whose time from the current time until the set expected shipping date and time is equal to or less than a predetermined time.

6. The control device according to claim 2, characterized in that: The target mobile object includes a mobile object on which no stationary object exists on the expected travel route.

7. The control device according to claim 2, characterized in that: The target mobile objects include mobile objects of models planned to be produced with priority.

8. The control device according to claim 2, characterized in that: The target mobile object includes a mobile object currently located outdoors.

9. The control device according to claim 2, characterized in that: The target mobile object includes a mobile object currently located indoors.

10. The control device according to any one of claims 2 to 9, characterized in that: The processor is configured as follows: in a case where the target mobile object is a mobile object that enters the abnormal state while executing the autonomous driving, executing control of starting the autonomous driving executed by the remote control on the target mobile object; as well as In a case where the target moving object is a moving object that enters the abnormal state while the autonomous driving is performed by the remote control, control of starting the autonomous driving is performed on the target moving object.

11. The control device according to claim 1, characterized in that: The automatic driving performed by the remote control includes: wherein the plurality of moving bodies travel automatically based on remote operation performed by an operator using an operator input device provided at a position different from the moving bodies, and wherein the plurality of moving bodies drive automatically by receiving the driving conditions generated by the control device without being based on the remote operation of the operator; and The processor is configured as follows: designating, from among a plurality of moving objects detected as being in the abnormal state, at least one moving object for which the autonomous driving based on the remote operation of the operator is to be started with priority as the target moving object, and Control of starting the automatic driving based on the remote operation by the operator is executed on the target moving object.

12. The control device according to claim 11, characterized in that: The target moving body includes a moving body whose distance from a current position to a position where a downstream process is performed is equal to or smaller than a predetermined distance.

13. The control device according to claim 11, characterized in that: The target moving body includes a moving body whose distance from a current position to a position where an upstream process is performed is equal to or smaller than a predetermined distance.

14. The control device according to claim 11, characterized in that: The target mobile object includes a mobile object capable of providing visual information to be used by an operator located at a remote location to remotely operate the mobile object.

15. The control device according to claim 11, characterized in that: The target mobile object includes a mobile object located on a path along which an operator at a remote location has caused the mobile object to travel based on the remote operation.

16. The control device according to claim 11, characterized in that: The target moving object includes the moving object where no object that hinders the moving of the moving object exists around the moving object when the moving of the moving object is started by remote operation of an operator at a remote location.

17. The control device according to claim 1, characterized in that: The processor is configured as follows: designating, from among a plurality of moving objects detected as being in the abnormal state, at least one moving object to be collected preferentially by driving of an operator who is riding, as the target moving object; and An instruction to collect the target moving object is notified to the operator together with information required for collection.

18. The control device according to claim 17, characterized in that: The target moving object includes the moving object in which there is no object that hinders the moving of the moving object when the moving object starts to be driven by the operator riding on the moving object.

19. A method for a computer-controlled mobile body to perform automatic driving autonomously or by remote control without remote operation by an operator at a remote location, characterized in that: The method comprises: In a case where an abnormal state occurs in a plurality of mobile bodies that perform the autonomous driving autonomously or through the remote control, designating at least one mobile body that is to perform the autonomous driving autonomously or through the remote control with priority as the target mobile body from among the plurality of mobile bodies; and A control is performed to cause the target moving object to perform the automatic driving autonomously or through the remote control.

20. A method for a computer-controlled mobile object to perform automatic driving autonomously or by remote control without relying on remote operation by an operator at a remote location, characterized in that: The method comprises: In the event that an abnormal state occurs in a plurality of mobile bodies that perform the autonomous driving autonomously or through the remote control, designating at least one mobile body for which the autonomous driving based on the remote operation of the operator is to be started with priority as the target mobile body from among the plurality of mobile bodies; and Control of starting the automatic driving based on the remote operation by the operator is executed on the target moving object.

Images