Remote driving system, remote driving terminal, and method for remote driving

Abstract

To provide a technique which makes it possible to finish an initial check early and start remote driving smoothly in a remote driving system.SOLUTION: The present disclosure relates to a remote driving system for performing remote driving of a vehicle on the basis of a manipulated variable input to a remote driving terminal. The remote driving system detects a first situation in which the remote driving of the vehicle is required, and acquires a degree of urgency of the first situation. When the first situation is detected, the remote driving system performs an initial check for checking that the remote driving can be started at the remote driving terminal. The remote driving system omits part of the initial check according to the degree of urgency.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to a technique for remotely driving a vehicle. [Background technology]

[0002] Patent Document 1 discloses a vehicle control device. When a transition from one driving state to another driving state is scheduled, the vehicle control device notifies occupants and other vehicles of information related to the transition of the driving state. The driving states of a vehicle include manual driving, automatic driving, and remote driving. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-018486 Summary of the Invention [Problem to be solved by the invention]

[0004] Systems are known that remotely drive a vehicle based on operation amounts input by a remote operator to a remote driving terminal. Such systems are sometimes called remote driving systems. The remote driving system starts remote driving when the vehicle is in a situation requiring remote driving. However, at this time, the remote driving system does not immediately start remote driving. Before starting remote driving, the remote driving system first performs an initial check to confirm that remote driving can be started. The remote driving system then starts remote driving only after the results of the initial check are positive. Therefore, in order to start remote driving smoothly, it is necessary to complete the initial check as soon as possible.

[0005] One object of the present disclosure is to provide a technology that enables an initial check to be completed early and remote operation to be started smoothly in a remote operation system. [Means for solving the problem]

[0006] The first aspect relates to a remote driving system that remotely drives a vehicle based on an operation amount input to a remote driving terminal. The remote operation system includes one or more processors. The one or more processors detect a first situation in which remote operation of the vehicle is required. The one or more processors obtain an urgency of the first situation. When a first condition is detected, the one or more processors perform an initial check to confirm that the remote driving can be initiated at the remote driving terminal. The one or more processors omit part of the initial check depending on the degree of urgency.

[0007] The second aspect relates to a remote driving terminal that remotely drives a vehicle based on an operation amount input by a remote operator. The remote operating terminal comprises one or more processors. The one or more processors obtain information indicating that a first situation requiring remote operation of the vehicle has been detected. The one or more processors obtain an urgency of the first situation. The one or more processors perform an initial check to verify that the remote driving terminal is able to begin remote driving if a first condition is detected. The one or more processors omit part of the initial check depending on the degree of urgency.

[0008] A third aspect relates to a method for remotely driving a vehicle based on an operation amount input to a remote driving terminal. The above method is Detecting a first situation requiring remote operation of the vehicle; obtaining an urgency of a first situation; When the first situation is detected, an initial check is performed to confirm that remote operation can be started at the remote operation terminal; Depending on the urgency, some initial checks may be omitted. Includes: [Effects of the Invention]

[0009] According to the present disclosure, the urgency of a situation requiring remote driving of a vehicle is acquired. Then, a portion of the initial check is omitted depending on the urgency. By omitting a portion of the initial check, the initial check can be completed early, allowing remote driving to be started smoothly. In other words, remote driving can be started smoothly, taking into account the urgency of a situation requiring remote driving of a vehicle. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating a configuration example of a remote driving system according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the configuration of a vehicle according to an embodiment of the present invention; [Figure 3] 2 is a block diagram showing an example of the configuration of a remote driving terminal according to the present embodiment; FIG. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a management device according to the present embodiment. [Figure 5] FIG. 10 is a schematic diagram for explaining deviation from a target path. [Figure 6] FIG. 10 is a diagram for explaining a specific example of an initial check. [Figure 7] FIG. 10 is a schematic diagram showing a state of steering position adjustment. [Figure 8] FIG. 4 is a diagram showing a priority map according to the present embodiment. [Figure 9] 1 is a block diagram showing an example of a functional configuration of a remote operation system according to an embodiment of the present invention; [Figure 10] 4 is a flowchart illustrating an example of processing performed by the remote operation system according to the present embodiment. [Figure 11] 4 is a time chart illustrating the effect of the processing performed by the remote operation system according to the present embodiment. [Figure 12] FIG. 10 is a diagram for explaining a first modified example. [Figure 13] FIG. 10 is a block diagram showing an example of the functional configuration of a remote operation system according to a second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0012] 1. Remote operation system 1-1. System configuration 1 is a schematic diagram showing an example configuration of a remote driving system 1 according to this embodiment. The remote driving system 1 includes a vehicle 100, a remote driving terminal 200, and a management device 300. The vehicle 100, the remote driving terminal 200, and the management device 300 can communicate with each other via a communication network.

[0013] The vehicle 100 is a vehicle that is to be remotely driven by a remote operator X. The vehicle 100 may be an autonomous vehicle. The remote driving terminal 200 is a terminal device that the remote operator X operates when remotely driving the vehicle 100. The remote driving terminal 200 can also be called a remote cockpit. The management device 300 manages the remote driving system 1. Typically, the management device 300 is a management server on the cloud. The management device 300 may be configured from multiple servers that perform distributed processing.

[0014] The remote driving system 1 may also include an infrastructure sensor 400. The infrastructure sensor 400 includes an infrastructure camera. The infrastructure sensor 400 may also include a rainfall sensor, etc. The infrastructure sensor 400 is installed in an area where the vehicle 100 travels. The infrastructure sensor 400 and the management device 300 can communicate with each other via a communication network. The infrastructure sensor 400 may also communicate with the vehicle 100 and the remote driving terminal 200 via the management device 300 or directly.

[0015] 2 to 4 are block diagrams showing examples of the configuration of the vehicle 100, the remote driving terminal 200, and the management device 300, respectively.

[0016] 2 shows an example configuration of a vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, an actuator 130, and a control device 150. In this example, the vehicle 100 is equipped with an autonomous driving system and is capable of autonomous driving.

[0017] The communication device 110 communicates with devices outside the vehicle 100. The communication devices 110 communicate with include the remote driving terminal 200 and the management device 300.

[0018] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include an on-board camera, a LIDAR (Laser Imaging Detection and Ranging), a radar, etc. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, a brake oil pressure sensor, etc. The position sensor detects the position and direction of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System) sensor. The sensor group 120 may also include a rainfall sensor.

[0019] The actuators 130 include a steering actuator, a drive actuator, and a brake actuator. The steering actuator steers the wheels. For example, the steering actuator includes an electric power steering (EPS) device. The drive actuator is a power source that generates drive force. Examples of drive actuators include an engine, an electric motor, and an in-wheel motor. The brake actuator generates braking force. For example, the brake actuator controls brake hydraulic pressure to activate the brakes.

[0020] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processors 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processor 160 executes various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various programs and various information required for processing by the processor 160. The processor 160 executes the programs stored in the storage device 170, thereby realizing the functions of the control device 150. Examples of the storage device 170 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The control device 150 may include one or more ECUs (Electronic Control Units).

[0021] The control device 150 controls the actuators 130 to control the traveling of the vehicle 100. The control device 150 also acquires vehicle information VCL from the sensor group 120. The vehicle information VCL includes recognition sensor information indicating the recognition results from the recognition sensors, vehicle state information obtained from the vehicle state sensors, and position information obtained from the position sensors. The recognition sensor information includes images captured by an on-board camera. The vehicle state information includes speed information, acceleration information, steering angle information, brake oil pressure information, etc. obtained from a speed sensor, an acceleration sensor, a steering angle sensor, a brake oil pressure sensor, etc. The vehicle information VCL may further include highly accurate position information obtained by a self-localization process. The control device 150 can acquire highly accurate position information by performing a self-localization process using map information and recognition sensor information stored in the storage device 170. The vehicle information VCL acquired by the control device 150 can be used for automatic driving or remote driving of the vehicle 100.

[0022] For example, the control device 150 controls the automatic driving of the vehicle 100 based on the vehicle information VCL. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the vehicle information VCL. Furthermore, the control device 150 generates a target path required for the vehicle 100 to drive according to the driving plan based on the vehicle information VCL. The target path is a set of target positions for the vehicle 100. The target path may be set to extend along the center of the lane. Then, the control device 150 controls the driving of the vehicle 100 so that the vehicle 100 follows the target path.

[0023] Furthermore, the control device 150 communicates via the communication device 110 and can transmit the vehicle information VCL to the remote driving terminal 200 and the management device 300. At least when the vehicle 100 is being remotely driven, the vehicle information VCL is transmitted to the remote driving terminal 200. The vehicle information VCL transmitted to the remote driving terminal 200 is referenced by the remote operator X, who then performs remote driving.

[0024] 3 shows an example of the configuration of the remote driving terminal 200. The remote driving terminal 200 includes a communication device 210, an output device 220, a remote control member 230, and a control device 250.

[0025] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0026] The output device 220 outputs various information and presents it to the remote operator X. For example, the output device 220 includes a display device. The display device displays the various information toward the remote operator X. As another example, the output device 220 may include a speaker.

[0027] The remote control member 230 is a member that the remote operator X operates when remotely driving the vehicle 100. For example, the remote control member includes a steering operation member, an accelerator pedal, a brake pedal, a turn signal, etc. The steering operation member is, for example, a steering wheel. The remote control member may include a touch panel, buttons, etc. The amount of operation input by the remote operator X when remotely driving the vehicle 100 is detected by a sensor installed in the remote control member 230.

[0028] The control device 250 controls the remote operation terminal 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as processor 260) and one or more storage devices 270 (hereinafter simply referred to as storage device 270). The processor 260 executes various processes. For example, the processor 260 includes a CPU. The storage device 270 stores various programs and various information required for processing by the processor 260. The processor 260 executes the programs stored in the storage device 270, thereby realizing the functions of the control device 250. Examples of the storage device 270 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0029] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information VCL transmitted from the vehicle 100. The control device 250 presents the vehicle information VCL, including images, to the remote operator X by displaying the vehicle information VCL on a display device. The remote operator X can recognize the state of the vehicle 100 and the surrounding circumstances based on the vehicle information VCL displayed on the display device.

[0030] Furthermore, the control device 250 may acquire infrastructure information detected by the infrastructure sensor 400 via the management device 300 or directly. The infrastructure information acquired by the control device 250 may include, for example, images obtained by an infrastructure camera capturing images of the vehicle 100 and its surroundings. The acquired infrastructure information is displayed on a display device. The remote operator X may recognize the state of the vehicle 100 and the surrounding circumstances by referring to the infrastructure information. By presenting the remote operator X with images captured by the infrastructure camera in addition to images captured by the on-board camera, the accuracy of remote driving and convenience for the remote operator X are improved.

[0031] The control device 250 also generates remote driving information REM based on the amount of operation input by the remote operator X. The remote driving information REM is information for controlling the vehicle 100 by remote driving. The remote driving information REM includes the amount of operation of the remote operation member 230 input by the remote operator X. The control device 250 transmits the remote driving information REM to the vehicle 100 as necessary.

[0032] 4 shows an example of the configuration of the management device 300. The management device 300 includes a communication device 310 and a control device 350.

[0033] The communication device 310 communicates with the vehicle 100 , the remote driving terminal 200 , and the infrastructure sensor 400 .

[0034] The control device 350 controls the management device 300. The control device 350 includes one or more processors 360 (hereinafter simply referred to as processors 360) and one or more storage devices 370 (hereinafter simply referred to as storage devices 370). The processor 360 executes various processes. For example, the processor 360 includes a CPU. The storage device 370 stores various programs and various information required for processing by the processor 360. The processor 360 executes the programs stored in the storage device 370, thereby realizing the functions of the control device 350. Examples of the storage device 370 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0035] The control device 350 communicates with the vehicle 100 and the remote driving terminal 200 via the communication device 310. Furthermore, the control device 350 communicates with the infrastructure sensor 400 via the communication device 310 as necessary to acquire infrastructure information. The infrastructure information acquired by the control device 350 from the infrastructure sensor 400 includes images captured by an infrastructure camera.

[0036] 1-2. Remote operation using a remote operation system The remote driving system 1 remotely drives the vehicle 100 based on the operation amount input by the remote operator X to the remote driving terminal 200. When the remote driving system 1 detects a "first situation," remote driving of the vehicle 100 begins. The first situation is a situation in which remote driving of the vehicle 100 is required. The first situation may be detected by the vehicle 100 itself, or may be detected by the management device 300.

[0037] For example, the first situation may be a situation in which a remote driving request (RFO: Request for Operation) is transmitted from the vehicle 100 to the management device 300. For example, when the autonomous driving system of the vehicle 100 determines that it is difficult to continue autonomous driving, the remote driving request is transmitted from the vehicle 100 to the management device 300. Note that the remote driving request from the vehicle 100 may be transmitted while the vehicle 100 is traveling, or may be transmitted while the vehicle 100 is stopped.

[0038] Alternatively, the remote driving system 1 may detect the first situation based on infrastructure information acquired from the infrastructure sensor 400. For example, when a rainfall sensor of the infrastructure sensor 400 indicates that the average amount of rainfall at the location where the vehicle 100 is traveling is greater than a threshold, it is predicted that it will be difficult for the vehicle 100 to continue autonomous driving. Alternatively, when an abnormal state of the vehicle 100 is detected from an image captured by an infrastructure camera of the infrastructure sensor 400, it is predicted that it will be difficult for the vehicle 100 to continue autonomous driving. For example, the abnormal state of the vehicle 100 is a state in which the vehicle 100 is traveling unsteadily. As another example, the abnormal state of the vehicle 100 is a state in which the vehicle 100, which should be traveling, is stopped. The remote driving system 1 may detect such an abnormal state of the vehicle 100 as the first situation.

[0039] Alternatively, the remote driving system 1 may detect the first situation based on deviation of the vehicle 100 from the target path during autonomous driving control. As shown in FIG. 5, there is a situation in which the driving position of the vehicle 100 deviates from the target path and the deviation amount is greater than a threshold. The remote driving system 1 may detect such a situation as the first situation. The deviation amount can also be expressed as the distance between the vehicle 100 and the target path. The remote driving system 1 can calculate the deviation amount by comparing the position of the vehicle 100 with the target path. The remote driving system 1 may obtain the position of the vehicle 100 from position information of the vehicle 100 obtained from the vehicle information VCL, or may obtain the position from an image captured by an infrastructure camera of the infrastructure sensor 400. Information regarding the target path to be followed by the vehicle 100 is obtained from the vehicle information VCL.

[0040] When the remote driving system 1 detects the first situation, the management device 300 assigns a remote operator X from among multiple candidates to the vehicle 100. The management device 300 manages the assignment relationship between the vehicle 100 and the remote operator X, and provides information about the assignment relationship to the vehicle 100 and the remote driving terminal 200. Upon receiving the information about the assignment relationship, the vehicle 100 and the remote driving terminal 200 establish communication.

[0041] During remote driving, the vehicle 100 and the remote driving terminal 200 transmit and receive information via a communication network. Note that communication between the vehicle 100 and the remote driving terminal 200 may be performed directly or via the management device 300.

[0042] The vehicle 100 transmits the vehicle information VCL to the remote driving terminal 200. The remote driving terminal 200 receives the vehicle information VCL transmitted from the vehicle 100 and presents it to the remote operator X. For example, the remote driving terminal 200 presents the vehicle information VCL by displaying an image on the display device of the output device 220. The remote operator X looks at the displayed information, recognizes the situation around the vehicle 100, and remotely drives the vehicle 100 by operating the remote control member 230.

[0043] The control device 250 generates remote driving information REM based on the amount of operation of the remote control member 230 input by the remote operator X. Then, the control device 250 transmits the remote driving information REM to the vehicle 100 via the communication device 210.

[0044] The vehicle 100 receives the remote driving information REM transmitted from the remote driving terminal 200. The vehicle 100 controls the traveling of the vehicle in accordance with the received remote driving information REM. In this way, the vehicle 100 is remotely driven.

[0045] 2. Initial check As described above, remote driving of vehicle 100 is initiated by remote driving system 1 when the first situation is detected. However, the detection of the first situation does not mean that remote driving is initiated immediately. When the first situation is detected, remote driving system 1 first performs an "initial check" to confirm that remote driving can be initiated. Then, if the result of the initial check indicates that remote driving can be initiated, remote driving system 1 initiates remote driving.

[0046] Fig. 6 shows a specific example of the initial check. The initial check is performed in the remote operation terminal 200. As shown in Fig. 6, the initial check includes several steps.

[0047] The first step of the initial check is a "communication establishment confirmation" to confirm that communication between the vehicle 100 and the remote driving terminal 200 has been established normally. More specifically, the communication establishment confirmation confirms that the vehicle 100 and the remote driving terminal 200 assigned to that vehicle 100 are normally connected and are in a state where communication is possible. For example, the remote driving terminal 200 transmits a test signal to the vehicle 100 to confirm communication establishment. The test signal may be a Ping. If the remote driving terminal 200 normally receives a response signal sent from the vehicle 100 in response to the test signal, it is determined that communication has been established normally. If it is determined that communication has been established normally, this means that necessary information can be accurately exchanged between the vehicle 100 and the remote driving terminal 200.

[0048] Once the communication establishment confirmation is complete, it is guaranteed that the remote driving terminal 200 can normally receive the vehicle information VCL from the vehicle 100. The initial check proceeds to the next step, where "steering position alignment" is performed. Steering position alignment means matching the steering angle of the vehicle 100 with the steering angle of the steering operation member of the remote driving terminal 200. The steering angle of the vehicle 100 is the steering angle of the steering wheel of the vehicle 100. Alternatively, the steering angle of the vehicle 100 may be calculated from the steering angle of the wheels of the vehicle 100. Information on the steering angle of the steering wheel and the steering angle of the wheels of the vehicle 100 is obtained from the vehicle information VCL sent from the vehicle 100 to the remote driving terminal 200. The steering angle of the steering operation member of the remote driving terminal 200 is detected by a sensor installed on the steering operation member.

[0049] FIG. 7 is a diagram showing an example of a screen displayed on the display device of the remote driving terminal 200 during steering position adjustment. The driving trajectory 221 shown by a dotted line represents the driving trajectory of the vehicle 100 calculated from the current steering angle of the vehicle 100. On the other hand, the driving trajectory 222 shown by a solid line represents the driving trajectory of the vehicle 100 calculated from the steering angle of the steering operation member of the remote driving terminal 200. The remote driving terminal 200 draws the driving trajectories 221 and 222 so that they are superimposed on the image included in the vehicle information VCL. The remote driving terminal 200 then displays the image with the driving trajectories 221 and 222 superimposed on it on the display device. Steering position adjustment is performed by adjusting the steering angle of the steering operation member so that these two driving trajectories 221 and 222 coincide with each other. Typically, the remote operator X performs the alignment by moving the steering operation member so that the driving trajectory 221 and the driving trajectory 222 coincide with each other. The coincidence of the two trajectories may be determined automatically by the remote driving terminal 200 or manually by the remote operator X. When the determination is made manually, the remote operator X inputs by means of a button or the like on the remote control member 230 that the two trajectories match.

[0050] Completion of the steering position alignment completes preparations for reflecting the operation amount of the steering operation member in the steering of the wheels on the vehicle 100 side. After that, it becomes possible to transmit remote driving information REM from the remote driving terminal 200 to the vehicle 100.

[0051] Referring again to Figure 6, once steering position alignment is complete and the steering angle of the steering operation member corresponds to the steering angle of the wheels, an operation check is performed. The operation check is performed to check that the operation amount of the remote operator X transmitted to the vehicle 100 as the remote driving information REM is reflected in the operation of the actuator 130, that is, to check that the actuator 130 operates in response to the operation amount of the remote operator X. The operation check includes an operation check for the brakes and an operation check for the steering. Note that the order in which the operation check for the brakes and the operation check for the steering are performed does not matter.

[0052] The brake operation check is performed to confirm that the amount of brake operation input by the remote operator X to the brake pedal of the remote control member 230 is reflected in the operation of the brake actuator of the vehicle 100. The remote driving terminal 200 transmits a test signal to the vehicle 100 requesting the operation of the brake actuator. For example, the remote driving terminal 200 transmits remote driving information REM (test signal) including the amount of operation of the brake pedal of the remote control member 230 to the vehicle 100. If the brake actuator operates normally in response to the test signal, the brake hydraulic pressure should change normally. Therefore, the remote driving terminal 200 receives vehicle information VCL from the vehicle 100 and monitors the brake hydraulic pressure information included in the vehicle information VCL. If the brake hydraulic pressure changes as expected in response to the transmission of the test signal, the remote driving terminal 200 determines that the brake actuator is operating. If the brake actuator is operating normally, the brake operation check is completed.

[0053] The steering operation check is performed to confirm that the amount of operation input by the remote operator X to the steering operation member of the remote control unit 230 is reflected in the operation of the steering actuator, thereby steering the wheels of the vehicle 100. The remote driving terminal 200 transmits a test signal to the vehicle 100 requesting the operation of the steering actuator. For example, the remote driving terminal 200 transmits remote driving information REM (test signal) including the amount of operation of the steering operation member of the remote control unit 230 to the vehicle 100. If the steering actuator operates normally in response to the test signal, the steering angle of the vehicle 100 should change normally. Therefore, the remote driving terminal 200 receives vehicle information VCL from the vehicle 100 and monitors the steering angle information included in the vehicle information VCL. If the steering angle of the vehicle 100 changes as expected in response to the transmission of the test signal, the remote driving terminal 200 determines that the steering actuator is operating. If the steering actuator is operating normally, the steering operation check is completed.

[0054] The above is a specific example of an initial check. When the remote driving system 1 detects the first situation, it first performs the initial check illustrated in FIG. 6. Then, after the result of the initial check confirms that remote driving can be started, the remote driving system 1 starts remote driving. In other words, the remote driving system 1 cannot start remote driving until the initial check is complete. Therefore, if the initial check takes a long time, the start of remote driving will be delayed accordingly.

[0055] However, when the first situation is a highly urgent situation, it may be necessary to start remote driving of the vehicle 100 early. For example, when the vehicle 100 stops in a dangerous place, it is necessary to start remote driving early. In order to start remote driving early, it is necessary to complete the initial check early.

[0056] Therefore, when the remote driving system 1 according to this embodiment detects a first situation in which remote driving of the vehicle 100 is required, it acquires an "urgency" that is an index showing the degree of urgency of the first situation. Then, the remote driving system 1 omits part of the initial check according to the urgency. By omitting part of the initial check, the time required for the initial check can be shortened, and remote driving can be started earlier. In other words, it is possible to start remote driving smoothly, taking into account the urgency of the situation in which remote driving of the vehicle 100 is required.

[0057] 3. Omission of initial check Omitting part of the initial check means omitting one or more of the multiple steps included in the initial check. Which steps are omitted is determined based on the urgency and "priority."

[0058] First, the priority will be explained. An omission priority is set for each step included in the initial check according to the content of the step. FIG. 8 is a priority map showing the omission priority. The priority map is stored in at least one of the storage device 170 of the vehicle 100, the storage device 270 of the remote driving terminal 200, and the storage device 370 of the management device 300. When the initial check is to be omitted, steps with higher priorities are preferentially omitted according to this priority map.

[0059] As shown in the priority map of FIG. 8, the omission priority is set so that the operation confirmation is higher than the communication establishment confirmation, and the steering position alignment is higher than the operation confirmation.

[0060] Confirming communication establishment is the most important step to perform in advance when starting remote driving of the vehicle 100. If communication between the vehicle 100 and the remote driving terminal 200 is not established normally, vehicle information VCL and remote driving information REM cannot be transmitted or received between the vehicle 100 and the remote driving terminal 200. If the vehicle information VCL is not transmitted normally from the vehicle 100 to the remote driving terminal 200, the remote operator X cannot obtain information for remote driving. Furthermore, if the remote driving information REM is not transmitted normally from the remote driving terminal 200 to the vehicle 100, remote driving cannot be performed in the first place. For this reason, the priority of omitting confirmation of communication establishment is set to the lowest.

[0061] Conversely, it is not necessarily necessary to perform steering position alignment before starting remote driving. If steering position alignment is performed as an initial check, remote operator X can start operating the steering operation member from a steering angle that corresponds to the steering angle of the wheels, making it easier to grasp the steering feel. However, even if the steering angle of the steering operation member and the steering angle of the wheels are misaligned, it is still possible to operate the steering operation member and input the operation amount. Therefore, the omission priority is set so that steering position alignment is given the highest priority.

[0062] Furthermore, among the operation checks, the priority of omitting the steering operation check is set higher than the priority of omitting the brake operation check. Of the brakes and steering, it is the brakes that are more likely to be checked in advance. This is because if the vehicle 100 gets into a situation where it must urgently avoid danger, it is likely that the remote operator X will operate the brakes first. Therefore, the priority of omitting the steering operation check is set higher than the priority of omitting the brake operation check.

[0063] Next, the degree of urgency will be described. When a portion of the initial check is omitted, the extent to which the initial check is omitted is determined based on the degree of urgency. The higher the degree of urgency, the earlier it is required to start remote operation. Therefore, the higher the degree of urgency, the greater the amount of the initial check that is omitted, and the lower the degree of urgency, the smaller the amount of the initial check that is omitted.

[0064] The following is an example of a method for calculating the urgency: For example, as shown in Fig. 5, when a situation in which the position of vehicle 100 deviates from the target path is detected as the first situation, the urgency may be calculated based on the amount of deviation of the position of vehicle 100 from the target path. In this case, the urgency is calculated so that it increases as the amount of deviation increases.

[0065] Alternatively, the urgency level may be calculated based on the situation in which the vehicle 100 is located when the first situation is detected. For example, the urgency level may be calculated as follows: When the vehicle 100 is in a highly dangerous situation, the urgency level is calculated to be the highest. When the vehicle 100 is in a situation that is not highly dangerous but may obstruct other traffic participants, the urgency level is calculated to be lower than that. When the situation does not fall into either of these categories, the urgency level is calculated to be even lower. An example of a highly dangerous situation is when it is raining heavily and the vehicle 100 is passing through or stopped in an underpass. Other examples of highly dangerous situations include when a disaster such as a fire occurs at or near the location of the vehicle 100, when the vehicle 100 is located at a railroad crossing and an alarm is sounding, etc. An example of a situation that may obstruct other traffic participants is when an emergency vehicle is approaching the vehicle 100.

[0066] The remote driving system 1 can determine such a situation based on information acquired from the infrastructure sensor 400 and the sensor group 120. For example, the remote driving system 1 can detect a situation in which heavy rain is falling based on information acquired from the sensor group 120 and a rainfall sensor of the infrastructure sensor 400. Furthermore, the remote driving system 1 can determine the situation around the vehicle 100, such as whether the location where the vehicle 100 is passing or stopped is an underpass or whether an emergency vehicle is approaching, based on information acquired from an in-vehicle camera or an infrastructure camera.

[0067] As another example of a method for calculating the urgency, if the vehicle 100 is stopped or is expected to stop when the first situation is detected, the urgency may be calculated based on the vehicle's stopped location. For example, the urgency may be calculated as follows: If the vehicle 100 is stopped within a hazardous area, the urgency is calculated to be the highest. If the vehicle 100 is stopped in a location that is not within a hazardous area but may be an obstruction to other traffic participants, the urgency is calculated to be the next highest. If the vehicle 100 is stopped in a location that does not fall into either of these categories, the urgency is calculated to be the lowest.

[0068] The dangerous area here refers to a location where the vehicle 100 may be exposed to danger if it continues to be stopped there. Examples of dangerous areas include on tram tracks, in railroad crossings, no-stopping zones, etc. Examples of no-stopping zones include in front of a fire station, etc. Examples of locations that may obstruct other traffic participants include near intersections, roads with heavy traffic, etc.

[0069] The remote driving system 1 can determine the stopping position of the vehicle 100 based on information detected by the infrastructure sensor 400 and the sensor group 120. For example, the remote driving system 1 can determine that the stopping position of the vehicle 100 is at a railroad crossing by acquiring images captured by an on-board camera or an infrastructure camera. Alternatively, the stopping position of the vehicle 100 may be acquired by combining map information indicating the positions of railroad tracks, railroad crossings, etc. with position information acquired from a GNSS sensor of the sensor group 120. The map information is stored in the storage device 170 of the vehicle 100, the storage device 270 of the remote driving terminal 200, or the storage device 370 of the management device 300. The remote driving system 1 may also determine whether traffic volume is heavy at the stopping position of the vehicle 100 based on road traffic information held by the management device 300 or road traffic information acquired from an external server.

[0070] 4. Processing example The processing performed by the remote operation system 1 will be described with reference to FIGS.

[0071] Fig. 9 is a block diagram showing an example of the functional configuration of the remote driving system 1. Fig. 10 is a flowchart showing an example of processing performed by the remote driving system 1. The remote driving system 1 includes, as functional blocks, a first situation detection unit 11, an urgency calculation unit 12, an initial check unit 13, and a remote driving start determination unit 14.

[0072] In step S110, first situation detection unit 11 detects a first situation. The first situation is a situation in which remote driving of vehicle 100 is required. A processing entity that realizes first situation detection unit 11 may be processor 160 of vehicle 100 or processor 360 of management device 300. For example, processor 160 of vehicle 100 may detect the first situation based on information acquired from sensor group 120. Alternatively, processor 360 of management device 300 may detect the first situation based on information acquired from infrastructure sensor 400. First situation detection unit 11 transmits information indicating that the first situation has been detected to urgency calculation unit 12, and the process proceeds to step S120.

[0073] In step S120, the urgency calculation unit 12 calculates the urgency. The processing entity that realizes the urgency calculation unit 12 may be the processor 160, the processor 260, or the processor 360. Alternatively, the urgency calculation unit 12 may be realized by cooperation of these processors.

[0074] For example, the processor 160 of the vehicle 100 acquires information about the stopping position of the vehicle 100 from a GNSS sensor of the sensor group 120. The processor 160 also acquires location information of a dangerous area from map information stored in the storage device 170. The processor 160 then calculates the urgency level based on whether the stopping position of the vehicle 100 is included in the dangerous area. As another example, the processor 360 of the management device 300 may detect the traveling position of the vehicle 100 based on infrastructure information obtained by the infrastructure sensor 400. The processor 160 or the processor 360 may calculate the urgency level by comparing the traveling position of the vehicle 100 with the target path. The urgency level calculation unit 12 transmits information about the calculated urgency level to the initial check unit 13, and the process proceeds to step S130.

[0075] In step S130, the initial check unit 13 performs an initial check. The processing entity that realizes the initial check unit 13 is the processor 260 of the remote driving terminal 200. When performing the initial check, the initial check unit 13 omits some of the initial check depending on the urgency calculated by the urgency calculation unit 12. Which steps of the initial check are omitted is determined depending on the urgency and priority, as described above. When the initial check is completed, the initial check unit 13 transmits the result of the initial check to the remote driving start determination unit 14, and the process proceeds to step S140.

[0076] In step S140, the remote operation start determination unit 14 determines whether or not remote operation can be started. The processing entity that realizes the remote operation start determination unit 14 may be processor 160, processor 260, or processor 360. Alternatively, the remote operation start determination unit 14 may be realized by cooperation of these. The remote operation start determination unit 14 determines whether or not remote operation can be started based on the result of the initial check received from the initial check unit 13. If the result of the initial check indicates that remote operation cannot be started (step S140; No), the series of processes ends.

[0077] On the other hand, if the result of the initial check indicates that remote driving can be started (step S140; Yes), the process proceeds to step S 150. In step S150, the remote driving start determination unit 14 starts remote driving of the vehicle 100.

[0078] 5.Effects 11 is a time chart showing the effect of omitting some of the initial checks by the remote operation system 1. The upper part is a time chart for a comparative example, and the lower part is a time chart showing an embodiment using the remote operation system 1. At time T1, a first situation is detected. In both the comparative example and this embodiment, the initial check begins in response to the detection of the first situation.

[0079] In the comparative example, after the initial check is completed, remote operation starts at time T3. On the other hand, in this embodiment, the time required for the initial check is shortened because part of the initial check is omitted. Therefore, remote operation starts at time T2, which is earlier than time T3.

[0080] As described above, according to the remote operation system 1 of this embodiment, some of the initial checks are omitted depending on the level of urgency, allowing the initial checks to be completed earlier. This shortens the time from when remote operation becomes necessary to when remote operation starts. Furthermore, when the level of urgency is particularly high, the amount of omission can be increased, further shortening the time until remote operation starts. Furthermore, which steps of the initial checks are omitted is determined according to priority. Particularly important steps are given a lower priority, so they are not omitted, preventing defects from being discovered after remote operation has started. This allows remote operation to be started smoothly.

[0081] 6. Application Examples As an example of a scene in which the remote driving system 1 according to this embodiment is applied, consider applying the remote driving system 1 to a scene of "intra-factory self-propelled transport."

[0082] In intra-factory autonomous transport, an autonomous vehicle automatically drives within the factory grounds. For example, an autonomous vehicle assembled at an assembly factory automatically drives from the assembly factory to the yard along a predetermined route. One or more infrastructure cameras are installed on the road from the assembly factory to the yard. By using these infrastructure cameras, the management device 300 of the remote driving system 1 can remotely monitor the autonomously driving autonomous vehicle.

[0083] If, for some reason, it becomes difficult for the autonomous vehicle to continue autonomous driving and a situation is detected in which the vehicle deviates from the target route, an initial check is performed to start remote driving. When the route the autonomous vehicle will travel is predetermined, such as in a factory self-driving transport system, the management device 300 can detect this situation based on route information stored in the storage device 370 and images captured by an infrastructure camera. When performing the initial check, parts of the initial check are omitted depending on the degree of deviation from the target route. By omitting parts of the initial check, remote driving can be started quickly. When an autonomous vehicle encounters a situation in which autonomous driving is difficult, it is possible to dispatch staff to the site and take over manual driving, but this requires time and effort. Omitting parts of the initial check and starting remote driving quickly is more convenient and also saves time and effort.

[0084] 7. Variations 7-1. First variant FIG. 12 is a diagram illustrating a first modified example. In the first modified example, the on-board camera includes a front camera that captures an image in front of the vehicle 100 and a side camera that captures an image to the side of the vehicle 100. The initial check also includes a camera check. The camera check is a check that the on-board camera is operating normally. In other words, the camera check is a check that the images captured by the on-board camera are being acquired normally. The camera check also includes a front camera check for the front camera and a side camera check for the side camera.

[0085] The priority of omission is set so that checking the side camera is higher than checking the front camera. It is considered that the remote operator X obtains more information from images captured by the front camera than from the side camera when remotely driving. Therefore, the priority of omission of checking the side camera is set higher. Note that, similarly, when the on-board camera includes cameras other than the side cameras, the priority of omission of checking the cameras other than the front camera is set higher than the priority of omission of checking the front camera.

[0086] As a specific example of a situation, consider a case where vehicle 100 makes an emergency stop at a railroad crossing. Because the urgency of the first situation is high, camera checks of cameras other than the front camera are omitted, and an initial check confirms that at least the front camera is operating normally. Remote operator X starts driving vehicle 100 by remote control, referring to at least the image captured by the front camera, and can allow vehicle 100 to quickly exit the railroad crossing.

[0087] 7-2. Second variant 13 is a block diagram showing an example of the functional configuration of the remote driving system 1 in the second modified example. In the second modified example, the remote driving system 1 includes a vehicle speed limiting unit 15 as a functional block.

[0088] The vehicle speed limiting unit 15 acquires information on whether or not the initial check has been omitted from the initial check unit 13. When a part of the initial check has been omitted, the vehicle speed limiting unit 15 sets an upper limit on the vehicle speed of the vehicle 100. The upper limit on the vehicle speed set here may be determined uniformly or may be set depending on the location where the vehicle 100 is located. For example, the upper limit may be set to a vehicle speed that is a predetermined amount lower than the speed limit of the road on which the vehicle 100 is located.

[0089] The vehicle speed limiting unit 15 may be included in the vehicle 100. In this case, the control device 150 controls the vehicle 100 so that the vehicle speed does not exceed the upper limit. Alternatively, the vehicle speed limiting unit 15 may be included in the remote driving terminal 200. In this case, the vehicle speed limiting unit 15 limits the input of the remote operator X so that the vehicle speed does not exceed the upper limit. Alternatively, the vehicle speed limiting unit 15 may be included in the management device 300, and information about the set vehicle speed may be transmitted to the vehicle 100 or the remote driving terminal 200.

[0090] In this modification, the vehicle speed is limited, so that the safety of the vehicle 100 during travel can be improved even if some of the initial checks are omitted. [Explanation of symbols]

[0091] 1 Remote operation system 11 First situation detection unit 12 Urgency Calculation Department 13 Initial Check Section 14 Remote operation start decision unit 15 Speed ​​limit section 100 vehicles 110 Communication equipment 120 sensors 130 Actuator 150 control device 160 processors 170 Storage device 200 Remote driving terminal 210 Communication equipment 220 Output Device 230 Remote control components 250 control device 260 processor 270 Storage device 300 Management device 310 Communication Equipment 350 control device 360 processor 370 Storage device 400 Infrastructure Sensors VCL vehicle information X Remote Operator

Claims

1. A remote driving system that remotely drives a vehicle based on an operation amount input to a remote driving terminal, one or more processors; the one or more processors: Detecting a first situation in which the remote operation of the vehicle is required; obtaining an urgency of the first situation; When the first situation is detected, an initial check is performed in the remote driving terminal to confirm that the remote driving can be started; Part of the initial check is omitted depending on the urgency. Remote driving system.

2. The remote driving system according to claim 1, The one or more processors increase the amount of omission of the initial check as the urgency increases. Remote driving system.

3. The remote driving system according to claim 2, the first situation is a situation in which a deviation amount of the vehicle from a target path exceeds a threshold; The degree of urgency is higher when the deviation amount is a second amount that is larger than the first amount than when the deviation amount is a first amount. Remote driving system.

4. The remote driving system according to claim 2, the degree of urgency when the stopping position of the vehicle is within a danger area is higher than the degree of urgency when the stopping position of the vehicle is outside the danger area; the danger area includes at least one of on tram tracks, in a railroad crossing, and a no-stopping zone; Remote driving system.

5. The remote driving system according to any one of claims 1 to 4, The initial check at the remote driving terminal includes at least confirming that communication with the vehicle is established and aligning the steering positions of the vehicle and the remote driving terminal. The steering position adjustment is omitted in favor of the confirmation of the communication establishment. Remote driving system.

6. The remote driving system according to any one of claims 1 to 4, The initial check in the remote driving terminal includes at least confirmation of establishment of communication with the vehicle and confirmation of operation of an actuator of the vehicle in response to the operation amount sent to the vehicle. The operation confirmation is omitted in favor of the confirmation of the communication establishment. Remote driving system.

7. The remote driving system according to claim 6, The operation amount includes a steering operation amount and a brake operation amount. The steering operation check is omitted in favor of the brake operation check. Remote driving system.

8. The remote driving system according to any one of claims 1 to 4, The initial check in the remote driving terminal includes at least confirmation of establishment of communication with the vehicle, alignment of the steering of the vehicle and the remote driving terminal, and confirmation of operation of the actuator of the vehicle in response to the operation amount sent to the vehicle. The steering position adjustment is omitted in favor of the confirmation of the communication establishment and the confirmation of the operation. Remote driving system.

9. The remote driving system according to any one of claims 1 to 4, The initial check in the remote driving terminal includes at least camera confirmation that an on-board camera mounted on the vehicle is operating normally; The vehicle-mounted camera includes a front camera that captures an image of a front side of the vehicle and a side camera that captures an image of a side side of the vehicle, The camera check for the side camera is omitted in favor of the camera check for the front camera. Remote driving system.

10. The remote driving system according to any one of claims 1 to 4, the one or more processors: If a part of the initial check is omitted, an upper limit is set on the vehicle speed. Remote driving system.

11. The remote driving system according to any one of claims 1 to 4, the one or more processors: Acquire infrastructure information including images captured by infrastructure cameras installed in an area in which the vehicle travels; Detecting the first situation based on the infrastructure information Remote driving system.

12. A remote driving terminal that remotely drives a vehicle based on an operation amount input by a remote operator, one or more processors; the one or more processors: obtaining information indicating that a first situation requiring the remote operation of the vehicle has been detected; obtaining an urgency of the first situation; When the first situation is detected, an initial check is performed to confirm that the remote driving terminal is capable of starting the remote driving; Part of the initial check is omitted depending on the urgency. Remote driving terminal.

13. A method for remotely driving a vehicle based on an operation amount input to a remote driving terminal, comprising: Detecting a first situation in which the remote operation of the vehicle is required; obtaining an urgency of the first situation; When the first situation is detected, an initial check is performed in the remote driving terminal to confirm that the remote driving can be started. omitting a part of the initial check depending on the urgency; A method comprising:

Images