Remote driving system, remote driving terminal, and method for remote driving

Abstract

To provide a technique which makes it possible to finish an initial check early and start remote driving smoothly in a remote driving system.SOLUTION: The present disclosure relates to a remote driving system for performing remote driving of a vehicle on the basis of a manipulated variable input to a remote driving terminal. The remote driving system detects a second situation showing a sign of a first situation in which the remote driving of the vehicle is required. When the second situation is detected, the remote driving system performs at least part of an initial check for checking that the remote driving can be started at the remote driving terminal, before the first situation is detected. When the first situation is detected, the remote driving system starts the remote driving.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a technique for remotely driving a vehicle. [Background technology]

[0002] Patent Document 1 discloses a vehicle control device. When a transition from one driving state to another driving state is scheduled, the vehicle control device notifies occupants and other vehicles of information related to the transition of the driving state. The driving states of a vehicle include manual driving, automatic driving, and remote driving. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-018486 Summary of the Invention [Problem to be solved by the invention]

[0004] Systems are known that remotely drive a vehicle based on operation amounts input by a remote operator to a remote driving terminal. Such systems are sometimes called remote driving systems. The remote driving system starts remote driving when the vehicle is in a situation requiring remote driving. However, at this time, the remote driving system does not immediately start remote driving. Before starting remote driving, the remote driving system first performs an initial check to confirm that remote driving can be started. The remote driving system then starts remote driving only after the results of the initial check are positive. Therefore, in order to start remote driving smoothly, it is necessary to complete the initial check as soon as possible.

[0005] One object of the present disclosure is to provide a technology that enables an initial check to be completed early and remote operation to be started smoothly in a remote operation system. [Means for solving the problem]

[0006] The first aspect relates to a remote driving system that remotely drives a vehicle based on an operation amount input to a remote driving terminal. The remote operation system includes one or more processors. The one or more processors detect a second condition that is predictive of a first condition that requires remote operation of the vehicle. When the second condition is detected, the one or more processors perform at least a portion of an initial check to verify that remote operation can be initiated at the remote operation terminal before the first condition is detected. The one or more processors initiate remote operation when a first condition is detected.

[0007] The second aspect relates to a remote driving terminal that remotely drives a vehicle based on an operation amount input by a remote operator. The remote operating terminal comprises one or more processors. The one or more processors obtain information indicating that a second condition has occurred that is predictive of a first condition that requires remote operation of the vehicle. The one or more processors perform at least a portion of an initial check to verify that the remote driving terminal is capable of initiating remote driving before the first condition occurs when the second condition occurs. The one or more processors initiate remote operation when a first condition occurs.

[0008] A third aspect relates to a method for remotely driving a vehicle based on an operation amount input to a remote driving terminal. The above method is Detecting a second condition that is predictive of a first condition that requires remote operation of the vehicle; When the second situation is detected, performing at least a part of an initial check to confirm that remote operation can be initiated at the remote operation terminal before the first situation is detected; Initiating remote operation when a first condition is detected; Includes. [Effects of the Invention]

[0009] According to the present disclosure, a second situation is detected that indicates a first situation that requires remote driving of the vehicle. Then, when the second situation is detected, at least a part of an initial check is performed before the first situation occurs. By performing at least a part of the initial check in advance, the time from when the first situation is detected to when the initial check is completed is shortened, allowing remote driving to be started smoothly. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating a configuration example of a remote driving system according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the configuration of a vehicle according to an embodiment of the present invention; [Figure 3] 2 is a block diagram showing an example of the configuration of a remote driving terminal according to the present embodiment; FIG. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a management device according to the present embodiment. [Figure 5] FIG. 10 is a diagram for explaining a specific example of an initial check. [Figure 6] FIG. 10 is a schematic diagram showing a state of steering position adjustment. [Figure 7] 1A and 1B are schematic diagrams illustrating examples of a first situation and a second situation. [Figure 8] 1 is a block diagram showing an example of a functional configuration of a remote operation system according to an embodiment of the present invention; [Figure 9] 4 is a flowchart illustrating an example of processing performed by the remote operation system according to the present embodiment. [Figure 10] 5 is a time chart illustrating the effect of the processing performed by the remote operation system according to the present embodiment. [Figure 11] 10 is a flowchart illustrating an example of processing in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0012] 1. Remote operation system 1-1. System configuration 1 is a schematic diagram showing an example configuration of a remote driving system 1 according to this embodiment. The remote driving system 1 includes a vehicle 100, a remote driving terminal 200, and a management device 300. The vehicle 100, the remote driving terminal 200, and the management device 300 can communicate with each other via a communication network.

[0013] The vehicle 100 is a vehicle that is to be remotely driven by a remote operator X. The vehicle 100 may be an autonomous vehicle. The remote driving terminal 200 is a terminal device that the remote operator X operates when remotely driving the vehicle 100. The remote driving terminal 200 can also be called a remote cockpit. The management device 300 manages the remote driving system 1. Typically, the management device 300 is a management server on the cloud. The management device 300 may be configured from multiple servers that perform distributed processing.

[0014] The remote driving system 1 may also include an infrastructure sensor 400. The infrastructure sensor 400 includes an infrastructure camera. The infrastructure sensor 400 may also include a rainfall sensor, etc. The infrastructure sensor 400 is installed in an area where the vehicle 100 travels. The infrastructure sensor 400 and the management device 300 can communicate with each other via a communication network. The infrastructure sensor 400 may also communicate with the vehicle 100 and the remote driving terminal 200 via the management device 300 or directly.

[0015] 2 to 4 are block diagrams showing examples of the configuration of the vehicle 100, the remote driving terminal 200, and the management device 300, respectively.

[0016] 2 shows an example configuration of a vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, an actuator 130, and a control device 150. In this example, the vehicle 100 is equipped with an autonomous driving system and is capable of autonomous driving.

[0017] The communication device 110 communicates with devices outside the vehicle 100. The communication devices 110 communicate with include the remote driving terminal 200 and the management device 300.

[0018] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include an on-board camera, a LIDAR (Laser Imaging Detection and Ranging), a radar, etc. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, a brake oil pressure sensor, etc. The position sensor detects the position and direction of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System) sensor. The sensor group 120 may also include a rainfall sensor.

[0019] The actuators 130 include a steering actuator, a drive actuator, and a brake actuator. The steering actuator steers the wheels. For example, the steering actuator includes an electric power steering (EPS) device. The drive actuator is a power source that generates drive force. Examples of drive actuators include an engine, an electric motor, and an in-wheel motor. The brake actuator generates braking force. For example, the brake actuator controls brake hydraulic pressure to activate the brakes.

[0020] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processors 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processor 160 executes various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various programs and various information required for processing by the processor 160. The processor 160 executes the programs stored in the storage device 170, thereby realizing the functions of the control device 150. Examples of the storage device 170 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The control device 150 may include one or more ECUs (Electronic Control Units).

[0021] The control device 150 controls the actuators 130 to control the traveling of the vehicle 100. The control device 150 also acquires vehicle information VCL from the sensor group 120. The vehicle information VCL includes recognition sensor information indicating the recognition results from the recognition sensors, vehicle state information obtained from the vehicle state sensors, and position information obtained from the position sensors. The recognition sensor information includes images captured by an on-board camera. The vehicle state information includes speed information, acceleration information, steering angle information, brake oil pressure information, etc. obtained from a speed sensor, an acceleration sensor, a steering angle sensor, a brake oil pressure sensor, etc. The vehicle information VCL may further include highly accurate position information obtained by a self-localization process. The control device 150 can acquire highly accurate position information by performing a self-localization process using map information and recognition sensor information stored in the storage device 170. The vehicle information VCL acquired by the control device 150 can be used for automatic driving or remote driving of the vehicle 100.

[0022] For example, the control device 150 controls the automatic driving of the vehicle 100 based on the vehicle information VCL. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the vehicle information VCL. Furthermore, the control device 150 generates a target path required for the vehicle 100 to drive according to the driving plan based on the vehicle information VCL. The target path is a set of target positions for the vehicle 100. The target path may be set to extend along the center of the lane. Then, the control device 150 controls the driving of the vehicle 100 so that the vehicle 100 follows the target path.

[0023] Furthermore, the control device 150 communicates via the communication device 110 and can transmit the vehicle information VCL to the remote driving terminal 200 and the management device 300. At least when the vehicle 100 is being remotely driven, the vehicle information VCL is transmitted to the remote driving terminal 200. The vehicle information VCL transmitted to the remote driving terminal 200 is referenced by the remote operator X, who then performs remote driving.

[0024] 3 shows an example of the configuration of the remote driving terminal 200. The remote driving terminal 200 includes a communication device 210, an output device 220, a remote control member 230, and a control device 250.

[0025] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0026] The output device 220 outputs various information and presents it to the remote operator X. For example, the output device 220 includes a display device. The display device displays the various information toward the remote operator X. As another example, the output device 220 may include a speaker.

[0027] The remote control member 230 is a member that the remote operator X operates when remotely driving the vehicle 100. For example, the remote control member includes a steering operation member, an accelerator pedal, a brake pedal, a turn signal, etc. The steering operation member is, for example, a steering wheel. The remote control member may include a touch panel, buttons, etc. The amount of operation input by the remote operator X when remotely driving the vehicle 100 is detected by a sensor installed in the remote control member 230.

[0028] The control device 250 controls the remote operation terminal 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as processor 260) and one or more storage devices 270 (hereinafter simply referred to as storage device 270). The processor 260 executes various processes. For example, the processor 260 includes a CPU. The storage device 270 stores various programs and various information required for processing by the processor 260. The processor 260 executes the programs stored in the storage device 270, thereby realizing the functions of the control device 250. Examples of the storage device 270 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0029] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information VCL transmitted from the vehicle 100. The control device 250 presents the vehicle information VCL, including images, to the remote operator X by displaying the vehicle information VCL on a display device. The remote operator X can recognize the state of the vehicle 100 and the surrounding circumstances based on the vehicle information VCL displayed on the display device.

[0030] Furthermore, the control device 250 may acquire infrastructure information detected by the infrastructure sensor 400 via the management device 300 or directly. The infrastructure information acquired by the control device 250 may include, for example, images obtained by an infrastructure camera capturing images of the vehicle 100 and its surroundings. The acquired infrastructure information is displayed on a display device. The remote operator X may recognize the state of the vehicle 100 and the surrounding circumstances by referring to the infrastructure information. By presenting the remote operator X with images captured by the infrastructure camera in addition to images captured by the on-board camera, the accuracy of remote driving and convenience for the remote operator X are improved.

[0031] The control device 250 also generates remote driving information REM based on the amount of operation input by the remote operator X. The remote driving information REM is information for controlling the vehicle 100 by remote driving. The remote driving information REM includes the amount of operation of the remote operation member 230 input by the remote operator X. The control device 250 transmits the remote driving information REM to the vehicle 100 as necessary.

[0032] 4 shows an example of the configuration of the management device 300. The management device 300 includes a communication device 310 and a control device 350.

[0033] The communication device 310 communicates with the vehicle 100 , the remote driving terminal 200 , and the infrastructure sensor 400 .

[0034] The control device 350 controls the management device 300. The control device 350 includes one or more processors 360 (hereinafter simply referred to as processors 360) and one or more storage devices 370 (hereinafter simply referred to as storage devices 370). The processor 360 executes various processes. For example, the processor 360 includes a CPU. The storage device 370 stores various programs and various information required for processing by the processor 360. The processor 360 executes the programs stored in the storage device 370, thereby realizing the functions of the control device 350. Examples of the storage device 370 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0035] The control device 350 communicates with the vehicle 100 and the remote driving terminal 200 via the communication device 310. Furthermore, the control device 350 communicates with the infrastructure sensor 400 via the communication device 310 as necessary to acquire infrastructure information. The infrastructure information acquired by the control device 350 from the infrastructure sensor 400 includes images captured by an infrastructure camera.

[0036] The control device 350 may also manage various types of information related to the remote driving of the vehicle 100. For example, the control device 350 may acquire weather forecast information, traffic information for the area in which the vehicle 100 is traveling, and manage this information. The weather forecast information includes a prediction of future weather in the area in which the vehicle 100 is traveling. The weather forecast information may also include information about the current weather in the area in which the vehicle 100 is traveling.

[0037] 1-2. Remote operation using a remote operation system The remote driving system 1 remotely drives the vehicle 100 based on the operation amount input by the remote operator X to the remote driving terminal 200. When the remote driving system 1 detects a "first situation", remote driving of the vehicle 100 begins. The first situation is a situation in which remote driving of the vehicle 100 is required.

[0038] When the remote driving system 1 detects a first situation, the management device 300 assigns a remote operator X from among multiple candidates to the vehicle 100. The management device 300 manages the assignment relationship between the vehicle 100 and the remote operator X, and provides information about the assignment relationship to the vehicle 100 and the remote driving terminal 200. Upon receiving the information about the assignment relationship, the vehicle 100 and the remote driving terminal 200 establish communication.

[0039] During remote driving, the vehicle 100 and the remote driving terminal 200 transmit and receive information via a communication network. Note that communication between the vehicle 100 and the remote driving terminal 200 may be performed directly or via the management device 300.

[0040] The vehicle 100 transmits the vehicle information VCL to the remote driving terminal 200. The remote driving terminal 200 receives the vehicle information VCL transmitted from the vehicle 100 and presents it to the remote operator X. For example, the remote driving terminal 200 presents the vehicle information VCL by displaying an image on the display device of the output device 220. The remote operator X looks at the displayed information, recognizes the situation around the vehicle 100, and remotely drives the vehicle 100 by operating the remote control member 230.

[0041] The control device 250 generates remote driving information REM based on the amount of operation of the remote control member 230 input by the remote operator X. Then, the control device 250 transmits the remote driving information REM to the vehicle 100 via the communication device 210.

[0042] The vehicle 100 receives the remote driving information REM transmitted from the remote driving terminal 200. The vehicle 100 controls the traveling of the vehicle in accordance with the received remote driving information REM. In this way, the vehicle 100 is remotely driven.

[0043] 2. Initial check As described above, remote driving of vehicle 100 is initiated by remote driving system 1 when the first situation is detected. However, just because the first situation is detected does not necessarily mean that remote driving can be initiated immediately. Before remote driving can be initiated, an "initial check" must be performed to confirm that remote driving can be initiated. Based on the results of the initial check, remote driving system 1 confirms that remote driving can be initiated before starting remote driving.

[0044] Fig. 5 shows a specific example of the initial check. The initial check is performed in the remote operation terminal 200. As shown in Fig. 5, the initial check includes several steps.

[0045] The first step of the initial check is a "communication establishment confirmation" to confirm that communication between the vehicle 100 and the remote driving terminal 200 has been established normally. More specifically, the communication establishment confirmation confirms that the vehicle 100 and the remote driving terminal 200 assigned to that vehicle 100 are normally connected and are in a state where communication is possible. For example, the remote driving terminal 200 transmits a test signal to the vehicle 100 to confirm communication establishment. The test signal may be a Ping. If the remote driving terminal 200 normally receives a response signal sent from the vehicle 100 in response to the test signal, it is determined that communication has been established normally. If it is determined that communication has been established normally, this means that necessary information can be accurately exchanged between the vehicle 100 and the remote driving terminal 200.

[0046] Once the communication establishment confirmation is complete, it is guaranteed that the remote driving terminal 200 can normally receive the vehicle information VCL from the vehicle 100. The initial check proceeds to the next step, where "steering position alignment" is performed. Steering position alignment means matching the steering angle of the vehicle 100 with the steering angle of the steering operation member of the remote driving terminal 200. The steering angle of the vehicle 100 is the steering angle of the steering wheel of the vehicle 100. Alternatively, the steering angle of the vehicle 100 may be calculated from the steering angle of the wheels of the vehicle 100. Information on the steering angle of the steering wheel and the steering angle of the wheels of the vehicle 100 is obtained from the vehicle information VCL sent from the vehicle 100 to the remote driving terminal 200. The steering angle of the steering operation member of the remote driving terminal 200 is detected by a sensor installed on the steering operation member.

[0047] FIG. 6 is a diagram showing an example of a screen displayed on the display device of the remote driving terminal 200 during steering position adjustment. The driving trajectory 221 shown by a dotted line represents the driving trajectory of the vehicle 100 calculated from the current steering angle of the vehicle 100. On the other hand, the driving trajectory 222 shown by a solid line represents the driving trajectory of the vehicle 100 calculated from the steering angle of the steering operation member of the remote driving terminal 200. The remote driving terminal 200 draws the driving trajectories 221 and 222 so that they are superimposed on the image included in the vehicle information VCL. The remote driving terminal 200 then displays the image with the driving trajectories 221 and 222 superimposed on it on the display device. Steering position adjustment is performed by adjusting the steering angle of the steering operation member so that these two driving trajectories 221 and 222 coincide with each other. Typically, the remote operator X performs the alignment by moving the steering operation member so that the driving trajectory 221 and the driving trajectory 222 coincide with each other. The coincidence of the two trajectories may be determined automatically by the remote driving terminal 200 or manually by the remote operator X. When the determination is made manually, the remote operator X inputs by means of a button or the like on the remote control member 230 that the two trajectories match.

[0048] Completion of the steering position alignment completes preparations for reflecting the operation amount of the steering operation member in the steering of the wheels on the vehicle 100 side. After that, it becomes possible to transmit remote driving information REM from the remote driving terminal 200 to the vehicle 100.

[0049] Referring again to Figure 5, once steering position alignment is complete and the steering angle of the steering operation member corresponds to the steering angle of the wheels, an operation check is performed. The operation check is performed to confirm that the operation amount of the remote operator X transmitted to the vehicle 100 as the remote driving information REM is reflected in the operation of the actuator 130, that is, to confirm that the actuator 130 operates in response to the operation amount of the remote operator X. The operation check includes an operation check for the brakes and an operation check for the steering. Note that the order in which the operation check for the brakes and the operation check for the steering are performed does not matter.

[0050] The brake operation check is performed to confirm that the amount of brake operation input by the remote operator X to the brake pedal of the remote control member 230 is reflected in the operation of the brake actuator of the vehicle 100. The remote driving terminal 200 transmits a test signal to the vehicle 100 requesting the operation of the brake actuator. For example, the remote driving terminal 200 transmits remote driving information REM (test signal) including the amount of operation of the brake pedal of the remote control member 230 to the vehicle 100. If the brake actuator operates normally in response to the test signal, the brake hydraulic pressure should change normally. Therefore, the remote driving terminal 200 receives vehicle information VCL from the vehicle 100 and monitors the brake hydraulic pressure information included in the vehicle information VCL. If the brake hydraulic pressure changes as expected in response to the transmission of the test signal, the remote driving terminal 200 determines that the brake actuator is operating. If the brake actuator is operating normally, the brake operation check is completed.

[0051] The steering operation check is performed to confirm that the amount of operation input by the remote operator X to the steering operation member of the remote control unit 230 is reflected in the operation of the steering actuator, thereby steering the wheels of the vehicle 100. The remote driving terminal 200 transmits a test signal to the vehicle 100 requesting the operation of the steering actuator. For example, the remote driving terminal 200 transmits remote driving information REM (test signal) including the amount of operation of the steering operation member of the remote control unit 230 to the vehicle 100. If the steering actuator operates normally in response to the test signal, the steering angle of the vehicle 100 should change normally. Therefore, the remote driving terminal 200 receives vehicle information VCL from the vehicle 100 and monitors the steering angle information included in the vehicle information VCL. If the steering angle of the vehicle 100 changes as expected in response to the transmission of the test signal, the remote driving terminal 200 determines that the steering actuator is operating. If the steering actuator is operating normally, the steering operation check is completed.

[0052] The above is a specific example of the initial check. Before starting remote operation, the remote driving system 1 performs the initial check illustrated in FIG. 5. Then, the remote driving system 1 starts remote operation after the initial check confirms that remote operation can be started. In other words, the remote driving system 1 cannot start remote operation until the initial check is completed. Therefore, if it takes a long time from the detection of the first situation to the completion of the initial check, the start of remote operation will be delayed by that amount.

[0053] However, it is desirable that the time from when the first situation is detected until remote driving is started be short. In particular, depending on the state of vehicle 100, it may be necessary to start remote driving of vehicle 100 early. For example, if vehicle 100 stops at a railroad crossing, it is necessary to start remote driving early. In order to start remote driving early, it is necessary to complete the initial check early.

[0054] Therefore, the remote driving system 1 according to this embodiment detects a "second situation." The second situation is a situation that indicates a precursor to the first situation. In other words, the second situation indicates that remote driving of the vehicle 100 is not currently required, but may be required in the future. When the second situation is detected, the remote driving system 1 performs at least a part of the initial check before the first situation is detected. By performing at least a part of the initial check before the first situation is detected, the time from the detection of the first situation to the completion of the initial check can be shortened, allowing remote driving to begin earlier.

[0055] As in the case of the first situation, when the remote driving system 1 detects the second situation, the management device 300 assigns a remote operator X from among multiple candidates to the vehicle 100. The management device 300 manages the assignment relationship between the vehicle 100 and the remote operator X, and provides information about the assignment relationship to the vehicle 100 and the remote driving terminal 200. Upon receiving the information about the assignment relationship, the vehicle 100 and the remote driving terminal 200 establish communication. Thereafter, the remote driving system 1 starts at least a part of an initial check.

[0056] 3. First and second situations Specific examples of the first situation and the second situation will be described. The first situation and the second situation may be detected by the vehicle 100 or by the management device 300. Furthermore, the entity that detects the first situation and the entity that detects the second situation may be the same or different.

[0057] The first situation may be, for example, a situation in which a remote driving request (RFO: Request for Operation) is transmitted from the vehicle 100 to the management device 300. For example, when the autonomous driving system of the vehicle 100 determines that it is difficult to continue autonomous driving, the remote driving request is transmitted from the vehicle 100 to the management device 300. Note that the remote driving request from the vehicle 100 may be transmitted while the vehicle 100 is traveling, or may be transmitted while the vehicle 100 is stopped.

[0058] Alternatively, the remote driving system 1 may detect the first situation based on infrastructure information acquired from the infrastructure sensor 400. For example, when an abnormal state of the vehicle 100 is detected from an image captured by an infrastructure camera of the infrastructure sensor 400, it is predicted that it is difficult for the vehicle 100 to continue autonomous driving. For example, the abnormal state of the vehicle 100 is a state in which the vehicle 100 continues to travel while deviating from its lane by more than a threshold. As another example, the abnormal state is a state in which the vehicle 100, which should be traveling, is stopped. The remote driving system 1 may detect such an abnormal state of the vehicle 100 as the first situation.

[0059] Furthermore, the remote driving system 1 may detect the second situation based on infrastructure information acquired from the infrastructure sensor 400. For example, a situation in which a predictive state is detected from an image captured by an infrastructure camera of the infrastructure sensor 400 may be regarded as the second situation. An example of the predictive state is a state in which the vehicle 100 is traveling unsteadily. As another example, the predictive state may be a state in which the vehicle speed of the vehicle 100 is extremely low. For example, if the vehicle 100 is traveling unsteadily, there is a possibility that a malfunction has occurred in the control of the vehicle 100, and there is a possibility that remote driving of the vehicle 100 will be required in the future. Therefore, such a state may be regarded as the second situation.

[0060] As another example, the remote driving system 1 may detect the first situation and the second situation from the weather of the place where the vehicle 100 is traveling. This is an example in which the vehicle 100 is an autonomous vehicle.

[0061] For example, the first situation may be that the amount of rainfall in the area where the vehicle 100 is traveling is equal to or greater than a first threshold. When the amount of rainfall is large, it becomes difficult for the vehicle 100 to continue autonomous driving, and remote driving is required, and therefore such a situation is considered to be the first situation. The remote driving system 1 can detect such a situation using a rainfall sensor in the infrastructure sensor 400 or a rainfall sensor in the sensor group 120. For example, the remote driving system 1 may detect, as the first situation, a situation indicating that the average amount of rainfall in the area where the vehicle 100 is traveling is greater than a first threshold.

[0062] In this case, the second situation may be that the amount of rainfall in the location where the vehicle 100 is traveling is equal to or greater than a second threshold and less than a first threshold. The second threshold is smaller than the first threshold. For example, the remote driving system 1 may detect, as the second situation, a situation indicating that the average amount of rainfall in the area where the vehicle 100 is traveling is greater than the second threshold.

[0063] Alternatively, the second situation may be a situation in which it is predicted that rain will start falling in the location where the vehicle 100 is traveling. If such a situation is detected, there is a possibility that the amount of rainfall in the location where the vehicle 100 is traveling will be equal to or greater than a threshold in the future. The remote driving system 1 can acquire the location where the vehicle 100 will travel in the future from the vehicle information VCL. In addition, the prediction of rainfall is obtained from weather forecast information. The remote driving system 1 compares this information and, if it is predicted that rain will start falling in the location where the vehicle 100 is traveling, detects this as the second situation.

[0064] As yet another example, the remote driving system 1 may detect the first situation and the second situation based on deviation of the vehicle 100 from a target path during autonomous driving control. This is also an example in which the vehicle 100 is an autonomous vehicle. The target path is a path that the vehicle 100 should follow during autonomous driving control. As shown in FIG. 7 , there are situations in which the traveling position of the vehicle 100 deviates from the target path and the deviation amount is greater than a first threshold. In such a situation in which the vehicle 100 deviates significantly from the target path, it is difficult to continue autonomous driving, and it is considered necessary to start remote driving of the vehicle 100. Therefore, the remote driving system 1 may detect such a situation as the first situation.

[0065] In addition, at this time, the remote driving system 1 may detect a situation in which the deviation amount becomes larger than a second threshold as the second situation. The second threshold is set to be smaller than the first threshold. When the deviation amount becomes larger than the second threshold, there is a possibility that the autonomous driving system will subsequently return the vehicle 100 to the target path, and the remote driving system may then detect the second situation. Turn However, there is a possibility that the departure amount will increase further and exceed the first threshold, making it necessary to remotely drive the vehicle 100. Therefore, such a situation may be detected as the second situation.

[0066] The deviation amount can also be expressed as the distance between the vehicle 100 and the target path. The remote driving system 1 can calculate the deviation amount by comparing the position of the vehicle 100 with the target path. The remote driving system 1 may obtain the position of the vehicle 100 from position information of the vehicle 100 obtained from the vehicle information VCL, or may obtain the position from an image captured by an infrastructure camera of the infrastructure sensor 400. Information regarding the target path that the vehicle 100 should follow is obtained from the vehicle information VCL.

[0067] 4. Processing example The processing performed by the remote operation system 1 will be described with reference to FIGS.

[0068] 8 is a block diagram showing an example of the functional configuration of the remote driving system 1. The remote driving system 1 includes, as functional blocks, a first situation detection unit 11, a second situation detection unit 12, an initial check unit 13, and a remote driving start determination unit 14.

[0069] The first situation detection unit 11 detects a first situation. The first situation is a situation in which remote driving of the vehicle 100 is required. The processing entity that realizes the first situation detection unit 11 may be the processor 160 of the vehicle 100 or the processor 360 of the management device 300. For example, the processor 160 of the vehicle 100 may detect the first situation based on information acquired from the sensor group 120. Alternatively, the processor 360 of the management device 300 may detect the first situation based on information acquired from the infrastructure sensor 400. When the first situation detection unit 11 detects the first situation, it transmits information indicating that the first situation has been detected to the initial check unit 13.

[0070] The second situation detection unit 12 detects a second situation. The processing entity that realizes the second situation detection unit 12 may be the processor 160 or the processor 360. For example, the processor 160 of the vehicle 100 may detect the second situation based on weather forecast information acquired from the management device 300. Alternatively, the processor 360 of the management device 300 may detect the second situation based on information acquired from the infrastructure sensor 400. When the second situation detection unit 12 detects the second situation, it transmits information indicating that the second situation has been detected to the initial check unit 13.

[0071] The initial check unit 13 performs an initial check. The processing entity that realizes the initial check unit 13 is the processor 260 of the remote driving terminal 200. The initial check unit 13 performs the initial check when a first situation is detected based on information transmitted from the first situation detection unit 11. Furthermore, when a second situation is detected based on information transmitted from the second situation detection unit 12, the initial check unit 13 performs at least a part of the initial check before the first situation is detected. When the initial check is completed, the initial check unit 13 transmits the result of the initial check to the remote driving start determination unit 14.

[0072] The remote driving start determination unit 14 determines whether to start remote driving. The processing entity that realizes the remote driving start determination unit 14 may be processor 160, processor 260, or processor 360. Alternatively, the remote driving start determination unit 14 may be realized by a collaboration of these. The remote driving start determination unit 14 determines whether to start remote driving based on the result of the initial check received from the initial check unit 13 and information about the first situation received from the first situation detection unit 11. If the first situation has been detected and the result of the initial check indicates that remote driving can be started, the remote driving start determination unit 14 starts remote driving of the vehicle 100. Note that the remote driving start determination unit 14 does not start remote driving unless the first situation is detected. In other words, even if the initial check is completed before the first situation is detected, remote driving actually starts after the first situation is detected.

[0073] FIG. 9 is a flowchart showing an example of processing performed by the remote driving system 1 when the second situation is detected.

[0074] In step S110, the remote driving system 1 detects a second situation. The processing of step S110 is executed by the second situation detection unit 12. When the second situation is detected, communication is established between the vehicle 100 and the remote driving terminal 200. Then, the processing proceeds to step S120.

[0075] In step S120, the remote driving system 1 starts an initial check. By starting the initial check in advance in step S120, the remote driving system 1 can perform at least a part of the initial check before the first situation is detected. The processing of step S120 is executed by the initial check unit 13. Once the initial check is started and at least a part of the initial check is performed, the processing proceeds to step S125.

[0076] In step S125, the remote driving system 1 determines whether the second situation has been resolved before the first situation has been detected. If the second situation has been resolved before the first situation has been detected (step S125; Yes), the remote driving system 1 ends the initial check without starting remote driving. Otherwise (step S125; No), the process proceeds to step S130.

[0077] In step S130, the remote operation system 1 determines whether or not a first situation has been detected. The processing of step S130 is executed by the first situation detection unit 11. If the first situation has been detected (step S130; Yes), the processing proceeds to step S140. On the other hand, if the first situation has not been detected (step S130; No), the processing returns to step S125 again. During this time, the initial check continues.

[0078] In step S140, the remote driving system 1 performs the remaining initial checks, if any. When the initial checks are completed, the process proceeds to step S150. The process of step S140 is executed by the initial check unit 13. Note that, because at least a portion of the initial checks has already been performed before step S130, the remote driving system 1 only needs to perform the remaining initial checks in step S140. Therefore, the time until the process of step S150 starts is shorter than the time required when the entire initial check process is performed in step S140. Furthermore, the initial checks may be completed before step S130. In this case, step S140 is skipped and the process proceeds to step S150. Because step S140 is skipped, the time required until the process of step S150 starts is further shortened.

[0079] In step S150, the remote driving system 1 determines whether to start remote driving. The processing of step S150 is executed by the remote driving start determination unit 14. If the result of the initial check indicates that remote driving can be started, the remote driving system 1 starts remote driving. Once the processing of step S150 is performed, the series of processing steps ends.

[0080] The series of processes shown in the flowchart of FIG. 9 is an example of the process when the second situation is detected. However, a situation may arise in which the first situation is detected without the second situation being detected. For example, a situation may arise in which heavy rain suddenly begins to fall in the area where the vehicle 100 is traveling, even though the weather forecast information did not predict rain. In such a case, in response to the detection of the first situation, an initial check is performed as usual. Then, if the initial check confirms that remote driving can be started, remote driving is started.

[0081] 5.Effects 10 is a time chart showing the effect of the processing performed by the remote driving system 1. The top time chart is a time chart for a comparative example. Example 1 and Example 2 are examples in which the second situation is detected by the remote driving system 1.

[0082] In the comparative example, example 1, and example 2, the first situation is detected at time T3. In the comparative example, an initial check is started in response to the detection of the first situation. Then, in the comparative example, remote operation starts at time T5 after the initial check is completed.

[0083] On the other hand, in Example 1, the remote driving system 1 detects the second situation at time T1, which is before time T3. Then, the remote driving system 1 starts the initial check in response to the detection of the second situation. This allows the initial check to be completed before the first situation is detected. Because the initial check has already been completed, the remote driving system 1 can start remote driving immediately after the first situation is detected at time T3.

[0084] In Example 2, the second situation is detected at time T2, which is after time T1 but before time T3. As in Example 1, the remote driving system 1 starts the initial check in response to the detection of the second situation. As a result, at time T3, when the first situation is detected, part of the initial check has already been performed. Because part of the initial check has already been performed, the time from when the first situation is detected to when the remaining initial check is completed is shorter than the time until the initial check is completed in the comparative example. In this way, the remote driving system 1 can start remote driving at time T4, which is earlier than time T5.

[0085] In this way, with the remote driving system 1 according to this embodiment, upon detection of the second situation, at least a part of the initial check is performed before the first situation is detected. This allows the initial check to be completed early, shortening the time from when remote driving becomes necessary to when remote driving starts. This allows remote driving to start smoothly.

[0086] 6. Application Examples As an example of a scene in which the remote driving system 1 according to this embodiment is applied, consider applying the remote driving system 1 to a scene of "intra-factory self-propelled transport."

[0087] In intra-factory autonomous transport, an autonomous vehicle automatically drives within the factory grounds. For example, an autonomous vehicle assembled at an assembly factory automatically drives from the assembly factory to the yard along a predetermined route. One or more infrastructure cameras are installed on the road from the assembly factory to the yard. By using these infrastructure cameras, the management device 300 of the remote driving system 1 can remotely monitor the autonomously driving autonomous vehicle.

[0088] If, for some reason, the vehicle deviates significantly from the target route and the deviation exceeds a first threshold, it is determined that continuing autonomous driving is difficult, and remote driving is initiated. In cases where the route the autonomous vehicle will travel is predetermined, such as in autonomous transport within a factory, the management device 300 can detect this situation based on route information stored in the storage device 370 and images captured by an infrastructure camera. At this time, at least some of the initial checks required to start remote driving have already been performed when the deviation exceeds the second threshold. Therefore, the remote driving system 1 can quickly begin remote driving. When an autonomous vehicle encounters a situation where autonomous driving is difficult, it is possible to dispatch staff to the site and take over manual driving, but this requires time and effort. Remote driving is more convenient and also reduces time and effort.

[0089] 7. Variations A modified example will now be described. In this modified example, a "standby state" is defined as the state of the remote driving terminal 200. As in the above-described embodiment, the remote driving system 1 starts an initial check when the second situation is detected. Then, if the initial check is completed before the first situation is detected and the result indicates that remote driving can be started, the remote driving system 1 transitions the remote driving terminal 200 to the standby state. The standby state indicates that the remote driving terminal 200 is ready for remote driving. In other words, the standby state is a state in which remote driving can be started immediately once the first situation is detected. In this modified example, the standby state of the remote driving terminal 200 is also terminated under certain conditions. Specifically, the remote driving system 1 terminates the standby state when the second situation is no longer detected and the first situation is not detected either.

[0090] 11 is a flowchart showing an example of processing performed by the remote operation system 1 in the modified example. Note that processing similar to that in FIG. 9 is omitted in FIG. 11. The processing up to step S125 is similar to that in the flowchart in FIG.

[0091] In step S130, the remote operation system 1 determines whether or not a first situation has been detected. If the first situation has been detected (step S130; Yes), the process proceeds to step S140, as in Fig. 9. The process from step S140 onwards is the same as in Fig. 9. On the other hand, if the first situation has not been detected (step S130; No), the process proceeds to step S160.

[0092] In step S160, the remote driving system 1 determines whether remote driving of the vehicle 100 can be started based on the result of the initial check. If the result of the initial check indicates that remote driving can be started (step S160; Yes), the process proceeds to step S170. On the other hand, if the result of the initial check indicates that remote driving cannot be started or if the initial check has not been completed (step S160; No), the process returns to step S125. During this time, the initial check continues.

[0093] In step S170, the remote driving system 1 transitions the remote driving terminal 200 to a standby state. The standby state indicates that preparation for remote driving is complete. When the remote driving terminal 200 transitions to the standby state, the process proceeds to step S180.

[0094] In step S180, the remote operation system 1 determines whether or not a first situation has been detected. If the first situation has been detected (step S180; Yes), the process proceeds to step S210. On the other hand, if the first situation has not been detected (step S180; No), the process proceeds to step S190.

[0095] In step S210, the remote driving system 1 starts remote driving of the vehicle 100. Note that in step S210, the remote driving terminal 200 is in a standby state, so the remote driving system 1 can immediately start remote driving without any time loss due to an initial check.

[0096] In step S190, the remote operation system 1 determines whether the second situation continues to be detected. If the second situation continues to be detected (step S190; Yes), the process returns to step S180. On the other hand, if the second situation is no longer detected (step S190; No), the process proceeds to step S200.

[0097] In step S200, the remote driving system 1 ends the standby state of the remote driving terminal 200. When the standby state of the remote driving terminal 200 is ended, the process ends.

[0098] Even if the second situation is detected, the first situation is not necessarily detected thereafter. For example, even if the deviation of the vehicle 100 from the target path exceeds the second threshold, if the cause is a temporary control disturbance or the like, the vehicle 100 may subsequently return to the target path normally. In this case, remote driving of the vehicle 100 is no longer necessary. Therefore, in a modified example, if the first situation is not detected and the second situation is no longer detected, the standby state of the remote driving terminal 200 is terminated without starting remote driving of the vehicle 100. This reduces the processing load on the remote driving system 1, which would otherwise be caused by keeping the remote driving terminal 200 in standby even when remote driving is no longer necessary. This also prevents the remote driving terminal 200 from continuing to store outdated initial check results. [Explanation of symbols]

[0099] 1 Remote operation system 11 First situation detection unit 12 Second situation detection unit 13 Initial Check Section 14 Remote operation start decision unit 100 vehicles 110 Communication equipment 120 sensors 130 Actuator 150 control device 160 processors 170 Storage device 200 Remote driving terminal 210 Communication equipment 220 Output Device 230 Remote control components 250 control device 260 processor 270 Storage device 300 Management device 310 Communication Equipment 350 control device 360 processor 370 Storage device 400 Infrastructure Sensors VCL vehicle information X Remote Operator

Claims

1. A remote driving system that remotely drives a vehicle based on an operation amount input to a remote driving terminal, one or more processors; the one or more processors detect a second condition that is predictive of a first condition that requires the remote operation of the vehicle; When the second condition is detected, the one or more processors perform at least a portion of an initial check to confirm that the remote driving can be initiated at the remote driving terminal before the first condition is detected; and the one or more processors initiate the remote operation when the first condition is detected; The vehicle is an autonomous vehicle that can be controlled by autonomous driving, The one or more processors obtain an amount of deviation of the vehicle from a target path; the first situation is a situation in which the deviation amount exceeds a first threshold; the second situation is a situation in which the deviation amount exceeds a second threshold; The first threshold is greater than the second threshold. Remote driving system.

2. The remote driving system according to claim 1, If the second condition is resolved before the first condition is detected, the one or more processors do not initiate the remote operation. Remote driving system.

3. The remote driving system according to claim 1, If the result of the initial check indicates that the remote operation can be started, transition the remote operation terminal to a standby state indicating that preparation for the remote operation is complete; If the second situation is not detected after the initial check and the first situation is not detected, the standby state is ended without starting the remote operation. Remote driving system.

4. The remote driving system according to any one of claims 1 to 3, the one or more processors: Acquire infrastructure information including images captured by infrastructure cameras installed in an area in which the vehicle travels; Detecting the second situation based on the infrastructure information Remote driving system.

5. The remote driving system according to any one of claims 1 to 3, The initial check at the remote driving terminal includes at least confirmation of establishment of communication with the vehicle. Remote driving system.

6. The remote driving system according to claim 5, The initial check at the remote driving terminal further includes verifying steering alignment between the vehicle and the remote driving terminal, and verifying that actuators of the vehicle operate in response to the operation amount sent to the vehicle. Remote driving system.

7. A method for remotely driving a vehicle based on an operation amount input to a remote driving terminal, comprising: The vehicle is an autonomous vehicle that can be controlled by autonomous driving, The method comprises: detecting a second condition that is predictive of a first condition that requires remote operation of the vehicle; When the second situation is detected, performing at least a part of an initial check to confirm that the remote driving can be started in the remote driving terminal before the first situation is detected; Initiating the remote operation when the first situation is detected; obtaining a deviation of the vehicle from a target path; Including, the first situation is a situation in which the deviation amount exceeds a first threshold; the second situation is a situation in which the deviation amount exceeds a second threshold; The first threshold is greater than the second threshold. method.

Images