Mobile body control device, mobile body control method, remote operation device, and remote operation control method

Abstract

The present technology pertains to a mobile body control device, a mobile body control method, a remote operation device, and a remote operation control method that make it possible for a remote operation operator to appropriately respond when a drive assist function is activated in a mobile body. This mobile body control device is provided with: a communication quality detection unit that detects communication quality between a mobile body and a remote operation device that performs an operation for remotely operating the mobile body; a recognition unit that acquires recognition information pertaining to the state of the mobile body and the conditions around the mobile body; and a drive assist control unit that controls the drive assist function of the mobile body on the basis of the recognition information, and controls the transmission of activation-related information, which is information pertaining to the activation of the drive assist function, to the remote operation device, on the basis of the recognition information and the communication quality. The present technology is applicable, for example, to a remote operation system.

Description

Mobile object control device, mobile object control method, remote driving device, and remote driving control method

[0001] The present technology relates to a mobile object control device, a mobile object control method, a remote driving device, and a remote driving control method, and in particular to a mobile object control device, a mobile object control method, a remote driving device, and a remote driving control method that are suitable for use when remotely driving a mobile object equipped with a driving assistance function.

[0002] In recent years, technological developments have been underway for remote driving systems in which a remote driving operator drives a vehicle in a remote location via communication (see, for example, Patent Document 1). Typically, the remote driving operator uses a remote driving controller to perform all operations required for driving, such as accelerating, decelerating, and steering the vehicle.

[0003] On the other hand, when a driving assistance function provided by a driving assistance system (ADAS (Advanced Driver-Assistance System)) equipped in the vehicle is activated, the driving assistance system intervenes in vehicle control regardless of the intentions of the remote driving operator.

[0004] Here, since the remote driving operator cannot feel the intervention of the driving assistance system, it is necessary for the vehicle to notify the remote driving controller of the operation of the driving assistance function (intervention of the driving assistance system).

[0005] International Publication No. 2020 / 202378

[0006] However, because communication delays occur between the vehicle and the remote driving system, it takes time for the remote driving operator to recognize the operation of the driving assistance function. In particular, if the communication quality between the vehicle and the remote driving system deteriorates, it takes a long time for the remote driving operator to recognize the operation of the driving assistance function.

[0007] This can cause a large discrepancy between the situation perceived by the remote driving operator and the actual situation the vehicle is in, which can lead to the remote driving operator performing inappropriate operations. For example, if the driving assistance function is stopped and remote driving is enabled again, the remote driving operator may perform operations without accurately recognizing the situation, which could lead to a dangerous situation.

[0008] The present technology has been developed in light of such circumstances, and enables a remote driving operator to respond appropriately when a driving assistance function is activated in a moving body such as a vehicle.

[0009] A mobile body control device according to a first aspect of the present technology includes a communication quality detection unit that detects communication quality between a mobile body and a remote driving device that operates the remote driving of the mobile body, a recognition unit that acquires recognition information regarding the state of the mobile body and the situation around the mobile body, and a driving assistance control unit that controls a driving assistance function of the mobile body based on the recognition information and controls the transmission of operation-related information, which is information regarding the operation of the driving assistance function, to the remote driving device based on the recognition information and the communication quality.

[0010] A mobile body control method according to a first aspect of the present technology includes a mobile body control device that detects communication quality between a mobile body and a remote driving device that operates the remote driving of the mobile body, acquires recognition information regarding the state of the mobile body and the situation around the mobile body, controls a driving assistance function of the mobile body based on the recognition information, and controls the transmission of operation-related information, which is information regarding the operation of the driving assistance function, to the remote driving device based on the recognition information and the communication quality.

[0011] A remote driving device according to a second aspect of the present technology includes a remote driving control unit that remotely controls the driving of a mobile body, and a presentation control unit that controls the presentation of remote driving information used for remotely driving the mobile body, and the presentation control unit controls the presentation of driving assistance function operation information that notifies the user that the driving assistance function is in operation in the mobile body, based on operation notification information transmitted from the mobile body to notify the user that the driving assistance function of the mobile body is in operation.

[0012] A remote driving control method according to a second aspect of the present technology is a method in which a remote driving device that operates the remote driving of a mobile body controls the presentation of driving assistance function operation information that notifies the user that the driving assistance function of the mobile body is in operation, based on operation notification information transmitted from the mobile body to notify the user that the driving assistance function of the mobile body is in operation.

[0013] In a first aspect of the present technology, the communication quality between a mobile body and a remote driving device that operates the remote driving of the mobile body is detected, recognition information regarding the state of the mobile body and the situation around the mobile body is obtained, a driving assistance function of the mobile body is controlled based on the recognition information, and the transmission of operation-related information, which is information regarding the operation of the driving assistance function, to the remote driving device is controlled based on the recognition information and the communication quality.

[0014] In a second aspect of the present technology, the presentation of driving assistance function operation information notifying that the driving assistance function is in operation in the mobile body is controlled based on operation notification information transmitted from the mobile body to notify that the driving assistance function of the mobile body is in operation.

[0015] 1 is a block diagram showing an embodiment of a remote driving system to which the present technology is applied; FIG. 1 is a block diagram showing an example configuration of a vehicle and a remote driving controller; FIG. 2 is a schematic diagram showing an example configuration of a remote driving controller; FIG. 3 is a flowchart for explaining a first embodiment of processing of a vehicle; FIG. 4 is a flowchart for explaining a first embodiment of processing of a vehicle; FIG. 5 is a diagram showing a first embodiment of a state transition diagram of each driving assistance function of a vehicle; FIG. 6 is a graph showing examples of activation conditions and advance notice conditions of an AEB function; FIG. 7 is a flowchart for explaining a first embodiment of processing of a remote driving controller; FIG. 8 is a sequence diagram showing a first processing example related to an operation display of the AEB function; FIG. 9 is a diagram showing an example of display control commands; FIG. 10 is a graph showing examples of activation conditions and advance notice conditions of a collision warning function; FIG. 11 is a sequence diagram showing an example of processing related to an operation display of the collision warning function; FIG. 12 is a flowchart for explaining a second embodiment of processing of a vehicle; FIG. 13 is a flowchart for explaining a second embodiment of processing of a vehicle; FIG. 14 is a diagram showing a second embodiment of a state transition diagram of each driving assistance function of a vehicle; FIG. 15 is a flowchart for explaining a second embodiment of processing of a remote driving controller; 1 is a sequence diagram showing a third processing example relating to an operation display of the AEB function. FIG. 2 is a diagram showing an example of the relationship between a driving assistance function and a control entity. FIG. 3 is a flowchart for explaining a third embodiment of processing in a vehicle. FIG. 4 is a flowchart for explaining a third embodiment of processing in a vehicle. FIG. 5 is a flowchart for explaining a third embodiment of processing in a remote driving controller. FIG. 6 is a sequence diagram showing a fourth processing example relating to an operation display of the AEB function. FIG. 7 is a diagram showing a first display example of the status display area of ​​the display device. FIG. 8 is a flowchart for explaining a fourth embodiment of processing in a vehicle. FIG. 9 is a flowchart for explaining a fourth embodiment of processing in a remote driving controller. FIG. 10 is a sequence diagram showing a fifth processing example relating to an operation display of the AEB function. FIG. 11 is a diagram showing a second display example of the status display area of ​​the display device.10 is a diagram illustrating a third display example of the status display area of ​​the display device. FIG. 11 is a block diagram illustrating an example of the configuration of a computer.

[0016] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Embodiment 2. Modification 3. Other

[0017] <<1. Embodiment>> An embodiment of the present technology will be described with reference to FIGS. 1 to 31 .

[0018] <Configuration Example of Remote Driving System> FIG. 1 shows an embodiment of a remote driving system 1 to which the present technology is applied.

[0019] The remote driving system 1 includes a vehicle 11 and a remote driving controller 12. The vehicle 11 and the remote driving controller 12 are connected to each other via a base station 21 and a communication network 22. The vehicle 11 and the base station 21 communicate wirelessly (e.g., Long Term Evolution (LTE), 5G (5th Generation), etc.). Wired communication is performed between the base station 21, the communication network 22, and the remote driving controller 12.

[0020] The vehicle 11 is a vehicle that is the target of remote driving.

[0021] The remote driving controller 12 is a remote driving device that performs remote driving of the vehicle 11 in accordance with operations by a remote driving operator.

[0022] Although the remote driving operator is assumed to be a person, it may also be a device such as a computer.

[0023] The vehicle 11 and the remote driving controller 12 communicate via a base station 21 and a communication network 22, sending and receiving video data, status information, remote driving commands, etc., to realize remote driving of the vehicle 11.

[0024] The status information is, for example, information indicating the interior and exterior conditions of the vehicle 11. The interior and exterior conditions of the vehicle 11 include, for example, the conditions around the vehicle 11, the conditions of the vehicle 11, and the conditions inside the vehicle 11.

[0025] The remote driving command is a control command used to control remote driving, and includes, for example, various control amounts of the vehicle 11 (for example, acceleration / deceleration amount, steering amount, etc.).

[0026] FIG. 2 shows an example of the functional configuration of the vehicle 11 and the remote driving controller 12.

[0027] The vehicle 11 includes a vehicle control system 101 that controls the vehicle 11. The vehicle control system 101 includes a communication unit 111, a communication quality detection unit 112, an input unit 113, an external sensor 114, an internal sensor 115, a recognition unit 116, a driving and braking device 117, a steering device 118, a remote driving execution unit 119, a driving assistance control unit 120, and an output unit 121. The communication unit 111, the communication quality detection unit 112, the input unit 113, the external sensor 114, the internal sensor 115, the recognition unit 116, the driving and braking device 117, the steering device 118, the remote driving execution unit 119, the driving assistance control unit 120, and the output unit 121 are connected to each other via a bus.

[0028] The communication unit 111 performs wireless communication (e.g., LTE, 5G, etc.) with the base station 21.

[0029] The communication quality detection unit 112 detects the communication quality between the vehicle 11 and the remote driving controller 12 based on the information from the communication unit 111. For example, the communication quality detection unit 112 detects the communication delay time between the vehicle 11 and the remote driving controller 12 as the communication quality. Note that the communication quality detection unit 112 may detect other types of communication quality such as throughput, or may detect multiple types of communication quality.

[0030] The input unit 113 includes an input device for the occupant to input data, instructions, etc. For example, the input unit 13 may include a touch panel, a button, a switch, a lever, etc. For example, the input unit 13 may include an input device that allows information to be input by a method other than manual operation, such as voice or gesture. For example, the input unit 13 may include a remote control device that uses infrared rays or radio waves, or an externally connected device that supports operation of the vehicle control system 101.

[0031] The external sensor 114 includes various sensors for recognizing the situation around the vehicle 11. For example, the external sensor 114 includes at least one type of sensor selected from the group consisting of a camera, a light detection and ranging (LiDAR), a millimeter-wave radar, etc. The external sensor 114 outputs information including sensor data from each sensor (hereinafter referred to as external sensor information).

[0032] The internal sensor 115 includes various sensors for recognizing the state of the vehicle 11 and the internal situation of the vehicle 11. The internal sensor 115 outputs information including sensor data from each sensor (hereinafter referred to as internal sensor information).

[0033] Possible sensors for recognizing the state of the vehicle 11 include, for example, a speed sensor, an acceleration sensor, an angular velocity sensor, an inertial measurement unit (IMU), a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, a brake sensor that detects the amount of brake pedal operation, a rotation sensor that detects the engine or motor rotation speed, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, a battery sensor that detects the remaining battery charge and temperature, and an impact sensor that can detect external impacts.

[0034] Possible sensors for recognizing the internal conditions of the vehicle 11 include, for example, a camera, radar, a seating sensor, a microphone, and a biosensor that detects biometric information of the occupants.

[0035] The recognition unit 116 recognizes the internal and external conditions of the vehicle 11 based on external sensor information from the external sensor 114 and internal sensor information from the internal sensor 115. For example, the recognition unit 116 executes a recognition process for at least one of the conditions around the vehicle 11, the conditions of the vehicle 11, and the conditions inside the vehicle 11, and acquires recognition information regarding at least one of the conditions around the vehicle 11, the conditions of the vehicle 11, and the conditions inside the vehicle 11.

[0036] For example, the situation around the vehicle 11 includes the presence or absence, size, shape, position, movement, etc. of objects around the vehicle 11. The objects around the vehicle 11 include, for example, vehicles (including bicycles), people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc.

[0037] For example, the state of the vehicle 11 includes speed, acceleration, angular velocity, steering angle, yaw rate, accelerator pedal operation amount, brake pedal operation amount, engine and motor rotation speed, tire air pressure, tire slip ratio, battery remaining charge and temperature, external impact, etc.

[0038] The internal conditions of the vehicle 11 include, for example, the state of the occupants, temperature, humidity, brightness, odor, etc. The state of the occupants includes, for example, physical condition, alertness, concentration, fatigue, line of sight, level of intoxication, driving operation, posture, etc.

[0039] The recognition unit 116 may recognize the interior and exterior conditions of the vehicle 11 based on information from devices external to the vehicle 11 (for example, sensors on other vehicles, sensors installed on the road, etc.).

[0040] The driving and braking device 117 includes, for example, a driving system. The driving system includes, for example, an accelerator pedal, a driving force generating device for generating driving force such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, etc. The driving and braking device 117 also includes, for example, a driving ECU for controlling the driving system, an actuator for driving the driving system, etc.

[0041] The driving and braking device 117 includes, for example, a braking system. The braking system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative braking mechanism, etc. The driving and braking device 117 also includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0042] The steering device 118 includes, for example, a steering system. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering device 118 also includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0043] The remote driving execution unit 119 receives a remote driving command from the remote driving controller 12 via the communication unit 111. The remote driving execution unit 119 controls the acceleration / deceleration and steering of the vehicle 11 by controlling the driving / braking device 117 and the steering device 118 based on the remote driving command.

[0044] The remote driving execution unit 119 also transmits information necessary for remote driving to the remote driving controller 12. For example, the remote driving execution unit 119 generates status information including information regarding at least one of the surrounding conditions of the vehicle 11, the state of the vehicle 11, and the internal conditions of the vehicle 11, based on the recognition information acquired by the recognition unit 116. The remote driving execution unit 119 transmits the status information to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22. For example, the remote driving execution unit 119 transmits sensor data necessary for remote driving, from among sensor data from the external sensor 114 and sensor data from the internal sensor 115, to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22. For example, the remote driving execution unit 119 transmits video data of the surroundings of the vehicle 11 (e.g., the direction of travel of the vehicle 11, etc.) captured by a camera equipped in the external sensor 114 to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22.

[0045] The driving assistance control unit 120 is configured, for example, by a driving assistance system (ADAS). The driving assistance control unit 120 controls the driving assistance function based on the interior and exterior conditions of the vehicle 11 recognized by the recognition unit 116. For example, the driving assistance control unit 120 determines whether or not activation of the driving assistance function is necessary based on the interior and exterior conditions of the vehicle 11 recognized by the recognition unit 116. If the driving assistance control unit 120 determines that activation of the driving assistance function is necessary, it intervenes in the control of at least one of the driving and braking device 117 and the steering device 118. Then, the driving assistance control unit 120 controls at least one of the driving and braking device 117 and the steering device 118 to activate the driving assistance function. In addition, the driving assistance control unit 120 causes the output unit 121 to present information indicating the activation status of the driving assistance function, etc.

[0046] For example, the driving assistance control unit 120 controls notification of operation-related information related to the operation of each driving assistance function to the remote driving controller 12 based on the quality of communication with the remote driving controller 12 and the recognition information (recognition results of the interior and exterior conditions of the vehicle 11) acquired by the recognition unit 116. More specifically, for example, the driving assistance control unit 120 generates a control command (hereinafter referred to as a display control command) including status information indicating the operation status of each driving assistance function based on the quality of communication with the remote driving controller 12 and the recognition information (recognition results of the interior and exterior conditions of the vehicle 11) acquired by the recognition unit 116. The control command is used to control the display of information related to the operation status of each driving assistance function on the remote driving controller 12. The driving assistance control unit 120 transmits the display control command to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22.

[0047] The driving assistance control unit 120 includes one or more controllers according to the type of driving assistance function to activate different driving assistance functions depending on the situation. For example, the driving assistance control unit 120 includes an AEB (Autonomous Emergency Braking) controller 131 and an LKA (Lane Keeping Assist) controller 132.

[0048] The AEB controller 131 is a controller that controls the collision warning function and the AEB function. The collision warning function includes, for example, a function of presenting a warning display, a warning sound, or the like to the occupants when there is a possibility that the vehicle 11 will collide with an obstacle. The AEB function also includes, for example, a function of controlling the drive / braking device 117 to decelerate the vehicle 11 and mitigate damage from a collision when there is a high possibility of a collision with an obstacle.

[0049] Here, an obstacle is, for example, an object that exists in the traveling direction of the vehicle 11, and includes moving objects such as other vehicles, pedestrians, and animals.

[0050] In addition, the AEB controller 131 generates a display control command related to the collision warning function or the AEB function and transmits it to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22.

[0051] The LKA controller 132 is a controller that controls the LKA function. The LKA function includes, for example, a function of controlling the steering device 118 to control the traveling direction of the vehicle 11 and prevent the vehicle 11 from deviating from the lane when there is a possibility that the vehicle 11 will deviate from the lane in which it is traveling.

[0052] In addition, the LKA controller 132 generates a display control command related to the LKA function and transmits it to the remote operation controller 12 via the communication unit 111, the base station 21, and the communication network 22.

[0053] The output unit 121 includes an output device that outputs, for example, one or more of visual information, auditory information, and tactile information. For example, the output unit 121 may include a display device that presents visual information by displaying an image itself, or a projector device that presents visual information by projecting an image. For example, the display device may be a device that displays visual information within the user's field of view, such as a head-up display, a see-through display, or a wearable device with an augmented reality (AR) function, in addition to or instead of a typical display device. For example, the output unit 121 may include a display device included in a navigation device, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like. For example, the output unit 121 may include an audio speaker, headphones, or earphones. For example, the output unit 121 may include a haptic element using haptic technology.

[0054] The remote driving controller 12 includes a communication unit 211, an input unit 212, a remote driving control unit 213, a presentation control unit 214, and an output unit 215. The communication unit 211, the input unit 212, the remote driving control unit 213, the presentation control unit 214, and the output unit 215 are connected to each other via a bus.

[0055] The communication unit 211 communicates with the vehicle 11 via the communication network 22 and the base station 21 .

[0056] The input unit 212 includes various input devices for inputting operation information required for remote driving of the vehicle 11. For example, the input unit 212 includes a brake pedal 251, an accelerator pedal 252, and a steering wheel 253 shown in FIG.

[0057] The remote driving control unit 213 executes control of the remote driving of the vehicle 11. That is, the remote driving control unit 213 remotely controls the driving of the vehicle 11. For example, the remote driving control unit 213 generates a remote driving command for executing remote driving of the vehicle 11 based on operation information input by the remote driving operator via the input unit 212. For example, the remote driving command includes the amount of acceleration / deceleration, steering amount, etc. of the vehicle 11. The remote driving control unit 213 transmits the remote driving command to the vehicle 11 via the communication unit 211, the communication network 22, and the base station 21.

[0058] The presentation control unit 214 controls the presentation of information (e.g., remote driving information used for remote driving of the vehicle 11) by the output unit 215. For example, the presentation control unit 214 receives video data, status information, etc. from the vehicle 11 via the base station 21, the communication network 22, and the communication unit 211. The presentation control unit 214 controls the presentation of information necessary for remote driving operation by the output unit 215 based on the video data and status information received from the vehicle 11.

[0059] The output unit 215 includes various output devices for outputting remote driving information (e.g., visual information, auditory information, tactile information, etc.) used for remote driving of the vehicle 11. For example, the output unit 215 includes a display device 254 and a speaker 255 shown in FIG. 3 . For example, under the control of the presentation control unit 214, the output unit 215 presents an image of the surroundings of the vehicle 11 (e.g., the direction of travel) based on video data. For example, under the control of the presentation control unit 214, the output unit 215 presents at least one of the conditions around the vehicle 11, the conditions inside the vehicle 11, and the state of the vehicle 11 based on status information.

[0060] FIG. 3 is a schematic diagram showing an example of the configuration of the remote operation controller 12. As shown in FIG.

[0061] The remote driving controller 12 includes a brake pedal 251, an accelerator pedal 252, a steering wheel 253, a display device 254, and a speaker 255. For example, the brake pedal 251, the accelerator pedal 252, and the steering wheel 253 constitute the input unit 212 of the remote driving controller 12 in Fig. 2. For example, the display device 254 and the speaker 255 constitute the output unit 215 of the remote driving controller 12 in Fig. 2.

[0062] The brake pedal 251 is used to remotely control the braking (deceleration) of the vehicle 11. Note that, for example, the brake pedal 251 may include a haptic element or the like that simulates vibrations, stresses, and the like that occur when the actual vehicle 11 is being driven.

[0063] The accelerator pedal 252 is used to remotely control the driving (acceleration) of the vehicle 11. Note that, for example, the accelerator pedal 252 may be provided with a haptic element or the like that simulates vibrations, stresses, and the like that occur when the vehicle 11 is actually being driven.

[0064] The steering wheel 253 is used to remotely control the steering of the vehicle 11. Note that, for example, the steering wheel 253 may include a haptic element or the like that simulates vibrations, stresses, and the like that occur while the actual vehicle 11 is being driven.

[0065] The display device 254 displays, for example, visual information necessary for remote driving of the vehicle 11. For example, the display device 254 displays an image of the surroundings of the vehicle 11 based on video data received from the vehicle 11. For example, the display device 254 displays at least one of the surrounding conditions of the vehicle 11, the internal conditions of the vehicle 11, and the state of the vehicle 11 in the status display area 254A based on status information received from the vehicle 11.

[0066] The speaker 255 outputs, for example, auditory information necessary for operating the remote driving of the vehicle 11. For example, the speaker 255 outputs audio guidance, warning sounds, sound effects, etc. to the remote driving operator. For example, the speaker 255 outputs environmental sounds collected around or inside the vehicle 11.

[0067] <First Embodiment of Processing in Remote Driving System 1> Next, a first embodiment of processing in the remote driving system 1 will be described with reference to FIGS.

[0068] For simplicity of explanation, the following description will be focused on processing related to the AEB function among the driving assistance functions.

[0069] <Processing of Vehicle 11> First, processing of the vehicle 11 will be described with reference to the flowcharts of FIGS. 4 and 5 and the state transition diagram of FIG.

[0070] This process is started, for example, when the vehicle 11 becomes capable of being remotely driven, and is ended when the vehicle 11 becomes unable to be remotely driven.

[0071] 6 shows a state transition diagram of each driving assistance function of the vehicle 11. The state of each driving assistance function (collision warning function, AEB function, LKA function) of the vehicle 11 independently transitions between an operation OFF state, an operation notice state, an operation ON state, and an operation OFF and display state, regardless of the state of the other driving assistance functions.

[0072] The operation OFF state is a state in which the driving assistance function is not operating and the display of operation information (hereinafter referred to as operation display) notifying that the driving assistance function is operating is not executed in the remote driving controller 12. The initial state of each driving assistance function is set to the operation OFF state.

[0073] The activation notice state is a state in which a notice of the driving assistance function is given. Specifically, before the driving assistance function is executed, the remote driving controller 12 displays a notice indicating that the driving assistance function is in operation.

[0074] The ON operation state is a state in which the driving assistance function is activated and the remote driving controller 12 displays a message indicating that the driving assistance function is activated.

[0075] The operation OFF and displaying state is a state in which the remote driving controller 12 continues to display that the driving assistance function is in operation after the operation of the driving assistance function has stopped.

[0076] In step S1, the vehicle 11 starts traveling by remote driving. Specifically, the vehicle 11 starts the following process.

[0077] The remote driving execution unit 119 receives a remote driving command from the remote driving controller 12 via the communication network 22, the base station 21, and the communication unit 111. The remote driving execution unit 119 controls the drive / braking device 117 and the steering device 118 based on the remote driving command, thereby causing the vehicle 11 to travel in accordance with the operation of the remote driving operator in the remote driving controller 12.

[0078] In step S2, the recognition unit 116 acquires sensor information. Specifically, the recognition unit 116 acquires internal sensor information from the internal sensor 115 and external sensor information from the external sensor 114.

[0079] In step S3, the recognition unit 116 recognizes the internal and external conditions of the vehicle 11 based on the internal sensor information and the external sensor information. For example, the recognition unit 116 recognizes one or more of the conditions around the vehicle 11, the conditions of the vehicle 11, and the conditions inside the vehicle 11.

[0080] In step S4, the communication quality detection unit 112 detects the communication quality. For example, the communication quality detection unit 112 detects the communication delay time between the vehicle 11 and the remote driving controller 12 as the communication quality between the vehicle 11 and the remote driving controller 12. Specifically, for example, the communication quality detection unit 112 periodically sends a ping command to the remote driving controller 12 via the communication unit 111 and measures the round trip time (RTT). The communication quality detection unit 112 calculates the communication delay time based on the average value of the RTT over a predetermined period of time.

[0081] In step S5, the driving assistance control unit 120 sets the activation conditions and advance notice conditions of the driving assistance function.

[0082] The activation conditions are conditions used to determine whether or not to activate the driving assistance function.

[0083] The advance notice condition is a condition used to determine whether or not to give advance notice of the driving assistance function when the driving assistance function is expected to be activated.

[0084] For example, the AEB controller 131 of the driving assistance control unit 120 calculates the advance notification conditions for each driving assistance function based on the surrounding conditions of the vehicle 11, the state of the vehicle 11, and the communication quality between the vehicle 11 and the remote driving controller 12.

[0085] 7 is a graph showing an example of the activation conditions and advance notice conditions of the AEB function. The horizontal axis of the graph in FIG. 7 represents the relative speed va of the vehicle 11 with respect to an obstacle (the speed at which the vehicle 11 approaches the obstacle), and the vertical axis represents the distance da between the vehicle 11 and the obstacle.

[0086] The activation condition of the AEB function is set based on the movement of the vehicle 11 relative to an obstacle. Specifically, the activation condition of the AEB function is determined, for example, by the distance da from the obstacle, and the AEB function is activated when the distance da from the obstacle becomes less than a threshold value θ1. The threshold value θ1 is calculated, for example, using the following equations (1) and (2).

[0087] θ1=α1×va-C1 (where va>VA1) (1) θ1=0 (where va≦VA1) (2)

[0088] α1 denotes a coefficient, and C1 denotes a constant. VA1 is set to, for example, C1 / α1.

[0089] A straight line L1 in FIG. 7 indicates the relationship between the threshold value θ1, which is the activation condition of the AEB function, and the relative velocity va, expressed by the formulas (1) and (2).

[0090] The advance notification conditions for the AEB function are set based on the movement of the vehicle 11 relative to an obstacle and the communication quality between the vehicle 11 and the remote driving controller 12. Specifically, the advance notification conditions for the AEB function are determined, for example, by the distance da between the vehicle 11 and an obstacle, and when the distance da from the obstacle becomes less than a threshold value θr1, the advance notification for the AEB function is executed. The threshold value θr1 is calculated, for example, using the following equations (3) and (4).

[0091] θr1=θ1+va×Td=α1×va−C1+va×Td=(α1+Td)×va−C1 (where va>VAr1) (3) θr1=0 (where va≦VAr1) (4)

[0092] Td indicates a communication delay time between the vehicle 11 and the remote driving controller 12. VAr1 is set to, for example, C1 / (α1+Td).

[0093] A straight line Lr1 in FIG. 7 indicates the relationship between the threshold value θr1, which is the advance notice condition for the AEB function, and the relative velocity va, expressed by the formulas (3) and (4).

[0094] In this way, the threshold value θr1 is set to be longer than the threshold value θ1 by the distance traveled by the vehicle 11 during the communication delay time Td.

[0095] The AEB controller 131 calculates a threshold value θ1 based on the relative velocity va and sets it as an activation condition of the AEB function. The AEB controller 131 calculates a threshold value θr1 based on the relative velocity va and the communication delay time Td and sets it as an advance notice condition of the AEB function.

[0096] In step S6, the driving assistance control unit 120 determines whether the activation conditions for the driving assistance function are met. For example, if the distance da between the vehicle 11 and an obstacle in the traveling direction is less than the threshold value θ1, the AEB controller 131 determines that the activation conditions for the AEB function are met, and the process proceeds to step S7.

[0097] In step S7, the vehicle 11 executes the driving assistance function.

[0098] For example, the AEB controller 131 instructs the drive / brake device 117 to control the AEB function.

[0099] In response to this, the driving / braking device 117 intervenes in the remote driving by the remote driving controller 12 and executes the AEB function, thereby automatically applying the brakes of the vehicle 11 according to the distance to the obstacle in the traveling direction of the vehicle 11.

[0100] The AEB controller 131 also instructs the output unit 121 to display the operation of the AEB function.

[0101] In response to this, the output unit 121 executes an operation display of the AEB function, thereby notifying the passengers in the vehicle 11 that the AEB function is operating.

[0102] In step S8, the driving assistance control unit 120 determines whether the remote driving controller 12 is displaying an operation indication. For example, the AEB controller 131 of the driving assistance control unit 120 determines whether the remote driving controller 12 is displaying an operation indication of the AEB function based on the transmission history of display control commands. If it is determined that the remote driving controller 12 is not displaying an operation indication of the AEB function, the process proceeds to step S9.

[0103] This is the case when the AEB function is in the OFF state.

[0104] In step S9, the driving assistance control unit 120 issues an instruction to display an operation display. For example, the AEB controller 131 of the driving assistance control unit 120 transmits an "AEB" operation display ON command, which is one of the display control commands, to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22.

[0105] In step S10, the driving assistance control unit 120 starts measuring the display time. For example, the AEB controller 131 of the driving assistance control unit 120 starts measuring the display time of the AEB function.

[0106] As a result, the state of the AEB function transitions from the OFF state to the ON state.

[0107] Thereafter, the process returns to step S2, and the processes from step S2 onwards are executed.

[0108] On the other hand, if it is determined in step S8 that the remote driving controller 12 is displaying the operation of the driving assistance function (AEB function), the process returns to step S2, and the processes from step S2 onward are then executed.

[0109] This is the case when the AEB function is in the ON state.

[0110] On the other hand, if it is determined in step S6 that the conditions for operating the driving assistance function (AEB function) are not met, the process proceeds to step S11.

[0111] In step S11, the driving assistance control unit 120 determines whether the driving assistance function is in operation. For example, if the AEB controller 131 of the driving assistance control unit 120 determines that the AEB function is in operation, the process proceeds to step S12.

[0112] This is the case when the AEB function is in the ON state.

[0113] In step S12, the vehicle 11 stops the driving assistance function. For example, the AEB controller 131 of the driving assistance control unit 120 instructs the drive / brake device 117 to stop control of the AEB function.

[0114] In response to this, the driving / braking device 117 stops the AEB function.

[0115] In step S13, the driving assistance control unit 120 determines whether the display time of the remote driving controller 12 has exceeded the minimum display time. For example, the AEB controller 131 of the driving assistance control unit 120 checks the elapsed time since starting to measure the display time after sending the "AEB" operation display ON command in the process of step S9 or the process of step S23 described below. If the elapsed time is equal to or exceeds the predetermined minimum display time, the AEB controller 131 determines that the display time of the AEB function operation display of the remote driving controller 12 has exceeded the minimum display time, and the process proceeds to step S14.

[0116] In step S14, the driving assistance control unit 120 instructs the display to be stopped.

[0117] For example, the AEB controller 131 of the driving assistance control unit 120 controls the output unit 121 to stop the operation display of the AEB function.

[0118] Also, for example, the AEB controller 131 of the driving assistance control unit 120 transmits an "AEB" display OFF command, which is one of the display control commands, to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22. Also, the driving assistance control unit 120 stops measuring the display time of the AEB function.

[0119] As a result, the state of the AEB function transitions from the operation ON state to the operation OFF state.

[0120] Thereafter, the process returns to step S2, and the processes from step S2 onwards are executed.

[0121] On the other hand, in step S13, if the predetermined minimum display time has not elapsed since the AEB controller 131 started measuring the display time of the AEB function, the AEB controller 131 determines that the display time of the remote operation controller 12 has not elapsed for the minimum display time, and the process returns to step S2. Then, the process from step S2 onward is executed.

[0122] As a result, the state of the AEB function transitions from the operation ON state to the operation OFF and displaying state.

[0123] On the other hand, if it is determined in step S11 that the driving assistance function (AEB function) is not in operation, the process proceeds to step S15.

[0124] In step S15, similar to the process in step S8, it is determined whether or not the remote operation controller 12 is displaying an operation. If it is determined that the remote operation controller 12 is displaying an operation, the process proceeds to step S16.

[0125] In step S16, the driving assistance control unit 120 determines whether the driving assistance function has been stopped. For example, if the AEB controller 131 of the driving assistance control unit 120 determines that the AEB function has been stopped, the process proceeds to step S17.

[0126] This is the case when the AEB function is in the OFF state and the display is in progress.

[0127] In step S17, similar to the process of step S13, it is determined whether or not the display time of the remote operation controller 12 has exceeded the minimum display time. If it is determined that the display time of the remote operation controller 12 has exceeded the minimum display time, the process proceeds to step S18.

[0128] In step S18, an instruction to stop the display is issued in the same manner as in step S14.

[0129] As a result, the state of the AEB function transitions from the operation-off and displaying state to the operation-off state.

[0130] Thereafter, the process returns to step S2, and the processes from step S2 onwards are executed.

[0131] On the other hand, if it is determined in step S17 that the display time of the remote operation controller 12 has not elapsed beyond the minimum display time, the process returns to step S2, whereupon the processes from step S2 onward are executed.

[0132] On the other hand, if it is determined in step S16 that the driving assistance function (AEB function) has not been stopped, the process proceeds to step S19.

[0133] This is the case when the AEB function is in the operation notice state.

[0134] In step S19, the driving assistance control unit 120 determines whether the advance notice condition of the driving assistance function is satisfied. For example, if the distance da to an obstacle in the traveling direction of the vehicle 11 is equal to or greater than the threshold value θr1, the AEB controller 131 of the driving assistance control unit 120 determines that the advance notice condition of the AEB function is not satisfied, and the process proceeds to step S20.

[0135] This is the case, for example, when the advance notice condition for the AEB function is satisfied, but then the activation condition for the AEB function is not satisfied, and the advance notice condition for the AEB function is not satisfied.

[0136] In step S20, an instruction to stop the display is issued, similar to the processing in step S14.

[0137] As a result, the state of the AEB function transitions from the operation notice state to the operation OFF state.

[0138] Thereafter, the process returns to step S2, and the processes from step S2 onwards are executed.

[0139] On the other hand, if it is determined in step S19 that the advance notice condition for the driving support function (AEB function) is satisfied, that is, if the operation of the driving support function (AEB function) is expected, the process returns to step S2. Then, the processes from step S2 onward are executed.

[0140] On the other hand, if it is determined in step S15 that the remote operation controller 12 is not displaying an operation, the process proceeds to step S21.

[0141] In step S21, similar to the process of step S19, it is determined whether the advance notice condition of the remote support function (AEB function) is satisfied. If it is determined that the advance notice condition of the remote support function is satisfied, the process proceeds to step S22.

[0142] In step S22, similar to the process in step S8, it is determined whether or not the remote operation controller 12 is displaying an operation. If it is determined that the remote operation controller 12 is not displaying an operation, the process proceeds to step S23.

[0143] This is the case when the AEB function is in the OFF state.

[0144] In step S23, an operation display is instructed in the same manner as in the process of step S9.

[0145] In step S24, similar to the process in step S10, an instruction to measure the display time is issued.

[0146] As a result, the state of the AEB function transitions from the operation OFF state to the operation notice state.

[0147] Thereafter, the process returns to step S2, and the processes from step S2 onwards are executed.

[0148] On the other hand, if it is determined in step S22 that the remote operation controller 12 is displaying an operation, the process returns to step S2, whereupon the processes from step S2 onward are executed.

[0149] This is the case when the AEB function is in the operation notice state.

[0150] On the other hand, if it is determined in step S21 that the advance notice condition for the remote support function is not met, the process returns to step S2, whereupon the processes from step S2 onwards are executed.

[0151] This is the case when the AEB function is in the OFF state.

[0152] <Processing of Remote Driving Controller 12> Next, with reference to the flowchart of FIG. 8, a process executed by the remote driving controller 12 in response to the process of the vehicle 11 shown in FIGS. 4 and 5 will be described.

[0153] In step S51, the remote driving controller 12 acquires operation information. For example, the remote driving operator operates the input unit 212 (e.g., the brake pedal 251, the accelerator pedal 252, and the steering wheel 253) to remotely drive the vehicle 11. The input unit 212 supplies the remote driving control unit 213 with operation information indicating the operation content of the remote driving operator.

[0154] In step S52, the remote driving control unit 213 transmits a remote driving command. Specifically, the remote driving control unit 213 generates a remote driving command including the acceleration / deceleration amount and steering amount of the vehicle 11 based on the operation information. The remote driving control unit 213 transmits the remote driving command to the vehicle 11 via the communication unit 211, the communication network 22, and the base station 21.

[0155] In step S53, the presentation control unit 214 determines whether an operation display has been instructed. For example, when the presentation control unit 214 receives an “AEB” operation display ON command from the vehicle 11 via the base station 21, the communication network 22, and the communication unit 211, the presentation control unit 214 determines that an operation display has been instructed, and the process proceeds to step S54.

[0156] In step S54, the output unit 215 performs an operation display under the control of the presentation control unit 214. For example, the display device 254 of the output unit 215 displays information indicating that the AEB function of the vehicle 11 is in operation in the status display area 254A.

[0157] Thereafter, the process proceeds to step S55.

[0158] On the other hand, if it is determined in step S53 that the operation display has not been instructed, the process of step S54 is skipped and the process proceeds to step S55.

[0159] In step S55, the presentation control unit 214 determines whether or not an instruction to stop the display has been issued. For example, when the presentation control unit 214 receives an “AEB” display OFF command from the vehicle 11 via the base station 21, the communication network 22, and the communication unit 211, the presentation control unit 214 determines that an instruction to stop the display has been issued, and the process proceeds to step S56.

[0160] In step S56, the output unit 215 stops the instructed display under the control of the presentation control unit 214. For example, the display device 254 of the output unit 215 stops displaying the operation of the AEB function of the vehicle 11.

[0161] Thereafter, the process returns to step S51, and the processes from step S51 onwards are executed.

[0162] On the other hand, if it is determined in step S55 that the display has not been stopped, the process returns to step S51, and the processes from step S51 onwards are then executed.

[0163] FIG. 9 is a sequence diagram showing an example of a process related to an operation display of the AEB function.

[0164] For example, when the advance notice condition for the AEB function is met, the AEB controller 131 of the vehicle 11 transmits an “AEB” operation display ON command to the remote driving controller 12 via the communication unit 111 .

[0165] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" operation display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, starts displaying the operation of the AEB function.

[0166] Next, when the conditions for operating the AEB function are met, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to control the AEB function. In response, the drive / brake device 117 executes the AEB function. The AEB controller 131 also instructs the output unit 121 to display an operation indication of the AEB function. In response, the output unit 121 executes an operation indication of the AEB function.

[0167] Next, when the activation conditions for the AEB function are no longer satisfied, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to stop control of the AEB function. In response to this, the drive / brake device 117 stops the AEB function.

[0168] At this time, if the display time of the AEB function operation indication on the remote driving controller 12 has not elapsed the minimum display time, the AEB function operation indication continues on the vehicle 11 and the remote driving controller 12 .

[0169] Next, when the display time of the AEB function operation indication of the remote driving controller 12 has exceeded the minimum display time, the AEB controller 131 of the vehicle 11 instructs the output unit 121 to stop the AEB function operation indication. In response, the output unit 121 stops the AEB function operation indication. The AEB controller 131 also transmits an "AEB" display OFF command to the remote driving controller 12 via the communication unit 111.

[0170] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" display OFF command via the communication unit 211. The output unit 215 stops the display related to the AEB function under the control of the presentation control unit 214. In this case, the display of the operation of the AEB function is stopped.

[0171] In addition, if the conditions for operating the AEB function are not met and the display time of the AEB function on the remote driving controller 12 (display time of the AEB function operation display) has already exceeded the minimum display time, the AEB function operation display on the output unit 121 of the vehicle 11 is immediately stopped and an "AEB" display OFF command is sent to the remote driving controller 12.

[0172] In this way, the remote driving operator is notified of the operation of the AEB function at an appropriate timing regardless of the quality of communication between the vehicle 11 and the remote driving controller 12. As a result, the remote driving operator can quickly recognize the operation of the AEB function and respond appropriately (for example, prepare for the next driving operation), thereby improving safety.

[0173] For example, if the advance notification conditions for the AEB function are met, and then the AEB function operation conditions are not met and the advance notification conditions for the AEB function are not met, even if the display time of the AEB function operation display on the remote driving controller 12 has not elapsed the minimum display time, the AEB function operation display on the output unit 121 of the vehicle 11 is stopped and an "AEB" display OFF command is sent to the remote driving controller 12.

[0174] In this way, if the AEB function does not operate after the advance notice condition for the AEB function is met, the AEB function operation display is immediately turned off in the remote driving controller 12. In response to this, the remote driving operator may become temporarily wary when he or she sees the AEB function operation display, but this does not pose a particular problem because the AEB function is not actually operating and no dangerous situation has occurred in the vehicle 11.

[0175] The same processing as described above is also executed for driving assistance functions other than the AEB function. Here, a brief description will be given of the processing for the other driving assistance functions.

[0176] 10 shows examples of display control commands related to each driving assistance function transmitted from the vehicle 11 to the remote driving controller 12. The display control command includes the type of driving assistance function to be targeted and operation-related information related to the operation of the driving assistance function, and is a command for instructing the remote driving controller 12 to display the operation-related information, etc.

[0177] The display control commands for the collision warning function include a "collision warning" display ON command and a "collision warning" display OFF command.

[0178] The "collision warning" display ON command includes activation notification information for notifying that the collision warning function is activated, and is a command for instructing the activation display of the collision warning function. When the remote driving controller 12 receives the "collision warning" display ON command, it executes a display for notifying the "collision warning." In other words, the remote driving controller 12 warns the remote driving operator that the vehicle 11 may collide with an obstacle.

[0179] The "collision warning" display OFF command includes stop notification information for notifying that the collision warning function has been stopped, and is a command for instructing the stop of the display related to the collision warning function. When the remote driving controller 12 receives the "collision warning" display OFF command, it stops the display related to the "collision warning function."

[0180] The display control commands for the AEB function include an "AEB" advance notice display ON command, an "AEB" operation display ON command, an "AEB" takeover command, and an "AEB" display OFF command.

[0181] The "AEB" warning display ON command includes operation warning information for warning of the operation of the AEB function, and is a command for instructing the warning display of the AEB function. When the remote driving controller 12 receives the "AEB" warning display ON command, it executes a display to notify the "AEB operation warning." In other words, the remote driving controller 12 warns the remote driving operator that the AEB function will be activated in the vehicle 11.

[0182] The "AEB" operation display ON command includes operation notification information for notifying that the AEB function is operating, and is a command for instructing to display an operation display of the AEB function. When the remote driving controller 12 receives the "AEB" operation display ON command, it executes a display to notify that "AEB is operating." In other words, the remote driving controller 12 notifies the remote driving operator that the AEB function is operating in the vehicle 11.

[0183] The "AEB" takeover command includes takeover information for notifying the takeover from the AEB function to remote driving, and is a command for instructing a takeover display of the AEB function. When the remote driving controller 12 receives the "AEB" takeover command, it executes a display for notifying that "AEB takeover is in progress." In other words, the remote driving controller 12 notifies the remote driving operator that the AEB function in the vehicle 11 is being taken over to remote driving.

[0184] The "AEB" display OFF command includes stop notification information for notifying that the AEB function has been stopped, and is a command for instructing the stop of the display related to the AEB function. When the remote operation controller 12 receives the "AEB" display OFF command, it stops the display related to the "AEB function."

[0185] The display control commands for the LKA function include an "LKA" advance notice display ON command, an "LKA" operation display ON command, an "LKA" takeover command, and an "LKA" display OFF command.

[0186] The "LKA" warning display ON command includes operation warning information for warning of the activation of the LKA function, and is a command for instructing the display of a warning of the LKA function. When the remote driving controller 12 receives the "LKA" warning display ON command, it executes a display to notify the "LKA activation warning." In other words, the remote driving controller 12 warns the remote driving operator that the LKA function will be activated in the vehicle 11.

[0187] The "LKA" operation display ON command includes operation notification information for notifying that the LKA function is operating, and is a command for instructing the display of the operation of the LKA function. When the remote driving controller 12 receives the "LKA" operation display ON command, it executes a display to notify that "LKA is operating." In other words, the remote driving controller 12 notifies the remote driving operator that the LKA function is operating in the vehicle 11.

[0188] The "LKA" handover command includes handover information for notifying the handover from the LKA function to remote driving, and is a command for instructing a handover display of the LKA function. When the remote driving controller 12 receives the "LKA" handover command, it executes a display to notify "LKA handover in progress." In other words, the remote driving controller 12 notifies the remote driving operator that the LKA function in the vehicle 11 is being handed over to remote driving.

[0189] The "LKA" display OFF command includes stop notification information for notifying that the LKA function has been stopped, and is a command for instructing the stop of the display related to the LKA function. When the remote operation controller 12 receives the "LKA" display OFF command, it stops the display related to the "LKA function."

[0190] The "collision warning" display ON command, the "AEB" operation display ON command, and the "LKA" advance warning display ON command are commands used for similar purposes in each driving assistance function. The "AEB" advance warning display ON command and the "LKA" advance warning display ON command are commands used for similar purposes in each driving assistance function. The "AEB" takeover command and the "LKA" takeover command are commands used for similar purposes in each driving assistance function. The "collision warning" display OFF command, the "AEB" display OFF command, and the "LKA" operation display ON command are commands used for similar purposes in each driving assistance function.

[0191] 11 is a graph showing an example of the activation conditions and advance notice conditions of the collision warning function. The horizontal axis of the graph in FIG. 11 represents the relative speed va of the vehicle 11 with respect to an obstacle, and the vertical axis represents the distance da between the vehicle 11 and the obstacle.

[0192] The activation condition of the collision warning function is set based on the movement of the vehicle 11 relative to an obstacle. Specifically, the activation condition of the collision warning function is determined, for example, by the distance da between the vehicle 11 and the obstacle, and the collision warning function is activated when the distance da from the obstacle becomes less than a threshold value θ2. The threshold value θ2 is calculated, for example, using the following equations (5) and (6).

[0193] θ2 = α2 × va - C2 (where va > VA2) (5) θ2 = 0 (where va ≦ VA2) (6)

[0194] α2 denotes a coefficient, and C2 denotes a constant. VA2 is set to, for example, C2 / α2.

[0195] A straight line L1 in FIG. 11 indicates the relationship between the threshold value θ2, which is the activation condition of the collision warning function, and the relative speed va, expressed by the equations (5) and (6).

[0196] The advance notification conditions of the collision warning function are set based on the movement of the vehicle 11 relative to an obstacle and the communication quality between the vehicle 11 and the remote driving controller 12. Specifically, the advance notification conditions of the collision warning function are determined, for example, by the distance da from the obstacle, and the advance notification of the collision warning function is executed when the distance da from the obstacle becomes less than a threshold value θr2. The threshold value θr2 is calculated, for example, using the following equations (7) and (8).

[0197] θr2=θ2+va×Td=α2×va-C2+va×Td=(α2+Td)×va-C2 (where va>VAr2) (7) θr2=0 (where va≦VAr2) (8)

[0198] VAr2 is set to, for example, C2 / (α2+Td).

[0199] A straight line Lr1 in FIG. 11 indicates the relationship between the threshold value θr2, which is the advance notice condition of the collision warning function, and the relative speed va, expressed by the formulas (7) and (8).

[0200] In this way, the threshold value θr2 is set to be longer than the threshold value θ2 by the distance traveled by the vehicle 11 during the communication delay time Td.

[0201] FIG. 12 is a sequence diagram showing an example of a process related to an operation display of the collision warning function.

[0202] For example, when the advance notice condition of the collision warning function is met, the AEB controller 131 of the vehicle 11 transmits a “collision warning” operation display ON command to the remote driving controller 12 via the communication unit 111 .

[0203] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "collision warning" activation display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, starts displaying the activation of the collision warning function.

[0204] Next, when the activation condition of the collision warning function is met, the AEB controller 131 of the vehicle 11 instructs the output unit 121 to display a collision warning. In response to this, the output unit 121 executes the display of the collision warning.

[0205] Thereafter, when the activation condition for the collision warning function is no longer satisfied and the display time of the activation display of the collision warning function of the remote driving controller 12 has exceeded the minimum display time, the AEB controller 131 of the vehicle 11 instructs the output unit 121 to stop the collision warning display. In response, the output unit 121 stops the collision warning display. The AEB controller 131 also transmits a "collision warning" display OFF command to the remote driving controller 12 via the communication unit 111.

[0206] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "collision warning" display OFF command via the communication unit 211. The output unit 215 stops the display related to the collision warning function under the control of the presentation control unit 214. In this case, the display indicating that the collision warning function is activated is stopped.

[0207] In this way, the remote driving operator is notified of the activation of the collision warning function at an appropriate time, regardless of the quality of communication between the vehicle 11 and the remote driving controller 12. As a result, the remote driving operator can quickly recognize the activation of the collision warning function and respond appropriately (for example, prepare for the next driving operation), thereby improving safety.

[0208] For example, if the advance notification conditions for the collision warning function are met, and then the activation conditions for the collision warning function are not met and the advance notification conditions for the collision warning function are not met, a "collision warning" display OFF command is sent to the remote driving controller 12 even if the display time of the activation display of the collision warning function on the remote driving controller 12 has not elapsed the minimum display time.

[0209] In this way, if the collision warning function does not operate after the advance notification condition of the collision warning function is met, the collision warning display is immediately turned off in the remote driving controller 12. In response to this, the remote driving operator may temporarily brace himself when he sees the display indicating that the collision warning function is operating, but this does not pose a particular problem because the collision warning function is not actually operating and no dangerous situation has occurred in the vehicle 11.

[0210] 13 is a graph showing examples of the activation conditions and advance notice conditions of the LKA function. The horizontal axis of the graph in FIG. 13 represents the relative speed vb of the vehicle 11 with respect to the boundary line of the lane in which the vehicle 11 is traveling, and the vertical axis represents the distance db between the vehicle 11 and the lane boundary line. More specifically, the relative speed vb represents the relative speed of the vehicle 11 with respect to the boundary line of the lane in a direction perpendicular to the lane (a direction crossing the lane).

[0211] The activation condition of the LKA function is set based on the movement of the vehicle 11 relative to the lane in which the vehicle 11 is traveling. Specifically, the activation condition of the LKA function is determined, for example, by the distance d b from the boundary line of the lane, and the LKA function is activated when the distance d b from the obstacle becomes less than a threshold value θ3. The threshold value θ3 is calculated, for example, using the following equations (9) and (10).

[0212] θ3=α3×vb−C3 (where vb>VB1) (9) θ3=0 (where vb≦VB1) (10)

[0213] α3 denotes a coefficient, and C3 denotes a constant. VB1 is set to, for example, C3 / α3.

[0214] A straight line L3 in FIG. 13 indicates the relationship between the threshold value θ3, which is the activation condition of the LKA function, and the relative speed vb, expressed by the equations (9) and (10).

[0215] The advance notification conditions for the LKA function are set based on the movement of the vehicle 11 relative to the lane in which it is traveling and the quality of communication between the vehicle 11 and the remote driving controller 12. The advance notification conditions for the LKA function are determined, for example, by the distance db from the lane boundary line, and the advance notification of the LKA function is executed when the distance db from the obstacle becomes less than a threshold value θr2. The threshold value θr2 is calculated, for example, using the following equations (11) and (12).

[0216] θr3=θ3+vb×Td=α3×vb−C3+vb×Td=(α3+Td)×vb−C3(where vb>VBr1) (11) θr3=0(where vb≦VBr1) (12)

[0217] VBr1 is set to, for example, C3 / (α3+Td).

[0218] A straight line Lr1 in FIG. 13 indicates the relationship between the threshold value θr3, which is the advance notice condition of the LKA function, and the relative speed vb, expressed by the formulas (11) and (12).

[0219] In this way, the threshold value θr3 is set to be longer than the threshold value θ3 by the distance that the vehicle 11 travels in the direction perpendicular to the lane during the communication delay time Td.

[0220] The processing related to the LKA function is almost the same as the processing related to the AEB function described above, except that the steering device 118 and the LKA controller 132 perform the processing instead of the drive / braking device 117 and the AEB controller 131, and that display control commands for the LKA function are used instead of the display control commands for the AEB function.

[0221] Therefore, similar to the AEB function, the operation of the LKA function is notified to the remote driving operator at an appropriate timing regardless of the quality of communication between the vehicle 11 and the remote driving controller 12. As a result, the remote driving operator can quickly recognize the operation of the LKA function and respond appropriately (for example, prepare for the next driving operation), thereby improving safety.

[0222] <Second Embodiment of Processing in Remote Driving System 1> Next, a second embodiment of processing in the remote driving system 1 will be described with reference to FIGS.

[0223] In a second embodiment of the processing of the remote driving system 1, when the advance notification conditions for the driving assistance function are met, the remote driving controller 12 displays (advance display) operation advance notification information that notifies the operation of the driving assistance function.

[0224] For simplicity of explanation, the following description will be focused on processing related to the AEB function among the driving assistance functions.

[0225] <Processing of Vehicle 11> Processing of the vehicle 11 will be described with reference to the flowcharts of FIGS. 14 and 15 and the state transition diagram of FIG.

[0226] This process is started, for example, when the vehicle 11 becomes capable of being remotely driven, and is ended when the vehicle 11 becomes unable to be remotely driven.

[0227] 16 shows a state transition diagram of each driving assistance function of the vehicle 11. The state of each driving assistance function of the vehicle 11 (collision warning function, AEB function, LKA function) transitions independently and independently of the other driving assistance functions among an operation-off state, an operation notice state, an operation-on state, and an operation-off and display state.

[0228] The state transition diagram of Fig. 16 has the same types of states and the same conditions for transition between states as the state transition diagram of Fig. 6. However, the operation advance notice state is different from the operation advance notice state of Fig. 6.

[0229] 16 is a state in which a warning is given to the driver before the driving assistance function is activated. Specifically, the remote driving controller 12 displays a warning to the driver before the driving assistance function is executed.

[0230] On the other hand, as described above, in the operation notice state of FIG. 6, a display indicating that the driving assistance function is in operation is displayed.

[0231] In steps S101 to S105, the same processes as those in steps S1 to S5 in FIG. 4 are executed.

[0232] In step S106, it is determined whether or not the activation conditions for the driving assistance function are met, similarly to the processing in step S6 of Fig. 4. If it is determined that the activation conditions for the driving assistance function are met, the process proceeds to step S107.

[0233] In step S107, the driving assistance function is executed in the same manner as in step S7 of FIG.

[0234] In step S108, it is determined whether or not the remote operation controller 12 is displaying an operation status, similar to the process in step S8 of Fig. 4. If it is determined that the remote operation controller 12 is not displaying an operation status, the process proceeds to step S109.

[0235] In step S109, the driving assistance control unit 120 determines whether the remote driving controller 12 is displaying a warning. For example, the AEB controller 131 of the driving assistance control unit 120 determines whether the remote driving controller 12 is displaying a warning of the AEB function based on the transmission history of display control commands. If it is determined that the remote driving controller 12 is displaying a warning of the AEB function, the process proceeds to step S110.

[0236] This is the case when the AEB function is in the operation notice state.

[0237] In step S110, an operation display is instructed in the same manner as in step S9 of FIG.

[0238] As a result, the state of the AEB function transitions from the operation notice state to the operation ON state.

[0239] Thereafter, the process returns to step S102, and the processes from step S102 onwards are executed.

[0240] On the other hand, if it is determined in step S109 that the remote operation controller 12 is not displaying a notice, the process proceeds to step S111.

[0241] This is the case when the AEB function is in the OFF state.

[0242] In step S111, an operation display is instructed in the same manner as in step S9 of FIG.

[0243] In step S112, measurement of the display time begins, similarly to the process in step S10 of FIG.

[0244] As a result, the state of the AEB function transitions from the OFF state to the ON state.

[0245] Thereafter, the process returns to step S102, and the processes from step S102 onwards are executed.

[0246] On the other hand, if it is determined in step S106 that the conditions for operating the driving assistance function are not met, the process proceeds to step S113.

[0247] In step S113, it is determined whether or not the driving assistance function is in operation, similarly to the process of step S11 in Fig. 4. If it is determined that the driving assistance function is in operation, the process proceeds to step S114.

[0248] In step S114, the driving assistance function is stopped, similarly to the process in step S12 of FIG.

[0249] As a result, the state of the AEB function transitions from the operation ON state to the operation OFF and displaying state.

[0250] Thereafter, the process proceeds to step S115.

[0251] On the other hand, if it is determined in step S113 that the driving assistance function is not in operation, the process of step S114 is skipped and the process proceeds to step S115.

[0252] In step S115, it is determined whether or not the remote operation controller 12 is displaying an operation, similar to the process in step S8 of Fig. 4. If it is determined that the remote operation controller 12 is displaying an operation, the process proceeds to step S116.

[0253] This is the case when the AEB function is in the OFF state and the display is in progress.

[0254] In step S116, the driving assistance control unit 120 determines whether the display time of the remote driving controller 12 has exceeded the minimum display time. For example, the AEB controller 131 of the driving assistance control unit 120 checks the elapsed time from when it started measuring the display time after sending the "AEB" operation display ON command in the processing of step S111, or after sending the "AEB" advance notice display ON command in the processing of step S122 described below. If the elapsed time is equal to or greater than the predetermined minimum display time, the AEB controller 131 determines that the total time of the advance notice display and operation display of the AEB function of the remote driving controller 12 has exceeded the minimum display time, and the processing proceeds to step S117.

[0255] In step S117, an instruction to stop the display is issued, similar to the processing in step S14 of FIG.

[0256] As a result, the state of the AEB function transitions from the operation-off and displaying state to the operation-off state.

[0257] Thereafter, the process returns to step S102, and the processes from step S102 onwards are executed.

[0258] On the other hand, if it is determined in step S116 that the display time of the remote operation controller 12 has not elapsed beyond the minimum display time, the process of step S117 is skipped and the process returns to step S102. Then, the processes from step S102 onward are executed.

[0259] On the other hand, if it is determined in step S115 that the remote operation controller 12 is not displaying an operation, the process proceeds to step S118.

[0260] In step S118, it is determined whether the advance notice condition for the driving assistance function is satisfied, similar to the processing in step S19 of Fig. 5. If it is determined that the advance notice condition for the driving assistance function is not satisfied, the processing proceeds to step S119.

[0261] In step S119, similar to the process of step S109, it is determined whether or not the remote driving controller 12 is displaying a warning. If it is determined that the remote driving controller 12 is displaying a warning, the process proceeds to step S120.

[0262] This is the case when the AEB function is in the operation notice state.

[0263] In step S120, an instruction to stop the display is issued, similar to the processing in step S14 of FIG.

[0264] As a result, the state of the AEB function transitions from the operation notice state to the operation OFF state.

[0265] Thereafter, the process returns to step S102, and the processes from step S102 onwards are executed.

[0266] On the other hand, if it is determined in step S119 that the remote operation controller 12 is not displaying a notice, the process returns to step S102, whereupon the processes from step S102 onward are executed.

[0267] This is the case when the AEB function is in the OFF state.

[0268] On the other hand, if it is determined in step S118 that the advance notice condition for the driving assistance function is met, the process proceeds to step S121.

[0269] In step S121, similar to the process of step S109, it is determined whether or not the remote driving controller 12 is displaying a warning. If it is determined that the remote driving controller 12 is not displaying a warning, the process proceeds to step S122.

[0270] This is the case when the AEB function is in the OFF state.

[0271] In step S122, the driving assistance control unit 120 instructs the warning display. For example, the AEB controller 131 of the driving assistance control unit 120 transmits an “AEB” warning display ON command to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22.

[0272] In step S123, measurement of the display time starts, similarly to the process in step S10 of FIG.

[0273] As a result, the state of the AEB function transitions from the operation OFF state to the operation notice state.

[0274] Thereafter, the process returns to step S102, and the processes from step S102 onwards are executed.

[0275] On the other hand, if it is determined in step S121 that the remote operation controller 12 is displaying a notice, the process returns to step S102, whereupon the processes from step S102 onward are executed.

[0276] This is the case when the AEB function is in the operation notice state.

[0277] <Processing of Remote Driving Controller 12> Next, with reference to the flowchart of FIG. 17, a process executed by the remote driving controller 12 in response to the process of the vehicle 11 shown in FIGS. 14 and 15 will be described.

[0278] In step S151, operation information is acquired in the same manner as in step S51 of FIG.

[0279] In step S152, a remote operation command is transmitted in the same manner as in step S52 of FIG.

[0280] In step S153, the presentation control unit 214 determines whether a warning display command has been issued. For example, when the presentation control unit 214 receives an “AEB” warning display ON command from the vehicle 11 via the base station 21, the communication network 22, and the communication unit 211, the presentation control unit 214 determines that a warning display command has been issued, and the process proceeds to step S154.

[0281] In step S154, the output unit 215 performs a warning display under the control of the presentation control unit 214. For example, the display device 254 of the output unit 215 displays information warning of the activation of the AEB function of the vehicle 11 in the status display area 254A.

[0282] Thereafter, the process proceeds to step S155.

[0283] On the other hand, if it is determined in step S153 that advance notice display has not been instructed, the process of step S154 is skipped and the process proceeds to step S155.

[0284] In step S155, it is determined whether or not an operation display command has been issued, similarly to the process in step S53 of Fig. 8. If it is determined that an operation display command has been issued, the process proceeds to step S156.

[0285] In step S156, an operation display is performed in the same manner as in step S54 of FIG.

[0286] Thereafter, the process proceeds to step S157.

[0287] On the other hand, if it is determined in step S155 that the activation display has not been instructed, the process of step S156 is skipped and the process proceeds to step S157.

[0288] In step S157, it is determined whether or not an instruction to stop the display has been issued, similarly to the processing in step S55 of Fig. 8. If it is determined that an instruction to stop the display has been issued, the processing proceeds to step S158.

[0289] In step S158, the output unit 215 stops the instructed display under the control of the presentation control unit 214. For example, the display device 254 of the output unit 215 stops the advance notice display or the operation display of the AEB function of the vehicle 11.

[0290] Thereafter, the process returns to step S151, and the processes from step S151 onwards are executed.

[0291] On the other hand, if it is determined in step S157 that the display has not been stopped, the process returns to step S151, and the processes from step S151 onwards are then executed.

[0292] FIG. 18 is a sequence diagram showing an example of a process related to an operation display of the AEB function.

[0293] For example, when the advance notice condition of the AEB function is met, the AEB controller 131 of the vehicle 11 transmits an “AEB” advance notice display ON command to the remote driving controller 12 via the communication unit 111 .

[0294] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" advance notice display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, starts the advance notice display of the AEB function.

[0295] Next, when the conditions for operating the AEB function are met, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to control the AEB function. In response, the drive / brake device 117 executes the AEB function. The AEB controller 131 also instructs the output unit 121 to display an operation indication of the AEB function. In response, the output unit 121 executes an operation indication of the AEB function.

[0296] Furthermore, the AEB controller 131 transmits an “AEB” operation display ON command to the remote operation controller 12 via the communication unit 111 .

[0297] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" operation display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, starts displaying the operation of the AEB function.

[0298] Next, when the activation conditions for the AEB function are no longer satisfied, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to stop control of the AEB function. In response to this, the drive / brake device 117 stops the AEB function.

[0299] At this time, if the total time of the AEB function warning display and operation display on the remote driving controller 12 (the elapsed time from the start of the AEB function warning display) has not exceeded the minimum display time, the AEB function operation display will continue in the vehicle 11 and the remote driving controller 12.

[0300] Next, when the total time of the AEB function warning display and operation display of the remote driving controller 12 has exceeded the minimum display time, the AEB controller 131 of the vehicle 11 instructs the output unit 121 to stop the operation display of the AEB function. In response to this, the output unit 121 stops the operation display of the AEB function.

[0301] In addition, the AEB controller 131 transmits an “AEB” display OFF command to the remote operation controller 12 via the communication unit 111 .

[0302] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" display OFF command via the communication unit 211. The output unit 215 stops the display related to the AEB function under the control of the presentation control unit 214. In this case, the display of the operation of the AEB function is stopped.

[0303] In addition, if the conditions for operating the AEB function are not met and the total time of the AEB function warning display and operation display of the remote driving controller 12 has already exceeded the minimum display time, the operation display of the AEB function of the output unit 121 of the vehicle 11 is immediately stopped and an "AEB" display OFF command is sent to the remote driving controller 12.

[0304] In this way, the remote driving operator is notified of the activation of the AEB function at an appropriate time regardless of the quality of communication between the vehicle 11 and the remote driving controller 12. As a result, the remote driving operator is able to quickly recognize that the AEB function may be activated and respond appropriately (for example, to prepare for the next driving operation), thereby improving safety.

[0305] FIG. 19 is a sequence diagram showing an example of processing when the advance notice condition for the AEB function is not met during the advance notice display of the AEB function.

[0306] For example, when the advance notice condition of the AEB function is met, the AEB controller 131 of the vehicle 11 transmits an “AEB” advance notice display ON command to the remote driving controller 12 via the communication unit 111 .

[0307] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" advance notice display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, starts the advance notice display of the AEB function.

[0308] Next, if the advance notice condition of the AEB function is not met, the AEB controller 131 of the vehicle 11 transmits an “AEB” display OFF command to the remote driving controller 12 via the communication unit 111 .

[0309] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" display OFF command via the communication unit 211. The output unit 215 stops the display related to the AEB function under the control of the presentation control unit 214. In this case, the advance notice display of the AEB function is stopped.

[0310] This may cause the remote driving operator to temporarily brace himself when he sees the warning display of the AEB function, but this does not pose a particular problem because the AEB function is not actually activated and no dangerous situation has occurred in the vehicle 11.

[0311] The processing related to the LKA function is almost the same as the processing related to the AEB function described above, except that the steering device 118 and the LKA controller 132 perform the processing instead of the drive / braking device 117 and the AEB controller 131, and that display control commands for the LKA function are used instead of the display control commands for the AEB function.

[0312] Therefore, similar to the AEB function, the remote driving operator is notified of the activation of the LKA function at an appropriate time regardless of the quality of communication between the vehicle 11 and the remote driving controller 12. As a result, the remote driving operator is able to quickly recognize that the LKA function may be activated and respond appropriately (for example, to prepare for the next driving operation), thereby improving safety.

[0313] It should be noted that the above processing does not apply to the collision warning function, since the remote driving controller 12 does not provide a warning display.

[0314] <Third Embodiment of Processing in Remote Driving System 1> Next, a third embodiment of processing in the remote driving system 1 will be described with reference to FIGS. 20 to 25. FIG.

[0315] When the driving assistance function is activated during remote driving by the remote driving operator, the vehicle 11 (driving assistance control unit 120) intervenes in the control of the vehicle 11.

[0316] As described above, there are various types of driving assistance functions. Furthermore, the control target in which the vehicle 11 intervenes varies depending on the type of driving assistance function. For example, depending on the type of driving assistance function, the vehicle 11 may be the main controller of acceleration / deceleration control, the main controller of steering control, or the main controller of both acceleration / deceleration control and steering control, instead of the remote driving controller 12 (remote driving operator).

[0317] FIG. 20 shows an example of the relationship between driving assistance functions and control entities.

[0318] For example, during remote driving, the remote driving controller 12 (remote driving operator) is the main controller of acceleration / deceleration control and steering control.

[0319] On the other hand, when the AEB function is activated, the vehicle 11 (AEB controller 131) becomes the main controller of acceleration / deceleration control, and the remote driving controller 12 (remote driving operator) becomes the main controller of steering control.

[0320] Furthermore, when the LKA function is activated, the remote driving controller 12 (remote driving operator) becomes the main controller of acceleration / deceleration control, and the vehicle 11 (LKA controller 132) becomes the main controller of steering control.

[0321] Therefore, the control entity for acceleration / deceleration control and steering control is switched depending on the operating state of each remote driving function. This switching of the control entity is performed at the discretion of the driving assistance control unit 120, so the control entity may be switched at a timing unexpected by the remote driving operator.

[0322] For example, when a driver is sitting in the driver's seat of the vehicle 11 and driving, the driver can immediately recognize the intervention and method of control (e.g., direction, acceleration / deceleration, etc.) by the driving assistance function by experiencing the movement of the vehicle 11 (e.g., acceleration, etc.) and recognizing information presented on the instrument panel of the vehicle 11. Therefore, the driver can quickly make a decision on the next driving operation.

[0323] On the other hand, since the remote driving operator is outside the vehicle 11 and cannot physically feel the movement of the vehicle 11, he or she cannot quickly recognize the intervention of control by the driving assistance function. Therefore, the remote driving operator may be slow in assessing the situation, which may delay the remote driving operator's decision on the next driving operation. For example, if a communication delay of 200 ms occurs while the vehicle 11 is traveling at 60 km / h, the vehicle 11 will move 3 meters during that time, so a delayed response in an emergency increases the possibility of a fatal accident occurring.

[0324] In contrast to this, in this embodiment, the remote driving controller 12 presents the control status of the driving assistance function.

[0325] For simplicity of explanation, the following description will be focused on processing related to the AEB function among the driving assistance functions.

[0326] <Processing of Vehicle 11> Processing of the vehicle 11 will be described with reference to the flowcharts of FIGS. 21 and 22.

[0327] This process is started, for example, when the vehicle 11 becomes capable of being remotely driven, and is ended when the vehicle 11 becomes unable to be remotely driven.

[0328] In steps S201 to S205, the same processes as in steps S1 to S5 in FIG. 4 are executed.

[0329] In step S206, it is determined whether or not the activation conditions for the driving assistance function are met, similar to the processing in step S6 of Fig. 4. If it is determined that the activation conditions for the driving assistance function are met, the processing proceeds to step S207.

[0330] In step S207, the driving assistance function is executed in the same manner as in step S7 of FIG.

[0331] In step S208, it is determined whether or not the remote operation controller 12 is displaying an operation, similar to the process in step S8 of Fig. 4. If it is determined that the remote operation controller 12 is not displaying an operation, the process proceeds to step S209.

[0332] In step S209, measurement of the display time starts, similarly to the process in step S9 of FIG.

[0333] The process then proceeds to step S210.

[0334] On the other hand, if it is determined in step S208 that the remote operation controller 12 is displaying an operation, the process of step S209 is skipped and the process proceeds to step S210.

[0335] In step S210, the driving assistance control unit 120 instructs the display of an operation display and driving assistance control information.

[0336] For example, the AEB controller 131 of the driving assistance control unit 120 transmits an "AEB" operation display ON command to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22. At this time, the AEB controller 131 adds driving assistance control information to the "AEB" operation display ON command. As a result, the "AEB" operation display ON command includes the driving assistance control information.

[0337] The driving assistance control information includes, for example, acceleration / deceleration control information and steering control information.

[0338] The acceleration / deceleration control information includes, for example, whether or not to intervene in acceleration control and the control amount (acceleration amount), and whether or not to intervene in braking control and the control amount (deceleration amount).

[0339] The steering control information includes, for example, whether or not to intervene in steering control and the amount of control (steering amount).

[0340] Thereafter, the process returns to step S202, and the processes from step S202 onward are executed.

[0341] On the other hand, if it is determined in step S206 that the conditions for operating the driving assistance function are not met, the process proceeds to step S211.

[0342] In step S211, it is determined whether or not the driving assistance function is in operation, similarly to the process of step S11 in Fig. 4. If it is determined that the driving assistance function is in operation, the process proceeds to step S212.

[0343] In step S212, the driving assistance function is stopped, similarly to the process in step S12 of FIG.

[0344] In step S213, similar to the process of step S13 in Fig. 4, it is determined whether or not the display time of the remote operation controller 12 has elapsed beyond the minimum display time. If it is determined that the display time of the remote operation controller 12 has not elapsed beyond the minimum display time, the process proceeds to step S214.

[0345] In step S214, similar to the process in step S210, an instruction to display the operation display and the driving support control information is issued, thereby notifying the remote driving controller 12 that the control amount by the driving support function has become zero.

[0346] Thereafter, the process returns to step S202, and the processes from step S202 onward are executed.

[0347] On the other hand, if it is determined in step S213 that the display time of the remote operation controller 12 has elapsed for more than the minimum display time, the process proceeds to step S215.

[0348] In step S215, an instruction to stop the display is issued, similar to the processing in step S14 of FIG.

[0349] Thereafter, the process returns to step S202, and the processes from step S202 onward are executed.

[0350] On the other hand, if it is determined in step S211 that the driving assistance function is not in operation, the process proceeds to step S216.

[0351] In steps S216 to S225, the same processes as steps S15 to S24 in Fig. 5 are executed, except for the process of step S224. That is, the process of step S224 is executed instead of the process of step S23 in Fig. 4.

[0352] In step S224, similar to the processing in step S210, an instruction is given to display the operation display and the driving assistance control information. That is, when the advance notice condition is met, the remote driving controller 12 is instructed to display the operation display and the driving assistance control information.

[0353] <Processing of Remote Driving Controller 12> Next, with reference to the flowchart of FIG. 23, a process executed by the remote driving controller 12 in response to the process of the vehicle 11 shown in FIGS. 21 and 22 will be described.

[0354] In step S251, operation information is acquired in the same manner as in step S51 of FIG.

[0355] In step S252, a remote operation command is transmitted in the same manner as in step S52 of FIG.

[0356] In step S253, the presentation control unit 214 determines whether or not an instruction to display the operation display and the driving assistance control information has been issued. For example, when the presentation control unit 214 receives an "AEB" operation display ON command from the vehicle 11 via the base station 21, the communication network 22, and the communication unit 211, the presentation control unit 214 determines that an instruction to display the operation display and the driving assistance control information has been issued, and the process proceeds to step S254.

[0357] In step S254, the output unit 215 displays the operation indication and the driving assistance control information under the control of the presentation control unit 214. For example, the display device 254 of the output unit 215 displays information indicating that the AEB function of the vehicle 11 is in operation and information indicating the operation state of the AEB function in the status display area 254A.

[0358] Thereafter, the process proceeds to step S255.

[0359] On the other hand, if it is determined in step S253 that the display of the operation display and the driving support control information has not been instructed, the process of step S254 is skipped and the process proceeds to step S255.

[0360] In step S255, it is determined whether or not an instruction to stop the display has been issued, similarly to the processing in step S55 of Fig. 8. If it is determined that an instruction to stop the display has been issued, the processing proceeds to step S256.

[0361] In step S256, the output unit 215 stops the instructed display under the control of the presentation control unit 214. For example, the display device 254 of the output unit 215 stops displaying the operation display of the AEB function of the vehicle 11 and the driving assistance control information.

[0362] Thereafter, the process returns to step S251, and the processes from step S251 onwards are executed.

[0363] On the other hand, if it is determined in step S255 that the display has not been stopped, the process returns to step S251, and the processes from step S251 onwards are then executed.

[0364] FIG. 24 is a sequence diagram showing an example of a process related to an operation display of the AEB function.

[0365] For example, when the advance notice condition for the AEB function is met, the AEB controller 131 of the vehicle 11 transmits an "AEB" operation display ON command to the remote driving controller 12 via the communication unit 111. As described above, the "AEB" operation display ON command includes driving assistance control information.

[0366] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" operation display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, starts displaying the operation display of the AEB function and the driving assistance control information.

[0367] Next, when the conditions for operating the AEB function are met, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to control the AEB function. In response, the drive / brake device 117 executes the AEB function. The AEB controller 131 also instructs the output unit 121 to display an operation indication of the AEB function. In response, the output unit 121 executes an operation indication of the AEB function.

[0368] Furthermore, while the AEB function is in operation, the AEB controller 131 of the vehicle 11 periodically transmits an "AEB" operation display ON command to the remote driving controller 12 via the communication unit 111. As described above, the "AEB" operation display ON command includes driving assistance control information.

[0369] In response to this, the presentation control unit 214 of the remote driving controller 12 periodically receives the "AEB" operation display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, periodically updates the display of the driving assistance control information.

[0370] Next, when the activation conditions for the AEB function are no longer satisfied, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to stop control of the AEB function. In response to this, the drive / brake device 117 stops the AEB function.

[0371] At this time, if the display time of the AEB function operation display of the remote driving controller 12 has not elapsed the minimum display time, the AEB controller 131 of the vehicle 11 transmits an "AEB" operation display ON command to the remote driving controller 12 via the communication unit 111. As described above, the "AEB" operation display ON command has driving assistance control information added thereto.

[0372] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" operation display ON command via the communication unit 211. The output unit 215 updates the display of the driving assistance control information under the control of the presentation control unit 214. This notifies the remote driving operator that the control amount of the vehicle 11 by the AEB function has become zero.

[0373] Next, when the display time of the AEB function operation indication of the remote driving controller 12 has exceeded the minimum display time, the AEB controller 131 of the vehicle 11 instructs the output unit 121 to stop the AEB function operation indication. In response, the output unit 121 stops the AEB function operation indication. The AEB controller 131 also transmits an "AEB" display OFF command to the remote driving controller 12 via the communication unit 111.

[0374] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" display OFF command via the communication unit 211. The output unit 215 stops displaying information related to the AEB function under the control of the presentation control unit 214. In this case, the display of the operation of the AEB function and the display of the driving assistance control information are stopped.

[0375] In addition, if the conditions for operating the AEB function are not met and the display time of the AEB function on the remote driving controller 12 (display time of the AEB function operation display) has already exceeded the minimum display time, the AEB function operation display on the output unit 121 of the vehicle 11 is immediately stopped and an "AEB" display OFF command is sent to the remote driving controller 12.

[0376] FIG. 25 shows an example of the driving assistance control information displayed in the status display area 254A of the display device 254.

[0377] The display device 254 displays an image of the direction of travel (forward) of the vehicle 11, and the preceding vehicle 301 is displayed within the image.

[0378] FIG. 25A shows an example of the status display area 254A when the distance between the vehicle 301 and the vehicle is large and none of the driving assistance functions are operating.

[0379] In this example, icons 321A to 321C and bar graphs 322B to 322C are displayed. Icons 321A to 321C are arranged side by side in the horizontal direction. Bar graphs 322B to 322C are arranged side by side in the horizontal direction. Icon 321A and bar graph 322A are arranged side by side in the vertical direction. Icon 321B and bar graph 322B are arranged side by side in the vertical direction. Icon 321C and bar graph 322C are arranged side by side in the vertical direction.

[0380] Hereinafter, when it is not necessary to distinguish between the icons 321A to 321C, they will be simply referred to as icons 321. Hereinafter, when it is not necessary to distinguish between the bar graphs 322B to 322C, they will be simply referred to as bar graphs 322.

[0381] The icon 321A has a design that resembles the steering wheel of the vehicle 11. The bar graph 322A indicates the steering direction and steering amount by the driving assistance function.

[0382] When the steering amount is 0, the scale of the bar graph 322A is positioned in the center. As the steering amount to the left increases, the scale of the bar graph 322A moves more to the left, and as the steering amount to the right increases, the scale of the bar graph 322A moves more to the right.

[0383] The icon 321B has a design that resembles the brake pedal of the vehicle 11. The bar graph 322B indicates the amount of deceleration caused by the driving assistance function.

[0384] The icon 321C has a design that resembles the accelerator pedal of the vehicle 11. The bar graph 322C indicates the amount of acceleration achieved by the driving assistance function.

[0385] When none of the driving assistance functions are operating, for example, each icon 321 and each bar graph 322 are displayed in an inconspicuous manner (for example, in a light color).

[0386] In this way, the operating status of each driving assistance function is presented by the icon 321 and the bar graph 322, and for example, the remote driving operator can recognize the type of driving assistance function currently in operation and its control status.

[0387] FIG. 25B shows an example of the status display area 254A when the distance to the vehicle 301 is narrowed and the advance notice condition of the AEB function is met.

[0388] In this case, for example, the display mode of icon 321B and bar graph 322B changes, and they are displayed more highlighted than the other icons 321 and bar graph 322. Note that because the AEB function has not yet been activated, the value of bar graph 322B (the amount of deceleration due to the driving assistance function) is 0. Also, the words "Watch out for distance between vehicles" are displayed above icon 321B. In this way, activation of the AEB function is announced in advance, and the amount of deceleration due to the AEB function is displayed.

[0389] FIG. 25C shows an example of the status display area 254A in which the distance to the vehicle 301 has become even closer, the conditions for activating the AEB function are met, and the AEB function is in operation.

[0390] In this case, the value of the bar graph 322B changes in accordance with the change in the amount of deceleration due to the AEB function.

[0391] The processing related to the LKA function is almost the same as the processing related to the AEB function described above, except that the steering device 118 and the LKA controller 132 perform the processing instead of the drive / braking device 117 and the AEB controller 131, and that display control commands for the LKA function are used instead of the display control commands for the AEB function.

[0392] In addition, in the advance notice display and operation display of the LKA function, the icon 321A and the bar graph 322A are highlighted, and the value of the bar graph 322A changes as the steering amount of the LKA function changes.

[0393] It should be noted that the collision warning function does not involve control of the vehicle 11, and therefore the above processing does not apply.

[0394] This allows the remote driving operator to recognize the operation of each driving assistance function at the appropriate time and quickly and accurately recognize the control amount of each driving assistance function. As a result, the remote driving operator can quickly and accurately recognize whether the remote driving operator or the vehicle 11 (driving assistance function) is in control of the steering, deceleration, and acceleration of the vehicle 11, and can respond appropriately (for example, prepare for the next driving operation), thereby improving safety.

[0395] For example, in the third embodiment of the processing of the remote driving system 1, advance notice of each driving assistance function may be displayed, similar to the second embodiment.

[0396] <Fourth Embodiment of Processing in Remote Driving System 1> Next, a fourth embodiment of processing in the remote driving system 1 will be described with reference to FIGS.

[0397] For example, when the driving assistance function of the vehicle 11 is activated and the control entity is switched to the vehicle 11, a discrepancy may occur between the control amount of the vehicle 11 based on the operation of the input unit 212 (e.g., the brake pedal 251, the accelerator pedal 252, the steering wheel 253, etc.) of the remote driving controller 12 by the remote driving operator and the control amount of the vehicle 11 by the driving assistance function. For example, depending on the situation, the remote driving operator may become confused and accidentally press the brake pedal 251 or the accelerator pedal 252 or steer the steering wheel 253 in an inappropriate direction.

[0398] On the other hand, when the control of the vehicle 11 by the driving assistance function is completed and the control entity is switched to the remote driving controller 12 (remote driving operator), vehicle control is performed based on the operation of the input unit 212 of the remote driving controller 12. At this time, if the operation of the remote driving operator is inappropriate, the vehicle 11 may perform an unintended sudden operation (for example, sudden acceleration, sudden steering, sudden braking, etc.), which may result in a dangerous situation.

[0399] In contrast to this, this embodiment is intended to safely hand over driving to a remote driving operator after the driving assistance function is activated.

[0400] <Processing of Vehicle 11> Processing of the vehicle 11 will be described with reference to the flowcharts of FIGS. 26 and 27.

[0401] This process is started, for example, when the vehicle 11 becomes capable of being remotely driven, and is ended when the vehicle 11 becomes unable to be remotely driven.

[0402] In steps S301 to S305, the same processes as in steps S1 to S5 in FIG. 4 are executed.

[0403] In step S306, it is determined whether or not the activation conditions for the driving assistance function are met, similar to the processing in step S6 of Fig. 4. If it is determined that the activation conditions for the driving assistance function are met, the processing proceeds to step S307.

[0404] In step S307, the driving assistance function is executed in the same manner as in step S7 of FIG.

[0405] In step S308, similar to the processing in step S210 of FIG. 21, an instruction is given to display the operation display and the driving support control information.

[0406] Thereafter, the process returns to step S302, and the processes from step S302 onwards are executed.

[0407] On the other hand, if it is determined in step S306 that the conditions for operating the driving assistance function are not met, the process proceeds to step S309.

[0408] In step S309, it is determined whether or not the driving assistance function is in operation, similarly to the process of step S11 in Fig. 4. If it is determined that the driving assistance function is in operation, the process proceeds to step S310.

[0409] In step S310, the vehicle 11 determines whether the conditions for taking over to remote driving are met.

[0410] Specifically, the remote driving execution unit 119 detects the control amounts of acceleration / deceleration control and steering control of the vehicle 11 by the remote driving controller 12 (remote driving operator) based on the remote driving command received from the remote driving controller 12. The remote driving execution unit 119 supplies information indicating the detection results of the control amounts of acceleration / deceleration control and steering control by the remote driving controller 12 to the driving assistance control unit 120.

[0411] For example, when the AEB function is operating, if the difference between the control amount of acceleration / deceleration control by the AEB function and the control amount of acceleration / deceleration control by the remote driving controller 12 is less than a predetermined threshold, the AEB controller 131 determines that the remote driving handover condition is met, and processing proceeds to step S311.

[0412] For example, when the LKA function is operating, if the difference between the control amount of steering control by the LKA function and the control amount of steering control by the remote driving controller 12 is less than a predetermined threshold, the LKA controller 132 determines that the remote driving handover condition has been met, and processing proceeds to step S311.

[0413] In step S311, the driving assistance function is stopped, similarly to the process in step S12 of FIG.

[0414] In step S312, an instruction to stop the display is issued, similar to the processing in step S14 of FIG.

[0415] Thereafter, the process returns to step S302, and the processes from step S302 onwards are executed.

[0416] On the other hand, in step S310, for example, if the AEB function is operating, the AEB controller 131 determines that the remote driving handover condition is not met if the difference between the control amount of acceleration / deceleration control by the AEB function and the control amount of acceleration / deceleration control by the remote driving controller 12 is greater than or equal to a predetermined threshold, and processing proceeds to step S313.

[0417] Also, in step S310, for example, when the LKA function is operating, if the difference between the control amount of steering control by the LKA function and the control amount of steering control by the remote driving controller 12 is equal to or greater than a predetermined threshold, the LKA controller 132 determines that the remote driving handover condition is not met, and processing proceeds to step S313.

[0418] In step S313, the driving assistance control unit 120 instructs the driver to display handover information that notifies the driver that the driving assistance function currently in operation will be handed over to remote driving.

[0419] For example, when the AEB function is in operation, the AEB controller 131 of the driving assistance control unit 120 transmits an “AEB” takeover command to the remote driving controller 12 via the communication unit 111 , the base station 21 , and the communication network 22 .

[0420] For example, when the LKA function is in operation, the LKA controller 132 of the driving assistance control unit 120 transmits an “LKA” takeover command to the remote driving controller 12 via the communication unit 111, the base station 21, and the communication network 22.

[0421] Thereafter, the process returns to step S302, and the processes from step S302 onwards are executed.

[0422] On the other hand, if it is determined in step S309 that the driving assistance function is not in operation, the process proceeds to step S314.

[0423] In steps S314 to S316, the same processes as in steps S222 to S224 in FIG. 22 are executed.

[0424] Thereafter, the process returns to step S302, and the processes from step S302 onwards are executed.

[0425] <Processing of Remote Driving Controller 12> Next, with reference to the flowchart of FIG. 28, a process executed by the remote driving controller 12 in response to the process of the vehicle 11 shown in FIGS. 26 and 27 will be described.

[0426] In steps S351 to S354, the same processes as in steps S251 to S254 in FIG. 23 are executed.

[0427] In step S355, the presentation control unit 214 determines whether a takeover display has been instructed. For example, when the presentation control unit 214 receives an “AEB” takeover command or an “LKA” takeover command from the vehicle 11 via the base station 21, the communication network 22, and the communication unit 211, the presentation control unit 214 determines that a takeover display has been instructed, and the process proceeds to step S356.

[0428] In step S356, the output unit 215 performs a transition display under the control of the presentation control unit 214. Details of the transition display will be described later with reference to Figs.

[0429] Thereafter, the process proceeds to step S357.

[0430] On the other hand, if it is determined in step S355 that the takeover display has not been instructed, the process of step S356 is skipped and the process proceeds to step S357.

[0431] In step S357, it is determined whether or not an instruction to stop the display has been issued, similarly to the processing in step S55 of Fig. 8. If it is determined that an instruction to stop the display has been issued, the processing proceeds to step S358.

[0432] In step S358, the output unit 215 stops the instructed display under the control of the presentation control unit 214. For example, the display device 254 of the output unit 215 stops the operation display of the driving assistance function and the driving assistance control information, or the handover display.

[0433] Thereafter, the process returns to step S351, and the processes from step S351 onwards are executed.

[0434] On the other hand, if it is determined in step S357 that the display has not been stopped, the process returns to step S351, and the processes from step S351 onwards are then executed.

[0435] FIG. 29 is a sequence diagram showing an example of a process related to an operation display of the AEB function.

[0436] For example, when the advance notice condition for the AEB function is met, the AEB controller 131 of the vehicle 11 transmits an "AEB" operation display ON command to the remote driving controller 12 via the communication unit 111. As described above, the "AEB" operation display ON command includes driving assistance control information.

[0437] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" operation display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, starts displaying the operation display of the AEB function and the driving assistance control information.

[0438] Next, when the conditions for operating the AEB function are met, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to control the AEB function. In response, the drive / brake device 117 executes the AEB function. The AEB controller 131 also instructs the output unit 121 to display an operation indication of the AEB function. In response, the output unit 121 executes an operation indication of the AEB function.

[0439] Furthermore, while the AEB function is in operation, the AEB controller 131 of the vehicle 11 periodically transmits an "AEB" operation display ON command to the remote driving controller 12 via the communication unit 111. As described above, the "AEB" operation display ON command includes driving assistance control information.

[0440] In response to this, the presentation control unit 214 of the remote driving controller 12 periodically receives the "AEB" operation display ON command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, periodically updates the display of the driving assistance control information.

[0441] Next, when the conditions for operating the AEB function are no longer met, the AEB controller 131 of the vehicle 11 periodically transmits an “AEB” takeover command to the remote driving controller 12 .

[0442] In response to this, the presentation control unit 214 of the remote driving controller 12 periodically receives the “AEB” takeover command via the communication unit 211. The output unit 215 starts and updates the takeover display of the AEB function under the control of the presentation control unit 214.

[0443] Next, when the conditions for taking over the AEB function are met, the AEB controller 131 of the vehicle 11 instructs the drive / brake device 117 to stop control of the AEB function. In response, the drive / brake device 117 stops the AEB function. The AEB controller 131 also instructs the output unit 121 to stop displaying the operation of the AEB function. In response, the output unit 121 stops displaying the operation of the AEB function.

[0444] Furthermore, the AEB controller 131 transmits an “AEB” display OFF command to the remote operation controller 12 via the communication unit 111 .

[0445] In response to this, the presentation control unit 214 of the remote driving controller 12 receives the "AEB" display OFF command via the communication unit 211. The output unit 215, under the control of the presentation control unit 214, stops the takeover display of the AEB function.

[0446] FIG. 30 shows an example of the driving assistance control information and handover display displayed in the status display area 254A of the display device 254 when the AEB function is activated.

[0447] In the status display area 254A, icons 321A to 321C and bar graphs 322B to 322C are displayed, similar to the example of FIG.

[0448] Similar to FIG. 25B, FIG. 30A shows an example of the display in the status display area 254A when the AEB function is activated.

[0449] FIG. 30B shows an example of the takeover display of the AEB function.

[0450] In this example, a bar graph 323B is displayed below the bar graph 322B. The bar graph 323B indicates the amount of deceleration of the vehicle 11 caused by the remote driving controller 12. Note that, because the AEB function is still in operation, the amount of deceleration of the vehicle 11 caused by the remote driving controller 12 is calculated based on the amount of operation of the brake pedal 251 of the remote driving controller 12, and is not actually used to control the vehicle 11.

[0451] On the other hand, although the AEB function is in operation, the AEB function's operating conditions are not met, so the amount of deceleration by the AEB function is zero.

[0452] Also displayed is information about how to switch from the AEB function to remote driving. That is, a message is displayed notifying the user that the AEB function is being switched over to remote driving, and urging the user to ease up on the brake pedal 251 of the remote driving controller 12. Specifically, messages such as "Returning control to remote driving" and "Release the brake" are displayed.

[0453] This allows the remote driving operator to easily recognize the difference between the amount of deceleration caused by the vehicle 11 (AEB function) and the amount of deceleration caused by the remote driving operator's own operation of the vehicle 11. The remote driving operator can then adjust the operation of the brake pedal 251 to match the amount of deceleration caused by the vehicle 11, allowing for a swift and safe handover of driving.

[0454] C in Fig. 30 shows an example of the display in the status display area 254A after the handover of driving to the remote driving operator. Since none of the driving support functions are active, the display of each icon 321 and each bar graph 322 is the same as A in Fig. 25. In addition, a message indicating the completion of the handover of driving is displayed. Specifically, the message displayed reads, "Control has been returned to remote driving."

[0455] FIG. 31 shows an example of the driving assistance control information and handover display displayed in the status display area 254A of the display device 254 when the LKA function is activated.

[0456] In the status display area 254A, icons 321A to 321C and bar graphs 322B to 322C are displayed, similar to the example of FIG.

[0457] FIG. 31A shows an example of the display in the status display area 254A when the LKA function is activated.

[0458] In this case, for example, the display mode of the icon 321A and the bar graph 322A changes, and they are displayed more highlighted than the other icons 321 and bar graph 322. Furthermore, as the steering amount changes due to the LKA function, the direction and amount of rotation of the icon 321A and the position of the scale of the bar graph 322A change.

[0459] FIG. 31B shows an example of a display of the LKA function takeover.

[0460] In this example, a bar graph 323A is displayed below the bar graph 322A. The bar graph 323A shows the steering direction and steering amount of the vehicle 11 by the remote driving controller 12 in the same manner as the bar graph 322A. Note that, because the LKA function is still operating, the steering amount of the vehicle 11 by the remote driving controller 12 is calculated based on the operation amount of the steering wheel 253 of the remote driving controller 12, and is not actually used to control the vehicle 11.

[0461] On the other hand, although the LKA function is in operation, the condition for operating the LKA function is not met, so the steering amount by the LKA function is zero.

[0462] Also displayed is information about how to switch from the LKA function to remote driving. That is, a message is displayed notifying the user that control is being switched from the LKA function to remote driving, and prompting the user to adjust the operation of the steering wheel 253 of the remote driving controller 12. Specifically, messages such as "Returning control to remote driving" and "Adjust the steering wheel" are displayed.

[0463] This allows the remote driving operator to easily recognize the difference between the amount of steering by the vehicle 11 (LKA function) and the amount of steering by the remote driving operator himself / herself of the vehicle 11. The remote driving operator can then adjust the operation of the steering wheel 253 to match the amount of steering by the vehicle 11, enabling a swift and safe handover of driving to be carried out.

[0464] C in Fig. 31 shows an example of the display in the status display area 254A after the handover of driving to the remote driving operator. Since none of the driving support functions are active, the display of each icon 321 and each bar graph 322 is the same as A in Fig. 25. In addition, a message indicating the completion of the handover of driving is displayed. Specifically, the message displayed reads, "Control has been returned to remote driving."

[0465] For example, in the fourth embodiment of the processing of the remote driving system 1, advance notice of each driving assistance function may be displayed, similar to the second embodiment.

[0466] <<2. Modifications>> Modifications of the above-described embodiments of the present technology will now be described.

[0467] The number and types of driving assistance functions are not limited to the above examples and can be changed as appropriate. For example, a driving assistance function may be used that is executed based on the internal conditions of the vehicle 11. For example, when an occupant of the vehicle 11 performs part of the driving, a driving assistance function may be used that is executed based on the state of the occupant (driver) (for example, looking away, falling asleep, etc.).

[0468] The above-described activation conditions and advance notice conditions of each driving assistance mechanism are merely examples and can be changed as appropriate. Furthermore, when a driving assistance function is executed based on the internal conditions of the vehicle 11 (for example, the state of the occupants), the activation conditions and advance notice conditions may be based on the internal conditions of the vehicle 11.

[0469] The above description shows an example in which the vehicle 11 transmits a display control command to the remote driving controller 12 to instruct the display related to the operation of each driving assistance function. On the other hand, for example, the vehicle 11 may transmit operation-related information (e.g., operation status) related to the operation of each driving assistance function to the remote driving controller 12, and the remote driving controller 12 may control the display related to the operation of each driving assistance function based on the operation-related information.

[0470] For example, when the operation status and control amount of each driving assistance function are presented in the remote driving controller 12, auditory information or tactile information may be used in addition to or instead of the visual information described above. For example, when the remote driving controller 12 notifies the driver of a driving takeover, auditory information or tactile information may be used in addition to or instead of the visual information described above.

[0471] Examples of auditory information include voice messages, notification sounds, warning sounds, etc. output from the speaker 255. Examples of tactile information include vibrations and stresses of the brake pedal 251, the accelerator pedal 252, and the steering wheel 253.

[0472] For example, the remote driving operator is assumed to be a person, but as described above, it may be a device such as a computer. In this case, the input unit 212 and the output unit 215 (interface) of the remote driving controller 12 are configured, for example, by electronic information input and output.

[0473] This technology can be applied to mobile objects, mobile object control devices, and remote driving devices that can be remotely driven and in which a driving assistance system that executes driving assistance functions in emergencies, etc. Examples of such mobile objects include drones and flying vehicles.

[0474] <<3. Others>> <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0475] FIG. 32 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0476] In the computer 1000 , a CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0477] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0478] The input unit 1006 includes input switches, buttons, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0479] In the computer 1000 configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program recorded in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0480] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0481] In the computer 1000, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0482] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0483] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0484] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0485] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0486] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0487] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0488] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0489] (1) A mobile body control device comprising: a communication quality detection unit that detects communication quality between a mobile body and a remote driving device that remotely operates the mobile body; a recognition unit that acquires recognition information regarding the state of the mobile body and a situation around the mobile body; and a driving assistance control unit that controls a driving assistance function of the mobile body based on the recognition information, and controls transmission of operation-related information that is information regarding operation of the driving assistance function to the remote driving device based on the recognition information and the communication quality. (2) The mobile body control device described in (1), wherein, when operation of the driving assistance function is predicted based on the recognition information, the driving assistance control unit transmits to the remote driving device, before the driving assistance function is activated, operation notification information that is the operation-related information for notifying the operation of the driving assistance function, or operation advance notice information that is the operation-related information for advance notice of the operation of the driving assistance function. (3) The mobile body control device described in (2), wherein the driving assistance control unit sets an advance notice condition that advances notice of operation of the driving assistance function based on the recognition information and the communication quality, and transmits the operation notification information or the operation advance notice information to the remote driving device when the advance notice condition is met. (4) The mobile body control device according to (3), wherein the driving assistance control unit transmits the operation advance notice information to the remote driving device when the advance notice condition is met, and transmits the operation notification information to the remote driving device when an operation condition of the driving assistance function is met. (5) The mobile body control device according to (3) or (4), wherein the mobile body is a vehicle, the driving assistance function includes a collision damage mitigation brake function, and the driving assistance control unit sets the advance notice condition based on a distance to the obstacle based on a relative speed of the vehicle with respect to the obstacle and the communication quality. (6) The mobile body control device according to any of (3) to (5), wherein the mobile body is a vehicle, the driving assistance function includes a lane keeping assist function, and the driving assistance control unit sets the advance notice condition based on a distance to the lane boundary line based on movement of the vehicle with respect to the lane in which the vehicle is traveling and the communication quality.(7) The mobile body control device according to any of (2) to (6), wherein, after the driving assistance function is stopped, the driving assistance control unit transmits stop notification information to the remote driving device, which is the operation-related information for notifying the remote driving device of the stop of the driving assistance function. (8) The mobile body control device according to (7), wherein, after the driving assistance function is stopped, if a predetermined time has not elapsed since the driving assistance control unit transmitted the operation notification information or the operation advance notice information to the remote driving device before the driving assistance function is activated, the driving assistance control unit transmits the stop notification information to the remote driving device after the predetermined time has elapsed. (9) The mobile body control device according to any of (2) to (8), wherein the operation notification information includes driving assistance control information indicating the type of the driving assistance function and a control state of the mobile body by the driving assistance function, and the driving assistance control unit repeatedly transmits the operation notification information to the remote driving device while the driving assistance function is activated. (10) The mobile body control device according to any of (1) to (9), wherein the driving assistance control unit stops the driving assistance function when an activation condition for the driving assistance function is not met during execution of the driving assistance function and a difference between a control amount of the moving body by the driving assistance control unit and a control amount of the moving body by the remote driving device becomes less than a predetermined threshold. (11) The mobile body control device according to (10), wherein the driving assistance control unit transmits handover information, which is the operation-related information for notifying the remote driving device of a handover from the driving assistance function to the remote driving, to the remote driving device when an activation condition for the driving assistance function is not met during execution of the driving assistance function and a difference between a control amount of the moving body by the driving assistance control unit and a control amount of the moving body by the remote driving device is equal to or greater than a predetermined threshold. (12) The mobile body control device according to (10) or (11), wherein the moving body is a vehicle, and the control amount of the moving body includes at least one of a control amount of acceleration / deceleration control and a control amount of steering control of the vehicle. (13) The mobile body control device according to any one of (1) to (12), wherein the recognition information further includes information relating to an internal state of the mobile body.(14) The mobile body control device according to any one of (1) to (13), further comprising: a communication unit that communicates with the remote driving device; and a remote driving execution unit that executes remote driving of the mobile body under the control of the remote driving device. (15) A mobile body control method, in which a mobile body control device detects communication quality between a mobile body and a remote driving device that operates the remote driving of the mobile body, acquires recognition information regarding the state of the mobile body and the circumstances around the mobile body, controls a driving assistance function of the mobile body based on the recognition information, and controls transmission of operation-related information that is information regarding the operation of the driving assistance function to the remote driving device based on the recognition information and the communication quality. (16) A remote driving device comprising: a remote driving control unit that remotely controls driving of the mobile body, and a presentation control unit that controls presentation of remote driving information used for remote driving of the mobile body, (17) The remote driving device described in (16), wherein the presentation control unit controls the presentation of driving assistance function operation notice information that notifies the driver of operation of the driving assistance function, based on operation notice information that notifies the driver of operation of the driving assistance function, transmitted from the vehicle. (18) The remote driving device described in (16) or (17), wherein the operation notification information includes driving assistance control information that indicates the type of the driving assistance function and a control state of the vehicle by the driving assistance function, and the driving assistance function operation information includes the type of the driving assistance function and the control state of the vehicle by the driving assistance function. (19) The remote driving device described in any of (16) to (18), wherein the control state of the vehicle by the driving assistance function includes a control amount of the vehicle by the driving assistance function. (20) The remote driving device described in (19), wherein the vehicle is a vehicle, and the control amount of the vehicle includes at least one of a control amount of acceleration / deceleration control and a control amount of steering control of the vehicle.(21) The remote driving device according to (20), wherein the driving assistance function includes at least one of a collision mitigation braking function and a lane keeping assistance function. (22) The remote driving device according to any of (16) to (21), wherein the presentation control unit controls the presentation of driving takeover information notifying the handover from the driving assistance function to remote driving, based on takeover information transmitted from the moving object for notifying the handover from the driving assistance function to remote driving. (23) The remote driving device according to (22), wherein the driving takeover information includes information indicating the control amount of the moving object by each of the driving assistance functions and the control amount of the moving object by the remote driving device. (24) The remote driving device according to (22) or (23), wherein the driving takeover information includes information on a method of handing over from the driving assistance function to remote driving. (25) The remote driving device according to any of (16) to (24), wherein the remote driving information includes an image of the surroundings of the moving object, and wherein the driving assistance function activation information is presented together with the image of the surroundings of the moving object. (26) A remote driving control method in which a remote driving device that operates the remote driving of a mobile body controls the presentation of driving assistance function operation information that notifies that the driving assistance function of the mobile body is in operation, based on operation notification information that is transmitted from the mobile body and that notifies that the driving assistance function of the mobile body is in operation.

[0490] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0491] 1 Remote driving system, 11 Vehicle, 12 Remote driving controller, 21 Base station, 22 Communication network, 101 Vehicle control system, 111 Communication unit, 112 Communication quality detection unit, 114 External sensor, 115 Internal sensor, 116 Recognition unit, 117 Driving / braking device, 118 Steering device, 119 Remote driving execution unit, 120 Driving assistance control unit, 121 Output unit, 131 AEB controller, 132 LKA controller, 211 Communication unit, 212 Input unit, 213 Remote driving control unit, 214 Presentation control unit, 215 Output unit, 251 Brake pedal, 252 Accelerator pedal, 253 Steering wheel, 254 Display device, 254A Status display area, 255 Speaker

Claims

1. A mobile body control device comprising: a communication quality detection unit that detects the communication quality between a mobile body and a remote driving device that operates the remote driving of the mobile body; a recognition unit that acquires recognition information regarding the state of the mobile body and the situation around the mobile body; and a driving assistance control unit that controls the driving assistance function of the mobile body based on the recognition information and controls the transmission of operation-related information, which is information regarding the operation of the driving assistance function, to the remote driving device based on the recognition information and the communication quality.

2. The mobile body control device described in claim 1, wherein when the driving assistance control unit predicts that the driving assistance function will be activated based on the recognition information, it transmits to the remote driving device, before the driving assistance function is activated, activation notification information, which is the activation-related information for notifying the remote driving device of the activation of the driving assistance function, or activation warning information, which is the activation-related information for warning the remote driving device of the activation of the driving assistance function.

3. The mobile control device described in claim 2, wherein the driving assistance control unit sets advance notification conditions to notify the operation of the driving assistance function based on the recognition information and the communication quality, and when the advance notification conditions are met, transmits the operation notification information or the operation advance information to the remote driving device.

4. The mobile body control device described in claim 3, wherein the driving assistance control unit transmits the activation notification information to the remote driving device when the advance notification condition is met, and transmits the activation notification information to the remote driving device when the activation condition of the driving assistance function is met.

5. The mobile body control device described in claim 3, wherein the mobile body is a vehicle, the driving assistance function includes a collision damage mitigation brake function, and the driving assistance control unit sets the advance notification condition based on the distance to the obstacle based on the relative speed of the vehicle to the obstacle and the communication quality.

6. The mobile body control device described in claim 3, wherein the mobile body is a vehicle, the driving assistance function includes a lane keeping assistance function, and the driving assistance control unit sets the advance notification conditions based on the distance from the lane boundary line based on the movement of the vehicle relative to the lane in which it is traveling and the communication quality.

7. The mobile body control device described in claim 2, wherein the driving assistance control unit transmits stop notification information, which is the operation-related information for notifying the remote driving device that the driving assistance function has been stopped, to the remote driving device after the driving assistance function has been stopped.

8. The mobile body control device described in claim 7, wherein, after the driving assistance function is stopped, if a predetermined time has not elapsed since the operation notification information or the operation advance notice information was sent to the remote driving device before the driving assistance function was activated, the driving assistance control unit sends the stop notification information to the remote driving device after the predetermined time has elapsed.

9. A mobile body control device as described in claim 2, wherein the operation notification information includes driving assistance control information indicating the type of driving assistance function and the control status of the mobile body by the driving assistance function, and the driving assistance control unit repeatedly transmits the operation notification information to the remote driving device while the driving assistance function is operating.

10. The mobile body control device described in claim 1, wherein the driving assistance control unit stops the driving assistance function when the activation conditions of the driving assistance function are not met while the driving assistance function is being executed, and when the difference between the control amount of the mobile body by the driving assistance control unit and the control amount of the mobile body by the remote driving device becomes less than a predetermined threshold.

11. A mobile body control device as described in claim 10, wherein when the activation conditions of the driving assistance function become unsatisfied while the driving assistance function is being executed, the driving assistance control unit transmits handover information, which is the operation-related information for notifying the remote driving device of a handover from the driving assistance function to the remote driving, to the remote driving device when the difference between the control amount of the mobile body by the driving assistance control unit and the control amount of the mobile body by the remote driving device is equal to or greater than a predetermined threshold.

12. The mobile body control device according to claim 10, wherein the mobile body is a vehicle, and the control amount of the mobile body includes at least one of a control amount of acceleration / deceleration control and a control amount of steering control of the vehicle.

13. The mobile body control device according to claim 1, wherein the recognition information further includes information regarding the internal state of the mobile body.

14. A mobile body control device according to claim 1, further comprising: a communication unit that communicates with the remote driving device; and a remote driving execution unit that executes remote driving of the mobile body under the control of the remote driving device.

15. A mobile body control method in which a mobile body control device detects communication quality between a mobile body and a remote driving device that remotely controls the mobile body, acquires recognition information regarding the state of the mobile body and the situation around the mobile body, controls a driving assistance function of the mobile body based on the recognition information, and controls the transmission of operation-related information, which is information regarding the operation of the driving assistance function, to the remote driving device based on the recognition information and the communication quality.

16. A remote driving device comprising: a remote driving control unit that remotely controls the driving of a mobile body; and a presentation control unit that controls the presentation of remote driving information used for remotely driving the mobile body, wherein the presentation control unit controls the presentation of driving assistance function operation information that notifies that the driving assistance function is in operation in the mobile body, based on operation notification information sent from the mobile body that notifies that the driving assistance function of the mobile body is in operation.

17. The remote driving device described in claim 16, wherein the presentation control unit controls the presentation of driving assistance function operation notice information that notifies the driver of the operation of the driving assistance function based on operation notice information that notifies the driver of the operation of the driving assistance function, the operation notice information being transmitted from the moving body.

18. The remote driving device described in claim 16, wherein the operation notification information includes driving assistance control information indicating the type of driving assistance function and the control status of the moving body by the driving assistance function, and the driving assistance function operation information includes the type of driving assistance function and the control status of the moving body by the driving assistance function.

19. The remote driving device according to claim 16, wherein the control state of the moving object by the driving assistance function includes a control amount of the moving object by the driving assistance function.

20. The remote driving device according to claim 19, wherein the moving body is a vehicle, and the control amount of the moving body includes at least one of a control amount of acceleration / deceleration control and a control amount of steering control of the vehicle.

21. The remote driving device according to claim 20, wherein the driving assistance function includes at least one of a collision mitigation braking function and a lane keeping assistance function.

22. The remote driving device described in claim 16, wherein the presentation control unit controls the presentation of driving takeover information notifying the handover from the driving assistance function to remote driving based on takeover information transmitted from the mobile body notifying the handover from the driving assistance function to remote driving.

23. The remote driving device according to claim 22, wherein the driving takeover information includes information indicating the amount of control of the moving object by the driving assistance function and the amount of control of the moving object by the remote driving device.

24. The remote driving device according to claim 22, wherein the driving takeover information includes information regarding a method of taking over from the driving assistance function to remote driving.

25. The remote driving device according to claim 16, wherein the remote driving information includes an image of the surroundings of the moving body, and the information that the driving assistance function is operating is presented together with the image of the surroundings of the moving body.

26. A remote driving control method in which a remote driving device that operates the remote driving of a mobile body controls the presentation of driving assistance function operation information that notifies that the driving assistance function of the mobile body is in operation, based on operation notification information sent from the mobile body that notifies that the driving assistance function of the mobile body is in operation.

Images