Information processing method and information processing system

Abstract

The information processing method is an information processing method executed by a computer, detects a first operation of a monitor who monitors a mobile body from a distance where the mobile body cannot be directly monitored (S106), if the first operation is detected, makes an execution condition of automatic driving of the mobile body be relaxed compared with an execution condition in a state where the first operation is not detected, and makes the mobile body automatically drive under the relaxed execution condition (S107).

Description

Technical Field

[0001] This disclosure relates to information processing methods and information processing systems. Background Technology

[0002] In recent years, research has been underway on a remote operating system that allows remote operators to indirectly control autonomous vehicles as needed via wireless communication, such as wireless LAN (Local Area Network) or mobile communication networks. For example, if a vehicle becomes unable to drive autonomously, the remote operator can send control signals to the vehicle to control its movement from a distance.

[0003] For example, Patent Document 1 discloses a remote operating system that displays the predicted movement path of the mobile body on an image reflecting the direction of the mobile body's movement, based on the communication delay time between the mobile body and the remote operating device.

[0004] (Existing technical literature)

[0005] (Patent Documents)

[0006] Patent Document 1: Japanese Patent Application Publication No. 2010-61346

[0007] However, in remote operating systems like those in Patent Document 1, the burden on the remote operator is significant. For example, the field of view in a remote operating screen is narrower than that in a moving body; therefore, the remote operator needs to operate more carefully compared to operating in a moving body. Summary of the Invention

[0008] Therefore, the purpose of this disclosure is to provide an information processing method and an information processing apparatus that can reduce the burden on remote operators.

[0009] One aspect of this disclosure relates to an information processing method executed by a computer, which detects a first operation by a monitor who is monitoring the mobile body from a distance and cannot directly monitor the mobile body in autonomous driving. When the first operation is detected, the execution conditions for autonomous driving of the mobile body are relaxed compared to the execution conditions in a state where the first operation is not detected, so that the mobile body can drive autonomously under the relaxed execution conditions.

[0010] One aspect of the information processing system disclosed herein includes: a detection unit that detects a first operation by a monitor who is monitoring the mobile body remotely from a location where the mobile body cannot be directly monitored; and a mode changing unit that, upon detecting the first operation, relaxes the execution conditions for the mobile body's automatic driving compared to the execution conditions in a state where the first operation is not detected, thereby enabling the mobile body to drive automatically under the relaxed execution conditions.

[0011] According to one embodiment of the information processing method disclosed herein, the burden on remote operators can be reduced. Attached Figure Description

[0012] Figure 1 This is a diagram showing the schematic structure of the vehicle control system involved in the embodiment.

[0013] Figure 2 This is a block diagram illustrating the functional structure of the vehicle control system involved in the implementation method.

[0014] Figure 3 This is a flowchart illustrating the operation of the vehicle involved in the implementation method.

[0015] Figure 4 This is a flowchart illustrating the operation of the remote operation device involved in the embodiment.

[0016] Figure 5 It is shown Figure 4 The flowchart for step S102, which shows the process of notifying the remote operator, is shown.

[0017] Figure 6 This is a diagram showing an example of information about the second vehicle.

[0018] Figure 7 This is a diagram illustrating the switch to a more relaxed driving mode for ODD.

[0019] Figure 8 It is shown Figure 4 The flowchart for the pattern retrieval process shown in step S103 is as follows.

[0020] Figure 9 This is a graph showing an example of ODD data.

[0021] Figure 10 This is a diagram illustrating an example of a constraint condition.

[0022] Figure 11 This is a diagram showing an example of information about the first vehicle.

[0023] Figure 12 This is a diagram illustrating an example of the first remote operation information.

[0024] Figure 13 This is a diagram illustrating an example of information from the first system.

[0025] Figure 14 It is shown Figure 4 The flowchart shows an example of the switching process shown in step S107.

[0026] Figure 15 It is shown Figure 4 The flowchart shows an example of the judgment processing operation shown in step S108.

[0027] Figure 16 It is shown Figure 15 The flowchart for the judgment process shown in step S407.

[0028] Figure 17 This is a diagram illustrating an example of the second remote operation information.

[0029] Figure 18 This is a diagram illustrating an example of information from the second system.

[0030] Figure 19 It is shown Figure 15 The flowchart for the judgment process shown in step S408.

[0031] Figure 20 This is a diagram illustrating an example of an instruction.

[0032] Figure 21 It is shown Figure 4 The flowchart shows another example of the switching process shown in step S107.

[0033] Figure 22 It is shown Figure 4 The flowchart shows another example of the judgment processing work shown in step S108.

[0034] Figure 23 This is a diagram used to illustrate the situation of the first applicable example of relaxing ODD.

[0035] Figure 24A This is a diagram illustrating various conditions in the automatic driving mode of the first applicable example.

[0036] Figure 24B This is a diagram illustrating various conditions in the ODD relaxed driving mode of the first applicable example.

[0037] Figure 25 This is a diagram used to illustrate the situation of the second applicable example of relaxing ODD.

[0038] Figure 26A This is a diagram illustrating various conditions in the automatic driving mode of the second applicable example.

[0039] Figure 26B This is a diagram illustrating various conditions in the ODD relaxed driving mode of the second applicable example.

[0040] Figure 27 This is a diagram used to illustrate the situation of the third applicable example of the relaxed ODD.

[0041] Figure 28A This is a diagram illustrating various conditions in the automatic driving mode of the third applicable example.

[0042] Figure 28B This is a diagram illustrating various conditions in the ODD relaxed driving mode of the third applicable example.

[0043] Figure 29 This is a diagram used to illustrate the situation of the fourth applicable example of the relaxed ODD.

[0044] Figure 30A This is a diagram illustrating various conditions in the automatic driving mode of the fourth applicable example.

[0045] Figure 30B This is a diagram illustrating various conditions in the ODD relaxed driving mode of the fourth applicable example.

[0046] Figure 31 This is a diagram used to illustrate the situation of the fifth applicable example of the relaxed ODD.

[0047] Figure 32A This is a diagram illustrating various conditions in the automatic driving mode of the fifth applicable example.

[0048] Figure 32B This is a diagram illustrating various conditions in the first ODD relaxed driving mode of the fifth applicable example.

[0049] Figure 32C This is a diagram illustrating various conditions in the second ODD relaxed driving mode of the fifth applicable example.

[0050] Figure 33 This is a block diagram illustrating the functional structure of a vehicle control system according to a variation of the implementation. Detailed Implementation

[0051] (The process by which this disclosure was made)

[0052] In autonomous driving systems, an Operational Design Domain (ODD) is defined from a safety perspective, restricting the conditions under which autonomous driving is possible. The higher the level of autonomous driving, the stricter the ODD conditions; at level 4, the driving area and conditions are significantly restricted. Therefore, the higher the level of autonomous driving, the more likely it is that the vehicle will fail to drive autonomously due to deviating from the ODD conditions. Remote operation of the vehicle by a remote operator in such a situation was discussed. However, if the remote operator performs remote operation every time the ODD conditions are deviated from, the burden on the remote operator is heavy. For example, in Patent Document 1, remote operation is improved by using a joystick or similar device; however, even with improvements to the user interface, remote operation is difficult due to delays and limited visibility, resulting in a heavy burden on the remote operator.

[0053] Furthermore, when driving autonomously, it is necessary to ensure a minimum level of driving safety. Minimum driving safety refers to things like the vehicle not colliding with objects.

[0054] Therefore, the inventors of this application carefully studied information processing methods that can reduce the burden on remote operators, and further studied information processing methods that can reduce the burden on remote operators while ensuring minimum security, and created the information processing method described below.

[0055] In addition, ODD is an example of the execution conditions for autonomous driving.

[0056] One aspect of this disclosure relates to an information processing method executed by a computer, which detects a first operation by a monitor who is monitoring the mobile body from a distance and cannot directly monitor the mobile body in autonomous driving. When the first operation is detected, the execution conditions for autonomous driving of the mobile body are relaxed compared to the execution conditions in a state where the first operation is not detected, so that the mobile body can drive autonomously under the relaxed execution conditions.

[0057] Accordingly, for example, if a mobile vehicle becomes unable to drive autonomously, the monitor performs the first operation, enabling the mobile vehicle to drive autonomously under relaxed execution conditions. In other words, even if the mobile vehicle becomes unable to drive autonomously, the monitor can still make the mobile vehicle move without remote operation, depending on the situation. As a result, the number of remote operations by the monitor can be reduced. Therefore, the burden on remote monitors can be alleviated.

[0058] Furthermore, for example, when the first operation is detected, the restrictions on the driving mode of autonomous driving can be further strengthened by relaxing the execution conditions.

[0059] Therefore, during autonomous driving under relaxed execution conditions, the safety of the mobile vehicle can be more easily ensured because the restrictions on the driving mode of the mobile vehicle are strengthened.

[0060] Furthermore, for example, the driving mode may include at least one of the speed, steering angle, and acceleration of the moving body, and according to the relaxation of the execution conditions, at least one of the restrictions on the maximum vehicle speed, maximum steering angle, and maximum acceleration during autonomous driving under the relaxed execution conditions may be strengthened.

[0061] Accordingly, during autonomous driving under relaxed execution conditions, the safety of the vehicle can be more easily ensured because at least one of the limits on the maximum speed, maximum steering angle, and maximum acceleration of the vehicle can be strengthened.

[0062] Furthermore, for example, upon detection of the first operation, the monitoring conditions of the monitor targeting the moving body may be further strengthened by relaxing the execution conditions.

[0063] Accordingly, during autonomous driving under relaxed execution conditions, the safety of the moving body can be more easily ensured because the monitoring conditions of the monitor can be strengthened.

[0064] Furthermore, for example, the monitoring conditions may include at least one of the area surrounding the moving body that needs to be monitored by the monitor, i.e., the monitoring area, and the object that needs to be monitored by the monitor, i.e., the monitoring object. Strengthening the monitoring conditions may include strengthening the monitoring of at least one of the monitoring area and the monitoring object, according to the relaxation of the execution conditions.

[0065] Accordingly, during autonomous driving under relaxed execution conditions, the safety of the moving body can be more easily ensured because it is possible to strengthen the restrictions on at least one of the areas monitored by the monitor and the objects monitored.

[0066] Furthermore, for example, the monitoring conditions may include the operation of the mobile body that needs to be monitored by the monitor, and the enhancement of the monitoring conditions may include, in accordance with the relaxation of the execution conditions, the additional monitoring of the mobile body's driving plan or driving control information.

[0067] Therefore, during autonomous driving under relaxed execution conditions, the safety of the vehicle can be more easily ensured because the monitor can carefully observe the behavior of the vehicle.

[0068] Furthermore, for example, the enhanced monitoring conditions may be prompted to the monitor.

[0069] Accordingly, the monitor can identify the enhanced surveillance conditions. Therefore, by enabling the monitor to conduct surveillance based on the surveillance conditions, security can be ensured. Furthermore, for example, if the enhanced surveillance conditions are indicated before the first operation, the monitor can determine whether to proceed with the first operation after confirming the enhanced surveillance conditions.

[0070] Furthermore, for example, the mobile body may be driven autonomously under relaxed execution conditions only during periods when it is being monitored by the remote operator under enhanced monitoring conditions.

[0071] Therefore, during the period of autonomous driving under relaxed execution conditions, the monitor is responsible for oversight, making it easier to ensure the safety of the mobile vehicle during this period. Furthermore, since the monitor is monitoring the mobile vehicle, the remote operation mode for switching to monitor-operated driving can be smoothly determined based on the mobile vehicle's driving status under the relaxed execution conditions.

[0072] Furthermore, for example, the period during which the monitoring is performed under the enhanced monitoring conditions is the period during which a second operation is detected, the second operation being an operation indicating the continuation of the autonomous driving under the second condition, and being an operation performed by the remote operator.

[0073] Accordingly, the mobile body can be automatically driven under relaxed execution conditions only during the period when the monitor performs the second operation. In other words, the mobile body can be automatically driven under relaxed execution conditions only during the period when the monitor determines that automatic driving under relaxed execution conditions should continue. Therefore, unnecessary automatic driving under relaxed execution conditions can be suppressed, thus, for example, reducing the monitoring burden on the monitor caused by unnecessary monitoring. Therefore, the burden on remote monitors can be further reduced. Furthermore, the safety of automatic driving under relaxed execution conditions may be lower than that of normal automatic driving; therefore, by performing it only during the period when the monitor performs the second operation, safety can be ensured.

[0074] Furthermore, for example, upon detection of the first operation, monitoring conditions could be strengthened to address delays in processing or communication related to the monitor's surveillance, in accordance with the relaxation of the execution conditions.

[0075] Accordingly, during autonomous driving under relaxed execution conditions, it is possible to strengthen monitoring conditions related to delays in processing or communication with the monitor, thereby making it easier to ensure the safety of the moving body during this period.

[0076] Furthermore, one aspect of the information processing system disclosed herein includes: a detection unit that detects a first operation by a monitor who is monitoring the mobile body from a distance and cannot directly monitor the mobile body; and a mode change unit that, upon detecting the first operation, relaxes the execution conditions for the mobile body's automatic driving compared to the execution conditions in a state where the first operation is not detected, thereby enabling the mobile body to drive automatically under the relaxed execution conditions.

[0077] Accordingly, the same effect as the aforementioned information processing system can be achieved.

[0078] Furthermore, these general or specific forms can be realized by non-transitory recording media such as systems, devices, methods, integrated circuits, computer programs, or computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0079] The following description, with reference to the accompanying drawings, provides specific examples of an information processing method and system according to one aspect of this disclosure. Each embodiment shown herein is merely an example of this disclosure. Therefore, the numerical values, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Furthermore, constituent elements not described in the embodiments below are described as arbitrary constituent elements. Moreover, the various elements can be combined in all embodiments.

[0080] Furthermore, the figures are illustrative and not rigorous diagrams. Therefore, for example, the scales in the figures may not be consistent. Also, in the figures, substantially identical components are assigned the same number, and sometimes repeated descriptions are omitted or simplified.

[0081] Furthermore, in this specification, numerical values ​​and their ranges do not merely represent a strict meaning, but rather imply a range that includes substantially the same quantity, such as a difference of approximately a few percentage points.

[0082] (Implementation Method)

[0083] Hereinafter, the information processing methods, etc., involved in this embodiment will be referred to... Figures 1 to 32C Please provide an explanation.

[0084] [1. Structure of Vehicle Control System]

[0085] First, regarding the structure of the vehicle control system 10 involved in this embodiment, refer to... Figure 1 as well as Figure 2 Please provide an explanation. Figure 1 This is a diagram showing a schematic structure of the vehicle control system 10 according to this embodiment.

[0086] like Figure 1 As shown, the vehicle control system 10 includes a remote operating system 100, a network 300, a wireless base station 310, and a vehicle 200. The vehicle control system 10 is a system that connects the vehicle 200 to the remote operating system 100 (specifically, the remote operating device 130) in a communicative manner via the wireless base station 310 and the network 300, such as a wireless LAN or communication terminal. The wireless base station 310 and the network 300 are examples of a communication network. Furthermore, the vehicle 200 is an example of a moving body that is at least remotely monitored by a remote operator H. The vehicle 200 can also be a vehicle that is remotely monitored and remotely operated by the remote operator H. Additionally, the vehicle control system 10 is an example of an information processing system. Furthermore, the remote operator H is an example of a monitor who monitors the vehicle 200 from a distance, where direct monitoring of the vehicle 200 is not possible. "Not directly monitorable" means that the vehicle 200 cannot be observed with the naked eye. In other words, the remote operator H remotely monitors and remotely operates the vehicle 200 from a location different from the vehicle's surroundings.

[0087] The remote operating system 100 is a system for a remote operator H to remotely monitor and operate the vehicle 200 as needed. The remote operating system 100 includes a display device 110, an operation input device 120, and a remote operation device 130.

[0088] Display device 110 provides various information to remote operator H. Display device 110 is a monitor connected to remote operating device 130 and displays information about vehicle 200. Display device 110 displays information used by remote operator H to remotely monitor or operate vehicle 200. Display device 110 may, for example, display images captured by a camera unit provided with vehicle 200. Furthermore, display device 110 displays information processed by remote operating device 130. Display device 110 may also display buttons (images) used by remote operator H to switch driving modes or maintain driving modes of vehicle 200. Display device 110 may, for example, display information such as the monitoring area that needs to be monitored by remote operator H. The information displayed by display device 110 will be described in detail later.

[0089] The display device 110 is implemented, for example, by a liquid crystal display or the like.

[0090] The operation input device 120 accepts various operations from the remote operator H. When driving in an ODD-relaxed driving mode (where the ODD is relaxed compared to the automatic driving mode), the operation input device 120 accepts driving permission commands from the remote operator H. Driving permission commands are commands that allow the vehicle 200 to continue driving in the ODD-relaxed driving mode. The operation input device 120 may be implemented, for example, by a touchscreen, but it may also be implemented by a hardware keyboard (hardware button) or a slide switch. Furthermore, the operation input device 120 can also accept various inputs based on information such as the remote operator H's voice, gestures, and gaze. The operation input device 120 can also accept input via swiping, destination input, etc.

[0091] Furthermore, the operation input device 120 accepts remote operation input from the remote operator H when driving in remote operation mode. The operation input device 120 can be, for example, a steering wheel, pedals (such as the accelerator pedal and the brake pedal), but it can also be implemented by a joystick or the like.

[0092] Furthermore, the ODD relaxed driving mode and remote operation mode will be explained later, but the vehicle 200 has the feature of being able to drive in the ODD relaxed driving mode.

[0093] When vehicle 200 becomes unable to automate its driving, remote control device 130 performs processing to switch the driving mode of vehicle 200. Even when vehicle 200 is unable to automate its driving, remote control device 130 reduces the burden on remote operator H by decreasing the frequency of remote operation. Specifically, remote control device 130 switches the driving mode to an ODD relaxed driving mode, an intermediate level between Level 4 of fully automated driving (where vehicle 200 is in fully automated driving) and Level 1 of automated driving (where remote operator H is in remote operation), allowing vehicle 200 to continue driving automatically in the switched ODD relaxed driving mode. Furthermore, remote control device 130 can also allow remote operator H to perform remote operation when switching to ODD relaxed driving mode is not possible.

[0094] Thus, the remote operation device 130 not only has the existing driving modes, namely the automatic driving mode and the remote operation mode, but also an ODD relaxed driving mode. The ODD relaxed driving mode is a mode in which the conditions of the ODD are relaxed to allow the vehicle 200 to drive automatically. In other words, the ODD relaxed driving mode is a mode in which the remote operator H does not remotely operate the vehicle 200. During driving in the ODD relaxed driving mode, the remote operator H, for example, remotely monitors the vehicle 200 based on images obtained from the vehicle 200. In other words, the vehicle 200 continues to drive automatically under the remote monitoring of the remote operator H, according to the relaxed ODD conditions. Accordingly, the frequency of reaching Level 1 of autonomous driving, which requires remote operation by the remote operator H, can be reduced, thus alleviating the burden on the remote operator H. Furthermore, as described above, the remote operation device 130 can also function as a remote monitoring device.

[0095] Furthermore, in this embodiment, the remote operating device 130 also imposes constraints on the vehicle 200 while driving in the ODD relaxed driving mode. In other words, the remote operating device 130 makes the constraints in the ODD relaxed driving mode more stringent than those in the automatic driving mode. These constraints are the basis for determining whether the vehicle 200 can continue driving in the ODD relaxed driving mode. The remote operating device 130 can make the vehicle 200 drive more safely by imposing constraints. Accordingly, the remote operating device 130, for example, through remote monitoring by a remote operator H and the imposition of constraints, can safely drive automatically on paths that are not traversable at fully automated driving level 4.

[0096] In addition, the level of autonomous driving in this specification is defined as follows.

[0097] Level 1 of automated driving is a level that automatically performs any one of the following operations: accelerator (acceleration), steering (steering angle), and braking (control). Level 2 of automated driving is a level that automatically performs multiple operations of the accelerator, steering, and braking. Level 3 of automated driving is a level that automatically performs all operations of the accelerator, steering, and braking, with the driver only responding when necessary. Level 4 of automated driving is a level that automatically performs all operations of the accelerator, steering, and braking, without driver intervention. Furthermore, Level 4 of automated driving may be a level that does not require remote monitoring by a remote operator (H). Level 3 of automated driving may be a level where the driver has a monitoring obligation, and Level 4 of automated driving may be a level where the driver does not have a monitoring obligation. Furthermore, Levels 3 and 4 of automated driving are levels that can reach the destination without driver intervention. Additionally, the number of automated driving levels is not limited to the above four levels; for example, it can be defined as five levels.

[0098] Vehicle 200 is an example of a mobile body that includes a driver and other passengers, and can be remotely monitored or operated by a remote operator H as needed. Vehicle 200 is an autonomous vehicle. Vehicle 200 can also be an autonomous vehicle capable of switching between autonomous and manual driving. Furthermore, regarding autonomous vehicles, there are no particular restrictions as long as a person can ride in it and it can operate autonomously; it can also be an autonomous bus, an autonomous taxi, an autonomous private car, an autonomous truck, an autonomous construction vehicle (e.g., a dump truck), etc.

[0099] For the structure of the remote operating device 130 and the vehicle 200, refer to Figure 2 Please provide an explanation. Figure 2 This is a block diagram illustrating the functional structure of the vehicle control system 10 according to this embodiment. Additionally, in Figure 2 The diagram shows only the remote operation device 130 and the vehicle 200, which are among the various components of the vehicle control system 10.

[0100] like Figure 2 As shown, the remote operation device 130 includes a communication unit 140, a remote operation unit 150, and a security judgment unit 160.

[0101] The communication unit 140 communicates with the vehicle 200 via the network 300 and the wireless base station 310. For example, the communication unit 140 obtains vehicle information from the vehicle 200 and outputs information for changing the driving mode to the vehicle 200. The communication unit 140 is implemented, for example, by a communication circuit (communication module).

[0102] The remote operation unit 150 performs processing related to switching the driving mode of the vehicle 200. The remote operation unit 150 includes a mode change unit 151, a prompt unit 152, a command receiving unit 153, an operation receiving unit 154, and an operation information acquisition unit 155.

[0103] The mode change unit 151 switches the driving mode of the vehicle 200. The mode change unit 151 outputs information for switching the driving mode of the vehicle 200 to the vehicle 200 via the communication unit 140. The mode change unit 151 switches the driving mode of the vehicle 200 from the current driving mode to any one of the following driving modes: automatic driving mode (e.g., fully automatic driving mode), ODD relaxed driving mode, and remote operation mode. If the vehicle is in automatic driving mode but cannot drive, the mode change unit 151 switches to either the ODD relaxed driving mode or the remote operation mode. Furthermore, the mode change unit 151 performs the driving mode switch, for example, by receiving an operation from a remote operator H indicating permission to switch driving modes.

[0104] The mode change unit 151 changes the conditions of the vehicle 200's ODD according to the switched driving mode.

[0105] Furthermore, the mode-changing unit 151 can also perform processing to ensure the safety of the vehicle 200 when switching the driving mode of the vehicle 200 to the ODD relaxed driving mode, along with the change of ODD. For example, the mode-changing unit 151 can also change the settings related to the driving capability determination of the driving capability determination unit 231 of the vehicle 200 based on the constraints of the relaxed capability determination unit 162 or the remote operation capability determination unit 163. The settings related to the driving capability determination include, for example, at least one of the following: maximum vehicle speed, maximum steering angle, maximum acceleration, etc. The mode-changing unit 151 can also update (rewrite) the settings related to the driving capability determination based on the constraints of the ODD relaxed driving mode when switching from the automatic driving mode to the ODD relaxed driving mode. For example, the mode-changing unit 151 can also reduce at least one of the maximum vehicle speed, maximum steering angle, maximum acceleration, etc., used by the driving capability determination unit 231 to determine the driving capability. Alternatively, the mode-changing unit 151 may be described as imposing stricter restrictions on at least one of the following parameters—maximum speed, maximum steering angle, and maximum acceleration—when switching from autonomous driving mode to ODD relaxed driving mode, compared to the autonomous driving mode. For example, whenever the driving mode of the vehicle 200 is switched, the mode-changing unit 151 changes the settings related to the driving feasibility determination of the driving feasibility determination unit 231.

[0106] Thus, the mode change unit 151 can also relax the ODD, but change the driving mode of the vehicle 200 to the ODD relaxed driving mode where the constraints are limited. Accordingly, the remote operation device 130 can ensure minimum safety while continuing to drive automatically in the ODD relaxed driving mode.

[0107] Furthermore, the mode change unit 151 outputs, for example, changes to the ODD and constraints to the area determination unit 161 and the driving continuation determination unit 164. Additionally, the mode change unit 151 may, for example, enable the operation acceptance unit 154 to accept the operation when switching to the ODD relaxed driving mode.

[0108] Furthermore, the speed, steering angle, and acceleration of vehicle 200 are examples of autonomous driving modes. A driving mode must include at least one of the following: vehicle 200's speed, steering angle, and acceleration.

[0109] The prompting unit 152 causes the display device 110 to provide a prescribed prompt. For example, the prompting unit 152 generates information to prompt the remote operator H to remotely monitor or operate the vehicle 200. Furthermore, the prompting unit 152 can overlay prescribed information (e.g., auxiliary information described later) with images captured by a camera unit equipped on the vehicle 200 to prompt the display device 110. The prompting unit 152 can also cause the display device 110 to display information processed by the remote operation device 130. Additionally, the prompting unit 152 can also cause the display device 110 to display buttons, etc., for the remote operator H to switch the driving mode of the vehicle 200 or maintain the driving mode.

[0110] When driving in ODD relaxed driving mode, the command receiving unit 153 receives information (e.g., commands) corresponding to the operations input by the remote operator H to the operation input device 120 (e.g., a touchscreen). The command receiving unit 153 is an example of a detection unit.

[0111] In remote operation mode, the operation receiving unit 154 receives information from the operation input device 120 corresponding to the operation input device 120 (e.g., steering wheel, pedal, etc.) input by the remote operator H.

[0112] The operation information acquisition unit 155 acquires information about the remote operating system 100, information about the remote operator H, etc. The information about the remote operator H includes, for example, the remote operator's proficiency and permissions during remote operation. The operation information acquisition unit 155 may read the information about the remote operator H from the storage unit 167. Furthermore, the information about the remote operator H may also include monitoring results of the current state of the remote operator H. The current state is information that allows determination of whether the remote operator H is in a state suitable for necessary monitoring, and includes at least one of, for example, the presence or absence of drowsiness, concentration level, or fatigue level. The operation information acquisition unit 155 may also, for example, acquire a captured image of the remote operator H and perform image analysis on that image to obtain the monitoring results of the remote operator H's current state.

[0113] The safety determination unit 160 performs processing related to the driving safety of the vehicle 200 in the vehicle control system 10. For example, the safety determination unit 160 performs processing to ensure safety when driving in ODD relaxed driving mode. For example, while the vehicle 200 is driving in ODD relaxed driving mode, the safety determination unit 160 monitors each of the vehicle 200, the remote operator H, and the remote operating system 100, and performs processing to stop the vehicle 200 due to reduced safety if predetermined conditions (e.g., constraints) are not met.

[0114] The safety determination unit 160 includes a region determination unit 161, a relaxation permission determination unit 162, a remote operation permission determination unit 163, a driving continuation permission determination unit 164, a reversing unit 165, a system information acquisition unit 166, and a storage unit 167.

[0115] The area determination unit 161 calculates the area that should be monitored by the remote operator H when driving in ODD relaxed driving mode or remote operation mode, based on the required monitoring area information from the mode change unit 151 and the vehicle information obtained from the vehicle 200.

[0116] The relaxation permission determination unit 162 determines whether switching to the ODD relaxed driving mode is possible when changing driving modes. The relaxation permission determination unit 162, for example, calculates the constraints for driving in the ODD relaxed driving mode and makes the determination based on whether the constraints are met in the current situation. The constraints include at least one of the following: maximum vehicle speed, maximum steering angle, communication delay, and necessary monitoring area. Furthermore, the communication delay is the communication delay between the vehicle 200 and the remote control device 130. The necessary monitoring area is the area that needs to be monitored by the remote operator H for the vehicle 200 to drive autonomously, and is the area surrounding the vehicle 200. The monitoring area monitored by the remote operator H is an example of the monitor's monitoring conditions. Additionally, the monitoring conditions may also include an object that needs to be monitored by the remote operator H, i.e., an object that needs to be remotely monitored. An object that needs to be remotely monitored is an example of an object that needs to be monitored. Furthermore, the current situation includes at least one of the following: the situation of the vehicle 200, the situation of the remote operator H, and the situation of the system.

[0117] The remote operation feasibility determination unit 163 determines whether it can switch to remote operation mode when changing driving modes. The remote operation feasibility determination unit 163 calculates, for example, the constraints for driving in remote operation mode and makes the determination based on whether the constraints are met in the current situation.

[0118] The driving continuation feasibility determination unit 164 determines whether the constraints in the current driving mode are met while the vehicle 200 is in motion. If the current driving mode is an ODD relaxed driving mode, the driving continuation feasibility determination unit 164 determines whether the constraints defined by the relaxation feasibility determination unit 162 are met while the vehicle 200 is in motion. Furthermore, if the current driving mode is a remote operation mode, the driving continuation feasibility determination unit 164 determines whether the constraints defined by the remote operation feasibility determination unit 163 are met while the vehicle 200 is in motion.

[0119] The reversing unit 165 controls the movement of the vehicle 200 based on the determination result of the driving continuation feasibility determination unit 164. If the driving continuation feasibility determination unit 164 determines that the constraints in the driving mode are not met, the reversing unit 165 reverses the movement of the vehicle 200. Reversing can be, for example, stopping the vehicle, restricting the vehicle's movement, or forcibly switching the driving mode. Alternatively, if the driving continuation feasibility determination unit 164 determines that the constraints in the driving mode are not met, the reversing unit 165 may perform one of the following actions: stopping the vehicle 200, further restricting the vehicle 200's movement, or forcibly switching the driving mode.

[0120] System information acquisition unit 166 acquires various information about vehicle control system 10. System information acquisition unit 166 acquires specification information of remote operating system 100 (e.g., described later). Figure 13 The system obtains information on the current health status of the vehicle control system 10, including network status such as communication delay and communication band, as well as information on the computing resources and program operation status of the remote control device 130 (e.g., as described later). Figure 18 (The second system information shown). The system information acquisition unit 166 can also store the acquired information in the storage unit 167.

[0121] Storage unit 167 stores data related to the vehicle 200 for each of the automatic driving mode, relaxed driving mode, and remote operation mode, as well as data related to remote operation (here, ODD, constraints, etc., for example, as described later). Figure 9 as well as Figure 10 Storage unit 167, for example, stores sets of ODDs and constraints corresponding to multiple relaxed driving modes. Alternatively, storage unit 167 can be described as storing sets of conditions enabling autonomous driving and the constraints under those conditions. Storage unit 167 may be implemented, for example, with a semiconductor memory, but is not limited thereto.

[0122] The vehicle 200 includes a communication unit 210, a vehicle information acquisition unit 220, an automatic driving system 230, a vehicle control unit 240, and a command acquisition unit 250.

[0123] The communication unit 210 communicates with the remote operating device 130 via the network 300 and the wireless base station 310. For example, the communication unit 210 outputs vehicle information to the remote operating device 130 and receives information from the remote operating device 130 for changing the driving mode. The communication unit 210 is implemented, for example, by a communication circuit (communication module).

[0124] The vehicle information acquisition unit 220 acquires vehicle information of the vehicle 200. The vehicle information includes at least one of the following: vehicle 200 specifications, current speed, steering angle, acceleration, autonomous driving path information, ODD relaxed driving mode path information, and sensor information. The specifications include at least one of the following: vehicle 200 size, wheelbase, maximum steering angle, maximum speed, maximum acceleration, maximum deceleration, and obstacle detection performance. The sensor information includes at least one of the following: information related to the vehicle 200's current position, surrounding objects, and information indicating the state of the vehicle interior.

[0125] The vehicle information acquisition unit 220 can also acquire vehicle information from external devices (e.g., a server device that manages the specification information of the vehicle 200) or from sensing information from various sensors.

[0126] The autonomous driving system 230 performs processing for autonomous driving of the vehicle 200 according to the driving mode of the vehicle 200. The autonomous driving system 230 includes a driving feasibility determination unit 231 and an autonomous driving unit 232.

[0127] The driving feasibility determination unit 231 performs judgments on whether driving is permitted in automatic driving mode and whether driving is permitted in ODD relaxed driving mode. The driving feasibility determination unit 231 determines whether driving is permitted in automatic driving mode and whether driving is permitted in ODD relaxed driving mode based on the set ODD and vehicle information. The driving feasibility determination unit 231 repeatedly performs these judgments at predetermined time intervals during driving in automatic driving mode and ODD relaxed driving mode.

[0128] The autonomous driving unit 232 generates an autonomous driving plan based on vehicle information. The driving plan includes the driving route, speed, etc. Furthermore, if the driving continuation feasibility determination unit 164 determines that driving can continue, the autonomous driving unit 232 outputs control information based on the driving plan to the vehicle control unit 240 in order to execute the generated driving plan. The control information based on the driving plan is an example of driving control information.

[0129] The vehicle control unit 240 controls the driving of the vehicle 200 according to the autonomous driving driving plan or remote operation control commands.

[0130] The command acquisition unit 250 receives various commands from the remote operating device 130 via the communication unit 210, generates commands corresponding to the received commands for controlling the vehicle 200, and outputs them to the vehicle control unit 240. When the vehicle 200 is traveling in ODD relaxed driving mode, the command acquisition unit 250 receives information from the remote operating device 130 indicating permission to continue traveling in ODD relaxed driving mode (driving continuation command). The command acquisition unit 250 receives this information, for example, periodically. If the command acquisition unit 250 receives this information, it outputs a command indicating continued ODD relaxed driving mode to the vehicle control unit 240. Alternatively, the command acquisition unit 250 may stop the vehicle 200 from traveling if it does not receive a driving continuation command within a specified period, if the delay of the driving continuation command is large (e.g., large communication network delay), or if it receives information indicating the end of ODD relaxed driving mode.

[0131] Furthermore, when the vehicle 200 is driving in remote operation mode, the instruction acquisition unit 250 acquires remote operation instructions generated by the remote operator H operating the operation receiving unit 154, generates driving control instructions for the vehicle 200 based on the acquired remote operation instructions, and outputs the generated instructions to the vehicle control unit 240.

[0132] [2. How the vehicle control system works]

[0133] Next, regarding the operation of the vehicle control system 10 described above, refer to... Figures 3 to 22 Please provide an explanation. Figure 3 This is a flowchart illustrating the operation of the vehicle 200 according to this embodiment. Figure 3 The image shows vehicle 200 operating in autonomous driving mode.

[0134] like Figure 3 As shown, the driving capability determination unit 231 determines whether automatic driving can continue (S11). The driving capability determination unit 231 determines, for example, whether driving in the automatic driving mode can continue based on the ODD corresponding to the automatic driving mode and vehicle information.

[0135] If the driving feasibility determination unit 231 can continue driving automatically (S11 "Yes"), it continues driving automatically (S12). Specifically, the driving feasibility determination unit 231 outputs information indicating that it will continue driving automatically to the automatic driving unit 232.

[0136] The travel feasibility determination unit 231 determines whether the destination has been reached (S13). For example, the travel feasibility determination unit 231 can also determine whether the destination has been reached based on the current position of the vehicle 200. The travel feasibility determination unit 231 can also obtain the current position of the vehicle 200 from the outside via the communication unit 210, or it can obtain the current position of the vehicle 200 based on GPS (Global Positioning System) signals (i.e., radio waves transmitted from satellites). The travel feasibility determination unit 231 is configured to include a GPS module that measures the current position of the vehicle 200 based on GPS signals to obtain the current position. In addition, the destination is obtained in advance.

[0137] If the destination is reached (S13 "Yes"), the driving capability determination unit 231 terminates the driving of vehicle 200. If the destination is not reached (S13 "No"), the driving capability determination unit 231 returns to step S11 and continues driving.

[0138] If the vehicle cannot continue to drive automatically (S11 "No"), the driving ability determination unit 231 stops the vehicle 200 from driving (S14). Then, the driving ability determination unit 231 outputs information indicating that the vehicle cannot drive (cannot drive automatically) to the remote operation device 130 via the communication unit 210 (S15).

[0139] Next, if the remote operating device 130 receives information from the vehicle 200 indicating that it cannot drive, it performs a process to switch driving modes. The process of switching driving modes by the remote operating device 130 will be explained later. If the remote operating device 130 determines the new driving mode, it outputs information indicating the new driving mode to the vehicle 200. Hereinafter, the new driving mode will be explained using the ODD (Optical Discharge) relaxed driving mode as an example.

[0140] Next, the autonomous driving unit 232, upon receiving a request to switch to the ODD relaxed driving mode ("Yes" in S16), switches the driving mode to the ODD relaxed driving mode and begins driving (S17). Specifically, the autonomous driving unit 232 generates a driving plan for the ODD relaxed driving mode and outputs instructions for executing the generated driving plan to the vehicle control unit 240.

[0141] Next, the instruction acquisition unit 250 determines whether a driving stop instruction has been received from the remote operation device 130 (S18). In this embodiment, the instruction acquisition unit 250 periodically receives instructions (hereinafter also referred to as driving continue instructions) allowing continued driving in ODD relaxed driving mode during driving, for example. The instruction acquisition unit 250 may also determine "yes" in step S18 if, for example, a driving continue instruction is not received within a specified period, a driving continue instruction is delayed significantly, or information indicating the end of ODD relaxed driving mode has been received. An example of a driving stop instruction being received within a specified period and information indicating the end of ODD relaxed driving mode being received is a situation where a driving continue instruction is not received within a specified period.

[0142] If the command acquisition unit 250 receives a driving stop instruction from the remote operating device 130 (S18 "Yes"), the vehicle control unit 240 returns to step S14 and continues subsequent processing. If the command acquisition unit 250 does not receive a driving stop instruction from the remote operating device 130 (S18 "No"), the vehicle control unit 240 continues driving in the ODD relaxed driving mode (S19).

[0143] Next, the travel feasibility determination unit 231 determines whether the destination has been reached (S20). The determination process in step S20 is the same as in step S13, therefore, the explanation is omitted. In addition, if the result in step S20 is "no", the process returns to step S18 to continue.

[0144] Next, the operation of the remote operation device 130 will be explained. Figure 4 This is a flowchart illustrating the operation of the remote operation device 130 according to this embodiment.

[0145] like Figure 4 As shown, if the notification unit 152 receives information from the vehicle 200 via the communication unit 140 indicating that the vehicle cannot drive (cannot drive automatically) (S101 "Yes"), it notifies the remote operator H (S102). The notification unit 152 notifies the remote operator H, for example, by having the display device 110 display the information obtained in step S101 (in this embodiment, "notification" means "display"). Furthermore, the manner in which the notification unit 152 displays this information to the remote operator H is not particularly limited.

[0146] Here, the processing of notifications to remote operator H is referred to... Figure 5 Please provide an explanation. Figure 5 It is shown Figure 4 The flowchart for step S102, which shows the process of notifying the remote operator H, is as follows.

[0147] like Figure 5 As shown, firstly, the prompting unit 152 obtains ODD detachment information (S201). For example, the prompting unit 152 may also obtain ODD detachment information by obtaining second vehicle information including ODD detachment information from vehicle 200. Figure 6 This is a diagram illustrating an example of information about a second vehicle. The remote operating device 130 sequentially obtains information about the second vehicle from vehicle 200.

[0148] like Figure 6 As shown, the second vehicle information indicates the current status of vehicle 200 and is output from vehicle 200 to remote control device 130 at predetermined time intervals. The second vehicle information includes the current vehicle speed, current steering angle, driving mode, driving status, and ODD disengagement information. The second vehicle information may also include the current acceleration. The driving mode is automatic driving mode, but the driving status is stationary. Furthermore, the ODD disengagement information indicates that it is necessary to drive in the oncoming lane. That is, vehicle 200 is in a situation where it needs to drive in the oncoming lane due to obstacles ahead, but if it does so, it will disengage from the ODD and cannot continue automatic driving.

[0149] Alternatively, the prompting unit 152 may also obtain ODD detachment information in addition to the information of the second vehicle.

[0150] Figure 7 This is a diagram illustrating the switch to a more relaxed driving mode for ODD. Figure 7 (a) shows the state of vehicle 200 stopped due to detachment from the ODD. Additionally, arrows extending from vehicle 200 indicate the travel path. Figure 7 The image shown is an overhead view; however, the image displayed by the prompting unit 152 is not limited to an overhead view. When the vehicle 200 is equipped with a surround view system, the prompting unit 152 can cause the display device 110 to display an overhead view.

[0151] Figure 7 (a) shows a prompt image displayed by the display device 110, indicating that vehicle 200 is stopped because vehicle 400 is stopped in front of it on road L. The image also indicates the current driving mode of vehicle 200 and the result of a judgment on whether automatic driving can continue. The driving mode is automatic driving, and the result of the judgment on whether automatic driving can continue is STOP (automatic driving is not allowed). Vehicle 400 is blocking road L, and vehicle 200's ODD includes being able to travel only in its own lane and in the opposite lane. Therefore, vehicle 200 cannot travel in the opposite lane to avoid vehicle 400. Furthermore, Figure 7The status shown in (a) can also be displayed to the remote operator H by the display device 110.

[0152] Refer again Figure 5 The prompting unit 152 then issues an ODD detachment alarm (S202). In other words, the prompting unit 152 causes the display device 110 to display information indicating that the vehicle 200 has detached from the ODD. Next, the prompting unit 152 displays the detached ODD information (S203). Based on steps S202 and S203, the remote operator H is aware of the ODD detachment alarm and the detached ODD information, thus enabling successful remote monitoring or operation of the vehicle 200 by the remote operator H.

[0153] Figure 7 (b) shows the image displayed by the prompting unit 15, indicating the ODD disengagement alarm and the disengaged ODD information. This image includes the reason why vehicle 400 is stopped (i.e., vehicle 200 is stopped), the disengaged ODD information, the current driving mode, the result of the judgment on whether autonomous driving can continue, and a switch button for changing the driving mode. Thus, by indicating to the remote operator H the reason why vehicle 200 cannot drive autonomously, the remote operator H can appropriately identify the condition of vehicle 200. Furthermore, the ODD disengagement alarm displays the reason for stopping, the disengaged ODD information, and the result of the judgment on whether autonomous driving can continue.

[0154] The switching buttons include, for example, a button 112 indicating "Opposite Lane Driving Mode" and a button 113 indicating "Remote Operation" (a so-called software button). Here, "Opposite Lane Driving Mode" is an example of a relaxed ODD driving mode, a driving mode that relaxes the ODD and strengthens restrictions such as limitations. Furthermore, "Remote Operation" means a remote operation mode. Additionally, if the switching buttons are hardware buttons, the display device 110 may not display a prompt indicating the switching buttons.

[0155] Remote operator H, just need to watch Figure 7 The image shown in (b) makes it easy to identify when vehicle 200 is stopped.

[0156] Refer again Figure 4 Next, the mode change unit 151 performs mode retrieval processing (S103). In other words, the mode change unit 151 determines whether there is a switchable driving mode when driving in autonomous driving mode is not possible. The mode change unit 151 determines whether there is a switchable ODD relaxed driving mode or remote operation mode.

[0157] Here, for pattern retrieval processing, refer to Figures 8 to 13 Please provide an explanation. Figure 8 It is shown Figure 4The flowchart for the pattern retrieval process shown in step S103 is as follows.

[0158] The mode change unit 151 obtains ODD data and constraints (S301). The mode change unit 151 also obtains ODD data and constraints for the relaxed driving mode from the storage unit 167. Figure 7 In the example, the mode change unit 151 obtains the ODD data and constraints for the oncoming lane driving mode. The oncoming lane driving mode is, for example, a driving mode that includes the oncoming lane in the driving possibility area of ​​the vehicle 200's ODD. For example, if multiple ODD relaxed driving modes are stored in the storage unit 167, the mode change unit 151 can also extract one or more ODD relaxed driving modes that include the oncoming lane in the driving possibility area of ​​the ODD (an example of a switchable ODD relaxed driving mode).

[0159] Here is an example of ODD data and constraints, see [reference]. Figure 9 as well as Figure 10 Please provide an explanation. Figure 9 This is a graph showing an example of ODD data. Figure 9 This displays the ODD data in the opposite lane driving mode (ODD relaxed driving mode). Figure 10 This is a diagram illustrating an example of a constraint condition. Figure 10 The constraints shown are related to Figure 9 The constraints corresponding to the ODD data shown.

[0160] Figure 9 (a) shows the possible driving area. In the opposite lane driving mode, the possible driving area is the current lane and the opposite lane. Furthermore, Figure 9 (b) shows the objects being overtaken. In the oncoming lane driving mode, the objects being overtaken are vehicles (parked on the side of the road), colored traffic cones (registered trademark), and vehicles (vehicles ahead). Additionally, in the ODD data of the automatic driving mode, for example, the driving area is only the driver's own lane, and the objects being overtaken are only vehicles (parked on the side of the road) and colored traffic cones (registered trademark). In other words, Figure 9 In the ODD data shown, the driving possibility area in the ODD data of the automatic driving mode has the opposite lane added, and the overtaken object has the vehicle (vehicle in front) added. This is an example of a relaxed ODD.

[0161] Figure 10 (a) shows the vehicle conditions in the oncoming lane driving mode. The maximum speed is 10 km / h, the maximum steering angle is 45 degrees, the maximum acceleration is 0.3G, and the system response time is 0.8s.

[0162] Figure 10 (b) refers to the conditions for safety monitoring. In the opposite lane driving mode, the area to be monitored is the vehicle's own lane and the opposite lane, which can be replaced by monitoring by the remote operator H (however, limited to a person with low fatigue). This means that the area to be monitored can also be monitored by the vehicle 200 itself (e.g., detecting objects), and the remote operator H can also remotely monitor at least a portion of the area to be monitored. In addition, the fatigue level of the remote operator H can be obtained, for example, by capturing images of the remote operator H. Furthermore, the area to be monitored by the remote operator H can also be an area where the vehicle 200 itself cannot detect objects or an area where the reliability of object detection is low.

[0163] Figure 10 (c) represents the system condition: in the opposite lane driving mode, the communication delay is 0.2s (maximum 0.2s), and the system status is normal.

[0164] Refer again Figure 8 Next, the mode change unit 151 obtains the first vehicle information (S302), the first remote operation information regarding remote operation (S303), and the first system information (S304). The sources for obtaining the first vehicle information, the first remote operation information, and the first system information are not particularly limited. Taking the first vehicle information as an example, the mode change unit 151 can obtain the first vehicle information from the vehicle 200, or if the first vehicle information is stored in the storage unit 167, it can be read from that storage unit 167, or it can be obtained from a server device that manages the vehicle information of the vehicle 200. Furthermore, the first system information can also be obtained via the system information acquisition unit 166. Obtaining the first vehicle information, the first remote operation information, and the first system information at least once is sufficient; if the first vehicle information, the first remote operation information, and the first system information have already been obtained, the processing steps S302 to S304 can be omitted.

[0165] The following refers to the information on the first vehicle, the first remote operation, and the first system. Figures 11 to 13 Please provide an explanation. Figure 11 This is a diagram showing an example of information about the first vehicle. Figure 12 This is a diagram illustrating an example of the first remote operation information. Figure 13 This is a diagram illustrating an example of information from the first system.

[0166] like Figure 11 As shown, the first vehicle information displays the specifications of vehicle 200, including maximum speed, maximum steering angle, maximum acceleration, system response time, and the possible monitoring area. The possible monitoring area is its own lane; therefore, vehicle 200 can only monitor its own lane.

[0167] like Figure 12 As shown in (a) and (b), the first remote operation information includes information about the remote operation device 130 and information (specifications) about the remote operator H. The information about the remote operation device 130 includes information about the capabilities of the remote operation device 130, such as the possible monitoring areas that the remote operation device 130 can monitor. Figure 12 Example (a) shows that the remote operating device 130 has the ability to monitor the opposite lane. Information about the remote operator H includes the remote operator H's permissions during remote operation. Figure 12 Example (b) shows that remote operator H is granted permission to perform remote operations. Pre-configured Figure 12 The contents recorded in (a) and (b).

[0168] like Figure 13 As shown, the first system information is information illustrating the specifications of the vehicle control system 10, and includes information on minimum communication latency. Figure 13 In the example, the minimum communication delay is 0.1s.

[0169] Refer again Figure 8 The mode change unit 151 retrieves drivable driving modes based on the various information obtained in steps S301 to S304 (S305). The mode change unit 151, based on each of the first vehicle information, the first remote operation information, and the first system information, retrieves the drivable driving mode as the constraint condition and the corresponding ODD if the constraint condition is satisfied. The mode change unit 151 may also retrieve, for example, multiple drivable driving modes.

[0170] Furthermore, the mode-changing unit 151 may also determine, in step S305, whether it can switch to the remote operation mode. The mode-changing unit 151 may also determine whether it can switch to the remote operation mode based on whether the constraints corresponding to the remote operation mode are met. The constraints corresponding to the remote operation mode include, for example, communication delay.

[0171] Refer again Figure 4 Next, if a driving mode is available (S104 "Yes"), the prompting unit 152 prompts the remote operator H with the driving mode (S105). The prompting unit 152 may also prompt the remote operator H with multiple driving modes. The prompting unit 152 may also, for example, [display multiple driving modes]. Figure 7 As shown in (b), buttons 112 and 113 are superimposed on the image to provide a prompt.

[0172] Alternatively, the prompting unit 152 may, when the mode change unit 151 searches for multiple driving modes, prioritize prompting the driving mode whose constraints are relaxed.

[0173] Furthermore, the prompting unit 152 may also, when prompting the remote operator H about the ODD relaxed driving mode, further overlap the area that the remote operator H should remotely monitor in the ODD relaxed driving mode (e.g., is...). Figure 7 (c) shows the area that needs to be monitored, and is an example of enhanced monitoring conditions. Thus, the display device 110 can also indicate the area that the remote operator H should remotely monitor in this ODD relaxed driving mode before the remote operator H operates button 112.

[0174] Accordingly, the remote operator H can determine whether to switch to the ODD (Operation Distance Controller) mode after confirming the area they should monitor. Additionally, the prompting unit 152 obtains the area to be monitored from the area determination unit 161.

[0175] Next, the command receiving unit 153 receives the driving mode selection from the remote operator H (S106). The command receiving unit 153 detects, for example, operations on buttons 112 or 113 (e.g., touch operations). Alternatively, the command receiving unit 153 receives an operation from the remote operator H allowing the switching of driving modes. Thus, the remote operating device 130, for example, performs the driving mode switching process upon receiving the operation from the remote operator H allowing the switching of driving modes. For example, if the remote operating device 130 can drive in ODD-enhanced automatic driving mode and the remote operator H allows switching to ODD-enhanced driving mode, it switches to ODD-enhanced driving mode.

[0176] If the command receiving unit 153 receives a driving mode selection from the remote operator, the mode changing unit 151 executes a driving mode switching process (S107) to switch to the accepted driving mode. If an operation on button 112 is detected, the mode changing unit 151 switches the vehicle 200 to ODD relaxed driving mode; if an operation on button 113 is detected, it executes a process to switch the vehicle 200 to remote operation mode. The operation on button 112 is an example of the first operation.

[0177] Here, regarding the operation of the instruction receiving unit 153 when it receives the selection of the driving mode for the opposite lane, please refer to... Figure 14 Please provide an explanation. Figure 14 It is shown Figure 4 The flowchart shows an example of the switching process shown in step S107.

[0178] like Figure 14As shown, the mode change unit 151 changes the ODD of the vehicle 200 (S401). The mode change unit 151 changes the conditions of the ODD of the vehicle 200 to the ODD in the switched driving mode (for example, refer to...). Figure 9 The mode change unit 151, via the communication unit 140, outputs information showing the ODD changed to the ODD of the switched driving mode to the vehicle 200. If the mode change unit 151 detects operation of button 112 by the remote operator H, it changes the ODD of the vehicle 200 from the ODD of the automatic driving mode to the ODD of the ODD relaxed driving mode. In other words, if the mode change unit 151 detects operation of button 112 by the remote operator H, it relaxes the ODD of the vehicle 200 compared to the ODD of the automatic driving mode when no operation was detected.

[0179] Next, the mode change unit 151 changes the settings of the vehicle 200 (S402). The mode change unit 151 changes the criteria used by the vehicle 200 driving feasibility determination unit 231 to determine whether driving is possible. The mode change unit 151 changes this criterion to the vehicle conditions in the switched driving mode (for example, referring to...). Figure 10 (a)). The mode change unit 151, via the communication unit 140, outputs information showing the vehicle conditions in the changed driving mode to the vehicle 200. Furthermore, the mode change unit 151 outputs information regarding the constraints of the changed ODD (for example, referring to...). Figure 10 Output to the region judgment unit 161.

[0180] Furthermore, the autonomous driving unit 232 generates an autonomous driving plan for the ODD-relaxed driving mode based on the relaxed ODD and the modified vehicle 200 settings. In other words, the autonomous driving unit 232 updates the driving plan according to the relaxed ODD. And, in order to execute this driving plan, the autonomous driving unit 232 outputs control information based on the driving plan to the vehicle control unit 240.

[0181] The remote operator H can also monitor the driving plan or control information of vehicle 200 in ODD relaxed driving mode. The information showing the driving plan or control information is an example of auxiliary information, which can also be prompted by the display device 110.

[0182] Thus, monitoring conditions may also include the operation of vehicle 200 that needs to be monitored by remote operator H, and enhanced monitoring conditions may also include monitoring of vehicle 200’s driving plan or driving control information in accordance with the relaxation of ODD.

[0183] Next, the area determination unit 161 sets the area to be monitored by the remote operator H (S403). The area determination unit 161 sets the area to be monitored by the remote operator H in a manner that satisfies the constraints for the switched ODD. The area determination unit 161 sets the area to be monitored by the remote operator H based on these constraints, the first vehicle information, and the first remote operation information. For example, the area determination unit 161 sets an area that cannot be monitored by the vehicle 200, within the areas that are set as areas to be monitored under safety monitoring conditions, as the monitoring area for the remote operator H.

[0184] like Figure 10 As shown in (b), the area that needs to be monitored is the lane itself and the opposite lane. Furthermore, as... Figure 11 As shown, the monitoring area of ​​vehicle 200 is its own lane. In this case, vehicle 200 cannot monitor the opposite lane, therefore, the opposite lane is set as the monitoring area of ​​remote operator H.

[0185] The relaxation approval determination unit 162 can also determine, based on the first remote operation information, whether the remote operation device 130 has the capability to monitor the monitoring area set by the area determination unit 161. If it does not have this capability, it outputs information indicating that the area to be monitored as shown by the constraint conditions cannot be monitored to the mode change unit 151. In this case, the mode change unit 151 can also re-explore other driving modes (e.g., other ODD relaxation driving modes).

[0186] Refer again Figure 14 Next, the driving continuation determination unit 164 sets the driving continuation conditions in the switched ODD relaxed driving mode (S404). Here, the driving continuation conditions are the basis for determining whether to continue driving in the switched ODD relaxed driving mode. The driving continuation determination unit 164 may, for example, set the constraint conditions of the switched relaxed driving mode as driving continuation conditions. The driving continuation determination unit 164 can, for example, determine that driving cannot continue and stop the vehicle 200 when the constraint conditions corresponding to the switched driving mode are not met during driving in the switched driving mode, such as when the vehicle speed exceeds 10 km / h, when the remote operator H cannot monitor the opposite lane, or when the communication delay exceeds 0.2 s. The driving continuation determination unit 164 performs the determination in step S404 when the vehicle 200 is driving in the ODD relaxed driving mode; however, the determination in step S404 can also be performed when the vehicle 200 is driving in the remote operation mode.

[0187] Next, the prompting unit 152 prompts the remote operator H with the area to be monitored and auxiliary information (S405). The area to be monitored is the area that needs to be monitored by the remote operator H. The auxiliary information is information that assists in remote monitoring, and includes at least one of the following: vehicle speed, driving path during automatic driving, current driving mode, and the judgment result of the driving continuation determination unit 164.

[0188] Figure 7 (c) is a diagram showing the state of switching to ODD relaxed driving mode. Figure 7 (c) shows Figure 7 The state after selecting the opposite lane driving mode in (b). Additionally, Figure 7 (c) shows vehicle 410 traveling in the opposite lane. And, in Figure 7 In (c), the areas that need to be monitored are indicated by dotted shading, which are the areas that need to be monitored by the remote operator H.

[0189] like Figure 7 As shown in (c), the display unit 152 displays the areas that need to be monitored by the remote operator H due to the relaxation of the ODD, as well as auxiliary information related to the operation of the vehicle 200. For example, the display unit 152 overlays the areas to be monitored and the auxiliary information onto an image from the vehicle 200, causing the display device 110 to display the information. Thus, the display unit 152 displays the areas that need to be monitored due to the relaxation of the ODD, making it easy for the remote operator H to know the areas they should monitor. Furthermore, the areas that need to be monitored due to the relaxation of the ODD represent an example of enhanced monitoring conditions.

[0190] Furthermore, the prompting unit 152 can also prompt the display device 110 to indicate the driving route and the current driving mode. Figure 7 (c) shows "ODD relaxed driving", and the judgment result of the driving continuation judgment unit 164. Figure 7 (c) shows "RUN" etc. Accordingly, the remote operator H can remotely monitor the vehicle 200 by referring to this auxiliary information. Furthermore, the prompting unit 152 can also prompt the display device 110 to display information such as "Caution ahead" when there is a vehicle 410 on the driving path, as an alarm for an approaching object. The warning information is also included in the auxiliary information.

[0191] Furthermore, in Figure 7In (c), button 114, which indicates "Normal Driving," serves as a driving mode switching button. Normal driving refers to automatic driving mode; if button 114 is operated, vehicle 200 switches to automatic driving mode. In other words, when button 114 is operated, mode changing unit 151 switches from the current driving mode (e.g., ODD relaxed driving mode) to automatic driving mode. Operation of button 114 signifies the end of ODD relaxed driving mode.

[0192] Furthermore, in Figure 7 (c) also indicates a button 115 indicating "Continue Driving". Button 115 is used to accept operations from the remote operator H when continuing in ODD relaxed driving mode. Additionally, button 115 is displayed when the driving mode is switched to ODD relaxed driving mode. Furthermore, the operation of button 115 by the remote operator H is an example of the second operation.

[0193] Refer again Figure 14 The mode change unit 151 causes the vehicle 200 to begin automatic driving in the ODD relaxed driving mode (S406). The mode change unit 151, for example, outputs information indicating the start of automatic driving in the ODD relaxed driving mode to the vehicle 200. The mode change unit 151 may also, for example, output information indicating the start of automatic driving in the ODD relaxed driving mode to the vehicle 200. Figure 7 After the image shown in (c) is displayed to the remote operator H, when an operation indicating the start of automatic driving in the ODD relaxed driving mode is detected from the remote operator H (e.g., operation of touching button 115), the vehicle 200 starts automatic driving in the ODD relaxed driving mode. Alternatively, the operation of button 115 for starting the ODD relaxed driving mode can also be the first operation.

[0194] Thus, the mode change unit 151 enables the vehicle 200 to drive automatically (autonomous driving) with a relaxed ODD.

[0195] If the automatic driving unit 232 of vehicle 200 receives information indicating the commencement of automatic driving in ODD relaxed driving mode, it generates a command to commence such automatic driving and outputs it to vehicle control unit 24. Accordingly, vehicle 200 commences automatic driving in ODD relaxed driving mode.

[0196] Thus, when the driving mode selection is accepted in step S106, the switching process in step S107 is performed once.

[0197] Refer again Figure 4 Next, the driving continuation determination unit 164 performs driving continuation determination processing (S108) on the vehicle 200 that is automatically driving in ODD relaxed driving mode. Figure 15 It is shown Figure 4The flowchart shows an example of the judgment processing operation shown in step S108.

[0198] like Figure 15 As shown, the driving continuation determination unit 164 determines whether driving in the current driving mode can continue, provided that the current driving mode is either an ODD relaxed driving mode or a remote operation mode. The driving continuation determination unit 164 makes this determination based on whether the driving continuation conditions corresponding to the current driving mode are met, according to the current state of the vehicle control system 10. The driving continuation determination unit 164 performs this determination repeatedly, for example, at predetermined time intervals. For example, if the current driving mode is an ODD relaxed driving mode, the driving continuation determination unit 164 determines whether automatic driving in the ODD relaxed driving mode can continue (S407).

[0199] Here, the judgment process in step S407 is referred to Figure 16 Please provide an explanation. Figure 16 It is shown Figure 15 The flowchart for the judgment process shown in step S407.

[0200] like Figure 16 As shown, the driving continuation determination unit 164 obtains second vehicle information from vehicle 200 (S501). Regardless of the driving mode, second vehicle information is obtained sequentially from vehicle 200 (see reference). Figure 6 ).

[0201] Next, the driving continuation determination unit 164 obtains the second remote operation information (S502). The second remote operation information shows the current status of the remote operator H. Figure 17 This is a diagram illustrating an example of the second remote operation information.

[0202] like Figure 17 As shown, the second remote operation information includes the fatigue level of the remote operator H, which serves as the current state of the remote operator H. Alternatively, the driving continuation feasibility determination unit 164 obtains the fatigue level of the remote operator H as its current state. The fatigue level indicates the current degree of fatigue of the remote operator H, for example, using terms such as "high," "medium," or "low" to represent a level or numerical value. Figure 17 In the example, can the driver determine whether the fatigue level of the remote operator H is "low" if the driving continues?

[0203] Furthermore, the second remote operation information may also include information other than fatigue level. The second remote operation information may also include information related to remote monitoring by the remote operator H. The second remote monitoring information may also include, for example, a determination of whether the remote operator H is viewing the display device 110, or information related to the direction of the remote operator H's gaze. In other words, the second remote operation information may also include a determination of whether the remote operator H is performing the required remote monitoring. Additionally, the second remote operation information may include at least one of fatigue level and information regarding remote monitoring.

[0204] Refer again Figure 16 Next, the driving continuation feasibility determination unit 164 obtains second system information (S503). The second system information displays the current state of the vehicle control system 10. This second system information displays, for example, the network status such as communication latency and communication band, and the computing resources and program operation status of the remote operation device 130, indicating the current integrity of the vehicle control system 10. The driving continuation feasibility determination unit 164 may also obtain the second system information, for example, via the system information acquisition unit 166. Figure 18 This is a diagram illustrating an example of information from the second system.

[0205] like Figure 18 As shown, the second system information includes the communication delay of the vehicle control system 10 and the system state of the vehicle control system 10, serving as the current state of the vehicle control system 10. In other words, the driving continuation determination unit 164 obtains the current communication delay and system state as the current state of the vehicle control system 10. Figure 18 In the example, the driving continuation determination unit 164 uses a communication delay of 0.2s and a normal (healthy) system status as the criteria for determining whether driving is permissible. Furthermore, the method for obtaining the second system information is not particularly limited. The driving continuation determination unit 164 may, for example, calculate the current communication delay based on the time of obtaining the information and the timestamp included in the information, after obtaining information from the vehicle 200. Also, for example, the driving continuation determination unit 164 may obtain the current system status from a monitoring device (not shown) that monitors the system status of the vehicle control system 10.

[0206] Refer again Figure 16 The driving continuation determination unit 164 determines, based on the information obtained in steps S501 to S503, whether the driving continuation conditions corresponding to the current driving mode are met (S504). In other words, the driving continuation determination unit 164 determines whether the current state of the vehicle control system 10 meets the driving continuation conditions.

[0207] If the driving continuation determination unit 164 determines that the automatic driving in the ODD relaxed driving mode can continue if the second vehicle information, the second remote operation information, and the second system information all meet the driving continuation conditions (S504 "Yes"), then proceeds to the next step. Figure 15 Step S408. Furthermore, the driving continuation determination unit 164 may also determine "yes" in step S504 if, based on the determination result of whether the remote operator H is viewing the display device 110, or if information related to the direction of the remote operator H's gaze is included in the second remote monitoring information, the remote operator H is viewing the display device 110, or the remote operator H's gaze is directed towards the display device 110 (e.g., towards the area of ​​the display device 110 that needs to be monitored). In other words, the driving continuation determination unit 164 determines "yes" in step S504 if the remote operator H is monitoring under enhanced monitoring conditions. Accordingly, the vehicle 200 can automatically drive with a relaxed ODD only when the remote operator H is monitoring under enhanced monitoring conditions.

[0208] Furthermore, if at least one of the second vehicle information, the second remote operation information, and the second system information fails to meet the conditions for continuing driving ("No" in S504), the driving continuation determination unit 164 determines that automatic driving in the ODD relaxed driving mode cannot continue (S506), and proceeds to... Figure 15 Step S410.

[0209] Refer again Figure 15 If the driving continuation determination unit 164 determines that automatic driving in the ODD relaxed driving mode can continue, i.e., the driving continuation conditions are met (S407 "Yes"), then it determines whether the command receiving unit 153 has received a driving continuation command from the remote operator H (S408). Furthermore, if the driving continuation determination unit 164 determines that automatic driving in the ODD relaxed driving mode cannot continue, i.e., the driving continuation conditions are not met (S407 "No"), then it stops the vehicle 200 from driving (S410) and enters... Figure 4 In step S102, the remote operator H is notified. However, if step S407 returns "No," the process is not limited to stopping the vehicle 200; switching driving modes can also be performed. In this case, at least one of the switchable ODD relaxed driving modes and the remote operation mode is notified to the remote operator H. Here, the switchable ODD relaxed driving mode is the driving mode that was "Yes" in the determination of step S407, for example, a driving mode that increases the monitoring burden or driving restrictions for the remote operator H compared to the current ODD relaxed driving mode. Furthermore, in this case, the vehicle 200 may not need to be stopped.

[0210] Figure 7 (d) is a diagram showing the state of driving in ODD relaxed driving mode. Figure 7 (d) shows that from Figure 7 The state of (c) changes to driving in the opposite lane in ODD relaxed driving mode and exceeding the vehicle's speed limit by 400.

[0211] like Figure 7 As shown in (d), the driving continuation determination unit 164, for example, determines whether the remote operator H has operated the button 115 that displays "Continue Driving" in step S408.

[0212] Alternatively, driving in the relaxed ODD mode can continue while the remote operator H continuously operates button 115 (e.g., continuously touches it). In this case, driving in the relaxed ODD mode only occurs while the remote operator H operates button 115. Detecting the remote operator H's operation on button 115 is equivalent to the remote operator H being monitored. Monitoring can also refer to, for example, the period during which the operation of button 115 is detected. Accordingly, the vehicle 200 can be driven automatically in the relaxed ODD only during the period of enhanced monitoring by the remote operator H (only during the operation of button 115).

[0213] Furthermore, whenever the remote operator H operates button 115, the ODD relaxed driving mode can be activated for a certain period of time, a certain range, or a certain distance. The remote operator H can continue the automatic driving in the ODD relaxed driving mode by continuously operating button 115 at specified time intervals.

[0214] Here, for the processing of step S408, refer to Figure 19 as well as Figure 20 Please provide an explanation. Figure 19 It is shown Figure 15 The flowchart for the judgment process shown in step S408. Figure 20 This is a diagram illustrating an example of an instruction.

[0215] like Figure 19 As shown, the driving continuation determination unit 164 receives instructions from the remote operator H from the instruction receiving unit 153 (S601). For example, the driving continuation determination unit 164 receives information from the instruction receiving unit 153 indicating that the remote operator H has operated button 115.

[0216] Next, the driving continuation determination unit 164 confirms the timestamp included in the instruction (S602). That is, the driving continuation determination unit 164 confirms that the received instruction is the instruction obtained by the remote operator H when he / she operated the button 115.

[0217] The driving continuation determination unit 164 may also determine, by checking the timestamp, whether the instruction was obtained through operation by remote operator H within a specified time, or whether it is the latest instruction. Therefore, the driving continuation determination unit 164 may proceed to step S603 if the instruction was obtained through operation by remote operator H within a specified time or is the latest instruction, and proceed to step S411 if the instruction was obtained through operation by remote operator H outside the specified time or is an old instruction.

[0218] like Figure 20 As shown, the driving permission command is generated by the remote operator H operating button 115, and includes a timestamp and the command itself. The timestamp indicates, for example, the date and time when the remote operator H operated button 115. Furthermore, the command is information corresponding to the operated button, indicating that driving should continue when button 115 is operated.

[0219] Refer again Figure 19 Next, the driving continuation determination unit 164 determines whether the communication delay is below a certain threshold (S603). The driving continuation determination unit 164, for example, obtains the current communication delay time, and if the obtained communication delay meets the conditions for continuing driving, determines that the communication delay is below a certain threshold. Accordingly, it is possible to determine whether to continue driving in the ODD relaxed driving mode based on the real-time operation of the button 115 by the remote operator H.

[0220] If the communication delay is below a certain threshold ("Yes" in S603) and the instruction obtained in step S601 is an instruction to continue driving ("Yes" in S604), then the system determines that driving can continue. Figure 15 The step S409 is shown. Furthermore, the driving continuation determination unit 164 determines whether to proceed if the communication delay is not below a certain threshold ("No" in S603) or if the instruction obtained in step S601 is not an instruction to continue driving ("No" in S604). Figure 15 Step S411 is shown.

[0221] Refer again Figure 15 If the driving continues, the determination unit 164 determines whether the driving continues. If the result of step S408 is "yes", the automatic driving of the ODD relaxed driving mode continues (S409), and it is determined that the driving has entered the driving mode. Figure 4 Step S109. Accordingly, automatic driving in the ODD relaxed driving mode continues only if the remote operator H permits continued automatic driving in the ODD relaxed driving mode. Therefore, the safety of automatic driving in the ODD relaxed driving mode can be further ensured.

[0222] Furthermore, if the driving continuation determination unit 164 does not receive a driving continuation instruction ("No" in S408), it stops the driving of vehicle 200 (S411) and proceeds to step S408. Here, the stopping of driving is a pause. If a driving continuation instruction is received again after the driving stops ("Yes" in step S408), vehicle 200 can resume driving in ODD relaxed driving mode.

[0223] Thus, remote operator H, while continuing driving in ODD relaxed driving mode, can maintain the ODD relaxed driving mode simply by operating (e.g., touching) button 115. Therefore, even while continuing in ODD relaxed driving mode, the workload of remote operator H can be reduced. Furthermore, Figure 15 The judgment and processing shown is a process that is repeatedly executed during driving in the ODD relaxed driving mode.

[0224] Furthermore, if the current driving mode is remote operation mode, the judgment process in step S408 is not performed (see...). Figure 22 In other words, if the driving mode is remote operation mode and the driving continuation determination unit 164 determines that driving in remote operation mode is possible, then driving in remote operation mode will continue.

[0225] Refer again Figure 4 If the mode-changing unit 151 receives a switch operation to the automatic driving mode (S109 "Yes") while driving in the ODD relaxed driving mode or remote operation mode, it switches the driving mode to the automatic driving mode (S111) and returns to step S101 to continue subsequent processing. If the mode-changing unit 151 does not receive a switch operation to the automatic driving mode (S109 "No") while driving in the ODD relaxed driving mode or remote operation mode, it determines whether the destination has been reached (S110). If the destination has been reached (S110 "Yes"), the remote monitoring and remote operation of the vehicle 200 by the remote operator H ends. If the destination has not been reached (S110 "No"), the mode-changing unit 151 proceeds to step S108 to continue subsequent processing.

[0226] Furthermore, if there is no driving mode (S104 "No"), the prompt unit 152 prompts the display device 110 that driving is not possible, and ends the driving of the vehicle 200 (S113).

[0227] in addition, Figure 4 The processes shown in steps S103 to S105, S112 and S113 can also be described as determining whether a mode switch is possible.

[0228] Next, regarding the operation of the instruction receiving unit 153 when it receives the selection of the remote operation mode, for example, in Figure 7 In image (b), the remote operator H operates button 113, referring to... Figure 21 Please provide an explanation. Figure 21 It is shown Figure 4 The flowchart for another example of the operation of controlling driving is shown in step S107.

[0229] like Figure 21 As shown, the area determination unit 161 sets the area to be monitored by the remote operator H (S701). The area determination unit 161 sets the area to be monitored by the remote operator H in a manner that satisfies the constraints for the remote operation mode. For example, the area to be monitored under safety monitoring conditions is set as the monitoring area for the remote operator H. The area to be monitored under safety monitoring conditions can also be the entire area surrounding the vehicle 200.

[0230] The remote operation capability determination unit 163 may also determine, based on the first remote operation information, whether the remote operation device 130 has the capability to monitor the monitoring area set by the area determination unit 161. If it does not have this capability, it outputs information indicating that the area to be monitored as shown by the constraint conditions cannot be monitored to the mode change unit 151. In this case, the mode change unit 151 may also re-explore other driving modes (e.g., ODD relaxed driving mode) or stop the vehicle 200 from moving.

[0231] Next, the driving continuation determination unit 164 sets the driving continuation conditions in the remote operation mode (S702). Here, the driving continuation conditions are the basis for determining whether to continue driving in the remote operation mode. The driving continuation determination unit 164 may, for example, set the constraint conditions of the remote operation mode as driving continuation conditions. The constraint conditions of the remote operation mode include, for example... Figure 10 At least a portion of the conditions shown in (a) to (c).

[0232] Next, the prompting unit 152 prompts the remote operator H with the area to be monitored and auxiliary information (S703). The area to be monitored is the area that needs to be monitored by the remote operator H. The auxiliary information is information that assists in remote operation, and includes at least one of the following: vehicle speed, current driving mode, and the judgment result of the driving continuation determination unit 164.

[0233] Next, the mode change unit 151 causes the vehicle 200 to begin automatic driving in remote operation mode (S704). The operation receiving unit 154, for example, if it receives an operation received by the operation input device 120, generates control information based on the received operation and outputs it to the vehicle 200, thereby starting driving in remote operation mode.

[0234] Thus, if the driving mode selection is accepted in step S106, the switching process in step S107 is performed, for example.

[0235] Refer again Figure 4 Next, the driving continuation determination unit 164 performs driving continuation determination processing (S108) on the vehicle 200 in remote operation mode. Figure 22 It is shown Figure 4 The flowchart shows another example of the judgment processing work shown in step S108.

[0236] like Figure 22 As shown, next, the driving continuation possibility determination unit 164 determines whether driving in remote operation mode can continue (S705). If the driving continuation possibility determination unit 164 determines that driving in remote operation mode can continue, that is, if the driving continuation condition is met (S705 "Yes"), the driving in remote operation mode continues (S706), and proceeds to... Figure 4 Step S109. Furthermore, if the driving continuation determination unit 164 determines that driving in remote operation mode cannot continue, i.e., the driving continuation conditions are not met ("No" in S705), it stops the driving of vehicle 200 (S707) and proceeds to... Figure 4 In step S102, the remote operator H is notified.

[0237] Furthermore, if step S705 selects "No," the process is not limited to stopping the vehicle 200; switching driving modes can also be performed. For example, if the vehicle 200 is a manually drivable vehicle, switching to manual driving mode can also be performed. In this case, the remote operator H is notified that the manual driving mode has been switched. Also, in this case, the vehicle 200 may not need to be stopped. Alternatively, if an option is available, switching to ODD relaxed mode can be performed.

[0238] Furthermore, during remote operation mode driving, the process is repeatedly executed. Figure 22 The judgment and processing are shown below.

[0239] [3. Application Examples]

[0240] The following are applicable examples for relaxing the switching of driving modes to ODD under various conditions, with reference to... Figures 23 to 32CAn explanation will be provided below. Furthermore, the examples provided are merely illustrative; the ODD and constraints corresponding to the condition of vehicle 200 are not limited to these. Also, for ease of explanation, the following describes the case of switching from automatic driving mode to ODD relaxed driving mode.

[0241] First, for the first applicable example, refer to Figures 23 to 24B Please provide an explanation. Figure 23 This is a diagram used to illustrate the situation of the first applicable example of relaxing ODD. Figure 23 It is, for example, an image displayed by display device 110, and an image confirmed by, for example, a remote operator H. Figure 24A This is a diagram illustrating the various conditions (ODD and constraints) in the automatic driving mode of the first applicable example. Figure 24B This is a diagram illustrating the various conditions (ODD and constraints) in the ODD relaxed driving mode of the first applicable example.

[0242] As a first applicable example, we will explain the case where the types of objects that the ODD (Operational Distance) can exceed and travel over are relaxed. Specifically, the case to be explained is that a vehicle 420 is parked in front of vehicle 200 but not on the side of the road, and vehicles not on the side of the road are not included in the objects that the ODD can exceed in automatic driving mode. In addition, in order to show that vehicle 200 detects vehicle 420 by sensing, a box surrounding vehicle 420 is shown in the figure.

[0243] like Figure 23 As shown, there is a vehicle 420 (the vehicle in front) parked ahead that is not parked on the side of the road. However, the vehicle 420 that is not parked on the side of the road is not included in the objects exceeding the limit of the ODD. Therefore, vehicle 200 cannot overtake vehicle 420 in automatic driving mode. Additionally, Figure 23 The × mark indicates that the vehicle cannot be driven. The × mark can also be included in auxiliary information.

[0244] like Figure 24A As shown, in automatic driving mode, the objects that the ODD (Operation Distance) can overtake include vehicles (parked on the side of the road) and colored traffic cones (registered trademark). That is, in automatic driving mode, vehicle 200 cannot overtake vehicle 420 that is not parked on the side of the road.

[0245] Therefore, the remote control device 130 switches from automatic driving mode to ODD (Optical Discharge Discharge) relaxed driving mode. Specifically, the remote control device 130 switches to an ODD relaxed driving mode that allows it to overtake vehicles not parked on the side of the road. Figure 4 In step S103, the ODD relaxed driving mode that can exceed the vehicle that is not parked on the side of the road is retrieved from the multiple ODD relaxed driving modes stored in the storage unit 167. Figure 24BAs shown, for example, the various conditions in the ODD relaxed driving mode retrieved in step S103.

[0246] like Figure 24B As shown, in the ODD relaxed driving mode, vehicles (vehicles ahead) and pedestrians (stationary) are added to the list of objects that can be overtaken by the ODD. That is, in this ODD relaxed driving mode, vehicle 200 can overtake vehicle 420 that is not parked on the side of the road. Furthermore, although the conditions of the ODD are relaxed, the constraints in this ODD relaxed driving mode are stricter than those in the automatic driving mode. For example, regarding vehicle conditions, the maximum speed is limited from 20 km / h to 5 km / h.

[0247] Thus, by relaxing the restrictions on the Operating Disclosure (ODD), the limitations on autonomous driving modes are strengthened. For example, by relaxing the ODD, at least one of the maximum vehicle speed, maximum steering angle, and maximum acceleration during autonomous driving under the relaxed ODD is strengthened. Specifically, at least one value of the maximum vehicle speed, maximum steering angle, and maximum acceleration is changed to a smaller value. Figure 24B In the example, the maximum speed limit was strengthened.

[0248] Furthermore, regarding the conditions for safety monitoring, the vehicle ahead is added to the list of objects requiring remote monitoring. In other words, in order to drive in the relaxed driving mode of this ODD, it is necessary for the remote operator H to monitor the vehicle ahead.

[0249] Thus, in accordance with the relaxation of the ODD, the monitoring conditions for the remote operator H of vehicle 200 are strengthened. Strengthening the monitoring conditions includes, for example, strengthening the monitoring of at least one of the monitoring areas and objects monitored by the remote operator H during autonomous driving under the relaxed ODD. Alternatively, the monitoring areas and objects monitored by the remote operator H can be changed to those monitored during autonomous driving under the relaxed ODD. Specifically, this includes expanding the monitoring area or adding monitoring objects. Figure 24B In the example, the surveillance of the monitored object was strengthened.

[0250] If satisfied Figure 24B Under the constraints shown, vehicle 200 can automatically drive past vehicle 420, which is not parked on the side of the road, under the supervision of remote operator H.

[0251] Next, for the second applicable example, refer to Figures 25 to 26B Please provide an explanation. Figure 25 This is a diagram used to illustrate the situation of the second applicable example of relaxing ODD. Figure 26A This is a diagram illustrating various conditions in the automatic driving mode of the second applicable example. Figure 26BThis is a diagram illustrating various conditions in the ODD relaxed driving mode of the second applicable example.

[0252] As a second applicable example, we will explain the situation where the driving possibility area of ​​the ODD is widened. Specifically, we will explain the situation where there is a vehicle 430 parked in front of vehicle 200 that is not parked on the side of the road, and the driving possibility area in the automatic driving mode does not include the opposite lane.

[0253] like Figure 25 As shown, there is a vehicle 430 parked ahead that is not parked on the side of the road. In order to pass the vehicle 430, it is necessary to drive in the opposite lane. However, if the opposite lane is not included in the driving possibility area of ​​the ODD, the vehicle 200 cannot pass the vehicle 430 in automatic driving mode.

[0254] like Figure 26A As shown, in automatic driving mode, the driving possibility area of ​​the ODD only includes its own lane. That is to say, in automatic driving mode, vehicle 200 cannot travel in the opposite lane to overtake vehicle 430.

[0255] Therefore, the remote control device 130 switches from automatic driving mode to ODD (Optical Discharge Drive) relaxed driving mode. Specifically, the remote control device 130 switches to the ODD relaxed driving mode, which allows driving in the opposite lane. The remote control device 130... Figure 4 In step S103, the ODD relaxed driving mode that allows driving on the opposite lane is retrieved from the multiple ODD relaxed driving modes stored in the storage unit 167. Figure 26B As shown, for example, the various conditions in the ODD relaxed driving mode retrieved in step S103.

[0256] like Figure 26B As shown, in the driving possibility area of ​​the ODD relaxed driving mode, the oncoming lane is added. That is, in this ODD relaxed driving mode, vehicle 200 can overtake vehicle 430 by traveling in the oncoming lane. Furthermore, although the conditions for ODD are relaxed, the constraints in this ODD relaxed driving mode are stricter than those in the automatic driving mode. Regarding vehicle conditions, the maximum speed is limited from 20 km / h to 5 km / h. Thus, in Figure 26B In the example, the maximum speed limit was strengthened.

[0257] Furthermore, regarding the conditions for safety monitoring, the opposite lane is added to the monitoring area. That is, in order to drive in this ODD relaxed driving mode, the opposite lane needs to be monitored by the remote operator H. Thus, in Figure 26B In the example, the surveillance area is expanded (surveillance area).

[0258] In satisfying Figure 26BUnder the constraints shown, under the supervision of remote operator H, vehicle 200 is able to drive automatically in the opposite lane to overtake vehicle 430.

[0259] Next, for the third applicable example, refer to Figures 27 to 28B Please provide an explanation. Figure 27 This is a diagram used to illustrate the situation of the third applicable example of the relaxed ODD. Figure 28A This is a diagram illustrating various conditions in the automatic driving mode of the third applicable example. Figure 28B This is a diagram illustrating various conditions in the ODD relaxed driving mode of the third applicable example.

[0260] As a third applicable example, we will explain the situation where the conditions for the ODD's inability to drive are relaxed. The inability to drive conditions are conditions that prohibit the vehicle 200 from driving. Specifically, we will explain the situation where an earthquake occurs while the vehicle is driving in automatic driving mode, and the inability to drive in automatic driving mode includes an earthquake (magnitude: 3 or higher). Furthermore, the vehicle 200, for example, obtains information related to the occurrence of an earthquake or other weather or disaster from an external device via communication.

[0261] like Figure 27 As shown, if an earthquake of magnitude 3 or higher occurs, vehicle 200 will stop because the driving conditions of ODD include the occurrence of an earthquake (magnitude: 3 or higher).

[0262] like Figure 28A As shown, in automatic driving mode, the driving inability conditions under the ODD include conditions related to weather and disasters. Specifically, the driving inability conditions under the ODD include earthquakes (magnitude 3 or higher), heavy rain (rainfall 200mm or more), dense fog (visibility less than 60m), and strong winds (wind speed 15m / s or higher). In other words, in automatic driving mode, the vehicle 200 cannot drive in the event of an earthquake of magnitude 3 or higher.

[0263] Therefore, the remote control device 130 switches from automatic driving mode to ODD relaxed driving mode. Specifically, the remote control device 130 switches to ODD relaxed driving mode, which allows driving even in the event of an earthquake of magnitude 3 or higher. Figure 4 In step S103, an ODD relaxed driving mode that can be driven even in the event of an earthquake of magnitude 3 or higher is retrieved from the multiple ODD relaxed driving modes stored in the storage unit 167. Figure 28B As shown, for example, the various conditions in the ODD relaxed driving mode retrieved in step S103.

[0264] like Figure 28BAs shown, earthquakes have been removed from the driving inability conditions in the ODD relaxed driving mode. That is, in this ODD relaxed driving mode, vehicle 200 can continue driving even during an earthquake. Furthermore, although the conditions for ODD are relaxed, the constraints in this ODD relaxed driving mode are more stringent than those in the automatic driving mode. Regarding vehicle conditions, the maximum speed is limited from 20 km / h to 10 km / h. Figure 28B In the example, the maximum speed limit was strengthened.

[0265] Furthermore, as a condition for safety monitoring, a dual check of vehicle 200 and remote operator H was added to the monitored area. That is, in order to drive in this ODD relaxed driving mode, remote operator H must also monitor their own lane. Figure 28B In the example, the restrictions on the surveillance area were strengthened.

[0266] In satisfying Figure 28B Under the constraints shown, vehicle 200 can automatically drive even during an earthquake under the monitoring of a remote operator H. Accordingly, for example, vehicle 200 can take shelter in a safe location during an earthquake.

[0267] Next, for the fourth applicable example, refer to Figures 29 to 30B Please provide an explanation. Figure 29 This is a diagram used to illustrate the situation of the fourth applicable example of the relaxed ODD. Figure 30A This is a diagram illustrating various conditions in the automatic driving mode of the fourth applicable example. Figure 30B This is a diagram illustrating various conditions in the ODD relaxed driving mode of the fourth applicable example. Additionally, in Figure 29 In the image, dotted shadows are used to indicate the icy areas on road L.

[0268] As a fourth applicable example, we will explain the situation where the driving-restricted area status of the ODD is relaxed. The driving-restricted area status is the road surface condition that prohibits the vehicle 200 from driving. Specifically, we will explain the situation where the road surface is icy, and the driving-restricted area status in the automatic driving mode includes the situation of icing.

[0269] like Figure 29 As shown, if the road surface in the direction of travel is icy, since the driving inaccessible area of ​​ODD includes icy road surface, vehicle 200 will stop.

[0270] like Figure 30A As shown, in automatic driving mode, the driving-in-progress areas in ODD include icy roads and flooded roads (more than 10cm). In other words, in automatic driving mode, vehicle 200 cannot drive on icy roads.

[0271] Therefore, the remote control device 130 switches from automatic driving mode to ODD relaxed driving mode. Specifically, the remote control device 130 switches to ODD relaxed driving mode, which allows driving even on icy roads. The remote control device 130, in Figure 4 In step S103, the ODD relaxed driving mode that can be driven even on icy roads is retrieved from the multiple ODD relaxed driving modes stored in the storage unit 167. Figure 30B As shown, for example, the various conditions in the ODD relaxed driving mode retrieved in step S103.

[0272] like Figure 30B As shown, in the "No Driving Area" state of the ODD relaxed driving mode, the condition of icy road surface has been removed. That is, in this ODD relaxed driving mode, vehicle 200 can drive even on icy roads. Furthermore, although the conditions for ODD are relaxed, the constraints in this ODD relaxed driving mode are stricter than those in the automatic driving mode. Regarding vehicle conditions, the maximum speed is limited from 20 km / h to 5 km / h, and the maximum acceleration is limited from 0.3G to 0.1G. Figure 30B In the example, the limits on maximum speed and maximum acceleration were strengthened.

[0273] Furthermore, regarding the conditions for safety monitoring, a dual check is added to the monitored area for both vehicle 200 and remote operator H. In other words, in order to drive in this ODD relaxed driving mode, remote operator H must also monitor their own lane. Figure 30B In the example, the restrictions on the surveillance area were strengthened.

[0274] In satisfying Figure 30B Under the constraints shown, vehicle 200 can automatically drive at low speed even when the road surface is icy, under the monitoring of remote operator H.

[0275] Next, for the fifth applicable example, refer to Figures 31 to 32C Please provide an explanation. Figure 31 This is a diagram used to illustrate the situation of the fifth applicable example of the relaxed ODD. Figure 32A This is a diagram illustrating various conditions in the automatic driving mode of the fifth applicable example. Figure 32B This is a diagram illustrating various conditions in the first ODD relaxed driving mode of the fifth applicable example. Figure 32C This is a diagram illustrating various conditions in the second ODD relaxed driving mode of the fifth applicable example. Additionally, Figure 31 (a), and Figure 27 same.

[0276] As a fifth applicable example, we will explain the case where the ODD is further relaxed during driving in the ODD relaxed driving mode. That is, as a fifth applicable example, we will explain the case where the ODD is relaxed in stages. Specifically, we will explain the following situation: an earthquake occurs while driving in automatic driving mode, so we switch to the ODD relaxed driving mode (first ODD relaxed driving mode) that allows driving even in the event of an earthquake. While driving in this ODD relaxed driving mode, because vehicle 440 is parked on the side of the road, we further switch to the ODD relaxed driving mode (second ODD relaxed driving mode) that allows driving even in the event of an earthquake and because vehicle 440 is parked on the side of the road.

[0277] First, let me explain the situation where an earthquake occurs while driving in automatic mode.

[0278] like Figure 31 As shown in (a), since the driving conditions of the ODD include the occurrence of an earthquake (magnitude: 3 or higher), if an earthquake of magnitude 3 or higher occurs, vehicle 200 stops. At this time, by switching to the ODD relaxed driving mode, which allows driving even in the event of an earthquake of magnitude 3 or higher, vehicle 200 can drive.

[0279] like Figure 32A As shown, in automatic driving mode, the conditions under which the ODD (Optical Distributor) cannot drive include earthquakes (magnitude 3 or higher), heavy rain (rainfall 200mm or more), dense fog (visibility less than 60m), and strong winds (wind speed 15m / s or higher). In other words, in automatic driving mode, the vehicle cannot drive in the event of an earthquake of magnitude 3 or higher. Furthermore, in automatic driving mode, there are no designated objects among the objects that the ODD can detect.

[0280] Therefore, the remote operating device 130 switches from the automatic driving mode to the ODD relaxed driving mode. Regarding this driving mode switch, since it is the same as in the third applicable example, a simplified explanation will be provided. Assume that the remote operating device 130, in... Figure 4 In step S103, the following was retrieved: Figure 32B The first ODD relaxed driving mode is shown.

[0281] like Figure 32B As shown, earthquakes have been removed from the driving inability conditions of the first ODD relaxed driving mode; therefore, driving is possible even in the event of an earthquake. Furthermore, the constraints in the first ODD relaxed driving mode are more stringent than those in the automatic driving mode. Figure 32B In the example, the limits on maximum vehicle speed and monitored area were strengthened.

[0282] In satisfying Figure 32BUnder the constraints shown, vehicle 200 can drive automatically even in the event of an earthquake, under the monitoring of remote operator H.

[0283] Next, it will be explained that when driving in the first ODD relaxed driving mode, a vehicle 440 that is parked on the side of the road appears in front.

[0284] like Figure 31 As shown in (b), vehicle 200 has vehicle 440 parked on the side of the road ahead. However, vehicle 440 is not included in the objects that can be overtaken in the ODD. Therefore, vehicle 440 cannot be overtaken in the first ODD relaxed driving mode.

[0285] like Figure 32B As shown, in the first relaxed driving mode of the ODD, the objects that the ODD exceeds do not include vehicle 440 parked on the side of the road. That is to say, in the first relaxed driving mode, vehicle 200 cannot exceed vehicle 440 parked on the side of the road.

[0286] Therefore, the remote control device 130 switches from the first ODD relaxed driving mode to the second ODD relaxed driving mode. Specifically, the remote control device 130 switches to an ODD relaxed driving mode that allows it to overtake vehicles parked on the side of the road. The remote control device 130, in... Figure 4 In step S103, the ODD relaxed driving mode that can be driven even in the event of an earthquake and can overtake vehicles parked on the side of the road is retrieved from the multiple ODD relaxed driving modes stored in the storage unit 167. Figure 32C As shown, for example, the various conditions in the ODD relaxed driving mode retrieved in step S103.

[0287] like Figure 32C As shown, in the second ODD relaxed driving mode, an additional vehicle (parked on the side of the road) is added to the list of objects that the ODD exceeds. That is, in this second ODD relaxed driving mode, vehicle 200 can overtake vehicle 440 parked on the side of the road in the event of an earthquake.

[0288] Furthermore, although the conditions for the ODD are relaxed, the constraints in the relaxed driving mode of this second ODD are stricter than those in the relaxed driving mode of the first ODD. Regarding vehicle conditions, the maximum speed is limited from 10 km / h to 5 km / h. Figure 32C In the example, the maximum speed limit is further strengthened.

[0289] Furthermore, regarding the conditions for safety monitoring, a vehicle ahead is added to the list of objects requiring remote monitoring. That is, in order to drive in the relaxed driving mode of this second ODD, the remote operator H needs to monitor the vehicle ahead. Figure 32C In the example, restrictions on objects that need to be remotely monitored are strengthened.

[0290] In satisfying Figure 32C Under the constraints shown, under the monitoring of remote operator H, vehicle 200 is able to automatically drive past vehicle 440 parked on the side of the road in the event of an earthquake.

[0291] Therefore, the higher the relaxation level of the ODD, the stricter the constraints.

[0292] The remote operating device 130 can, for example, switch from a second ODD relaxed driving mode to a third ODD relaxed driving mode, which further relaxes the ODD, based on the conditions around the vehicle 200. Thus, the remote operating device 130 can also periodically change the ODD relaxation level based on the driving environment of the vehicle 200. Even when changing the ODD relaxation level, the processing in step S106 is performed. That is, when switching from the automatic driving mode to the first ODD relaxed driving mode, and when switching from the first ODD relaxed driving mode to the second ODD relaxed driving mode, the remote operator H is received with an instruction indicating that the switching of the driving mode is permitted.

[0293] Furthermore, the examples described above illustrate instances where system condition restrictions are not strengthened, but the system is not limited to these. System condition restrictions can also be strengthened by relaxing the ODD. For example, communication latency restrictions during autonomous driving can be strengthened by relaxing the ODD. Specifically, the permissible communication latency value can be changed to a smaller value.

[0294] Furthermore, the restrictions related to the processing load of the remote operating system 100 can be strengthened by relaxing the ODD. For example, the threshold (e.g., upper limit) of the processing load of the remote operating system 100 can be reduced by relaxing the ODD. The processing load, for example, includes at least one of the processing volume of the remote operating system 100 and the processing latency (e.g., the processing time of the remote operating device 130). Strengthening the restrictions related to the processing load of the remote operating system 100 is an example of strengthening the constraints on the processing related to the monitoring of the remote operator H.

[0295] Furthermore, relaxing ODD can also mean relaxing at least one of the vehicle conditions. Relaxing ODD can also mean, for example, relaxing the maximum steering angle, i.e., making sharp steering wheel turns possible. Furthermore, relaxing ODD can also mean relaxing the weather conditions under which ODD prohibits driving. Relaxing ODD can also mean, for example, removing driving restrictions in rainy weather, or changing the precipitation level that determines driving is prohibited in rainy weather to a higher value.

[0296] Furthermore, in automated driving with ODD relaxed driving mode, the driving range can also be limited. For example, the system conditions can be set for the time, interval, or distance during which the remote operator H presses button 115 to continue driving in ODD relaxed driving mode. Moreover, the time, interval, or distance can be changed to a smaller value according to the relaxed ODD settings to further restrict the driving range.

[0297] (Modifications of the implementation method)

[0298] The following refers to the vehicle control system involved in this variation. Figure 33 Please provide an explanation. Figure 33 This is a block diagram illustrating the functional structure of the vehicle control system 10a according to this modification. The main difference between the vehicle control system 10a according to this modification and the vehicle control system 10 according to the embodiment is that the vehicle 200a has multiple automatic driving systems. Hereinafter, the vehicle control system 10a according to this modification will be described focusing on the differences between it and the vehicle control system 10 according to the embodiment. Furthermore, structures that are the same as or similar to those in the vehicle control system 10 according to the embodiment will be given the same reference numerals, and descriptions will be omitted or simplified.

[0299] like Figure 33 As shown, the vehicle control system 10a includes a remote operation device 130a and a vehicle 200a.

[0300] Vehicle 200a has an automatic driving system 260, replacing the automatic driving system 230 of vehicle 200 in the embodiment. The automatic driving system 260 is configured to include multiple automatic driving systems.

[0301] The autonomous driving system 260 involved in this variation has a first autonomous driving system 230a and a second autonomous driving system 230b.

[0302] The first autonomous driving system 230a is a system that operates when driving in autonomous driving mode, and includes a first driving feasibility determination unit 231a and a first autonomous driving unit 232a.

[0303] The first driving feasibility determination unit 231a performs a determination on whether the automatic driving mode is feasible. The first driving feasibility determination unit 231a stores the constraints of the automatic driving mode, and determines whether the automatic driving mode is feasible based on the constraints and vehicle information.

[0304] The first automated driving unit 232a generates a driving plan for an automated driving mode based on vehicle information. The driving plan includes the driving route, speed, etc. Furthermore, if the driving continuation determination unit 164 determines that driving can continue, the first automated driving unit 232a outputs control information based on the driving plan to the vehicle control unit 240 in order to execute the generated driving plan.

[0305] The second autonomous driving system 230b is a system that operates when driving in the ODD relaxed driving mode, and includes a second driving permission determination unit 231b and a second autonomous driving unit 232b.

[0306] The second driving permission determination unit 231b performs a driving permission determination for the ODD relaxed driving mode. The second driving permission determination unit 231b stores the constraints of the ODD relaxed driving mode, and determines whether the ODD relaxed driving mode is driving permission based on the constraints and vehicle information.

[0307] The second automated driving unit 232b generates a driving plan with relaxed ODD driving mode based on vehicle information. The driving plan includes driving route, speed, etc. Furthermore, if the driving continuation determination unit 164 determines that driving can continue, the second automated driving unit 232b outputs control information based on the driving plan to the vehicle control unit 240 in order to execute the generated driving plan.

[0308] In addition, it was explained Figure 33 The example of the autonomous driving system 260 has two autonomous driving systems, but there is no particular limit to the number of autonomous driving systems, and it is also possible to have more than three autonomous driving systems.

[0309] The mode-changing unit 151 outputs a signal to the vehicle 200a to activate the autonomous driving system corresponding to the driving mode to be switched. For example, when driving in automatic driving mode, the mode-changing unit 151 outputs a signal to the vehicle 200a to activate only the first autonomous driving system 230a among the multiple autonomous driving systems. Furthermore, when driving in ODD relaxed driving mode, the mode-changing unit 151 outputs a signal to the vehicle 200a to activate only the second autonomous driving system 230b among the multiple autonomous driving systems. Thus, among the multiple autonomous driving systems included in the autonomous driving system 260, the mode-changing unit 151 activates only the autonomous driving system corresponding to the switched driving mode.

[0310] Furthermore, the safety judgment unit 160a of the remote operator H may not have a storage unit that stores the ODD of the relaxed driving mode and the constraint conditions, etc.

[0311] Additionally, the remote operating device 130a may also have a storage unit (not shown) that stores information indicating which driving mode each of the multiple automatic driving systems of the vehicle 200a corresponds to.

[0312] As described above, vehicle 200a can also be structured as an autonomous driving system with each driving mode. For example, in the case of multiple ODD relaxed driving modes, an autonomous driving system is set up for each of the multiple ODD relaxed driving modes.

[0313] Accordingly, the mode-changing unit 151 only needs to send information related to the autonomous driving system to be operated to the vehicle 200a, thereby reducing the amount of communication between the vehicle 200a and the remote operating device 130a. Furthermore, the processing of changing settings related to the driving feasibility determination unit can be omitted, thus shortening the time required to switch driving modes. Therefore, compared to situations where time is required for switching, the safety of the vehicle 200a during driving mode switching can be improved.

[0314] (Other implementation methods)

[0315] The present disclosure has been described above based on embodiments and variations (hereinafter also referred to as embodiments, etc.). However, the present disclosure is not limited to the embodiments described above. Various modifications that can be conceived by those skilled in the art, or combinations of different components of embodiments, may be implemented in accordance with the spirit of the present disclosure and may be included within the scope of one or more variations.

[0316] For example, the above-described embodiments illustrate an example where the constraints are strengthened by relaxing the ODD, but this is not the only possibility. Depending on the degree of relaxation of the ODD, if a remote operator is performing remote monitoring, the constraints may not need to be strengthened.

[0317] Furthermore, in the above-described embodiments, when the constraints are strengthened according to the relaxation of ODD, it is sufficient to strengthen at least one of the constraints: the vehicle condition, the safety monitoring condition, and the system condition.

[0318] Furthermore, while the above embodiments describe an example where a remote operator presses a button to continue driving in the ODD relaxed driving mode, the method is not limited to this. For example, the decision to continue driving in the ODD relaxed driving mode can be based on the remote operator's posture during monitoring. The driving continuation determination unit can also, for example, determine whether to continue driving in the ODD relaxed driving mode based on the remote operator's gaze. Accordingly, if the remote operator is not looking at the display screen during driving in the ODD relaxed driving mode, the safety monitoring conditions that satisfy the constraints are not met, and therefore, driving in the ODD relaxed driving mode can be automatically stopped.

[0319] Furthermore, while the above embodiments illustrate an example of remote monitoring by a remote operator during ODD relaxed driving mode, the method is not limited to this. The remote operator can also perform some driving operations during ODD relaxed driving mode. For example, the remote operator can control the speed of the vehicle in ODD relaxed driving mode by operating the accelerator and brake pedals. In this case, the burden on the remote operator can be reduced compared to driving in remote operation mode.

[0320] Furthermore, while examples of display devices displaying overhead images have been described in the above embodiments, the display device is not limited to these. For example, the display device may also display 360-degree images, or it may re-synthesize images to display third-person viewpoint images. Additionally, the display device may display object information detected by sensors mounted on the vehicle (e.g., object detection sensors). The object information includes at least one of the following: object position, size, speed, etc.

[0321] Furthermore, the order of the multiple processes described in the above embodiments is just one example. The order of the multiple processes can be changed, and the multiple processes can be executed in parallel. Also, it is possible to omit some of the multiple processes.

[0322] Furthermore, the various components described in the above embodiments can be implemented as software or as typical integrated circuits, i.e., LSIs. These can be individually implemented on a single chip, or partially or entirely on a single chip. Here, we refer to it as a system LSI; however, depending on the level of integration, it may be called an IC, system LSI, super LSI, or extra-large LSI. Moreover, the method of integrated circuit implementation is not limited to LSIs; it can also be implemented using dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that can reconfigure the connections or settings of the circuit units within an LSI, can also be used. Furthermore, if advancements in semiconductor technology or other derived technologies lead to integrated circuit technologies that replace LSIs, then these technologies can certainly be used for the integration of components.

[0323] Furthermore, taking the division of functional blocks in the block diagram as an example, multiple functional blocks can be implemented as a single functional block, and a single functional block can also be divided into multiple functional blocks. A portion of the functionality can also be transferred to other functional blocks. Moreover, the functions of multiple functional blocks with similar capabilities can be processed in parallel or time-divided by a single piece of hardware or software.

[0324] Furthermore, the remote operation device included in the vehicle control system can be implemented as a single device or by multiple devices. For example, the processing units of the remote operation device can be implemented by two or more devices. For example, the remote operation unit and the safety judgment unit can be implemented by different devices (e.g., server devices). When the remote operating system is implemented by multiple devices, the components of the remote operating system can be distributed among the multiple devices in any way. Moreover, there are no particular restrictions on the communication methods between the multiple devices.

[0325] Furthermore, the technology disclosed herein can also be the aforementioned program, or a non-transitory computer-readable recording medium on which the aforementioned program is recorded. And, of course, the aforementioned program can be distributed via transmission media such as the Internet. For example, the aforementioned program and the digital signals constituted by the aforementioned program can also be transmitted via electrical communication lines, wireless or wired communication lines, networks such as the Internet, data broadcasting, etc. Furthermore, the aforementioned program and the digital signals constituted by the aforementioned program can also be recorded on a recording medium for transmission, or transmitted via a network, etc., so that they can be executed by other independent computer systems.

[0326] Furthermore, in each embodiment, each component can be constructed by dedicated hardware or implemented by executing software programs suitable for each component. Alternatively, each component can be implemented by a program execution unit such as a CPU or processor reading and executing software programs recorded on a recording medium such as a hard disk or semiconductor memory.

[0327] This disclosure can be widely used in systems that utilize mobile bodies capable of autonomous driving.

[0328] Symbol Explanation

[0329] 10, 10a Vehicle control system (information processing system)

[0330] 100 Remote Operating Systems

[0331] 110 display device

[0332] Buttons 112, 113, 114, 115

[0333] 120 Operation Input Device

[0334] 130, 130a Remote Operating Device

[0335] 140, 210 Ministry of Communications

[0336] 150 Remote Operations Department

[0337] 151 Model Change Department

[0338] 152 Reminder Department

[0339] 153 Command Receiving Department (Detection Department)

[0340] 154 Operation Receiving Unit

[0341] 155 Operational Information Acquisition Department

[0342] Safety Judgment Department 160, 160a

[0343] 161 Area Judgment Department

[0344] 162 Relaxing the Judgment Department

[0345] 163 Remote Operation Capability Assessment Department

[0346] 164. Determination of whether driving can continue.

[0347] 165 Rear Section

[0348] 166 System Information Acquisition Department

[0349] 167 Storage Department

[0350] Vehicles 200, 200a, 400, 410, 420, 430, 440

[0351] 220 Vehicle Information Acquisition Department

[0352] 230, 260 autonomous driving systems

[0353] 230a First Automated Driving System

[0354] 230b Second Automated Driving System

[0355] 231 Driving Permission Determination Department

[0356] 231a First Driving Permission Determination Section

[0357] 231b Second Driving Permission Determination Section

[0358] 232 Autonomous Driving Department

[0359] 232a First Automated Driving Section

[0360] 232b Second Automated Driving Unit

[0361] 240 Vehicle Control Department

[0362] 250 Instruction Acquisition Unit

[0363] 300 Network

[0364] 310 Wireless Base Station

[0365] H Remote Operator

[0366] L Road

Claims

1. An information processing method, which is an information processing method in an information processing device. In the information processing method, The system obtains first information, which indicates whether a mobile body capable of autonomous driving based on a first condition related to its safety can continue to drive autonomously, wherein the mobile body is connected to the information processing device via a network. If the first information indicates that the mobile body cannot continue to drive automatically, a first instruction is output to the mobile body to enable it to operate in a first mode. This first mode is a mode in which the mobile body is remotely monitored but not operated remotely by the remote operator, and the mobile body is then driven automatically according to a second condition. This second condition is a condition related to the safety of the mobile body that is relaxed compared to the first condition. When the mobile vehicle is moving automatically, a first monitoring condition is set, which indicates the monitoring conditions when the remote operator monitors the mobile vehicle. When the mobile body operates in the first mode, a second monitoring condition is set according to the second condition, which strengthens the monitoring compared to the first monitoring condition. The monitoring conditions when the enhanced monitoring is displayed on the display device used by the remote operator. The mobile body is made to drive automatically under the second condition only during the period when it is monitored by the remote operator under the enhanced monitoring conditions.

2. The information processing method as described in claim 1, When the mobile body operates in the first mode, the restrictions on the autonomous driving mode of movement are strengthened according to the second condition.

3. The information processing method as described in claim 2, The movement method includes at least one of the following: the speed, the turning angle, and the acceleration of the moving body. According to the second condition, at least one of the following should be restricted during autonomous driving: maximum speed, maximum steering angle, and maximum acceleration.

4. The information processing method as described in claim 1, The monitoring conditions during monitoring include at least one of a monitoring area and a monitored object. The monitoring area is the area surrounding the moving body that needs to be monitored by the remote operator, and the monitored object is the object that needs to be monitored by the remote operator. The enhancement of the monitoring conditions during monitoring includes, according to the second condition, enhancing the monitoring of at least one of the monitoring area and the monitoring object.

5. The information processing method as described in claim 1, The monitoring conditions include the operation of the mobile body that needs to be monitored by the remote operator. The enhanced monitoring conditions during monitoring include, according to the second condition, adding monitoring of the vehicle's travel plan or travel control information.

6. The information processing method as described in claim 1, The period during which the enhanced monitoring is conducted under the monitoring conditions is the period during which the operation performed by the remote operator indicates the continuation of the automatic driving under the second condition.

7. The information processing method as described in claim 1, When the mobile body is operated in the first mode, the monitoring conditions for delays in processing or communication related to the monitoring of the remote operator are strengthened in accordance with the second condition.

8. The information processing method according to any one of claims 1 to 7, If the first information indicates that the mobile body cannot continue to drive automatically, the first instruction or a second instruction for making the mobile body work in a second mode is output, the second mode being the mode in which the remote operator remotely operates the mobile body.

9. The information processing method as described in claim 8, If, based on the first information, it is determined that the mobile unit cannot continue to move automatically, the display device used by the remote operator displays second information. This second information is used to switch the operation of the mobile unit to either the first mode or the second mode. Based on the information input by the remote operator in response to the second information, an instruction is output to cause the mobile body to operate in either the first mode or the second mode.

10. The information processing method according to any one of claims 1 to 7, When the mobile vehicle drives automatically according to the second condition, the display device used by the remote operator displays an image, on which the driving area of ​​the mobile vehicle and the area of ​​the driving area that needs to be monitored by the remote operator are superimposed in a form that can be visually confirmed.

11. The information processing method as described in claim 10, Furthermore, the display device displays at least one of the information related to the movement path of the moving body and the judgment result of whether it can move in the current working mode.

12. The information processing method according to any one of claims 1 to 7, The first piece of information is determined based on the first condition and information related to the state or specifications of the moving body.

13. An information processing system, comprising: The acquiring unit acquires first information, which indicates whether a mobile body that is automatically driving according to a first condition related to the safety of the mobile body can continue to drive automatically, wherein the mobile body is connected to an information processing device via a network; and The output unit, upon receiving the first information indicating that the mobile body cannot continue autonomous driving, outputs a first instruction to the mobile body to operate in a first mode. This first mode is a mode in which the mobile body is remotely monitored rather than remotely operated by a remote operator, and the mobile body is automatically driven according to a second condition. This second condition is a condition that relaxes the safety-related conditions of the mobile body compared to the first condition. In the information processing system, When the mobile vehicle is moving automatically, a first monitoring condition is set, which indicates the monitoring conditions when the remote operator monitors the mobile vehicle. When the mobile body operates in the first mode, a second monitoring condition is set according to the second condition, which strengthens the monitoring compared to the first monitoring condition. The monitoring conditions when the enhanced monitoring is displayed on the display device used by the remote operator. The mobile body is made to drive automatically under the second condition only during the period when it is monitored by the remote operator under the enhanced monitoring conditions.

Images