Information processing method and information processing system

Abstract

An information processing method executed by a computer, wherein information pertaining to a state of a mobile body capable of automatic driving and remote manual driving is acquired (S20), and when a changeover from the automatic driving to the remote manual driving occurs in the mobile body (No in step S30), first operation control, which is of the operation control of the mobile body and is carried out by the remote manual operation, is determined on the basis of the information pertaining to the state of the mobile body, the mobile body is operated on the basis of the remote manual operation corresponding to the first operation control, and the mobile body is made to execute second operation control other than the first operation control (S60).

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method and an information processing system. [Background technology]

[0002] In recent years, remote control systems have been studied in which a remote operator in a remote location indirectly controls an autonomously driven vehicle as needed using wireless communication such as a wireless local area network (LAN) or a mobile phone line. For example, when the vehicle reaches a situation where it cannot drive autonomously, the remote operator controls the vehicle's driving from a remote location by transmitting a control signal related to the vehicle's driving to the vehicle.

[0003] For example, Patent Document 1 discloses a semi-autonomous driving system (remote control system) that changes the allocation of driving decision-making between the autonomous driving function and remote control based on driving environment conditions and communication environment conditions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5506423 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with a remote control system such as that described in Patent Document 1, it is difficult to reduce the burden of remote control on the remote operator. For example, if it is determined that the remote operator will perform remote control, the remote operator is expected to perform all actual vehicle operations. Therefore, with the technology described in Patent Document 1, it is difficult to reduce the burden of remote control itself.

[0006] Therefore, the present disclosure provides an information processing method and an information processing device that can reduce the burden on a remote operator who operates a mobile object from a remote location while ensuring the safety of the mobile object. [Means for solving the problem]

[0007] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which acquires information regarding the state of a mobile body capable of automatic driving and remote manual driving, and when a switch from automatic driving to remote manual driving occurs in the mobile body, determines a first operation control to be remotely manually operated among the operation controls of the mobile body based on information regarding the state of the mobile body, operates the mobile body based on the remote manual operation corresponding to the first operation control, and causes the mobile body to execute a second operation control other than the first operation control.

[0008] An information processing system according to one embodiment of the present disclosure includes an acquisition unit that acquires information regarding the state of a mobile body capable of automatic driving and remote manual driving; a determination unit that, when switching from automatic driving to remote manual driving occurs in the mobile body, determines a first operation control to be remotely manually operated among the operation controls of the mobile body based on information regarding the state of the mobile body; and a switching instruction unit that operates the mobile body based on the remote manual operation corresponding to the first operation control and causes the mobile body to execute a second operation control other than the first operation control. [Effects of the Invention]

[0009] According to an information processing method and the like according to one aspect of the present disclosure, it is possible to reduce the burden on a remote operator who operates a mobile object from a remote location while ensuring the safety of the mobile object. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a vehicle control system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of the vehicle control system according to the embodiment. [Figure 3] FIG. 3 is a diagram showing an example of vehicle control allocation in each driving mode. [Figure 4]FIG. 4 is a diagram showing an example of driving conditions in each driving mode. [Figure 5] FIG. 5 is a flowchart showing the operation of the vehicle control system according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the details of the running mode switching process shown in FIG. [Figure 7] FIG. 7 is a schematic diagram for explaining switching to the accelerator manual driving mode. [Figure 8] FIG. 8 is a block diagram showing a functional configuration of a vehicle control system according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Background to this disclosure) For example, when a vehicle is unable to autonomously drive, a remote operator controls the vehicle's driving from a remote location by transmitting a control signal related to the vehicle's driving to the vehicle. At this time, the remote operator must perform remote control under conditions such as communication delays and limited field of view, which can result in poor operability. For example, the remote operator may not be able to perform remote control safely and efficiently depending on the communication state, vehicle state, vehicle location, movement task, and conditions around the vehicle. The communication state may be, for example, communication delay or bandwidth. The vehicle state may be, for example, a malfunction or speed. The movement task may be, for example, turning right or left or parking. The conditions around the vehicle may be, for example, the degree of congestion or the positional relationship with surrounding vehicles. In other words, in the past, the burden of remote control on the remote operator may be heavy. Furthermore, even when remotely controlled, the vehicle is required to drive safely. Note that when the moving body is a vehicle, movement may also be referred to as driving, and the movement task may also be referred to as a driving task.

[0012] Therefore, the inventors of the present application have conducted extensive research into information processing methods and information processing devices that can reduce the burden on remote operators in remote locations while ensuring the safety of the vehicle, and have discovered that by appropriately dividing vehicle control by the autonomous driving function and vehicle control by the remote operator according to the various situations described above, it is possible to reduce the burden on the remote operator and ensure safety throughout the entire system.

[0013] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which acquires information regarding the state of a mobile body capable of automatic driving and remote manual driving, and when a switch from automatic driving to remote manual driving occurs in the mobile body, determines a first operation control to be remotely manually operated among the operation controls of the mobile body based on information regarding the state of the mobile body, operates the mobile body based on the remote manual operation corresponding to the first operation control, and causes the mobile body to execute a second operation control other than the first operation control.

[0014] As a result, when switching from automatic driving to remote manual driving occurs, the automatic driving function and the remote operator can share the operational control of the moving object, thereby reducing the burden on the remote operator compared to when the remote operator performs all operational control. Also, since the first operational control and the second operational control are determined based on the state of the moving object, it becomes easier to ensure safety in the movement of the moving object. Therefore, according to the information processing method, it is possible to reduce the burden on the remote operator who operates the moving object from a remote location while ensuring the safety of the moving object.

[0015] Also, for example, the information regarding the state of the mobile body may be communication information indicating the state of communication between the mobile body and a remote control device for remotely controlling the mobile body via communication.

[0016] As a result, the first operation control and the second operation control are determined based on the state of communication, which makes it easier to ensure safety during movement of the moving body.

[0017] Furthermore, for example, the first operation control may be determined based on a correspondence relationship between the state of the communication and the operation control of the moving object for which the remote manual operation is permitted.

[0018] Thus, according to the information processing method, by setting the correspondence relationship in advance, the first operation control can be easily determined simply by acquiring the communication state.

[0019] Furthermore, for example, in determining the first operation control, the better the state of the communication is, the more the first operation control that is determined such that the permitted remote manual operations increase, become more complex, or become more advanced.

[0020] As a result, when remote operation can be performed more safely from the viewpoint of the communication state, the first operation control that is increased, complicated, or advanced can be determined as the remote manual operation by the remote operator. Therefore, according to the information processing method, it is possible to limit the control operations that can be controlled by the autonomous driving function and the remote operator depending on the communication state, thereby ensuring safety according to the communication state and reducing the burden on the remote operator.

[0021] Furthermore, for example, the communication status may include a communication delay or a communication band status between the mobile object and the remote control device.

[0022] As a result, according to the information processing method, by simply obtaining the communication delay or communication bandwidth, it is possible to determine a first operation control and a second operation control that can reduce the burden on a remote operator in a remote location while ensuring the safety of the mobile body.

[0023] Furthermore, for example, sensing information indicating a sensing state of the periphery of the moving object may be acquired, and the first operation control may be determined based on the communication information and the sensing information.

[0024] This allows the control operations that can be controlled by the autonomous driving function and the remote operator to be determined according to the sensing state, and therefore the first operation control and the second operation control that can ensure safety in the movement of the mobile object and reduce the burden on the remote operator can be more appropriately determined. Therefore, according to the information processing method, the burden on the remote operator in a remote location can be reduced while ensuring the safety of the mobile object.

[0025] Furthermore, for example, when determining the first operation control, the first operation control may be determined based on the correspondence between the communication state and the sensing state and the operation control of the mobile body for which the remote manual operation is permitted, such that the better the sensing state, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced.

[0026] As a result, when remote operation can be performed more safely from the viewpoint of the sensing state, an increased, complicated, or advanced first operation control can be determined as the control operation of the remote operator. Therefore, according to the information processing method, the control operations that can be controlled by the autonomous driving function and the remote operator can be limited according to the communication state and the sensing state, thereby ensuring safety according to the communication state and the sensing state and reducing the burden on the remote operator.

[0027] Furthermore, for example, the sensing state may include processing delays or missing of image data, sound data, or point cloud data.

[0028] As a result, according to the information processing method, more appropriate first and second action controls can be determined simply by further acquiring processing delays or missing pieces of image data, sound data, or point cloud data.

[0029] Furthermore, for example, location information indicating the location of the moving object may be further acquired, and the first operation control may be determined based on the communication information and the location information.

[0030] This allows the control operations that can be controlled by the autonomous driving function and the remote operator to be determined according to the location of the moving body, and therefore the first operation control and the second operation control that can ensure the safety of the moving body and reduce the burden on the remote operator can be more appropriately determined. Therefore, according to the information processing method, the burden on the remote operator in a remote location can be reduced while ensuring the safety of the moving body.

[0031] Furthermore, for example, when determining the first operation control, the first operation control may be determined based on the correspondence between the communication state and the location of the mobile body and the operation control of the mobile body for which remote manual operation is permitted, and the lower the safety of the autonomous driving of the mobile body at the location of the mobile body, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced.

[0032] As a result, when the safety of the autonomous driving of the mobile body is low from the viewpoint of the safety of the mobile body at its location, it is possible to determine an increased, complicated, or advanced first operation control as the control operation of the remote operator. Therefore, according to the information processing method, it is possible to limit the control operations that can be controlled by the autonomous driving function and the remote operator depending on the communication state and the location of the mobile body, thereby ensuring safety depending on the communication state and the location of the mobile body and reducing the burden on the remote operator.

[0033] Furthermore, for example, the location of the mobile object may include a position of the mobile object or a position on a moving route of the mobile object.

[0034] As a result, according to the information processing method, it is possible to determine more appropriate first and second movement controls simply by further acquiring the position of the moving body or a position on the moving route of the moving body.

[0035] Furthermore, for example, movement state information indicating a movement state of the moving object may be further acquired, and the first operation control may be determined based on the communication information and the movement state information.

[0036] This makes it possible to determine the control operations that can be controlled by the autonomous driving function and the remote operator according to the moving state of the moving body, and therefore it is possible to more appropriately determine the first operation control and the second operation control that can ensure the safety of the moving body and reduce the burden on the remote operator. Therefore, according to the information processing method, it is possible to further reduce the burden on the remote operator who operates the moving body from a remote location while ensuring the safety of the moving body.

[0037] Furthermore, for example, when determining the first operation control, the first operation control may be determined based on the correspondence between the communication state and the movement state of the mobile body and the operation control of the mobile body for which remote manual operation is permitted, such that the higher the safety of the mobile body in the movement state of the mobile body, the more increased, complex, or advanced the remote manual operation that is permitted.

[0038] As a result, when the safety of the mobile body is high from the viewpoint of the safety of the moving state of the mobile body, it is possible to determine the increased, complicated, or advanced first operation control as the control operation of the remote operator. Therefore, according to the information processing method, it is possible to limit the control operations that can be controlled by the automatic driving function and the remote operator depending on the communication state and the moving state of the mobile body, thereby ensuring safety depending on the communication state and the moving state of the mobile body and reducing the burden on the remote operator.

[0039] Furthermore, for example, the moving state of the moving object may include the speed of the moving object.

[0040] As a result, according to the information processing method, more appropriate first and second movement controls can be determined simply by further acquiring the speed of the moving object.

[0041] Furthermore, for example, environmental information indicating the surrounding environment of the moving object may be acquired, and the first operation control may be determined based on the communication information and the environmental information.

[0042] This makes it possible to determine the control operations that can be controlled by the autonomous driving function and the remote operator according to the surrounding environment of the moving body, and therefore to more appropriately determine the first operation control and the second operation control that can ensure the safety of the moving body during movement and reduce the burden on the remote operator. Therefore, according to the information processing method, it is possible to further reduce the burden on the remote operator in a remote location while ensuring the safety of the moving body.

[0043] Furthermore, for example, when determining the first operation control, the first operation control may be determined based on the correspondence between the communication state and the surrounding environment of the mobile body and the operation control of the mobile body for which remote manual operation is permitted, and the lower the safety of the autonomous driving of the mobile body in the surrounding environment of the mobile body, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced.

[0044] As a result, when the safety of the autonomous driving of the mobile body is low from the viewpoint of the safety of the surrounding environment of the mobile body, it is possible to determine the increased, complicated, or advanced first operation control as the control operation of the remote operator. Therefore, according to the information processing method, it is possible to limit the control operations that can be controlled by the autonomous driving function and the remote operator depending on the communication state and the surrounding environment of the mobile body, thereby ensuring safety depending on the communication state and the surrounding environment of the mobile body and reducing the burden on the remote operator.

[0045] Furthermore, for example, the surrounding environment of the moving object may include the state of obstacles, the state of the road surface, or the weather.

[0046] As a result, according to the information processing method, more appropriate first and second action controls can be determined simply by further acquiring the state of the obstacle, the state of the road surface, or the weather.

[0047] Furthermore, for example, movement task information indicating a movement task for moving the moving body may be acquired, and the first operation control may be determined based on the communication information and the movement task information.

[0048] This allows the control operations that can be controlled by the autonomous driving function and the remote operator to be determined according to the travel task for travel, and therefore the first and second operation controls that can ensure safety in travel of the moving body and reduce the burden on the remote operator can be more appropriately determined. Therefore, according to the information processing method, the burden on the remote operator who operates the moving body from a remote location can be reduced while ensuring the safety of the vehicle.

[0049] Furthermore, for example, when determining the first operation control, the first operation control may be determined based on the correspondence between the communication state and the movement task and the operation control of the moving body for which the remote manual operation is permitted, such that the higher the complexity of the movement task, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced.

[0050] As a result, when the complexity of the travel task is further increased, an increased, more complicated, or more advanced first motion control can be determined as the control action of the remote operator. Therefore, according to the information processing method, the control actions that can be controlled by the autonomous driving function and the remote operator can be limited depending on the communication state and the permitted travel task, so that it is possible to ensure safety and reduce the burden on the remote operator depending on the communication state and the executable travel task.

[0051] Furthermore, for example, the movement task may include going straight, turning right or left, avoiding parking, backing up, or pulling over to the side of the road.

[0052] As a result, according to the information processing method, more appropriate first and second operation controls can be determined simply by further obtaining whether or not it is possible to go straight, turn right or left, avoid parking on the road, back up, or pull over to the shoulder.

[0053] In addition, an information processing system according to one embodiment of the present disclosure includes an acquisition unit that acquires information regarding the state of a mobile body capable of automatic driving and remote manual driving; a decision unit that, when switching from automatic driving to remote manual driving occurs in the mobile body, determines a first operation control to be remotely manually operated among the operation controls of the mobile body based on information regarding the state of the mobile body; and a switching instruction unit that operates the mobile body based on the remote manual operation corresponding to the first operation control and causes the mobile body to execute a second operation control other than the first operation control.

[0054] This provides the same effect as the above-described information processing method.

[0055] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0056] Specific examples of an information processing method and an information processing system according to one aspect of the present disclosure will be described below with reference to the drawings. Each of the embodiments shown here represents a specific example of the present disclosure. Therefore, the numerical values, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components. Furthermore, the contents of each of the embodiments can be combined.

[0057] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0058] Furthermore, in this specification, numerical values ​​and numerical ranges are not expressions that express only the strict meaning, but are expressions that mean that a substantially equivalent range, for example, a difference of about several percent, is also included.

[0059] (Embodiment) Hereinafter, an information processing method and the like according to this embodiment will be described with reference to FIGS.

[0060] [1. Vehicle control system configuration] First, the configuration of a vehicle control system 10 according to this embodiment will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing a schematic configuration of a vehicle control system 10 according to this embodiment.

[0061] 1, the vehicle control system 10 includes a remote operation system 100, a network 300, a wireless base station 310, and a vehicle 200. The vehicle control system 10 is a system that communicatively connects the vehicle 200 and the remote operation system 100 (for example, a remote operation device 130) via the wireless base station 310, such as a wireless LAN or a communication terminal, and the network 300.

[0062] The wireless base station 310 and the network 300 are an example of a communication network. The vehicle 200 is an example of a moving body that is at least remotely manually operated by a remote operator H. The vehicle 200 is capable of automatic driving and remote manual driving by remote control. The vehicle control system 10 is an example of an information processing system. Hereinafter, remote manual operation will also be simply referred to as remote operation.

[0063] Although details will be described later, the remote operation system 100 switches the driving mode of the vehicle 200 from one of an autonomous driving mode (for example, a fully autonomous driving mode) and a remote control mode to the other. The remote operation modes include a fully remote control mode in which the remote operator H controls all remotely operable operations, as well as a shared driving mode which is an intermediate driving mode between the autonomous driving mode and the fully remote control mode. This embodiment is characterized in that the remote operation system 100 can control the vehicle 200 in the shared driving mode. Remotely operable operations include, for example, the control operations shown in FIG. 3 described later, such as accelerator operation, brake operation, steering operation, and presentation operation, but are not limited to these.

[0064] The autonomous driving mode is a driving mode in which the passenger and the remote operator H are not involved in the vehicle control of the vehicle 200. The autonomous driving mode is also referred to as an automatic driving mode. In the autonomous driving mode, the remote operator H, for example, performs remote monitoring but does not perform remote operation. The complete remote control mode is a driving mode in which the passenger and the automatic driving system are not involved in the vehicle control of the vehicle 200. In the complete remote control mode, the remote operator H, for example, performs all operations that can be remotely operated. The shared driving mode is a driving mode in which both the remote operator H and the automatic driving system are involved in the vehicle control of the vehicle 200. In the shared driving mode, the remote operator H, for example, performs only some of the operations that can be remotely operated (e.g., accelerator operation), and the remaining operations (e.g., braking operation, steering operation, and presentation operation) are performed by the automatic driving system.

[0065] The remote operator H is an example of a monitor who monitors the automatically driven vehicle 200 from a remote location where the monitor cannot directly monitor the vehicle 200. "Not being able to directly monitor" means that the vehicle 200 cannot be seen with the naked eye. In other words, the remote operator H remotely monitors and remotely operates the vehicle 200 from a location that is different from the surroundings of the vehicle 200.

[0066] The configuration of the vehicle control system 10 will be described below.

[0067] The remote operation system 100 is a system that allows a remote operator H at a remote location to remotely monitor and remotely operate the traveling of a vehicle 200 as needed. The remote operation system 100 includes a presentation unit 110, a vehicle operation unit 120, and a remote operation device 130. The presentation unit 110 and the vehicle operation unit 120 constitute an input / output system (see input / output system 100a in FIG. 2). It can also be said that the remote operation system 100 includes the input / output system 100a and the remote operation device 130.

[0068] The presentation unit 110 presents various information to the remote operator H. The presentation unit 110 is connected to the remote control device 130 and includes a display device that displays information related to the vehicle 200. The display device may be, for example, a liquid crystal display device, but is not limited to this. The presentation unit 110 displays information that the remote operator H uses to remotely monitor or remotely control the traveling of the vehicle 200. The presentation unit 110 may, for example, display an image captured by an imaging unit provided in the vehicle 200. The presentation unit 110 may also display information processed by the remote control device 130. The presentation unit 110 may also display buttons that the remote operator H uses to switch the traveling mode of the vehicle 200 or to maintain the traveling mode, using a GUI (Graphical User Interface). The presentation unit 110 may also display information such as a monitoring area that the remote operator H is requested to monitor. Details of the information displayed by the presentation unit 110 will be described later.

[0069] It should be noted that the presenting unit 110 is not limited to presenting various types of information by displaying them, but may also present various types of information by voice, projection, or the like, for example.

[0070] The vehicle operation unit 120 accepts various operations related to the remote operation of the vehicle 200 from the remote operator H. The vehicle operation unit 120 accepts operations for remotely operating the vehicle 200 from the remote operator H in the remote operation mode or the shared driving mode. The vehicle operation unit 120 is, for example, a steering wheel, foot pedals (e.g., an accelerator pedal and a brake pedal), etc., but may also be realized by a joystick or the like.

[0071] Furthermore, the vehicle operation unit 120 may have a reception unit for receiving various inputs such as selections made by the remote operator H. The reception unit is realized by a touch panel or the like, but may also be realized by a hardware key (hardware button), a slide switch, or the like. Furthermore, the vehicle operation unit 120 may receive various inputs based on information based on the voice, gestures, line of sight, etc. of the remote operator H.

[0072] The remote control device 130 performs various processes related to vehicle control (e.g., remote operation) of the vehicle 200. When the vehicle 200 is in a situation where it cannot be driven autonomously, the remote control device 130 executes a process to switch the driving mode of the vehicle 200. Even when the vehicle 200 is in a situation where it cannot be driven autonomously, the remote control device 130 reduces the burden on the remote operator H by reducing the frequency with which the remote operator H remotely controls the vehicle. Specifically, when the vehicle 200 is in a situation where it cannot be driven autonomously and predetermined conditions are satisfied, the remote control device 130 can switch to a shared driving mode instead of a full remote control mode, and by switching to the shared driving mode, the burden on the remote operator H related to vehicle control can be reduced compared to when switching to the full remote control mode.

[0073] Note that the driving mode of the vehicle 200 may be switched not only when the vehicle 200 is in a situation where it cannot be driven autonomously, but may also be executed, for example, depending on the situation at the time of remote operation by the remote operator H, the remote operation skill of the remote operator H, etc. For example, the driving mode may be switched depending on the number of vehicles 200 that the remote operator H is remotely monitoring or remotely operating. Furthermore, if the remote operator H has low remote operation skill, the driving mode may be switched to one that has a lower operational load than other remote operators with high remote operation skill.

[0074] In this way, the remote control device 130 has a shared driving mode in addition to the conventional driving modes of an autonomous driving mode and a fully remote controlled mode. The remote control device 130 is an example of an information processing system. The remote control device 130 may also be realized by a server device.

[0075] Vehicle 200 is an example of a moving object in which people including a driver ride, and is remotely monitored or remotely operated by remote operator H as necessary. Vehicle 200 is an autonomous vehicle. Vehicle 200 may be an autonomous vehicle that can switch between autonomous driving and manual driving. Note that the autonomous vehicle is not particularly limited as long as it is capable of autonomous driving and remote control driving, and may be an autonomously driven bus, an autonomously driven taxi, an autonomously driven private car, an autonomously driven truck, an autonomously driven construction vehicle (e.g., a dump truck), an unmanned transport robot that is capable of autonomous driving and remote control, or the like.

[0076] Next, the functional configuration of each component of the vehicle control system 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the vehicle control system 10 according to this embodiment. Of the various components included in the vehicle control system 10, Fig. 2 only shows the input / output system 100a, the remote control device 130, and the vehicle 200. The functional configuration of the input / output system 100a has been described above, so a description thereof will be omitted here.

[0077] As shown in FIG. 2, the remote control device 130 includes a response content determination unit 131, a driving continuation determination unit 132, a control signal switching instruction unit 133, and a communication state measurement unit 134.

[0078] The response content determination unit 131 determines a driving task for the vehicle 200 to travel. When the vehicle 200 is unable to travel in an autonomous driving mode, the response content determination unit 131 determines a current driving task for the vehicle 200 to travel, based on at least one of the surrounding environment of the vehicle 200 and the sensing result acquired from the environment information acquisition unit 210. The driving task is an example of response content information.

[0079] The response content determination unit 131 may determine the response content using, for example, a table in which the surrounding environment of the vehicle 200 and the response content are associated, or may obtain the response content by inputting the surrounding environment and the like into a trained model that has been trained to input the surrounding environment of the vehicle 200 and output the response content.

[0080] While vehicle 200 is traveling, continuation of traveling feasibility determination unit 132 determines whether or not traveling conditions (ODD: Operational Design Domain, see FIG. 5, described later) that are conditions necessary for traveling in the current traveling mode are satisfied. For example, when vehicle 200 is traveling in an autonomous traveling mode, continuation of traveling feasibility determination unit 132 determines at predetermined time intervals whether or not the traveling conditions corresponding to the autonomous traveling mode are satisfied.

[0081] When traveling in the autonomous traveling mode becomes impossible, the control signal switching instruction unit 133 performs processing to determine a traveling mode to which the vehicle 200 can switch, based on at least one of the communication information, sensing information, location information, traveling information, environmental information, and traveling task information, and the traveling enablement conditions set for each traveling mode for traveling in that traveling mode. In this embodiment, the control signal switching instruction unit 133 determines the traveling mode to which the vehicle 200 can switch, based on at least the communication information and the traveling enablement conditions. Note that the traveling information is an example of movement state information.

[0082] Although details will be described later, in this embodiment, the control signal switching instruction unit 133 has three driving modes as shared driving modes: accelerator manual driving mode, brake manual driving mode, and steering manual driving mode (see FIG. 3 described later). The control signal switching instruction unit 133 may determine the driving mode after switching based on a driving mode that satisfies driving enable conditions from among the accelerator manual driving mode, brake manual driving mode, steering manual driving mode, and full remote control mode. The control signal switching instruction unit 133 is an example of a determination unit.

[0083] The control signal switching instruction unit 133 outputs information indicating the determined driving mode to the vehicle 200. The process by which the control signal switching instruction unit 133 determines the switchable driving mode will be described later.

[0084] The communication status measurement unit 134 measures the status of communication between the remote control device 130 and the vehicle 200. The communication status measurement unit 134 measures the status of communication, such as the communication delay or the communication bandwidth between the remote control device 130 and the vehicle 200. Note that the communication status may include at least one of the status of communication delay or the communication bandwidth. The communication status may also include a packet loss rate, etc. Note that information indicating the communication status is an example of communication information.

[0085] The communication status measurement unit 134 receives, for example, information for calculating a delay time of a communication network (for example, an RTT (Round Trip Time) measurement packet), and measures a communication delay based on the received TRR measurement packet. For example, the communication status measurement unit 134 receives an RTT measurement packet with a timestamp from the vehicle 200, and measures a communication delay based on the timestamp and the time at which the RTT measurement packet is received. Note that the communication delay may be a one-way delay time or a round-trip delay time.

[0086] The communication state measurement unit 134 may also transmit data to a specific device (e.g., vehicle 200) connected to the communication network and measure the communication bandwidth based on the amount of data that the device was able to receive within a predetermined unit time. The communication state measurement unit 134 may also receive data from a specific device (e.g., vehicle 200) connected to the communication network and measure the communication bandwidth based on the amount of data that the communication state measurement unit 134 was able to receive within a predetermined unit time.

[0087] The timing at which the communication status measuring unit 134 measures the communication status is not particularly limited, and the communication status measuring unit 134 may measure the communication status periodically or may measure the communication status when a change in driving mode occurs. Furthermore, the method by which the communication status measuring unit 134 measures the communication status is not limited to the above, and any known method may be used.

[0088] The communication status measurement unit 134 may also measure sensing information indicating the state of sensing of the surroundings of the vehicle 200. The sensing state includes processing delays or omissions of image data, sound data, or point cloud data. The processing delays refer to processing delays that occur in the detection process in response to reception by the reception unit of the vehicle operation unit 120, the internal processing of the computer, and the presentation process to the presentation unit 110. Taking image data as an example, omissions refer to a phenomenon in which part of a frame is not displayed for some reason, causing the image to be momentarily interrupted. Note that defects include, for example, dropped frames. Furthermore, the presentation process includes, for example, display processing.

[0089] The communication state measurement unit 134 may acquire the sensing state of the vehicle control system 10 based on the sensing state of the vehicle 200 and the sensing state of the remote operation system 100. Taking processing delay as an example, the communication state measurement unit 134 may acquire the time from the time when the environmental information acquisition unit 210 acquires the sensing result to the time when the presentation unit 110 presents information based on the sensing result, excluding the communication delay, as the processing delay of the vehicle control system 10. The processing delay is a one-way delay. The sensing result includes, for example, at least one of image, sound, and point cloud data.

[0090] The vehicle 200 is a mobile body that can be driven automatically and manually by remote control. The vehicle 200 includes an environmental information acquisition unit 210, a vehicle information acquisition unit 220, an automatic driving system 230, a control signal switching unit 240, and a vehicle control unit 250.

[0091] The environmental information acquisition unit 210 acquires environmental information indicating the surrounding environment including the state of the surroundings of the vehicle 200. The environmental information acquisition unit 210 acquires the state of the surroundings of the vehicle 200 from various sensors mounted on the vehicle 200. The state of the surroundings of the vehicle 200 includes the state of obstacles, the state of the road surface, or the weather. The various sensors include, for example, at least one of an image sensor, a microphone, and a point cloud detection sensor.

[0092] The state of the obstacle includes at least one of the size, position, speed, etc. of the obstacle, and can be acquired based on the sensing results of a sensor such as a camera or a LiDAR (Light Detection and Ranging).

[0093] The road surface condition includes at least one of dryness, wetness, snow cover, and ice on the road surface on which the vehicle 200 travels, and can be acquired based on the sensing results obtained by sensing the road surface with a sensor such as a polarization camera. The sensing results may be, for example, an image. Note that "dry" means that the road surface is dry, "wet" means that the road surface is wet, "snow cover" means that snow has accumulated on the road surface, and "frozen" means that the road surface is frozen. The road surface condition may also include depressions in the road, etc.

[0094] The weather includes at least one of sunny, cloudy, foggy, rainy, snowy, thunder, and the like, and can be acquired based on the results of sensing the sky with a sensor such as a polarization camera, for example.

[0095] The method of acquiring the state of the surroundings of the vehicle 200 is not limited to the above, and any known technology may be used. Furthermore, the road surface condition and weather are not limited to being acquired based on sensing results, and may be acquired from an external device or the like. The surrounding condition may also include information about earthquakes, information about wind, etc. The surrounding condition may also include images of the surroundings of the vehicle 200 captured by a camera. The external device may be, for example, a server device that manages the road surface condition and weather. The information about earthquakes may include, for example, seismic intensity, epicenter, etc., and the information about wind may include, for example, wind speed, etc.

[0096] The vehicle information acquisition unit 220 acquires vehicle information indicating the vehicle state of the vehicle 200. The vehicle state includes the driving state of the vehicle 200. The driving state of the vehicle 200 includes the speed of the vehicle 200. The speed of the vehicle 200 is, for example, the current speed. The driving state of the vehicle 200 may further include the steering angle, acceleration, etc. of the vehicle 200. The vehicle information may further include specification information, etc. of the vehicle 200. The specification information may include driving task information indicating a driving task for driving the vehicle 200, and may include at least one of the size, wheelbase, maximum steering angle, maximum vehicle speed, maximum acceleration, maximum deceleration, and obstacle detection performance of the vehicle 200. The driving task includes going straight, turning right or left, avoiding parking on the road, reversing, or pulling over to the shoulder of the road. The driving state is an example of a moving state.

[0097] The vehicle information acquisition unit 220 may acquire the vehicle state from an external device or may acquire the vehicle state based on the sensing results of various sensors. The external device may be, for example, a server device that manages specification information of the vehicle 200.

[0098] Furthermore, the vehicle information acquisition unit 220 may acquire location information indicating the location of the vehicle 200. The location information is also referred to as position information. The location of the vehicle 200 includes at least one of the location of the vehicle 200 and a location on the driving route of the vehicle 200. The location of the vehicle 200 may be the current location. The vehicle information acquisition unit 220 may be configured to include, for example, a GPS (Global Positioning System) module that acquires a GPS signal (i.e., a radio wave output from a satellite) and measures the current location of the vehicle 200 based on the acquired GPS signal to acquire the current location. Furthermore, the vehicle information acquisition unit 220 may acquire the location on the driving route of the vehicle 200 by acquiring a driving route based on a driving plan generated by the autonomous driving system 230. The location may be expressed as an area. The area may be a lane, a road section, a block, or the like. The driving route is an example of a movement route.

[0099] Furthermore, the vehicle information acquisition unit 220 may acquire, as vehicle information, the state of the automatic driving system 230. The state of the automatic driving system 230 includes information indicating whether the automatic driving system 230 is normal or not.

[0100] The automatic driving system 230 executes processing for automatic driving of the vehicle 200. The automatic driving system 230 generates a driving plan for automatic driving based on vehicle information and the like. The driving plan includes a driving route, a speed, and the like. When driving in the autonomous driving mode, the automatic driving system 230 generates a control signal based on the driving plan and outputs it to the control signal switching unit 240 in order to execute the generated driving plan.

[0101] Control signal switching unit 240 outputs a control signal to vehicle control unit 250 to realize driving in accordance with the driving mode determined by control signal switching instruction unit 133. When driving in the autonomous driving mode, control signal switching unit 240 outputs a control signal from automatic driving system 230 to vehicle control unit 250. When driving in the autonomous driving mode, even if control signal switching unit 240 receives a control signal from vehicle operation unit 120, it does not output the control signal to vehicle control unit 250.

[0102] Furthermore, when traveling in a complete remote control mode among the remote control modes, control signal switching unit 240 outputs a control signal from vehicle operation unit 120 to vehicle control unit 250. When traveling in the complete remote control mode, even if control signal switching unit 240 receives a control signal from automatic driving system 230, it does not output the control signal to vehicle control unit 250. It can also be said that when traveling in the complete remote control mode, control signal switching unit 240 outputs a control signal from vehicle operation unit 120 to vehicle control unit 250 instead of a control signal from automatic driving system 230.

[0103] Furthermore, when driving in the shared driving mode of the remote operation mode, control signal switching unit 240 extracts necessary information from each of the control signals from vehicle operation unit 120 and the control signals from automatic driving system 230, and outputs the extracted information to vehicle control unit 250. When driving in the accelerator manual driving mode, control signal switching unit 240 extracts information other than the accelerator and parking brake from the control signals from automatic driving system 230, and extracts information related to the accelerator and parking brake from the control signals from vehicle operation unit 120, and outputs a control signal including the extracted information to vehicle control unit 250. It can also be said that when driving in the accelerator manual driving mode, control signal switching unit 240 replaces information related to the accelerator and parking brake from the control signals from automatic driving system 230 with information related to the accelerator and parking brake from vehicle operation unit 120 and outputs the replaced information.

[0104] The vehicle control unit 250 controls the running and presentation of the vehicle 200 based on the control signal from the control signal switching unit 240.

[0105] [2. Explanation of each driving mode] Next, each driving mode, including the shared driving mode, will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of vehicle control sharing in each driving mode. FIG. 3 shows the correspondence between each driving mode and the control operation for each driving mode. Here, each driving mode includes an autonomous driving mode, a manual accelerator driving mode, a manual brake driving mode, a manual steering wheel driving mode, and a fully remote control mode. In addition, the control operation includes parking brake operation, accelerator operation, brake operation, steering wheel operation, and operation of presentation devices such as hazard lights, turn signals, and vehicle surrounding presentation devices. Hereinafter, operation of a presentation device will also be referred to as a presentation operation.

[0106] As shown in FIG. 3, the autonomous driving mode is a driving mode in which all control operations are performed by automatic driving (automatic driving system 230).

[0107] The accelerator manual driving mode is a driving mode in which, among the control operations, the parking brake operation and accelerator operation are performed by remote control, and the other control operations are performed by automatic driving. The parking brake operation and accelerator operation are examples of first operation control that is permitted to the remote operator H or executed by remote control in the accelerator manual driving mode. Furthermore, the other control operations (brake operation, steering operation, and presented operation) are examples of second operation control other than the first operation control in the accelerator manual driving mode.

[0108] The manual braking driving mode is a driving mode in which, among the control operations, the parking brake operation and the brake operation are performed by remote control, and the other control operations are performed by automatic driving. The parking brake operation and the brake operation are examples of first operational control in the manual braking driving mode. Furthermore, the other control operations (accelerator operation, steering operation, and presentation operation) are examples of second operational control other than the first operational control in the manual braking driving mode.

[0109] The manual steering mode is a driving mode in which, among the control operations, the parking brake operation, steering operation, and presentation operation are performed by remote control, and the other control operations are performed by automatic driving. The parking brake operation, steering operation, and presentation operation are examples of first operational control in the manual steering mode. Furthermore, the other control operations (accelerator operation and brake operation) are examples of second operational control other than the first operational control in the manual steering mode.

[0110] The complete remote control mode is a driving mode in which all control operations are performed by remote control. The parking brake operation, accelerator operation, brake operation, steering operation, and presentation operation are examples of first operational control in the complete remote control mode.

[0111] As described above, FIG. 3 can also be said to show the operational control in which remote manual operation is permitted for each driving mode.

[0112] Furthermore, for example, the burden on the remote operator H tends to increase in the order of accelerator manual driving mode, brake manual driving mode, steering manual driving mode, and full remote control mode. For example, it can be said that the remote operations permitted to the remote operator H or performed in the remote control modes increase, become more complex, or become more advanced in this order. "Increase" refers to the number of remote operations performed by the remote operator H, which are 2, 2, 3, and 5, respectively. "Complexity" refers to the number of remote operations performed in parallel by the remote operator H, which are 1, 1, 2, and 4, respectively. Note that the number of remote operations performed in parallel is calculated assuming that the parking brake operation is an operation performed when the vehicle 200 is stopped and is not performed in parallel with other operations. "Advanced" refers to the difficulty of the remote operation, and the steering manual driving mode requires precise steering operation, and in that respect it can be said to be more difficult than the accelerator manual driving mode and the brake manual driving mode.

[0113] Furthermore, operation control includes driving control and presentation control. Driving control includes deceleration control, acceleration control, and direction control. Deceleration control is, for example, control of the parking brake or brake. Acceleration control is, for example, control of the accelerator. Direction control is, for example, control of the steering wheel. Furthermore, presentation control includes light control, display control, and projection control. Light control is, for example, control of hazard lights or turn signals. Display control is, for example, control of a display device such as a display device. Projection control is, for example, control of a projection device such as a projector. Note that operation control may also be referred to as operation, and driving control may also be referred to as driving.

[0114] It can also be said that when switching from automatic driving to remote manual driving (from autonomous driving mode to remote operation mode) occurs in the vehicle 200, the control signal switching instruction unit 133 determines a first operation control to be remotely manually operated among the operation controls of the vehicle 200 based on at least one of the above information. Then, the control signal switching instruction unit 133 outputs instruction information to the control signal switching unit 240 so as to operate the vehicle 200 based on remote manual operation corresponding to the first operation control in the determined remote driving mode, and to cause the vehicle 200 (for example, the automatic driving system 230) to execute a second operation control other than the first operation control. In this way, the control signal switching instruction unit 133 functions as a switching instruction unit. Note that the automatic control here includes causing the automatic driving system 230 to control at least one of the driving control and the presentation control.

[0115] The allocation of functions in the shared driving mode is not limited to the allocation shown in Fig. 3. The shared driving mode may include driving modes other than the above three driving modes. For example, the shared driving mode may include a driving mode in which the accelerator and brake operations are performed by remote control and other functions are performed by automatic driving. The control operations are also not limited to the operations shown in Fig. 3.

[0116] [3. Explanation of driving conditions] Next, the driving enable conditions for each of the driving modes will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the driving enable conditions for each of the driving modes. Fig. 4 shows the correspondence between each of the driving modes and the conditions for each item required to allow driving in that driving mode. The remote control device 130 stores in advance, for example, a table showing the driving enable conditions for each of the driving modes shown in Fig. 4.

[0117] The driving conditions include communication delay (RTT), video delay (one way), bandwidth (communication bandwidth), area, vehicle speed, autonomous driving system state, pedestrians in the driving route, and driving task. Communication delay and bandwidth are examples of communication states, video delay is an example of a sensing state, and area is an example of the location of vehicle 200. Furthermore, vehicle speed and autonomous driving system state are examples of driving states, pedestrians in the driving route are an example of the surrounding environment, and driving task is an example of a movement task.

[0118] When determining the driving mode based on the communication state, the control signal switching instruction unit 133 determines the driving mode based on the correspondence between the communication state and the driving enable conditions for each driving mode (for example, communication delay, bandwidth conditions). Determining the driving mode corresponds to determining the first operation control to be remotely operated. When determining the driving mode based on the driving enable conditions shown in FIG. 4, the control signal switching instruction unit 133 determines a driving mode in which the permitted remote operations increase, become more complex, or become more advanced, as the communication state improves. A good communication state means, for example, little communication delay or a wide bandwidth. In other words, it can also be said that the communication speed is high. A good communication state can also be said to mean that the communication environment has strict constraints.

[0119] Furthermore, when determining the driving mode based on the sensing state, the control signal switching instruction unit 133 determines the driving mode based on the correspondence between the sensing state and the driving enable conditions for each driving mode (for example, conditions for video delay). When determining the driving mode based on the driving enable conditions shown in Fig. 4, the control signal switching instruction unit 133 determines a driving mode in which the permitted remote operations increase, become more complex, or become more advanced, as the sensing state improves. A good sensing state means, for example, that there is little video delay.

[0120] Furthermore, when determining the driving mode based on the location information of vehicle 200, control signal switching instruction unit 133 determines the driving mode based on the correspondence between the location of vehicle 200 and driving enable conditions for each driving mode (e.g., area conditions). When determining the driving mode based on the driving enable conditions shown in FIG. 4, control signal switching instruction unit 133 determines a driving mode in which the permitted remote operations increase, become more complex, or become more advanced, as the safety of autonomous driving of vehicle 200 in the location indicated by the location information decreases. In other words, the higher the difficulty of autonomous driving, the more the permitted remote operations increase, become more complex, or become more advanced, as the driving mode is determined. Low safety of autonomous driving by vehicle 200 means, for example, traveling in a lane other than the vehicle's own lane (e.g., an oncoming lane, or an overtaking lane if the current lane is a driving lane). In other words, low safety of autonomous driving by vehicle 200 means a high risk in autonomous driving. Low safety of autonomous driving by vehicle 200 may also mean that remote manual driving is relatively less dangerous than autonomous driving.

[0121] Furthermore, when determining the driving mode based on the driving state of vehicle 200, control signal switching instruction unit 133 determines the driving mode based on the correspondence between the driving state and the driving enable conditions for each driving mode (for example, vehicle speed, conditions for the autonomous driving system state). When determining the driving mode based on the driving enable conditions shown in Fig. 4, control signal switching instruction unit 133 determines a driving mode in which the permitted remote operations increase, become more complex, or become more advanced as the safety of vehicle 200 at the vehicle speed increases. A high safety of vehicle 200 means, for example, that the vehicle speed is slow, that is, the risk of driving is low.

[0122] 4, the control signal switching instruction unit 133 determines a driving mode in which the remote operations to be executed increase, become more complex, or become more advanced as the safety of the vehicle 200 in the autonomous driving system state decreases. Low safety of the autonomous driving of the vehicle 200 means, for example, that there is an abnormality in the autonomous driving system state. In other words, low safety of the autonomous driving of the vehicle 200 means that remote manual driving is relatively less dangerous during driving.

[0123] Furthermore, when determining the driving mode based on the surrounding environment of the vehicle 200, the control signal switching instruction unit 133 determines the driving mode based on the correspondence between the surrounding environment and the driving enable conditions for each driving mode (for example, conditions for pedestrians in the walking path). When determining the driving mode based on the driving enable conditions shown in FIG. 4, the control signal switching instruction unit 133 determines a driving mode in which the permitted remote operations increase, become more complex, or become more advanced, as the safety of autonomous driving of the vehicle 200 in the surrounding environment decreases. In other words, the higher the difficulty of autonomous driving, the more the permitted remote operations increase, become more complex, or become more advanced, as determined. Low safety of autonomous driving of the vehicle 200 means, for example, the presence of an obstacle on the driving path, such as the presence of a pedestrian on the driving path. In other words, low safety of autonomous driving of the vehicle 200 means a high risk in autonomous driving. Low safety of autonomous driving of the vehicle 200 may also mean that remote manual driving is relatively less dangerous than autonomous driving.

[0124] Furthermore, when determining a driving mode based on a driving task of vehicle 200, control signal switching instruction unit 133 determines the driving mode based on the correspondence between the driving task and the driving enable conditions (e.g., applicable tasks) for each driving mode. When determining a driving mode based on the driving enable conditions shown in Fig. 4, control signal switching instruction unit 133 determines a driving mode in which the permitted remote operations increase, become more complex, or become more advanced as the driving task becomes more complex. A complex driving task means, for example, that the operation control is sophisticated or the decisions for the operation control are complex.

[0125] In this way, the control signal switching instruction unit 133 can partially automate vehicle operation (cause automatic control by the automatic driving system 230) based on at least one of the communication status, sensing status, location information of the vehicle 200, the driving status of the vehicle 200, the surrounding environment of the vehicle 200, and executable driving tasks, and can also limit the vehicle functions (control operations) that can be controlled by the automatic driving function and the remote operator H, respectively.

[0126] As shown in FIGS. 4 and 5, it can also be said that the control signal switching instruction unit 133 determines the driving mode based on the correspondence between various information and the operational control executed by remote manual operation.

[0127] In the above, an example has been described in which the control signal switching instruction unit 133 determines the driving mode based on one of the communication state, sensing state, location information, driving state, surrounding environment, and driving task, but the driving mode may also be determined based on two or more of the communication state, sensing state, location information, driving state, surrounding environment, and driving task.

[0128] [4. Vehicle Control System Operation] Next, the operation of the vehicle control system 10 described above (an example of an information processing method) will be described with reference to Figs. 5 to 7. Fig. 5 is a flowchart showing the operation of the vehicle control system 10 according to this embodiment. Specifically, Fig. 5 shows the operation of the remote control device 130. Note that, below, an example will be described in which the control signal switching instruction unit 133 determines the driving mode based on communication information indicating at least the state of communication between the remote control device 130 and the vehicle 200.

[0129] As shown in FIG. 5, the vehicle 200 starts traveling in the autonomous traveling mode (S10).

[0130] Next, the control signal switching instruction unit 133 acquires communication information, location information, environmental information, and vehicle information (S20). The control signal switching instruction unit 133 acquires communication information from the communication state measurement unit 134, and acquires location information, environmental information, and vehicle information from the vehicle 200. The control signal switching instruction unit 133 may output necessary information from the acquired various types of information to the traveling continuation determination unit 132. Note that at least one of the communication information, location information, environmental information, and vehicle information may be acquired in step S20. In this embodiment, it is sufficient that at least communication information is acquired in step S20. Furthermore, the control signal switching instruction unit 133 may acquire response content information from the response content determination unit 131. Furthermore, the control signal switching instruction unit 133 also functions as an acquisition unit that acquires the above various types of information. At least one of the communication information, location information, environmental information, and vehicle information is an example of information regarding the state of the moving object.

[0131] The timing at which the control signal switching instruction unit 133 acquires the communication information, location information, environmental information, vehicle information, and response content information is not particularly limited, and may be acquired periodically or based on an operation by the remote operator H. Furthermore, the communication information, location information, environmental information, vehicle information, and response content information may be acquired at different times or at the same time.

[0132] Next, the driving continuation possibility determination unit 132 determines whether driving in the autonomous driving mode can be continued (S30). The driving continuation possibility determination unit 132 determines whether driving in the autonomous driving mode can be continued, for example, based on the driving possibility conditions corresponding to the autonomous driving mode and the various information acquired in step S20. For example, the driving continuation possibility determination unit 132 determines that continuing driving in the autonomous driving mode is possible if the driving possibility conditions are met for all items in the autonomous driving mode, and that continuing driving in the autonomous driving mode is impossible if at least one item is not met. The driving continuation possibility determination unit 132 outputs the determination result to the control signal switching instruction unit 133.

[0133] If the driving continuation possibility determination unit 132 determines that it is possible to continue driving in the autonomous driving mode (Yes in S30), the control signal switching instruction unit 133 continues driving in the autonomous driving mode (S40). In other words, the control signal switching instruction unit 133 does not change the driving mode.

[0134] Next, the control signal switching instruction unit 133 determines whether or not the traveling has ended (S50). The control signal switching instruction unit 133 may make the determination in step S50 based on, for example, whether or not the destination has been reached.

[0135] If the vehicle has finished traveling (Yes in S50), the control signal switching instruction unit 133 ends the processing related to the vehicle 200. If the vehicle has not finished traveling (No in S50), the control signal switching instruction unit 133 returns to step S20 and continues the processing.

[0136] Furthermore, when the driving continuation possibility determination unit 132 determines that it is impossible to continue driving in the autonomous driving mode (No in S30), the control signal switching instruction unit 133 executes a process of switching the driving mode (S60). Details of step S60 will be described later with reference to FIG. 6. Note that, for example, when the result is No in step S30, the control signal switching instruction unit 133 may cause the presentation unit 110 to present information indicating that it is impossible to continue driving in the autonomous driving mode. Furthermore, for example, when the result is No in step S30, the control signal switching instruction unit 133 may cause the presentation unit 110 to display information for prompting the remote operator H to select a driving mode to switch to, or may cause the presentation unit 110 to display information indicating switchable driving modes based on various current information. When displaying information indicating switchable driving modes, the control signal switching instruction unit 133 may highlight a driving mode that places less strain on the remote operator H.

[0137] Note that a determination of No in step S30, that is, the occurrence of a state in which automatic driving cannot be continued, is an example of the occurrence of a switch from automatic driving to remote manual driving.

[0138] In the following, an example will be described in which the control signal switching instruction unit 133 causes the presentation unit 110 to display a message for prompting the remote operator H to select a destination driving mode.

[0139] Next, the control signal switching instruction unit 133 determines whether the driving mode determined in step S60 is a fallback mode (S70). The fallback mode is a driving mode that is determined when there is no driving mode that satisfies the driving enable condition, and is, for example, a driving mode that stops the vehicle 200 from traveling.

[0140] If the driving mode determined in step S60 is the fallback mode (Yes in S70), the control signal switching instruction unit 133 stops the vehicle 200 (for example, stops the vehicle on the shoulder of the road) and ends the processing. If the driving mode determined in step S60 is not the fallback mode (No in S70), the control signal switching instruction unit 133 causes the vehicle 200 to drive in the driving mode determined in step S60. That is, the driving mode of the vehicle 200 is switched from the autonomous driving mode to the driving mode determined in step S60. Then, the process returns to step S20 and continues.

[0141] Next, details of the process (S60) for switching the driving mode shown in Fig. 5 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing details of the process for switching the driving mode shown in Fig. 5. Note that vehicle 200 is assumed to be stopped because it is not capable of traveling in the autonomous driving mode.

[0142] As shown in Fig. 6, the remote operator H selects one driving mode from the driving modes displayed on the presentation unit 110. In other words, the presentation unit 110 displays information for selecting a driving mode, and the vehicle operation unit 120 accepts the selection of the driving mode from the remote operator H (S110). The information for selecting a driving mode is, for example, information indicating each of the driving modes set in advance, and in this embodiment includes information indicating five driving modes: an autonomous driving mode, a manual accelerator driving mode, a manual brake driving mode, a manual steering driving mode, and a full remote control mode. The information for selecting a driving mode may also include a fallback mode.

[0143] When the vehicle operation unit 120 accepts the selection of a driving mode, the control signal switching instruction unit 133 determines whether the autonomous driving mode has been selected, and, if the autonomous driving mode has been selected, whether driving in the autonomous driving mode is possible (S120). When the autonomous driving mode has been selected and driving in the autonomous driving mode is possible (Yes in S120), the control signal switching instruction unit 133 switches to the autonomous driving mode (S130). The control signal switching instruction unit 133 outputs instruction information to the vehicle 200 indicating that the vehicle should drive in the autonomous driving mode. When the control signal switching unit 240 acquires the instruction information, it outputs a control signal from the automatic driving system 230 to the vehicle control unit 250, thereby causing the vehicle 200 to drive in the autonomous driving mode.

[0144] As a result, for example, when the cause of the determination "No" in step S30 is resolved, the control signal switching instruction unit 133 can resume driving in the autonomous driving mode.

[0145] The determination of whether or not driving in the autonomous driving mode is possible is made based on the driving enable conditions for the autonomous driving mode shown in Fig. 4. For example, the driving enable conditions for determining whether or not driving in the autonomous driving mode is possible in steps S120 and S30 may be the same.

[0146] Furthermore, if the autonomous driving mode has not been selected (No in S120), the control signal switching instruction unit 133 further determines whether or not the accelerator manual driving mode has been selected, and if the accelerator manual driving mode has been selected, whether or not driving in the accelerator manual driving mode is possible (S140). The control signal switching instruction unit 133 determines whether or not driving in the accelerator manual driving mode is possible, for example, based on the driving enable conditions corresponding to the accelerator manual driving mode and the various information acquired in step S20. For example, in step S140, the control signal switching instruction unit 133 determines whether or not the driving enable conditions in the accelerator manual driving mode are met.

[0147] The control signal switching instruction unit 133 may, for example, determine the driving mode based on communication information and sensing information, may determine the driving mode based on communication information and location information, may determine the driving mode based on communication information and driving information, may determine the driving mode based on communication information and environmental information, or may determine the driving mode based on communication information and driving task information.

[0148] If the accelerator manual driving mode is selected and driving in the accelerator manual driving mode is possible (Yes in S140), the control signal switching instruction unit 133 switches to the accelerator manual driving mode (S150). The control signal switching instruction unit 133 outputs instruction information to the vehicle 200 indicating that driving is to be performed in the accelerator manual driving mode. Upon receiving this instruction information, the control signal switching unit 240 replaces the portions of the control signals from the automatic driving system 230 that correspond to the operation of the parking brake and accelerator with control signals from the vehicle operation unit 120 and outputs the replaced control signals to the vehicle control unit 250, thereby causing the vehicle 200 to drive in the accelerator manual driving mode.

[0149] Furthermore, if the accelerator manual traveling mode has not been selected (No in S140), the control signal switching instruction unit 133 further determines whether the brake manual traveling mode has been selected, and if the brake manual traveling mode has been selected, whether traveling in the brake manual traveling mode is possible (S160). The control signal switching instruction unit 133 determines whether traveling in the brake manual traveling mode is possible, for example, based on the traveling enablement conditions corresponding to the brake manual traveling mode and the various information acquired in step S20. For example, in step S160, the control signal switching instruction unit 133 determines whether the traveling enablement conditions in the brake manual traveling mode are met.

[0150] When the manual brake driving mode is selected and driving in the manual brake driving mode is possible (Yes in S160), the control signal switching instruction unit 133 switches to the manual brake driving mode (S170). The control signal switching instruction unit 133 outputs instruction information to the vehicle 200 indicating that driving is to be performed in the manual brake driving mode. Upon receiving the instruction information, the control signal switching unit 240 replaces the portions of the control signals from the automatic driving system 230 that correspond to the operation of the parking brake and brakes with control signals from the vehicle operation unit 120 and outputs the replaced control signals to the vehicle control unit 250, thereby causing the vehicle 200 to drive in the manual brake driving mode.

[0151] Furthermore, if the brake manual traveling mode has not been selected (No in S160), the control signal switching instruction unit 133 further determines whether the handle manual traveling mode has been selected, and if the handle manual traveling mode has been selected, whether traveling in the handle manual traveling mode is possible (S180). The control signal switching instruction unit 133 determines whether traveling in the handle manual traveling mode is possible, for example, based on the traveling enablement conditions corresponding to the handle manual traveling mode and the various information acquired in step S20. For example, in step S180, the control signal switching instruction unit 133 determines whether the traveling enablement conditions in the handle manual traveling mode are met.

[0152] When the manual steering wheel driving mode is selected and driving in the manual steering wheel driving mode is possible (Yes in S180), the control signal switching instruction unit 133 switches to the manual steering wheel driving mode (S190). The control signal switching instruction unit 133 outputs instruction information to the vehicle 200 indicating that driving is to be performed in the manual steering wheel driving mode. Upon receiving the instruction information, the control signal switching unit 240 replaces the portions of the control signals from the automatic driving system 230 that are related to the operation of the parking brake, steering wheel, and presentation device with control signals from the vehicle operation unit 120 and outputs the replaced control signals to the vehicle control unit 250, thereby causing the vehicle 200 to drive in the manual steering wheel driving mode.

[0153] Furthermore, if the manual handle driving mode has not been selected (No in S180), the control signal switching instruction unit 133 further determines whether the complete remote control mode has been selected, and if the complete remote control mode has been selected, whether driving in the complete remote control mode is possible (S200). The control signal switching instruction unit 133 determines whether driving in the complete remote control mode is possible, for example, based on the driving enablement conditions corresponding to the complete remote control mode and the various information acquired in step S20. For example, in step S200, the control signal switching instruction unit 133 determines whether the driving enablement conditions in the complete remote control mode are met.

[0154] When the complete remote control mode is selected and driving is possible in the complete remote control mode (Yes in S200), the control signal switching instruction unit 133 switches to the complete remote control mode (S210). The control signal switching instruction unit 133 outputs instruction information to the vehicle 200 indicating that the vehicle should drive in the complete remote control mode. Upon receiving the instruction information, the control signal switching unit 240 discards the control signal from the automatic driving system 230 and outputs the control signal from the vehicle operation unit 120 to the vehicle control unit 250, thereby causing the vehicle 200 to drive in the complete remote control mode.

[0155] Furthermore, if the complete remote control mode is not selected (No in S200), or if the complete remote control mode is selected but driving in the complete remote control mode is not possible, the control signal switching instruction unit 133 determines to switch to the fallback mode (S220). The control signal switching instruction unit 133 outputs instruction information to the vehicle 200 indicating that the vehicle should drive in the fallback mode. Upon receiving the instruction information, the autonomous driving system 230 outputs a control signal to stop the vehicle 200 on the shoulder of the road to the vehicle control unit 250 via the control signal switching unit 240, thereby stopping the vehicle 200.

[0156] Note that if the accelerator manual driving mode is selected in step S140 but driving in the accelerator manual driving mode is not possible, if the brake manual driving mode is selected in step S160 but driving in the brake manual driving mode is not possible, or if the handle manual driving mode is selected in step S180 but driving in the handle manual driving mode is not possible, the control signal switching instruction unit 133 may cause the presentation unit 110 to display information indicating that the selected driving mode is not possible. In this case, the control signal switching instruction unit 133 may return to step S110 and accept the selection of the driving mode again, or may cause the presentation unit 110 to display information indicating the selectable driving modes.

[0157] In this embodiment, the control signal switching instruction unit 133 uses at least the communication information to determine whether or not traveling is permitted in steps S120, S140, S160, S180, and S200.

[0158] The control signal switching instruction unit 133 may determine the driving mode (an example of a first operational control) based on, for example, the correspondence between only the communication state and the operational control for which remote manual operation is permitted. In this case, the control signal switching instruction unit 133 may determine the post-switching driving mode to be one in which the permitted remote manual operations increase, become more complex, or become more advanced as the communication state improves.

[0159] Furthermore, the control signal switching instruction unit 133 may determine the driving mode (an example of a first operational control) based on, for example, the correspondence between the communication state and the sensing state and the operational control for which remote manual operation is permitted. In this case, the control signal switching instruction unit 133 may determine the post-switching driving mode to be one in which the permitted remote manual operations increase, become more complex, or become more advanced the better the sensing state. For example, the control signal switching instruction unit 133 may determine the post-switching driving mode to be one in which the permitted remote manual operations increase, become more complex, or become more advanced the better the communication state and the sensing state.

[0160] Furthermore, the control signal switching instruction unit 133 may determine the driving mode (an example of a first operational control) based on, for example, the correspondence between the state of communication and the location of the vehicle 200, and the operational control for which remote manual operation is permitted. In this case, the control signal switching instruction unit 133 may determine the post-switching driving mode to be a driving mode in which the number, complexity, or sophistication of remote manual operations to be executed increases as the safety of the vehicle 200 at the location of the vehicle 200 decreases.

[0161] Furthermore, the control signal switching instruction unit 133 may determine the driving mode (an example of the first operational control) based on, for example, a correspondence relationship between the state of communication and the driving state of the vehicle 200 and the operational control for which remote manual operation is permitted. In this case, the control signal switching instruction unit 133 may determine the driving mode after switching to a driving mode in which the permitted remote manual operations increase, become more complex, or become more advanced as the safety of the vehicle 200 in the driving state of the vehicle 200 increases.

[0162] Furthermore, the control signal switching instruction unit 133 may determine the driving mode (an example of the first operational control) based on, for example, a correspondence relationship between the communication state and the surrounding environment of the vehicle 200 and the operational control for which remote manual operation is permitted. In this case, the control signal switching instruction unit 133 may determine the post-switching driving mode to be a driving mode in which the number, complexity, or sophistication of remote manual operations to be executed increases as the safety of the vehicle 200 in the surrounding environment of the vehicle 200 decreases.

[0163] Furthermore, the control signal switching instruction unit 133 may determine the driving mode (an example of a first operation control) based on, for example, the correspondence between the state of communication and executable driving tasks, and operation controls for which remote manual operation is permitted. In this case, the control signal switching instruction unit 133 may determine the post-switching driving mode to be a driving mode in which the number of executable driving tasks of the vehicle 200 increases, becomes more complex, or becomes more advanced, as the number of remote manual operations that can be performed by the vehicle 200 decreases (for example, only turning right or left and going in a straight line).

[0164] Although the above describes an example in which the control signal switching instruction unit 133 determines the driving mode using two pieces of information, the driving mode may be determined using three or more pieces of information. The control signal switching instruction unit 133 may determine the driving mode using, for example, the communication state, the sensing state, the location of the vehicle 200, the driving state of the vehicle 200, the surrounding environment of the vehicle 200, and the executable driving tasks. The information used to determine the driving mode may be set in advance or may be set by the remote operator H.

[0165] It should be noted that after the determination in step S200 is No and the process of step S220 is performed, the process of step S110 may be performed again after a predetermined time interval.

[0166] An example of switching to the accelerator manual driving mode will now be described with reference to FIG. 7. FIG. 7 is a schematic diagram for explaining switching to the accelerator manual driving mode. Specifically, FIG. 7 shows an example of an image displayed on the presentation unit 110 in each state of the vehicle 200. Note that an arrow or a broken line from the vehicle 200 indicates the driving route of the vehicle 200. The image displayed by the presentation unit 110 is, for example, an overhead image, but is not limited to this. If the vehicle 200 is equipped with a surround view system, the presentation unit 110 can display an overhead image.

[0167] 7(a) shows a state in which vehicle 200 has stopped because it has deviated from the conditions under which vehicle 200 can travel in autonomous driving mode. Specifically, FIG. 7(a) shows a state in which vehicle 400 (another vehicle) is stopped in front of vehicle 200 traveling on road L, blocking the vehicle's own lane, and an image showing the state of vehicle 200 immediately after it has stopped is displayed on display unit 110. The current driving mode of vehicle 200 and the result of a determination as to whether vehicle 200 can continue traveling in the autonomous driving mode may also be displayed in the image. The driving mode is automatic traveling (autonomous driving mode), and the most recent result of a determination as to whether vehicle 200 can continue traveling is "travel possible."

[0168] Vehicle 400 is blocking road L, and the only area included in the driving conditions for vehicle 200 is the own lane out of the own lane and the oncoming lane. Therefore, vehicle 200 cannot pass by avoiding vehicle 400 by driving around the oncoming vehicle.

[0169] Furthermore, since vehicle 400 is stopped ahead of vehicle 200, the action content determination unit 131 determines from this situation that vehicle 400 will be overtaken. The action content determination unit 131 calculates a route for overtaking. The overtaking route includes an oncoming lane. Therefore, the location information at the time of overtaking includes the oncoming lane as a driving area.

[0170] 7(b) shows an image indicating a state in which an alert and a deviation from the driving conditions are presented, displayed on the presentation unit 110. FIG. 7(b) shows a state in which the vehicle shown in FIG. 7(a) has stopped for a certain period of time (a stuck state).

[0171] The image includes information that vehicle 400 is the cause of vehicle 200 stopping ("Road Closed" in FIG. 7(b)), the driving conditions that have been deviated from ("Driving in Oncoming Lane Not Possible" in FIG. 7(b)), the current driving mode ("Outside Driving Conditions" in FIG. 7(b)), the result of a determination as to whether driving can continue ("STOP" in FIG. 7(b)), and a switch button for switching the driving mode (button 111 in FIG. 7(b)). For example, the cause of stopping, the driving conditions that have been deviated from, the result of a determination as to whether driving can continue, etc. are displayed as an alert.

[0172] In this way, the reason why the vehicle 200 is unable to travel automatically is displayed to the remote operator H, allowing the remote operator H to appropriately recognize the situation of the vehicle 200. The remote operator H can easily recognize that the vehicle 200 is stopped just by looking at the image shown in (b) of Fig. 7.

[0173] When the remote operator H operates (e.g., presses) the button 111, the control signal switching instruction unit 133 causes, for example, a button (not shown) for the remote operator H to select the driving mode after switching to be displayed on the presentation unit 110. As a result, the control signal switching instruction unit 133 only needs to determine whether or not switching to the driving mode selected by the remote operator H is possible, and the amount of processing required for the determination can be reduced.

[0174] If the switching button is implemented as a hardware button, the switching button does not need to be presented on the presentation unit 110.

[0175] (c) of Fig. 7 is a diagram showing a state in which the mode has been switched to the accelerator manual driving mode. Specifically, (c) of Fig. 7 shows a state in which the accelerator manual driving mode has been selected by the remote operator H, driving in the accelerator manual driving mode is possible (Yes in S140), and the mode has been switched to the accelerator manual driving mode (S150). (c) of Fig. 7 shows that the vehicle 410 is traveling in the oncoming lane. (c) of Fig. 7 also shows monitoring areas that require monitoring by the remote operator H, which are indicated by dotted hatching.

[0176] The area to be monitored is, for example, an area that the remote operator H should monitor while the vehicle 200 is traveling in the accelerator manual traveling mode, and is an area around the vehicle 200. The area to be monitored by the remote operator H may also be an area where the vehicle 200 itself cannot detect an object or where the reliability of object detection is low.

[0177] As shown in (c) of Fig. 7, the monitoring area that needs to be monitored by the remote operator H in response to the switch to the accelerator manual driving mode and auxiliary information related to the driving of the vehicle 200 are displayed on the presentation unit 110. For example, the presentation unit 110 displays the monitoring area and auxiliary information by superimposing them on an image acquired from the vehicle 200. In this way, the monitoring area that needs to be monitored in response to the switch to the accelerator manual driving mode is displayed, so that the remote operator H can easily know the area that he or she should monitor.

[0178] The presentation unit 110 may also display the driving route (dashed line shown in (c) of FIG. 7), the current driving mode ("Manual accelerator" shown in (c) of FIG. 7), the result of determination as to whether driving can be continued ("Driving OK" shown in (c) of FIG. 7), etc. This allows the remote operator H to remotely operate the vehicle 200 by referring to the auxiliary information. Furthermore, since a vehicle 410 is present on the driving route, the presentation unit 110 may display information to warn the driver, such as "Watch out ahead," as an alert for the approaching vehicle 410 or an alert that there is a possibility of a collision with the vehicle 410. The information to warn the driver is included in the auxiliary information. The driving route is an example of a position on the vehicle's driving route. In the example of (c) of FIG. 7, the position on the driving route includes an oncoming lane.

[0179] Also, in (c) of FIG. 7, a button 112 indicating "normal driving" and a button 113 indicating "emergency stop" are displayed as driving mode switching buttons. Normal driving means driving in the autonomous driving mode, and when button 112 is operated, vehicle 200 switches to the automatic driving mode and drives in the autonomous driving mode. In other words, when button 112 is operated, control signal switching instruction unit 133 switches from the current accelerator manual driving mode to the autonomous driving mode. Note that in the state of (c) of FIG. 7, the driving enablement conditions for the autonomous driving mode are not met (driving in the oncoming lane is not possible), so button 112 is in an inoperable state. The inoperable state can also be said to be a state in which operation is invalid. In this way, a button that selects a driving mode that does not meet the driving enablement conditions is controlled so as not to be selectable. In (c) of FIG. 7, inoperable button 112 is indicated by a dashed line.

[0180] For example, the control signal switching instruction unit 133 may determine whether or not the driving enable conditions for the accelerator manual driving mode are satisfied at predetermined time intervals after the accelerator manual driving mode is selected by the remote operator H. Furthermore, for example, the control signal switching instruction unit 133 may determine whether or not the driving enable conditions for the autonomous driving mode are satisfied at predetermined time intervals after the accelerator manual driving mode is selected by the remote operator H.

[0181] Furthermore, the emergency stop button is a button for bringing vehicle 200 to an emergency stop, and when button 113 is operated, vehicle 200 comes to an emergency stop. In other words, when control signal switching unit 240 acquires a signal indicating that button 113 has been operated, control signal switching unit 240 forcibly stops vehicle 200 regardless of the current driving mode. For example, control signal switching unit 240 forcibly stops vehicle 200 regardless of instruction information from control signal switching instruction unit 133.

[0182] When the driving mode is switched to the accelerator manual driving mode and driving in the accelerator manual driving mode begins, the remote operator H performs the parking brake operation and accelerator operation as control operations, and the brake operation, steering operation and presentation operation are controlled by the automatic driving system 230.

[0183] Fig. 7(d) shows a state in which the vehicle is traveling in the manual accelerator driving mode. Fig. 7(d) shows a state in which the vehicle has traveled in the oncoming lane in the manual accelerator driving mode from the state in Fig. 7(c) and has overtaken vehicle 400.

[0184] 7(d), for example, when it is determined that vehicle 200 has overtaken vehicle 400 in the manual accelerator driving mode, returned to its own lane, and is now in a state that satisfies the conditions for driving in the autonomous driving mode, control signal switching instruction unit 133 makes button 112 operable. In FIG. 7(d), operable buttons 112 are indicated by solid lines.

[0185] For example, when button 112 is operated by remote operator H, control signal switching instruction unit 133 switches the driving mode from the accelerator manual driving mode to the autonomous driving mode. Specifically, control signal switching instruction unit 133 outputs instruction information instructing driving in the autonomous driving mode to control signal switching unit 240. Upon acquiring the instruction information, control signal switching unit 240 outputs a control signal from automatic driving system 230 to vehicle control unit 250, thereby switching vehicle 200 to the autonomous driving mode.

[0186] In this way, when the vehicle 200 is unable to travel in the autonomous travel mode, the remote operation device 130 can partially automate vehicle operation by switching to the shared travel mode, thereby reducing the burden on the remote operator H compared to switching to the full remote operation mode. Furthermore, the remote operator H can limit the automatic driving functions and the vehicle functions that each of the remote operator H can control depending on the communication state, vehicle state, response content, and the state around the vehicle, thereby ensuring the safety of the travel of the vehicle 200.

[0187] (Modification of the embodiment) The configuration of the vehicle control system according to this modification will be described with reference to FIG. 8. FIG. 8 is a block diagram showing the functional configuration of the vehicle control system according to this modification. The vehicle control system according to this modification differs from the vehicle control system 10 according to the embodiment mainly in that the vehicle 200a has a remote-controlled edge module 260. Hereinafter, the vehicle control system according to this modification will be described, focusing on the differences from the vehicle control system 10 according to the embodiment. Furthermore, components that are the same as or similar to those in the vehicle control system 10 according to the embodiment will be assigned the same reference numerals as those in the vehicle control system 10 according to the embodiment, and descriptions thereof will be omitted or simplified.

[0188] Of the various components included in the vehicle control system, only the input / output system 100a, the remote control device 130, and the vehicle 200a are shown in FIG.

[0189] As shown in FIG. 8, a vehicle 200a according to this modification does not have the control signal switching unit 240 of the vehicle 200 according to the embodiment, but has a remote-controlled edge module 260 and a driving mode switching unit 270.

[0190] The remote control edge module 260 is a module provided to enable both automatic driving and remote manual driving of the vehicle 200a, which is capable of automatic driving but does not have a function for remote manual driving. The remote control edge module 260 may be attached to the vehicle 200a capable of automatic driving, for example. The remote control edge module 260 is also communicably connected to the vehicle operation unit 120 and the remote control device 130 via a communication network.

[0191] The remote control edge module 260 includes an auxiliary automatic driving system 261 and a control signal switching unit 262.

[0192] The auxiliary automatic driving system 261 executes processing for the vehicle 200a to drive automatically. The auxiliary automatic driving system 261 generates a driving plan for automatic driving based on vehicle information and the like. The driving plan includes a driving route, a speed, and the like. When driving in autonomous driving mode, the auxiliary automatic driving system 261 generates a control signal based on the driving plan and outputs it to the control signal switching unit 262 in order to execute the generated driving plan. The auxiliary automatic driving system 261 is, for example, a system independent of the automatic driving system 230 and has the same functions as the automatic driving system 230.

[0193] The driving plan and control signals generated by the assisted automatic driving system 261 may be the same as the driving plan and control signals generated by the automatic driving system 230, or may be different from each other.

[0194] The assisted automatic driving system 261 may, for example, acquire a driving plan and a control signal from the automatic driving system 230. In other words, the assisted automatic driving system 261 may output a control signal generated by the automatic driving system 230 to the control signal switching unit 262.

[0195] The control signal switching unit 262 has the same function as the control signal switching unit 240 of the embodiment. The control signal switching unit 262 outputs a control signal to the driving mode switching unit 270 to realize driving in accordance with the driving mode determined by the control signal switching instruction unit 133. When driving in the full remote control mode among the remote control modes, the control signal switching unit 262 outputs a control signal from the vehicle operation unit 120 to the driving mode switching unit 270. When driving in the shared driving mode among the remote control modes, the control signal switching unit 262 extracts necessary information from the control signal from the vehicle operation unit 120 and the control signal from the auxiliary automatic driving system 261, and outputs the extracted information to the driving mode switching unit 270. When driving in the accelerator manual driving mode, the control signal switching unit 262 extracts information other than the accelerator and parking brake from the control signal from the auxiliary automatic driving system 261, and information about the accelerator and parking brake from the control signal from the vehicle operation unit 120, and outputs the extracted information to the driving mode switching unit 270 as a control signal.

[0196] Furthermore, when the control signal switching unit 262 acquires instruction information indicating a switch to the remote control mode from the control signal switching instruction unit 133, the control signal switching unit 262 may output the instruction information to the traveling mode switching unit 270.

[0197] Based on the driving mode determined by the control signal switching instruction unit 133, that is, based on the instruction information, the driving mode switching unit 270 outputs a control signal for either the automatic driving system 230 or the remote-operated edge module 260 to the vehicle control unit 250. In other words, the driving mode switching unit 270 switches the control signal to be output to the vehicle control unit 250 depending on whether the driving mode is the autonomous driving mode or the remote-operated mode. The remote-operated mode includes any of the shared driving modes or the complete remote-operated mode.

[0198] When the control signal switching instruction unit 133 determines the driving mode to be the autonomous driving mode, the driving mode switching unit 270 outputs a control signal from the automatic driving system 230 to the vehicle control unit 250. At this time, the driving mode switching unit 270 does not output a control signal from the remote operation edge module 260 to the vehicle control unit 250. Furthermore, when the control signal switching instruction unit 133 determines the driving mode to be the remote operation mode, the driving mode switching unit 270 outputs a control signal from the remote operation edge module 260 to the vehicle control unit 250. At this time, the driving mode switching unit 270 does not output a control signal from the automatic driving system 230 to the vehicle control unit 250.

[0199] As a result, even if the vehicle 200a does not have the function of switching to a remote control mode, simply by attaching the remote control edge module 260, it becomes possible to switch to the remote control mode determined by the remote control device 130 in the vehicle 200a.

[0200] (Other embodiments) The present disclosure has been described above based on the embodiments and modifications (hereinafter also referred to as embodiments, etc.), but the present disclosure is not limited to the above-described embodiments, etc. As long as it does not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art may make to the present embodiments, etc., or forms constructed by combining components of different embodiments, etc., may also be included within the scope of one or more aspects of the present disclosure.

[0201] For example, in the above-described embodiments, the control signal switching instruction unit determines the driving mode based at least on communication information. However, this is not limiting. The control signal switching instruction unit may determine the driving mode without using communication information. The control signal switching instruction unit may determine the driving mode based on at least one of communication information, sensing status, location information, driving information, environmental information, and driving task information. This allows the remote operator's burden of remote control to be reduced and safety to be ensured throughout the system by appropriately allocating vehicle control by the autonomous driving function and vehicle control by the remote operator according to the various conditions, even if the remote operator's burden of remote control increases due to the sensing status of the surroundings of the moving object, the position of the moving object, the driving status of the moving object, the surrounding conditions of the moving object, or the driving task of the moving object. Note that the communication information, sensing status, location information, driving information, environmental information, and driving task information are examples of information related to the moving object.

[0202] In the above-described embodiment, an example has been described in which the shared driving modes include three driving modes: a manual accelerator driving mode, a manual brake driving mode, and a manual steering driving mode. However, the driving modes are not limited to these. For example, a mode in which the moving object speaks to its surroundings, a mode in which autonomous driving continues while the remote operator monitors (approaching pedestrians, approaching vehicles, and approaching bicycles), etc. may be provided instead of or in addition to the above-described three driving modes, and switching to these modes may be performed depending on the load of the remote operator's remote operation.

[0203] Furthermore, in the above embodiments, the shared driving mode is performed by sharing vehicle control between two parties: the automatic driving function equipped in the vehicle and vehicle control by a remote operator; however, it may also be performed by sharing vehicle control between three parties: the automatic driving function equipped in the vehicle, vehicle control by a remote operator, and vehicle control by commands generated by a remote control device.

[0204] In the above-described embodiment, the communication status measurement unit measures the communication status, but the present invention is not limited to this. For example, the communication status measurement unit may acquire information indicating the communication status of the communication network from an external device communicatively connected to the communication network.

[0205] Furthermore, in the above-described embodiment and the like, an example has been described in which the monitoring-required area is displayed on the display unit as shown in FIG. 7(c), but it is not essential that the monitoring-required area be displayed.

[0206] In the above-described embodiments, the presenting unit presents an overhead image, but the presenting unit is not limited to this. For example, the presenting unit may present a 360-degree image, or may recombine the image to present a third-person perspective image. The presenting unit may also present object information (obstacle information) detected by a sensor (for example, an object detection sensor) mounted on the vehicle. The object information includes at least one of the object's position, size, speed, etc.

[0207] In addition, in the above embodiment, an example of a switch from automatic driving to remote manual driving is given as a No judgment in step S30, but this is not limited to this, and it may also be that the remote operator performs an operation indicating a switch from automatic driving to remote manual driving.

[0208] The order of the processes described in the above embodiments is merely an example. The order of the processes may be changed, the processes may be executed in parallel, or some of the processes may not be executed.

[0209] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, an integrated circuit. These components may be integrated individually on a single chip, or some or all of them may be integrated on a single chip. While LSI is used here, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI depending on the level of integration. Furthermore, the integration method is not limited to LSI; it may be implemented using dedicated circuits or general-purpose processors. Field-programmable gate arrays (FPGAs), which can be programmed after LSI fabrication, or reconfigurable processors, which allow the connection or settings of circuit cells within an LSI to be reconfigured, may also be used. Furthermore, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or other derivative technologies, that technology may naturally be used to integrate the components.

[0210] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

[0211] Furthermore, the remote control device provided in the vehicle control system may be realized as a single device or may be realized by multiple devices. For example, each processing unit of the remote control device may be realized by two or more devices. For example, the function of at least one processing unit of the remote control device may be realized by a device mounted on the vehicle. When the remote control device is realized by multiple devices, the components provided in the remote control device may be allocated to the multiple devices in any manner. Furthermore, the communication method between the multiple devices is not particularly limited.

[0212] In the above embodiment, the remote control edge module is mounted on a vehicle, but the present invention is not limited to this. The remote control edge module may be mounted on a remote control device, for example.

[0213] Furthermore, the technology of the present disclosure may be the above-mentioned program, or a non-transitory computer-readable recording medium on which the above-mentioned program is recorded. It goes without saying that the above-mentioned program can be distributed via a transmission medium such as the Internet. For example, the above-mentioned program and a digital signal comprising the above-mentioned program may be transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc. Furthermore, the above-mentioned program and a digital signal comprising the above-mentioned program may be executed by another independent computer system by being recorded on a recording medium and transferred, or by being transferred via a network, etc.

[0214] In the embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. [Industrial Applicability]

[0215] The present disclosure is widely applicable to systems that operate mobile bodies that can travel in autonomous driving mode and remote control mode. [Explanation of symbols]

[0216] 10 Vehicle control system (information processing system) 100 Remote Control System 100a Input / Output System 110 Presentation section 111, 112, 113 buttons 120 Vehicle operation unit 130 Remote Control Device 131 Response Content Judgment Department 132 Determination unit for determining whether or not driving can continue 133 Control signal switching instruction unit 134 Communication status measurement unit 200, 200a, 400, 410 vehicles 210 Environmental Information Acquisition Department 220 Vehicle Information Acquisition Unit 230 Autonomous Driving System 240, 262 control signal switching unit 250 Vehicle control unit 260 Remote Control Edge Module 261 Assisted Automated Driving Systems 270 Driving mode switching section 300 Network 310 Wireless Base Station H Remote Operator L road

Claims

1. An information processing method executed by a computer, Acquires communication information regarding the status of mobile objects that can be driven automatically or remotely, When switching from the automatic operation to the remote manual operation occurs in the moving body, a first operation control to be remotely manually operated among operation controls of the moving body is determined based on communication information regarding a state of the moving body; operating the moving body based on the remote manual operation corresponding to the first motion control, and causing the moving body to execute a second motion control other than the first motion control; the communication information regarding the state of the mobile object is communication information indicating a state of communication between the mobile object and a remote control device for remotely controlling the mobile object via communication, Furthermore, sensing information indicating a sensing state of the surroundings of the moving object is acquired; determining the first operation control based on the communication information and the sensing information; Information processing methods.

2. An information processing method executed by a computer, comprising: Acquires communication information regarding the status of mobile objects that can be driven automatically or remotely, When switching from the automatic operation to the remote manual operation occurs in the moving body, a first operation control to be remotely manually operated among operation controls of the moving body is determined based on communication information regarding a state of the moving body; operating the moving body based on the remote manual operation corresponding to the first motion control, and causing the moving body to execute a second motion control other than the first motion control; the communication information regarding the state of the mobile object is communication information indicating a state of communication between the mobile object and a remote control device for remotely controlling the mobile object via communication, Furthermore, location information indicating the location of the moving body is acquired, determining the first operation control based on the communication information and the location information Information processing methods.

3. An information processing method executed by a computer, comprising: Acquires communication information regarding the status of mobile objects that can be driven automatically or remotely, When switching from the automatic operation to the remote manual operation occurs in the moving body, a first operation control to be remotely manually operated among operation controls of the moving body is determined based on communication information regarding a state of the moving body; operating the moving body based on the remote manual operation corresponding to the first motion control, and causing the moving body to execute a second motion control other than the first motion control; the communication information regarding the state of the mobile object is communication information indicating a state of communication between the mobile object and a remote control device for remotely controlling the mobile object via communication, Furthermore, acquiring movement state information indicating the movement state of the moving body; determining the first operation control based on the communication information and the movement state information Information processing methods.

4. An information processing method executed by a computer, comprising: Acquires communication information regarding the status of mobile objects that can be driven automatically or remotely, When switching from the automatic operation to the remote manual operation occurs in the moving body, a first operation control to be remotely manually operated among operation controls of the moving body is determined based on communication information regarding a state of the moving body; operating the moving body based on the remote manual operation corresponding to the first motion control, and causing the moving body to execute a second motion control other than the first motion control; the communication information regarding the state of the mobile object is communication information indicating a state of communication between the mobile object and a remote control device for remotely controlling the mobile object via communication, Furthermore, environmental information indicating the surrounding environment of the moving body is acquired, determining the first operation control based on the communication information and the environmental information Information processing methods.

5. An information processing method executed by a computer, comprising: Acquires communication information regarding the status of mobile objects that can be driven automatically or remotely, When switching from the automatic operation to the remote manual operation occurs in the moving body, a first operation control to be remotely manually operated among operation controls of the moving body is determined based on communication information regarding a state of the moving body; operating the moving body based on the remote manual operation corresponding to the first motion control, and causing the moving body to execute a second motion control other than the first motion control; the communication information regarding the state of the mobile object is communication information indicating a state of communication between the mobile object and a remote control device for remotely controlling the mobile object via communication, Furthermore, acquiring movement task information indicating a movement task for movement of the moving body; determining the first motion control based on the communication information and the movement task information Information processing methods.

6. The first operation control is determined based on a correspondence relationship between the communication state and the operation control of the mobile object for which remote manual operation is permitted. The information processing method according to any one of claims 1 to 5.

7. In determining the first operation control, the better the state of the communication, the more the permitted remote manual operations increase, become more complex, or become more advanced. The information processing method according to claim 6.

8. The communication state includes a communication delay or a communication band state between the mobile object and the remote control device. The information processing method according to any one of claims 1 to 7.

9. In determining the first operation control, the first operation control is determined based on a correspondence relationship between the communication state, the sensing state, and the operation control of the moving object for which the remote manual operation is permitted, such that the better the sensing state, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced. The information processing method according to claim 1 .

10. The sensing state includes processing delays or missing image data, sound data, or point cloud data. The information processing method according to claim 9.

11. In determining the first operation control, the first operation control is determined based on a correspondence relationship between the state of communication and the location of the mobile body, and the operation control of the mobile body for which the remote manual operation is permitted, such that the lower the safety of the autonomous driving of the mobile body in the location of the mobile body, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced. The information processing method according to claim 2 .

12. The location of the moving object includes a position of the moving object or a position on a moving route of the moving object. The information processing method according to claim 11.

13. In determining the first operation control, the first operation control is determined based on a correspondence relationship between the state of communication and the movement state of the mobile body and the operation control of the mobile body for which the remote manual operation is permitted, such that the higher the safety of the mobile body in the movement state of the mobile body, the more the number, complexity, or sophistication of the remote manual operations permitted increases. The information processing method according to claim 3 .

14. The moving state of the moving object includes the speed of the moving object. The information processing method according to claim 13.

15. In determining the first operation control, the first operation control is determined based on a correspondence relationship between the communication state and the surrounding environment of the mobile body and the operation control of the mobile body for which the remote manual operation is permitted, such that the lower the safety of the autonomous driving of the mobile body in the surrounding environment of the mobile body, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced. The information processing method according to claim 4.

16. The surrounding environment of the moving object includes the state of obstacles, the state of the road surface, or the weather. The information processing method according to claim 15.

17. In determining the first operation control, the first operation control is determined based on a correspondence relationship between the communication state and the movement task and the operation control of the moving object for which the remote manual operation is permitted, such that the higher the complexity of the movement task, the more the permitted remote manual operation increases, becomes more complex, or becomes more advanced. The information processing method according to claim 5 .

18. The movement task includes going straight, turning right or left, avoiding parking, backing up, or pulling over to the shoulder of the road.

18. The information processing method according to claim 17.

19. an acquisition unit that acquires communication information regarding the state of a mobile body that can be automatically driven and remotely driven; a determination unit that, when switching from the automatic operation to the remote manual operation occurs in the moving body, determines a first operation control to be remotely manually operated among operation controls of the moving body based on communication information related to a state of the moving body; a switching instruction unit that operates the moving body based on the remote manual operation corresponding to the first operation control and causes the moving body to execute a second operation control other than the first operation control, the communication information regarding the state of the mobile object is communication information indicating a state of communication between the mobile object and a remote control device for remotely controlling the mobile object via communication, The acquisition unit further acquires sensing information indicating a sensing state of the surroundings of the moving object; The determination unit determines the first operation control based on the communication information and the sensing information. Information processing system.

Images