Computer program, remote control device, and remote control method

Abstract

To provide a computer program, a remote control device and a remote control method that can monitor a control state on vehicle side and a control state on remote control side and determine whether to permit switching depending on the difference between them.SOLUTION: With respect to a vehicle that is capable of traveling via autonomous control and via remote control, a computer is caused to execute processing for identifying a control state of the vehicle when autonomous control is being executed and a control state of the vehicle when autonomous control is switched to remote control, and determining whether or not to permit switching from the autonomous control to the remote control on the basis of a difference between the identified two control states.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a computer program, a remote control device, and a remote control method. [Background technology]

[0002] Conventional remote driving systems for vehicles often use steering wheels, accelerator pedals, brake pedals, and other control components that mimic real vehicles in order to make it easier to adapt operating techniques. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-5123 A Summary of the Invention [Problem to be solved by the invention]

[0004] In many cases, with these operating members, the physical state of the operating member (the rotation angle of the steering wheel or the amount of depression of the pedal) corresponds one-to-one with the control parameter.

[0005] For this reason, if the control right is handed over to the remote control side in a state different from the control state on the vehicle side, sudden steering or sudden acceleration or deceleration may occur.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide a computer program, a remote control device, and a remote control method that can monitor the control state on the vehicle side and the control state on the remote control side, and determine whether or not to allow switching based on the difference between the two. [Means for solving the problem]

[0007] A computer program according to one embodiment of the present invention is a computer program for causing a computer to execute a process of identifying, for a vehicle capable of running under autonomous control and remote control, a control state of the vehicle when the autonomous control is being executed and a control state of the vehicle when switching from the autonomous control to the remote control, and determining whether or not to allow switching from the autonomous control to the remote control based on the difference between the two identified control states.

[0008] A remote control device according to one embodiment of the present invention includes, for a vehicle capable of running under autonomous control and remote control, an identification unit that identifies a control state of the vehicle when the autonomous control is being executed and a control state of the vehicle when switching from the autonomous control to the remote control, a judgment unit that determines whether or not to permit switching from the autonomous control to the remote control based on the difference between the two identified control states, and a control execution unit that executes the remote control instead of the autonomous control when it is determined that the switching is permitted.

[0009] A remote control method according to one embodiment of the present invention uses a computer to identify, for a vehicle capable of running under both autonomous control and remote control, the control state of the vehicle when the autonomous control is being executed and the control state of the vehicle when switching from the autonomous control to the remote control, and based on the difference between the two identified control states, determines whether or not to allow switching from the autonomous control to the remote control, and if it determines that the switching is permitted, executes the remote control instead of the autonomous control. Effect of the Invention

[0010] According to the present invention, the control state on the vehicle side and the control state on the remote control side are monitored, and it is possible to determine whether or not to permit switching based on the difference between the two. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing an overall configuration of a remote control system. [Diagram 2] FIG. 2 is a block diagram illustrating a configuration of a control system of the vehicle. [Diagram 3] FIG. 2 is a block diagram illustrating a configuration of a remote control device. [Figure 4] FIG. 4 is a schematic diagram showing an example of a monitoring screen. [Diagram 5] FIG. 2 is a schematic diagram showing an example of a steering indicator. [Figure 6] FIG. 2 is a schematic diagram showing an example of a vehicle speed indicator. [Figure 7] FIG. 2 is a schematic diagram showing an example of a pedal indicator. [Figure 8] FIG. 13 is a schematic diagram illustrating an example of a delay indicator. [Figure 9] 13 is a flowchart showing a procedure for measuring the amount of delay. [Figure 10] 11 is a flowchart illustrating a procedure for switching from autonomous control to remote control. [Figure 11] 11 is a flowchart illustrating a procedure for switching from remote control to autonomous control. [Figure 12] 11 is a flowchart illustrating a switching procedure in the second embodiment. [Figure 13] FIG. 13 is an explanatory diagram illustrating a procedure for gradually changing a control ratio. [Figure 14] 13 is a flowchart illustrating a control procedure in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the present invention will be described in detail with reference to the drawings showing embodiments thereof. (Embodiment 1) 1 is a schematic diagram showing the overall configuration of a remote control system. The remote control system according to this embodiment includes a vehicle 1 and a remote control device 2 that remotely controls the vehicle 1 via a communication network NW. The communication network NW is composed of communication lines such as 3G, 4G, 5G, LTE (Long Term Evolution), WAN (Wide Area Network), LAN (Local Area Network), Internet lines, leased lines, and satellite lines, as well as communication facilities such as base stations.

[0013] The vehicle 1 is a vehicle that runs using power from a drive source. As the drive source of the vehicle 1, an internal combustion engine such as a gasoline engine or a diesel engine, an electric motor that operates using power supplied from a secondary battery or a generator, or a combination of these can be used. An example of the vehicle 1 is a four-wheeled passenger car. The vehicle 1 is not limited to a four-wheeled passenger car, and may be a two-wheeled, three-wheeled, or other vehicle, or may be an unmanned vehicle.

[0014] The vehicle 1 in this embodiment is a vehicle capable of traveling by autonomous control and remote control. Here, the autonomous control refers to generating a control command required for traveling inside the vehicle according to the traveling situation of the vehicle, and controlling the traveling of the vehicle based on the generated control command. The autonomous control may be a fully autonomous control that completely controls the traveling of the vehicle, or a semi-autonomous control that partially controls the traveling of the vehicle. The remote control refers to monitoring the traveling situation of the vehicle 1 by the remote control device 2, generating a control command required for traveling inside the remote control device 2 according to the monitored traveling situation of the vehicle 1, and transmitting the generated control command to the vehicle 1 via the communication network NW to control the traveling of the vehicle 1. The remote control may be a complete control that executes all control related to the traveling of the vehicle 1, or a partial control that partially controls the traveling of the vehicle 1.

[0015] In this embodiment, the vehicle 1 is a vehicle capable of running under autonomous control and remote control, but manual control by a passenger is not excluded. That is, the vehicle 1 may receive a driving operation by a passenger while the autonomous control or remote control is being executed, or while the autonomous control and remote control are not being executed, and the running may be controlled according to the received driving operation. For this reason, the vehicle 1 may be equipped with a steering wheel 11 that receives a steering operation by the passenger, an accelerator pedal 12 that receives an accelerator operation, a brake pedal 13 that receives a brake operation, and the like.

[0016] The remote control device 2 is a control device for remotely controlling the traveling of the vehicle 1 via the communication network NW. The remote control device 2 receives a remote driving operation by an operator, generates a control command for the vehicle 1 in response to the received remote driving operation, and transmits the generated control command to the vehicle 1 via the communication network NW, thereby remotely controlling the traveling of the vehicle 1.

[0017] The remote control device 2 includes a remote controller 210 that accepts remote driving operations by an operator. The remote controller 210 includes, for example, a steering controller 211, an accelerator controller 212, and a brake controller 213. The steering controller 211 includes, for example, a handle unit 211a configured to accept an operation simulating the steering wheel 11 of the vehicle 1, and is configured to accept a remote operation related to the steering of the vehicle 1 by accepting an operation (rotation operation) at the handle unit 211a. Similarly, the accelerator controller 212 includes, for example, an accelerator pedal unit 212a configured to accept an operation simulating the accelerator pedal 12 of the vehicle 1, and is configured to accept a remote operation related to the acceleration and deceleration of the vehicle 1 by accepting an operation (depression operation) at the accelerator pedal unit 212a. In addition, the brake controller 213 is provided with a brake pedal unit 213a configured to accept an operation simulating the brake pedal 13 of the vehicle 1, and is configured to accept remote control related to braking of the vehicle 1 by accepting an operation (depressing operation) in the brake pedal unit 213a.

[0018] The remote control device 2 also includes a monitor 220 that displays various information. The monitor 220 includes a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The remote control device 2 acquires information indicating the control state of the vehicle 1, a field of view image of the vehicle 1 obtained by the in-vehicle camera 116 (see FIG. 2), etc. via the communication network NW, and displays the acquired information indicating the control state, the field of view image of the vehicle 1, etc. on the monitor 220.

[0019] An operator of the remote control device 2 can remotely control the vehicle 1 with the same operational feeling as if he or she were driving the vehicle 1 by operating the remote controller 210 while viewing the field of view image displayed on the monitor 220.

[0020] In the remote control, for example, the following steps are executed. (1) The surroundings of the vehicle 1 are captured using the on-board camera 116. (2) The video data output by the vehicle-mounted camera 116 is encoded so as to conform to a transmission format in the communication network NW. (3) The encoded video data is transmitted from the vehicle 1 to the remote control device 2. (4) The remote control device 2 decodes the received video data. (5) Based on the decoded video data, a field of view image of vehicle 1 is displayed on monitor 220. (6) The operator who recognizes the visual field image performs remote driving operations using the remote controller 210. (7) The remote control device 2 generates a control command in response to a remote driving operation by an operator. (8) The generated control command is encoded so as to conform to a transmission format in the communication network NW. (9) The encoded control command is transmitted from the remote control device 2 to the vehicle 1. (10) Decode the control command received in vehicle 1. (11) Controlling the running of vehicle 1 based on the decoded control command.

[0021] On the other hand, in the autonomous control, for example, a control command for controlling driving can be generated inside the vehicle 1 by analyzing a field of view image output from the in-vehicle camera 116. That is, the vehicle 1 can generate a control command at the point in time when the above-mentioned procedure (1) is executed. In contrast, in the remote control, the above-mentioned procedures (1) to (11) need to be executed, and the vehicle 1 needs to wait for a control command to be transmitted from the remote control device 2. For this reason, a time lag (amount of delay) occurs between the autonomous control and the remote control via the communication network NW due to communication processing, image processing, and the like. The time lag is, for example, about 200 msec to 1 sec.

[0022] Since driving control is possible by both autonomous control and remote control, and there is a time lag between the autonomous control and the remote control, when switching from the autonomous control to the remote control, it is considered that a difference occurs between the control state by the autonomous control immediately before switching and the control state by the remote control immediately after switching. If this difference is large, the driving state of the vehicle 1 may change suddenly when switching from the autonomous control to the remote control. For example, if there is a large difference between the steering control by the autonomous control and the steering control by the remote control, there is a possibility that the steering wheel will be turned suddenly when switching from the autonomous control to the remote control. In addition, if there is a large difference between the acceleration / deceleration control by the autonomous control and the acceleration / deceleration control by the remote control, there is a possibility that sudden acceleration or sudden deceleration will occur when switching from the autonomous control to the remote control. Furthermore, if there is a large difference between the brake control by the autonomous control and the brake control by the remote control, there is a possibility that sudden braking or sudden start will occur when switching from the autonomous control to the remote control. As a result, there is a possibility that the occupants of the vehicle 1 will feel uncomfortable and that the surrounding vehicles will be affected.

[0023] Therefore, the remote control device 2 according to this embodiment monitors the control state of the vehicle 1 under autonomous control, and judges whether or not to permit switching from autonomous control to remote control based on the difference between the control state under autonomous control and the control state under remote control. If the remote control device 2 judges that the difference between autonomous control and remote control is small, it permits switching from autonomous control to remote control, and if the difference between the control state under autonomous control and the control state under remote control is large, it prohibits switching from autonomous control to remote control.

[0024] 2 is a block diagram illustrating the configuration of a control system of the vehicle 1. The vehicle 1 includes an on-board control device 100 that executes various controls related to the vehicle 1 based on information input from the outside. The on-board control device 100 is, for example, an ECU (Electronic Control Unit) or a BCM (Body Control Module). The on-board control device 100 includes a control unit 101, a storage unit 102, an input unit 103, an external vehicle communication unit 104, an internal vehicle communication unit 105, and the like.

[0025] The control unit 101 includes a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like. The CPU in the control unit 101 refers to information stored in the storage unit 102 and executes a control program stored in the ROM and a computer program stored in the storage unit 102, which will be described later, to control the operation of the above hardware and execute various controls related to the vehicle 1. The RAM in the control unit 101 stores various data generated during the execution of the control program. The control unit 101 may also include a clock that outputs date and time information, a timer that measures elapsed time, a counter that counts numbers, and the like.

[0026] The storage unit 102 is configured with a non-volatile memory such as an EEPROM (Electronically Erasable Programmable Read Only Memory), and stores computer programs executed by the control unit 101, various information necessary for control, etc. The computer programs stored in the storage unit 102 include an autonomous control program that autonomously controls the traveling of the vehicle 1, a remote control program that controls the traveling of the vehicle 1 according to instructions from the remote control device 2, etc.

[0027] The input unit 103 includes an interface for connecting various sensors and devices, and acquires information output from the connected sensors and devices. The sensors and devices connected to the input unit 103 include, for example, a steering angle sensor 111, an accelerator sensor 112, a brake sensor 113, a vehicle speed sensor 114, an acceleration sensor 115, an in-vehicle camera 116, an obstacle detection sensor 117, a GPS (Global Positioning System) receiver 118, and the like.

[0028] The steering angle sensor 111 is a sensor for detecting a steering angle (a rotation angle of the steering wheel 11 ), and outputs information indicating the detected steering angle to the on-board control device 100 .

[0029] The accelerator sensor 112 is a sensor for detecting the accelerator opening (the depression amount of the accelerator pedal 12), and outputs information indicating the detected accelerator opening to the on-board control device 100.

[0030] The brake sensor 113 is a sensor for detecting a brake stroke (amount of depression of the brake pedal 13), and outputs information indicating the detected brake stroke to the on-board control device 100.

[0031] The vehicle speed sensor 114 is a sensor for measuring the traveling speed (vehicle speed) of the vehicle 1, and outputs information indicating the measured vehicle speed to the on-board control device 100.

[0032] The acceleration sensor 115 is a sensor for measuring the acceleration of the vehicle 1, and outputs information indicating the measured acceleration to the on-board control device 100.

[0033] The in-vehicle camera 116 is an imaging device for capturing images of the periphery of the vehicle 1 and generating a field of view image (video data), and includes, for example, a complementary metal oxide semiconductor (CMOS) or a charge-coupled device (CCD). The in-vehicle camera 116 is provided at an appropriate position on the vehicle 1 so that its field of view includes the front, sides, rear, etc. of the vehicle 1. The in-vehicle camera 116 outputs video data obtained by capturing images of the periphery of the vehicle 1 to the in-vehicle control device 100.

[0034] The obstacle detection sensor 117 is a sensor for detecting an obstacle present around the vehicle 1. The obstacle detection sensor 117 includes, for example, an irradiation unit that irradiates a millimeter wave radar or an ultrasonic radar to the front (or side or rear) of the vehicle 1, a receiving unit that receives a reflected wave reflected by an object, and detects an obstacle present around the vehicle 1 by measuring the distance and angle to the object present around the vehicle 1, and the relative speed and relative acceleration with respect to the object. When the obstacle detection sensor 117 detects an obstacle present around the vehicle 1, it outputs information such as the distance and angle to the obstacle, and the relative speed and relative acceleration with respect to the obstacle to the on-board control device 100.

[0035] The GPS receiver 118 is a device for receiving radio waves transmitted from a GPS satellite (not shown) and locating the current position of the vehicle 1 based on the received radio waves. The GPS receiver 118 outputs information related to the located current position of the vehicle 1 to the on-board control device 100.

[0036] The vehicle exterior communication unit 104 includes a communication module such as a DCM (Data Communication Module), an antenna 104a for transmitting and receiving wireless signals, and performs wireless communication with external devices including the remote control device 2 via the communication network NW.

[0037] The in-vehicle communication unit 105 includes a communication interface that connects to an in-vehicle communication line CN. As the in-vehicle communication line CN, a controller area network (CAN) or a local interconnect network (LIN) can be used. In addition to the in-vehicle control device 100, the engine ECU 120, the transmission ECU 130, the brake ECU 140, the steering ECU 150, the vehicle light ECU 160, and the like are connected to the in-vehicle communication line CN.

[0038] The engine ECU 120 controls the operation of the engine mounted on the vehicle 1 based on a control signal received from the on-board control device 100 via the in-vehicle communication line CN. The engine ECU 120 controls, for example, the operation of a throttle actuator that opens and closes an electronic throttle valve provided in an intake pipe of the engine, and adjusts the opening of the electronic throttle valve so that the engine speed becomes a target speed.

[0039] When the driving source of the vehicle 1 is an electric motor, a motor ECU is connected to the in-vehicle communication line CN instead of the engine ECU 120. The motor ECU controls the operation of the electric motor, which is the driving source, based on a control signal from the in-vehicle control device 100 received via the in-vehicle communication line CN.

[0040] The transmission ECU 130 controls the operation of the transmission mounted on the vehicle 1 based on a control signal received from the on-board control device 100 via the in-vehicle communication line CN. The transmission ECU 130 switches the gear ratio of the transmission, for example, by controlling the operation of a hydraulic control device that adjusts the hydraulic pressure of the hydraulic oil supplied to the transmission. The transmission is connected to the output shaft of the engine and is configured to change the rotation speed and torque transmitted from the engine.

[0041] The brake ECU 140 controls the operation of the brake devices provided on each wheel of the vehicle 1 based on a control signal received from the on-board control device 100 via the in-vehicle communication line CN, thereby braking the vehicle 1.

[0042] The steering ECU 150 controls the operation of the steering wheel provided in the vehicle 1 based on a control signal received from the on-board control device 100 via the in-vehicle communication line CN. The steering ECU 150 is connected to, for example, a steering actuator that applies an assist force to the steering wheel, and controls the operation of the steering actuator based on the control signal from the on-board control device 100 so as to output an assist torque required for steering the vehicle 1.

[0043] The vehicle light ECU 160 controls the operation of headlamps, tail lamps, hazard lamps, and the like equipped on the vehicle 1 based on a control signal received from the in-vehicle control device 100 via the in-vehicle communication line CN. The vehicle lights controlled by the vehicle light ECU 160 may include a projection lamp that projects a specific detection pattern.

[0044] The control unit 101 of the on-board control device 100 reads out and executes an autonomous control program stored in the storage unit 102, and performs autonomous control of the traveling of the vehicle 1 by outputting control signals to the various ECUs 120-160 based on various information input through the input unit 103. The control unit 101 also reads out and executes a remote control program stored in the storage unit 102, and performs remote control of the traveling of the vehicle 1 by outputting control signals to the various ECUs 120-160 based on instructions from the remote control device 2 received through the exterior communication unit 104.

[0045] 3 is a block diagram illustrating the configuration of the remote control device 2. The remote control device 2 is configured by a dedicated or general-purpose computer, and includes a control unit 201, a storage unit 202, a communication unit 203, an input unit 204, and an output unit 205.

[0046] The control unit 201 includes a CPU, a ROM, a RAM, etc. The ROM included in the control unit 201 stores a control program for controlling the operation of each of the above hardware units. The CPU in the control unit 201 executes the control program stored in the ROM and various programs stored in a storage unit 202 (described later) and controls the operation of each of the above hardware units, thereby causing the entire device to function as the remote control device of the present application. The RAM included in the control unit 201 stores data that is temporarily used during the execution of the various programs.

[0047] Note that the control unit 201 is not limited to the above configuration, and may be one or more processing circuits or control circuits including a single-core CPU, a multi-core CPU, a GPU (Graphics Processing Unit), a microcomputer, a volatile or non-volatile memory, etc. The control unit 201 may also have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when an instruction to start measurement is given until when an instruction to end measurement is given, and a counter that counts numbers.

[0048] The storage unit 202 is configured with an SSD (Solid State Drive), a hard disk device, etc. The storage unit 202 stores various programs to be executed by the control unit 201. The programs stored in the storage unit 202 include a computer program for executing remote control of the vehicle 1, a computer program for measuring a delay amount related to remote control, a computer program for executing control related to switching between autonomous control and remote control of the vehicle 1, and the like.

[0049] The computer program stored in the storage unit 202 may be provided by a recording medium M on which the computer program is readably recorded. The recording medium M is, for example, a portable memory such as a Secure Digital (SD) card, a micro SD card, or a Compact Flash (registered trademark). In this case, the control unit 201 may read the computer program from the recording medium M using a reading device (not shown) and install the read computer program in the storage unit 202. The computer program stored in the storage unit 202 may be provided by communication via the communication unit 203. In this case, the control unit 201 may obtain the computer program through the communication unit 203 and install the obtained computer program in the storage unit 202.

[0050] The communication unit 203 includes a communication interface for connecting to the communication network NW. When information to be notified to the vehicle 1 is input from the control unit 201, the communication unit 203 transmits the information via the communication network NW. When the communication unit 203 receives information transmitted from the vehicle 1 via the communication network NW, the communication unit 203 outputs the received information to the control unit 201.

[0051] The input unit 204 includes a connection interface for connecting the remote controller 210. Information relating to the amount of operation of the connected remote controller 210 is input to the input unit 204. Specifically, for the steering controller 211, information relating to the rotation angle of the handle unit 211a is input. For the accelerator controller 212, information relating to the amount of depression of the accelerator pedal unit 212a is input. For the brake controller 213, information relating to the amount of depression of the brake pedal unit 213a is input. The input information relating to the amount of operation is output to the control unit 201.

[0052] In addition to the remote controller 210, appropriate input devices such as a keyboard, a mouse, and a touch panel may be connected to the input unit 204.

[0053] The output unit 205 includes a connection interface for connecting the monitor 220. The control unit 201 generates screen data including information to be notified to an operator based on information acquired through the communication unit 203 and the input unit 204. The control unit 201 outputs the generated screen data from the output unit 205 to the monitor 220, thereby displaying a desired screen on the monitor 220. For example, the remote control device 2 monitors the control state of the vehicle 1, and displays the monitoring screen on the monitor 220 by outputting screen data of the monitoring screen from the output unit 205.

[0054] In addition to the monitor 220, an appropriate output device such as a speaker may be connected to the output unit 205.

[0055] 4 is a schematic diagram showing an example of a monitoring screen 500. The monitoring screen 500 shown in FIG. 4 is composed of, for example, a field of view image display field 510, a control status display field 520, a steering indicator 530, a vehicle speed indicator 540, and a delay indicator 550.

[0056] The field of view image display field 510 displays a field of view image of the vehicle 1 captured by the vehicle-mounted camera 116. The control unit 201 of the remote control device 2 acquires the video data output from the vehicle-mounted camera 116 through communication, and displays the field of view image of the vehicle 1 in the field of view image display field 510 based on the acquired video data. Since the control unit 201 acquires the video data output from the vehicle-mounted camera 116 through communication, the field of view image display field 510 displays not a real-time field of view image, but a field of view image from a point in time back corresponding to a time lag.

[0057] 4 shows a field of view image captured in front of the vehicle 1, but a field of view image captured in the side or rear of the vehicle 1 may be divided and displayed in the field of view image display field 510. The control unit 201 may also display field of view images captured in the front, rear, and side of the vehicle 1 in the field of view image display field 510 in sequence.

[0058] The control status display field 520 displays information regarding the control state of the vehicle 1. When the control unit 201 of the remote control device 2 is not remotely controlling the vehicle 1 but is monitoring the control state of the vehicle 1, the control unit 201 displays text information such as "remote monitoring in progress" in the control status display field 520. When the control unit 201 is in the process of switching from autonomous control to remote control of the vehicle 1, the control unit 201 displays text information such as "switching to remote operation in progress" in the control status display field 520. When the control unit 201 is remotely controlling the vehicle 1, the control unit 201 displays text information such as "remote operation in progress" in the control status display field 520. In addition, the control unit 201 may display text information such as "speed control possible" in the control status display field 520 to indicate a state in which only the target value of the vehicle speed can be set. In the example of FIG. 4, the control state is displayed by text information, but the control state may be displayed by an icon or color determined according to the control state.

[0059] The steering indicator 530 displays information related to the operation state of the steering wheel 11 in the vehicle 1 and the operation state of the handle portion 211a in the remote controller 210. Fig. 5 is a schematic diagram showing an example of the steering indicator 530. The steering indicator 530 is, for example, a rectangular indicator that is long in the horizontal direction, and includes a steering bar 531, a center line 532, a vehicle-side indicator 533, a remote control-side indicator 534, and an allowable range indicator 535.

[0060] The steering bar 531 is a display bar that indicates the rotation range of the steering wheel 11. The right side of the steering bar 531 extending horizontally from the center line 532 indicates a state in which the steering wheel 11 is rotated to the right (clockwise), and the left side of the center line 532 indicates a state in which the steering wheel 11 is rotated to the left (counterclockwise). The right end of the steering bar 531 indicates a position where the steering wheel 11 is rotated to the right to the maximum. For example, if the steering wheel 11 can rotate two times to the right, the right end of the steering bar 531 corresponds to a position where the rotation angle of the steering wheel 11 is +720 degrees. Similarly, the left end of the steering bar 531 indicates a position where the steering wheel 11 is rotated to the left to the maximum. For example, if the steering wheel 11 can rotate two times to the left, the rotation angle of the steering wheel 11 corresponds to a position where the rotation angle of the steering wheel 11 is -720 degrees. The center line 532 indicates a state in which the steering wheel 11 is not rotated (a position where the rotation angle of the steering wheel 11 is 0 degrees). In this embodiment, the rotational position of the handlebar 11 is displayed on the linear steering bar 531, so that it is possible to clearly distinguish and display states that are difficult to distinguish from the appearance of the handlebar 11, such as 0 degrees, 360 degrees, and 720 degrees.

[0061] The vehicle-side indicator 533 is an indicator that indicates the rotational position of the vehicle-side steering wheel 11. The rotational angle of the steering wheel 11 is measured by the steering angle sensor 111 of the vehicle 1. The control unit 201 of the remote control device 2 acquires information on the rotational angle of the steering wheel 11 through communication with the vehicle 1, and moves the vehicle-side indicator 533 in the left-right direction based on the acquired information on the rotational angle of the steering wheel 11. Since the control unit 201 acquires information on the rotational angle of the steering wheel 11 through communication, the rotational position of the steering wheel 11 indicated by the vehicle-side indicator 533 indicates the rotational position of the steering wheel 11 at a point back in time corresponding to the time lag.

[0062] The remote control side indicator 534 is an indicator that indicates the rotational position of the handle unit 211a on the remote control side. Information on the rotation angle, which is the amount of operation of the handle unit 211a, is input to the remote control device 2 through the input unit 204. The control unit 201 acquires the information on the rotation angle of the handle unit 211a from the input unit 204, and moves the remote control side indicator 534 in the left and right directions based on the acquired information on the rotation angle of the handle unit 211a. The rotational position of the handle unit 211a indicated by the remote control side indicator 534 indicates the rotational position of the handle unit 211a in approximately real time.

[0063] The allowable range indicator 535 indicates the allowable range of error between the rotation angle of the handle 11 on the vehicle side and the rotation angle of the handle unit 211a on the remote control side. In the example of Fig. 5, the allowable range of error is indicated by a rectangular area. The allowable range of error may be a preset constant range (for example, 5 degrees in each of the left and right directions), or may be a range that varies according to an index that indicates the difficulty of driving control, such as a time lag or a vehicle speed. In the latter case, for example, the allowable range of error may be set wider as the time lag is smaller or the vehicle speed is lower, and narrower as the time lag is larger or the vehicle speed is higher.

[0064] In the example of FIG. 5, a range of allowable error is set for the rotation angle of the handle 11 on the vehicle side. That is, the rectangular area showing the allowable range indicator 535 is configured to move following the movement of the vehicle side indicator 533. In this embodiment, if the rotation angle of the handle unit 211a shown by the remote control side indicator 534 is within the range of the rectangular area shown by the allowable range indicator 535, switching between autonomous control and remote control is permitted, and if it is not within the range, switching between autonomous control and remote control is prohibited. If the rotation angle of the handle unit 211a on the remote control side is outside the range of the rectangular area shown by the allowable range indicator 535, the control unit 201 may display guidance information on the monitoring screen 500 that guides the operation of the handle unit 211a so that the rotation angle of the handle unit 211a falls within the range of the allowable error.

[0065] 5 shows an allowable range indicator 535 that moves following the movement of the vehicle-side indicator 533, but an allowable range indicator that moves following the remote control-side indicator 534 may also be used. In this case, if the rotation position of the steering wheel 11 indicated by the vehicle-side indicator 533 falls within the range of the allowable range indicator, switching between autonomous control and remote control is permitted, and if it does not fall within the range, switching between autonomous control and remote control is prohibited. In addition, if the rotation position of the steering wheel 111 on the vehicle side is outside the range of the rectangular area indicated by the allowable range indicator, the control unit 201 may display guidance information on the monitoring screen 500 that guides the operation of the steering wheel unit 211a so that the rotation position of the steering wheel 11 falls within the allowable error range.

[0066] 5 is merely one example of the steering indicator 530. As the steering indicator, other display methods may be used as long as they can display the rotation angle of the handle 11 on the vehicle side, the rotation angle of the handle portion 211a on the remote control side, and the difference between them. For example, in order to clearly distinguish between the vehicle side indicator 533 and the remote control side indicator 534, an icon or text information for distinguishing between them may be added.

[0067] The vehicle speed indicator 540 constituting the monitoring screen 500 displays information related to the vehicle speed of the vehicle 1. Fig. 6 is a schematic diagram showing an example of the vehicle speed indicator 540. The vehicle speed indicator 540 is, for example, a vertically long rectangular indicator, and includes a vehicle speed bar 541, a vehicle speed display field 542, a vehicle-side indicator 543, a remote control-side indicator 544, and an allowable range indicator 545.

[0068] The vehicle speed bar 541 is a display bar for displaying the actual measured value of the vehicle speed as a level. The vehicle speed is measured by a vehicle speed sensor 114 mounted on the vehicle 1. The control unit 201 of the remote control device 2 acquires information on the vehicle speed measured by the vehicle speed sensor 114 through communication with the vehicle 1. The control unit 201 displays a level according to the vehicle speed on the vehicle speed bar 541.

[0069] The vehicle speed display field 542 is a display field that displays the actual vehicle speed as text information. The control unit 201 of the remote control device 2 displays the vehicle speed information acquired through communication with the vehicle 1 in the vehicle speed display field 542 as text information.

[0070] The vehicle-side indicator 543 is an indicator that indicates a target value of the vehicle speed set by the autonomous control of the vehicle 1. When the in-vehicle control device 100 mounted on the vehicle 1 is executing the autonomous control of the vehicle 1, the in-vehicle control device 100 sets the target speed of the vehicle 1 as needed based on various information input from the input unit 103. The control unit 201 of the remote control device 2 acquires information on the target speed set by the in-vehicle control device 100 through communication with the vehicle 1. The control unit 201 moves the vehicle-side indicator 543 in the up and down directions according to the target speed on the vehicle side.

[0071] The remote control indicator 544 is an indicator that indicates a target value of the vehicle speed set by remote control of the vehicle 1. When remote control of the vehicle 1 is being executed, the remote control device 2 sets the target speed of the vehicle 1 as needed based on various information acquired by communication with the vehicle 1. The control unit 201 of the remote control device 2 moves the remote control indicator 544 up and down according to the set target speed.

[0072] The allowable range indicator 545 indicates the allowable range of error between the target speed set on the vehicle side and the target speed set on the remote control side. The allowable range of error may be a preset constant range (e.g., ±5 km / h) or may be a range that varies depending on the time lag, the actual measured value of the vehicle speed, etc. In the latter case, for example, the allowable range of error may be set narrower as the time lag increases or the vehicle speed increases, and may be set wider as the time lag decreases or the vehicle speed decreases.

[0073] 6, a range of allowable error is set for the target speed on the vehicle side. That is, the allowable range indicated by the allowable range indicator 545 is configured to move in accordance with the target speed set on the vehicle side. In this embodiment, if the target speed indicated by the remote control side indicator 544 is within the range indicated by the allowable range indicator 545, Switching between autonomous control and remote control is permitted, and if it is not within the range, switching between autonomous control and remote control is prohibited. In addition, if the target speed on the remote control side is outside the range of the rectangular area indicated by the allowable range indicator 545, the control unit 201 may display guidance information on the monitoring screen 500 that guides the operation of the accelerator pedal unit 212a and the brake pedal unit 123 so as to fall within the allowable error range.

[0074] 6 shows an allowable range indicator 545 that moves following the movement of the vehicle-side indicator 543, but an allowable range indicator that moves following the remote control-side indicator 544 may also be used. In this case, if the target speed indicated by the vehicle-side indicator 543 falls within the range of the allowable range indicator, switching between autonomous control and remote control is permitted, and if it does not fall within the range, switching between autonomous control and remote control is prohibited. If the target speed on the vehicle side is outside the range of the rectangular area indicated by the allowable range indicator, the control unit 201 may display guidance information on the monitoring screen 500 that guides the operation of the accelerator pedal unit 212a and the brake pedal unit 123 so that the target speed on the vehicle side falls within the allowable error range.

[0075] 6 is merely one example of the vehicle speed indicator 540. As the vehicle speed indicator, other display methods may be used as long as they can display the actual vehicle speed, the target speed set on the vehicle side, the target speed set on the remote control side, and the difference between them. For example, in order to clearly distinguish between the vehicle side indicator 543 and the remote control side indicator 544, an icon or text information for distinguishing between them may be added.

[0076] In this embodiment, the vehicle speed indicator 540 is used to display information related to the vehicle speed, but the operation amount of the accelerator pedal 12 on the vehicle side, the operation amount of the accelerator pedal section 212a on the remote control side, and the allowable error ranges for these operation amounts may also be displayed. Also, the operation amount of the brake pedal 13 on the vehicle side, the operation amount of the brake pedal section 213a on the remote control side, and the allowable error ranges for these operation amounts may also be displayed.

[0077] Fig. 7 is a schematic diagram showing an example of a pedal indicator. The pedal indicator 560 shown in Fig. 7 may be displayed on the monitoring screen 500 in place of the vehicle speed indicator 540 or together with the vehicle speed indicator 540. The pedal indicator 560 includes an accelerator pedal indicator 561 and a brake pedal indicator 562.

[0078] The accelerator pedal indicator 561 includes, for example, a vehicle-side indicator 561A, a remote control-side indicator 561B, and a tolerance range indicator 561C.

[0079] The vehicle-side indicator 561A is an indicator that indicates the operation amount (depression amount) of the accelerator pedal 12 on the vehicle side. The depression amount of the accelerator pedal 12 is measured by the accelerator sensor 112 of the vehicle 1. The control unit 201 of the remote control device 2 acquires information on the depression amount of the accelerator pedal 12 through communication with the vehicle 1. The control unit 201 causes the vehicle-side indicator 561A to display a level according to the depression amount of the accelerator pedal 12.

[0080] The remote control side indicator 561B is an indicator that indicates the operation amount (depression amount) of the accelerator pedal section 212a on the remote control side. Information on the depression amount of the accelerator pedal section 212a is input to the remote control device 2 through the input unit 204. The control unit 201 of the remote control device 2 causes the remote control side indicator 561B to display a level according to the depression amount of the accelerator pedal section 212a.

[0081] The allowable range indicator 561C indicates the allowable range of error between the depression amount of the accelerator pedal 12 on the vehicle side and the depression amount of the accelerator pedal unit 212a on the remote control side. The allowable range of error may be a constant range set in advance, or may be a range that varies according to an index indicating the difficulty of driving control, such as a time lag or vehicle speed.

[0082] In the example of FIG. 7, a range of allowable error is set for the depression amount of the accelerator pedal 12 on the vehicle side. That is, the allowable range indicated by the allowable range indicator 561C is configured to move following the depression amount of the accelerator pedal 12 on the vehicle side. In this embodiment, if the depression amount of the accelerator pedal unit 212a indicated by the remote control side indicator 561B is within the range indicated by the allowable range indicator 561C, switching between autonomous control and remote control is permitted, and if it is not within the range, switching between autonomous control and remote control is prohibited. If the depression amount of the accelerator pedal unit 212a on the remote control side is out of the range indicated by the allowable range indicator 561C, the control unit 201 may display guidance information on the monitoring screen 500 that guides the operation of the accelerator pedal unit 212a so that the depression amount of the accelerator pedal unit 212a falls within the range of the allowable error.

[0083] The same applies to the brake pedal indicator 562. That is, the vehicle-side indicator 562A displays the operation amount (depression amount) of the brake pedal 13 measured by the brake sensor 113, and the remote control-side indicator 562B displays the operation amount (depression amount) of the brake pedal unit 213a input through the input unit 204. In this embodiment, if the depression amount of the brake pedal unit 213a displayed by the remote control-side indicator 562B is within the range indicated by the allowable range indicator 562C, switching between the autonomous control and the remote control is permitted, and if it is not within the range, switching between the autonomous control and the remote control is prohibited. If the depression amount of the brake pedal unit 213a on the remote control side is out of the range indicated by the allowable range indicator 562C, the control unit 201 may display guidance information on the monitoring screen 500 that guides the operation of the brake pedal unit 213a so that the depression amount of the brake pedal unit 213a falls within the range of the allowable error.

[0084] In the example of FIG. 7, tolerance range indicators 561C and 562C that follow the operation on the vehicle side are shown, but a tolerance range indicator that follows the operation on the remote control side may also be used.

[0085] The delay indicator 550 constituting the monitoring screen 500 displays information regarding the amount of delay between the autonomous control and the remote control. FIG. 8 is a schematic diagram showing an example of the delay indicator 550. The delay indicator 550 is, for example, an indicator that displays the amount of delay between the autonomous control and the remote control by an arc-shaped gauge graph. The delay indicator 550 shown in FIG. 8 employs a gauge graph in which the value (amount of delay) increases clockwise from a position corresponding to 0 o'clock. A line 551 shown on the delay indicator 550 indicates the current amount of delay. The example in FIG. 8 indicates that there is a delay of about 3 / 7 of the maximum expected amount of delay (for example, 2 seconds).

[0086] In remote control, the greater the delay amount, the greater the difficulty of control, so the display may be changed in color for each section, for example from cool colors to warm colors, to indicate the increasing difficulty. Also, in remote control, the greater the difficulty of control as the vehicle speed increases, so the product of the delay amount and the vehicle speed may be displayed instead of the delay amount.

[0087] The delay indicator 550 is not limited to an arc-shaped gauge graph, but may be an indicator using a bar graph, a pie chart, or the like, or may be an indicator imitating a level meter.

[0088] The operation of the remote control system will now be described. 9 is a flowchart showing the procedure for measuring the amount of delay. The remote control device 2 executes the following process at appropriate timing, such as regular timing or timing instructed by an operator. The control unit 201 of the remote control device 2 transmits an instruction to project the detection pattern to the vehicle 1 (step S101) and stores the transmission time of the projection instruction in the storage unit 202 (step S102). The control unit 201 can obtain the transmission time by referring to time information output by the built-in clock at the timing of issuing the transmission instruction. The projection instruction is transmitted from the communication unit 203 of the remote control device 2 and arrives at the external communication unit 104 of the in-vehicle control device 100 via the communication network NW.

[0089] The control unit 101 of the on-vehicle control device 100 receives the projection instruction transmitted from the remote control device 2 through the vehicle exterior communication unit 104 (step S103). Based on the received projection instruction, the control unit 101 gives an instruction to the vehicle light ECU 160 through the vehicle interior communication unit 105 to project the detection pattern outside the vehicle (step S104).

[0090] The detection pattern may be a pattern having any shape and color as long as it can be detected using an existing image analysis method such as pattern matching or template matching. The detection pattern to be projected does not have to be one type. A configuration may be adopted in which a plurality of types of patterns are prepared in the on-board control device 100 and projected sequentially. Data on the detection pattern may be stored in the storage unit 102 of the on-board control device 100 that projects the detection pattern and in the storage unit 202 of the remote control device 2 that detects the pattern. The projection location of the detection pattern may be within the field of view of the on-board camera 116, and may be, for example, on the road surface or on a structure around the vehicle. In addition, the number of times the detection pattern is projected is not limited to one, and may be multiple times with an interval.

[0091] The on-vehicle control device 100 captures an image of the outside of the vehicle using the on-vehicle camera 116 (step S105). For convenience of explanation, in this flowchart, the procedure is such that the detection pattern is projected and then the outside of the vehicle is captured by the on-vehicle camera 116, but the outside of the vehicle may be constantly captured by the on-vehicle camera 116 regardless of the processing of each step. Video data obtained by capturing an image of the outside of the vehicle using the on-vehicle camera 116 is input to the control unit 101 through the input unit 103. The control unit 101 encodes the input video data into a format compatible with the transmission format of the communication network NW, and transmits the encoded video data to the remote control device 2 from the outside-vehicle communication unit 104 (step S106).

[0092] The control unit 201 of the remote control device 2 receives the video data transmitted from the in-vehicle control device 100 through the communication unit 203 (step S107). The control unit 201 decodes the received video data and causes the monitor 220 to display a video based on the decoded video data.

[0093] The control unit 201 detects a detection pattern from the captured image (each frame image constituting the video data) of the in-vehicle camera 116 (step S108). The control unit 201 may detect the detection pattern from the captured image of the in-vehicle camera 116 using an existing image analysis method such as pattern matching or template matching.

[0094] When the control unit 201 detects a detection pattern from an image captured by the in-vehicle camera 116, the control unit 201 acquires the detection time by referring to the time information output by the built-in clock. The control unit 201 calculates a delay time from the difference between the detection time of the detection pattern and the transmission time stored in step S102 (step S109). The control unit 201 stores the calculated delay time in the storage unit 202. Furthermore, when the monitoring screen 500 as shown in Fig. 4 is displayed, the control unit 201 generates display data for the delay indicator 550 based on the calculated delay time, and updates the monitoring screen 500 based on the generated display data.

[0095] In this way, the remote control device 2 can measure an approximate amount of system delay by calculating the difference between the time when a command to project the detection pattern is issued and the time when the detection pattern is detected.

[0096] In this embodiment, the amount of delay is estimated from the difference between the time when the projection of the detection pattern is instructed and the time when the detection pattern is detected. However, the control unit 201 may also operate a built-in timer at the timing when the detection pattern is projected, and use the built-in timer to measure the time until the detection pattern is detected.

[0097] 10 is a flowchart for explaining the procedure for switching from autonomous control to remote control. When autonomous control is being executed in the vehicle 1, the control unit 201 of the remote control device 2 executes the following process.

[0098] The control unit 201 monitors the control state of the vehicle 1 by the autonomous control (step S121). Specifically, the control unit 201 acquires various sensing data such as the steering angle detected by the steering angle sensor 111 of the vehicle 1, the accelerator opening detected by the accelerator sensor 112, the brake stroke detected by the brake sensor 113, the vehicle speed detected by the vehicle speed sensor 114, and the video data output from the vehicle-mounted camera 116 through communication with the vehicle-mounted control device 100. The remote control device 2 transmits a transmission request for the sensing data and the video data to the vehicle-mounted control device 100, and receives the sensing data and the video data transmitted from the vehicle-mounted control device 100 in response to the transmission request through the communication unit 203. In addition, the vehicle-mounted control device 100 may transmit the sensing data and the video data to the remote control device 2 on its own initiative. When transmitting these sensing data and video data from the in-vehicle control device 100 to the remote control device 2, the sensing data and video data are encoded in the in-vehicle control device 100 so as to conform to the transmission format of the communication network NW. The encoded sensing data and video data are decoded in the remote control device 2 and used to generate, for example, a monitoring screen 500.

[0099] The control unit 201 acquires information on the amount of operation of the remote controller 210 (step S122). Specifically, the control unit 201 may acquire information on the rotation angle of the handle unit 211a, the amount of depression of the accelerator pedal unit 212a, and the amount of depression of the brake pedal unit 213a through the input unit 204.

[0100] In the flowchart of FIG. 10, the control state of the vehicle 1 is monitored, and then information regarding the amount of operation of the remote controller 210 is obtained. However, these steps may be performed in any order, and may also be performed simultaneously in parallel.

[0101] The control unit 201 generates screen data for the monitoring screen 500 based on the information acquired in steps S121 and S122, and outputs the data from the output unit 205 to the monitor 220, thereby causing the monitor 220 to display the monitoring screen 500 (step S123). Since remote monitoring of the vehicle 1 is being performed at this time, the control unit 201 causes the text information "Remote monitoring in progress" to be displayed in the control status display field 520 of the monitoring screen 500.

[0102] Next, the control unit 201 determines whether or not a switching instruction has been given (step S124). For example, the control unit 201 determines that a switching instruction has been given when a predetermined operation for giving a switching instruction is received. The control unit 201 may also determine that a switching instruction has been given when some operation using the remote controller 210 is received. If a switching instruction has not been given (S124: NO), the control unit 201 returns the process to step S121.

[0103] When it is determined that a switching instruction has been given (S124: YES), the control unit 201 determines whether or not the difference between the control state of the vehicle 1 under the autonomous control and the control state of the vehicle 1 under the remote control is within an allowable range (step S125). In step S125, the control state of the vehicle 1 controlled under the autonomous control may be compared with the control state of the vehicle 1 under the remote control that is assumed when switching from the autonomous control to the remote control.

[0104] For example, the control unit 201 judges whether or not the difference between the rotation angle of the steering wheel 11 in the vehicle 1 and the rotation angle of the steering wheel unit 211a in the remote controller 210 is within the range of the allowable error set for the steering angle (i.e., within the range indicated by the allowable range indicator 535). The control unit 201 also judges whether or not the difference between the target speed set by the autonomous control of the vehicle 1 and the target speed set by the remote control of the vehicle 1 is within the range of the allowable error set for the vehicle speed (i.e., within the range indicated by the allowable range indicator 545). When the control unit 201 judges that both the rotation angle and the target speed are within the allowable error range, it judges that the difference between the two control states is within the allowable range. When the control unit 201 judges that one or both are outside the allowable error range, it judges that the difference between the two control states is outside the allowable range.

[0105] The control unit 201 may compare the difference between the depression amount of the accelerator pedal 12 and the operation amount of the accelerator pedal section 212a, or the difference between the depression amount of the brake pedal 13 and the operation amount of the brake pedal section 213a, to determine whether the difference in the control state between the autonomous control and the remote control is within an acceptable range.

[0106] When it is determined that the difference between the control states of the autonomous control and the remote control is within the allowable range (S125: YES), the control unit 201 executes the remote control of the vehicle 1 (step S126). The control unit 201 transmits an instruction to stop the autonomous control to the vehicle 1, and transmits information necessary for remotely controlling the vehicle 1 to the vehicle 1. As the information necessary for remotely controlling the vehicle 1, the control unit 201 may transmit, for example, information on the steering angle of the vehicle 1 set in response to the operation of the handle unit 211a, information on the target speed set in response to the operation of the accelerator pedal unit 212a and the brake pedal unit 213a, etc. to the vehicle 1. When the in-vehicle control device 100 receives an instruction to stop the autonomous control from the remote control device 2, it stops the autonomous control and controls the traveling of the vehicle 1 based on the steering angle, the target speed, etc. instructed from the remote control device 2. Note that the in-vehicle control device 100 may stop the autonomous control related to the steering control and the speed control of the vehicle 1, and continue other autonomous controls such as the control of the turn signal illumination. When the control mode is changed to remote control, the control unit 201 displays text information stating "remote operation in progress" in the control status display field 520 of the monitoring screen 500.

[0107] When it is determined that the difference between the control states of the autonomous control and the remote control is outside the allowable range (S125: NO), the control unit 201 prohibits the remote control of the vehicle 1 (step S127). In order to prohibit the remote control of the vehicle 1, a lock mechanism (not shown) that locks the operation of the handle unit 211a, the accelerator pedal unit 212a, and the brake pedal unit 213a may be provided. When the control unit 201 determines that the remote control of the vehicle 1 is to be prohibited, it may operate the lock mechanism to lock the operation of the handle unit 211a, the accelerator pedal unit 212a, and the brake pedal unit 213a. In addition, the control unit 201 may prohibit the remote control in a software manner by invalidating the remote control after the remote control is prohibited.

[0108] When the control unit 201 prohibits the remote control of the vehicle 1, the control unit 201 may display text information saying "Remote control prohibited" in the control status display field 520 of the monitoring screen 500. Furthermore, when a speaker is connected to the output unit 205, the control unit 201 may output an alarm sound from the speaker to notify that the remote control is prohibited. Furthermore, when a vibrator (not shown) is built in the handle unit 211a, the control unit 201 may vibrate the vibrator to notify that the remote control is prohibited.

[0109] When the remote control of the vehicle 1 is prohibited, the control unit 201 returns the process to step S125, and executes the remote control at a timing when the difference in the control state between the autonomous control and the remote control falls within an allowable range.

[0110] 11 is a flowchart explaining the procedure for switching from remote control to autonomous control. The control unit 201 executes switching from remote control to autonomous control in the same procedure as the procedure for switching from autonomous control to remote control. The control unit 201 executes the procedure of steps S121 to S125 described above, and determines whether or not the difference between the control state of the vehicle 1 by the autonomous control and the control state of the vehicle 1 by the remote control is within an allowable range.

[0111] When it is determined that the difference between the control states of the autonomous control and the remote control is within an allowable range (S125: YES), the control unit 201 executes the autonomous control of the vehicle 1 (step S128). Specifically, the control unit 201 stops the remote control and transmits an instruction to the vehicle 1 to start the autonomous control. When the in-vehicle control device 100 receives an instruction to start the autonomous control from the remote control device 2, it executes the autonomous control of the vehicle 1 based on various information input via the input unit 103.

[0112] On the other hand, when it is determined that the difference between the control states of the autonomous control and the remote control is outside the allowable range (S125: NO), the control unit 201 prohibits the autonomous control of the vehicle 1 (step S129). When prohibiting the autonomous control of the vehicle 1, the control unit 201 may display text information stating "AUTONOMOUS CONTROL PROHIBITED" in the control status display field 520 of the monitoring screen 500. In addition, the control unit 201 may output an alarm sound or vibrate a vibrator to notify the user that the autonomous control has been prohibited.

[0113] If autonomous control of the vehicle 1 is prohibited, the control unit 201 returns the process to step S125, and when the difference in the control state between the autonomous control and the remote control falls within the allowable range, it instructs the in-vehicle control device 100 to start autonomous control.

[0114] As described above, in this embodiment, when the difference between autonomous control and remote control is relatively large, switching between the two is prohibited, thereby preventing the occurrence of sudden steering, sudden acceleration (or sudden deceleration), sudden braking, sudden starts, etc., that accompany switching between autonomous control and remote control.

[0115] (Embodiment 2) In the second embodiment, a configuration will be described in which a part of remote control is permitted even when the difference between the control state of autonomous control and the control state of remote control is not within an allowable range. The overall configuration of the remote control system, the configuration of the control system of the vehicle 1, and the internal configuration of the remote control device 2 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0116] In remote control, the difficulty of control increases as the vehicle speed increases, so if the vehicle speed can be reduced, the difficulty of control can be reduced. Therefore, even if it is determined that the difference between the control state of the autonomous control and the control state of the remote control is not within the allowable range, remote control that reduces the vehicle speed may be permitted.

[0117] Fig. 12 is a flow chart for explaining the switching procedure in the second embodiment. Fig. 12 shows the switching procedure from autonomous control to remote control. The control unit 201 of the remote control device 2 executes each process of steps S201 to S205 in the same procedure as in the first embodiment, and judges whether or not the difference between the control state of the vehicle 1 by the autonomous control and the control state of the vehicle 1 by the remote control is within an allowable range (step S205). The allowable range is a range that becomes narrower as the running speed (vehicle speed) of the vehicle 1 is higher and becomes wider as the running speed (vehicle speed) of the vehicle 1 is lower.

[0118] When it is determined that the difference between the control states of the autonomous control and the remote control is within an allowable range (S205: YES), the control unit 201 executes remote control of the vehicle 1 (step S206). That is, the control unit 201 transmits an instruction to stop the autonomous control to the vehicle 1, and transmits information required for remotely controlling the vehicle 1 to the vehicle 1. When the in-vehicle control device 100 receives an instruction to stop the autonomous control from the remote control device 2, it stops the autonomous control and controls the traveling of the vehicle 1 based on the steering angle, target speed information, etc., instructed from the remote control device 2. When shifting to remote control, the control unit 201 displays text information saying "Remote driving in progress" in the control status display field 520 of the monitoring screen 500.

[0119] When it is determined that the difference between the control states of the autonomous control and the remote control is outside the allowable range (S205: NO), the control unit 201 allows the remote control to reduce the vehicle speed (step S207). At this time, the control unit 201 displays text information of "speed control possible" in the control status display field 520 of the monitoring screen 500. The remote control to reduce the vehicle speed is executed by an operation of reducing the depression of the accelerator pedal unit 212a or an operation of depressing the brake pedal unit 213a. Conversely, the remote control to increase the vehicle speed and the remote control related to steering that is not allowed in step S207 may be prohibited. For example, the control unit 201 may handle an operation to increase the vehicle speed and an operation to change the steering angle of the vehicle 1 as invalid.

[0120] After allowing the remote control to reduce the vehicle speed, the control unit 201 returns the process to step S205. By reducing the vehicle speed, the tolerance range used in step S205 can be widened. When the difficulty of remote control is reduced and the difference in the control state between the autonomous control and the remote control falls within the tolerance range, the control unit 201 can execute remote control including steering.

[0121] As described above, in this embodiment, if the difference between the control state of autonomous control and the control state of remote control is not within an acceptable range, remote control that reduces the vehicle speed is permitted, so that it becomes possible to switch from autonomous control to remote control once the difficulty of remote control has decreased.

[0122] (Embodiment 3) In the third embodiment, a configuration will be described in which the control ratio is changed stepwise when switching from autonomous control to remote control (or from remote control to autonomous control). The overall configuration of the remote control system, the configuration of the control system of the vehicle 1, and the internal configuration of the remote control device 2 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0123] Fig. 13 is an explanatory diagram explaining the procedure for gradually changing the control ratio. The horizontal axis of the graph shown in Fig. 13 represents time, and the vertical axis represents the remote control ratio. Here, the remote control ratio represents the control ratio between remote control and autonomous control. When autonomous control is being executed in vehicle 1 and remote control is not being executed, this represents a state in which the remote control ratio is 0%. Also, when remote control is being executed in vehicle 1 and autonomous control is not being executed, this represents a state in which the remote control ratio is 100%.

[0124] 13 shows that only autonomous control of the vehicle 1 is executed until time T1, switching from autonomous control to remote control is started at time T1, and switching to remote control is completed at time T2. That is, in this embodiment, a transition time ΔT (=T2-T1) for switching from autonomous control to remote control is provided, and the control ratio is switched stepwise within this transition time ΔT.

[0125] In the example of FIG. 13, it is assumed that the control ratio is changed linearly according to the elapsed time within the transition time ΔT. That is, at time Tx within the transition time ΔT, the control ratio RC(Tx) is given by 100×(Tx-T1) / (T2-T1). For example, if the remote control ratio is 40%, the target steering angle by remote control is 100 degrees, and the target steering angle by autonomous control is 50 degrees, the control unit 201 calculates 100×0.4+50×0.6 (=70 degrees) and sets the final control value of the steering angle to 70 degrees. The control unit 201 changes the control ratio stepwise according to the elapsed time within the transition time ΔT, and executes switching from autonomous control to remote control.

[0126] In the example of Fig. 13, the control ratio is changed linearly according to the elapsed time within the transition time ΔT, but it may be changed nonlinearly or discontinuously (in steps). Also, in the example of Fig. 13, the switching from autonomous control to remote control is described, but the control ratio may be changed stepwise in the same manner when switching from remote control to autonomous control.

[0127] 14 is a flow chart explaining the control procedure in the embodiment 3. When switching from autonomous control to remote control (remote control to autonomous control) starts (step S301), the control unit 201 of the remote control device 2 calculates a control ratio according to the elapsed time within the transition time ΔT (step S302). If the switching start time is T1 and the switching completion time is T2, the control unit 201 can calculate the control ratio from 100×(Tx-T1) / (T2-T1).

[0128] Next, the control unit 201 allocates the control target value in the autonomous control and the control target value by the remote control according to the control ratio calculated in step S302, and calculates a final control value (step S303). The control unit 201 controls the traveling of the vehicle 1 according to the calculated final control value (step S304). That is, the control unit 201 transmits the calculated control value to the in-vehicle control device 100 via the communication unit 203. The in-vehicle control device 100 controls the traveling of the vehicle 1 according to the control value received from the remote control device 2.

[0129] Next, the control unit 201 determines whether the switching is complete by determining whether the transition time ΔT has elapsed (step S305). If the switching is not complete (S305: NO), the control unit 201 returns the process to step S302, determines a control ratio according to the elapsed time within the transition time ΔT, and continues the process of notifying the in-vehicle control device 100 of the control value allocated according to the determined control ratio. If it is determined that the switching is complete (S305: YES), the control unit 201 ends the process according to this flowchart.

[0130] As described above, in embodiment 3, a transition time is provided for switching between autonomous control and remote control, and control is switched gradually within this transition time, making it possible to achieve smooth switching even if there is a difference between the control state in autonomous control and the control in remote control.

[0131] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0132] 1 vehicle 2 Remote control device 11 Handle 12 Accelerator pedal 13. Brake pedal 100 Vehicle control device 101 Control section 102 Storage section 103 Input section 104 External communication unit 105 In-vehicle communication unit 201 Control section 202 Storage section 203 Communications Department 204 Input section 205 Output section 211 Steering Controller 212 Accelerator Controller 213 Brake controller 220 Monitor

Claims

1. On the computer, For a vehicle capable of running under autonomous control and remote control, a control state of the vehicle when the autonomous control is being executed and a control state of the vehicle when switching from the autonomous control to the remote control are identified; determining whether or not to permit switching from the autonomous control to the remote control based on the difference between the two identified control states; Even when it is determined that the switching from the autonomous control to the remote control is not to be performed, the remote control for reducing the traveling speed of the vehicle is permitted to the computer. A computer program for executing a process.

2. The computer includes: acquiring information regarding a steering angle from the vehicle, and identifying a control state by the autonomous control based on the acquired information; and acquiring information about a steering angle from a remote control device that executes the remote control, and identifying a control state by the remote control based on the acquired information.

2. A computer program product as claimed in claim 1 for carrying out a process.

3. The computer includes: acquiring information regarding an accelerator opening from the vehicle, and identifying a control state by the autonomous control based on the acquired information; Acquire information about an accelerator opening from a remote control device that executes the remote control, and identify a control state by the remote control based on the acquired information.

2. A computer program product as claimed in claim 1 for carrying out a process.

4. The computer includes: acquiring information regarding a braking amount from the vehicle, and identifying a control state by the autonomous control based on the acquired information; Obtaining information regarding a brake amount from a remote control device that executes the remote control, and identifying a control state by the remote control based on the obtained information 2. A computer program product as claimed in claim 1 for carrying out a process.

5. The computer includes: Display information about the two control states A computer program product according to any one of claims 1 to 4 for executing a process.

6. The computer includes: Displaying information on a tolerance range for allowing switching between the autonomous control and the remote control with respect to a difference between a control state by the autonomous control and a control state by the remote control A computer program product according to any one of claims 1 to 5 for executing a process.

7. The computer includes: Measure a delay amount related to the remote control; Display the measured delay information A computer program product according to any one of claims 1 to 6 for carrying out a process.

8. The computer includes: Transmitting a projection instruction to the vehicle to project a specific detection pattern within a field of view of an on-board camera of the vehicle; receiving, from the vehicle, an image including the detection pattern captured by the vehicle-mounted camera; Detecting the detection pattern from the received captured image; The amount of delay is measured by measuring the time required from transmitting the projection command to detecting the detection pattern.

8. A computer program product as claimed in claim 7, for carrying out a process.

9. The computer includes: Transmitting a projection instruction to the vehicle to project a specific detection pattern within a field of view of an on-board camera of the vehicle; receiving, from the vehicle, an image including the detection pattern captured by the vehicle-mounted camera; Detecting the detection pattern from the received captured image; The amount of delay related to the remote control is measured by measuring the time required from transmitting the projection command to detecting the detection pattern. A computer program product according to any one of claims 1 to 6 for carrying out a process.

10. The computer includes: A tolerance range for allowing switching between the autonomous control and the remote control is set for a difference between a control state by the autonomous control and a control state by the remote control according to the delay amount or a vehicle speed of the vehicle. A computer program product according to claim 7 or 8 for executing a process.

11. The computer includes: Measure a delay amount related to the remote control; A tolerance range for allowing switching between the autonomous control and the remote control is set for a difference between a control state by the autonomous control and a control state by the remote control according to the delay amount or a vehicle speed of the vehicle. A computer program product according to any one of claims 1 to 6 for carrying out a process.

12. A transition time is provided for switching from the autonomous control to the remote control, A control ratio between the autonomous control and the remote control is changed stepwise according to an elapsed time within the transition time. A computer program product according to any one of claims 1 to 11, for causing a computer to carry out a process.

13. The computer includes: Identifying a control state of the vehicle when the remote control is being executed and a control state of the vehicle when switching from the remote control to the autonomous control; Based on the difference between the two identified control states, it is determined whether or not to permit switching from the remote control to the autonomous control. A computer program product according to any one of claims 1 to 12 for carrying out a process.

14. The computer includes: Displaying information indicating a switching status between the autonomous control and the remote control A computer program product according to any one of claims 1 to 13 for carrying out a process.

15. The computer includes: A warning is output according to the difference between the two control states. A computer program product according to any one of claims 1 to 14 for carrying out a process.

16. an identification unit that identifies, for a vehicle capable of running under autonomous control and remote control, a control state of the vehicle when the autonomous control is being executed and a control state of the vehicle when switching from the autonomous control to the remote control; a determination unit that determines whether or not to permit switching from the autonomous control to the remote control based on a difference between the two identified control states; a control execution unit that executes the remote control instead of the autonomous control when it is determined that the switching is permitted; Equipped with Even if the determination unit determines that the switching should not be performed, the control execution unit is permitted to perform remote control for reducing the traveling speed of the vehicle. Remote control device.

17. The computer For a vehicle capable of running under autonomous control and remote control, a control state of the vehicle when the autonomous control is being executed and a control state of the vehicle when switching from the autonomous control to the remote control are identified; determining whether or not to permit switching from the autonomous control to the remote control based on the difference between the two identified control states; When it is determined that the switching is permitted, the remote control is executed instead of the autonomous control; Even if it is determined that the switching should not be performed, the computer is permitted to execute remote control to reduce the traveling speed of the vehicle. Remote control method.

Images